OpenSearch: Up and Running in 10 minutes — On local or Amazon EKS

Say Hello to OpenSearch Release Candidate 1

Step by Step guide to run OpenSearch “Release Candidate 1' on local and on Amazon Elastic Kubernetes Service (EKS)

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, OpenSearch, and a visualization and user interface, OpenSearch Dashboards. OpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more.

https://opensearch.org/

Now let’s get started and set up OpenSearch on your local machine.

OpenSearch team publishes all docker containers on docker hub, that makes it super simple to run it anywhere.

Install Docker from“here” and Verify the installation using

docker --version
Docker version 19.03.2, build 6a30dfc

Now OpenSearch has two main components. First opensearch itself the engine and second open search dashboards. Just like Elasticsearch and Kibana. To run both together and have them talk to each other we will use docker compose.

To created docker compose file follow this

mkdir opensearch; cd ./opensearch

Create docker compose file, below file will start 1 opensearch container and 1 opensearch-dashboard container

cat << EOF > docker-compose.yaml 
---
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:1.0.0-rc1
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1
- cluster.initial_master_nodes=opensearch-node1
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:1.0.0-rc1
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: https://opensearch-node1:9200
networks:
- opensearch-net

volumes:
opensearch-data1:

networks:
opensearch-net:
EOF

Lets fire it up

docker-compose up

Thats it.. if you started web server like million times like me look for this in rolling logs

opensearch-dashboards | {“type”:”log”,”@timestamp”:”2021–06–13T00:01:47Z”,”tags”:[“listening”,”info”],”pid”:1,”message”:”Server running at http://0:5601"}

Lets look at the Dashboards

On your browser open

http://localhost:5601/
Username: admin
Password: admin

voilà

Thats it. You have OpenSearch Running on your local machine with OpenSearch Dashboard

Lets Add some Data: Click on Add data.

Lets add Sample Web Logs to simulate log aggregation use case

View Data

For details logs ust like in Kibana, click on discover

And there you go all your logs are here (from Sample Data)

Lets get this going on EKS

To create EKS Cluster I am using eksctl from friends at weaveworks https://eksctl.io/

cat << EOF > ekscluster.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ekscluster-eksctl
region: us-east-1
version: "1.17"
availabilityZones: ["us-east-1a", "us-east-1b", "us-east-1c"]managedNodeGroups:
- name: nodegroup
desiredCapacity: 3
instanceType: t3.small
ssh:
enableSsm: true
secretsEncryption:
keyARN: "<<Your Role ARN>>"
EOF

Download eksctl from

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp  sudo mv -v /tmp/eksctl /usr/local/bin

Create Cluster

eksctl create cluster -f eksworkshop.yaml

Once EKS cluster is created, create namespace

kubectl create ns opensearch

Deploy opensearch and opensearch dashboard to EKS

cat << EOF > opensearch-on-eks.yaml
apiVersion: v1
kind: Service
metadata:
labels:
service: opensearch-dashboards
name: opensearch-dashboards
spec:
ports:
- name: "5601"
port: 80
targetPort: 5601
selector:
service: opensearch-dashboards
type: LoadBalancer
status:
loadBalancer: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
service: opensearch-node1
name: opensearch-node1
spec:
ports:
- name: "9200"
port: 9200
targetPort: 9200
- name: "9600"
port: 9600
targetPort: 9600
selector:
service: opensearch-node1
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
service: opensearch-dashboards
name: opensearch-dashboards
spec:
replicas: 1
selector:
matchLabels:
service: opensearch-dashboards
strategy: {}
template:
metadata:
labels:
network/opensearch-net: "true"
service: opensearch-dashboards
spec:
containers:
- env:
- name: OPENSEARCH_HOSTS
value: https://opensearch-node1:9200
image: opensearchproject/opensearch-dashboards:1.0.0-rc1
name: opensearch-dashboards
ports:
- containerPort: 5601
resources: {}
restartPolicy: Always
status: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
service: opensearch-node1
name: opensearch-node1
spec:
replicas: 1
selector:
matchLabels:
service: opensearch-node1
strategy:
type: Recreate
template:
metadata:
labels:
network/opensearch-net: "true"
service: opensearch-node1
spec:
containers:
- env:
- name: OPENSEARCH_JAVA_OPTS
value: -Xms512m -Xmx512m
- name: bootstrap.memory_lock
value: "true"
- name: cluster.initial_master_nodes
value: opensearch-node1
- name: cluster.name
value: opensearch-cluster
- name: discovery.seed_hosts
value: opensearch-node1
- name: node.name
value: opensearch-node1
image: opensearchproject/opensearch:1.0.0-rc1
name: opensearch-node1
ports:
- containerPort: 9200
- containerPort: 9600
resources: {}
restartPolicy: Always
status: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
creationTimestamp: null
name: opensearch-net
spec:
ingress:
- from:
- podSelector:
matchLabels:
network/opensearch-net: "true"
podSelector:
matchLabels:
network/opensearch-net: "true"
---
EOF

Deploy on Kubernetes

kubectl apply -f opensearch-on-eks.yaml -n opensearch

Check status of deployment.

kubectl get all -n opensearch
NAME READY STATUS RESTARTS AGE
pod/opensearch-dashboards-77dbd99765-m95rl 1/1 Running 0 8m13s
pod/opensearch-node1-6cc679ffdf-d88dq 1/1 Running 0 8m12s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/opensearch-dashboards LoadBalancer 10.100.199.160 a43ad4374f7b2449496b364ec2f2d254-540229199.us-east-1.elb.amazonaws.com 80:32317/TCP 8m13s
service/opensearch-node1 ClusterIP 10.100.83.173 <none> 9200/TCP,9600/TCP 8m13s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/opensearch-dashboards 1/1 1 1 8m13s
deployment.apps/opensearch-node1 1/1 1 1 8m12s
NAME DESIRED CURRENT READY AGE
replicaset.apps/opensearch-dashboards-77dbd99765 1 1 1 8m13s
replicaset.apps/opensearch-node1-6cc679ffdf 1 1 1 8m12s

Now lets get the endpoint for Dashboard

kubectl get service opensearch-dashboards -n opensearchNAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)        AGE
opensearch-dashboards LoadBalancer 10.100.199.160 a43ad4374f7b2449496b364ec2f2d254-540229199.us-east-1.elb.amazonaws.com 80:32317/TCP 10m

Open your browser and go to external IP url:

voilà

Code used in this blog is available at https://github.com/amulyas/opensearch

More Configuration options and details for OpenSearch at https://docs-beta.opensearch.org/

More information on creating EKS Cluster : https://www.eksworkshop.com/

More Information on eksctl: https://eksctl.io/

OpenSearch: https://aws.amazon.com/blogs/opensource/introducing-opensearch/

--

--

--

Cloud, Containers, SaaS, PaaS, Automation, Infrastructure, Dad, Husband, Sillicon Valley

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Immutable Infrastructure Using Packer, Ansible, and Terraform

Head First Java (Chapter 02) A Trip to Objectville

Aptos Tutorial Episode 4: Let’s Table This For Now (Part 1)

How to crop transparent areas of image?

Handling secrets in a k6 project

How to Ace a Software Engineering Interview at Vestmark

Technical Intuition

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Amulya Sharma

Amulya Sharma

Cloud, Containers, SaaS, PaaS, Automation, Infrastructure, Dad, Husband, Sillicon Valley

More from Medium

Upgrade Aurora PostgreSQL with Minimal Downtime using AWS DMS

Deployer — secure method of auto publishing new versions of applications

Bottlerocket: Operating System to Run Containers

Configure notification for new/updates available for AWS EKS Add-ons