Open in app

Sign in

Write

Sign in

OpenSearch: Up and Running in 10 minutes — On local or Amazon EKS

Amulya Sharma
5 min readJun 13, 2021

--

Say Hello to OpenSearch Release Candidate 1

Step by Step guide to run OpenSearch “Release Candidate 1' on local and on Amazon Elastic Kubernetes Service (EKS)

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, OpenSearch, and a visualization and user interface, OpenSearch Dashboards. OpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more.

— https://opensearch.org/

Now let’s get started and set up OpenSearch on your local machine.

OpenSearch team publishes all docker containers on docker hub, that makes it super simple to run it anywhere.

Install Docker from“here” and Verify the installation using

docker --version
Docker version 19.03.2, build 6a30dfc

Now OpenSearch has two main components. First opensearch itself the engine and second open search dashboards. Just like Elasticsearch and Kibana. To run both together and have them talk to each other we will use docker compose.

To created docker compose file follow this

mkdir opensearch; cd ./opensearch

Create docker compose file, below file will start 1 opensearch container and 1 opensearch-dashboard container

cat << EOF > docker-compose.yaml 
---
version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:1.0.0-rc1
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1
      - cluster.initial_master_nodes=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:1.0.0-rc1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: https://opensearch-node1:9200
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:
EOF

Lets fire it up

docker-compose up

Thats it.. if you started web server like million times like me look for this in rolling logs

opensearch-dashboards | {“type”:”log”,”@timestamp”:”2021–06–13T00:01:47Z”,”tags”:[“listening”,”info”],”pid”:1,”message”:”Server running at http://0:5601"}

Lets look at the Dashboards

On your browser open

http://localhost:5601/
Username: admin
Password: admin

voilà

Thats it. You have OpenSearch Running on your local machine with OpenSearch Dashboard

Lets Add some Data: Click on Add data.

Lets add Sample Web Logs to simulate log aggregation use case

View Data

For details logs ust like in Kibana, click on discover

And there you go all your logs are here (from Sample Data)

Lets get this going on EKS

To create EKS Cluster I am using eksctl from friends at weaveworks https://eksctl.io/

cat << EOF > ekscluster.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfigmetadata:
  name: ekscluster-eksctl
  region: us-east-1
  version: "1.17"availabilityZones: ["us-east-1a", "us-east-1b", "us-east-1c"]managedNodeGroups:
- name: nodegroup
  desiredCapacity: 3
  instanceType: t3.small
  ssh:
    enableSsm: truesecretsEncryption:
  keyARN: "<<Your Role ARN>>"
EOF

Download eksctl from

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp  sudo mv -v /tmp/eksctl /usr/local/bin

Create Cluster

eksctl create cluster -f eksworkshop.yaml

Once EKS cluster is created, create namespace

kubectl create ns opensearch

Deploy opensearch and opensearch dashboard to EKS

cat << EOF > opensearch-on-eks.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    service: opensearch-dashboards
  name: opensearch-dashboards
spec:
  ports:
  - name: "5601"
    port: 80
    targetPort: 5601
  selector:
    service: opensearch-dashboards
  type: LoadBalancer
status:
  loadBalancer: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: opensearch-node1
  name: opensearch-node1
spec:
  ports:
  - name: "9200"
    port: 9200
    targetPort: 9200
  - name: "9600"
    port: 9600
    targetPort: 9600
  selector:
    service: opensearch-node1
status:
  loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: opensearch-dashboards
  name: opensearch-dashboards
spec:
  replicas: 1
  selector:
    matchLabels:
      service: opensearch-dashboards
  strategy: {}
  template:
    metadata:
      labels:
        network/opensearch-net: "true"
        service: opensearch-dashboards
    spec:
      containers:
      - env:
        - name: OPENSEARCH_HOSTS
          value: https://opensearch-node1:9200
        image: opensearchproject/opensearch-dashboards:1.0.0-rc1
        name: opensearch-dashboards
        ports:
        - containerPort: 5601
        resources: {}
      restartPolicy: Always
status: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: opensearch-node1
  name: opensearch-node1
spec:
  replicas: 1
  selector:
    matchLabels:
      service: opensearch-node1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        network/opensearch-net: "true"
        service: opensearch-node1
    spec:
      containers:
      - env:
        - name: OPENSEARCH_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: bootstrap.memory_lock
          value: "true"
        - name: cluster.initial_master_nodes
          value: opensearch-node1
        - name: cluster.name
          value: opensearch-cluster
        - name: discovery.seed_hosts
          value: opensearch-node1
        - name: node.name
          value: opensearch-node1
        image: opensearchproject/opensearch:1.0.0-rc1
        name: opensearch-node1
        ports:
        - containerPort: 9200
        - containerPort: 9600
        resources: {}
      restartPolicy: Always
status: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  creationTimestamp: null
  name: opensearch-net
spec:
  ingress:
  - from:
    - podSelector:
        matchLabels:
          network/opensearch-net: "true"
  podSelector:
    matchLabels:
      network/opensearch-net: "true"
---
EOF

Deploy on Kubernetes

kubectl apply -f opensearch-on-eks.yaml -n opensearch

Check status of deployment.

kubectl get all -n opensearch
NAME                                         READY   STATUS    RESTARTS   AGE
pod/opensearch-dashboards-77dbd99765-m95rl   1/1     Running   0          8m13s
pod/opensearch-node1-6cc679ffdf-d88dq        1/1     Running   0          8m12sNAME                            TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)             AGE
service/opensearch-dashboards   LoadBalancer   10.100.199.160   a43ad4374f7b2449496b364ec2f2d254-540229199.us-east-1.elb.amazonaws.com   80:32317/TCP        8m13s
service/opensearch-node1        ClusterIP      10.100.83.173    <none>                                                                   9200/TCP,9600/TCP   8m13sNAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/opensearch-dashboards   1/1     1            1           8m13s
deployment.apps/opensearch-node1        1/1     1            1           8m12sNAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/opensearch-dashboards-77dbd99765   1         1         1       8m13s
replicaset.apps/opensearch-node1-6cc679ffdf        1         1         1       8m12s

Now lets get the endpoint for Dashboard

kubectl get service opensearch-dashboards -n opensearchNAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)        AGE
opensearch-dashboards   LoadBalancer   10.100.199.160   a43ad4374f7b2449496b364ec2f2d254-540229199.us-east-1.elb.amazonaws.com   80:32317/TCP   10m

Open your browser and go to external IP url:

voilà

Code used in this blog is available at https://github.com/amulyas/opensearch

More Configuration options and details for OpenSearch at https://docs-beta.opensearch.org/

More information on creating EKS Cluster : https://www.eksworkshop.com/

More Information on eksctl: https://eksctl.io/

OpenSearch: https://aws.amazon.com/blogs/opensource/introducing-opensearch/

Opensearch
Elasticsearch
AWS
Open Source

--

--

Amulya Sharma
Amulya Sharma

Written by Amulya Sharma

29 Followers
90 Following

Cloud, Containers, SaaS, PaaS, Automation, Infrastructure, Dad, Husband, Sillicon Valley

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams