Creating Elastic Search Cluster (EC2, cloud-aws Plugin)

Reading Time : ~ .

While handling Large amounts of data with elastic search, you may run out of server capacity or compute power, Forming an Elasticsearch cluster will reduce the load distribution. In Single, ElasticSearch Instance endpoint acts as Data Node + Master Node + Client Node.

Master Node:
The master node is responsible for actions such as creating or deleting an index, tracking actions, and deciding which shares to allocate to which nodes.

Data Node:
Data nodes hold the shards that contain the documents you have indexed. Data nodes handle data related operations like CRUD, search, and aggregations. These operations are I/O, memory, and CPU-intensive.

Client Node:
client node that can only route requests, handle the search reduce phase, and distribute bulk indexing.

Dedicated Nodes in a cluster has better performance. Let's look into how to set those up.

Installing ElasticSearch

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-2.3.2.deb
sudo dpkg -i elasticsearch-2.3.2.deb
sudo update-rc.d elasticsearch defaults 95 10

Installing cloud-aws plugin:

cd /usr/share/elasticsearch (ES_HOME Directory)
bin/plugin install cloud-aws
service elasticsearch restart

Configuring Nodes:

Create iam user who has privileges on all the cluster ec2 instances.(you can create a vpc setup all clusters there and give privileges over that vpc to the iam user).

Avoid default cluster name elastic search because in cases, like you need to have two clusters in the same VPS, the nodes of one cluster will mix with other. Since nodes with same cluster name form a cluster, choose apt cluster name to uniquely bind a node.

Master Node:

cluster.name: <cluster name>
node.name: <node name> #(optional)
node.master: true
node.data: true
cloud:
  aws:
    access_key: < your access key here > 
    secret_key: < your secret key here > 
region: us-east-1
discovery:
  type: ec2
availability_zones : us-east-1

Data Node:

cluster.name: <cluster name>
node.name: <node name> # (optional)
node.master: false 
node.data: true 
cloud: 
  aws: 
    access_key: < your access key here > 
    secret_key: < your secret key here > 
region: us-east-1 
discovery: 
  type: ec2 
availability_zones : us-east-1 

Client Node: 

cluster.name: <cluster name> 
node.name: <node name>    # (optional)
node.master: false
node.data: false
cloud:
  aws:
    access_key: < your access key here > 
    secret_key: < your secret key here > 
region: us-east-1
discovery:
    type: ec2
availability_zones : us-east-1

Restarting elastic search and curl localhost:9200/_nodes/process?pretty will list all the nodes that have been configured.

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
How to Mount S3 Bucket on Local Disk Dinesh Deshmukh

It all starts with FUSE, FUSE is File System User Space. Operating Systems have Kernel Space and User Space. Kernel Space is where low level ...

Continue Reading...
Configuring and Testing Load Balancer in AWS EC2 Jagadeesh V

When You have an application that is serving Huge Customer Base, so will be your Traffic. Sometimes The Application simply stops responding. We can use ...

Continue Reading...
Easy and Fast way to implement AWS Lambda service Dinesh Deshmukh

We are going to use a simple application called Gordan to prevent creating a lambda function and triggering actions which involves time taking and repetitive ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties