1

Install elasticsearch using enrollment tokens

Posted on by jlim0930

I have not installed elasticsearch using rpm/tarball in a super long time since I use my deploy-elastic.sh script to install elasticsearch onto docker containers. Decided to have a look today on the new way of standing up a cluster using enrollment tokens.

Following the steps from https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html - a bit adapated for my liking.

Install elasticsearch onto first host:

  • import GPG keys
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
  • create /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
  • install elasticsearch via yum
# yum install elasticsearch -y
...
Dependencies Resolved

==================================================================================================================================================================================================================
 Package                                               Arch                                           Version                                         Repository                                             Size
==================================================================================================================================================================================================================
Installing:
 elasticsearch                                         x86_64                                         8.3.3-1                                         elasticsearch                                         514 M

Transaction Summary
==================================================================================================================================================================================================================
Install  1 Package

Total download size: 514 M
Installed size: 1.1 G
Downloading packages:
elasticsearch-8.3.3-x86_64.rpm                                                                                                                                                             | 514 MB  00:00:15
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
  Installing : elasticsearch-8.3.3-1.x86_64                                                                                                                                                                   1/1
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : Vroka=umh7EndSDriQ*6

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
  Verifying  : elasticsearch-8.3.3-1.x86_64                                                                                                                                                                   1/1

Installed:
  elasticsearch.x86_64 0:8.3.3-1

Complete!

Things look a little different from 7.x days...

  • start elasticsearch
# systemctl start elasticsearch
# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-08-09 16:17:16 UTC; 47s ago
     Docs: https://www.elastic.co
 Main PID: 9058 (java)
    Tasks: 122
   Memory: 15.7G
   CGroup: /system.slice/elasticsearch.service
           ├─9058 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.hom...
           ├─9126 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless...
           └─9157 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 09 16:17:00 sl-justinlim-e5459f-host1 systemd[1]: Starting Elasticsearch...
Aug 09 16:17:16 sl-justinlim-e5459f-host1 systemd[1]: Started Elasticsearch.
  • verify elasticsearch
# curl -k -u "elastic:Vroka=umh7EndSDriQ*6" "https://localhost:9200"
{
  "name" : "sl-justinlim-e5459f-host1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "FaTxWQRtRD-P5OlQrPWX2Q",
  "version" : {
    "number" : "8.3.3",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "801fed82df74dbe537f89b71b098ccaff88d2c56",
    "build_date" : "2022-07-23T19:30:09.227964828Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

# curl -k -u "elastic:Vroka=umh7EndSDriQ*6" "https://localhost:9200/_cat/nodes?v"
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
127.0.0.1            1          60   4    0.31    0.36     0.21 cdfhilmrstw *      sl-justinlim-e5459f-host1

# curl --cacert /etc/elasticsearch/certs/http_ca.crt  -u "elastic:Vroka=umh7EndSDriQ*6" "https://localhost:9200/_cat/nodes?v"
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.16.0.15            2          61   0    0.00    0.00     0.04 cdfhilmrstw *      sl-justinlim-e5459f-host1

Looks good! - the new install method creates certificates and sets the password for the elastic user and secures the cluster by default and generates certificates!

Lets take a look at the settings

# cat /etc/elasticsearch/elasticsearch.yml | grep -v ^#
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["sl-justinlim-e5459f-host1"]

http.host: 0.0.0.0

#transport.host: 0.0.0.0

So far so good. Checking for network ports 9200 (http) and 9300 (transport)

# netstat -an | grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN

# netstat -an | grep 9300
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN
tcp6       0      0 ::1:9300                :::*                    LISTEN

hmm.. to add additional nodes you need the transport(9300) opened and listening so that other nodes can connect and communicate with it.

I edited /etc/elasticsearch/elasticsearch.yml and uncommented transport.host: 0.0.0.0 and restarted elasticsearch and now we can see that the transport port is listening

# netstat -an | grep 9300
tcp6       0      0 :::9300                 :::*                    LISTEN
tcp6       0      0 ::1:59984               ::1:9300                TIME_WAIT
tcp6       0      0 127.0.0.1:44372         127.0.0.1:9300          TIME_WAIT

THIS IS VERY IMPORTANT to do if you want to add nodes, and is not documented on the docs

  • Lets create the enrollment token
# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
eyJ2ZXIiOiI4LjMuMyIsImFkciI6WyIxNzIuMTYuMC4xNTo5MjAwIl0sImZnciI6IjAwYzQ5OWMxMmNiNDZhOTNlZmY2OWVkZDMwZTk4NTUyMjE4YTNmZjQ2MDg2YTg5ZDRiOTkzYWU1MzJkYmYzYzciLCJrZXkiOiIzNmxvZzRJQlVwSzhhSnMyc0hpYzpGU3IzaGlUZ1I2aWVLR1RqMWdPaDJRIn0=
  • Install elasticsearch on the 2nd node
# yum install elasticsearch -y
....
  • enroll the node
# /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjMuMyIsImFkciI6WyIxNzIuMTYuMC4xNTo5MjAwIl0sImZnciI6IjAwYzQ5OWMxMmNiNDZhOTNlZmY2OWVkZDMwZTk4NTUyMjE4YTNmZjQ2MDg2YTg5ZDRiOTkzYWU1MzJkYmYzYzciLCJrZXkiOiIzNmxvZzRJQlVwSzhhSnMyc0hpYzpGU3IzaGlUZ1I2aWVLR1RqMWdPaDJRIn0=

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically:
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y

Now before starting elasticsearch on the 2nd node I want to confirm that transport.host is open. Look in /etc/elasticsearch/elasticsearch.yml to ensure that transport.host is not commented out.

# cat /etc/elasticsearch/elasticsearch.yml  | grep -v ^#
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
discovery.seed_hosts: ["172.16.0.15:9300"]

http.host: 0.0.0.0

transport.host: 0.0.0.0
  • start elasticsearch
# systemctl start elasticsearch
# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-08-09 16:23:37 UTC; 10s ago
     Docs: https://www.elastic.co
 Main PID: 9883 (java)
    Tasks: 90
   Memory: 15.7G
   CGroup: /system.slice/elasticsearch.service
           ├─9883 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.hom...
           ├─9951 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless...
           └─9982 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 09 16:23:20 sl-justinlim-e5459f-host2 systemd[1]: Starting Elasticsearch...
Aug 09 16:23:37 sl-justinlim-e5459f-host2 systemd[1]: Started Elasticsearch.
  • verify to ensure that 2nd node joined
# curl -k -u "elastic:Vroka=umh7EndSDriQ*6" "https://localhost:9200/_cat/nodes?v"
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.16.0.78            3          60   0    0.81    0.41     0.26 cdfhilmrstw -      sl-justinlim-e5459f-host2
172.16.0.15            4          61   0    0.05    0.32     0.25 cdfhilmrstw *      sl-justinlim-e5459f-host1
  • Now rinse and repeat for other nodes.

I did test this a bit further by not uncommenting transport.host on the first node and enrolling 2ndary nodes and if you do that the 2ndary nodes will also have transport.host commeneted out so the cluster will never form. Please ensure that transport.host is uncommneted. You can use 0.0.0.0 which will bind to all interfaces or you can use specific IP addresses.

jlim0930

More Posts

One Comment

  1. Oh, thank you so much for “elasticsearch-reconfigure-node –enrollment-token” notice! What a relief! ES security is diving me nuts.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.