PLEASE use THIS POST instead, as many items were updated and changed.

Configuring snapshots for ECE(Elasticsearch Cloud Enterprise) or deployments for ECE from a S3 object storage or on a s3 compliant storage with publicly trusted certificates are easy but how do you configure the snapshot repository if you are using self-signed or internally signed certificates ?

There are 2 ways of configuring snapshot repository and snapshots on ECE.

First you can configure the snapshot repository on ECE. Once you configure the snapshot repository in the Admin UI, you can configure found-snapshots for each deployment via the Admin UI. Unfortunately, for now unless your endpoint is configured with a publicly trusted certificate or you are using a non TLS endpoint it is not possible to configure the repository in the ECE Admin UI. There are some "hacks" that uses apache/nginx to provide http endpoint for your https none public trusted S3 endpoint but we will not cover it here.

Second method is to configure a snapshot repository per deployment. Again if your s3 endpoint is not TLS or is signed by a publicly trusted CA then the steps are easy however if it is using a self-signed certificate or internal certificate then you would need to create a custom JVM trust store bundle and add it to the deployment so that the certificate can be trusted.

Part 1 - I will configure a minio server with SSL encrypted endpoint, configure s3cmd(mc alternative - since I am more used to s3cmd than mc) to work with my minio server. I will create a custom JVM trust store bundle and host the custom bundle on a web server.

Part 2 - I will configure my deployment to use the custom bundle and configure the snapshot repository

Lets get started

My ECE environment is a 3 node cluster where all the nodes have all roles. I will install the minio server and the httpd server onto the 3rd node. My hosts are running CentOS.

Part 1

Configure minio server

Lets export some variables into our environment

export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=minio123
export MINIO_BUCKET_NAME=elastic

Lets install docker-compose and some other needed tools

sudo yum install docker-compose unzip zip wget -y

Create a /tmp/minio directory - if you have limited /tmp you can create the directory anywhere just make sure to update the path as you go

mkdir /tmp/minio
cd /tmp/minio

Create the docker-compose.yml in /tmp/minio/ - we are running minio on port 9001 since port 9000 is taken already by ECE.

version: "2"
services:
  minio:
    image: minio/minio:latest
    ports:
      - "9001:9000"
    volumes:
      - ./data/minio/data:/export
      - ./data/minio/config:/root/.minio
    environment:
      - "MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY"
      - "MINIO_SECRET_KEY=$MINIO_SECRET_KEY"
    command: server /export
  createbuckets:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add myminio http://minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc mb myminio/$MINIO_BUCKET_NAME;
      /usr/bin/mc policy download myminio/$MINIO_BUCKET_NAME;
      exit 0;
      "

Start the container - from /tmp/minio

$ docker-compose up -d
Creating network "a_default" with the default driver
Pulling minio (minio/minio:latest)...
latest: Pulling from minio/minio
d46336f50433: Pull complete
be961ec68663: Pull complete
2d8918585761: Pull complete
234a02bcc461: Pull complete
40c7306e8d99: Pull complete
296dea563960: Pull complete
fae88b72ea9f: Pull complete
Digest: sha256:3755b00abde3ec763272d3fed1f4f4130bcbd56bc1fddc6e134249413d267fd8
Status: Downloaded newer image for minio/minio:latest
Pulling createbuckets (minio/mc:latest)...
latest: Pulling from minio/mc
d46336f50433: Already exists
be961ec68663: Already exists
929afd096525: Pull complete
d9ca8ede5805: Pull complete
8643f6c7725d: Pull complete
Digest: sha256:c3943bffa32e18a463a9283e053a9fcde66dd02ad1017224b01f632d2c3efb78
Creating minio_minio_1         ... done
Creating minio_minio_1         ...
Creating a_createbuckets_1 ... done
$ docker ps | grep minio
a4789174b406        minio/mc                                                            "/bin/sh -c ' until …"   6 seconds ago       Up 5 seconds                                                                                                                                                                                 a_createbuckets_1
98ce8a201610        minio/minio:latest                                                  "/usr/bin/docker-ent…"   6 seconds ago       Up 5 seconds        0.0.0.0:9001->9000/tcp                                                                                                                                                   minio_minio_1

Configure s3cmd to use with my minio server

Install & configure s3cmd

sudo yum install -y s3cmd
vi ~/.s3cfg

Fill with the following

host_base = localhost:9001
host_bucket = localhost:9001
use_https = False
access_key = minio
secret_key = minio123
signature_v2 = False

Lets take s3cmd for a spin

$ s3cmd ls
2021-12-06 20:44  s3://elastic
$ s3cmd ls s3://elastic
$

We can see that our bucket was already created and the contents of the bucket is empty

Configure TLS for the minio server

We will use the certutil to create the CA and CA signed certificate and apply it to our minio server. In a separate directory. For the IP I am using my host's internal IP address(GCP environment with an internal and external IP address).

$ cd /tmp
$ mkdir certs
$ chmod 777 /tmp/certs
$ cd /tmp/certs
$ export IMG=$(docker images docker.elastic.co/cloud-assets/elasticsearch --format "{{.Repository}}:{{.Tag}}"| head -n1)
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} /usr/share/elasticsearch/bin/elasticsearch-certutil  -s ca --pem --out /tmp/certs/ca.zip
$ sudo chmod 777 ca.zip
$ unzip ca.zip
$ ls -lah ca
total 8.0K
drwxrwxr-x. 2 jlim jlim   34 Dec  7 02:59 .
drwxrwxrwx. 4 jlim jlim   62 Dec  7 03:09 ..
-rw-rw-r--. 1 jlim jlim 1.2K Dec  7 02:59 ca.crt
-rw-rw-r--. 1 jlim jlim 1.7K Dec  7 02:59 ca.key
$ export DOMAIN=ece.local
$ export IP=172.16.0.61
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} /usr/share/elasticsearch/bin/elasticsearch-certutil cert -s --pem --name server --dns "localhost,${DOMAIN},*.${DOMAIN}" --ip "127.0.0.1,${IP}" --out /tmp/certs/server.zip --ca-cert /tmp/certs/ca/ca.crt --ca-key /tmp/certs/ca/ca.key
$ sudo chmod 777 server.zip
$ unzip server.zip
$ ls -lah server
total 8.0K
drwxrwxr-x. 2 jlim jlim   42 Dec  7 03:26 .
drwxrwxrwx. 4 jlim jlim   62 Dec  7 03:26 ..
-rw-rw-r--. 1 jlim jlim 1.3K Dec  7 03:26 server.crt
-rw-rw-r--. 1 jlim jlim 1.7K Dec  7 03:26 server.key

Lets copy server.crt to /tmp/minio/data/minio/config/certs/public.crt & copy server.key to /tmp/minio/data/minio/config/certs/private.key

Take down minio to re-configure it

$ cd /tmp/minio
$ docker-compose down

Edit docker-compose.yml line command: server /export to command: server /export --certs-dir /root/.minio/certs/

restart minio

$ cd /tmp/minio
$ docker-compose up -d

Now lets reconfigure our s3cmd so that we can access the minio server. Edit .s3cfg to

host_base = localhost:9001
host_bucket = localhost:9001
use_https = True
access_key = minio
secret_key = minio123
signature_v2 = False
ca_certs_file=/tmp/certs/ca/ca.crt

and now your s3cmd should work again.

Create a custom JVM trust store

The instructions to create a custom JVM trust store is located here

$ openssl s_client -connect localhost:9001 -showcerts
CONNECTED(00000003)
depth=0 CN = server
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=server
   i:/CN=Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDXDCCAkSgAwIBAgIUMCsBYVtbOd7oyxvHsGJBJ3yeYyMwDQYJKoZIhvcNAQEL
BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
cmF0ZWQgQ0EwHhcNMjExMjA3MDMwODQ5WhcNMjQxMjA2MDMwODQ5WjARMQ8wDQYD
VQQDEwZzZXJ2ZXIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCC9Tzv
lKUeT/YeEzxQJjksYGRyqA39hkQoOLm8IHEg3eTctnKsPy/Du2wraRmrxoSTooYJ
mt3/a/9Z8+bmGoS9Q/nckd/jxvT4cTgP0yqrbjRGNaEu+9IdHmzpcxtnV57lrX9l
pJ5dxIHXKVnZSK960leYLYII8dzVDibRGvgUmwfUFMiPG+V2VtEs1RsVWcgEHMzn
Rip+kaLFsSPuEDHjgj6VzTfQjuyCHb6CpDZjDwDGiXR5GucQSBy36sxLrj3xASnj
xZmDFxCZoiPMCFgYcI8ehKonAgMC+oCLEr80iCeQ2W8LyzHXa/fIpCiGRsmxFvFQ
Rqlod35OoIPOTk7FAgMBAAGjgYgwgYUwHQYDVR0OBBYEFDIyiLI+6w3XNy1vUeg8
O0W/Xj3HMB8GA1UdIwQYMBaAFJjHjVISVMTwK/t9n9/becaxJZILMDgGA1UdEQQx
MC+CCWVjZS5sb2NhbIILKi5lY2UubG9jYWyCCWxvY2FsaG9zdIcEfwAAAYcErAAA
ATAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQA5f4BPg1VjdYaQDir46B6+
1h2e7qPMlecCsiI/LgC6fDx9A2VDYDCs36tPKxjvW1N3kHhO3zBap9ANtQCrXTIS
2LiyKihR2Ok/4UNzVlrpBT//Axuf2ZhvSF57w3CpzKiWGfZjSw13byddbiZ1AzmE
7kmQCrZXB23IN/bCSoVnAC+BzLxH0RmEG4wlCbwrIagLJl7AYqF7i7wuV7EgSl4m
dL73AErLB2YaTFb6Ao/P2iNHLNKHYRP/iULpDhbMcgLblizKMPGdzmOo1ub4SOei
p8TL26oEGLwZu3JEmkJhEwmDpm816xCGxwqZswtAns21At8ulA1cPxHA/kXxRlGe
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=server
issuer=/CN=Elastic Certificate Tool Autogenerated CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1485 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: FED030D7D4685EE4010FE3D1FD0AFEB890EC8A9EB8E23D3C21A717120FA52D11
    Session-ID-ctx:
    Master-Key: DB99744CF8D56955F3C646052893F939A7FBB47A6C31C6BC4F0A1E6504F3591A7CF3BCF8E0D1BCC2EC6F095DD03D3DD2
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket:
    0000 - 17 4c 8b d6 3f c5 90 a6-aa b0 ea 23 5b 48 ca e3   .L..?......#[H..
    0010 - f5 5c f0 a7 07 78 fb ee-f4 f9 a9 5c dd 20 da f1   .\...x.....\. ..
    0020 - 40 f4 4e 8b c4 34 d3 e0-b5 b7 92 c2 28 5f 63 24   @.N..4......(_c$
    0030 - be cb ab 7a 78 86 a5 82-ac a5 f4 f3 b3 65 d4 a5   ...zx........e..
    0040 - a0 30 17 16 9a 4f 84 86-e5 5e c5 b5 d2 51 9d 1f   .0...O...^...Q..
    0050 - d4 2a fe 94 a0 0d 53 b5-ed 27 27 42 e3 8e 86 ae   .*....S..''B....
    0060 - 3d a7 45 a9 bd a3 9b d2-4d 4c 5e 17 4f 24 e3 fe   =.E.....ML^.O$..
    0070 - 3b db 18 76 75 eb 82 6c-3d 9a ca 0d 14 c9 93 87   ;..vu..l=.......
    0080 - 44                                                D

    Start Time: 1638847788
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

I am only going to grab lines from and including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- so it will be

Now lets take a look at this certificate you will see that it is the same file as your server.crt, so ideally you can skip the above and just use your server.crt for this exercise.

$ cd /tmp/certs
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} cp /usr/share/elasticsearch/jdk/lib/security/cacerts /tmp/certs/cacerts
$ sudo chmod 777 cacerts
$ docker run --rm -v "$(pwd)":/tmp/certs -w /tmp/certs ${IMG} /usr/share/elasticsearch/jdk/bin/keytool  -keystore cacerts -storepass changeit -noprompt -importcert -file server/server.crt -alias minio
$ zip cacerts.zip cacerts

Lets get the cacerts.zip staged for your httpd

mkdir /tmp/htdocs
cd /tmp/htdocs
cp /tmp/certs/cacerts.zip ./

Host the custom bundle on a web server

Lets now host a simple http server that will serve our cacerts.zip

$ docker run -dit --name apache -p 9002:80 -v /tmp/htdocs:/usr/local/apache2/htdocs/ httpd:2.4
$ docker ps | grep http
515e05b0a759        httpd:2.4                                                           "httpd-foreground"       About an hour ago   Up About an hour    0.0.0.0:9002->80/tcp                                                                                                                                                     apache

To test lets try to grab the cacerts file

$ wget http://localhost:9002/cacerts.zip
--2021-12-07 00:29:25--  http://localhost:9002/cacerts.zip
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:9002... connected.
HTTP request sent, awaiting response... 200 OK
Length: 111398 (109K) [application/zip]
Saving to: ‘cacerts.zip’

100%[===============================================================================================================================================>] 111,398     --.-K/s   in 0s

2021-12-07 00:29:25 (360 MB/s) - ‘cacerts.zip’ saved [111398/111398]

Now all the preparation is complete!

Part 2

We will stand up a deployment and configure custom bundles and configure the snapshot

Lets create a simple deployment named test. We will use the default elasticsearch deployment with 2 AZ.

After the deployment is created lets goto Edit -> Advanced Edit -> Deployment configuration and around line 56 look for

          "elasticsearch": {
            "version": "7.11.1"
          },

and change it to

          "elasticsearch": {
            "version": "7.11.1",
            "user_bundles": [
              {
                "name": "custom-ca-certs",
                "url": "http://172.16.0.61:9002/cacerts.zip",
                "elasticsearch_version": "*"
              }
            ]
          },

Please change 172.16.0.61 to the IP of your http server that we stood up in Part 1. I am using GCP instances so I used the internal IP of the GCP instance. IF you are on prem you can use the IP of the ECE host.

Click on SAVE - this will lead to a change plan and your cluster should restart. Once the change is complete goto the devtools in kibana for the test deployment

Lets create the snapshot repository

PUT _snapshot/minio01
{
  "type": "s3",
  "settings": {
    "bucket": "elastic",
    "endpoint": "172.16.0.61:9001",
    "protocol": "https",
    "path_style_access": "true",
    "access_key": "minio",
    "secret_key": "minio123"
  }
}

#! [access_key] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
#! [secret_key] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
#! Using s3 access/secret key from repository settings. Instead store these in named clients and the elasticsearch keystore for secure settings.
{
  "acknowledged" : true
}

Verify the repository

POST /_snapshot/minio01/_verify

{
  "nodes" : {
    "eFEJL77STGyv7PU0a8OtSw" : {
      "name" : "instance-0000000001"
    },
    "ORg5jorNT2-9jdeYcqEC-Q" : {
      "name" : "instance-0000000000"
    }
  }
}

You can now go to Stack Management -> Snapshot and restore -> Policy -> create or even create a new SLM policy from devtools.

Just for testing lets run a quick snapshot and verify it

PUT _snapshot/minio01/snapshot_1?wait_for_completion=true

GET _cat/snapshots/minio01?v

id          status start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
snapshot_1 SUCCESS 1638823923  20:52:03   1638823924 20:52:04       1s       9                 9             0            9

ECE(Elasticsearch Cloud Enterprise) snapshots with minio – TLS!