Elasticsearch on k8s (ECK) All about passwords

Lets talk about passwords!

How is it set, where is it, how do you reset it, how do you force it?

When you do a simple deployment such as

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.1.1
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

Where is your password stored and how do you access it? For this article we will assume that your deployment name is quickstart

Default

The elastic users password is stored in secrets along with all the other secrets. It usually takes form of <DEPLYMENT NAME>-es-elastic-user and to get the password you can run something like kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'

Set a static password

Now this is cumbersome when are you testing and want it to be set to something more static. You can force the password by creating the secret before you deploy your deployment with something like kubectl create secret generic quickstart-es-elastic-user --from-literal=elastic=changeme your password for the elastic user will be set to changeme

Forgot passwords & Resetting passwords

Lets say you forgot your elastic users password and need to reset it. You can reset the password just for the elastic user by deleting it. The operator will recreate the secret and update any references to the elastic users password on the resources that it controls, this can be done via kubectl delete secret quickstart-es-elastic-user.

What if you had a security event and need to reset the passwords for all system users on your stack? First of all what all system users are there that is controlled by the operator? You can get a list by kubectl get secret -l eck.k8s.elastic.co/credentials=true. To reset all the credentials you can delete the secret by label and the operator will re-create it kubectl delete secret -l eck.k8s.elastic.co/credentials=true

File Realm

File realm is great if you need to configure some users from the start or via automation to bootstrap new users or for recovery. You can run various API's to create native users but if your cluster is having issues or if your running your CI/CD pipeline and needed more automated way to get users added from your vault you can use a file realm to add users.

File realm will require 2 parts. 1. creating the secret and 2. referencing the secret in your elasticsearch deployment

with hash

You can easily configure the elasticsearch deployment to use the file realm. From the example above you can add configs to configure it. Example:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.1.1
  auth:
    fileRealm:
    - secretName: my-filerealm-secret
  nodeSets:
  - name: default
    count: 1

You would also need to create a secret named my-filerealm-secret and you can do this by

kind: Secret
apiVersion: v1
metadata:
  name: my-filerealm-secret
stringData:
  users: |-
    rdeniro:$2a$10$BBJ/ILiyJ1eBTYoRKxkqbuDEdYECplvxnqQ47uiowE7yGqvCEgj9W
    alpacino:$2a$10$cNwHnElYiMYZ/T3K4PvzGeJ1KbpXZp2PfoQD.gfaVdImnHOwIuBKS
    jacknich:{PBKDF2}50000$z1CLJt0MEFjkIK5iEfgvfnA6xq7lF25uasspsTKSo5Q=$XxCVLbaKDimOdyWgLCLJiyoiWpA/XDMe/xtVgn1r5Sg=
  users_roles: |-
    admin:rdeniro
    power_user:alpacino,jacknich
    user:jacknich

Unfortunately for now only password hash is supported.

automation from various vaults

Lets say your secrets are stored in a vault with un-hashed password, so how would you hash it and add it as a secret? Since cleartextpassword is currently not supported we can use the elasticsearch container and docker to create the secret

$ mkdir filerealm
$ touch filerealm/users fileream/users_roles

$ docker run \ -v $(pwd)/filerealm:/usr/share/elasticsearch/config docker.elastic.co/elasticsearch/elasticsearch:8.1.1 bin/elasticsearch-users useradd myuser -p mypassword -r superuser

$ kubectl create secret genertic my-filerealm-secret-1 --from-file filerealm

You can subsitute myuser & mypassword from your vault.

Also if you need to create a custom role to be used with your file realm you can create your roles secret. Example:

kind: Secret
apiVersion: v1
metadata:
  name: my-roles-secret
stringData:
  roles.yml: |-
    click_admins:
      run_as: [ 'clicks_watcher_1' ]
      cluster: [ 'monitor' ]
      indices:
      - names: [ 'events-*' ]
        privileges: [ 'read' ]
        field_security:
          grant: ['category', '@timestamp', 'message' ]
        query: '{"match": {"category": "click"}}'

and reference both of your secrets in the elasticsearch configuration like

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.1.1
  auth:
    fileRealm:
    - secretName: my-filerealm-secret
    roles:
    - secretName: my-roles-secret
  nodeSets:
  - name: default
    count: 1

Hope this helps and enjoy!