Mounting certificates/CAs for elasticsearch pods in k8s for custom configurations – ECK

When you deploy elasticsearch in k8s via ECK by default it takes care of the transport certificates/http certificates/CA creation if you don't use your own custom ones.

Lets say that your cluster is up and running and now you are trying to configure something like SAML/LDAP/OIDC or something else that requires content to be made available to the elasticsearch pod but its a bit sensitive so you need it to be stored as a secret such as CA, certificates, etc. How would you go about doing this?

Following example will add a custom certificate authority to be used with your xpack settings when configuring various authentication with elasticsearch so that even if your auth provider has custom/internal/self-signed certificate it will work.

Create a secret

There are multiple ways to create the secret.

• via command line

$ kubectl create secret generic ca --from-file=ca.crt

• via yaml

$ cat ca.crt
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVAMHMs72Dweteroshd5dObzjgtry+MA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIyMDcyNzEzNDkyOFoXDTI1MDcyNjEzNDkyOFowNDEyMDAG
A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxt6JV2y5PXhLyo5jU2RJw
g1E7ntt5an1sMXBg01Tgne0E3RH8ga4XSZnzPGqCP5w9WYvlIsBPRxj6tmQAL1V7
FS62e0WENbaIJyuPk4O3JhlsYIOe6ACOdjfq72w14gBdqPyRn6txIPZErXL2SmC5
uxd2ccAIW3FCMsvTgSLAKWkKpIgPTe3OznRJ7q0KJ5dNO7kCRukOrQ4eFrW6YboU
Iil8QtrjZqtG0IflkfKp+r7pzIgiZnn28hS7gz4wCVabzJlOy1Usgb71xOUyUNHP
cG1qLNBPEvxgqLP3VL6aI40Jta5IO4h4NOzRVFqgE+QvuELsW7scD8O1MVL3H+lp
AgMBAAGjUzBRMB0GA1UdDgQWBBR2eVQVdE6JmhNWi8jT9jitZdHB1TAfBgNVHSME
GDAWgBR2eVQVdE6JmhNWi8jT9jitZdHB1TAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQASnFScLgyFnxUsCx0pceQhBlRX5lvricMj3Z1PbJlPnWMY
IESLuuG0yOE4nutwuDcMf2+A3SmpXRCwcuLpHztJQ4w4Lo6jqOrlyFS6cfRfq+QP
mU2y5R7khhOT3vGSh6QFoHL5TyF2zFDv4Gy4b3Bp3kFaYhKnskrGx+CXjH8Dtxy5
mFWvtrodx2FoCy7RzPby368moqADuSuEWshO9t62IQs4JYL63NY1lxtI24x5beFv
H0fgi8Etcmc0oIdy8QYptrcHsZ6HZWD9sJZytK1+1jPTd3GQ8q3Mpx7qitRDCHCY
9rvLKyX8hQeeu10QjqxdbMQ5luxI8aQNv4oSvQNb
-----END CERTIFICATE-----

$ base64 ca.key
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBc2JlaVZkc3VUMTRTOHFPWTFOa1NjSU5STzU3YmVXcDliREZ3WU5OVTRKM3RCTjBSCi9JR3VGMG1aOHp4cWdqK2NQVm1MNVNMQVQwY1krclprQUM5VmV4VXV0bnRGaERXMmlDY3JqNU9EdHlZWmJHQ0QKbnVnQWpuWTM2dTlzTmVJQVhhajhrWityY1NEMlJLMXk5a3BndWJzWGRuSEFDRnR4UWpMTDA0RWl3Q2xwQ3FTSQpEMDN0enM1MFNlNnRDaWVYVFR1NUFrYnBEcTBPSGhhMXVtRzZGQ0lwZkVMYTQyYXJSdENINVpIeXFmcSs2Y3lJCkltWjU5dklVdTRNK01BbFdtOHlaVHN0VkxJRys5Y1RsTWxEUnozQnRhaXpRVHhMOFlLaXo5MVMrbWlPTkNiV3UKU0R1SWVEVHMwVlJhb0JQa0w3aEM3RnU3SEEvRHRURlM5eC9wYVFJREFRQUJBb0lCQUUyT1pxMVNQRGtOeEhZWQpXQk9YY2RraExidTNuVzgwdm15UGZNdjhuaDl5UENReHQrNDZ4N2RJK3dMdi9FbEFLbGFKdENmNGpQVnNBU1kwCm41cUhEYVpydlptTVhXK1BvMHR5MEt0VlJKZVNiR2prQzd5MjRidEtNeWYzZVlGZ1VpbU5uRCtPNzFvcytOeEsKMTQ5Tk1UNE12SFlIdVRiTWRUWmZCWjJWVHcyTnFXcDM5cGpIaVBxTGdjQUYvMS9RSlhKMkF3aW1SRVJaVE1jWQoxZXhUNi9zYmtGb3k0NURNMy9yUWo3U1Rvdk1QTHZwcFlOQ2tNclFnWXJFZmR3UlMrdXNoRGF3UlhiN1czNjI2CmNZV0pEWHEwQnhWVk9kWkpzU1prZ0I5MXl5K0QzN21nSm9UZmpNcGp2eDdwT1YwQytoSkhkOUFKTGJ4YktPc0cKTGwybklnc0NnWUVBdmZYYmZkVkhzQk1qYzIxNGhJd21OenZ4TjlSaWZXQWN3WmhvM3RyMDNMNHpPdXVMakxiMApaSzc0cTljaTJMZjlZOVJodU9QMHEvbWdZQXNOT3RzaW40OEw1b0czakRjcnluZXc5bVFTM0h6a3U3c0djOXBtClhUZkljUVRuekRaUWtDZnNMem4yOGQ1a3h0U1o0cGJOQVE5VDdWcDM4c1lKNGxIeHlWa0xQR2NDZ1lFQTc0QXEKY0VGMkhiVEE2SFRQL3FZQzNpUFF0YmxhZXF0THhqT1lDT2dpeDU3Y1BpZC9Bc1lxUWtPQXg5azN4cERpZlVaMAp1c1huRzRQSWZGeDdlTWhkQU4zcHQ3bzdHREc4UGNBS1ZkSlhReWc0cUpkZ3c2TVZuSnpGVlNZaXVHS285emgxCnJ1OFFnckhRaDFsaFpveHVhTDBha3RmYXdNN2FITkhwWXQ3ZUNhOENnWUVBc0NHc1dHcFBSQVVhMDhYbS93OXIKeDc3K2xFT0s1ckVkT0t2MllOd29PaHpwSjNLTE8vZlBkeU45VXVmeFdYeVBwK2Flalc1ZnlLMTJkRmNLOTAyRQpOdW5Ob1BjWmx4cjRzWXgxT3AvamxTa0FkTWllM1FEZDlRVHFCMUVERkNnd1h0bUZkdXlKK2cyNDUxcnRMRm1pCno1N0V0TWt6ZXBrQlJTV3h5SDJWbTNVQ2dZRUE2bUNMcDNWNGFOUEY0UGJId1pXbXdzdGI2U3J0NzVRSXdIc24KTXJneHFSYSt4QmlVeEJzM3FjY3psTi92SkdRUE1iZXNia2RUeG1wdDNiNEhtd2pCY3lLNElQT2Y5eXBjNURVMgpLQXlERnhhYWMrcElhTXdGVDFGZ0Q5Y1ZMVXBudmgvRGt5RWpFRE9CTmFOem5RNmEwZWNWeFZSMklDK041WnhMCkFGWElCc01DZ1lCUWRXdkFYbDUwYko3TURXb2h6MnQ4UGtrUDlsWUphRjg5YmNlSlRPM3Y3dXBucDEyWHFRUnoKK0JoRkI1K2V2WkJuTk8vbzlsWXE2ODExV3BDampyWHpkbmkyekM3VGxMdnNpSmZCbGt1TjM3dlBsL0dxLzU0QgpEQmFjVG1MSFFrWElTTjF5NWlJb0ZNZ1Z2YkN1ZmhFVW93MHlzdUxlRW02TjRkNjJrRG0vQkE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

Create secret.yml

apiVersion: v1
data:
 ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURTakNDQWpLZ0F3SUJBZ0lWQU1ITXM3MkR3ZXRlcm9zaGQ1ZE9iempndHJ5K01BMEdDU3FHU0liM0RRRUIKQ3dVQU1EUXhNakF3QmdOVkJBTVRLVVZzWVhOMGFXTWdRMlZ5ZEdsbWFXTmhkR1VnVkc5dmJDQkJkWFJ2WjJWdQpaWEpoZEdWa0lFTkJNQjRYRFRJeU1EY3lOekV6TkRreU9Gb1hEVEkxTURjeU5qRXpORGt5T0Zvd05ERXlNREFHCkExVUVBeE1wUld4aGMzUnBZeUJEWlhKMGFXWnBZMkYwWlNCVWIyOXNJRUYxZEc5blpXNWxjbUYwWldRZ1EwRXcKZ2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3h0NkpWMnk1UFhoTHlvNWpVMlJKdwpnMUU3bnR0NWFuMXNNWEJnMDFUZ25lMEUzUkg4Z2E0WFNabnpQR3FDUDV3OVdZdmxJc0JQUnhqNnRtUUFMMVY3CkZTNjJlMFdFTmJhSUp5dVBrNE8zSmhsc1lJT2U2QUNPZGpmcTcydzE0Z0JkcVB5Um42dHhJUFpFclhMMlNtQzUKdXhkMmNjQUlXM0ZDTXN2VGdTTEFLV2tLcElnUFRlM096blJKN3EwS0o1ZE5PN2tDUnVrT3JRNGVGclc2WWJvVQpJaWw4UXRyalpxdEcwSWZsa2ZLcCtyN3B6SWdpWm5uMjhoUzdnejR3Q1ZhYnpKbE95MVVzZ2I3MXhPVXlVTkhQCmNHMXFMTkJQRXZ4Z3FMUDNWTDZhSTQwSnRhNUlPNGg0Tk96UlZGcWdFK1F2dUVMc1c3c2NEOE8xTVZMM0grbHAKQWdNQkFBR2pVekJSTUIwR0ExVWREZ1FXQkJSMmVWUVZkRTZKbWhOV2k4alQ5aml0WmRIQjFUQWZCZ05WSFNNRQpHREFXZ0JSMmVWUVZkRTZKbWhOV2k4alQ5aml0WmRIQjFUQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUFTbkZTY0xneUZueFVzQ3gwcGNlUWhCbFJYNWx2cmljTWozWjFQYkpsUG5XTVkKSUVTTHV1RzB5T0U0bnV0d3VEY01mMitBM1NtcFhSQ3djdUxwSHp0SlE0dzRMbzZqcU9ybHlGUzZjZlJmcStRUAptVTJ5NVI3a2hoT1QzdkdTaDZRRm9ITDVUeUYyekZEdjRHeTRiM0JwM2tGYVloS25za3JHeCtDWGpIOER0eHk1Cm1GV3Z0cm9keDJGb0N5N1J6UGJ5MzY4bW9xQUR1U3VFV3NoTzl0NjJJUXM0SllMNjNOWTFseHRJMjR4NWJlRnYKSDBmZ2k4RXRjbWMwb0lkeThRWXB0cmNIc1o2SFpXRDlzSlp5dEsxKzFqUFRkM0dROHEzTXB4N3FpdFJEQ0hDWQo5cnZMS3lYOGhRZWV1MTBRanF4ZGJNUTVsdXhJOGFRTnY0b1N2UU5iCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
kind: Secret
metadata:
 name: ca
type: Opaque

kubectl apply -f secret.yml

Verify your secret

$ kubectl get secret
NAME                                    TYPE                                  DATA   AGE
ca                                      Opaque                                2      23m
...

$ kubectl describe secret ca
Name:         ca
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
ca.crt:  1200 bytes

Edit your elasticsearch deployment to create the volumeMount

Updated my podTemplate to include the volumeMount

    podTemplate:
      metadata:
        labels:
          scrape: es
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          volumeMounts:
          - name: ca
            mountPath: /usr/share/elasticsearch/config/certificates
            readOnly: true
        volumes:
          - name: ca
            secret:
              secretName: ca

Apply and once the change is done you can see the mount and the file

$ kubectl exec -it eck-lab-es-default-0 -- bash
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init), sysctl (init)

elasticsearch@eck-lab-es-default-0:~$ cd config/certificates/

elasticsearch@eck-lab-es-default-0:~/config/certificates$ pwd
/usr/share/elasticsearch/config/certificates

elasticsearch@eck-lab-es-default-0:~/config/certificates$ ls
ca.crt 

elasticsearch@eck-lab-es-default-0:~/config/certificates$ cat ca.crt
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIVAMHMs72Dweteroshd5dObzjgtry+MA0GCSqGSIb3DQEB
CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
ZXJhdGVkIENBMB4XDTIyMDcyNzEzNDkyOFoXDTI1MDcyNjEzNDkyOFowNDEyMDAG
A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxt6JV2y5PXhLyo5jU2RJw
g1E7ntt5an1sMXBg01Tgne0E3RH8ga4XSZnzPGqCP5w9WYvlIsBPRxj6tmQAL1V7
FS62e0WENbaIJyuPk4O3JhlsYIOe6ACOdjfq72w14gBdqPyRn6txIPZErXL2SmC5
uxd2ccAIW3FCMsvTgSLAKWkKpIgPTe3OznRJ7q0KJ5dNO7kCRukOrQ4eFrW6YboU
Iil8QtrjZqtG0IflkfKp+r7pzIgiZnn28hS7gz4wCVabzJlOy1Usgb71xOUyUNHP
cG1qLNBPEvxgqLP3VL6aI40Jta5IO4h4NOzRVFqgE+QvuELsW7scD8O1MVL3H+lp
AgMBAAGjUzBRMB0GA1UdDgQWBBR2eVQVdE6JmhNWi8jT9jitZdHB1TAfBgNVHSME
GDAWgBR2eVQVdE6JmhNWi8jT9jitZdHB1TAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQASnFScLgyFnxUsCx0pceQhBlRX5lvricMj3Z1PbJlPnWMY
IESLuuG0yOE4nutwuDcMf2+A3SmpXRCwcuLpHztJQ4w4Lo6jqOrlyFS6cfRfq+QP
mU2y5R7khhOT3vGSh6QFoHL5TyF2zFDv4Gy4b3Bp3kFaYhKnskrGx+CXjH8Dtxy5
mFWvtrodx2FoCy7RzPby368moqADuSuEWshO9t62IQs4JYL63NY1lxtI24x5beFv
H0fgi8Etcmc0oIdy8QYptrcHsZ6HZWD9sJZytK1+1jPTd3GQ8q3Mpx7qitRDCHCY
9rvLKyX8hQeeu10QjqxdbMQ5luxI8aQNv4oSvQNb
-----END CERTIFICATE-----

Now you can use this path/file to configure your settings

Example:

xpack:
  security:
    authc:
      realms:
        ldap:
          ldap1:
            order: 0
            url: "ldaps://ldap.example.com:636"
            ssl:
              certificate_authorities: [ "/usr/share/elasticsearch/config/certificates/ca.crt" ]