Whenever you install beats or update beats its best if you run the setup so that the setup will update your index templates, ILM, visualizations, etc. I found that its always best before installing or upgrading to use a seed host to just run the setup before deploying or updating across your environment and to turn off template updates and dashboard loading on the beats locally so that your elasticsearch cluster is not flooded.
In a baremetal environment this is easy to do but in docker and in kubernetes it gets a bit difficult.
Sure you can setup a initContainer for your beats to run the setup before starting your pods however the initContainer will run each time there is a change or update so if you have a lot of different beats deployed in a lot of places or if you are running this as a Beat or DaemonSet it can potentially flood your elasticsearch cluster if you have a lot of hosts.
So what can you do?
There are many ways to do this but this method worked for me.
In Docker you can run a docker image to run commands and let the container exit once the command is complete. You can also do similar in kubernetes.
This will work on all beats - filebeat, metricbeat, heartbeat, journalbeat, packetbeat, etc.
Lets go ahead and create a quick test:
Create a simple elastic stack
kubectl create -f https://download.elastic.co/downloads/eck/1.9.1/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/1.9.1/operator.yaml
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.17.0
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
EOF
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: quickstart
spec:
version: 7.17.0
count: 1
elasticsearchRef:
name: quickstart
EOF
kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/quickstart-es-default-0 1/1 Running 0 3m7s
pod/quickstart-kb-6d86f77f4b-fgjnn 1/1 Running 0 3m6s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4m33s
service/quickstart-es-default ClusterIP None <none> 9200/TCP 3m7s
service/quickstart-es-http ClusterIP 10.104.56.200 <none> 9200/TCP 3m10s
service/quickstart-es-transport ClusterIP None <none> 9300/TCP 3m10s
service/quickstart-kb-http ClusterIP 10.99.77.239 <none> 5601/TCP 3m10s
Grab the elastic users password:
kubectl get secrets
NAME TYPE DATA AGE
default-quickstart-kibana-user Opaque 3 3m45s
default-token-tb4f8 kubernetes.io/service-account-token 3 4m51s
quickstart-es-default-es-config Opaque 1 3m42s
quickstart-es-default-es-transport-certs Opaque 3 3m43s
quickstart-es-elastic-user Opaque 1 3m43s
quickstart-es-http-ca-internal Opaque 2 3m44s
quickstart-es-http-certs-internal Opaque 3 3m43s
quickstart-es-http-certs-public Opaque 2 3m43s
quickstart-es-internal-users Opaque 3 3m42s
quickstart-es-remote-ca Opaque 1 3m43s
quickstart-es-transport-ca-internal Opaque 2 3m43s
quickstart-es-transport-certs-public Opaque 1 3m43s
quickstart-es-xpack-file-realm Opaque 3 3m42s
quickstart-kb-config Opaque 1 3m41s
quickstart-kb-es-ca Opaque 2 3m43s
quickstart-kb-http-ca-internal Opaque 2 3m43s
quickstart-kb-http-certs-internal Opaque 3 3m43s
quickstart-kb-http-certs-public Opaque 2 3m42s
quickstart-kibana-user Opaque 1 3m45s
PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
echo $PASSWORD
43Q0G7Y6529VBQkSn0oQam2JTest to ensure that we are good to go
kubectl port-forward service/quickstart-es-http 9200
Forwarding from 127.0.0.1:9200 -> 9200
Forwarding from [::1]:9200 -> 9200
curl -u "elastic:$PASSWORD" -k "https://localhost:9200"
{
"name" : "quickstart-es-default-0",
"cluster_name" : "quickstart",
"cluster_uuid" : "CJGwkSjeQVm48YfQAhXWmg",
"version" : {
"number" : "7.17.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
"build_date" : "2022-01-28T08:36:04.875279988Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}Awesome! now we have a working elasticsearch cluster and we know the elastic users password and we also know CLUSTER-IP's and PORTS to access both elasticsearch and kibana
For this example we will use filebeats but this will work with any beats
Run a seed pod so that we can run the setup
kubectl run -i --tty filebeat-seed --image=docker.elastic.co/beats/filebeat:7.17.0 -- sh
If you don't see a command prompt, try pressing enter.
$This will take a while to run since the container image needs to be pulled first.
From https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#setup-command we see that there are various options you can add to the setup but we will just run the setup so that it will load the defaults.
Plug in all the information and run
./filebeat setup -e \
-E output.logstash.enabled=false \
-E output.elasticsearch.hosts=['10.104.56.200:9200'] \
-E output.elasticsearch.protocol=https \
-E output.elasticsearch.username=elastic \
-E output.elasticsearch.password=43Q0G7Y6529VBQkSn0oQam2J \
-E output.elasticsearch.ssl.verification_mode=none \
-E setup.kibana.host='10.99.77.239:5601' \
-E setup.kibana.protocol=https \
-E setup.kibana.username=elastic \
-E setup.kibana.password=43Q0G7Y6529VBQkSn0oQam2J \
-E setup.kibana.ssl.verification_mode=none
2022-02-09T04:22:28.972Z INFO instance/beat.go:686 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] Hostfs Path: [/]
2022-02-09T04:22:28.972Z INFO instance/beat.go:694 Beat ID: cef84f35-59a9-4fbc-baf8-ef3fbfbbf4ab
2022-02-09T04:22:31.974Z WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:79 read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-02-09T04:22:31.975Z INFO [beat] instance/beat.go:1040 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "cef84f35-59a9-4fbc-baf8-ef3fbfbbf4ab"}}}
2022-02-09T04:22:31.975Z INFO [beat] instance/beat.go:1049 Build info {"system_info": {"build": {"commit": "93708bd74e909e57ed5d9bea3cf2065f4cc43af3", "libbeat": "7.17.0", "time": "2022-01-28T09:53:30.000Z", "version": "7.17.0"}}}
2022-02-09T04:22:31.975Z INFO [beat] instance/beat.go:1052 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.5"}}}
2022-02-09T04:22:31.976Z INFO [beat] instance/beat.go:1056 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-01-16T03:22:05Z","containerized":true,"name":"filebeat-seed","ip":["127.0.0.1/8","172.17.0.7/16"],"kernel_version":"3.10.0-1160.49.1.el7.x86_64","mac":["02:42:ac:11:00:07"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.3 LTS (Focal Fossa)","major":20,"minor":4,"patch":3,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-02-09T04:22:31.976Z INFO [beat] instance/beat.go:1085 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 31, "ppid": 7, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2022-02-09T04:22:28.150Z"}}}
2022-02-09T04:22:31.976Z INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.0
2022-02-09T04:22:31.976Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.17.0' as ILM is enabled.
2022-02-09T04:22:31.976Z WARN [cfgwarn] tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2022-02-09T04:22:31.977Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.104.56.200:9200
2022-02-09T04:22:31.977Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:22:31.977Z INFO [publisher] pipeline/module.go:113 Beat name: filebeat-seed
2022-02-09T04:22:31.978Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.104.56.200:9200
2022-02-09T04:22:31.978Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:22:31.978Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:22:31.987Z INFO [esclientleg] eslegclient/connection.go:284 Attempting to connect to Elasticsearch version 7.17.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2022-02-09T04:22:32.024Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2022-02-09T04:22:32.030Z INFO [index-management.ilm] ilm/std.go:170 ILM policy filebeat exists already.
2022-02-09T04:22:32.030Z INFO [index-management] idxmgmt/std.go:397 Set setup.template.name to '{filebeat-7.17.0 {now/d}-000001}' as ILM is enabled.
2022-02-09T04:22:32.030Z INFO [index-management] idxmgmt/std.go:402 Set setup.template.pattern to 'filebeat-7.17.0-*' as ILM is enabled.
2022-02-09T04:22:32.030Z INFO [index-management] idxmgmt/std.go:436 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.0 {now/d}-000001} as ILM is enabled.
2022-02-09T04:22:32.030Z INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-02-09T04:22:32.047Z INFO template/load.go:197 Existing template will be overwritten, as overwrite is enabled.
2022-02-09T04:22:33.667Z INFO template/load.go:131 Try loading template filebeat-7.17.0 to Elasticsearch
2022-02-09T04:22:33.744Z INFO template/load.go:123 Template with name "filebeat-7.17.0" loaded.
2022-02-09T04:22:33.744Z INFO [index-management] idxmgmt/std.go:297 Loaded index template.
2022-02-09T04:22:33.746Z INFO [index-management.ilm] ilm/std.go:126 Index Alias filebeat-7.17.0 exists already.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2022-02-09T04:22:33.747Z INFO kibana/client.go:180 Kibana url: https://10.99.77.239:5601
2022-02-09T04:22:33.747Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:22:33.747Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:22:34.975Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2022-02-09T04:22:35.877Z INFO kibana/client.go:180 Kibana url: https://10.99.77.239:5601
2022-02-09T04:22:35.877Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:22:35.877Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.220Z INFO instance/beat.go:869 Kibana dashboards successfully loaded.
Loaded dashboards
2022-02-09T04:23:54.220Z WARN [cfgwarn] instance/beat.go:594 DEPRECATED: Setting up ML using Filebeat is going to be removed. Please use the ML app to setup jobs. Will be removed in version: 8.0.0
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
It is not possble to load ML jobs into an Elasticsearch 8.0.0 or newer using the Beat.
2022-02-09T04:23:54.221Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.104.56.200:9200
2022-02-09T04:23:54.221Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.221Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.230Z INFO [esclientleg] eslegclient/connection.go:284 Attempting to connect to Elasticsearch version 7.17.0
2022-02-09T04:23:54.230Z INFO kibana/client.go:180 Kibana url: https://10.99.77.239:5601
2022-02-09T04:23:54.230Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.230Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.330Z WARN fileset/modules.go:463 X-Pack Machine Learning is not enabled
Loaded machine learning job configurations
2022-02-09T04:23:54.333Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.104.56.200:9200
2022-02-09T04:23:54.333Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.333Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-09T04:23:54.342Z INFO [esclientleg] eslegclient/connection.go:284 Attempting to connect to Elasticsearch version 7.17.0
2022-02-09T04:23:54.342Z INFO cfgfile/reload.go:262 Loading of config files completed.
Loaded Ingest pipelines
$ exit
Session ended, resume using 'kubectl attach filebeat-seed -c filebeat-seed -i -t' command when the pod is running
kubectl get pod
NAME READY STATUS RESTARTS AGE
filebeat-seed 1/1 Running 1 9m50s
quickstart-es-default-0 1/1 Running 0 26m
quickstart-kb-6d86f77f4b-fgjnn 1/1 Running 0 26m
kubectl delete pod filebeat-seed
pod "filebeat-seed" deletedWent ahead and cleanup and deleted the pod.
Alternatively, you can edit filebeat.yml and just run ./filebeat setup -e if you don't want to pass all the env vars.
Now we can log into kibana and see that we have templates/ILM/dashboards!
ALTERNATIVE 1
Thanks to my friend Camilo Sierra! you can also perform the same using kubernets jobs
Create the manifest
apiVersion: batch/v1
kind: Job
metadata:
name: seed-filebeat
spec:
template:
spec:
containers:
- name: seed-filebeat
image: docker.elastic.co/beats/filebeat:7.17.0
command: ["/bin/sh", "-c"]
args: ["filebeat setup -e -E output.elasticsearch.hosts=['10.103.121.252:9200'] -E output.elasticsearch.protocol=https -E output.elasticsearch.username=elastic -E output.elasticsearch.password=d71gR579ON53p3rc7C2IHcVa -E output.elasticsearch.ssl.verification_mode=none -E setup.kibana.host='10.97.152.126:5601' -E setup.kibana.protocol=https -E setup.kibana.username=elastic -E setup.kibana.password=d71gR579ON53p3rc7C2IHcVa -E setup.kibana.ssl.verification_mode=none"]
restartPolicy: Never
backoffLimit: 1- side note: endpoints and the passwords are different in this example since this is a new test
Create the job
kubectl apply -f job.yamlOnce the job is applied it will create a pod
kubectl get pod
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 1/1 Running 0 48m
quickstart-kb-57955bbc9f-g9htz 1/1 Running 0 48m
seed-filebeat-zxggl 0/1 Completed 0 6m20sLets look at the logs from the pod
kubectl logs -f seed-filebeat-zxggl
2022-02-10T18:50:57.646Z INFO instance/beat.go:686 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] Hostfs Path: [/]
2022-02-10T18:50:57.650Z INFO instance/beat.go:694 Beat ID: 74377805-61a5-4d2f-af19-3e4b961cbafb
2022-02-10T18:51:00.652Z WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:79 read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-02-10T18:51:00.654Z INFO [beat] instance/beat.go:1040 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "74377805-61a5-4d2f-af19-3e4b961cbafb"}}}
2022-02-10T18:51:00.654Z INFO [beat] instance/beat.go:1049 Build info {"system_info": {"build": {"commit": "93708bd74e909e57ed5d9bea3cf2065f4cc43af3", "libbeat": "7.17.0", "time": "2022-01-28T09:53:30.000Z", "version": "7.17.0"}}}
2022-02-10T18:51:00.654Z INFO [beat] instance/beat.go:1052 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.5"}}}
2022-02-10T18:51:00.656Z INFO [beat] instance/beat.go:1056 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-01-16T03:22:05Z","containerized":true,"name":"seed-filebeat-zxggl","ip":["127.0.0.1/8","172.17.0.7/16"],"kernel_version":"3.10.0-1160.49.1.el7.x86_64","mac":["02:42:ac:11:00:07"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.3 LTS (Focal Fossa)","major":20,"minor":4,"patch":3,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-02-10T18:51:00.657Z INFO [beat] instance/beat.go:1085 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2022-02-10T18:50:56.810Z"}}}
2022-02-10T18:51:00.657Z INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.0
2022-02-10T18:51:00.658Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.17.0' as ILM is enabled.
2022-02-10T18:51:00.658Z WARN [cfgwarn] tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2022-02-10T18:51:00.659Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.103.121.252:9200
2022-02-10T18:51:00.660Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:51:00.660Z INFO [publisher] pipeline/module.go:113 Beat name: seed-filebeat-zxggl
2022-02-10T18:51:00.664Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.103.121.252:9200
2022-02-10T18:51:00.664Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:51:00.665Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:51:00.798Z INFO [esclientleg] eslegclient/connection.go:284 Attempting to connect to Elasticsearch version 7.17.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2022-02-10T18:51:00.839Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2022-02-10T18:51:00.878Z INFO [index-management.ilm] ilm/std.go:180 ILM policy filebeat successfully created.
2022-02-10T18:51:00.879Z INFO [index-management] idxmgmt/std.go:397 Set setup.template.name to '{filebeat-7.17.0 {now/d}-000001}' as ILM is enabled.
2022-02-10T18:51:00.879Z INFO [index-management] idxmgmt/std.go:402 Set setup.template.pattern to 'filebeat-7.17.0-*' as ILM is enabled.
2022-02-10T18:51:00.879Z INFO [index-management] idxmgmt/std.go:436 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.0 {now/d}-000001} as ILM is enabled.
2022-02-10T18:51:00.879Z INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-02-10T18:51:00.881Z INFO template/load.go:197 Existing template will be overwritten, as overwrite is enabled.
2022-02-10T18:51:02.619Z INFO template/load.go:131 Try loading template filebeat-7.17.0 to Elasticsearch
2022-02-10T18:51:02.988Z INFO template/load.go:123 Template with name "filebeat-7.17.0" loaded.
2022-02-10T18:51:02.988Z INFO [index-management] idxmgmt/std.go:297 Loaded index template.
2022-02-10T18:51:03.624Z INFO [index-management.ilm] ilm/std.go:140 Index Alias filebeat-7.17.0 successfully created.
2022-02-10T18:51:03.624Z INFO kibana/client.go:180 Kibana url: https://10.97.152.126:5601
2022-02-10T18:51:03.624Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:51:03.624Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2022-02-10T18:51:03.655Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2022-02-10T18:51:05.824Z INFO kibana/client.go:180 Kibana url: https://10.97.152.126:5601
2022-02-10T18:51:05.824Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:51:05.824Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
It is not possble to load ML jobs into an Elasticsearch 8.0.0 or newer using the Beat.
2022-02-10T18:52:24.446Z INFO instance/beat.go:869 Kibana dashboards successfully loaded.
2022-02-10T18:52:24.446Z WARN [cfgwarn] instance/beat.go:594 DEPRECATED: Setting up ML using Filebeat is going to be removed. Please use the ML app to setup jobs. Will be removed in version: 8.0.0
2022-02-10T18:52:24.446Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.103.121.252:9200
2022-02-10T18:52:24.447Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:52:24.447Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:52:24.456Z INFO [esclientleg] eslegclient/connection.go:284 Attempting to connect to Elasticsearch version 7.17.0
2022-02-10T18:52:24.456Z INFO kibana/client.go:180 Kibana url: https://10.97.152.126:5601
2022-02-10T18:52:24.456Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:52:24.456Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:52:24.550Z WARN fileset/modules.go:463 X-Pack Machine Learning is not enabled
Loaded machine learning job configurations
2022-02-10T18:52:24.552Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: https://10.103.121.252:9200
2022-02-10T18:52:24.552Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:52:24.552Z WARN [tls] tlscommon/tls_config.go:101 SSL/TLS verifications disabled.
2022-02-10T18:52:24.561Z INFO [esclientleg] eslegclient/connection.go:284 Attempting to connect to Elasticsearch version 7.17.0
2022-02-10T18:52:24.561Z INFO cfgfile/reload.go:262 Loading of config files completed.
Loaded Ingest pipelinesYou can add/change the setup to just load individual or module specfic items as well.
To clean up the pod and the job just delete the job
kubectl delete -f job.yamlALTERNATIVE 2
This is similar to the filebeat-seed method but you can do this with just 2 commands
$ kubectl run -it beats-setup --image=docker.elastic.co/beats/${1}:${VERSION} -- sh -c "${1} setup -E output.elasticsearch.hosts=\"${ESIP}:9200\" -E output.elasticsearch.protocol=https -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${PASSWORD} -E output.elasticsearch.ssl.verification_mode=none -E setup.kibana.host=\"${KIBANAIP}:5601\" -E setup.kibana.protocol=https -E setup.kibana.username=elastic -E setup.kibana.password=${PASSWORD} -E setup.kibana.ssl.verification_mode=none -E setup.ilm.overwrite=true"
$ kubectl delete pod beats-setupwhere:
${1}is the type of beat like filebeat, metricbeat${VERSION}is the version of your stack${ESIP}is the ES endpoint${PASSWORD}is the password for the elastic user${KIBANAIP}is the KB endpoint
