System Stuff

1

Elastic Fleet Server & elastic-agent common troubleshooting

Posted on by jlim0930

Common troubleshooting items for any fleet & elastic-agent issues can be found on https://www.elastic.co/guide/en/fleet/current/fleet-troubleshooting.html however there are a few items that I would like to add. Whitespaces When you start the elastic-agent on an endpoint and even if you have everything configured for your elastic-agent, the right policy, integrations, etc however no data is going into elasticsearch. The most common issue is that there is something wrong with your fleet settings. This can be edited from kibana -> fleet -> settings. Either your ES endpoint / Fleet server endpoint / Advanced YAML settings are wrong or there might even be… Continue Reading →

4

Fleet server with logstash output – elastic-agent

Posted on by jlim0930

I think one of the biggest issue with fleet and elastic-agent was that it was limited on the outputs so if you had tons and tons of elastic-agents in the wild it would all connect back to your elasticsearch and can overwhelm the cluster. Starting 8.2 of elasticsearch logstash output type was introduced.. it is still in BETA at this time but I just tried it and it worked very nicely! The steps to setup the logstash output is listed on https://www.elastic.co/guide/en/fleet/8.2/secure-logstash-connections.html but I wanted to try it out and document it. Install & configure logstash Install yum install logstash-8.2.2… Continue Reading →

1

Google Cloud scripts

Posted on by jlim0930

At work we use google cloud as our lab to stand up and take down instances. As I work, I noticed that I usually stand up the same type of instance and run the same post install all the time so instead of doing this manually I decided to create a script for the compute instance and another for the kubernetes cluster. Compute Instance gcp.sh – script to automate creating and deletion of compute instance script is located: https://github.com/jlim0930/scripts/blob/master/gcp.sh edit gcp_name & gcp_project to match your environment. Also zone and region as needed – The script will stand up a… Continue Reading →

2

Updated deploy-elastic script – version 8

Posted on by jlim0930

Last update to version 7 is listed here The original post for the deploy-elastic.sh script is here. Changes: fully tested from 6.x-8.2(will most likely work with future 8.x versions) decoupled non stack modes from checking versions so that you can add the secondary components like monitoring/fleet/apm/enterprise-search using different version than the stack version. Just install the stack first then install the secondary components with different versions. Secondary versions can not be higher than the stack version. This will allow you to test using different versions fleet setting is now auto-populated. The script will gather the machines external IP and set… Continue Reading →

0

Elasticsearch on k8s (ECK) All about passwords

Posted on by jlim0930

Lets talk about passwords! How is it set, where is it, how do you reset it, how do you force it? When you do a simple deployment such as apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart spec: version: 8.1.1 nodeSets: – name: default count: 1 config: node.store.allow_mmap: false Where is your password stored and how do you access it? For this article we will assume that your deployment name is quickstart Default The elastic users password is stored in secrets along with all the other secrets. It usually takes form of <DEPLYMENT NAME>-es-elastic-user and to get the password you can… Continue Reading →

9

Updated deploy-elastic script – version 7

Posted on by jlim0930

UPDATE 3/30/2022 – another update was made to make fleet deployments easier. This is explained on this post The original post for the deploy-elastic.sh script is here. Made some updates so I thought I would post it on a new post. Changes: 8.x is now supported. You can use the script for any versions 6.x-8.x. full – modes are removed. The script is additive so if you deploy only the stack and want to add features on top you can run it again with the feature you want to add. You can add multiple features to your stack. Starting stack… Continue Reading →

0

Create secrets to use with your elastic stack to create keystore in kubernetes – minio example – helm example included

Posted on by jlim0930

For this example I will stand up a very simple minio server on my localhost. Create kubernetes secrets for the s3.client.default.access_key and s3.client.default.secret_key. Configure my elasticsearch pod with initContainer to install the repository-s3 plugin and secureSettings to create the keystore. minio server This is a very simple, not secure setup just for testing $ mkdir data $ wget https://dl.min.io/server/minio/release/linux-amd64/minio $ chmod +x minio $ ./minio server ./data API: http://192.168.1.251:9000 http://172.17.0.1:9000 http://192.168.122.1:9000 http://192.168.49.1:9000 http://127.0.0.1:9000 RootUser: minioadmin RootPass: minioadmin Console: http://192.168.1.251:36012 http://172.17.0.1:36012 http://192.168.122.1:36012 http://192.168.49.1:36012 http://127.0.0.1:36012 RootUser: minioadmin RootPass: minioadmin Command-line: https://docs.min.io/docs/minio-client-quickstart-guide $ mc alias set myminio http://192.168.1.251:9000 minioadmin minioadmin Instead of getting… Continue Reading →

2

elasticsearch stack monitoring on kubernetes(ECK)

Posted on by jlim0930

UPDATE: Since ECK operator 1.7 there is a new way to deploy stack monitoring. I’ve added the section at the end to cover for this. The old method still works and depending on your use case you can still use the old method versus new. The main difference between the OLD and NEW is that the OLD method uses filebeat and metricbeat pods while the new method uses filebeat and metricbeat sidecar containers. Lets get started with monitoring your elastic stack in kubernetes. There are multiple ways of doing this and I will divide this up into 3 sections, using… Continue Reading →

1

elasticsearch 8.x configure roles and users for stack monitoring

Posted on by jlim0930

A lot of changes were made with elasticsearch 8.0 release but the main one that affected me the most was the ability to use the default elastic(superuser role) to access system indices. According to https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights.html#_better_protection_for_system_indices we no longer have write access to system indices with the superuser role. When configuring stack monitoring using metricbeats this is an issue if you’ve been just using the elastic user to configure the monitoring. You can add allow_restricted_indices to the role to enable it back however its the lazy and insecure method. I’ve summarized the creation of roles and users for metricbeats and filebeats… Continue Reading →