Fleet server with logstash output – elastic-agent

I think one of the biggest issue with fleet and elastic-agent was that it was limited on the outputs so if you had tons and tons of elastic-agents in the wild it would all connect back to your elasticsearch and can overwhelm the cluster.

Starting 8.2 of elasticsearch logstash output type was introduced.. it is still in BETA at this time but I just tried it and it worked very nicely!

The steps to setup the logstash output is listed on https://www.elastic.co/guide/en/fleet/8.2/secure-logstash-connections.html but I wanted to try it out and document it.

Install & configure logstash

Install yum install logstash-8.2.2 -y
Lets find a server that has elasticsearch so that we can use elasticsearch-certutil to create our certs.

Create the CA

cd /usr/share/elasticsearch
bin/elasticsearch-certutil ca --pem

This will create elastic-stack-ca.zip Lets unzip it and you will find
./ca/ca.crt & ./ca/ca.key

Lets create the certificate

bin/elasticsearch-certutil cert --name something --ca-cert ./ca/ca.crt --ca-key ./ca/ca.key --dns your.fqdn --ip x.x.x.x --pem

This will create certificate-bundle.zip and when you unzip it you will find
./something/something.crt & ./something/something.key

Placing the certificates and reformat the logstash key to pkcs8

mkdir -p /etc/logstash/certs
mv ca/ca.* /etc/logstash/certs/
mv something/something.* /etc/logstash/certs/
cd /etc/logstash/certs
openssl pkcs8 -inform PEM -in something.key -topk8 -nocrypt -outform PEM -out something.pkcs8.key

Create logstash pipeline

create elastic-agent-pipeline.conf in /etc/logstash/conf.d with:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
    ssl_certificate => "/etc/logstash/certs/something.crt"
    ssl_key => "/etc/logstash/certs/something.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    hosts => "https://xxxxxx:9200"
    api_key => "xxxxxx:yyyyy"
    data_stream => true
    ssl => true
  }
}

You can generate your API key from Kibana -> Stack management -> APIKeys

Lets start logstash and monitor it to ensure that the pipeline starts.

systemctl start logstash
journalctl -u logstash -f

Configure the new output

Kibana -> Fleet -> Settings -> Add new output

mine looks like

One word of caution: Client SSL certificate key is the something.key NOT something.pkcs8.key

If your SAN is not set properly you might want to add the following into advanced YAML configuration

ssl:
  verification_mode: none

Lets create a new policy and for Output for integrations & Output for agent monitoring select our logstash output.

SAVE!!

Enroll and start your elastic-agents

elastic-agent enroll -f --url=https://xxxxxx:8220 --enrollment-token=xxxxxxxxxxxxxxxxxxxx

Wait until the agent is healthy and then goto /var/lib/elastic-agent/data/elastic-agent-xxxxxx and look at state.yml

The output section should look like

    outputs:
      211025e0-dd3e-11ec-b7c0-993e284ab2dc:
        hosts:
        - xxxxxxxxx:5044
        ssl:
          certificate: |-
            -----BEGIN CERTIFICATE-----
            MIIDXDCCAkSgAwIBAgIVAJUSUSKS8BNbsmhY8nblVbD9kNocMA0GCSqGSIb3DQEB
            CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
            ZXJhdGVkIENBMB4XDTIyMDUyNjIxNDE1MVoXDTI1MDUyNTIxNDE1MVowETEPMA0G
            A1UEAxMGY2xpZW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp0U3
            EBAB4GPLRRW/X/XU60rdTgn8G4aGfl6gKYEdgLOhmMd6StaI7AyhXsv0r/tFje2y
            2LgyKYs4QFnYEe+1k8nAs5ndpLxKFG/Ac8XsKMGTPsU7XFzzFoRI1fZEQNa/R95I
            v3Q3iKOa/VRZwqBD3Rzigj7/8rK8jQ/lz5Q5AyOV/r6bj38IR8/Zgu88APlU3ymR
            ds8YZQyHNM9r05J4iFtjH8heOjmyoZo+31u1hwC+jXPcv8Nx1AaGpNpy7J4sQDOJ
            V/XU4WBGoEaQ/WxuixKt3HVfpVJXxLbzJ0Q6u0rpliWaMrqHVAxlPjO3qNdePcGT
            qhxRBRU69FgNIOZzYwIDAQABo4GHMIGEMB0GA1UdDgQWBBQcDd9qpqLD50x7yHjM
            m6FT6LQRaDAfBgNVHSMEGDAWgBTde4R72qfhbJ9EqngJ31qmSJZUGDA3BgNVHREE
            MDAuhwQiR3mkgiYxNjQuMTIxLjcxLjM0LmJjLmdvb2dsZXVzZXJjb250ZW50LmNv
            bTAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQA+IFfMxZ9ry//nNiej/rNK
            gfq06OqTAvSpIL2vZXDenfFftvtkWWP80dakxeqnzA1CI/rFRkFv6wHniCm3X1Bk
            /pqAJoD4Zw3MJGKgI6qZrXdbcIS4/8MKxBvES7dUDNjcE72M6XOB7hDtfNlWUZ6y
            vgwK9mZNGBZAbFQD0J1IAEeEfAUlWyw/+mdJmu1ud+7Scky60Da8qgauJ2RtBkeY
            KG4cvaQ/qBQuPmjrs7K3fk0rtDt5ZWK2Oo1+iD3hxn360HQLMYPIwk6xsZDN2KsQ
            uWlTJ+Cv/QAWLaqB1DrCu31uJ0BXRms6pct1kOULKbdbB8ZcxOl45VtgcekIcIz2
            -----END CERTIFICATE-----
          certificate_authorities:
          - |-
            -----BEGIN CERTIFICATE-----
            MIIDSjCCAjKgAwIBAgIVAMC0VKcqarq3pGwdHcEtTgierh0kMA0GCSqGSIb3DQEB
            CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
            ZXJhdGVkIENBMB4XDTIyMDUyNjIxMzkyN1oXDTI1MDUyNTIxMzkyN1owNDEyMDAG
            A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
            ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCriPXcuhL8LVWXoX4RNCNB
            QeyiskMKyPTQb28HD2tXH+waeuN5BYe6HqJbfXC5AKyFb6C55N8yj11ikEnnghuU
            HeulPi6cUETRqOwDUgs8DlJo116ScuNaxfuDKGTefC7djIPDHHOiVwso8PVch3he
            WOg786yXeO7PROY8W5iD4f7QzuqVluV6kJxjuroLT2iRSvB21Fnzk/wwKEiCCXxG
            9/GcSO6gSQu+sQWp4KI36dHHM/V9ZORTvw5hc1Z4NyibfXbPPPJrEH22TNZEKETY
            TDQUQ7duhlFv/kiALm4Rrbl/X8a7tr4Z5Okuu+MQBs50dGS443Xxc2gVEBw3Vr31
            AgMBAAGjUzBRMB0GA1UdDgQWBBTde4R72qfhbJ9EqngJ31qmSJZUGDAfBgNVHSME
            GDAWgBTde4R72qfhbJ9EqngJ31qmSJZUGDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
            SIb3DQEBCwUAA4IBAQB1iujiV1axVRfn/eID1Oj0x7UNv7z/Md3f1/5Hm1vwTSc3
            RNHsnULUJBDss7zzV/duxYtstVtFD/6uri/0ScN8tu1CNk2QYeBF/3j+oNpRJ6GH
            t9Lkh68Z8A40sKmvue7aIPRnNqpy5vg/YQKlWiLyegdGQGFXGvW1GxU984VblmuA
            kV5tsJtjUHRtgU5GnDDRNI48DG42XTYhNV5Q7prQjN98/fBI2PXmanHWkrWDkCo+
            4FNp9P7wnEOaDMZwdsRdp3uLGxD/QK+/QX6kkVFGt4HsDQld2PS9FUu2V3zODHf/
            lSOUGRYomG64y7eLC/nSZfVgUNVi1gUw5yIu375r
            -----END CERTIFICATE-----
          key: |-
            -----BEGIN RSA PRIVATE KEY-----
            MIIEpAIBAAKCAQEAp0U3EBAB4GPLRRW/X/XU60rdTgn8G4aGfl6gKYEdgLOhmMd6
            StaI7AyhXsv0r/tFje2y2LgyKYs4QFnYEe+1k8nAs5ndpLxKFG/Ac8XsKMGTPsU7
            XFzzFoRI1fZEQNa/R95Iv3Q3iKOa/VRZwqBD3Rzigj7/8rK8jQ/lz5Q5AyOV/r6b
            j38IR8/Zgu88APlU3ymRds8YZQyHNM9r05J4iFtjH8heOjmyoZo+31u1hwC+jXPc
            v8Nx1AaGpNpy7J4sQDOJV/XU4WBGoEaQ/WxuixKt3HVfpVJXxLbzJ0Q6u0rpliWa
            MrqHVAxlPjO3qNdePcGTqhxRBRU69FgNIOZzYwIDAQABAoIBAAupcosppTUuRU6V
            6Lz5fYnsCHLiP6lG9Jsx4LZpZz5WXRwIPaP4bMe8TQc4GWFS7PNWEQPgD7XqxsC3
            ItkunmO6i2IE27maW7XXYWe7na6XvxAqHcHfmckbmZ0P6Yv1t5SnUD4c4v72nR3a
            uClZascrNjWWIS/Gu15AWf7iGs7M8DdEjHmMyuc3Q/UN7CjeDpjgKhwtv2fBbfX3
            gusUz258ouPRubKPGFtjxIjGx+SLlAxyJEkGrB4rSOD1bQr/8cY1ITbFsN84MU40
            mCDBBmQwSRUhdwm3e5fuTvpG1xs7TCORRI3RDqMjYanIA3/J4jASYSu7N8L4ID+4
            TD7CaCUCgYEA4dzFiTe17pH1buG9/JVhmo5O8ufdibgDedw6XrbG2hKzAzF8TX6k
            k32lZhhZR8CYsL8qZLtwqcdj2F4Sz7U3sRN3d6Xfb0QGM1WHcaC70Oxavy4Ia1S8
            HC3uTVVqeKgH3BDxiCvY95mTzbFC1EDoXKngFD0/AG1Wa6ifKrDdP+0CgYEAvZcA
            AWhdNlvV0AM4OHTj9v9kyNcPKcw+IreRx7BecsAruE7HnWlpKcvAohnnTUnnt5ix
            9PfNtSFVdmmcrG2Ds/NImbL4eFU7MX8jWylXXV9LZpqixNSYqXRxvDUV/pTGgEgx
            xpvcpvpxv8KPFeXmXExivXhA84BlvdB+9pEq9o8CgYEAoBiTVH5O0TpYlC4jMwTX
            2GwC/h7oY5QxspEuNrCsJyuWBC9xgqTAF2O6d7HOpxOXp8D5VcmfRFj3JwkXoG89
            eKDpz7+drwqysnRvu1VJi//dGyKQCTRY2kicX+ipvbitSYohCnq6IleT8vsw+a7A
            hd5L3UzEeZMG7OzrVaFkUskCgYB/oKUaTX1iWJQpAeHY76d+4RKhuVS12I6DpHv4
            p5wNN5y26CGssAdhYY5+EV4KQ5Q/ngU+WYsbQiNcIWCdGuQsQbw/66EksIu2mFXe
            yofLjZkeqz6jrlJO0Nq3zizOQq0WMoN/pfD2X2YvpvPX0/otbCaUUd23jlnvW+n8
            ZmE7tQKBgQCCEiwrGNeaVHe5P3odqx0Q8DSsSvZXPFT48teyQkciu/jO0PTLaeAk
            kW1lgzG0DthtaJZ52ZL/UdZN9IIbvT0IZJOaA2cfW53qsR4CwcbtT92lizfDCMiP
            Ug0ArMV93jT86zzS76U229AvC3PgKeEzayWYAZbo+NedmdbYHAta+A==
            -----END RSA PRIVATE KEY-----
        type: logstash
    revision: 2

Now to check the status of the agent

# elastic-agent status
Status: HEALTHY
Message: (no message)
Applications:
  * filebeat_monitoring    (HEALTHY)
                           Running
  * metricbeat_monitoring  (HEALTHY)
                           Running
  * metricbeat             (HEALTHY)
                           Running
  * filebeat               (HEALTHY)
                           Running

and to verify in kibana

and we are alive!!!!