It is highly recommended to disable self-monitoring on your production Elasticsearch deployment for performance and built-in support in Elastic Cloud on Kubernetes. In this article we go over configuring the monitoring cluster.

I. Stack Monitoring configuration

a. Pod Discovery

When you run applications on containers, they become moving targets to the monitoring system so set the autodiscovery tag for your resource. We will use the stack-monitoring.elastic.co/type label for that purpose.

stack-monitoring.elastic.co/type: es # ElasticSearch tag
stack-monitoring.elastic.co/type: kb # Kibana tag

b. Log collection by Filebeat

Filebeat gets logs from all containers by default, you can set this hint to false to ignore the output of the container so that Filebeat won’t read or send logs from it. If default config is disabled, you can use this annotation to enable log retrieval only for containers with this set to true. The PROD deployment we will monitor will have this set to TRUE.

co.elastic.logs/enabled: "true"

c. Disable Legacy metric collections

We will be using Metricbeat to collect and ship logs from our PROD cluster to the monitoring cluster so legacy collections will be disabled with the following settings:

xpack.monitoring.elasticsearch.collection.enabled xpack.monitoring.kibana.collection.enabled to false

d. Enable Metrics collection

Since legacy collections were disabled, enable Metricbeat with the following setting:

xpack.monitoring.collection.enabled to true

e. Endpoints.

In this article we are only monitoring ElasticSearch and Kibana so metricbeats will use the default endpoints already created. If you add APM you must expose the endpoint.

II. Monitoring Cluster Deployment

Since we are deploying to a private AKS cluster that’s running our ECK we will deploy via Azure DevOps pipelines on a hosted agent that can access our cluster.

a. Deploy your monitoring ElasticSearch / Kibana cluster:

Deploying an ElasticSearch / Kibana Monitoring Cluster for Elastic Cloud on Kubernetes

b. ElasticSearch / Kibana

In the manifest we specify the version, # of pods and add the annotations for internal load balancer so that Kibana can generate and endpoint to Load Balance for us to login and use.

ElasticSearch / Kibana Monitoring Cluster with an Internal IP

c. Metricbeat and Filebeat

Metricbeat will use kubernetes as a provider and will do autodiscovery ONLY on pods with the setting: stack-monitoring.elastic.co/type: xx . Then filebeat will collect logs from pods that have the annotation: co.elastic.logs/enabled: “true”

Conditional Monitoring for ElasticSearch Monitoring Cluster with filebeat

Specify the username and secret that contains the password for the PROD cluster

Reference secret to monitor a PROD elasticsearch cluster.

III. Result:

Sources:

The following sources are directly from ElasticSearch and they are outdated. Please download and modify the manifest before deployment.

Deploy ES/Kibana:

kubectl apply -f https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt15e75034cdf96fbc/5f717762419b304ab54c37d8/monitoring-es-kb.yaml

Deploy Metricbeat:

kubectl apply -f https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/bltebb3cb94ae14c511/5f717762747ff04fb790086a/monitoring-beats-eck.yaml