Monitor Etcd Cluster with Grafana and Prometheus

We are going to configure Prometheus to monitor Etcd cluster.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configure Etcd Metrics URL

By default, Etcd cluster metrics are exposed locally. You can get the metrics endpoint URL by checking the file /etc/kubernetes/manifests/etcd.yaml on an Etcd cluster.

$ grep metrics /etc/kubernetes/manifests/etcd.yaml 
    - --listen-metrics-urls=http://127.0.0.1:2381

What we want to do is to make the metrics URL accessible from our local network so that we can configure Prometheus to use static configs.

For each of our control plane nodes, we are going to add an additional endpoint that resolves to the server’s internal IP address like this:

--listen-metrics-urls=http://127.0.0.1:2381,http://10.11.1.31:2381

Configure Prometheus Scraping

Add the following to your Prometheus config map and restart the pod:

  - job_name: 'etcd'
    static_configs:
      - targets: ['10.11.1.31:2381','10.11.1.32:2381','10.11.1.33:2381']

Add Grafana Dashboard for Etcd

Install a dashboard to monitor Etcd: https://grafana.com/grafana/dashboards/3070

The end result should look something like this:

Leave a Reply

Your email address will not be published.