Reload Prometheus ConfigMap without Restarting the Kubernetes Pod

We run Prometheus on Kubernetes. Every time we make changes to Prometheus ConfigMap, we end up restarting the pod so that the new configuration would be picked up.

While Prometheus configuration does not change very often, we would prefer to have a way to do this without downtime.

The Solution

Use Prometheus web flags to enable HTTP reloads and add configmap-reload container to reload the ConfigMap. The configmap-reload is a simple binary to trigger a reload when Kubernetes ConfigMaps are updated.

Starting with Prometheus 2.0, the --web.enable-lifecycle flag controls HTTP reloads and shutdowns of Prometheus. This is disabled by default, therefore we need to enable it by adding the parameter to our Prometheus deployment definition. When enabled, the reload endpoint will be accessible under the /-/reload path.

We have to update our Kubernetes Deployment configuration and add a new container for prometheus-server-configmap-reload.

See example below. Note that most of the deployment configuration that is not relevant to this change has been omitted.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: monitoring
spec:
  template:
    spec:
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-server
            defaultMode: 420
      containers:
        - name: prometheus-server-configmap-reload
          image: jimmidyson/configmap-reload:v0.5.0
          imagePullPolicy: IfNotPresent
          args:
            - '--volume-dir=/etc/config'
            - '--webhook-url=http://127.0.0.1:9090/-/reload'
          volumeMounts:
            - name: config-volume
              readOnly: true
              mountPath: /etc/config
        - name: prometheus-server
          image: quay.io/prometheus/prometheus:v2.37.0
          imagePullPolicy: IfNotPresent
          args:
            - '--config.file=/etc/config/prometheus.yml'
            - '--enable-feature=expand-external-labels'
            - '--storage.tsdb.path=/data'
            - '--storage.tsdb.retention.time=14d'
            - '--web.enable-lifecycle'
          [...]

Deploy the changes to Kubernetes. After that, the prometheus-server-configmap-reload container should reload all changes to the ConfigMap automatically without a need of restarting the Prometheus pod.

References

https://github.com/jimmidyson/configmap-reload