Install and Configure Prometheus Monitoring on Kubernetes

We are going to deploy Prometheus to monitor Kubernetes nodes and more.

Pre-requisites

We are using our Kubernetes homelab to deploy Prometheus.

A working NFS server is required to create persistent volumes. Note that NFS server configuration is not covered in this article, but the way we set it up can be found here.

Our NFS server IP address is 10.11.1.20, and we have the following export configured for Prometheus:

/mnt/storage-k8s/nfs/prometheus

The owner:group of the NFS folder is set to 65534:65534, because of Prometheus deployment runAsUser: 65534.

Download Files from GitHub

Configuration files used in this article are hosted on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/

TLDR; Install and Configure Prometheus: All in One Go

Create a monitoring namespace:

$ kubectl create ns monitoring

Create everything with a single command:

$ kubectl apply -f ./kubernetes/prometheus/

Note to self: this could be a Helm chart.

Install and Configure Prometheus: Step by Step

Step by step instructions. Note that this homelab project is under development, therefore please refer to GitHub for any source code changes.

Create a Namespace

Create a monitoring namespace:

$ kubectl create ns monitoring

Create a Cluster Role and a Cluster Role Binding

Allow Prometheus fetch metrics from Kubernetes API:

$ kubectl apply -f ./kubernetes/prometheus/prometheus-cluster-role.yml

This is what the code looks like:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: monitoring

Create a Config Map

Prometheus scrapes metrics from instrumented jobs. The config map is the place to define your scrape config, what needs monitoring and how often.

My scrape config will not match yours, and it is therefore important that you modify the file prometheus-config-map.yml to meet your environment needs.

$ kubectl apply -f ./kubernetes/prometheus/prometheus-config-map.yml

This is what the code looks like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.rules: |-
    groups:
    - name: node.alerts
      rules:
      - alert: KubernetesHostHighCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 15m
        labels:
          severity: warning
          context: node
        annotations:
          summary: High load on node
          description: "Node {{ $labels.instance }} has more than 90% CPU load"
      - alert: KubernetesNodeDiskUsagePercentage
        expr: (100 - 100 * sum(node_filesystem_avail_bytes{device!~"tmpfs|by-uuid",fstype=~"xfs|ext"} / node_filesystem_size_bytes{device!~"tmpfs|by-uuid",fstype=~"xfs|ext"}) BY (instance,device)) > 85
        for: 5m
        labels:
          severity: warning
          context: node
        annotations:
          description: Node disk usage above 85%
          summary: Disk usage on target {{ $labels.instance }} at 85%
      - alert: KubernetesNodeContainerOOMKilled
        expr: sum by (instance) (changes(node_vmstat_oom_kill[24h])) > 3
        labels:
          severity: warning
          context: node
        annotations:
          description: More than 3 OOM killed pods on a node within 24h
          summary: More than 3 OOM killed pods on node {{ $labels.instance }} within 24h
  prometheus.yml: |-
    global:
      evaluation_interval: 60s
      scrape_interval: 15s
      scrape_timeout: 10s
    rule_files:
      - /etc/prometheus/prometheus.rules

    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - 'alertmanager.monitoring.svc:9093'

    scrape_configs:
      - job_name: 'kubernetes-apiservers'

        kubernetes_sd_configs:
        - role: endpoints
        scheme: https

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      #--------------------------------------------
      # Scrape config for nodes (kubelet).
      #--------------------------------------------
      - job_name: 'kubernetes-nodes'

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics

        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      #--------------------------------------------
      # Scrape config for service endpoints. This will
      # scrape node-exporter and kube-state-metrics services.
      # You need this for the following Grafana dashboards:
      # - Kubernetes Cluster Summary
      # - Node Exporter Full
      #--------------------------------------------
      - job_name: kubernetes-service-endpoints

        kubernetes_sd_configs:
        - role: endpoints

        relabel_configs:
        - action: keep
          regex: true
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_scrape
        - action: replace
          regex: (https?)
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_scheme
          target_label: __scheme__
        - action: replace
          regex: (.+)
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_path
          target_label: __metrics_path__
        - action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          source_labels:
          - __address__
          - __meta_kubernetes_service_annotation_prometheus_io_port
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_service_name
          target_label: kubernetes_name
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: kubernetes_node

      #--------------------------------------------
      # Scrape config for pods.
      #--------------------------------------------
      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

      #--------------------------------------------
      # Scrape static config for homelab.
      #--------------------------------------------
      - job_name: 'dns-master'
        static_configs:
          - targets: ['10.11.1.2:9153']
            labels:
              alias: admin1

      - job_name: 'dns-slave1'
        static_configs:
          - targets: ['10.11.1.3:9153']
            labels:
              alias: admin2

      - job_name: 'etcd'
        static_configs:
          - targets: ['10.11.1.31:2381','10.11.1.32:2381','10.11.1.33:2381']

      - job_name: 'haproxy'
        static_configs:
          - targets: ['10.11.1.30:9101']

      - job_name: 'admin1'
        static_configs:
          - targets: ['10.11.1.2:9100']
            labels:
              alias: admin1

      - job_name: 'admin2'
        static_configs:
          - targets: ['10.11.1.3:9100']
            labels:
              alias: admin2

      - job_name: 'kvm1'
        static_configs:
          - targets: ['10.11.1.21:9100']
            labels:
              alias: kvm1

      - job_name: 'kvm2'
        static_configs:
          - targets: ['10.11.1.22:9100']
            labels:
              alias: kvm2

      - job_name: 'kvm3'
        static_configs:
          - targets: ['10.11.1.23:9100']
            labels:
              alias: kvm3

      - job_name: 'k8s-master1'
        static_configs:
          - targets: ['10.11.1.31:9100']
            labels:
              alias: k8s-master1

      - job_name: 'k8s-master2'
        static_configs:
          - targets: ['10.11.1.32:9100']
            labels:
              alias: k8s-master2

      - job_name: 'k8s-master3'
        static_configs:
          - targets: ['10.11.1.33:9100']
            labels:
              alias: k8s-master3

      - job_name: 'k8s-node1'
        static_configs:
          - targets: ['10.11.1.34:9100']
            labels:
              alias: k8s-node1

      - job_name: 'k8s-node2'
        static_configs:
          - targets: ['10.11.1.35:9100']
            labels:
              alias: k8s-node2

      - job_name: 'k8s-node3'
        static_configs:
          - targets: ['10.11.1.36:9100']
            labels:
              alias: k8s-node3

      - job_name: 'mikrotik-exporter'
        scrape_interval: 30s
        static_configs:
          - targets:
            - 10.11.1.1 # your Mikrotik router IP you wish to monitor
        metrics_path: /metrics
        params:
          module: [my_router]
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: mikrotik-exporter:9436

Our Prometheus config map contains Alertmanager configuration that we’ll cover in this article.

Create a Persistent Volume

We want to keep monitoring data and store it on a persistent volume.

$ kubectl apply -f ./kubernetes/prometheus/prometheus-pv.yml

This is what the code looks like:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-prometheus
  namespace: monitoring
spec:
  capacity:
    storage: 32Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /mnt/storage-k8s/nfs/prometheus
    server: 10.11.1.20

Create a Persistent Volume Claim

Allow Prometheus to request persistent storage.

$ kubectl apply -f ./prometheus/prometheus-pvc.yml

This is what the code looks like:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc-prometheus
  namespace: monitoring
spec:
  storageClassName: nfs
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 32Gi

Create a Deployment Configuration

$ kubectl apply -f ./prometheus/prometheus-deployment.yml

This is what the code looks like:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      containers:
        - name: prometheus
          image: prom/prometheus:v2.24.1
          imagePullPolicy: IfNotPresent
          args:
            - "--storage.tsdb.retention.time=28d"
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
              protocol: TCP
          resources:
            limits:
              memory: "1Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "100m"
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
              readOnly: true
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      # See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
      #initContainers:
      #  - name: fix-nfs-permissions
      #    image: busybox
      #    command: ["sh", "-c", "chown -R 65534:65534 /prometheus"]
      #    securityContext:
      #      runAsUser: 0
      #      runAsNonRoot: false
      #    volumeMounts:
      #      - name: prometheus-storage-volume
      #        mountPath: /prometheus
      restartPolicy: Always
      terminationGracePeriodSeconds: 60
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
  
        - name: prometheus-storage-volume
          persistentVolumeClaim:
            claimName: nfs-pvc-prometheus

Create a Service

$ kubectl apply -f ./prometheus/prometheus-service.yml

This is what the code looks like:

---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: monitoring
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port:   '9090'
  labels:
    app: prometheus-server
spec:
  selector: 
    app: prometheus-server
  type: NodePort  
  ports:
    - port: 9090
      targetPort: 9090 
      nodePort: 30000

Check Monitoring Namespace

$ kubectl -n monitoring get all -l app=prometheus-server
NAME                              READY   STATUS    RESTARTS   AGE
pod/prometheus-7d8dc64fdc-2wzt2   1/1     Running   0          6d1h

NAME                         TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/prometheus-service   NodePort   10.104.220.139   none          9090:30000/TCP   14d

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus   1/1     1            1           6d5h

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-7d8dc64fdc   1         1         1       6d3h

We can access the Prometheus dashboard by using its service node port 30000.

What’s Next?

We will configure Grafana to use Prometheus as a data source.

Leave a Reply

Your email address will not be published. Required fields are marked *