Deploy Elasticsearch and Kibana on Kubernetes with Helm

We will install Elasticsearch and Kibana as well as set up basic security for the Elastic Stack plus secured HTTPS traffic.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configuration files used in this article can be found on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/logging/

The Plan

1. Install Helm.
2. Create an internal Certificate Authority (CA).
3. Create a wildcard certificate for Elasticsearch signed by the CA.
4. Install Elasticsearch 7.17 using Helm.
5. Install Kibana 7.17 using Helm.

Install Helm

On a Debian-based OS, do the following:

$ curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
$ sudo apt-get install -y apt-transport-https
$ echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm

Add Helm repository:

$ helm repo add elastic https://helm.elastic.co

Create Internal Certificate Authority (CA)

This section covers steps required to create a Root CA. Note that we have done this for the homelab environment here.

Generate a Root CA that is valid for 10 years:

$ openssl req -newkey rsa:2048 -keyout homelab-ca.key -nodes -x509 -days 3650 -out homelab-ca.crt

Verify X509v3 extensions:

$ openssl x509 -text -noout -in homelab-ca.crt | grep CA
     CA:TRUE

Create a wildcard certificate signed by the Root CA to be used with Elasticsearch and Kibana:

$ DOMAIN="wildcard.hl.test"
$ openssl genrsa -out "${DOMAIN}".key 2048 && chmod 0600 "${DOMAIN}".key

Generate a Certificate Sign Request (CSR):

$ openssl req -new -sha256 -key "${DOMAIN}".key -out "${DOMAIN}".csr

Sign the request with the Root CA:

$ openssl x509 -req -in "${DOMAIN}".csr -CA homelab-ca.crt -CAkey homelab-ca.key -CAcreateserial -out "${DOMAIN}".crt -days 1825 -sha256

Optional: import the Root CA in to your browser.

Install Elasticsearch on Kubernetes

Create logging namespace:

$ kubectl create namespace logging

Create a secret to store Elasticsearch credentials:

$ kubectl apply -f ./elastic-credentials-secret.yml

Create a secret to store Elasticsearch SSL certificates. We are using the Root CA to sign the certificate.

$ kubectl apply -f ./elastic-certificates-secret.yml

By default, the Elasticsearch security features are disabled when we have a basic license. To enable security features, we will use the xpack.security.enabled setting.

In order to enable TLS/SSL on the HTTP networking layer, which Elasticsearch uses to communicate with other clients, we will use the xpack.security.http.ssl.enabled setting.

Create a values file values-elasticsearch.yml for Elasticsearch:

---
clusterName: "elasticsearch"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"
  remote_cluster_client: "true"
  ml: "true"

replicas: 1
minimumMasterNodes: 1

protocol: https
httpPort: 9200
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"
  - name: "ELASTIC_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: "true"
    xpack.security.transport.ssl.enabled: "true"
    xpack.security.transport.ssl.supported_protocols: "TLSv1.2"
    xpack.security.transport.ssl.client_authentication: "none"
    xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.transport.ssl.verification_mode: "certificate"
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.supported_protocols: "TLSv1.2"
    xpack.security.http.ssl.client_authentication: "none"
    xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/elasticsearch/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "250m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "4Gi"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "freenas-nfs-csi"
  resources:
    requests:
      storage: 64Gi

service:
  enabled: true
  labels: {}
  labelsHeadless: {}
  type: LoadBalancer
  nodePort: ""
  annotations: {}
  httpPortName: https
  transportPortName: transport
  loadBalancerIP: "10.11.1.59"
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""

clusterHealthCheckParams: "wait_for_status=yellow&timeout=2s"

Deploy a single node Elasticsearch with authentication, certificates for TLS and custom values:

$ helm upgrade --install elasticsearch \
  elastic/elasticsearch \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-elasticsearch.yml

Elasticsearch endpoint will be available at https://10.11.1.59:9200/.

You can test it by using curl:

$ curl -sk -u "username:password" https://10.11.1.59:9200/ | jq
{
  "name": "elasticsearch-master-0",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "t6rPuP6NSn6IDaW98J0VWw",
  "version": {
    "number": "7.17.1",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "e5acb99f822233d62d6444ce45a4543dc1c8059a",
    "build_date": "2022-02-23T22:20:54.153567231Z",
    "build_snapshot": false,
    "lucene_version": "8.11.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}

Install Kibana on Kubernetes

Create a values file values-kibana.yml for Kibana:

---
elasticsearchHosts: "https://elasticsearch-master:9200"

replicas: 1

protocol: https
httpPort: 5601
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: "ELASTICSEARCH_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"
  - name: "ELASTICSEARCH_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: "true"
      key: "/usr/share/kibana/config/certs/tls.key"
      certificate: "/usr/share/kibana/config/certs/tls.crt"
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      clientAuthentication: "none"
      supportedProtocols: [ "TLSv1.2", "TLSv1.3" ]
    elasticsearch.ssl:
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      verificationMode: "certificate"
    newsfeed.enabled: "false"
    telemetry.enabled: "false"
    telemetry.optIn: "false"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/kibana/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "55m"
    memory: "512Mi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

service:
  type: LoadBalancer
  loadBalancerIP: "10.11.1.58"
  port: 5601
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  httpPortName: http

Deploy Kibana using authentication and TLS to connect to Elasticsearch:

$ helm upgrade --install kibana \
  elastic/kibana \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-kibana.yml

Kibana endpoint will be available at https://10.11.1.58:5601/.

Verify that pods are running:

$ kubectl get po -n logging
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          23h
kibana-kibana-5d8dc78bfb-4fqr2   1/1     Running   0          23h

Verify services:

$ kubectl get svc -n logging
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
elasticsearch-master            LoadBalancer   10.105.182.194   10.11.1.59    9200:31657/TCP,9300:32405/TCP   3d22h
elasticsearch-master-headless   ClusterIP      None             none          9200/TCP,9300/TCP               3d22h
kibana-kibana                   LoadBalancer   10.105.176.223   10.11.1.58    5601:31251/TCP                  3d21h

References

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-stack-security.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-settings.html