Deploy Elasticsearch and Kibana on Kubernetes with Helm

We will install Elasticsearch and Kibana as well as set up basic security for the Elastic Stack plus secured HTTPS traffic.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configuration files used in this article can be found on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/logging/

The Plan

  1. Install Helm.
  2. Create an internal Certificate Authority (CA).
  3. Create a wildcard certificate for Elasticsearch signed by the CA.
  4. Install Elasticsearch 7.17 using Helm.
  5. Install Kibana 7.17 using Helm.

Install Helm

On a Debian-based OS, do the following:

$ curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
$ sudo apt-get install -y apt-transport-https
$ echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm

Add Helm repository:

$ helm repo add elastic https://helm.elastic.co

Create Internal Certificate Authority (CA)

This section covers steps required to create a Root CA. Note that we have done this for the homelab environment here.

Generate a Root CA that is valid for 10 years:

$ openssl req -newkey rsa:2048 -keyout homelab-ca.key -nodes -x509 -days 3650 -out homelab-ca.crt

Verify X509v3 extensions:

$ openssl x509 -text -noout -in homelab-ca.crt | grep CA
     CA:TRUE

Create a wildcard certificate signed by the Root CA to be used with Elasticsearch and Kibana:

$ DOMAIN="wildcard.hl.test"
$ openssl genrsa -out "${DOMAIN}".key 2048 && chmod 0600 "${DOMAIN}".key

Generate a Certificate Sign Request (CSR):

$ openssl req -new -sha256 -key "${DOMAIN}".key -out "${DOMAIN}".csr

Sign the request with the Root CA:

$ openssl x509 -req -in "${DOMAIN}".csr -CA homelab-ca.crt -CAkey homelab-ca.key -CAcreateserial -out "${DOMAIN}".crt -days 1825 -sha256

Optional: import the Root CA in to your browser.

Install Elasticsearch on Kubernetes

Create logging namespace:

$ kubectl create namespace logging

Create a secret to store Elasticsearch credentials:

$ kubectl apply -f ./elastic-credentials-secret.yml

Create a secret to store Elasticsearch SSL certificates. We are using the Root CA to sign the certificate.

$ kubectl apply -f ./elastic-certificates-secret.yml

By default, the Elasticsearch security features are disabled when we have a basic license. To enable security features, we will use the xpack.security.enabled setting.

In order to enable TLS/SSL on the HTTP networking layer, which Elasticsearch uses to communicate with other clients, we will use the xpack.security.http.ssl.enabled setting.

Create a values file values-elasticsearch.yml for Elasticsearch:

---
clusterName: "elasticsearch"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"
  remote_cluster_client: "true"
  ml: "true"

replicas: 1
minimumMasterNodes: 1

protocol: https
httpPort: 9200
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"
  - name: "ELASTIC_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: "true"
    xpack.security.transport.ssl.enabled: "true"
    xpack.security.transport.ssl.supported_protocols: "TLSv1.2"
    xpack.security.transport.ssl.client_authentication: "none"
    xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.transport.ssl.verification_mode: "certificate"
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.supported_protocols: "TLSv1.2"
    xpack.security.http.ssl.client_authentication: "none"
    xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/elasticsearch/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "250m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "4Gi"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "freenas-nfs-csi"
  resources:
    requests:
      storage: 64Gi

service:
  enabled: true
  labels: {}
  labelsHeadless: {}
  type: LoadBalancer
  nodePort: ""
  annotations: {}
  httpPortName: https
  transportPortName: transport
  loadBalancerIP: "10.11.1.59"
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""

clusterHealthCheckParams: "wait_for_status=yellow&timeout=2s"

Deploy a single node Elasticsearch with authentication, certificates for TLS and custom values:

$ helm upgrade --install elasticsearch \
  elastic/elasticsearch \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-elasticsearch.yml

Elasticsearch endpoint will be available at https://10.11.1.59:9200/.

You can test it by using curl:

$ curl -sk -u "username:password" https://10.11.1.59:9200/ | jq
{
  "name": "elasticsearch-master-0",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "t6rPuP6NSn6IDaW98J0VWw",
  "version": {
    "number": "7.17.1",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "e5acb99f822233d62d6444ce45a4543dc1c8059a",
    "build_date": "2022-02-23T22:20:54.153567231Z",
    "build_snapshot": false,
    "lucene_version": "8.11.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}

Install Kibana on Kubernetes

Create a values file values-kibana.yml for Kibana:

---
elasticsearchHosts: "https://elasticsearch-master:9200"

replicas: 1

protocol: https
httpPort: 5601
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: "ELASTICSEARCH_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"
  - name: "ELASTICSEARCH_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: "true"
      key: "/usr/share/kibana/config/certs/tls.key"
      certificate: "/usr/share/kibana/config/certs/tls.crt"
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      clientAuthentication: "none"
      supportedProtocols: [ "TLSv1.2", "TLSv1.3" ]
    elasticsearch.ssl:
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      verificationMode: "certificate"
    newsfeed.enabled: "false"
    telemetry.enabled: "false"
    telemetry.optIn: "false"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/kibana/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "55m"
    memory: "512Mi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

service:
  type: LoadBalancer
  loadBalancerIP: "10.11.1.58"
  port: 5601
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  httpPortName: http

Deploy Kibana using authentication and TLS to connect to Elasticsearch:

$ helm upgrade --install kibana \
  elastic/kibana \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-kibana.yml

Kibana endpoint will be available at https://10.11.1.58:5601/.

Verify that pods are running:

$ kubectl get po -n logging
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          23h
kibana-kibana-5d8dc78bfb-4fqr2   1/1     Running   0          23h

Verify services:

$ kubectl get svc -n logging
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
elasticsearch-master            LoadBalancer   10.105.182.194   10.11.1.59    9200:31657/TCP,9300:32405/TCP   3d22h
elasticsearch-master-headless   ClusterIP      None             none          9200/TCP,9300/TCP               3d22h
kibana-kibana                   LoadBalancer   10.105.176.223   10.11.1.58    5601:31251/TCP                  3d21h

References

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-stack-security.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-settings.html

5 thoughts on “Deploy Elasticsearch and Kibana on Kubernetes with Helm

  1. hello, can I have details about certificates. I create a rootCA and a certificate with key. I change values inside elastic-certificates-secret.yml with my values (I put base64 encoded rootca and crt and key)
    After applying all I obtain “io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca”, at elastic start. When I left intact your elastic-certificates-secret.yml, it’s ok but with your certificates and not my certificates.
    thanks for the help

    • Hi Bruno, the error message suggests that the CA cert has not been provided. Did you generate a root CA certificate, and then used it to sign the ElasticSearch certificate?

  2. Hello,
    I followed this article but used AWS CA for certs and my cert is bind by passphrase and when I run my es version 7.10.2, I am getting below error:ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: IllegalStateException[Error parsing Private Key from: /usr/share/elasticsearch/config/certs/tls.key]; nested: NoSuchAlgorithmException[PBES2 SecretKeyFactory not available];
    Likely root cause: java.security.NoSuchAlgorithmException: PBES2 SecretKeyFactory not available
    Not sure, why I am getting this error. I tried to follow the steps as it is.

    • Hi, does your config provide the passphrase to ElasticSearch to decrypt the private key in some way? The error suggests that ElasticSearch could not read the private key, probably because it is encrypted and may need a passphrase to decrypt it.

Leave a Reply

Your email address will not be published. Required fields are marked *