Deploy Elasticsearch and Kibana on Kubernetes with Helm

Posted on by

We will install Elasticsearch and Kibana as well as set up basic security for the Elastic Stack plus secured HTTPS traffic.

[UPDATE: 2023] We have migrated from Elasticsearch to Loki because Elastic no longer support deployment via Helm.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configuration files used in this article can be found on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/kubernetes/elasticsearch/

The Plan

  1. Install Helm.
  2. Create an internal Certificate Authority (CA).
  3. Create a wildcard certificate for Elasticsearch signed by the CA.
  4. Install Elasticsearch 7.17 using Helm (Elasticsearch 8.x has not been tested).
  5. Install Kibana 7.17 using Helm.

Install Helm

On a Debian-based OS, do the following:

$ curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
$ sudo apt-get install -y apt-transport-https
$ echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm

Add Helm repository:

$ helm repo add elastic https://helm.elastic.co

Create Internal Certificate Authority (CA)

This section covers steps required to create a Root CA. Note that we have done this for the homelab environment here.

Generate a Root CA that is valid for 10 years:

$ openssl req -newkey rsa:2048 -keyout homelab-ca.key -nodes -x509 -days 3650 -out homelab-ca.crt

Verify X509v3 extensions:

$ openssl x509 -text -noout -in homelab-ca.crt | grep CA
     CA:TRUE

Create a wildcard certificate signed by the Root CA to be used with Elasticsearch and Kibana:

$ openssl genrsa -out tls.key 2048 && chmod 0600 tls.key

Generate a Certificate Sign Request (CSR):

$ openssl req -new -sha256 -key tls.key -out tls.csr

Sign the request with the Root CA:

$ openssl x509 -req -in tls.csr -CA homelab-ca.crt -CAkey homelab-ca.key -CAcreateserial -out tls.crt -days 1825 -sha256

Optional: import the Root CA in to your browser.

Install Elasticsearch on Kubernetes

Create logging namespace:

$ kubectl create namespace logging

Create a secret to store Elasticsearch credentials:

$ kubectl apply -f ./elastic-credentials-secret.yml

Create a secret to store Elasticsearch SSL certificates. We are using the Root CA to sign the certificate.

$ kubectl apply -f ./elastic-certificates-secret.yml

By default, the Elasticsearch security features are disabled when we have a basic license. To enable security features, we will use the xpack.security.enabled setting.

In order to enable TLS/SSL on the HTTP networking layer, which Elasticsearch uses to communicate with other clients, we will use the xpack.security.http.ssl.enabled setting.

Create a values file values-elasticsearch.yml for Elasticsearch:

---
clusterName: "elasticsearch"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"
  remote_cluster_client: "true"
  ml: "true"

replicas: 1
minimumMasterNodes: 1

protocol: https
httpPort: 9200
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"
  - name: "ELASTIC_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: "true"
    xpack.security.transport.ssl.enabled: "true"
    xpack.security.transport.ssl.supported_protocols: "TLSv1.2"
    xpack.security.transport.ssl.client_authentication: "none"
    xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.transport.ssl.verification_mode: "certificate"
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.supported_protocols: "TLSv1.2"
    xpack.security.http.ssl.client_authentication: "none"
    xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/elasticsearch/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "250m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "4Gi"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "freenas-nfs-csi"
  resources:
    requests:
      storage: 64Gi

service:
  enabled: true
  labels: {}
  labelsHeadless: {}
  type: LoadBalancer
  nodePort: ""
  annotations: {}
  httpPortName: https
  transportPortName: transport
  loadBalancerIP: "10.11.1.59"
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""

clusterHealthCheckParams: "wait_for_status=yellow&timeout=2s"

Deploy a single node Elasticsearch with authentication, certificates for TLS and custom values:

$ helm upgrade --install elasticsearch \
  elastic/elasticsearch \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-elasticsearch.yml

Elasticsearch endpoint will be available at https://10.11.1.59:9200/.

You can test it by using curl:

$ curl -sk -u "username:password" https://10.11.1.59:9200/ | jq
{
  "name": "elasticsearch-master-0",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "t6rPuP6NSn6IDaW98J0VWw",
  "version": {
    "number": "7.17.1",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "e5acb99f822233d62d6444ce45a4543dc1c8059a",
    "build_date": "2022-02-23T22:20:54.153567231Z",
    "build_snapshot": false,
    "lucene_version": "8.11.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}

Install Kibana on Kubernetes

Create a values file values-kibana.yml for Kibana:

---
elasticsearchHosts: "https://elasticsearch-master:9200"

replicas: 1

protocol: https
httpPort: 5601
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: "ELASTICSEARCH_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"
  - name: "ELASTICSEARCH_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: "true"
      key: "/usr/share/kibana/config/certs/tls.key"
      certificate: "/usr/share/kibana/config/certs/tls.crt"
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      clientAuthentication: "none"
      supportedProtocols: [ "TLSv1.2", "TLSv1.3" ]
    elasticsearch.ssl:
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      verificationMode: "certificate"
    newsfeed.enabled: "false"
    telemetry.enabled: "false"
    telemetry.optIn: "false"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/kibana/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "55m"
    memory: "512Mi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

service:
  type: LoadBalancer
  loadBalancerIP: "10.11.1.58"
  port: 5601
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  httpPortName: http

Deploy Kibana using authentication and TLS to connect to Elasticsearch:

$ helm upgrade --install kibana \
  elastic/kibana \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-kibana.yml

Kibana endpoint will be available at https://10.11.1.58:5601/.

Verify that pods are running:

$ kubectl get po -n logging
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          23h
kibana-kibana-5d8dc78bfb-4fqr2   1/1     Running   0          23h

Verify services:

$ kubectl get svc -n logging
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
elasticsearch-master            LoadBalancer   10.105.182.194   10.11.1.59    9200:31657/TCP,9300:32405/TCP   3d22h
elasticsearch-master-headless   ClusterIP      None             none          9200/TCP,9300/TCP               3d22h
kibana-kibana                   LoadBalancer   10.105.176.223   10.11.1.58    5601:31251/TCP                  3d21h

References

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-stack-security.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-settings.html

44 thoughts on “Deploy Elasticsearch and Kibana on Kubernetes with Helm”

  2. hello, can I have details about certificates. I create a rootCA and a certificate with key. I change values inside elastic-certificates-secret.yml with my values (I put base64 encoded rootca and crt and key)
    After applying all I obtain “io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca”, at elastic start. When I left intact your elastic-certificates-secret.yml, it’s ok but with your certificates and not my certificates.
    thanks for the help

    Reply

    • Hi Bruno, the error message suggests that the CA cert has not been provided. Did you generate a root CA certificate, and then used it to sign the ElasticSearch certificate?

  3. Hello,
    I followed this article but used AWS CA for certs and my cert is bind by passphrase and when I run my es version 7.10.2, I am getting below error:ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: IllegalStateException[Error parsing Private Key from: /usr/share/elasticsearch/config/certs/tls.key]; nested: NoSuchAlgorithmException[PBES2 SecretKeyFactory not available];
    Likely root cause: java.security.NoSuchAlgorithmException: PBES2 SecretKeyFactory not available
    Not sure, why I am getting this error. I tried to follow the steps as it is.

    Reply

    • Hi, does your config provide the passphrase to ElasticSearch to decrypt the private key in some way? The error suggests that ElasticSearch could not read the private key, probably because it is encrypted and may need a passphrase to decrypt it.

  4. How can we add these elasticsearch SSL certificates( tls.key, tls.crt and homelab-ca.crt) to elastic-certificates-secret.yml file, Could you help me on this

    Reply

  5. Hello,
    How can we add elasticsearch SSL certificates to elastic-certificates-secret.yml file.
    Could you help on this.

    Reply

    • Hi ,
      We have converted ssl certificates from plain text to base64
      Command used for converting : openssl base64 -in elasticsearch-ca.crt -out elasticsearch-ca.b64
      Still elastic search pods are not running throwing like ssl certificate issue.
      Could you please help us to understand issue here .
      logs attached here for reference:

    • Please check if you can get certificate data from your Kubernetes secrets. See example command below:

      kubectl -n logging get secret elastic-certificates -o yaml

      Decode the secret data that you get from above using base64 -d and see if it has your certificate.

  6. logs attached here for reference:
    {“type”: “server”, “timestamp”: “2023-04-14T09:28:22,379Z”, “level”: “ERROR”, “component”: “o.e.b.ElasticsearchUncaughtExceptionHandler”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “uncaught exception in thread [main]”,
    “stacktrace”: [“org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL KeyManagerFactory]; nested: MalformedInputException[Input length = 1];”,
    “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]”,
    “at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]”,

    Reply

  7. Hi ,
    After decoding the secrets we got like this but still facing ssl authentication issue , anything do we need to add it
    apiVersion: v1
    data:
    elastic-ca.crt: MIID6TCCAtGgAwIBAgIJANVcM6ZYRpA7MA0GCSqGSIb3DQEBCwUAMIGKMQswCQYDVQQGEwJJTjELMAkGA1UECAwCS04xEjAQBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UECgwIRVJJQ1NTT04xDTALBgNVBAsMBElEVU4xEDAOBgNVBAMMB2hhaG4xMzAxJjAkBgkqhkiG9w0BCQEWF3NhcmFsYS5rOTlAZXJpY3Nzb24uY29tMB4XDTIzMDQxNDA2MzUxOFoXDTMzMDQxMTA2MzUxOFowgYoxCzAJBgNVBAYTAklOMQswCQYDVQQIDAJLTjESMBAGA1UEBwwJQkFOR0FMT1JFMREwDwYDVQQKDAhFUklDU1NPTjENMAsGA1UECwwESURVTjEQMA4GA1UEAwwHaGFobjEzMDEmMCQGCSqGSIb3DQEJARYXc2FyYWxhLms5OUBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCeBGr/D05pn5HIPN7rbUgL+7Txr0OEgZKBigG/EC1c6uQkv69NzvgSzzqnAyWmgbnDxDixuSKyeqyFXWoSUF3/FdCHUSVdBHsxP1lY0uaqp8JPit4Ym/lihZXCtczbHExlhFbsUqu30BI3D4s3QljwhCsy6LaEsAUKeMFgTWBYy8z3qSYlEzqntePbGfEMYPWGEslkFu05kiGcMs/FoWzbdp2hA+qOCGeRy1DXkvzGrEIsP3IX5+S1eBUYiVfm/yWVihPy9h5HcMapQ9nUI49L3cWWq2QPTRLC7epwfbC+uSxV1jDeLQmJ296jSYsm48izvQFGPSfcDegQkd/3rDXlAgMBAAGjUDBOMB0GA1UdDgQWBBReynQAkqa1GsvTULUJMSOK66Ks5TAfBgNVHSMEGDAWgBReynQAkqa1GsvTULUJMSOK66Ks5TAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBBNI7c6skjCpH5N08J9ezqIaNYtdo8aGCXCcAEOYRXer2mtLhE9OAzrHIWMSdHC/BUIhMmnJXBWGyjLzVJkdKD133sfk1y9XKqqVTfzo9+GKYRjfAdS5zdKi8PXVAivsnUeJZ9IpzLEznCEbhlrVA3AEkv0/zSvSqHwtWnAo459t1TZbku7esKfnGbhlz3d6+CL2tReuLDPJx9l4XLRS1w965j/mJoLaWhNerEDU4aogrKJxD+5rubRxIFezLh7ZjFcKlR6SsTgGnAAVNokAxGfBRIvs22uJTX7+0ecEJ5XyXMm1gIfrDFCjF3eHXG5Ci2u7mKXpQMBIImG7HPfhx2
    tls.crt: MIIDlDCCAnwCCQC4dEuHK0c6zzANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCSU4xCzAJBgNVBAgMAktOMRIwEAYDVQQHDAlCQU5HQUxPUkUxETAPBgNVBAoMCEVSSUNTU09OMQ0wCwYDVQQLDARJRFVOMRAwDgYDVQQDDAdoYWhuMTMwMSYwJAYJKoZIhvcNAQkBFhdzYXJhbGEuazk5QGVyaWNzc29uLmNvbTAeFw0yMzA0MTQwNzMwMzFaFw0yODA0MTIwNzMwMzFaMIGMMQswCQYDVQQGEwJJTjELMAkGA1UECAwCS04xEjAQBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UECgwIRVJJQ1NTT04xDTALBgNVBAsMBElEVU4xEDAOBgNVBAMMB2hhaG4xMzAxKDAmBgkqhkiG9w0BCQEWGXNhcmFsYS5rLmV4dEBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaUJr/RqvKYAHOEITeJTCsTxwwsEYKyKLAGSiAd5CHBM5nGRsoETEYbqypdJPtk9MgHrc84TmFgDdaiLA8ijH0XO8wmj0OaMK6W4MZLzFifjaPjn+3hvx401keqRt7NjGlCNy6ICTNfuMnYB1Ya0UNj6Edph+LrYTl56Te014amZyhqfr0vBjmUKTeafzVqp55JL1/3O9IQe5RihK5eq4qCDD7hLGRMmBmXtS1XLjyET4XPLYz4bhO+JxE3N+mYe9uuHUoHQYVKe8driM1Fn5qUMipxxAonFtGQ7QLSTE942iXgkwHDWFGXpAJfX/2O8bYTyFKh7Wf6wMJ5RDDyldzAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHjL7Gqges0WAU5wY1PHK85KdURiSrVgDx/aQneghocXPlf/pGXt6IAJdnMoT2MLuV4Ic8/Vv0fedUFcgmF5zuWd3wM0wDtz3SkAheIZymkBK4lfXuDK1phIJ0Ak8I6Y/maKoQDXAT9WIfZIeScq3vwdqCweK3AXOXg/PCIKCSSTg46Y96PLRhZGyQ/pLjnCJg8A4R50eyK911EbdfYWb/rfdEzJqf3v1q0+UuKoxknHkHiMty2S3d4Wi7bJWz+9zfI0EFZO70bQahbChFE0CONHMMvVRNErLNRJpDDqpI8myyEq4hp2aLT52X4//SeumylrjRgrU0sqMF9zFhA42Aw=
    tls.key: MIIEpAIBAAKCAQEAmlCa/0arymABzhCE3iUwrE8cMLBGCsiiwBkogHeQhwTOZxkbKBExGG6sqXST7ZPTIB63POE5hYA3WoiwPIox9FzvMJo9DmjCuluDGS8xYn42j45/t4b8eNNZHqkbezYxpQjcuiAkzX7jJ2AdWGtFDY+hHaYfi62E5eek3tNeGpmcoan69LwY5lCk3mn81aqeeSS9f9zvSEHuUYoSuXquKggw+4SxkTJgZl7UtVy48hE+Fzy2M+G4TvicRNzfpmHvbrh1KB0GFSnvHa4jNRZ+alDIqccQKJxbRkO0C0kxPeNol4JMBw1hRl6QCX1/9jvG2E8hSoe1n+sDCeUQw8pXcwIDAQABAoIBAGPectFvPVL2G3TvP+49B2kcsdPY4RutiZblMU8jEkgvlh0nJGoj5jA3wErTBcGl/+czuccOxBlgR3KyL8yea4IIe0xrJcSjjBLrksbDwiRKJql1wbZXCCJyNaUgMIbnJe329FMI5oiP7BbZn2RP2BrUr6Lulk/hdrcL2anUBX0UmALvkZFz7MiuDuGQjsb51DxZvEpE++XRVDwL2iNw3mub+GuJ0DjSOkPg/RAGahJFZL9eNsf8Me4WanMCPIyzekSB6b798vnEoYBmU7S09TwVtrt6vatkBv84zWbC88Y5zoQdNL0MUqHq6uCago2Cf8hIOVdrGimrP66ZZTRTX+ECgYEAx88GRbtFNPkV1ED+ALUD3XtaR122xLkinHiIlk1n+LatNP5gvSy7xxzfa0VqZvuVNhuvK+HS/SO5C1NiHQaq6H0+li8MQxv5P/+4xB/ZmCkg19BuWKSiXIoaP3IBYe0YE58M1uMDNNFAQXVGoi/zuSWDAhwgHN0eCabhemB3JGkCgYEAxbZPa/fxhewgS+c+Cep+usYCOlTLGRMg8vUeqWSIzhLUVruv/D9EsE6JOGWUj3NqYGO8b+gsMrklgz7gtOSYPS43+7jDM9fvg7a8elWJ5175Vihk0U++u9n3t5kfFueLpn1hLzWJeCscXucKXSQCnEtO94dSEE/xrOcbFGp98XsCgYEAiaAibON53uv2yjVd/3SvTd4KJ//3xPbUTTyEsCpvBBQnp8nTLlpimNpdVZEBoh+F/jgBZ1NrtarmaVOsz9to2yxxcJbFdnANNbTZOlXJ1hH2KlDJwMrdfqotPCg1pZLes50pBdZlvfqnrK6v0UUANjYNT+W5hMgVSYal5loNlokCgYBXJwMhi91KdIjUDK4bPuP4PvqSbfhNKFJ45rZKY0eu1zwEs51i6xzDPwtb8eMnzO+SZ8ST69s3zMeNcDUraCz3ox0IeCyL+N7ax72Ti4tLk2EDqrCuV6GzOuToaLX8qbq0fEZPwiDD+PT7nIrD/fCspsG7eUoiaKsW1ZTpTfwQawKBgQC20xXN5O8sSoJ3RCbMjxjxibrqSNxvEnpGNQx+P+15zqMKByajO6zGkk5b8svtftm3pFVdUe3t0EoaUHEDoKbzTiNIUCeDQSjbtd/qyDciEkgrI1FAGBR1N7PpWpUX1CsZ27KVREgLND+VIK5gq++40ASGi7pqIErfkoIWRpYKKQ==
    kind: Secret
    metadata:
    annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
    {“apiVersion”:”v1″,”data”:{“elastic-ca.crt”:”MIID6TCCAtGgAwIBAgIJANVcM6ZYRpA7MA0GCSqGSIb3DQEBCwUAMIGKMQswCQYD\nVQQGEwJJTjELMAkGA1UECAwCS04xEjAQBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UE\nCgwIRVJJQ1NTT04xDTALBgNVBAsMBElEVU4xEDAOBgNVBAMMB2hhaG4xMzAxJjAk\nBgkqhkiG9w0BCQEWF3NhcmFsYS5rOTlAZXJpY3Nzb24uY29tMB4XDTIzMDQxNDA2\nMzUxOFoXDTMzMDQxMTA2MzUxOFowgYoxCzAJBgNVBAYTAklOMQswCQYDVQQIDAJL\nTjESMBAGA1UEBwwJQkFOR0FMT1JFMREwDwYDVQQKDAhFUklDU1NPTjENMAsGA1UE\nCwwESURVTjEQMA4GA1UEAwwHaGFobjEzMDEmMCQGCSqGSIb3DQEJARYXc2FyYWxh\nLms5OUBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQCeBGr/D05pn5HIPN7rbUgL+7Txr0OEgZKBigG/EC1c6uQkv69NzvgSzzqnAyWm\ngbnDxDixuSKyeqyFXWoSUF3/FdCHUSVdBHsxP1lY0uaqp8JPit4Ym/lihZXCtczb\nHExlhFbsUqu30BI3D4s3QljwhCsy6LaEsAUKeMFgTWBYy8z3qSYlEzqntePbGfEM\nYPWGEslkFu05kiGcMs/FoWzbdp2hA+qOCGeRy1DXkvzGrEIsP3IX5+S1eBUYiVfm\n/yWVihPy9h5HcMapQ9nUI49L3cWWq2QPTRLC7epwfbC+uSxV1jDeLQmJ296jSYsm\n48izvQFGPSfcDegQkd/3rDXlAgMBAAGjUDBOMB0GA1UdDgQWBBReynQAkqa1GsvT\nULUJMSOK66Ks5TAfBgNVHSMEGDAWgBReynQAkqa1GsvTULUJMSOK66Ks5TAMBgNV\nHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBBNI7c6skjCpH5N08J9ezqIaNY\ntdo8aGCXCcAEOYRXer2mtLhE9OAzrHIWMSdHC/BUIhMmnJXBWGyjLzVJkdKD133s\nfk1y9XKqqVTfzo9+GKYRjfAdS5zdKi8PXVAivsnUeJZ9IpzLEznCEbhlrVA3AEkv\n0/zSvSqHwtWnAo459t1TZbku7esKfnGbhlz3d6+CL2tReuLDPJx9l4XLRS1w965j\n/mJoLaWhNerEDU4aogrKJxD+5rubRxIFezLh7ZjFcKlR6SsTgGnAAVNokAxGfBRI\nvs22uJTX7+0ecEJ5XyXMm1gIfrDFCjF3eHXG5Ci2u7mKXpQMBIImG7HPfhx2\n”,”tls.crt”:”MIIDlDCCAnwCCQC4dEuHK0c6zzANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMC\nSU4xCzAJBgNVBAgMAktOMRIwEAYDVQQHDAlCQU5HQUxPUkUxETAPBgNVBAoMCEVS\nSUNTU09OMQ0wCwYDVQQLDARJRFVOMRAwDgYDVQQDDAdoYWhuMTMwMSYwJAYJKoZI\nhvcNAQkBFhdzYXJhbGEuazk5QGVyaWNzc29uLmNvbTAeFw0yMzA0MTQwNzMwMzFa\nFw0yODA0MTIwNzMwMzFaMIGMMQswCQYDVQQGEwJJTjELMAkGA1UECAwCS04xEjAQ\nBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UECgwIRVJJQ1NTT04xDTALBgNVBAsMBElE\nVU4xEDAOBgNVBAMMB2hhaG4xMzAxKDAmBgkqhkiG9w0BCQEWGXNhcmFsYS5rLmV4\ndEBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCa\nUJr/RqvKYAHOEITeJTCsTxwwsEYKyKLAGSiAd5CHBM5nGRsoETEYbqypdJPtk9Mg\nHrc84TmFgDdaiLA8ijH0XO8wmj0OaMK6W4MZLzFifjaPjn+3hvx401keqRt7NjGl\nCNy6ICTNfuMnYB1Ya0UNj6Edph+LrYTl56Te014amZyhqfr0vBjmUKTeafzVqp55\nJL1/3O9IQe5RihK5eq4qCDD7hLGRMmBmXtS1XLjyET4XPLYz4bhO+JxE3N+mYe9u\nuHUoHQYVKe8driM1Fn5qUMipxxAonFtGQ7QLSTE942iXgkwHDWFGXpAJfX/2O8bY\nTyFKh7Wf6wMJ5RDDyldzAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHjL7Gqges0W\nAU5wY1PHK85KdURiSrVgDx/aQneghocXPlf/pGXt6IAJdnMoT2MLuV4Ic8/Vv0fe\ndUFcgmF5zuWd3wM0wDtz3SkAheIZymkBK4lfXuDK1phIJ0Ak8I6Y/maKoQDXAT9W\nIfZIeScq3vwdqCweK3AXOXg/PCIKCSSTg46Y96PLRhZGyQ/pLjnCJg8A4R50eyK9\n11EbdfYWb/rfdEzJqf3v1q0+UuKoxknHkHiMty2S3d4Wi7bJWz+9zfI0EFZO70bQ\nahbChFE0CONHMMvVRNErLNRJpDDqpI8myyEq4hp2aLT52X4//SeumylrjRgrU0sq\nMF9zFhA42Aw=\n”,”tls.key”:”MIIEpAIBAAKCAQEAmlCa/0arymABzhCE3iUwrE8cMLBGCsiiwBkogHeQhwTOZxkb\nKBExGG6sqXST7ZPTIB63POE5hYA3WoiwPIox9FzvMJo9DmjCuluDGS8xYn42j45/\nt4b8eNNZHqkbezYxpQjcuiAkzX7jJ2AdWGtFDY+hHaYfi62E5eek3tNeGpmcoan6\n9LwY5lCk3mn81aqeeSS9f9zvSEHuUYoSuXquKggw+4SxkTJgZl7UtVy48hE+Fzy2\nM+G4TvicRNzfpmHvbrh1KB0GFSnvHa4jNRZ+alDIqccQKJxbRkO0C0kxPeNol4JM\nBw1hRl6QCX1/9jvG2E8hSoe1n+sDCeUQw8pXcwIDAQABAoIBAGPectFvPVL2G3Tv\nP+49B2kcsdPY4RutiZblMU8jEkgvlh0nJGoj5jA3wErTBcGl/+czuccOxBlgR3Ky\nL8yea4IIe0xrJcSjjBLrksbDwiRKJql1wbZXCCJyNaUgMIbnJe329FMI5oiP7BbZ\nn2RP2BrUr6Lulk/hdrcL2anUBX0UmALvkZFz7MiuDuGQjsb51DxZvEpE++XRVDwL\n2iNw3mub+GuJ0DjSOkPg/RAGahJFZL9eNsf8Me4WanMCPIyzekSB6b798vnEoYBm\nU7S09TwVtrt6vatkBv84zWbC88Y5zoQdNL0MUqHq6uCago2Cf8hIOVdrGimrP66Z\nZTRTX+ECgYEAx88GRbtFNPkV1ED+ALUD3XtaR122xLkinHiIlk1n+LatNP5gvSy7\nxxzfa0VqZvuVNhuvK+HS/SO5C1NiHQaq6H0+li8MQxv5P/+4xB/ZmCkg19BuWKSi\nXIoaP3IBYe0YE58M1uMDNNFAQXVGoi/zuSWDAhwgHN0eCabhemB3JGkCgYEAxbZP\na/fxhewgS+c+Cep+usYCOlTLGRMg8vUeqWSIzhLUVruv/D9EsE6JOGWUj3NqYGO8\nb+gsMrklgz7gtOSYPS43+7jDM9fvg7a8elWJ5175Vihk0U++u9n3t5kfFueLpn1h\nLzWJeCscXucKXSQCnEtO94dSEE/xrOcbFGp98XsCgYEAiaAibON53uv2yjVd/3Sv\nTd4KJ//3xPbUTTyEsCpvBBQnp8nTLlpimNpdVZEBoh+F/jgBZ1NrtarmaVOsz9to\n2yxxcJbFdnANNbTZOlXJ1hH2KlDJwMrdfqotPCg1pZLes50pBdZlvfqnrK6v0UUA\nNjYNT+W5hMgVSYal5loNlokCgYBXJwMhi91KdIjUDK4bPuP4PvqSbfhNKFJ45rZK\nY0eu1zwEs51i6xzDPwtb8eMnzO+SZ8ST69s3zMeNcDUraCz3ox0IeCyL+N7ax72T\ni4tLk2EDqrCuV6GzOuToaLX8qbq0fEZPwiDD+PT7nIrD/fCspsG7eUoiaKsW1ZTp\nTfwQawKBgQC20xXN5O8sSoJ3RCbMjxjxibrqSNxvEnpGNQx+P+15zqMKByajO6zG\nkk5b8svtftm3pFVdUe3t0EoaUHEDoKbzTiNIUCeDQSjbtd/qyDciEkgrI1FAGBR1\nN7PpWpUX1CsZ27KVREgLND+VIK5gq++40ASGi7pqIErfkoIWRpYKKQ==\n”},”kind”:”Secret”,”metadata”:{“annotations”:{},”name”:”elastic-certificates”,”namespace”:”reg-eck”},”type”:”kubernetes.io/tls”}
    creationTimestamp: “2023-04-14T08:56:13Z”
    managedFields:
    – apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
    f:data:
    .: {}
    f:elastic-ca.crt: {}
    f:tls.crt: {}
    f:tls.key: {}
    f:metadata:
    f:annotations:
    .: {}
    f:kubectl.kubernetes.io/last-applied-configuration: {}
    f:type: {}
    manager: kubectl
    operation: Update
    time: “2023-04-14T08:56:13Z”
    name: elastic-certificates
    namespace: reg-eck
    resourceVersion: “203114252”
    uid: 755e3db4-a9ff-4b2d-8b1d-add16a2f5223
    type: kubernetes.io/tls

    Reply

    • When you decode your base64 data from the secret, do you get certificates/keys in plain text? If you don’t, then that would be the problem. You need to encode plain text certificates/keys using base64 and add them to the secret.

    • Used above commands for generating root ca certificates, domain crt and keys
      Passed Internal Certificate Authority (CA) commands for generating these crts and keys (domain.crt,domain.csr, root-ca.crt, root-ca.key and root-ca.srl) from these keys and crts converted to base.64 by using this command:

      After generating the crt.b64 files added to elasticertificate.yaml
      Applied to these secret to namespace .
      Did i missed any steps here

    • That looks OK to me. Have you verified that you can decode base64 encoded secret data and see plaintext content of you certificates/keys?

  8. Hi Lisenet,

    Decoded base64 to plain text .
    can see below text its showing valid details.

    Is there any other way to check this.

    Reply

  9. Have doubt on domain , which domain should provide whether it should be kibana url or elasticsearch url

    Reply

  10. Hi,

    Shiwa here, I had used your solution along with https://blog.searce.com/deploying-a-secure-elasticsearch-environment-on-kubernetes-deb0f981ddf5
    But when I try to bring up the elasticsearch pods those are failing with error like

    ERROR
    —

    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,904Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “stopping …”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,908Z”, “level”: “INFO”, “component”: “o.e.x.w.WatcherService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “stopping watch service, reason [shutdown initiated]”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,908Z”, “level”: “INFO”, “component”: “o.e.x.m.p.l.CppLogMessageHandler”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[controller/180] [Main.cc@174] ML controller exiting”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,909Z”, “level”: “INFO”, “component”: “o.e.x.m.p.NativeController”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Native controller process has stopped – no new native processes can be started”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,909Z”, “level”: “INFO”, “component”: “o.e.x.w.WatcherLifeCycleService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “watcher has stopped and shutdown”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,174Z”, “level”: “INFO”, “component”: “o.e.t.ClusterConnectionManager”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “transport connection to [{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw}] closed by remote”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,177Z”, “level”: “INFO”, “component”: “o.e.c.r.a.AllocationService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Cluster health status changed from [GREEN] to [YELLOW] (reason: [{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw} reason: disconnected]).”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,179Z”, “level”: “INFO”, “component”: “o.e.c.s.MasterService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “node-left[{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw} reason: disconnected], term: 24, version: 474, delta: removed {{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw}}”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,183Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “master node changed {previous [{elasticsearch-master-0}{IwKBzipXTeaOzFXgoj4x9Q}{C91kj8TuTpWcUVdeSWnk-w}{10.50.5.209}{10.50.5.209:9300}{cdfhilmrstw}], current []}, term: 24, version: 473, reason: becoming candidate: Publication.onCompletion(false)”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,184Z”, “level”: “WARN”, “component”: “o.e.c.s.MasterService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “failing [node-left[{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw} reason: disconnected]]: failed to commit cluster state version [474]”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” ,
    “stacktrace”: [“org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1772) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1679) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.start(Publication.java:61) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1380) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:305) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]”,
    “at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]”,
    “at java.lang.Thread.run(Thread.java:833) [?:?]”,
    “Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:163) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “… 14 more”] }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,187Z”, “level”: “ERROR”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “unexpected failure during [node-left]”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” ,
    “stacktrace”: [“org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1772) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1679) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.start(Publication.java:61) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1380) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:305) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]”,
    “at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]”,
    “at java.lang.Thread.run(Thread.java:833) [?:?]”,
    “Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:163) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “… 14 more”] }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,196Z”, “level”: “WARN”, “component”: “o.e.c.NodeConnectionsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “failed to connect to {elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw}{ml.machine_memory=2147483648, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true} (tried [1] times)”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” ,
    “stacktrace”: [“org.elasticsearch.transport.ConnectTransportException: [elasticsearch-master-1][10.50.9.173:9300] connect_exception”,
    “at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1047) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:279) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.17.1.jar:7.17.1]”,
    “at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]”,
    “at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]”,
    “at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]”,
    “at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]”,
    “at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:58) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]”,
    “at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]”,
    “at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]”,
    “at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]”,
    “at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]”,
    “at java.lang.Thread.run(Thread.java:833) [?:?]”,
    “Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.50.9.173/10.50.9.173:9300”,
    “Caused by: java.net.ConnectException: Connection refused”,
    “at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]”,
    “at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]”,
    “at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]”,
    “at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]”,
    “at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]”,
    “… 7 more”] }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,439Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “stopped”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,440Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “closing …”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,481Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “closed”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }

    Secrets I had used
    [azadmin@l21q1382s005001 gpp]$ kubectl get secret elastic-secret -o jsonpath='{.data}’
    {“ES_PASSWORD”:”cGFzc3dvcmQ=”,”ES_USERNAME”:”ZWxhc3RpYw==”}[azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$ echo “cGFzc3dvcmQ=” | base64 –decode
    password[azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$ echo “ZWxhc3RpYw==” | base64 –decode
    elastic[azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$ kubectl get secret elastic-certificates -o jsonpath='{.data}’
    {“elastic-certificates.p12″:”MIIOCAIBAzCCDbIGCSqGSIb3DQEHAaCCDaMEgg2fMIINmzCCBbIGCSqGSIb3DQEHAaCCBaMEggWfMIIFmzCCBZcGCyqGSIb3DQEMCgECoIIFQDCCBTwwZgYJKoZIhvcNAQUNMFkwOAYJKoZIhvcNAQUMMCsEFDzss3hKqQoNyk1i5YUSNgw8CUBkAgInEAIBIDAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQvQpeljqUxniTMTiX7Hq0FgSCBNDf60kzS3enXucBMvbUPRvsshPGiXdYuIMi82eTAxmE2Z88JX1bKyQG1b/bAAAqKOEOvEHTuear0bjQ0xqkl0YFWpnU87mxMGrFFDfdvocrpruwxLKKrUU+VZxwWkyhy64ev5No0IQjMZMhFy6Hl/kCu14iPPEtmPXDT+OMwJ49wENTx8H9B4mLfqNsp/rZJ4zOdqOeCw/Irr0TlJTiGBzfgtrKH9T0xhIaoFl49vbqnABWZZcZqGXIy0ap8jc0Vpp2DPb8V+/yqja/13mllGPx9GxYpkdSW2aNOwwcRXH00luVwu4pWTGlQNNPlHPYov+b29E4XevnA5NNjP25upuH19pMgfv0XAap2vssugVW538kL7g6WY0zD4eRpSYM7yzKEoaZH7CPMJ39EbD0eucoVFRhA0KB1JsErDdDBsj077NBYxJ+CQWIhbTnS7Kb7Rq2VpLYqr8cP4dUEgfBEvbg3MVpVH7F62nUqfnSKE7js4dy382Xpki1aL0LLktnMAUebensjl8oLg4o9P4f0VqJ1YAAXslJxOsNI8h4rR0Yff8qu80oaFprypkPVPqwdttRXMOoNb0lTlAA1zjctZAATgU4434VN3BHDfozjXk+EXoaoH7r2Ww5EzCKQFo/xGpVZW6fN8R8jLb6y3Eopa/pQUfscmVTyyE4IAlSfJMehU4urgbyfm7clBpmW5RtfyEgDaeZ1J+JYxrZNv6Z6BdaAxB1gShsSLikFPg6w3Mpca8UHrbIM1y/UTkBNjmFLDzSlAxq43oguURBTf8eipByhCW9fdQpcOwlIFdUtaUft9bvEwLZxjhviRO3il2KRCcuFxeAtYN7fTiqC1pm8+JFMyoAnBSzktzVFF4yZE6RxA0ShHE46zDh8sO4Q3rjmta0Z2J5rATcJ3h+NDrWNhg+yGhQH5ruwyqHoJOT+2bm8G0yZYN0JbqLTgC7vrjiR49z6nvnY6ZsAnlNMj+JLPAbf9aPpohf31XJ2KgujiTXvB74L4l9a5jWkYm1Nq7djf9Rt+dXGx/41hLhBOfqo/OLbO32g/vMASUtcHapGeOH7XYew1ImeVQd4aswJMNg4RE6E+RF5nOdoI8jbWhxHQqVTYEPSuHfTrrNM3mou6hCL9VMLlkxDQSvyGk73ynZRnZFp6uhtqWt3vSOr1KWnvIZCJZ8J4V9YoADmfiIcJg/o3BmbYqDt+k33UvxXdo/XakfCSPZLtGHrZgiKaahlmu+BvbDyEsCQsyDMznRCnHkNSfcgRplaikHIn7+E3WPyG/BNAgI1saza33r6nwcrckVMjtexsN/MQ3YrF3H0K5THvN0UcDjq0obvEXOOgHt6ItNi1jkGs4Jm9dGtu5dmfQ49Yl2eN1gi0aB1LEzSTqronSk/6SlYUJzxd7jjIWUl0x1hx25yyqBuyxcTlk7D4cSKskPbDok7w91Yq4qkTtuBDCr5UC+gTWEpYiwk03xHyO8KBqGr9tR8XIVNGX/9rAe9o5gL/pOlVhWDlAzOBbby3sU6Q4dL2m4JmsKy4BQvwAMwWMrGyCKcZuML9WEWteoX4lzu1vsbJgwE4Z42KjIAjf150A06Tf7woirja8CwejaL/JsWprfWBjvLJWyIwaDZAbmIZ8mBAcskwkD73jukTFEMB8GCSqGSIb3DQEJFDESHhAAaQBuAHMAdABhAG4AYwBlMCEGCSqGSIb3DQEJFTEUBBJUaW1lIDE3MDUwNTc4OTI5MjgwggfhBgkqhkiG9w0BBwagggfSMIIHzgIBADCCB8cGCSqGSIb3DQEHATBmBgkqhkiG9w0BBQ0wWTA4BgkqhkiG9w0BBQwwKwQUkNrezGXW41fq85vOMFlEOIBaVj4CAicQAgEgMAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBABtfdn4q80sOdnX/oecwMzgIIHUPmdEEg+gmpBUX1UJNJ/+dFIlgGPhMZbzminH1iPVgUollZo4An7oprdLioGQI7uh4BBIcn4s168rm+MmR0e0Kzts+Rt0Sl9QasgrYi0eAydxgLJd701KkJH8HWfjmWlW+LPXT/IPUnI+WaooSoHXy1RVOmP3T6vQAPrCEMzguuGUupReqqIQu+Kx95JQBfXDQZ2raQ0hsV75KkcVfkJt05LmkMD8GvQ0ydYBz8BX30iK+hriFdLUPXrE+SFyGHP5rz4H9u3TDZqDM2oDMAWwKbbfZCXko+yBWri3SkQ4Nhut8LgusbSfZwBWuG7tjpSim6oynyFXEqSiInPP3UZ/JSVMDPNAaPv8MLnCGKS+WlD0wGnffdCuQpnjRpeyK2UK5tOWYIUPtikh1IVaXLUlkW8McODx41KOwqC18KJND1y/SCJw35I41Y2BsaPtRvei5CfQ7Lu8L1M7OnQ0FzZwde2wlh9V2qONmcv978/dPBt8LLJ94Ke1pJSxmcoDEnTtPC4wonuYp3oc60B5emo2SfgwZOHN/3GDPAh9/AzbuXIPli3D9CD+Sz4JLAUXHkYNJs6bqvNJPi2GrXCIUWlKq6VmFo1XVKgzNo/NzfKyo24tJi5noJ6HfJUw1H0DGO0/kd3120BLSAsUPVul0b1EAOrEKEyQwn0utjWB7CItKPj3cKbvDjR6L7B30+kyl9JYhUydilVoSquRHY3gkzVEtafEo8tsswRb7zI/5giyLD7cwNRUjmKl6JP9z5Dc1/P46Sc0MSR1d0YAClCXrjmrBzxFnNZ6URvVWwATvpHyQgQreyqmo1zw8//BINf1IL7abCqFOKfrJENAL4frBffZGjXLAxUSAYDPjhx6QQh4GWZi2OBtQkGTdPn5DlW2UrVZMA9kzIJisSVAsqYtOMVATylYdpnwnX3U1wSIgn9wG3NVJyey+Puyj/FNsEU2x4194nF9AQIaSc4J9RNbCNI6k9RNTz97HzciNOt5VywinqzGIAYLfrGi685WlCnVdmjdRCczTc13e04CBaqPMQhS0M75uznSeS1UWFYIilQLbXvBDywRB0U7o02iKBj2iifu4XNfOeoOEEfT+FoQmtAjs8/eAYZ9fbaaiYqIjE9dhb+puKsiW8Pk2KB2KD1JjfKez1RN71IZH6SuHx2T60ZZqcerQVm7MWASU0C7ERth6uW+6yrVqHVpBRjG6j1x5K8r7RUoKjMIJVsgP+1mgZEGBfp7ZCmhkNVq1UhAh95s7w1zqJoa/JH2bv3PVUrfhFvjCMJfHkoC4iLgykwhq6tYZL9G3YDx3L7KZzRK6EVzsO7LfayQj81+BiDk/JagSV4oKQoxQUrzB4SCqsR9JeHXKWToltqFeTzEXk32e6Fxw6kV0hxB/QsrBE8DZ4D0Ifyf4RncfvSPYonV/j3BONx3FvQ4XfDzNmPBh2Sv9J1btnxGVPPCGtihpfzE8urNjlvQjuemQg4K09S+cleXXxGwCn4+3x3Sm+azJ4muw4ZlaTIkXHlASloUFYWoYKDUWHF6Q8nCOnD+BnI+akwvvA2zP4mX24ACldKtofxDPT1HWrC82hfmtYZQmD50OUR6zaTc3+Ju3bno5mLorw2SezxErNNTbRUIi1aI+3T1pDfXJouzgoQaM02wylB3YIWIxFLztN9HRChaKxRIpYI2ZStoz1CRhtgvBctFs3yhXSXgUYZ3Lk15Elt5F61MJaEZ9O8PUVPo5Wy6u++PepbqZUpUBkocxGkohwVJLdTO+zbt5QEi+3jvQ+0EfaEK6IbRC3WUic1zmVuGpFyPMWyifBDD2SVWKSSPmcUwb+naucdiiLLtSIa8YQV1roFiNlpZEaOC1QwSMA2z6CF0neg3YgCEtxglPcbgslw57ODJvgOHZY3IDSBktHU3V+jrTRJ1ufsvl3f6/jzGLoRrlIoR+w/azorKd4BpHcCSiJer/h7Qir+nA8hujuMzx5IicJqKudeiwSIyfjdUDXlZgNqDIEyL8gVEQeBVIFADufu4L7jm2ljkSTG0jKn/3TpqUrHuWc5vfJVogpBRaEmdwIl+jZXuAj8ej9rGHk7KqnLAWxLycY3m0vHyinY387zbtUNkJvmWB2pftsgkw6cpe+QSZqDlT/DqrCmkBiMqOVatNbmGZc2vFiIlsaHugR8VCiDZsdFJChx5cI/Vllz2eWGg7wa6m4Jj5q8ecdUJhrbLvqn6hywqsK3FRJBRbCoKmrRSxg9g0tKL9dYjFG1O2JgJbDN1hZfkGVkPFAeHew6wGeB9BwmYsGRGa7hujtJy+lSlkGe5mlWUDXA6PPlareg6IdDhQC8J6qzGA+E6ZY8b/M4ZAyIziSYn2p+Luc/wvTjcCNBc8/PBIWjXK4GWXiAxgYO4w/zCo8mqhXyq2n33mrU0W2dQ3zgdlYnnhuoSDCxjbAIcn6gyeRnz35IdbU2FXTfniW/7ewBixc2TMjbkEwDI5AlGZqT5Cu70MaRseeA1E1q7DBNMDEwDQYJYIZIAWUDBAIBBQAEIFZcXHS1nbOgP8gq29uuabJfNeQxecN2pcM5fFADa39MBBTZny4HajWnTma/diX6Gokivrnm9QICJxA=”}[azadmin@l21q1382s005001 gpp]$ echo “MIIOCAIBAzCCDbIGCSqGSIb3DQEHAaCCDaMEgg2fMIINmzCCBbIGCSqGSIb3DQEHAaCCBaMEggWfMIIFk1i5YUSNgw8CUBkAgInEAIBIDAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQvQpeljqUxniTMTiX7Hq0FgSCBNDf60kzS3enXucBMvbUPRvsshPGiXdYuIMi82eTAxmE2Z88JX1bKyQG1b/bAAAqKOEOvEHTuear0bjQ0xqkl0YFWpnU87mxMGrFFDfdvocrpruwxLKKrUU+VZxwWkyhy64ev5No0IQjMZMhFy6Hl/kCu14iPPEtmPXDT+OMwJ49wENTx8H9B4mLfqNsp/rZJ4zOdqOeCw/Irr0TlJTiGBzfgtrKH9T0xhIaoFl49vbqnABWZZcZqGXIy0ap8jc0Vpp2DPb8V+/yqja/13mllGPx9GxYpkdSW2aNOwwcRXH00luVwu4pWTGlQNNPlHPYov+b29E4XevnA5NNjP25upuH19pMgfv0XAap2vssugVW538kL7g6WY0zD4eRpSYM7yzKEoaZH7CPMJ39EbD0eucoVFRhA0KB1JsErDdDBsj077NBYxJ+CQWIhbTnS7Kb7Rq2VpLYqr8cP4dUEgfBEvbg3MVpVH7F62nUqfnSKE7js4dy382Xpki1aL0LLktnMAUebensjl8oLg4o9P4f0VqJ1YAAXslJxOsNI8h4rR0Yff8qu80oaFprypkPVPqwdttRXMOoNb0lTlAA1zjctZAATgU4434VN3BHDfozjXk+EXoaoH7r2Ww5EzCKQFo/xGpVZW6fN8R8jLb6y3Eopa/pQUfscmVTyyE4IAlSfJMehU4urgbyfm7clBpmW5RtfyEgDaeZ1J+JYxrZNv6Z6BdaAxB1gShsSLikFPg6w3Mpca8UHrbIM1y/UTkBNjmFLDzSlAxq43oguURBTf8eipByhCW9fdQpcOwlIFdUtaUft9bvEwLZxjhviRO3il2KRCcuFxeAtYN7fTiqC1pm8+JFMyoAnBSzktzVFF4yZE6RxA0ShHE46zDh8sO4Q3rjmta0Z2J5rATcJ3h+NDrWNhg+yGhQH5ruwyqHoJOT+2bm8G0yZYN0JbqLTgC7vrjiR49z6nvnY6ZsAnlNMj+JLPAbf9aPpohf31XJ2KgujiTXvB74L4l9a5jWkYm1Nq7djf9Rt+dXGx/41hLhBOfqo/OLbO32g/vMASUtcHapGeOH7XYew1ImeVQd4aswJMNg4RE6E+RF5nOdoI8jbWhxHQqVTYEPSuHfTrrNM3mou6hCL9VMLlkxDQSvyGk73ynZRnZFp6uhtqWt3vSOr1KWnvIZCJZ8J4V9YoADmfiIcJg/o3BmbYqDt+k33UvxXdo/XakfCSPZLtGHrZgiKaahlmu+BvbDyEsCQsyDMznRCnHkNSfcgRplaikHIn7+E3WPyG/BNAgI1saza33r6nwcrckVMjtexsN/MQ3YrF3H0K5THvN0UcDjq0obvEXOOgHt6ItNi1jkGs4Jm9dGtu5dmfQ49Yl2eN1gi0aB1LEzSTqronSk/6SlYUJzxd7jjIWUl0x1hx25yyqBuyxcTlk7D4cSKskPbDok7w91Yq4qkTtuBDCr5UC+gTWEpYiwk03xHyO8KBqGr9tR8XIVNGX/9rAe9o5gL/pOlVhWDlAzOBbby3sU6Q4dL2m4JmsKy4BQvwAMwWMrGyCKcZuML9WEWteoX4lzu1vsbJgwE4Z42KjIAjf150A06Tf7woirja8CwejaL/JsWprfWBjvLJWyIwaDZAbmIZ8mBAcskwkD73jukTFEMB8GCSqGSIb3DQEJFDESHhAAaQBuAHMAdABhAG4AYwBlMCEGCSqGSIb3DQEJFTEUBBJUaW1lIDE3MDUwNTc4OTI5MjgwggfhBgkqhkiG9w0BBwagggfSMIIHzgIBADCCB8cGCSqGSIb3DQEHATBmBgkqhkiG9w0BBQ0wWTA4BgkqhkiG9w0BBQwwKwQUkNrezGXW41fq85vOMFlEOIBaVj4CAicQAgEgMAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBABtfdn4q80sOdnX/oecwMzgIIHUPmdEEg+gmpBUX1UJNJ/+dFIlgGPhMZbzminH1iPVgUollZo4An7oprdLioGQI7uh4BBIcn4s168rm+MmR0e0Kzts+Rt0Sl9QasgrYi0eAydxgLJd701KkJH8HWfjmWlW+LPXT/IPUnI+WaooSoHXy1RVOmP3T6vQAPrCEMzguuGUupReqqIQu+Kx95JQBfXDQZ2raQ0hsV75KkcVfkJt05LmkMD8GvQ0ydYBz8BX30iK+hriFdLUPXrE+SFyGHP5rz4H9u3TDZqDM2oDMAWwKbbfZCXko+yBWri3SkQ4Nhut8LgusbSfZwBWuG7tjpSim6oynyFXEqSiInPP3UZ/JSVMDPNAaPv8MLnCGKS+WlD0wGnffdCuQpnjRpeyK2UK5tOWYIUPtikh1IVaXLUlkW8McODx41KOwqC18KJND1y/SCJw35I41Y2BsaPtRvei5CfQ7Lu8L1M7OnQ0FzZwde2wlh9V2qONmcv978/dPBt8LLJ94Ke1pJSxmcoDEnTtPC4wonuYp3oc60B5emo2SfgwZOHN/3GDPAh9/AzbuXIPli3D9CD+Sz4JLAUXHkYNJs6bqvNJPi2GrXCIUWlKq6VmFo1XVKgzNo/NzfKyo24tJi5noJ6HfJUw1H0DGO0/kd3120BLSAsUPVul0b1EAOrEKEyQwn0utjWB7CItKPj3cKbvDjR6L7B30+kyl9JYhUydilVoSquRHY3gkzVEtafEo8tsswRb7zI/5giyLD7cwNRUjmKl6JP9z5Dc1/P46Sc0MSR1d0YAClCXrjmrBzxFnNZ6URvVWwATvpHyQgQreyqmo1zw8//BINf1IL7abCqFOKfrJENAL4frBffZGjXLAxUSAYDPjhx6QQh4GWZi2OBtQkGTdPn5DlW2UrVZMA9kzIJisSVAsqYtOMVATylYdpnwnX3U1wSIgn9wG3NVJyey+Puyj/FNsEU2x4194nF9AQIaSc4J9RNbCNI6k9RNTz97HzciNOt5VywinqzGIAYLfrGi685WlCnVdmjdRCczTc13e04CBaqPMQhS0M75uznSeS1UWFYIilQLbXvBDywRB0U7o02iKBj2iifu4XNfOeoOEEfT+FoQmtAjs8/eAYZ9fbaaiYqIjE9dhb+puKsiW8Pk2KB2KD1JjfKez1RN71IZH6SuHx2T60ZZqcerQVm7MWASU0C7ERth6uW+6yrVqHVpBRjG6j1x5K8r7RUoKjMIJVsgP+1mgZEGBfp7ZCmhkNVq1UhAh95s7w1zqJoa/JH2bv3PVUrfhFvjCMJfHkoC4iLgykwhq6tYZL9G3YDx3L7KZzRK6EVzsO7LfayQj81+BiDk/JagSV4oKQoxQUrzB4SCqsR9JeHXKWToltqFeTzEXk32e6Fxw6kV0hxB/QsrBE8DZ4D0Ifyf4RncfvSPYonV/j3BONx3FvQ4XfDzNmPBh2Sv9J1btnxGVPPCGtihpfzE8urNjlvQjuemQg4K09S+cleXXxGwCn4+3x3Sm+azJ4muw4ZlaTIkXHlASloUFYWoYKDUWHF6Q8nCOnD+BnI+akwvvA2zP4mX24ACldKtofxDPT1HWrC82hfmtYZQmD50OUR6zaTc3+Ju3bno5mLorw2SezxErNNTbRUIi1aI+3T1pDfXJouzgoQaM02wylB3YIWIxFLztN9HRChaKxRIpYI2ZStoz1CRhtgvBctFs3yhXSXgUYZ3Lk15Elt5F61MJaEZ9O8PUVPo5Wy6u++PepbqZUpUBkocxGkohwVJLdTO+zbt5QEi+3jvQ+0EfaEK6IbRC3WUic1zmVuGpFyPMWyifBDD2SVWKSSPmcUwb+naucdiiLLtSIa8YQV1roFiNlpZEaOC1QwSMA2z6CF0neg3YgCEtxglPcbgslw57ODJvgOHZY3IDSBktHU3V+jrTRJ1ufsvl3f6/jzGLoRrlIoR+w/azorKd4BpHcCSiJer/h7Qir+nA8hujuMzx5IicJqKudeiwSIyfjdUDXlZgNqDIEyL8gVEQeBVIFADufu4L7jm2ljkSTG0jKn/3TpqUrHuWc5vfJVogpBRaEmdwIl+jZXuAj8ej9rGHk7KqnLAWxLycY3m0vHyinY387zbtUNkJvmWB2pftsgkw6cpe+QSZqDlT/DqrCmkBiMqOVatNbmGZc2vFiIlsaHugR8VCiDZsdFJChx5cI/Vllz2eWGg7wa6m4Jj5q8ecdUJhrbLvqn6hywqsK3FRJBRbCoKmrRSxg9g0tKL9dYjFG1O2JgJbDN1hZfkGVkPFAeHew6wGeB9BwmYsGRGa7hujtJy+lSlkGe5mlWUDXA6PPlareg6IdDhQC8J6qzGA+E6ZY8b/M4ZAyIziSYn2p+Luc/wvTjcCNBc8/PBIWjXK4GWXiAxgYO4w/zCo8mqhXyq2n33mrU0W2dQ3zgdlYnnhuoSDCxjbAIcn6gyeRnz35IdbU2FXTfniW/7ewBixc2TMjbkEwDI5AlGZqT5Cu70MaRseeA1E1q7DBNMDEwDQYJYIZIAWUDBAIBBQAEIFZcXHS1nbOgP8gq29uuabJfNeQxecN2pcM5fFADa39MBBTZny4HajWnTma/diX6Gokivrnm9QICJxA” | base64 –decode
    ▒▒▒▒▒0▒▒0▒▒▒▒
    *▒H▒▒

    0Y08▒<0f*▒H▒▒ *▒H▒▒
    0+<▒xJ▒
    ▒Mb▒6
    < @d' 0
    *▒H▒▒ 0 `▒He*▒
    ^▒:▒▒x▒18▒▒z▒▒▒▒▒I3Kw▒^▒2▒▒=▒ƉwX▒▒"▒g▒▒ٟU▒pZL▒ˮ▒▒hЄ#1▒!.▒▒▒▒^”<▒-▒▒▒O▒▒=▒CS▒▒▒▒▒~▒l▒▒▒'▒▒v▒▒
    Ȯ▒▒▒▒߂▒▒▒▒▒▒Yx▒▒▒Ve▒▒e▒▒F▒▒74V▒v
    ▒▒W▒▒6▒▒y▒▒c▒▒lX▒GR[f▒;
    Eq▒▒[▒▒▒)Y1▒@▒O▒sآ▒▒▒▒8]▒▒▒M▒▒▒▒L▒▒▒\▒▒▒,▒V$/▒:Y▒3▒▒▒&
    ▒,▒▒▒▒▒0▒▒▒▒z▒(TTaB▒ԛ▒7C▒▒▒Ac~ ▒▒▒▒K▒▒▒▒V▒ت▒?▒T▒▒▒▒▒iT~▒▒iԩ▒▒(N㳇r▒͗▒H▒h▒
    ▒▒ԟ▒c▒6▒▒▒Zu▒(lH▒▒▒:▒s)q▒▒▒3\▒Q969▒,▒hP▒▒▒*▒▒▒▒▒f▒▒m2e▒t%▒▒N▒▒▒▒G▒s▒{▒c▒lyM2?▒,▒▒▒_▒U▒ب.▒$׼▒/▒}k▒֑▒▒6▒ݍ▒Q▒▒W▒▒▒▒▒l▒▒▒▒%-pv▒▒▒v▒R&yT▒0$▒`▒:▒E▒s▒▒▒#mhq
    ▒▒i;▒)▒FvE▒▒▒▒▒▒▒▒R▒▒▒|’▒}b▒▒▒p▒?▒pfm▒▒▒▒7▒K▒]▒?]▒ #▒.ч▒▒”)▒▒▒k▒▒▒▒KB̃39▒
    ج]▒ЮS▒tQ▒▒JE▒:▒▒M▒X▒▒|▒▒▒▒F▒▒]▒▒8▒vx▒`▒F▒Ա3I:▒▒t▒▒▒▒aBs▒▒㌅▒▒Lu▒▒▒*▒▒,\NY;▒*▒l:$▒ub▒*▒;n0▒▒@▒▒5▒▒▒▒▒M▒#▒(▒▒▒Q▒r4e▒▒▒`/▒N▒XVP38▒▒{▒/i▒&k
    ˀP▒
    0Y08▒▒0▒*▒H▒▒ 17*▒H▒▒8929280▒▒▒�*▒H▒▒▒/▒lZ▒▒X▒,▒▒#▒d▒!▒&,▒ ▒x▒1D0 *▒H▒▒
    0+▒▒▒▒e▒▒W▒▒▒0YD8▒ZV>’ 0
    *▒H▒▒ 0 `▒He*▒▒g▒4▒▒g_▒s3▒▒P▒H>▒jAQ}T$▒▒H▒▒▒▒[▒h▒X▒V(▒Vh▒ ▒▒.*@▒A!▒▒^▒▒o▒▒Ь▒▒m▒)}A▒ ▒▒▒x
    v▒▒4▒▒{▒U▒u▒▒e▒[▒NK▒C▒k▒▒’X?_}”+▒k▒WKP▒▒▒▒a▒▒▒۷L6j▒▒I@▒
    ▒▒▒}▒▒▒▒▒j▒▒)▒▒n▒▒▒▒▒}▒ZỶ:R▒n▒▒|▒\J▒▒▒▒?u▒03▒▒▒▒▒b▒▒iC▒▒}▒B▒
    g▒^ȭ▒+▒NY▒>ؤ▒RirԖE▒1ÃǍJ;
    ▒▒�4=r▒ ▒▒~H▒V6Ə▒▒▒▒C▒▒▒L▒▒▒▒\▒▒׶▒X}Wj▒6g/▒?t▒m▒▒▒֒R▒g(
    IӴ▒�▒b▒▒s▒▒▒▒’▒▒▒7▒▒
    ▒!▒▒3n▒▒>X▒Ѓ▒,▒$▒\y4▒:n▒▒$▒▒▒!E▒*▒▒▒Z5]R▒▒▒?77▒ʍ▒▒▒▒▒▒z▒T▒Q▒
    ▒▒▒dh▒,- ,P▒n▒F▒▒▒2C ▒▒▒▒▒▒▒▒▒�▒8▒▒▒▒O▒▒_Ib2v)U▒*▒Dv7▒L▒֟▒-▒▒o▒▒▒▒”Ȱ▒sQR9▒▒▒O▒>Cs_▒㤜▒đ▒▒)B^▒▒▒sY▒DoUlN▒G▒쪚▒s▒▒▒▒_Ԃ▒i▒▒⟬▒
    TH>8q▒!▒e▒▒c▒▒ M▒▒▒9V▒J▒d▒=▒2 ▒ĕʘ▒▒<▒a▒g▒u▒S\" ▒▒m▒T▒▒▒▒▒▒?▒6▒▒5▒▒i'8'▒Ml#H▒OQ5<▒▒|܈ӭ▒\▒▒z▒▒-▒Ƌ▒9ZP▒U٣u▒▒75▒▒▒<▒!KC;▒▒▒I▒QaX")P-▒▒<▒D▒6▒▒c▒(▒▒▒▒|▒8AO▒hBk@▒▒?x▒▒▒j&*"1=v▒▒⬉o▒b▒ؠ▒&7▒{=Q7▒Hd~▒▒|vO▒f▒▒f▒ŀIM▒Dm▒▒▒▒V▒դc▒ǒ▒▒▒T▒▒▒ ▒l▒▒▒▒D▒퐦▒CU▒U!y▒▒5΢hk▒Gٻ▒=U+~o▒# |y(
    ▒▒▒)0▒▒▒a▒▒▒r▒)▒▒+▒▒û-▒B?5▒▒▒▒Z▒%x▒▒(▒+▒
    ▒Ї▒gq▒▒=▒'W▒▒▒q▒[▒▒w▒▒ُ▒▒▒un▒▒Skb▒▒▒˫69oB;▒8+OR▒▒^]|F▒)▒▒|wJo▒̞&▒▒▒ȑq▒)hPV▒▒▒Qa▒▒▒▒▒▒▒0▒▒6▒▒&_n
    WJ▒▒▒
    ▒▒j▒▒h_▒▒B`▒▒▒▒6▒▒▒v磙▒▒▒6I▒▒▒MM▒T"-Z#▒▒֐▒\▒.▒
    h▒6▒)A݂#K▒▒}▒h▒Q"ٔ▒▒=BF▒-▒▒t▒▒Fܹ5▒Im▒^▒0▒▒gӼ=EO▒▒▒▒▒=▒[▒▒)P(s▒▒$▒S;▒۷▒▒▒▒▒▒+▒
    -▒R'5▒en▒rg▒▒j▒▒”˵”▒ֺ▒▒idF▒
    T0H▒6Ϡ▒▒w▒݈▒`▒▒▒p糃&▒▒7 4▒▒▒▒▒_▒▒4I▒▒▒]▒▒▒▒▒▒R(G▒?k:+)▒▒wJ”^▒▒{B*▒▒!▒;▒▒H▒▒j*▒^▒▒▒▒▒P5▒fj
    ▒2/▒▒T▒@▒▒▒▒ic▒$▒▒2▒▒t▒Jǹg9▒▒U▒
    ▒▒▒X▒~▒ ▒▒▒▒I▒▒▒?ê▒▒▒▒▒▒Z▒▒▒▒6▒X▒▒Ƈ▒|T(▒f▒E$(q▒▒?VYs▒冃▒▒n ▒▒▒y▒T&▒.▒▒▒▒·AE▒▒*j▒K=▒KJ/▒X▒Q▒;b`%▒▒▒_▒ed<P▒:▒g▒▒&b▒▒▒;I▒▒R▒A▒▒iVP5▒▒▒▒j▒▒▒C▒▒'▒▒▒▒<o▒8d
    ▒▒$▒▒j~.▒?▒▒▒p#As▒▒▒▒\▒Yx▒▒▒▒
    ▒&▒▒i▒▒j`▒He V\\t▒▒▒▒?▒*▒ۮi▒_5▒1y▒v▒▒9|PLٟ.j5▒Nf▒v%▒▒"▒▒▒▒'base64: invalid input
    [azadmin@l21q1382s005001 gpp]$ xterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256color

    Contents of values.yaml file

    —
    clusterName: "elasticsearch"
    nodeGroup: "master"

    # These will be set as environment variables. E.g. node.master=true
    roles:
    master: "true"
    ingest: "true"
    data: "true"
    remote_cluster_client: "true"
    ml: "true"

    replicas: 3
    minimumMasterNodes: 2

    esMajorVersion: ""

    clusterDeprecationIndexing: "false"
    ## shiva added
    esConfig:
    elasticsearch.yml: |
    cluster.name: "docker-cluster"
    network.host: 0.0.0.0
    xpack.security.enabled: "true"
    xpack.security.transport.ssl.enabled: "true"
    xpack.security.transport.ssl.supported_protocols: "TLSv1.2"
    #xpack.security.transport.ssl.client_authentication: "none"
    xpack.security.transport.ssl.client_authentication: "required"
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    #xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/elastic-certificates.p12"
    xpack.security.transport.ssl.verification_mode: "certificate"
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.supported_protocols: "TLSv1.2"
    #xpack.security.http.ssl.client_authentication: "none"
    xpack.security.http.ssl.client_authentication: "required"
    xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    #xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"

    ## shiva added
    extraEnvs:
    – name: "ELASTIC_PASSWORD"
    valueFrom:
    secretKeyRef:
    name: "elastic-secret"
    key: "ES_PASSWORD"
    – name: "ELASTIC_USERNAME"
    valueFrom:
    secretKeyRef:
    name: "elastic-secret"
    key: "ES_USERNAME"

    ## shiva added
    secretMounts:
    – name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/elasticsearch/config/certs"
    defaultMode: "0755"

    image: "docker.elastic.co/elasticsearch/elasticsearch"
    imageTag: "7.17.1"
    imagePullPolicy: "IfNotPresent"

    protocol: https
    httpPort: 9200
    transportPort: 9300

    service:
    enabled: true
    labels: {}
    labelsHeadless: {}
    type: ClusterIP
    # Consider that all endpoints are considered "ready" even if the Pods themselves are not
    # https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec
    publishNotReadyAddresses: false
    nodePort: ""
    annotations: {}
    httpPortName: https
    transportPortName: transport
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    externalTrafficPolicy: ""

    Please help me I spend around 5 days still I didn't get expected output.

    Reply

    • Hi Shiwanand, you seem to be trying to deploy a clustered version of Elasticsearch:

      failing [node-left[{elasticsearch-master-1}
      non-failed nodes do not form a quorum

      This article was written for, and tested with, a single node deployment I’m afraid.

      Could you try deploying Elasticsearch using the following and post back if it does the trick?

      replicas: 1
minimumMasterNodes: 1

      If so, then the issue will be related to your cluster configuration.

  11. Hey @Lisenet,

    I tried with single node then also no luck, Yes something is missing. In my case certification generation is done through elasticsearch-certutil tool

    I follow below steps
    1. Login to Elasticsearch container
    2. Generate the elastic-stack-ca.p12 cert
    3. Generate elastic-certificates.p12
    4. Create a secret using elastic-certificates.p12 and mount it
    5. Ran the helm upgrade
    Once we run the helm-upgrade, it’s failed cause with error
    Error

    “`
    “at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “… 6 more”] }
    ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
    Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
    at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
    at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
    at java.base/java.security.KeyStore.load(KeyStore.java:1473)
    at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:99)
    at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66)
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:453)
    at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1220)
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:546)
    at jav
    “`

    Thanks for your reply. Can you guide me how to automate tls/ssl configuration using helm for 3 nodes.
    If you provide me path then also fine, I have issue with certificate generation only and this entire process I need to be automated using helm charts.

    Thank you so much for your response and time.

    Reply

    • Have you tried using openssl to generate SSL certificates as per this blog post? Did that work for you with a single node setup?

  13. hi @lisenet
    how you created elastic-credentials-secret.yaml and elastic-certificates-secret.yaml
    because by following your commands we create home-lab-ca.crt, home-lab-ca.key, wildcard.hl.test.csr and wildcard.hl.test.key

    and i cant directly take the yaml files from git since they got ur keys
    so please provide commands to create those two files
    and in elastic-certificates-secret.yml files i saw home-lab-ca.crt, tls.crt and tls.key
    where are we creating tls related keys and certificates!!!
    and in elastic-credentials-secret.yml username and password are provided wht r they

    can you elaborate more on these so we can smoothly deploy on out nodes

    thank you

    Reply

    • how you created elastic-credentials-secret.yaml and elastic-certificates-secret.yaml

      These are standard YAML files used by Kubernetes for secrets. See documentation here. I created them by hand using vim.

      and i cant directly take the yaml files from git since they got ur keys

      You can take the YAML files from GitHub and use them as a template where you replace my keys with your keys.

      where are we creating tls related keys and certificates!!!

      You are creating them with openssl whe you generate ${DOMAIN}.key, ${DOMAIN}.crt and homelab-ca.crt files. When you mount the secrets inside Elasticsearch pod, these secrets are called tls.key, tls.crt and homelab-ca.crt, respectively. See below.

      xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"

      and in elastic-credentials-secret.yml username and password are provided wht r they

      These are initial Elasticsearch login credentials for the elastic user that you will use to log into Elasticsearch. See documentation here.

  16. I have one issue with credentials i created my own
    please look into it
    Actually
    kubectl create secret generic elastic-credentials –namespace=logging –type=Opaque –from-literal=username=elastic –from-literal=password=Admin1234
    after running this and certificates
    then deployed elasticsearch and kibana
    kibana throwing this error

    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”plugins-service”],”pid”:7,”message”:”Plugin \”newsfeed\” is disabled.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”plugins-service”],”pid”:7,”message”:”Plugin \”telemetry\” is disabled.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”plugins-service”],”pid”:7,”message”:”Plugin \”telemetryManagementSection\” has been disabled since the following direct or transitive dependencies are missing, disabled, or have incompatible types: [telemetry]”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”plugins-service”],”pid”:7,”message”:”Plugin \”metricsEntities\” is disabled.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”http”,”server”,”Preboot”],”pid”:7,”message”:”http server running at https://0.0.0.0:5601“}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:7,”message”:”Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:7,”message”:”Kibana is configured to authenticate to Elasticsearch with the \”elastic\” user. Use a service account token instead.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:7,”message”:”The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \”xpack.reporting.roles.enabled\” to \”false\” to adopt the future behavior before upgrading.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:7,”message”:”User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:7,”message”:”Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”plugins-system”,”standard”],”pid”:7,”message”:”Setting up [111] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:57+00:00″,”tags”:[“info”,”plugins”,”taskManager”],”pid”:7,”message”:”TaskManager is identified by the Kibana UUID: db658c03-27c6-4152-a9e3-f7dbba219763″}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“warning”,”plugins”,”security”,”config”],”pid”:7,”message”:”Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“warning”,”plugins”,”security”,”config”],”pid”:7,”message”:”Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“warning”,”plugins”,”reporting”,”config”],”pid”:7,”message”:”Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“warning”,”plugins”,”encryptedSavedObjects”],”pid”:7,”message”:”Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“warning”,”plugins”,”actions”],”pid”:7,”message”:”APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“warning”,”plugins”,”alerting”],”pid”:7,”message”:”APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:58+00:00″,”tags”:[“info”,”plugins”,”ruleRegistry”],”pid”:7,”message”:”Installing common resources shared between all indices”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:56:59+00:00″,”tags”:[“info”,”plugins”,”reporting”,”config”],”pid”:7,”message”:”Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.”}
    {“type”:”log”,”@timestamp”:”2024-04-24T12:57:02+00:00″,”tags”:[“error”,”elasticsearch-service”],”pid”:7,”message”:”Unable to retrieve version information from Elasticsearch nodes. Client network socket disconnected before secure TLS connection was established – Local: unknown:unknown, Remote: unknown:unknown”}

    its telling
    Unable to retrieve version information from Elasticsearch nodes. Client network socket disconnected before secure TLS connection was established

    thank you

    Reply

  17. Hi Lisenet
    If i create elastic-credentials-secret.yml from by executing this command
    kubectl create secret generic elastic-credentials \
    –namespace=logging \
    –type=Opaque \
    –from-literal=username=username \
    –from-literal=password=password > elastic-credentials-secret.yaml

    —
    apiVersion: v1
    kind: Secret
    metadata:
    name: elastic-credentials
    namespace: logging
    type: Opaque
    data:
    password: dlhWV0Q4MW1zMnM2QjU2S1ZHUU8=
    username: ZWxhc3RpYw==

    elasticsearch deployed without any issue

    but while deploying kibana it throw this error
    {“type”:”log”,”@timestamp”:”2024-04-25T03:14:27+00:00″,”tags”:[“error”,”elasticsearch-service”],”pid”:7,”message”:”Unable to retrieve version information from Elasticsearch nodes. read ECONNRESET – Local: unknown:unknown, Remote: unknown:unknown”}
    Readiness probe failed: Error: Got HTTP code 503 but expected a 200

    can you please help me with it

    Reply

    • You could deploy a busybox pod and try accessing the URL with curl to gather more information. HTTP 503 means that the server is unavailable and there should be a reason for that. It’s difficult to say without having the service logs.

    • hi Lisenet

      please refer this kibana logs

      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”plugins-service”],”pid”:8,”message”:”Plugin \”newsfeed\” is disabled.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”plugins-service”],”pid”:8,”message”:”Plugin \”telemetry\” is disabled.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”plugins-service”],”pid”:8,”message”:”Plugin \”telemetryManagementSection\” has been disabled since the following direct or transitive dependencies are missing, disabled, or have incompatible types: [telemetry]”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”plugins-service”],”pid”:8,”message”:”Plugin \”metricsEntities\” is disabled.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”http”,”server”,”Preboot”],”pid”:8,”message”:”http server running at https://0.0.0.0:5601“}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:8,”message”:”Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:8,”message”:”Kibana is configured to authenticate to Elasticsearch with the \”elastic\” user. Use a service account token instead.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:8,”message”:”The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \”xpack.reporting.roles.enabled\” to \”false\” to adopt the future behavior before upgrading.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:8,”message”:”User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“warning”,”config”,”deprecation”],”pid”:8,”message”:”Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”plugins-system”,”standard”],”pid”:8,”message”:”Setting up [111] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:29+00:00″,”tags”:[“info”,”plugins”,”taskManager”],”pid”:8,”message”:”TaskManager is identified by the Kibana UUID: de309072-b586-46c1-8078-53f93dd0bc48″}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“warning”,”plugins”,”security”,”config”],”pid”:8,”message”:”Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“warning”,”plugins”,”security”,”config”],”pid”:8,”message”:”Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“warning”,”plugins”,”reporting”,”config”],”pid”:8,”message”:”Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“warning”,”plugins”,”encryptedSavedObjects”],”pid”:8,”message”:”Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“warning”,”plugins”,”actions”],”pid”:8,”message”:”APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“warning”,”plugins”,”alerting”],”pid”:8,”message”:”APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:30+00:00″,”tags”:[“info”,”plugins”,”ruleRegistry”],”pid”:8,”message”:”Installing common resources shared between all indices”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:31+00:00″,”tags”:[“info”,”plugins”,”reporting”,”config”],”pid”:8,”message”:”Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.”}
      {“type”:”log”,”@timestamp”:”2024-05-01T05:44:35+00:00″,”tags”:[“error”,”elasticsearch-service”],”pid”:8,”message”:”Unable to retrieve version information from Elasticsearch nodes. self signed certificate”}

    • Thanks. See the error message below.

      [“error”,”elasticsearch-service”],”pid”:8,”message”:”Unable to retrieve version information from Elasticsearch nodes. self signed certificate”}

      You need to provide the self signed CA for Elasticsearch to Kibana in kibana.yml.

    • kibanaConfig:
      kibana.yml: |
      server.ssl:
      enabled: “true”
      key: “/usr/share/kibana/config/certs/tls.key”
      certificate: “/usr/share/kibana/config/certs/tls.crt”
      certificateAuthorities: [ “/usr/share/kibana/config/certs/ca.crt” ]
      clientAuthentication: “none”
      supportedProtocols: [ “TLSv1.2”, “TLSv1.3” ]
      elasticsearch.ssl:
      certificateAuthorities: [ “/usr/share/kibana/config/certs/ca.crt” ]
      verificationMode: “certificate”
      newsfeed.enabled: “false”
      telemetry.enabled: “false”
      telemetry.optIn: “false”

      actually i have provided
      (i named homelab-ca.crt as ca.crt in my deployment
      even file name is also ca.crt)

  18. Hi Lisenet
    I want to deploy elasticsearch and kibana in latest helm version 8.5.1 can u help me deploying it
    bcz above method not working since its ssl configuration looks bit different

    Reply

  19. Hi Lisenet,
    thanks for wonderful source,
    I would like to request
    can u also share deployment of logstash and filebeat along with above deployment
    thank you

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *