Deploy Elasticsearch and Kibana on Kubernetes with Helm

We will install Elasticsearch and Kibana as well as set up basic security for the Elastic Stack plus secured HTTPS traffic.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configuration files used in this article can be found on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/kubernetes/elasticsearch/

The Plan

  1. Install Helm.
  2. Create an internal Certificate Authority (CA).
  3. Create a wildcard certificate for Elasticsearch signed by the CA.
  4. Install Elasticsearch 7.17 using Helm.
  5. Install Kibana 7.17 using Helm.

Install Helm

On a Debian-based OS, do the following:

$ curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
$ sudo apt-get install -y apt-transport-https
$ echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm

Add Helm repository:

$ helm repo add elastic https://helm.elastic.co

Create Internal Certificate Authority (CA)

This section covers steps required to create a Root CA. Note that we have done this for the homelab environment here.

Generate a Root CA that is valid for 10 years:

$ openssl req -newkey rsa:2048 -keyout homelab-ca.key -nodes -x509 -days 3650 -out homelab-ca.crt

Verify X509v3 extensions:

$ openssl x509 -text -noout -in homelab-ca.crt | grep CA
     CA:TRUE

Create a wildcard certificate signed by the Root CA to be used with Elasticsearch and Kibana:

$ DOMAIN="wildcard.hl.test"
$ openssl genrsa -out "${DOMAIN}".key 2048 && chmod 0600 "${DOMAIN}".key

Generate a Certificate Sign Request (CSR):

$ openssl req -new -sha256 -key "${DOMAIN}".key -out "${DOMAIN}".csr

Sign the request with the Root CA:

$ openssl x509 -req -in "${DOMAIN}".csr -CA homelab-ca.crt -CAkey homelab-ca.key -CAcreateserial -out "${DOMAIN}".crt -days 1825 -sha256

Optional: import the Root CA in to your browser.

Install Elasticsearch on Kubernetes

Create logging namespace:

$ kubectl create namespace logging

Create a secret to store Elasticsearch credentials:

$ kubectl apply -f ./elastic-credentials-secret.yml

Create a secret to store Elasticsearch SSL certificates. We are using the Root CA to sign the certificate.

$ kubectl apply -f ./elastic-certificates-secret.yml

By default, the Elasticsearch security features are disabled when we have a basic license. To enable security features, we will use the xpack.security.enabled setting.

In order to enable TLS/SSL on the HTTP networking layer, which Elasticsearch uses to communicate with other clients, we will use the xpack.security.http.ssl.enabled setting.

Create a values file values-elasticsearch.yml for Elasticsearch:

---
clusterName: "elasticsearch"
nodeGroup: "master"

roles:
  master: "true"
  ingest: "true"
  data: "true"
  remote_cluster_client: "true"
  ml: "true"

replicas: 1
minimumMasterNodes: 1

protocol: https
httpPort: 9200
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "ELASTIC_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"
  - name: "ELASTIC_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: "true"
    xpack.security.transport.ssl.enabled: "true"
    xpack.security.transport.ssl.supported_protocols: "TLSv1.2"
    xpack.security.transport.ssl.client_authentication: "none"
    xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.transport.ssl.verification_mode: "certificate"
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.supported_protocols: "TLSv1.2"
    xpack.security.http.ssl.client_authentication: "none"
    xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/elasticsearch/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "250m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "4Gi"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "freenas-nfs-csi"
  resources:
    requests:
      storage: 64Gi

service:
  enabled: true
  labels: {}
  labelsHeadless: {}
  type: LoadBalancer
  nodePort: ""
  annotations: {}
  httpPortName: https
  transportPortName: transport
  loadBalancerIP: "10.11.1.59"
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""

clusterHealthCheckParams: "wait_for_status=yellow&timeout=2s"

Deploy a single node Elasticsearch with authentication, certificates for TLS and custom values:

$ helm upgrade --install elasticsearch \
  elastic/elasticsearch \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-elasticsearch.yml

Elasticsearch endpoint will be available at https://10.11.1.59:9200/.

You can test it by using curl:

$ curl -sk -u "username:password" https://10.11.1.59:9200/ | jq
{
  "name": "elasticsearch-master-0",
  "cluster_name": "elasticsearch",
  "cluster_uuid": "t6rPuP6NSn6IDaW98J0VWw",
  "version": {
    "number": "7.17.1",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "e5acb99f822233d62d6444ce45a4543dc1c8059a",
    "build_date": "2022-02-23T22:20:54.153567231Z",
    "build_snapshot": false,
    "lucene_version": "8.11.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}

Install Kibana on Kubernetes

Create a values file values-kibana.yml for Kibana:

---
elasticsearchHosts: "https://elasticsearch-master:9200"

replicas: 1

protocol: https
httpPort: 5601
imagePullPolicy: "IfNotPresent"

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: "ELASTICSEARCH_USERNAME"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "username"
  - name: "ELASTICSEARCH_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: "elastic-credentials"
        key: "password"

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: "true"
      key: "/usr/share/kibana/config/certs/tls.key"
      certificate: "/usr/share/kibana/config/certs/tls.crt"
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      clientAuthentication: "none"
      supportedProtocols: [ "TLSv1.2", "TLSv1.3" ]
    elasticsearch.ssl:
      certificateAuthorities: [ "/usr/share/kibana/config/certs/homelab-ca.crt" ]
      verificationMode: "certificate"
    newsfeed.enabled: "false"
    telemetry.enabled: "false"
    telemetry.optIn: "false"

secretMounts:
  - name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/kibana/config/certs"
    defaultMode: "0755"

resources:
  requests:
    cpu: "55m"
    memory: "512Mi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

service:
  type: LoadBalancer
  loadBalancerIP: "10.11.1.58"
  port: 5601
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  httpPortName: http

Deploy Kibana using authentication and TLS to connect to Elasticsearch:

$ helm upgrade --install kibana \
  elastic/kibana \
  --namespace logging \
  --version "7.17.1" \
  --values ./values-kibana.yml

Kibana endpoint will be available at https://10.11.1.58:5601/.

Verify that pods are running:

$ kubectl get po -n logging
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          23h
kibana-kibana-5d8dc78bfb-4fqr2   1/1     Running   0          23h

Verify services:

$ kubectl get svc -n logging
NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
elasticsearch-master            LoadBalancer   10.105.182.194   10.11.1.59    9200:31657/TCP,9300:32405/TCP   3d22h
elasticsearch-master-headless   ClusterIP      None             none          9200/TCP,9300/TCP               3d22h
kibana-kibana                   LoadBalancer   10.105.176.223   10.11.1.58    5601:31251/TCP                  3d21h

References

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-stack-security.html

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-settings.html

22 thoughts on “Deploy Elasticsearch and Kibana on Kubernetes with Helm

  1. hello, can I have details about certificates. I create a rootCA and a certificate with key. I change values inside elastic-certificates-secret.yml with my values (I put base64 encoded rootca and crt and key)
    After applying all I obtain “io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca”, at elastic start. When I left intact your elastic-certificates-secret.yml, it’s ok but with your certificates and not my certificates.
    thanks for the help

    • Hi Bruno, the error message suggests that the CA cert has not been provided. Did you generate a root CA certificate, and then used it to sign the ElasticSearch certificate?

  2. Hello,
    I followed this article but used AWS CA for certs and my cert is bind by passphrase and when I run my es version 7.10.2, I am getting below error:ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: IllegalStateException[Error parsing Private Key from: /usr/share/elasticsearch/config/certs/tls.key]; nested: NoSuchAlgorithmException[PBES2 SecretKeyFactory not available];
    Likely root cause: java.security.NoSuchAlgorithmException: PBES2 SecretKeyFactory not available
    Not sure, why I am getting this error. I tried to follow the steps as it is.

    • Hi, does your config provide the passphrase to ElasticSearch to decrypt the private key in some way? The error suggests that ElasticSearch could not read the private key, probably because it is encrypted and may need a passphrase to decrypt it.

  3. How can we add these elasticsearch SSL certificates( tls.key, tls.crt and homelab-ca.crt) to elastic-certificates-secret.yml file, Could you help me on this

  4. Hello,
    How can we add elasticsearch SSL certificates to elastic-certificates-secret.yml file.
    Could you help on this.

    • Hi ,
      We have converted ssl certificates from plain text to base64
      Command used for converting : openssl base64 -in elasticsearch-ca.crt -out elasticsearch-ca.b64
      Still elastic search pods are not running throwing like ssl certificate issue.
      Could you please help us to understand issue here .
      logs attached here for reference:

    • Please check if you can get certificate data from your Kubernetes secrets. See example command below:

      kubectl -n logging get secret elastic-certificates -o yaml

      Decode the secret data that you get from above using base64 -d and see if it has your certificate.

  5. logs attached here for reference:
    {“type”: “server”, “timestamp”: “2023-04-14T09:28:22,379Z”, “level”: “ERROR”, “component”: “o.e.b.ElasticsearchUncaughtExceptionHandler”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “uncaught exception in thread [main]”,
    “stacktrace”: [“org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL KeyManagerFactory]; nested: MalformedInputException[Input length = 1];”,
    “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.0.jar:7.14.0]”,
    “Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]”,
    “at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]”,

  6. Hi ,
    After decoding the secrets we got like this but still facing ssl authentication issue , anything do we need to add it
    apiVersion: v1
    data:
    elastic-ca.crt: MIID6TCCAtGgAwIBAgIJANVcM6ZYRpA7MA0GCSqGSIb3DQEBCwUAMIGKMQswCQYDVQQGEwJJTjELMAkGA1UECAwCS04xEjAQBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UECgwIRVJJQ1NTT04xDTALBgNVBAsMBElEVU4xEDAOBgNVBAMMB2hhaG4xMzAxJjAkBgkqhkiG9w0BCQEWF3NhcmFsYS5rOTlAZXJpY3Nzb24uY29tMB4XDTIzMDQxNDA2MzUxOFoXDTMzMDQxMTA2MzUxOFowgYoxCzAJBgNVBAYTAklOMQswCQYDVQQIDAJLTjESMBAGA1UEBwwJQkFOR0FMT1JFMREwDwYDVQQKDAhFUklDU1NPTjENMAsGA1UECwwESURVTjEQMA4GA1UEAwwHaGFobjEzMDEmMCQGCSqGSIb3DQEJARYXc2FyYWxhLms5OUBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCeBGr/D05pn5HIPN7rbUgL+7Txr0OEgZKBigG/EC1c6uQkv69NzvgSzzqnAyWmgbnDxDixuSKyeqyFXWoSUF3/FdCHUSVdBHsxP1lY0uaqp8JPit4Ym/lihZXCtczbHExlhFbsUqu30BI3D4s3QljwhCsy6LaEsAUKeMFgTWBYy8z3qSYlEzqntePbGfEMYPWGEslkFu05kiGcMs/FoWzbdp2hA+qOCGeRy1DXkvzGrEIsP3IX5+S1eBUYiVfm/yWVihPy9h5HcMapQ9nUI49L3cWWq2QPTRLC7epwfbC+uSxV1jDeLQmJ296jSYsm48izvQFGPSfcDegQkd/3rDXlAgMBAAGjUDBOMB0GA1UdDgQWBBReynQAkqa1GsvTULUJMSOK66Ks5TAfBgNVHSMEGDAWgBReynQAkqa1GsvTULUJMSOK66Ks5TAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBBNI7c6skjCpH5N08J9ezqIaNYtdo8aGCXCcAEOYRXer2mtLhE9OAzrHIWMSdHC/BUIhMmnJXBWGyjLzVJkdKD133sfk1y9XKqqVTfzo9+GKYRjfAdS5zdKi8PXVAivsnUeJZ9IpzLEznCEbhlrVA3AEkv0/zSvSqHwtWnAo459t1TZbku7esKfnGbhlz3d6+CL2tReuLDPJx9l4XLRS1w965j/mJoLaWhNerEDU4aogrKJxD+5rubRxIFezLh7ZjFcKlR6SsTgGnAAVNokAxGfBRIvs22uJTX7+0ecEJ5XyXMm1gIfrDFCjF3eHXG5Ci2u7mKXpQMBIImG7HPfhx2
    tls.crt: MIIDlDCCAnwCCQC4dEuHK0c6zzANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMCSU4xCzAJBgNVBAgMAktOMRIwEAYDVQQHDAlCQU5HQUxPUkUxETAPBgNVBAoMCEVSSUNTU09OMQ0wCwYDVQQLDARJRFVOMRAwDgYDVQQDDAdoYWhuMTMwMSYwJAYJKoZIhvcNAQkBFhdzYXJhbGEuazk5QGVyaWNzc29uLmNvbTAeFw0yMzA0MTQwNzMwMzFaFw0yODA0MTIwNzMwMzFaMIGMMQswCQYDVQQGEwJJTjELMAkGA1UECAwCS04xEjAQBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UECgwIRVJJQ1NTT04xDTALBgNVBAsMBElEVU4xEDAOBgNVBAMMB2hhaG4xMzAxKDAmBgkqhkiG9w0BCQEWGXNhcmFsYS5rLmV4dEBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaUJr/RqvKYAHOEITeJTCsTxwwsEYKyKLAGSiAd5CHBM5nGRsoETEYbqypdJPtk9MgHrc84TmFgDdaiLA8ijH0XO8wmj0OaMK6W4MZLzFifjaPjn+3hvx401keqRt7NjGlCNy6ICTNfuMnYB1Ya0UNj6Edph+LrYTl56Te014amZyhqfr0vBjmUKTeafzVqp55JL1/3O9IQe5RihK5eq4qCDD7hLGRMmBmXtS1XLjyET4XPLYz4bhO+JxE3N+mYe9uuHUoHQYVKe8driM1Fn5qUMipxxAonFtGQ7QLSTE942iXgkwHDWFGXpAJfX/2O8bYTyFKh7Wf6wMJ5RDDyldzAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHjL7Gqges0WAU5wY1PHK85KdURiSrVgDx/aQneghocXPlf/pGXt6IAJdnMoT2MLuV4Ic8/Vv0fedUFcgmF5zuWd3wM0wDtz3SkAheIZymkBK4lfXuDK1phIJ0Ak8I6Y/maKoQDXAT9WIfZIeScq3vwdqCweK3AXOXg/PCIKCSSTg46Y96PLRhZGyQ/pLjnCJg8A4R50eyK911EbdfYWb/rfdEzJqf3v1q0+UuKoxknHkHiMty2S3d4Wi7bJWz+9zfI0EFZO70bQahbChFE0CONHMMvVRNErLNRJpDDqpI8myyEq4hp2aLT52X4//SeumylrjRgrU0sqMF9zFhA42Aw=
    tls.key: MIIEpAIBAAKCAQEAmlCa/0arymABzhCE3iUwrE8cMLBGCsiiwBkogHeQhwTOZxkbKBExGG6sqXST7ZPTIB63POE5hYA3WoiwPIox9FzvMJo9DmjCuluDGS8xYn42j45/t4b8eNNZHqkbezYxpQjcuiAkzX7jJ2AdWGtFDY+hHaYfi62E5eek3tNeGpmcoan69LwY5lCk3mn81aqeeSS9f9zvSEHuUYoSuXquKggw+4SxkTJgZl7UtVy48hE+Fzy2M+G4TvicRNzfpmHvbrh1KB0GFSnvHa4jNRZ+alDIqccQKJxbRkO0C0kxPeNol4JMBw1hRl6QCX1/9jvG2E8hSoe1n+sDCeUQw8pXcwIDAQABAoIBAGPectFvPVL2G3TvP+49B2kcsdPY4RutiZblMU8jEkgvlh0nJGoj5jA3wErTBcGl/+czuccOxBlgR3KyL8yea4IIe0xrJcSjjBLrksbDwiRKJql1wbZXCCJyNaUgMIbnJe329FMI5oiP7BbZn2RP2BrUr6Lulk/hdrcL2anUBX0UmALvkZFz7MiuDuGQjsb51DxZvEpE++XRVDwL2iNw3mub+GuJ0DjSOkPg/RAGahJFZL9eNsf8Me4WanMCPIyzekSB6b798vnEoYBmU7S09TwVtrt6vatkBv84zWbC88Y5zoQdNL0MUqHq6uCago2Cf8hIOVdrGimrP66ZZTRTX+ECgYEAx88GRbtFNPkV1ED+ALUD3XtaR122xLkinHiIlk1n+LatNP5gvSy7xxzfa0VqZvuVNhuvK+HS/SO5C1NiHQaq6H0+li8MQxv5P/+4xB/ZmCkg19BuWKSiXIoaP3IBYe0YE58M1uMDNNFAQXVGoi/zuSWDAhwgHN0eCabhemB3JGkCgYEAxbZPa/fxhewgS+c+Cep+usYCOlTLGRMg8vUeqWSIzhLUVruv/D9EsE6JOGWUj3NqYGO8b+gsMrklgz7gtOSYPS43+7jDM9fvg7a8elWJ5175Vihk0U++u9n3t5kfFueLpn1hLzWJeCscXucKXSQCnEtO94dSEE/xrOcbFGp98XsCgYEAiaAibON53uv2yjVd/3SvTd4KJ//3xPbUTTyEsCpvBBQnp8nTLlpimNpdVZEBoh+F/jgBZ1NrtarmaVOsz9to2yxxcJbFdnANNbTZOlXJ1hH2KlDJwMrdfqotPCg1pZLes50pBdZlvfqnrK6v0UUANjYNT+W5hMgVSYal5loNlokCgYBXJwMhi91KdIjUDK4bPuP4PvqSbfhNKFJ45rZKY0eu1zwEs51i6xzDPwtb8eMnzO+SZ8ST69s3zMeNcDUraCz3ox0IeCyL+N7ax72Ti4tLk2EDqrCuV6GzOuToaLX8qbq0fEZPwiDD+PT7nIrD/fCspsG7eUoiaKsW1ZTpTfwQawKBgQC20xXN5O8sSoJ3RCbMjxjxibrqSNxvEnpGNQx+P+15zqMKByajO6zGkk5b8svtftm3pFVdUe3t0EoaUHEDoKbzTiNIUCeDQSjbtd/qyDciEkgrI1FAGBR1N7PpWpUX1CsZ27KVREgLND+VIK5gq++40ASGi7pqIErfkoIWRpYKKQ==
    kind: Secret
    metadata:
    annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
    {“apiVersion”:”v1″,”data”:{“elastic-ca.crt”:”MIID6TCCAtGgAwIBAgIJANVcM6ZYRpA7MA0GCSqGSIb3DQEBCwUAMIGKMQswCQYD\nVQQGEwJJTjELMAkGA1UECAwCS04xEjAQBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UE\nCgwIRVJJQ1NTT04xDTALBgNVBAsMBElEVU4xEDAOBgNVBAMMB2hhaG4xMzAxJjAk\nBgkqhkiG9w0BCQEWF3NhcmFsYS5rOTlAZXJpY3Nzb24uY29tMB4XDTIzMDQxNDA2\nMzUxOFoXDTMzMDQxMTA2MzUxOFowgYoxCzAJBgNVBAYTAklOMQswCQYDVQQIDAJL\nTjESMBAGA1UEBwwJQkFOR0FMT1JFMREwDwYDVQQKDAhFUklDU1NPTjENMAsGA1UE\nCwwESURVTjEQMA4GA1UEAwwHaGFobjEzMDEmMCQGCSqGSIb3DQEJARYXc2FyYWxh\nLms5OUBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQCeBGr/D05pn5HIPN7rbUgL+7Txr0OEgZKBigG/EC1c6uQkv69NzvgSzzqnAyWm\ngbnDxDixuSKyeqyFXWoSUF3/FdCHUSVdBHsxP1lY0uaqp8JPit4Ym/lihZXCtczb\nHExlhFbsUqu30BI3D4s3QljwhCsy6LaEsAUKeMFgTWBYy8z3qSYlEzqntePbGfEM\nYPWGEslkFu05kiGcMs/FoWzbdp2hA+qOCGeRy1DXkvzGrEIsP3IX5+S1eBUYiVfm\n/yWVihPy9h5HcMapQ9nUI49L3cWWq2QPTRLC7epwfbC+uSxV1jDeLQmJ296jSYsm\n48izvQFGPSfcDegQkd/3rDXlAgMBAAGjUDBOMB0GA1UdDgQWBBReynQAkqa1GsvT\nULUJMSOK66Ks5TAfBgNVHSMEGDAWgBReynQAkqa1GsvTULUJMSOK66Ks5TAMBgNV\nHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBBNI7c6skjCpH5N08J9ezqIaNY\ntdo8aGCXCcAEOYRXer2mtLhE9OAzrHIWMSdHC/BUIhMmnJXBWGyjLzVJkdKD133s\nfk1y9XKqqVTfzo9+GKYRjfAdS5zdKi8PXVAivsnUeJZ9IpzLEznCEbhlrVA3AEkv\n0/zSvSqHwtWnAo459t1TZbku7esKfnGbhlz3d6+CL2tReuLDPJx9l4XLRS1w965j\n/mJoLaWhNerEDU4aogrKJxD+5rubRxIFezLh7ZjFcKlR6SsTgGnAAVNokAxGfBRI\nvs22uJTX7+0ecEJ5XyXMm1gIfrDFCjF3eHXG5Ci2u7mKXpQMBIImG7HPfhx2\n”,”tls.crt”:”MIIDlDCCAnwCCQC4dEuHK0c6zzANBgkqhkiG9w0BAQsFADCBijELMAkGA1UEBhMC\nSU4xCzAJBgNVBAgMAktOMRIwEAYDVQQHDAlCQU5HQUxPUkUxETAPBgNVBAoMCEVS\nSUNTU09OMQ0wCwYDVQQLDARJRFVOMRAwDgYDVQQDDAdoYWhuMTMwMSYwJAYJKoZI\nhvcNAQkBFhdzYXJhbGEuazk5QGVyaWNzc29uLmNvbTAeFw0yMzA0MTQwNzMwMzFa\nFw0yODA0MTIwNzMwMzFaMIGMMQswCQYDVQQGEwJJTjELMAkGA1UECAwCS04xEjAQ\nBgNVBAcMCUJBTkdBTE9SRTERMA8GA1UECgwIRVJJQ1NTT04xDTALBgNVBAsMBElE\nVU4xEDAOBgNVBAMMB2hhaG4xMzAxKDAmBgkqhkiG9w0BCQEWGXNhcmFsYS5rLmV4\ndEBlcmljc3Nvbi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCa\nUJr/RqvKYAHOEITeJTCsTxwwsEYKyKLAGSiAd5CHBM5nGRsoETEYbqypdJPtk9Mg\nHrc84TmFgDdaiLA8ijH0XO8wmj0OaMK6W4MZLzFifjaPjn+3hvx401keqRt7NjGl\nCNy6ICTNfuMnYB1Ya0UNj6Edph+LrYTl56Te014amZyhqfr0vBjmUKTeafzVqp55\nJL1/3O9IQe5RihK5eq4qCDD7hLGRMmBmXtS1XLjyET4XPLYz4bhO+JxE3N+mYe9u\nuHUoHQYVKe8driM1Fn5qUMipxxAonFtGQ7QLSTE942iXgkwHDWFGXpAJfX/2O8bY\nTyFKh7Wf6wMJ5RDDyldzAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHjL7Gqges0W\nAU5wY1PHK85KdURiSrVgDx/aQneghocXPlf/pGXt6IAJdnMoT2MLuV4Ic8/Vv0fe\ndUFcgmF5zuWd3wM0wDtz3SkAheIZymkBK4lfXuDK1phIJ0Ak8I6Y/maKoQDXAT9W\nIfZIeScq3vwdqCweK3AXOXg/PCIKCSSTg46Y96PLRhZGyQ/pLjnCJg8A4R50eyK9\n11EbdfYWb/rfdEzJqf3v1q0+UuKoxknHkHiMty2S3d4Wi7bJWz+9zfI0EFZO70bQ\nahbChFE0CONHMMvVRNErLNRJpDDqpI8myyEq4hp2aLT52X4//SeumylrjRgrU0sq\nMF9zFhA42Aw=\n”,”tls.key”:”MIIEpAIBAAKCAQEAmlCa/0arymABzhCE3iUwrE8cMLBGCsiiwBkogHeQhwTOZxkb\nKBExGG6sqXST7ZPTIB63POE5hYA3WoiwPIox9FzvMJo9DmjCuluDGS8xYn42j45/\nt4b8eNNZHqkbezYxpQjcuiAkzX7jJ2AdWGtFDY+hHaYfi62E5eek3tNeGpmcoan6\n9LwY5lCk3mn81aqeeSS9f9zvSEHuUYoSuXquKggw+4SxkTJgZl7UtVy48hE+Fzy2\nM+G4TvicRNzfpmHvbrh1KB0GFSnvHa4jNRZ+alDIqccQKJxbRkO0C0kxPeNol4JM\nBw1hRl6QCX1/9jvG2E8hSoe1n+sDCeUQw8pXcwIDAQABAoIBAGPectFvPVL2G3Tv\nP+49B2kcsdPY4RutiZblMU8jEkgvlh0nJGoj5jA3wErTBcGl/+czuccOxBlgR3Ky\nL8yea4IIe0xrJcSjjBLrksbDwiRKJql1wbZXCCJyNaUgMIbnJe329FMI5oiP7BbZ\nn2RP2BrUr6Lulk/hdrcL2anUBX0UmALvkZFz7MiuDuGQjsb51DxZvEpE++XRVDwL\n2iNw3mub+GuJ0DjSOkPg/RAGahJFZL9eNsf8Me4WanMCPIyzekSB6b798vnEoYBm\nU7S09TwVtrt6vatkBv84zWbC88Y5zoQdNL0MUqHq6uCago2Cf8hIOVdrGimrP66Z\nZTRTX+ECgYEAx88GRbtFNPkV1ED+ALUD3XtaR122xLkinHiIlk1n+LatNP5gvSy7\nxxzfa0VqZvuVNhuvK+HS/SO5C1NiHQaq6H0+li8MQxv5P/+4xB/ZmCkg19BuWKSi\nXIoaP3IBYe0YE58M1uMDNNFAQXVGoi/zuSWDAhwgHN0eCabhemB3JGkCgYEAxbZP\na/fxhewgS+c+Cep+usYCOlTLGRMg8vUeqWSIzhLUVruv/D9EsE6JOGWUj3NqYGO8\nb+gsMrklgz7gtOSYPS43+7jDM9fvg7a8elWJ5175Vihk0U++u9n3t5kfFueLpn1h\nLzWJeCscXucKXSQCnEtO94dSEE/xrOcbFGp98XsCgYEAiaAibON53uv2yjVd/3Sv\nTd4KJ//3xPbUTTyEsCpvBBQnp8nTLlpimNpdVZEBoh+F/jgBZ1NrtarmaVOsz9to\n2yxxcJbFdnANNbTZOlXJ1hH2KlDJwMrdfqotPCg1pZLes50pBdZlvfqnrK6v0UUA\nNjYNT+W5hMgVSYal5loNlokCgYBXJwMhi91KdIjUDK4bPuP4PvqSbfhNKFJ45rZK\nY0eu1zwEs51i6xzDPwtb8eMnzO+SZ8ST69s3zMeNcDUraCz3ox0IeCyL+N7ax72T\ni4tLk2EDqrCuV6GzOuToaLX8qbq0fEZPwiDD+PT7nIrD/fCspsG7eUoiaKsW1ZTp\nTfwQawKBgQC20xXN5O8sSoJ3RCbMjxjxibrqSNxvEnpGNQx+P+15zqMKByajO6zG\nkk5b8svtftm3pFVdUe3t0EoaUHEDoKbzTiNIUCeDQSjbtd/qyDciEkgrI1FAGBR1\nN7PpWpUX1CsZ27KVREgLND+VIK5gq++40ASGi7pqIErfkoIWRpYKKQ==\n”},”kind”:”Secret”,”metadata”:{“annotations”:{},”name”:”elastic-certificates”,”namespace”:”reg-eck”},”type”:”kubernetes.io/tls”}
    creationTimestamp: “2023-04-14T08:56:13Z”
    managedFields:
    – apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
    f:data:
    .: {}
    f:elastic-ca.crt: {}
    f:tls.crt: {}
    f:tls.key: {}
    f:metadata:
    f:annotations:
    .: {}
    f:kubectl.kubernetes.io/last-applied-configuration: {}
    f:type: {}
    manager: kubectl
    operation: Update
    time: “2023-04-14T08:56:13Z”
    name: elastic-certificates
    namespace: reg-eck
    resourceVersion: “203114252”
    uid: 755e3db4-a9ff-4b2d-8b1d-add16a2f5223
    type: kubernetes.io/tls

    • When you decode your base64 data from the secret, do you get certificates/keys in plain text? If you don’t, then that would be the problem. You need to encode plain text certificates/keys using base64 and add them to the secret.

    • Used above commands for generating root ca certificates, domain crt and keys
      Passed Internal Certificate Authority (CA) commands for generating these crts and keys (domain.crt,domain.csr, root-ca.crt, root-ca.key and root-ca.srl) from these keys and crts converted to base.64 by using this command:

      After generating the crt.b64 files added to elasticertificate.yaml
      Applied to these secret to namespace .
      Did i missed any steps here

    • That looks OK to me. Have you verified that you can decode base64 encoded secret data and see plaintext content of you certificates/keys?

  7. Hi Lisenet,

    Decoded base64 to plain text .
    can see below text its showing valid details.

    Is there any other way to check this.

  8. Have doubt on domain , which domain should provide whether it should be kibana url or elasticsearch url

  9. Hi,

    Shiwa here, I had used your solution along with https://blog.searce.com/deploying-a-secure-elasticsearch-environment-on-kubernetes-deb0f981ddf5
    But when I try to bring up the elasticsearch pods those are failing with error like

    ERROR

    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,904Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “stopping …”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,908Z”, “level”: “INFO”, “component”: “o.e.x.w.WatcherService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “stopping watch service, reason [shutdown initiated]”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,908Z”, “level”: “INFO”, “component”: “o.e.x.m.p.l.CppLogMessageHandler”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “[controller/180] [Main.cc@174] ML controller exiting”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,909Z”, “level”: “INFO”, “component”: “o.e.x.m.p.NativeController”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Native controller process has stopped – no new native processes can be started”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:00,909Z”, “level”: “INFO”, “component”: “o.e.x.w.WatcherLifeCycleService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “watcher has stopped and shutdown”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,174Z”, “level”: “INFO”, “component”: “o.e.t.ClusterConnectionManager”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “transport connection to [{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw}] closed by remote”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,177Z”, “level”: “INFO”, “component”: “o.e.c.r.a.AllocationService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “Cluster health status changed from [GREEN] to [YELLOW] (reason: [{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw} reason: disconnected]).”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,179Z”, “level”: “INFO”, “component”: “o.e.c.s.MasterService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “node-left[{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw} reason: disconnected], term: 24, version: 474, delta: removed {{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw}}”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,183Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “master node changed {previous [{elasticsearch-master-0}{IwKBzipXTeaOzFXgoj4x9Q}{C91kj8TuTpWcUVdeSWnk-w}{10.50.5.209}{10.50.5.209:9300}{cdfhilmrstw}], current []}, term: 24, version: 473, reason: becoming candidate: Publication.onCompletion(false)”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,184Z”, “level”: “WARN”, “component”: “o.e.c.s.MasterService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “failing [node-left[{elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw} reason: disconnected]]: failed to commit cluster state version [474]”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” ,
    “stacktrace”: [“org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1772) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1679) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.start(Publication.java:61) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1380) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:305) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]”,
    “at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]”,
    “at java.lang.Thread.run(Thread.java:833) [?:?]”,
    “Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:163) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “… 14 more”] }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,187Z”, “level”: “ERROR”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “unexpected failure during [node-left]”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” ,
    “stacktrace”: [“org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1772) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1679) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Publication.start(Publication.java:61) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1380) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:305) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.1.jar:7.17.1]”,
    “at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]”,
    “at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]”,
    “at java.lang.Thread.run(Thread.java:833) [?:?]”,
    “Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum”,
    “at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:163) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “… 14 more”] }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,196Z”, “level”: “WARN”, “component”: “o.e.c.NodeConnectionsService”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “failed to connect to {elasticsearch-master-1}{3yxrm2CBSYKQtGhHqeW0jA}{hKjH6eltT9GqcscqiH15OQ}{10.50.9.173}{10.50.9.173:9300}{cdfhilmrstw}{ml.machine_memory=2147483648, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true} (tried [1] times)”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” ,
    “stacktrace”: [“org.elasticsearch.transport.ConnectTransportException: [elasticsearch-master-1][10.50.9.173:9300] connect_exception”,
    “at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1047) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:279) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.17.1.jar:7.17.1]”,
    “at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]”,
    “at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]”,
    “at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]”,
    “at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]”,
    “at org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:58) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[?:?]”,
    “at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]”,
    “at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]”,
    “at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) ~[?:?]”,
    “at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]”,
    “at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]”,
    “at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]”,
    “at java.lang.Thread.run(Thread.java:833) [?:?]”,
    “Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.50.9.173/10.50.9.173:9300”,
    “Caused by: java.net.ConnectException: Connection refused”,
    “at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]”,
    “at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]”,
    “at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]”,
    “at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]”,
    “at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]”,
    “… 7 more”] }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,439Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “stopped”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,440Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “closing …”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }
    {“type”: “server”, “timestamp”: “2024-01-12T11:49:01,481Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “elasticsearch”, “node.name”: “elasticsearch-master-0”, “message”: “closed”, “cluster.uuid”: “Gk3KJQlMQ-e1OEGNuen6vw”, “node.id”: “IwKBzipXTeaOzFXgoj4x9Q” }

    Secrets I had used
    [azadmin@l21q1382s005001 gpp]$ kubectl get secret elastic-secret -o jsonpath='{.data}’
    {“ES_PASSWORD”:”cGFzc3dvcmQ=”,”ES_USERNAME”:”ZWxhc3RpYw==”}[azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$ echo “cGFzc3dvcmQ=” | base64 –decode
    password[azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$ echo “ZWxhc3RpYw==” | base64 –decode
    elastic[azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$
    [azadmin@l21q1382s005001 gpp]$ kubectl get secret elastic-certificates -o jsonpath='{.data}’
    {“elastic-certificates.p12″:”MIIOCAIBAzCCDbIGCSqGSIb3DQEHAaCCDaMEgg2fMIINmzCCBbIGCSqGSIb3DQEHAaCCBaMEggWfMIIFmzCCBZcGCyqGSIb3DQEMCgECoIIFQDCCBTwwZgYJKoZIhvcNAQUNMFkwOAYJKoZIhvcNAQUMMCsEFDzss3hKqQoNyk1i5YUSNgw8CUBkAgInEAIBIDAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQvQpeljqUxniTMTiX7Hq0FgSCBNDf60kzS3enXucBMvbUPRvsshPGiXdYuIMi82eTAxmE2Z88JX1bKyQG1b/bAAAqKOEOvEHTuear0bjQ0xqkl0YFWpnU87mxMGrFFDfdvocrpruwxLKKrUU+VZxwWkyhy64ev5No0IQjMZMhFy6Hl/kCu14iPPEtmPXDT+OMwJ49wENTx8H9B4mLfqNsp/rZJ4zOdqOeCw/Irr0TlJTiGBzfgtrKH9T0xhIaoFl49vbqnABWZZcZqGXIy0ap8jc0Vpp2DPb8V+/yqja/13mllGPx9GxYpkdSW2aNOwwcRXH00luVwu4pWTGlQNNPlHPYov+b29E4XevnA5NNjP25upuH19pMgfv0XAap2vssugVW538kL7g6WY0zD4eRpSYM7yzKEoaZH7CPMJ39EbD0eucoVFRhA0KB1JsErDdDBsj077NBYxJ+CQWIhbTnS7Kb7Rq2VpLYqr8cP4dUEgfBEvbg3MVpVH7F62nUqfnSKE7js4dy382Xpki1aL0LLktnMAUebensjl8oLg4o9P4f0VqJ1YAAXslJxOsNI8h4rR0Yff8qu80oaFprypkPVPqwdttRXMOoNb0lTlAA1zjctZAATgU4434VN3BHDfozjXk+EXoaoH7r2Ww5EzCKQFo/xGpVZW6fN8R8jLb6y3Eopa/pQUfscmVTyyE4IAlSfJMehU4urgbyfm7clBpmW5RtfyEgDaeZ1J+JYxrZNv6Z6BdaAxB1gShsSLikFPg6w3Mpca8UHrbIM1y/UTkBNjmFLDzSlAxq43oguURBTf8eipByhCW9fdQpcOwlIFdUtaUft9bvEwLZxjhviRO3il2KRCcuFxeAtYN7fTiqC1pm8+JFMyoAnBSzktzVFF4yZE6RxA0ShHE46zDh8sO4Q3rjmta0Z2J5rATcJ3h+NDrWNhg+yGhQH5ruwyqHoJOT+2bm8G0yZYN0JbqLTgC7vrjiR49z6nvnY6ZsAnlNMj+JLPAbf9aPpohf31XJ2KgujiTXvB74L4l9a5jWkYm1Nq7djf9Rt+dXGx/41hLhBOfqo/OLbO32g/vMASUtcHapGeOH7XYew1ImeVQd4aswJMNg4RE6E+RF5nOdoI8jbWhxHQqVTYEPSuHfTrrNM3mou6hCL9VMLlkxDQSvyGk73ynZRnZFp6uhtqWt3vSOr1KWnvIZCJZ8J4V9YoADmfiIcJg/o3BmbYqDt+k33UvxXdo/XakfCSPZLtGHrZgiKaahlmu+BvbDyEsCQsyDMznRCnHkNSfcgRplaikHIn7+E3WPyG/BNAgI1saza33r6nwcrckVMjtexsN/MQ3YrF3H0K5THvN0UcDjq0obvEXOOgHt6ItNi1jkGs4Jm9dGtu5dmfQ49Yl2eN1gi0aB1LEzSTqronSk/6SlYUJzxd7jjIWUl0x1hx25yyqBuyxcTlk7D4cSKskPbDok7w91Yq4qkTtuBDCr5UC+gTWEpYiwk03xHyO8KBqGr9tR8XIVNGX/9rAe9o5gL/pOlVhWDlAzOBbby3sU6Q4dL2m4JmsKy4BQvwAMwWMrGyCKcZuML9WEWteoX4lzu1vsbJgwE4Z42KjIAjf150A06Tf7woirja8CwejaL/JsWprfWBjvLJWyIwaDZAbmIZ8mBAcskwkD73jukTFEMB8GCSqGSIb3DQEJFDESHhAAaQBuAHMAdABhAG4AYwBlMCEGCSqGSIb3DQEJFTEUBBJUaW1lIDE3MDUwNTc4OTI5MjgwggfhBgkqhkiG9w0BBwagggfSMIIHzgIBADCCB8cGCSqGSIb3DQEHATBmBgkqhkiG9w0BBQ0wWTA4BgkqhkiG9w0BBQwwKwQUkNrezGXW41fq85vOMFlEOIBaVj4CAicQAgEgMAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBABtfdn4q80sOdnX/oecwMzgIIHUPmdEEg+gmpBUX1UJNJ/+dFIlgGPhMZbzminH1iPVgUollZo4An7oprdLioGQI7uh4BBIcn4s168rm+MmR0e0Kzts+Rt0Sl9QasgrYi0eAydxgLJd701KkJH8HWfjmWlW+LPXT/IPUnI+WaooSoHXy1RVOmP3T6vQAPrCEMzguuGUupReqqIQu+Kx95JQBfXDQZ2raQ0hsV75KkcVfkJt05LmkMD8GvQ0ydYBz8BX30iK+hriFdLUPXrE+SFyGHP5rz4H9u3TDZqDM2oDMAWwKbbfZCXko+yBWri3SkQ4Nhut8LgusbSfZwBWuG7tjpSim6oynyFXEqSiInPP3UZ/JSVMDPNAaPv8MLnCGKS+WlD0wGnffdCuQpnjRpeyK2UK5tOWYIUPtikh1IVaXLUlkW8McODx41KOwqC18KJND1y/SCJw35I41Y2BsaPtRvei5CfQ7Lu8L1M7OnQ0FzZwde2wlh9V2qONmcv978/dPBt8LLJ94Ke1pJSxmcoDEnTtPC4wonuYp3oc60B5emo2SfgwZOHN/3GDPAh9/AzbuXIPli3D9CD+Sz4JLAUXHkYNJs6bqvNJPi2GrXCIUWlKq6VmFo1XVKgzNo/NzfKyo24tJi5noJ6HfJUw1H0DGO0/kd3120BLSAsUPVul0b1EAOrEKEyQwn0utjWB7CItKPj3cKbvDjR6L7B30+kyl9JYhUydilVoSquRHY3gkzVEtafEo8tsswRb7zI/5giyLD7cwNRUjmKl6JP9z5Dc1/P46Sc0MSR1d0YAClCXrjmrBzxFnNZ6URvVWwATvpHyQgQreyqmo1zw8//BINf1IL7abCqFOKfrJENAL4frBffZGjXLAxUSAYDPjhx6QQh4GWZi2OBtQkGTdPn5DlW2UrVZMA9kzIJisSVAsqYtOMVATylYdpnwnX3U1wSIgn9wG3NVJyey+Puyj/FNsEU2x4194nF9AQIaSc4J9RNbCNI6k9RNTz97HzciNOt5VywinqzGIAYLfrGi685WlCnVdmjdRCczTc13e04CBaqPMQhS0M75uznSeS1UWFYIilQLbXvBDywRB0U7o02iKBj2iifu4XNfOeoOEEfT+FoQmtAjs8/eAYZ9fbaaiYqIjE9dhb+puKsiW8Pk2KB2KD1JjfKez1RN71IZH6SuHx2T60ZZqcerQVm7MWASU0C7ERth6uW+6yrVqHVpBRjG6j1x5K8r7RUoKjMIJVsgP+1mgZEGBfp7ZCmhkNVq1UhAh95s7w1zqJoa/JH2bv3PVUrfhFvjCMJfHkoC4iLgykwhq6tYZL9G3YDx3L7KZzRK6EVzsO7LfayQj81+BiDk/JagSV4oKQoxQUrzB4SCqsR9JeHXKWToltqFeTzEXk32e6Fxw6kV0hxB/QsrBE8DZ4D0Ifyf4RncfvSPYonV/j3BONx3FvQ4XfDzNmPBh2Sv9J1btnxGVPPCGtihpfzE8urNjlvQjuemQg4K09S+cleXXxGwCn4+3x3Sm+azJ4muw4ZlaTIkXHlASloUFYWoYKDUWHF6Q8nCOnD+BnI+akwvvA2zP4mX24ACldKtofxDPT1HWrC82hfmtYZQmD50OUR6zaTc3+Ju3bno5mLorw2SezxErNNTbRUIi1aI+3T1pDfXJouzgoQaM02wylB3YIWIxFLztN9HRChaKxRIpYI2ZStoz1CRhtgvBctFs3yhXSXgUYZ3Lk15Elt5F61MJaEZ9O8PUVPo5Wy6u++PepbqZUpUBkocxGkohwVJLdTO+zbt5QEi+3jvQ+0EfaEK6IbRC3WUic1zmVuGpFyPMWyifBDD2SVWKSSPmcUwb+naucdiiLLtSIa8YQV1roFiNlpZEaOC1QwSMA2z6CF0neg3YgCEtxglPcbgslw57ODJvgOHZY3IDSBktHU3V+jrTRJ1ufsvl3f6/jzGLoRrlIoR+w/azorKd4BpHcCSiJer/h7Qir+nA8hujuMzx5IicJqKudeiwSIyfjdUDXlZgNqDIEyL8gVEQeBVIFADufu4L7jm2ljkSTG0jKn/3TpqUrHuWc5vfJVogpBRaEmdwIl+jZXuAj8ej9rGHk7KqnLAWxLycY3m0vHyinY387zbtUNkJvmWB2pftsgkw6cpe+QSZqDlT/DqrCmkBiMqOVatNbmGZc2vFiIlsaHugR8VCiDZsdFJChx5cI/Vllz2eWGg7wa6m4Jj5q8ecdUJhrbLvqn6hywqsK3FRJBRbCoKmrRSxg9g0tKL9dYjFG1O2JgJbDN1hZfkGVkPFAeHew6wGeB9BwmYsGRGa7hujtJy+lSlkGe5mlWUDXA6PPlareg6IdDhQC8J6qzGA+E6ZY8b/M4ZAyIziSYn2p+Luc/wvTjcCNBc8/PBIWjXK4GWXiAxgYO4w/zCo8mqhXyq2n33mrU0W2dQ3zgdlYnnhuoSDCxjbAIcn6gyeRnz35IdbU2FXTfniW/7ewBixc2TMjbkEwDI5AlGZqT5Cu70MaRseeA1E1q7DBNMDEwDQYJYIZIAWUDBAIBBQAEIFZcXHS1nbOgP8gq29uuabJfNeQxecN2pcM5fFADa39MBBTZny4HajWnTma/diX6Gokivrnm9QICJxA=”}[azadmin@l21q1382s005001 gpp]$ echo “MIIOCAIBAzCCDbIGCSqGSIb3DQEHAaCCDaMEgg2fMIINmzCCBbIGCSqGSIb3DQEHAaCCBaMEggWfMIIFk1i5YUSNgw8CUBkAgInEAIBIDAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQvQpeljqUxniTMTiX7Hq0FgSCBNDf60kzS3enXucBMvbUPRvsshPGiXdYuIMi82eTAxmE2Z88JX1bKyQG1b/bAAAqKOEOvEHTuear0bjQ0xqkl0YFWpnU87mxMGrFFDfdvocrpruwxLKKrUU+VZxwWkyhy64ev5No0IQjMZMhFy6Hl/kCu14iPPEtmPXDT+OMwJ49wENTx8H9B4mLfqNsp/rZJ4zOdqOeCw/Irr0TlJTiGBzfgtrKH9T0xhIaoFl49vbqnABWZZcZqGXIy0ap8jc0Vpp2DPb8V+/yqja/13mllGPx9GxYpkdSW2aNOwwcRXH00luVwu4pWTGlQNNPlHPYov+b29E4XevnA5NNjP25upuH19pMgfv0XAap2vssugVW538kL7g6WY0zD4eRpSYM7yzKEoaZH7CPMJ39EbD0eucoVFRhA0KB1JsErDdDBsj077NBYxJ+CQWIhbTnS7Kb7Rq2VpLYqr8cP4dUEgfBEvbg3MVpVH7F62nUqfnSKE7js4dy382Xpki1aL0LLktnMAUebensjl8oLg4o9P4f0VqJ1YAAXslJxOsNI8h4rR0Yff8qu80oaFprypkPVPqwdttRXMOoNb0lTlAA1zjctZAATgU4434VN3BHDfozjXk+EXoaoH7r2Ww5EzCKQFo/xGpVZW6fN8R8jLb6y3Eopa/pQUfscmVTyyE4IAlSfJMehU4urgbyfm7clBpmW5RtfyEgDaeZ1J+JYxrZNv6Z6BdaAxB1gShsSLikFPg6w3Mpca8UHrbIM1y/UTkBNjmFLDzSlAxq43oguURBTf8eipByhCW9fdQpcOwlIFdUtaUft9bvEwLZxjhviRO3il2KRCcuFxeAtYN7fTiqC1pm8+JFMyoAnBSzktzVFF4yZE6RxA0ShHE46zDh8sO4Q3rjmta0Z2J5rATcJ3h+NDrWNhg+yGhQH5ruwyqHoJOT+2bm8G0yZYN0JbqLTgC7vrjiR49z6nvnY6ZsAnlNMj+JLPAbf9aPpohf31XJ2KgujiTXvB74L4l9a5jWkYm1Nq7djf9Rt+dXGx/41hLhBOfqo/OLbO32g/vMASUtcHapGeOH7XYew1ImeVQd4aswJMNg4RE6E+RF5nOdoI8jbWhxHQqVTYEPSuHfTrrNM3mou6hCL9VMLlkxDQSvyGk73ynZRnZFp6uhtqWt3vSOr1KWnvIZCJZ8J4V9YoADmfiIcJg/o3BmbYqDt+k33UvxXdo/XakfCSPZLtGHrZgiKaahlmu+BvbDyEsCQsyDMznRCnHkNSfcgRplaikHIn7+E3WPyG/BNAgI1saza33r6nwcrckVMjtexsN/MQ3YrF3H0K5THvN0UcDjq0obvEXOOgHt6ItNi1jkGs4Jm9dGtu5dmfQ49Yl2eN1gi0aB1LEzSTqronSk/6SlYUJzxd7jjIWUl0x1hx25yyqBuyxcTlk7D4cSKskPbDok7w91Yq4qkTtuBDCr5UC+gTWEpYiwk03xHyO8KBqGr9tR8XIVNGX/9rAe9o5gL/pOlVhWDlAzOBbby3sU6Q4dL2m4JmsKy4BQvwAMwWMrGyCKcZuML9WEWteoX4lzu1vsbJgwE4Z42KjIAjf150A06Tf7woirja8CwejaL/JsWprfWBjvLJWyIwaDZAbmIZ8mBAcskwkD73jukTFEMB8GCSqGSIb3DQEJFDESHhAAaQBuAHMAdABhAG4AYwBlMCEGCSqGSIb3DQEJFTEUBBJUaW1lIDE3MDUwNTc4OTI5MjgwggfhBgkqhkiG9w0BBwagggfSMIIHzgIBADCCB8cGCSqGSIb3DQEHATBmBgkqhkiG9w0BBQ0wWTA4BgkqhkiG9w0BBQwwKwQUkNrezGXW41fq85vOMFlEOIBaVj4CAicQAgEgMAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBABtfdn4q80sOdnX/oecwMzgIIHUPmdEEg+gmpBUX1UJNJ/+dFIlgGPhMZbzminH1iPVgUollZo4An7oprdLioGQI7uh4BBIcn4s168rm+MmR0e0Kzts+Rt0Sl9QasgrYi0eAydxgLJd701KkJH8HWfjmWlW+LPXT/IPUnI+WaooSoHXy1RVOmP3T6vQAPrCEMzguuGUupReqqIQu+Kx95JQBfXDQZ2raQ0hsV75KkcVfkJt05LmkMD8GvQ0ydYBz8BX30iK+hriFdLUPXrE+SFyGHP5rz4H9u3TDZqDM2oDMAWwKbbfZCXko+yBWri3SkQ4Nhut8LgusbSfZwBWuG7tjpSim6oynyFXEqSiInPP3UZ/JSVMDPNAaPv8MLnCGKS+WlD0wGnffdCuQpnjRpeyK2UK5tOWYIUPtikh1IVaXLUlkW8McODx41KOwqC18KJND1y/SCJw35I41Y2BsaPtRvei5CfQ7Lu8L1M7OnQ0FzZwde2wlh9V2qONmcv978/dPBt8LLJ94Ke1pJSxmcoDEnTtPC4wonuYp3oc60B5emo2SfgwZOHN/3GDPAh9/AzbuXIPli3D9CD+Sz4JLAUXHkYNJs6bqvNJPi2GrXCIUWlKq6VmFo1XVKgzNo/NzfKyo24tJi5noJ6HfJUw1H0DGO0/kd3120BLSAsUPVul0b1EAOrEKEyQwn0utjWB7CItKPj3cKbvDjR6L7B30+kyl9JYhUydilVoSquRHY3gkzVEtafEo8tsswRb7zI/5giyLD7cwNRUjmKl6JP9z5Dc1/P46Sc0MSR1d0YAClCXrjmrBzxFnNZ6URvVWwATvpHyQgQreyqmo1zw8//BINf1IL7abCqFOKfrJENAL4frBffZGjXLAxUSAYDPjhx6QQh4GWZi2OBtQkGTdPn5DlW2UrVZMA9kzIJisSVAsqYtOMVATylYdpnwnX3U1wSIgn9wG3NVJyey+Puyj/FNsEU2x4194nF9AQIaSc4J9RNbCNI6k9RNTz97HzciNOt5VywinqzGIAYLfrGi685WlCnVdmjdRCczTc13e04CBaqPMQhS0M75uznSeS1UWFYIilQLbXvBDywRB0U7o02iKBj2iifu4XNfOeoOEEfT+FoQmtAjs8/eAYZ9fbaaiYqIjE9dhb+puKsiW8Pk2KB2KD1JjfKez1RN71IZH6SuHx2T60ZZqcerQVm7MWASU0C7ERth6uW+6yrVqHVpBRjG6j1x5K8r7RUoKjMIJVsgP+1mgZEGBfp7ZCmhkNVq1UhAh95s7w1zqJoa/JH2bv3PVUrfhFvjCMJfHkoC4iLgykwhq6tYZL9G3YDx3L7KZzRK6EVzsO7LfayQj81+BiDk/JagSV4oKQoxQUrzB4SCqsR9JeHXKWToltqFeTzEXk32e6Fxw6kV0hxB/QsrBE8DZ4D0Ifyf4RncfvSPYonV/j3BONx3FvQ4XfDzNmPBh2Sv9J1btnxGVPPCGtihpfzE8urNjlvQjuemQg4K09S+cleXXxGwCn4+3x3Sm+azJ4muw4ZlaTIkXHlASloUFYWoYKDUWHF6Q8nCOnD+BnI+akwvvA2zP4mX24ACldKtofxDPT1HWrC82hfmtYZQmD50OUR6zaTc3+Ju3bno5mLorw2SezxErNNTbRUIi1aI+3T1pDfXJouzgoQaM02wylB3YIWIxFLztN9HRChaKxRIpYI2ZStoz1CRhtgvBctFs3yhXSXgUYZ3Lk15Elt5F61MJaEZ9O8PUVPo5Wy6u++PepbqZUpUBkocxGkohwVJLdTO+zbt5QEi+3jvQ+0EfaEK6IbRC3WUic1zmVuGpFyPMWyifBDD2SVWKSSPmcUwb+naucdiiLLtSIa8YQV1roFiNlpZEaOC1QwSMA2z6CF0neg3YgCEtxglPcbgslw57ODJvgOHZY3IDSBktHU3V+jrTRJ1ufsvl3f6/jzGLoRrlIoR+w/azorKd4BpHcCSiJer/h7Qir+nA8hujuMzx5IicJqKudeiwSIyfjdUDXlZgNqDIEyL8gVEQeBVIFADufu4L7jm2ljkSTG0jKn/3TpqUrHuWc5vfJVogpBRaEmdwIl+jZXuAj8ej9rGHk7KqnLAWxLycY3m0vHyinY387zbtUNkJvmWB2pftsgkw6cpe+QSZqDlT/DqrCmkBiMqOVatNbmGZc2vFiIlsaHugR8VCiDZsdFJChx5cI/Vllz2eWGg7wa6m4Jj5q8ecdUJhrbLvqn6hywqsK3FRJBRbCoKmrRSxg9g0tKL9dYjFG1O2JgJbDN1hZfkGVkPFAeHew6wGeB9BwmYsGRGa7hujtJy+lSlkGe5mlWUDXA6PPlareg6IdDhQC8J6qzGA+E6ZY8b/M4ZAyIziSYn2p+Luc/wvTjcCNBc8/PBIWjXK4GWXiAxgYO4w/zCo8mqhXyq2n33mrU0W2dQ3zgdlYnnhuoSDCxjbAIcn6gyeRnz35IdbU2FXTfniW/7ewBixc2TMjbkEwDI5AlGZqT5Cu70MaRseeA1E1q7DBNMDEwDQYJYIZIAWUDBAIBBQAEIFZcXHS1nbOgP8gq29uuabJfNeQxecN2pcM5fFADa39MBBTZny4HajWnTma/diX6Gokivrnm9QICJxA” | base64 –decode
    ▒▒▒▒▒0▒▒0▒▒▒▒
    *▒H▒▒

    0Y08▒<0f*▒H▒▒ *▒H▒▒
    0+<▒xJ▒
    ▒Mb▒6
    < @d' 0
    *▒H▒▒ 0 `▒He*▒
    ^▒:▒▒x▒18▒▒z▒▒▒▒▒I3Kw▒^▒2▒▒=▒ƉwX▒▒"▒g▒▒ٟU▒pZL▒ˮ▒▒hЄ#1▒!.▒▒▒▒^”<▒-▒▒▒O▒▒=▒CS▒▒▒▒▒~▒l▒▒▒'▒▒v▒▒
    Ȯ▒▒▒▒߂▒▒▒▒▒▒Yx▒▒▒Ve▒▒e▒▒F▒▒74V▒v
    ▒▒W▒▒6▒▒y▒▒c▒▒lX▒GR[f▒;
    Eq▒▒[▒▒▒)Y1▒@▒O▒sآ▒▒▒▒8]▒▒▒M▒▒▒▒L▒▒▒\▒▒▒,▒V$/▒:Y▒3▒▒▒&
    ▒,▒▒▒▒▒0▒▒▒▒z▒(TTaB▒ԛ▒7C▒▒▒Ac~ ▒▒▒▒K▒▒▒▒V▒ت▒?▒T▒▒▒▒▒iT~▒▒iԩ▒▒(N㳇r▒͗▒H▒h▒
    ▒▒ԟ▒c▒6▒▒▒Zu▒(lH▒▒▒:▒s)q▒▒▒3\▒Q969▒,▒hP▒▒▒*▒▒▒▒▒f▒▒m2e▒t%▒▒N▒▒▒▒G▒s▒{▒c▒lyM2?▒,▒▒▒_▒U▒ب.▒$׼▒/▒}k▒֑▒▒6▒ݍ▒Q▒▒W▒▒▒▒▒l▒▒▒▒%-pv▒▒▒v▒R&yT▒0$▒`▒:▒E▒s▒▒▒#mhq
    ▒▒i;▒)▒FvE▒▒▒▒▒▒▒▒R▒▒▒|’▒}b▒▒▒p▒?▒pfm▒▒▒▒7▒K▒]▒?]▒ #▒.ч▒▒”)▒▒▒k▒▒▒▒KB̃39▒
    ج]▒ЮS▒tQ▒▒JE▒:▒▒M▒X▒▒|▒▒▒▒F▒▒]▒▒8▒vx▒`▒F▒Ա3I:▒▒t▒▒▒▒aBs▒▒㌅▒▒Lu▒▒▒*▒▒,\NY;▒*▒l:$▒ub▒*▒;n0▒▒@▒▒5▒▒▒▒▒M▒#▒(▒▒▒Q▒r4e▒▒▒`/▒N▒XVP38▒▒{▒/i▒&k
    ˀP▒
    0Y08▒▒0▒*▒H▒▒ 17*▒H▒▒8929280▒▒▒�*▒H▒▒▒/▒lZ▒▒X▒,▒▒#▒d▒!▒&,▒ ▒x▒1D0 *▒H▒▒
    0+▒▒▒▒e▒▒W▒▒▒0YD8▒ZV>’ 0
    *▒H▒▒ 0 `▒He*▒▒g▒4▒▒g_▒s3▒▒P▒H>▒jAQ}T$▒▒H▒▒▒▒[▒h▒X▒V(▒Vh▒ ▒▒.*@▒A!▒▒^▒▒o▒▒Ь▒▒m▒)}A▒ ▒▒▒x
    v▒▒4▒▒{▒U▒u▒▒e▒[▒NK▒C▒k▒▒’X?_}”+▒k▒WKP▒▒▒▒a▒▒▒۷L6j▒▒I@▒
    ▒▒▒}▒▒▒▒▒j▒▒)▒▒n▒▒▒▒▒}▒ZỶ:R▒n▒▒|▒\J▒▒▒▒?u▒03▒▒▒▒▒b▒▒iC▒▒}▒B▒
    g▒^ȭ▒+▒NY▒>ؤ▒RirԖE▒1ÃǍJ;
    ▒▒�4=r▒ ▒▒~H▒V6Ə▒▒▒▒C▒▒▒L▒▒▒▒\▒▒׶▒X}Wj▒6g/▒?t▒m▒▒▒֒R▒g(
    IӴ▒�▒b▒▒s▒▒▒▒’▒▒▒7▒▒
    ▒!▒▒3n▒▒>X▒Ѓ▒,▒$▒\y4▒:n▒▒$▒▒▒!E▒*▒▒▒Z5]R▒▒▒?77▒ʍ▒▒▒▒▒▒z▒T▒Q▒
    ▒▒▒dh▒,- ,P▒n▒F▒▒▒2C ▒▒▒▒▒▒▒▒▒�▒8▒▒▒▒O▒▒_Ib2v)U▒*▒Dv7▒L▒֟▒-▒▒o▒▒▒▒”Ȱ▒sQR9▒▒▒O▒>Cs_▒㤜▒đ▒▒)B^▒▒▒sY▒DoUlN▒G▒쪚▒s▒▒▒▒_Ԃ▒i▒▒⟬▒
    TH>8q▒!▒e▒▒c▒▒ M▒▒▒9V▒J▒d▒=▒2 ▒ĕʘ▒▒<▒a▒g▒u▒S\" ▒▒m▒T▒▒▒▒▒▒?▒6▒▒5▒▒i'8'▒Ml#H▒OQ5<▒▒|܈ӭ▒\▒▒z▒▒-▒Ƌ▒9ZP▒U٣u▒▒75▒▒▒<▒!KC;▒▒▒I▒QaX")P-▒▒<▒D▒6▒▒c▒(▒▒▒▒|▒8AO▒hBk@▒▒?x▒▒▒j&*"1=v▒▒⬉o▒b▒ؠ▒&7▒{=Q7▒Hd~▒▒|vO▒f▒▒f▒ŀIM▒Dm▒▒▒▒V▒դc▒ǒ▒▒▒T▒▒▒ ▒l▒▒▒▒D▒퐦▒CU▒U!y▒▒5΢hk▒Gٻ▒=U+~o▒# |y(
    ▒▒▒)0▒▒▒a▒▒▒r▒)▒▒+▒▒û-▒B?5▒▒▒▒Z▒%x▒▒(▒+▒
    ▒Ї▒gq▒▒=▒'W▒▒▒q▒[▒▒w▒▒ُ▒▒▒un▒▒Skb▒▒▒˫69oB;▒8+OR▒▒^]|F▒)▒▒|wJo▒̞&▒▒▒ȑq▒)hPV▒▒▒Qa▒▒▒▒▒▒▒0▒▒6▒▒&_n
    WJ▒▒▒
    ▒▒j▒▒h_▒▒B`▒▒▒▒6▒▒▒v磙▒▒▒6I▒▒▒MM▒T"-Z#▒▒֐▒\▒.▒
    h▒6▒)A݂#K▒▒}▒h▒Q"ٔ▒▒=BF▒-▒▒t▒▒Fܹ5▒Im▒^▒0▒▒gӼ=EO▒▒▒▒▒=▒[▒▒)P(s▒▒$▒S;▒۷▒▒▒▒▒▒+▒
    -▒R'5▒en▒rg▒▒j▒▒”˵”▒ֺ▒▒idF▒
    T0H▒6Ϡ▒▒w▒݈▒`▒▒▒p糃&▒▒7 4▒▒▒▒▒_▒▒4I▒▒▒]▒▒▒▒▒▒R(G▒?k:+)▒▒wJ”^▒▒{B*▒▒!▒;▒▒H▒▒j*▒^▒▒▒▒▒P5▒fj
    ▒2/▒▒T▒@▒▒▒▒ic▒$▒▒2▒▒t▒Jǹg9▒▒U▒
    ▒▒▒X▒~▒ ▒▒▒▒I▒▒▒?ê▒▒▒▒▒▒Z▒▒▒▒6▒X▒▒Ƈ▒|T(▒f▒E$(q▒▒?VYs▒冃▒▒n ▒▒▒y▒T&▒.▒▒▒▒·AE▒▒*j▒K=▒KJ/▒X▒Q▒;b`%▒▒▒_▒ed<P▒:▒g▒▒&b▒▒▒;I▒▒R▒A▒▒iVP5▒▒▒▒j▒▒▒C▒▒'▒▒▒▒<o▒8d
    ▒▒$▒▒j~.▒?▒▒▒p#As▒▒▒▒\▒Yx▒▒▒▒
    ▒&▒▒i▒▒j`▒He V\\t▒▒▒▒?▒*▒ۮi▒_5▒1y▒v▒▒9|PLٟ.j5▒Nf▒v%▒▒"▒▒▒▒'base64: invalid input
    [azadmin@l21q1382s005001 gpp]$ xterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256colorxterm-256color

    Contents of values.yaml file


    clusterName: "elasticsearch"
    nodeGroup: "master"

    # These will be set as environment variables. E.g. node.master=true
    roles:
    master: "true"
    ingest: "true"
    data: "true"
    remote_cluster_client: "true"
    ml: "true"

    replicas: 3
    minimumMasterNodes: 2

    esMajorVersion: ""

    clusterDeprecationIndexing: "false"
    ## shiva added
    esConfig:
    elasticsearch.yml: |
    cluster.name: "docker-cluster"
    network.host: 0.0.0.0
    xpack.security.enabled: "true"
    xpack.security.transport.ssl.enabled: "true"
    xpack.security.transport.ssl.supported_protocols: "TLSv1.2"
    #xpack.security.transport.ssl.client_authentication: "none"
    xpack.security.transport.ssl.client_authentication: "required"
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    #xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/elastic-certificates.p12"
    xpack.security.transport.ssl.verification_mode: "certificate"
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.supported_protocols: "TLSv1.2"
    #xpack.security.http.ssl.client_authentication: "none"
    xpack.security.http.ssl.client_authentication: "required"
    xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/certs/tls.key"
    xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/certs/tls.crt"
    #xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/homelab-ca.crt"
    xpack.security.http.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/elastic-certificates.p12"

    ## shiva added
    extraEnvs:
    – name: "ELASTIC_PASSWORD"
    valueFrom:
    secretKeyRef:
    name: "elastic-secret"
    key: "ES_PASSWORD"
    – name: "ELASTIC_USERNAME"
    valueFrom:
    secretKeyRef:
    name: "elastic-secret"
    key: "ES_USERNAME"

    ## shiva added
    secretMounts:
    – name: "elastic-certificates"
    secretName: "elastic-certificates"
    path: "/usr/share/elasticsearch/config/certs"
    defaultMode: "0755"

    image: "docker.elastic.co/elasticsearch/elasticsearch"
    imageTag: "7.17.1"
    imagePullPolicy: "IfNotPresent"

    protocol: https
    httpPort: 9200
    transportPort: 9300

    service:
    enabled: true
    labels: {}
    labelsHeadless: {}
    type: ClusterIP
    # Consider that all endpoints are considered "ready" even if the Pods themselves are not
    # https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec
    publishNotReadyAddresses: false
    nodePort: ""
    annotations: {}
    httpPortName: https
    transportPortName: transport
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    externalTrafficPolicy: ""

    Please help me I spend around 5 days still I didn't get expected output.

    • Hi Shiwanand, you seem to be trying to deploy a clustered version of Elasticsearch:

      failing [node-left[{elasticsearch-master-1}
      non-failed nodes do not form a quorum

      This article was written for, and tested with, a single node deployment I’m afraid.

      Could you try deploying Elasticsearch using the following and post back if it does the trick?

      replicas: 1
      minimumMasterNodes: 1
      

      If so, then the issue will be related to your cluster configuration.

  10. Hey @Lisenet,

    I tried with single node then also no luck, Yes something is missing. In my case certification generation is done through elasticsearch-certutil tool

    I follow below steps
    1. Login to Elasticsearch container
    2. Generate the elastic-stack-ca.p12 cert
    3. Generate elastic-certificates.p12
    4. Create a secret using elastic-certificates.p12 and mount it
    5. Ran the helm upgrade
    Once we run the helm-upgrade, it’s failed cause with error
    Error

    “`
    “at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.1.jar:7.17.1]”,
    “… 6 more”] }
    ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
    Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
    at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
    at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
    at java.base/java.security.KeyStore.load(KeyStore.java:1473)
    at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:99)
    at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66)
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:453)
    at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1220)
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:546)
    at jav
    “`

    Thanks for your reply. Can you guide me how to automate tls/ssl configuration using helm for 3 nodes.
    If you provide me path then also fine, I have issue with certificate generation only and this entire process I need to be automated using helm charts.

    Thank you so much for your response and time.

    • Have you tried using openssl to generate SSL certificates as per this blog post? Did that work for you with a single node setup?

Leave a Reply to James Cancel reply

Your email address will not be published. Required fields are marked *