Blue/Green Deployment with Istio: Match Host Header and sourceLabels for Pod to Pod Communication

We are going to deploy two versions of the application, blue and green, and configure Istio routing so that both versions could be accessed at the same time.

Pre-requisites

We are using our Kubernetes homelab in this article. We will continue with the pii-demo application for blue/green that we had deployed some time ago with Istio mTLS.

Basic familiarity with Istio is required.

Blue/green deployment is an application release model that transfers user traffic from a previous version of a microservice to a new release, both of which are running in production, without downtime.

Source code is available on GitHub.

The Goal

To allow two versions of the pii-demo application to run simultaneously using blue/green deployment, and configure Istio routing rules to allow access to both versions based on a host header. Round robin routing should apply if no colour was specified by a user.

The following DNS names will be used:

  1. pii-demo.apps.hl.test – DNS of the application before blue/green, and round robin routing after. This can easily be managed at the DNS level to point to either blue or green using a CNAME, if required.
  2. pii-demo-blue.apps.hl.test – DNS of the blue version of the application.
  3. pii-demo-green.apps.hl.test – DNS of the green version of the application.

We should also configure Istio routing rules for the pod-to-pod communication within the mesh so that blue pods talk to other blue pods, and green pods talk to other green pods. For example, a blue version of the webserver should not be routed to the green version of the database to store data, and a blue webserver should not be routed to a green webserver if there was a need for such a pod-to-pod communication.

The PII Demo Application (Simple Database App)

The pii-demo application is a simple Apache/PHP/MySQL application that allows users to enter personally identifiable information (pii) and store it in a database. The application uses two containers: httpd and MySQL.

The block diagram showing Istio flow when an application request is made can be seen below. Yaml code of the application deployment can be found on GitHub.

We use a Gateway to describe a load balancer, operating at the edge of the mesh, that receives incoming HTTP connections. The specification describes a set of ports that should be exposed and the type of protocol to use, in our case 80 and HTTP, respectively (see below). A VirtualService can then be bound to a gateway to control the forwarding of traffic arriving at a particular host.

---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: pii-demo-gateway
  namespace: pii-demo
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

We use a VirtualService to define a set of traffic routing rules to apply when a host is addressed, in our case pii-demo.apps.hl.test. A routing rule defines matching criteria for traffic of a specific protocol (TCP, TLS or http), in our case http. If the traffic is matched, then it is sent to a named destination service (or version of it) defined in the registry, in our case httpd-server. See below.

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpd-server
  namespace: pii-demo
spec:
  hosts:
    - "pii-demo.apps.hl.test"
  gateways:
    - pii-demo-gateway
  http:
    - route:
      - destination:
          host: httpd-server
          port:
            number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpd-server
  namespace: pii-demo
  labels:
    app: httpd-server
spec:
  selector:
    app: httpd-server
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080

If we were to release a new version of the application, we would have to update the deployment yaml with a new docker image version and apply it.

This is how the frontend looks like:

The PII Demo Application: Blue/Green Deployment

We are going to change the Istio configuration to allow us to do blue/green deployment. The block diagram showing Istio flow when a blue application request is made can be seen below. The same diagram would apply if a green application request was made, it would just go to a different pod.

Add a Version Label to Kubernetes Deployment Configuration

The first thing that we want to do is to separate deployments by adding a version label. Istio recommends adding an explicit app label and version label to the specification of the pods deployed using a Kubernetes Deployment. In our case the versions won’t be standard application versions such as 1.0.0, but blue/green instead. We will have two versions of the application, a blue version, and a green version. See below.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-server-blue
  namespace: pii-demo
  labels:
    app: httpd-server
    version: blue
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd-server
      version: blue
  template:
    metadata:
      name: httpd-server
      labels:
        app: httpd-server
        version: blue
    spec:
      containers:
        - name: httpd-server
          image: lisenet/httpd-pii-demo:0.2

[... configuration omitted ...]

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-server-green
  namespace: pii-demo
  labels:
    app: httpd-server
    version: green
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd-server
      version: green
  template:
    metadata:
      name: httpd-server
      labels:
        app: httpd-server
        version: green
    spec:
      containers:
        - name: httpd-server
          image: lisenet/httpd-pii-demo:0.3

[... configuration omitted ...]

Although not displayed here, the same versioning logic applies to the MySQL deployment configuration as well.

Use Named Service Ports

Istio requires named service ports. See below.

---
apiVersion: v1
kind: Service
metadata:
  name: httpd-server
  namespace: pii-demo
  labels:
    app: httpd-server
spec:
  selector:
    app: httpd-server
  type: NodePort
  ports:
    - name: http-80 # istio requires named service ports
      port: 80
      protocol: TCP
      targetPort: 80
      nodePort: 30080
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-server
  namespace: pii-demo
  labels:
    app: mysql-server
spec:
  selector:
    app: mysql-server
  ports:
    - name: tcp-3306 # do not use "mysql" as a name, see https://github.com/istio/istio/issues/13484
      port: 3306
      protocol: TCP
      targetPort: 3306

Configure Istio Routing Rules for httpd

HttpMatchRequest allows us to specify a set of criterion to be met in order for the rule to be applied to the HTTP request. Then, each routing rule is further associated with one service version (blue or green) by using a DestinationRule. DestinationRule defines policies that apply to traffic intended for a service after routing has occurred.

Requests to the httpd webserver can either come from outside the Istio mesh via the Gateway, or from within the mesh when one pod communicates with another pod.

  1. When a request comes in to the Gateway, we want to match it based on HTTP headers.
  2. When a request is made from within the service mesh, we want to match it based on a sourceLabels.

When using sourceLabels, the top-level gateways field in the VirtualService definition must include the reserved gateway mesh for this field to be applicable.

A single VirtualService is used for sidecars inside the mesh as well as for one or more gateways. In order to apply the rules to both gateways and sidecars, we have to specify mesh as one of the gateway names.

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpd-server
  namespace: pii-demo
spec:
  # the destination hosts to which traffic is being sent to
  hosts:
    - pii-demo.apps.hl.test # cannot use "*" here since this is being combined with the mesh services
    - pii-demo-blue.apps.hl.test
    - pii-demo-green.apps.hl.test
    - httpd-server
  gateways:
    - mesh # applies internally as well as externally
    - pii-demo-gateway
  http:
    - match:
        - headers:
            host:
              exact: 'pii-demo-blue.apps.hl.test'
          gateways:
            - pii-demo-gateway # restricts this rule to apply only to ingress gateway
        - sourceLabels:
            version: blue # a blue httpd pod should only talk to another blue httpd pod
      route:
        - destination:
            host: httpd-server
            port:
              number: 80
            subset: blue
    - match:
        - headers:
            host:
              exact: 'pii-demo-green.apps.hl.test'
          gateways:
            - pii-demo-gateway
        - sourceLabels:
            version: green # a green httpd pod should only talk to another green httpd pod
      route:
        - destination:
            host: httpd-server
            port:
              number: 80
            subset: green
    - match:
        - headers:
            host:
              exact: 'pii-demo.apps.hl.test'
          gateways:
            - pii-demo-gateway
      route:
        - destination:
            host: httpd-server # no subset, round robin service routing applies
            port:
              number: 80

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpd-server
  namespace: pii-demo
spec:
  host: httpd-server
  subsets:
  - name: blue
    labels:
      version: blue
  - name: green
    labels:
      version: green

Configure Istio Routing Rules for MySQL

MySQL protocol uses TCP and not HTTP.

TCPRoute describes match conditions and actions for routing TCP traffic, and L4MatchAttributes describes L4 connection match attributes.

Requests to MySQL should never come from outside the service mesh, therefore the Gateway pii-demo-gateway has not been specified. See configuration below.

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: mysql-server
  namespace: pii-demo
spec:
  # the destination host to which traffic is being sent to
  hosts:
    - mysql-server
  gateways:
    - mesh # applies internally as well as externally
  tcp:
    - match:
        - sourceLabels:
            version: blue # a blue mysql pod should only talk to another blue mysql pod
      route:
        - destination:
            host: mysql-server
            port:
              number: 3306
            subset: blue
    - match:
        - sourceLabels:
            version: green # a green mysql pod should only talk to another green mysql pod
      route:
        - destination:
            host: mysql-server
            port:
              number: 3306
            subset: green
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: mysql-server
  namespace: pii-demo
spec:
  host: mysql-server
  subsets:
  - name: blue
    labels:
      version: blue
  - name: green
    labels:
      version: green

Test the Blue/Green Deployment

Now that we have everything configured, we should test the application. Kiali provides an easy way to get Istio traffic data visualised in real time.

When a user makes a request to pii-demo-blue.apps.hl.test, only blue versions of pods are used.

When a request is made to pii-demo-green.apps.hl.test, only green versions of pods are used.

See the Istio workload graph below. The padlock indicates that connections are mTLS encrypted.

See the versioned application graph below.

If a user were to make a request to pii-demo.apps.hl.test, it would be routed to either blue or green version using round robin because of our configuration. As mentioned earlier, this behaviour can be changed by managing routing at the DNS level, e.g. pii-demo.apps.hl.test could be a CNAME pointing to pii-demo-blue.apps.hl.test before the deployment of the green version, and changed to point to pii-demo-green.apps.hl.test after the green version has been deployed.

References

https://istio.io/latest/docs/ops/deployment/requirements/

https://istio.io/latest/docs/reference/config/networking/virtual-service/

Leave a Reply

Your email address will not be published.