We are going to deploy two versions of the application, blue and green, and configure Istio routing so that both versions could be accessed at the same time.
Pre-requisites
We are using our Kubernetes homelab in this article. We will continue with the pii-demo application for blue/green that we had deployed some time ago with Istio mTLS.
Basic familiarity with Istio is required.
Blue/green deployment is an application release model that transfers user traffic from a previous version of a microservice to a new release, both of which are running in production, without downtime.
Source code is available on GitHub.
The Goal
To allow two versions of the pii-demo application to run simultaneously using blue/green deployment, and configure Istio routing rules to allow access to both versions based on a host header. Round robin routing should apply if no colour was specified by a user.
The following DNS names will be used:
- pii-demo.apps.hl.test – DNS of the application before blue/green, and round robin routing after. This can easily be managed at the DNS level to point to either blue or green using a CNAME, if required.
- pii-demo-blue.apps.hl.test – DNS of the blue version of the application.
- pii-demo-green.apps.hl.test – DNS of the green version of the application.
We should also configure Istio routing rules for the pod-to-pod communication within the mesh so that blue pods talk to other blue pods, and green pods talk to other green pods. For example, a blue version of the webserver should not be routed to the green version of the database to store data, and a blue webserver should not be routed to a green webserver if there was a need for such a pod-to-pod communication.
The PII Demo Application (Simple Database App)
The pii-demo application is a simple Apache/PHP/MySQL application that allows users to enter personally identifiable information (pii) and store it in a database. The application uses two containers: httpd and MySQL.
The block diagram showing Istio flow when an application request is made can be seen below. Yaml code of the application deployment can be found on GitHub.
We use a Gateway to describe a load balancer, operating at the edge of the mesh, that receives incoming HTTP connections. The specification describes a set of ports that should be exposed and the type of protocol to use, in our case 80 and HTTP, respectively (see below). A VirtualService can then be bound to a gateway to control the forwarding of traffic arriving at a particular host.
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: pii-demo-gateway
namespace: pii-demo
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
We use a VirtualService to define a set of traffic routing rules to apply when a host is addressed, in our case pii-demo.apps.hl.test. A routing rule defines matching criteria for traffic of a specific protocol (TCP, TLS or http), in our case http. If the traffic is matched, then it is sent to a named destination service (or version of it) defined in the registry, in our case httpd-server. See below.
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: httpd-server
namespace: pii-demo
spec:
hosts:
- "pii-demo.apps.hl.test"
gateways:
- pii-demo-gateway
http:
- route:
- destination:
host: httpd-server
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: httpd-server
namespace: pii-demo
labels:
app: httpd-server
spec:
selector:
app: httpd-server
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 30080
If we were to release a new version of the application, we would have to update the deployment yaml with a new docker image version and apply it.
This is how the frontend looks like:
The PII Demo Application: Blue/Green Deployment
We are going to change the Istio configuration to allow us to do blue/green deployment. The block diagram showing Istio flow when a blue application request is made can be seen below. The same diagram would apply if a green application request was made, it would just go to a different pod.
Add a Version Label to Kubernetes Deployment Configuration
The first thing that we want to do is to separate deployments by adding a version label. Istio recommends adding an explicit app label and version label to the specification of the pods deployed using a Kubernetes Deployment. In our case the versions won’t be standard application versions such as 1.0.0, but blue/green instead. We will have two versions of the application, a blue version, and a green version. See below.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpd-server-blue
namespace: pii-demo
labels:
app: httpd-server
version: blue
spec:
replicas: 1
selector:
matchLabels:
app: httpd-server
version: blue
template:
metadata:
name: httpd-server
labels:
app: httpd-server
version: blue
spec:
containers:
- name: httpd-server
image: lisenet/httpd-pii-demo:0.2
[... configuration omitted ...]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpd-server-green
namespace: pii-demo
labels:
app: httpd-server
version: green
spec:
replicas: 1
selector:
matchLabels:
app: httpd-server
version: green
template:
metadata:
name: httpd-server
labels:
app: httpd-server
version: green
spec:
containers:
- name: httpd-server
image: lisenet/httpd-pii-demo:0.3
[... configuration omitted ...]
Although not displayed here, the same versioning logic applies to the MySQL deployment configuration as well.
Use Named Service Ports
Istio requires named service ports. See below.
---
apiVersion: v1
kind: Service
metadata:
name: httpd-server
namespace: pii-demo
labels:
app: httpd-server
spec:
selector:
app: httpd-server
type: NodePort
ports:
- name: http-80 # istio requires named service ports
port: 80
protocol: TCP
targetPort: 80
nodePort: 30080
---
apiVersion: v1
kind: Service
metadata:
name: mysql-server
namespace: pii-demo
labels:
app: mysql-server
spec:
selector:
app: mysql-server
ports:
- name: tcp-3306 # do not use "mysql" as a name, see https://github.com/istio/istio/issues/13484
port: 3306
protocol: TCP
targetPort: 3306
Configure Istio Routing Rules for httpd
HttpMatchRequest allows us to specify a set of criterion to be met in order for the rule to be applied to the HTTP request. Then, each routing rule is further associated with one service version (blue or green) by using a DestinationRule. DestinationRule defines policies that apply to traffic intended for a service after routing has occurred.
Requests to the httpd webserver can either come from outside the Istio mesh via the Gateway, or from within the mesh when one pod communicates with another pod.
- When a request comes in to the Gateway, we want to match it based on HTTP headers.
- When a request is made from within the service mesh, we want to match it based on a sourceLabels.
When using sourceLabels, the top-level gateways field in the VirtualService definition must include the reserved gateway mesh for this field to be applicable.
A single VirtualService is used for sidecars inside the mesh as well as for one or more gateways. In order to apply the rules to both gateways and sidecars, we have to specify mesh as one of the gateway names.
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: httpd-server
namespace: pii-demo
spec:
# the destination hosts to which traffic is being sent to
hosts:
- pii-demo.apps.hl.test # cannot use "*" here since this is being combined with the mesh services
- pii-demo-blue.apps.hl.test
- pii-demo-green.apps.hl.test
- httpd-server
gateways:
- mesh # applies internally as well as externally
- pii-demo-gateway
http:
- match:
- headers:
host:
exact: 'pii-demo-blue.apps.hl.test'
gateways:
- pii-demo-gateway # restricts this rule to apply only to ingress gateway
- sourceLabels:
version: blue # a blue httpd pod should only talk to another blue httpd pod
route:
- destination:
host: httpd-server
port:
number: 80
subset: blue
- match:
- headers:
host:
exact: 'pii-demo-green.apps.hl.test'
gateways:
- pii-demo-gateway
- sourceLabels:
version: green # a green httpd pod should only talk to another green httpd pod
route:
- destination:
host: httpd-server
port:
number: 80
subset: green
- match:
- headers:
host:
exact: 'pii-demo.apps.hl.test'
gateways:
- pii-demo-gateway
route:
- destination:
host: httpd-server # no subset, round robin service routing applies
port:
number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpd-server
namespace: pii-demo
spec:
host: httpd-server
subsets:
- name: blue
labels:
version: blue
- name: green
labels:
version: green
Configure Istio Routing Rules for MySQL
MySQL protocol uses TCP and not HTTP.
TCPRoute describes match conditions and actions for routing TCP traffic, and L4MatchAttributes describes L4 connection match attributes.
Requests to MySQL should never come from outside the service mesh, therefore the Gateway pii-demo-gateway has not been specified. See configuration below.
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: mysql-server
namespace: pii-demo
spec:
# the destination host to which traffic is being sent to
hosts:
- mysql-server
gateways:
- mesh # applies internally as well as externally
tcp:
- match:
- sourceLabels:
version: blue # a blue mysql pod should only talk to another blue mysql pod
route:
- destination:
host: mysql-server
port:
number: 3306
subset: blue
- match:
- sourceLabels:
version: green # a green mysql pod should only talk to another green mysql pod
route:
- destination:
host: mysql-server
port:
number: 3306
subset: green
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: mysql-server
namespace: pii-demo
spec:
host: mysql-server
subsets:
- name: blue
labels:
version: blue
- name: green
labels:
version: green
Test the Blue/Green Deployment
Now that we have everything configured, we should test the application. Kiali provides an easy way to get Istio traffic data visualised in real time.
When a user makes a request to pii-demo-blue.apps.hl.test, only blue versions of pods are used.
When a request is made to pii-demo-green.apps.hl.test, only green versions of pods are used.
See the Istio workload graph below. The padlock indicates that connections are mTLS encrypted.
See the versioned application graph below.
If a user were to make a request to pii-demo.apps.hl.test, it would be routed to either blue or green version using round robin because of our configuration. As mentioned earlier, this behaviour can be changed by managing routing at the DNS level, e.g. pii-demo.apps.hl.test could be a CNAME pointing to pii-demo-blue.apps.hl.test before the deployment of the green version, and changed to point to pii-demo-green.apps.hl.test after the green version has been deployed.
References
https://istio.io/latest/docs/ops/deployment/requirements/
https://istio.io/latest/docs/reference/config/networking/virtual-service/