Upgrading Homelab Kubernetes Cluster from 1.22 to 1.23

Calico 3.22 has been released with support for Kubernetes 1.23.

The Upgrade Path

Our cluster was originally built using Ansible (kubeadm). We will use kubeadm upgrade to upgrade it.

We will be upgrading from:

  1. kubeadm 1.22.4
  2. kubelet 1.22.4
  3. kubectl 1.22.4
  4. coredns 1.8.4
  5. etcd 3.5.0
  6. calico 3.21
  7. docker-ce 20.10.11
  8. Istio 1.11

to:

  1. kubeadm 1.23.5
  2. kubelet 1.23.5
  3. kubectl 1.23.5
  4. coredns 1.8.6
  5. etcd 3.5.1
  6. calico 3.22
  7. docker-ce 20.10.13
  8. Istio 1.13

Backup the Cluster

Kubernetes nodes run on KVM, therefore we have taken KVM snapshosts of each virtual machine before starting the upgrade.

Upgrade Control Plane Nodes

Cluster node status before proceeding:

$ kubectl get no -o wide
NAME    STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION                 CONTAINER-RUNTIME
srv31   Ready    control-plane,master   3d    v1.22.4   10.11.1.31    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.11
srv32   Ready    control-plane,master   3d    v1.22.4   10.11.1.32    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.11
srv33   Ready    control-plane,master   3d    v1.22.4   10.11.1.33    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.11
srv34   Ready    none                   3d    v1.22.4   10.11.1.34    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.11
srv35   Ready    none                   3d    v1.22.4   10.11.1.35    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.11
srv36   Ready    none                   3d    v1.22.4   10.11.1.36    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.11

Perform kubeadm upgrade

The upgrade procedure on control plane nodes should be executed one node at a time.

We will start with the control plane srv31. For the first control plane node srv31:

$ sudo yum install -y kubeadm-1.23.5-0 --disableexcludes=kubernetes
$ kubeadm version

Verify the upgrade plan:

$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.22.4
[upgrade/versions] kubeadm version: v1.23.5
[upgrade/versions] Target version: v1.23.5
[upgrade/versions] Latest version in the v1.22 series: v1.22.8

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     6 x v1.22.4   v1.22.8

Upgrade to the latest version in the v1.22 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.22.4   v1.22.8
kube-controller-manager   v1.22.4   v1.22.8
kube-scheduler            v1.22.4   v1.22.8
kube-proxy                v1.22.4   v1.22.8
CoreDNS                   v1.8.4    v1.8.6
etcd                      3.5.0-0   3.5.1-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.22.8

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     6 x v1.22.4   v1.23.5

Upgrade to the latest stable version:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.22.4   v1.23.5
kube-controller-manager   v1.22.4   v1.23.5
kube-scheduler            v1.22.4   v1.23.5
kube-proxy                v1.22.4   v1.23.5
CoreDNS                   v1.8.4    v1.8.6
etcd                      3.5.0-0   3.5.1-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply v1.23.5

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

Upgrade the cluster:

$ sudo kubeadm upgrade apply v1.23.5
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.23.5"
[upgrade/versions] Cluster version: v1.22.4
[upgrade/versions] kubeadm version: v1.23.5
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.23.5"...
Static pod: kube-apiserver-srv31 hash: ae455d05fb8089c3953fa3122ae8e2da
Static pod: kube-controller-manager-srv31 hash: d5b77f6fef7e23f93183e2f233065032
Static pod: kube-scheduler-srv31 hash: 0a3b5517126615967ae25294fe6cd714
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-srv31 hash: a8288882fa95e16f2c58bc7f181cefdb
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-03-17-23-24-32/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-srv31 hash: a8288882fa95e16f2c58bc7f181cefdb
Static pod: etcd-srv31 hash: 2bcd3e36b6f264a44730bca7ad5772d9
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3039953140"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-03-17-23-24-32/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-srv31 hash: ae455d05fb8089c3953fa3122ae8e2da
Static pod: kube-apiserver-srv31 hash: b80e9d3ff3d198ff4ef92906198ae945
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-03-17-23-24-32/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-srv31 hash: d5b77f6fef7e23f93183e2f233065032
Static pod: kube-controller-manager-srv31 hash: 199a72542c3eb17ea5d8d951f1d66bb7
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2022-03-17-23-24-32/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-srv31 hash: 0a3b5517126615967ae25294fe6cd714
Static pod: kube-scheduler-srv31 hash: 2715b9c95eebccc681f0ef1f6ff5bdc2
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.23.5". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

We are going to upgrade to Calico 3.22 which been tested against Kubernetes version 1.23.

$ kubectl apply -f https://docs.projectcalico.org/archive/v3.22/manifests/calico.yaml
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers configured
serviceaccount/calico-kube-controllers unchanged
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers configured

For the other control plane nodes:

$ sudo yum install -y kubeadm-1.23.5-0 --disableexcludes=kubernetes
$ kubeadm version
$ sudo kubeadm config images pull
$ sudo kubeadm upgrade node

According to Kubernetes documentation, calling kubeadm upgrade plan and upgrading the CNI provider plugin is no longer needed.

Drain the Nodes and Upgrade kubelet and kubectl

$ export CONTROL_PLANE="srv31"
$ kubectl drain ${CONTROL_PLANE} --ignore-daemonsets --delete-emptydir-data
$ sudo yum install -y kubelet-1.23.5-0 kubectl-1.23.5-0 --disableexcludes=kubernetes
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet
$ kubectl uncordon ${CONTROL_PLANE}

Repeat the process for control planes srv32 and srv33.

Upgrade Worker Nodes

We will start with the worker node srv34.

Upgrade kubeadm:

$ sudo yum install -y kubeadm-1.23.5-0 --disableexcludes=kubernetes
$ sudo kubeadm upgrade node

Drain the worker node:

$ export WORKER_NODE="srv34"
$ kubectl drain ${WORKER_NODE} --ignore-daemonsets --delete-emptydir-data

Upgrade kubelet and kubectl:

$ sudo yum install -y kubelet-1.23.5-0 kubectl-1.23.5-0 --disableexcludes=kubernetes
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

Uncordon the worker node:

$ kubectl uncordon ${WORKER_NODE}

Repeat the process for worker nodes srv35 and srv36.

Verify Cluster Status

Check cluster node status:

$ kubectl get no
NAME    STATUS   ROLES                  AGE   VERSION
srv31   Ready    control-plane,master   3d    v1.23.5
srv32   Ready    control-plane,master   3d    v1.23.5
srv33   Ready    control-plane,master   3d    v1.23.5
srv34   Ready    none                   3d    v1.23.5
srv35   Ready    none                   3d    v1.23.5
srv36   Ready    none                   3d    v1.23.5

Check Calico pods:

$ kubectl -n kube-system get po -l k8s-app=calico-node
NAME                READY   STATUS    RESTARTS   AGE
calico-node-55vhg   1/1     Running   0          9m9s
calico-node-f4ln7   1/1     Running   0          11m
calico-node-kc6w2   1/1     Running   0          10m
calico-node-lk65l   1/1     Running   0          12m
calico-node-phtff   1/1     Running   0          13m
calico-node-v8zzb   1/1     Running   0          11m

Update Docker

Update docker packages on control planes and worker nodes:

$ sudo yum install -y docker-ce-20.10.13 docker-ce-cli-20.10.13

Check docker version:

$ kubectl get no -o wide
NAME    STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION                 CONTAINER-RUNTIME
srv31   Ready    control-plane,master   3d    v1.23.5   10.11.1.31    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.13
srv32   Ready    control-plane,master   3d    v1.23.5   10.11.1.32    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.13
srv33   Ready    control-plane,master   3d    v1.23.5   10.11.1.33    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.13
srv34   Ready    none                   3d    v1.23.5   10.11.1.34    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.13
srv35   Ready    none                   3d    v1.23.5   10.11.1.35    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.13
srv36   Ready    none                   3d    v1.23.5   10.11.1.36    none          Rocky Linux 8.5 (Green Obsidian)   4.18.0-348.20.1.el8_5.x86_64   docker://20.10.13

Update Istio

Istio Canary updates are just bad as they don’t upgrade sidecars. In production we build a new Kubernetes cluster using red/black deployment and install a new version of Istio. For the sake of the homelab environment, we will do an in-place upgrade.

Download istioctl binary:

$ wget https://github.com/istio/istio/releases/download/1.13.2/istioctl-1.13.2-linux-amd64.tar.gz
$ tar xf istioctl-1.13.2-linux-amd64.tar.gz 
$ sudo mv istioctl /usr/local/bin/
$ sudo chown root: /usr/local/bin/istioctl

Ensure that the upgrade is compatible with our environment:

$ istioctl x precheck
✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
$ istioctl version
client version: 1.13.2
control plane version: 1.11.4
data plane version: 1.11.4 (21 proxies)

Generate a YAML manifest for Kubernetes:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/istio
$ istioctl manifest generate -f ./istio-operator.yml --set values.global.jwtPolicy=first-party-jwt > ./istio-kubernetes.yml

Upgrade Istio. The kubectl apply command may show transient errors due to resources not being available in the cluster in the correct order. If that happens, simply run the command again.

$ kubectl apply -f ./istio-kubernetes.yml

Verify:

$ kubectl get po -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-77b8764f44-8qvv7   1/1     Running   0          62s
istio-ingressgateway-77b8764f44-krkzt   1/1     Running   0          62s
istiod-f9d549d48-fwq94                  1/1     Running   0          62s

We should see the updated version on the control plane but a bunch of old proxies (sidecars) on the data plane:

$ istioctl version
client version: 1.13.2
control plane version: 1.13.2
data plane version: 1.11.4 (19 proxies), 1.13.2 (2 proxies)

Restart all pods that have Istio sidecards running to allow them to pick up a new version of Istio. When done, we should have no old versions of proxies running:

$ istioctl version
client version: 1.13.2
control plane version: 1.13.2
data plane version: 1.13.2 (21 proxies)

References

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

https://istio.io/latest/docs/setup/upgrade/in-place/

Leave a Reply

Your email address will not be published.