Backup Kubernetes Cluster Resources to AWS S3 with Velero

Posted on by

We will install Velero to backup and restore Kubernetes cluster resources to AWS S3.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configuration files used in this article can be found on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/kubernetes/helm/velero/

Access to AWS account is required. Free Tier provides 5GB of S3 storage for 12 months.

The Plan

  1. Install Helm.
  2. Configure AWS S3 bucket and IAM credentials.
  3. Install Velero to the Kubernetes cluster using Helm.
  4. Install Velero client.
  5. Create a backup and a backup schedule.
  6. Test the backup by restoring Kubernetes resources from it.

Install Helm

On a Debian-based OS, do the following:

$ curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
$ sudo apt-get install -y apt-transport-https
$ echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install -y helm

Add Helm repository:

$ helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

Configure AWS S3 Bucket and IAM Credentials

Create an S3 bucket. We use Terraform to do that. Also, create an IAM user for Velero that has the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}

Save IAM user’s access keys in a file velero-credentials.txt, e.g.:

[default]
aws_access_key_id = AWS_ACCESS_KEY_ID
aws_secret_access_key = AWS_SECRET_ACCESS_KEY

Deploy Velero

Create velero namespace inside the Kubernetes cluster:

$ kubectl create namespace velero

Install Velero using Helm. See the content of the values.yaml file on GitHub.

$ helm upgrade --install velero \
  vmware-tanzu/velero \
  --namespace velero \
  --version "2.28.1" \
  --set-file credentials.secretContents.cloud=velero-credentials.txt \
  --values ./values.yaml

Install Velero Client

When Velero server is up and running, we are going to need to install the client before we can use it:

$ export VELERO_VERSION="1.8.1"
$ wget https://github.com/vmware-tanzu/velero/releases/download/v${VELERO_VERSION}/velero-v${VELERO_VERSION}-linux-amd64.tar.gz
$ tar xf velero-v${VELERO_VERSION}-linux-amd64.tar.gz 
$ sudo mv velero-v${VELERO_VERSION}-linux-amd64/velero /usr/local/bin/
$ sudo chown root:root /usr/local/bin/velero

Create Backups and Backup Schedules

First of all, verify our backup location:

$ velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX                       PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        kubernetes-homelab-velero-backups   Available   2022-05-20 01:28:00 +0100 BST   ReadWrite     true

List all namespaces in the Kubernetes cluster:

$ kubectl get namespace -o name
namespace/argocd
namespace/default
namespace/democratic-csi
namespace/httpd-healthcheck
namespace/istio-system
namespace/kube-node-lease
namespace/kube-public
namespace/kube-system
namespace/kubecost
namespace/kubernetes-dashboard
namespace/logging
namespace/metallb-system
namespace/monitoring
namespace/openvpn
namespace/pii-demo
namespace/speedtest
namespace/velero

We want to backup all of them but the velero one. Create a Velero backup job for each namespace:

$ for i in $(kubectl get ns -o name|cut -d"/" -f2|grep -ve velero);do \
  velero backup create "${i}" --include-namespaces "${i}"; \
done

Create a backup schedule for each namespace to run daily at 2AM:

$ for i in $(kubectl get ns -o name|cut -d"/" -f2|grep -ve velero);do \
  velero create schedule "${i}" --schedule="0 2 * * *" --include-namespaces "${i}"; \
done

Verify schedules by running the following command:

$ velero schedule get
NAME                   STATUS    CREATED                         SCHEDULE    BACKUP TTL   LAST BACKUP   SELECTOR
default                Enabled   2022-03-20 23:24:26 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
democratic-csi         Enabled   2022-03-20 23:24:26 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
httpd-healthcheck      Enabled   2022-03-20 23:24:27 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
istio-system           Enabled   2022-03-20 23:24:27 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
kube-node-lease        Enabled   2022-03-20 23:24:27 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
kube-public            Enabled   2022-03-20 23:24:27 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
kube-system            Enabled   2022-03-20 23:24:28 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
kubecost               Enabled   2022-03-20 23:24:28 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
kubernetes-dashboard   Enabled   2022-03-20 23:24:28 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
logging                Enabled   2022-03-20 23:24:29 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
metallb-system         Enabled   2022-03-20 23:24:29 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
monitoring             Enabled   2022-03-20 23:24:29 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
openvpn                Enabled   2022-03-20 23:24:30 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
pii-demo               Enabled   2022-03-20 23:24:30 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none
speedtest              Enabled   2022-03-20 23:24:30 +0000 GMT   0 2 * * *   720h0m0s     22h ago       none

Test Velero Backup Restore

We are going to delete our Grafana deployment configuration from the monitoring namespace.

$ kubectl -n monitoring delete deploy grafana

$ kubectl -n monitoring get deploy grafana
Error from server (NotFound): deployments.apps "grafana" not found

Retrieve the name of the most recent backup for the monitoring namespace:

$ velero backup get | grep monitoring | head -n1
monitoring-20220519020037 Completed 0 0 2022-05-19 03:01:19 +0100 BST 29d default none

Submit a restore request to the server:

$ velero restore create monitoring --from-backup monitoring-20220519020037

Check the job status:

$ velero restore describe monitoring
Name:         monitoring
Namespace:    velero
Labels:       
Annotations:  

Phase:                       Completed
Total items to be restored:  153
Items restored:              153

Started:    2022-05-20 01:53:09 +0100 BST
Completed:  2022-05-20 01:53:31 +0100 BST

Grafana pod should now be running again:

$ kubectl -n monitoring get deploy grafana
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
grafana   1/1     1            1           112s

References

https://github.com/vmware-tanzu/velero-plugin-for-aws

Leave a Reply

Your email address will not be published. Required fields are marked *