Install Kubernetes Cluster with Ansible

We are going to install a Kubernetes control plane with two worker nodes using Ansible.

Note that installation of Ansible control node is not covered in this article.

Pre-requisites

This guide is based on Debian Stretch. You can use Ubuntu as well.

  1. Ansible control node with Ansible 2.9 to run playbooks.
  2. 3x Debian Stretch servers with the ansible user created for SSH.
  3. Each server should have 2x CPUs and 2GB of RAM.
  4. /etc/hosts file configured to resolve hostnames.

We are going to use the following ansible.cfg configuration:

[defaults]
inventory  = ./inventory

remote_user = ansible
host_key_checking = False
private_key_file = ./files/id_rsa

[privilege_escalation]
become=False
become_method=sudo
become_user=root
become_ask_pass=False

The content of the inventory file can be seen below:

[k8s-master]
10.11.1.101

[k8s-node]
10.11.1.102
10.11.1.103

[k8s:children]
k8s-master
k8s-node

The Goal

To deploy a specific version of a 3-node Kubernetes cluster (one master and two worker nodes) with Calico networking and Kubernetes Dashboard.

Package Installation

Docker

Configure packet forwarding and install a specific version of Docker so that we can test upgrades later.

---
- name: Install Docker Server
  hosts: k8s
  become: true
  gather_facts: yes
  vars:
    docker_dependencies:
    - ca-certificates
    - gnupg
    - gnupg-agent
    - software-properties-common
    - apt-transport-https
    docker_packages:
    - docker-ce=5:18.09.6~3-0~debian-stretch
    - docker-ce-cli=5:18.09.6~3-0~debian-stretch
    - containerd.io 
    - curl
    docker_url_apt_key: "https://download.docker.com/linux/debian/gpg"
    docker_repository: "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable"
  tasks:
  - name: Debian | Configure Sysctl
    sysctl:
      name: "net.ipv4.ip_forward"
      value: "1"
      state: present

  - name: Debian | Install Prerequisites Packages
    package: name={{ item }} state=present force=yes
    loop: "{{ docker_dependencies }}"

  - name: Debian | Add GPG Keys
    apt_key: 
      url: "{{ docker_url_apt_key }}"

  - name: Debian | Add Repo Source
    apt_repository: 
      repo: "{{ docker_repository }}"
      update_cache: yes

  - name: Debian | Install Specific Version of Docker Packages
    package: name={{ item }} state=present force=yes install_recommends=no
    loop: "{{ docker_packages }}"
    notify:
    - start docker

  - name: Debian | Start and Enable Docker Service
    service:
      name: docker
      state: started
      enabled: yes  

  handlers:
  - name: start docker
    service: name=docker state=started
...

Kubeadm and Kubelet

Install a specific version of kubeadm so that we can test upgrades later.

---
- name: Install Kubernetes Server
  hosts: k8s
  become: true
  gather_facts: yes
  vars:
    k8s_dependencies: 
    - kubernetes-cni=0.6.0-00 
    - kubelet=1.13.1-00
    k8s_packages: 
    - kubeadm=1.13.1-00 
    - kubectl=1.13.1-00 
    k8s_url_apt_key: "https://packages.cloud.google.com/apt/doc/apt-key.gpg"
    k8s_repository: "deb https://apt.kubernetes.io/ kubernetes-xenial main"
  tasks:
  - name: Disable SWAP K8S will not work with swap enabled (1/2)
    command: swapoff -a
    when: ansible_swaptotal_mb > 0

  - name: Debian | Remove SWAP from fstab K8S will not work with swap enabled (2/2)
    mount:
      name: "{{ item }}"
      fstype: swap
      state: absent
    with_items:
    - swap
    - none

  - name: Debian | Add GPG Key
    apt_key:
      url: "{{ k8s_url_apt_key }}"

  - name: Debian | Add Kubernetes Repository
    apt_repository: 
      repo: "{{ k8s_repository }}"
      update_cache: yes

  - name: Debian | Install Dependencies
    package: name={{ item }} state=present force=yes install_recommends=no
    loop: "{{ k8s_dependencies }}"

  - name: Debian | Install Kubernetes Packages
    package: name={{ item }} state=present force=yes install_recommends=no
    loop: "{{ k8s_packages }}"
...

Kubernetes Configuration

Create a Kubernetes control plane and join two other servers as worker nodes. Note the content of the file files/dashboard-adminuser.yaml below:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system	
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

Deploy Kubernetes cluster.

---
- name: Deploy Kubernetes Cluster
  hosts: k8s
  gather_facts: yes
  vars:
    k8s_pod_network: "192.168.192.0/18"
    k8s_user: "ansible"
    k8s_user_home: "/home/{{ k8s_user }}"
    k8s_token_file: "join-k8s-command"
    k8s_admin_config: "/etc/kubernetes/admin.conf"
    k8s_dashboard_adminuser_config: "dashboard-adminuser.yaml"
    k8s_kubelet_config: "/etc/kubernetes/kubelet.conf"
    k8s_dashboard_port: "6443"
    k8s_dashboard_url: "https://{{ ansible_default_ipv4.address }}:{{ k8s_dashboard_port }}/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login"
    calico_rbac_url: "https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml"
    calico_rbac_config: "rbac-kdd.yaml"
    calico_net_url: "https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml"
    calico_net_config: "calico.yaml"
    dashboard_url: "https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml"
    dashboard_config: "kubernetes-dashboard.yml"
  tasks:
  - name: Debian | Configure K8S Master Block
    block:
    - name: Debian | Initialise the Kubernetes cluster using kubeadm
      become: true
      command: kubeadm init --pod-network-cidr={{ k8s_pod_network }}
      args:
        creates: "{{ k8s_admin_config }}"

    - name: Debian | Setup kubeconfig for {{ k8s_user }} user
      file:
        path: "{{ k8s_user_home }}/.kube"
        state: directory
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0750"
    
    - name: Debian | Copy {{ k8s_admin_config }}
      become: true
      copy:
        src: "{{ k8s_admin_config }}"
        dest: "{{ k8s_user_home }}/.kube/config"
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0640"
        remote_src: yes
    
    - name: Debian | Download {{ calico_rbac_url }}
      get_url:
        url: "{{ calico_rbac_url }}"
        dest: "{{ k8s_user_home }}/{{ calico_rbac_config }}"
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0640"
    
    - name: Debian | Download {{ calico_net_url }}
      get_url:
        url: "{{ calico_net_url }}"
        dest: "{{ k8s_user_home }}/{{ calico_net_config }}"
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0640"     

    - name: Debian | Set CALICO_IPV4POOL_CIDR to {{ k8s_pod_network }}
      replace:
        path: "{{ k8s_user_home }}/{{ calico_net_config }}"
        regexp: "192.168.0.0/16"
        replace: "{{ k8s_pod_network }}"
    
    - name: Debian | Download {{ dashboard_url }}
      get_url:
        url: "{{ dashboard_url }}"
        dest: "{{ k8s_user_home }}/{{ dashboard_config }}"
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0640"     
    
    - name: Debian | Install calico pod network {{ calico_rbac_config }}
      become: false
      command: kubectl apply -f "{{ k8s_user_home }}/{{ calico_rbac_config }}"
    
    - name: Debian | Install calico pod network {{ calico_net_config }}
      become: false
      command: kubectl apply -f "{{ k8s_user_home }}/{{ calico_net_config }}"
    
    - name: Debian | Install K8S dashboard {{ dashboard_config }}
      become: false
      command: kubectl apply -f "{{ k8s_user_home }}/{{ dashboard_config }}"
    
    - name: Debian | Create service account
      become: false
      command: kubectl create serviceaccount dashboard -n default
      ignore_errors: yes
    
    - name: Debian | Create cluster role binding dashboard-admin
      become: false
      command: kubectl create clusterrolebinding dashboard-admin -n default --clusterrole=cluster-admin --serviceaccount=default:dashboard
      ignore_errors: yes
    
    - name: Debian | Create {{ k8s_dashboard_adminuser_config }} for service account
      copy:
        src: "files/{{ k8s_dashboard_adminuser_config }}"
        dest: "{{ k8s_user_home }}/{{ k8s_dashboard_adminuser_config }}"
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0640"
    
    - name: Debian | Create service account
      become: false
      command: kubectl apply -f "{{ k8s_user_home }}/{{ k8s_dashboard_adminuser_config }}"
      ignore_errors: yes
    
    - name: Debian | Create cluster role binding cluster-system-anonymous
      become: false
      command: kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
      ignore_errors: yes
    
    - name: Debian | Test K8S dashboard and wait for HTTP 200
      uri:
        url: "{{ k8s_dashboard_url }}"
        status_code: 200
        validate_certs: no
      ignore_errors: yes
      register: result_k8s_dashboard_page
      retries: 10
      delay: 6
      until: result_k8s_dashboard_page is succeeded

    - name: Debian | K8S dashboard URL
      debug:
        var: k8s_dashboard_url
    
    - name: Debian | Generate join command
      command: kubeadm token create --print-join-command
      register: join_command
    
    - name: Debian | Copy join command to local file
      become: false
      local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="{{ k8s_token_file }}"
    when: "'k8s-master' in group_names"

  - name: Debian | Configure K8S Node Block
    block:
    - name: Debian | Copy {{ k8s_token_file }} to server location
      copy: 
        src: "{{ k8s_token_file }}"
        dest: "{{ k8s_user_home }}/{{ k8s_token_file }}.sh"
        owner: "{{ k8s_user }}"
        group: "{{ k8s_user }}"
        mode: "0750"

    - name: Debian | Join the node to cluster unless file {{ k8s_kubelet_config }} exists
      become: true
      command: sh "{{ k8s_user_home }}/{{ k8s_token_file }}.sh"
      args:
        creates: "{{ k8s_kubelet_config }}"
    when: "'k8s-node' in group_names"
...

Verify Cluster Status

Cluster nodes:

$ kubectl get nodes -o wide
NAME          STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
debian9-101   Ready    master   5m30s   v1.13.1   10.11.1.101           Debian GNU/Linux 9 (stretch)   4.9.0-13-amd64   docker://18.9.6
debian9-102   Ready    < none>  2m30s   v1.13.1   10.11.1.102           Debian GNU/Linux 9 (stretch)   4.9.0-13-amd64   docker://18.9.6
debian9-103   Ready    < none>  2m29s   v1.13.1   10.11.1.103           Debian GNU/Linux 9 (stretch)   4.9.0-13-amd64   docker://18.9.6

Cluster pods:

$ kubectl get pods -n kube-system -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE          NOMINATED NODE   READINESS GATES
calico-node-fghpt                      2/2     Running   0          3m29s   10.11.1.101     debian9-101              
calico-node-stf8j                      2/2     Running   0          77s     10.11.1.103     debian9-103              
calico-node-tqx8r                      2/2     Running   0          78s     10.11.1.102     debian9-102              
coredns-54ff9cd656-52x2n               1/1     Running   0          3m46s   192.168.192.3   debian9-101              
coredns-54ff9cd656-vzg6b               1/1     Running   0          3m47s   192.168.192.2   debian9-101              
etcd-debian9-101                       1/1     Running   0          4m3s    10.11.1.101     debian9-101              
kube-apiserver-debian9-101             1/1     Running   0          3m55s   10.11.1.101     debian9-101              
kube-controller-manager-debian9-101    1/1     Running   0          3m53s   10.11.1.101     debian9-101              
kube-proxy-gc7r2                       1/1     Running   0          78s     10.11.1.103     debian9-103              
kube-proxy-tfvmj                       1/1     Running   0          79s     10.11.1.102     debian9-102              
kube-proxy-xf8pv                       1/1     Running   0          3m48s   10.11.1.101     debian9-101              
kube-scheduler-debian9-101             1/1     Running   0          3m44s   10.11.1.101     debian9-101              
kubernetes-dashboard-57df4db6b-gmmcx   1/1     Running   0          3m25s   192.168.192.4   debian9-101              

5 thoughts on “Install Kubernetes Cluster with Ansible

Leave a Reply

Your email address will not be published. Required fields are marked *