OpenVPN Server on Kubernetes

Posted on by

OpenVPN server in a Docker container running on Kubernetes.

Pre-requisites

We are using our Kubernetes homelab in this article.

Configuration files used in this article can be found on GitHub. Clone the following repositories:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ git clone https://github.com/lisenet/docker-openvpn.git

The Goal

We want to be able to access Kubernetes homelab subnet 10.11.1.0/24 from the Internet using a VPN connection on an Android device. This would allow us to access internally hosted services like:

  1. https://grafana.apps.hl.test
  2. https://prometheus.apps.hl.test

We also want to route all traffic through the VPN server (push default gateway).

The Plan

  1. Build a container image for the latest version of OpenVPN.
  2. Generate OpenVPN configuration files and certificates.
  3. Deploy OpenVPN on Kubernetes.
  4. Configure a destination NAT rule on Mikrotik router.
  5. Test VPN access from an Android client.

Build OpenVPN Docker Image (Optional)

This step is optional.

TL;DR: use lisenet/openvpn:latest docker image.

At the time of writing, OpenVPN image provided by kylemanna/openvpn:2.5 has not been updated since 2020, which makes it slightly out of date if you ask me.

We will build a new image and push it to Docker Hub.

$ git clone https://github.com/lisenet/docker-openvpn.git
$ cd ./docker-openvpn
$ docker build -t openvpn:latest .
$ docker tag openvpn:latest lisenet/openvpn:latest
$ docker push lisenet/openvpn:latest

Generate OpenVPN Configuration Files and Certificates

Pull scripts from GitHub:

$ git clone https://github.com/lisenet/docker-openvpn.git
$ cd ./docker-openvpn/

Create a Kubernetes namespace:

$ kubectl create namespace openvpn

Set environment variables to be used to generate OpenVPN config:

$ export VPN_HOSTNAME="vpn.example.com"
$ export DNS_SERVER="10.11.1.2"

Generate basic OpenVPN config:

$ ./bin/generate-config.sh

Change ownership of ovpn0 folder so that we can write to it:

$ sudo chown -R "${USER}:${USER}" ./ovpn0/

Generate a client config (can be repeated for any new client):

$ export CLIENT_NAME=android
$ ./bin/add-client.sh

Set the Kubernetes secrets. Prepend with REPLACE=true to update the existing ones:

$ ./bin/set-secrets.sh

Note: VPN config, certificates and keys are stored in the ovpn0 directory on the machine that was used to run the commands.

$ kubectl get cm -n openvpn
NAME                 DATA   AGE
ccd0                 0      1h
istio-ca-root-cert   1      1h
kube-root-ca.crt     1      1h
ovpn0-conf           2      1h

$ kubectl get secrets -n openvpn
NAME                  TYPE                                  DATA   AGE
default-token-wd9q8   kubernetes.io/service-account-token   3      1h
ovpn0-cert            Opaque                                1      1h
ovpn0-key             Opaque                                1      1h
ovpn0-pki             Opaque                                3      1h

Deploy OpenVPN on Kubernetes

Content of the file openvpn-deployment.yaml can be seen below.

Note a couple of things:

  1. We define initContainers to configure IP forwarding on the pod.
  2. We use a service type of LoadBalancer to receive an IP address from MetalLB.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for OpenVPN service pods only."
---
apiVersion: v1
kind: Service
metadata:
  name: openvpn
  namespace: openvpn
  labels:
    app: openvpn
spec:
  selector:
    app: openvpn
  ports:
  - name: openvpn
    port: 31194
    protocol: TCP
    targetPort: openvpn
  type: LoadBalancer
  loadBalancerIP: 10.11.1.53
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openvpn
  namespace: openvpn
  labels:
    app: openvpn
spec:
  replicas: 1
  selector:
    matchLabels:
      app: openvpn
  template:
    metadata:
      name: openvpn
      labels:
        app: openvpn
    spec:
      initContainers:
      - image: busybox:1.36
        imagePullPolicy: IfNotPresent
        name: init-busybox
        command:
        - sh
        - -c
        - sysctl -w net.ipv4.ip_forward=1 && sysctl -w net.ipv4.conf.all.forwarding=1
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
          privileged: true
      containers:
      - image: lisenet/openvpn:latest
        imagePullPolicy: Always
        name: openvpn
        ports:
        - containerPort: 1194
          name: openvpn
          protocol: TCP
        # The kubelet will send the first readiness probe 5 seconds after the container starts.
        # This will attempt to connect to the openvpn container on port 1194. If the probe succeeds,
        # the Pod will be marked as ready. The kubelet will continue to run this check every 30 seconds.
        readinessProbe:
          tcpSocket:
            port: 1194
          initialDelaySeconds: 5
          periodSeconds: 30
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        # In addition to the readiness probe, this configuration includes a liveness probe.
        # The kubelet will run the first liveness probe 15 seconds after the container starts.
        # Similar to the readiness probe, this will attempt to connect to the container on port 1194.
        # If the liveness probe fails, the container will be restarted.
        livenessProbe:
          tcpSocket:
            port: 1194
          initialDelaySeconds: 15
          periodSeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 2
        resources:
          limits:
            cpu: 1000m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 16Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
        volumeMounts:
        - mountPath: /etc/openvpn/pki/private
          name: ovpn0-key
        - mountPath: /etc/openvpn/pki/issued
          name: ovpn0-cert
        - mountPath: /etc/openvpn/pki
          name: ovpn0-pki
        - mountPath: /etc/openvpn
          name: ovpn0-conf
        - mountPath: /etc/openvpn/ccd
          name: ccd0
      priorityClassName: high-priority
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
      - name: ovpn0-key
        secret:
          defaultMode: 384
          secretName: ovpn0-key
      - name: ovpn0-cert
        secret:
          defaultMode: 420
          secretName: ovpn0-cert
      - name: ovpn0-pki
        secret:
          defaultMode: 384
          secretName: ovpn0-pki
      - configMap:
          defaultMode: 420
          name: ovpn0-conf
        name: ovpn0-conf
      - configMap:
          defaultMode: 420
          name: ccd0
        name: ccd0
...

Deploy OpenVPN:

$ kubectl apply -f openvpn-deployment.yaml

List pods and services to verify.

$ kubectl get pods -n openvpn
NAME                      READY   STATUS    RESTARTS   AGE
openvpn-8f548449f-cbqxm   1/1     Running   0          43m

$ kubectl get svc -n openvpn
NAME      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
openvpn   LoadBalancer   10.107.175.151   10.11.1.53    31194:31844/TCP   43m

Configure Mikrotik Router

We want to allow connections from the internet to the OpenVPN service IP 10.11.1.53. In this case, we have to configure a destination address translation rule on our Mikrotik router.

Create a destination NAT rule:

/ip firewall nat add chain=dstnat \
  action=dst-nat \
  in-interface=ether1_isp \
  dst-port=31194 \
  to-addresses=10.11.1.53 \
  to-ports=31194 \
  protocol=tcp \
  comment="Allow public to OpenVPN"

At this point, assuming that the DNS record for VPN_HOSTNAME points to the IP address of the router, we should be able to access the OpenVPN server on vpn.example.com:31194 from the Internet.

Test VPN Access from Android Device

Download OpenVPN Connect client from Google Play.

Import ./ovpn0/android.ovpn client configuration file and connect to the VPN server.

Check pod logs.

$ kubectl -n openvpn logs $(kubectl -n openvpn get pods -o name)
Checking IPv6 Forwarding
Sysctl error for disable_ipv6, please run docker with '--sysctl net.ipv6.conf.all.disable_ipv6=0'
Sysctl error for default forwarding, please run docker with '--sysctl net.ipv6.conf.default.forwarding=1'
Sysctl error for all forwarding, please run docker with '--sysctl net.ipv6.conf.all.forwarding=1'
Running 'openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd '
2022-02-16 01:36:44 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-256-CBC' to --data-ciphers or change --cipher 'AES-256-CBC' to --data-ciphers-fallback 'AES-256-CBC' to silence this warning.
2022-02-16 01:36:44 OpenVPN 2.5.4 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov 15 2021
2022-02-16 01:36:44 library versions: OpenSSL 1.1.1l  24 Aug 2021, LZO 2.10
2022-02-16 01:36:44 Diffie-Hellman initialized with 2048 bit key
2022-02-16 01:36:44 Outgoing Control Channel Authentication: Using 384 bit message hash 'SHA384' for HMAC authentication
2022-02-16 01:36:44 Incoming Control Channel Authentication: Using 384 bit message hash 'SHA384' for HMAC authentication
2022-02-16 01:36:44 TUN/TAP device tun0 opened
2022-02-16 01:36:44 /sbin/ip link set dev tun0 up mtu 1500
2022-02-16 01:36:44 /sbin/ip link set dev tun0 up
2022-02-16 01:36:44 /sbin/ip addr add dev tun0 10.8.0.1/24
2022-02-16 01:36:44 Could not determine IPv4/IPv6 protocol. Using AF_INET
2022-02-16 01:36:44 Socket Buffers: R=[87380->87380] S=[16384->16384]
2022-02-16 01:36:44 Listening for incoming TCP connection on [AF_INET][undef]:1194
2022-02-16 01:36:44 TCPv4_SERVER link local (bound): [AF_INET][undef]:1194
2022-02-16 01:36:44 TCPv4_SERVER link remote: [AF_UNSPEC]
2022-02-16 01:36:44 GID set to nogroup
2022-02-16 01:36:44 UID set to nobody
2022-02-16 01:36:44 MULTI: multi_init called, r=256 v=256
2022-02-16 01:36:44 IFCONFIG POOL IPv4: base=10.8.0.2 size=252
2022-02-16 01:36:44 MULTI: TCP INIT maxclients=1024 maxevents=1028
2022-02-16 01:36:44 Initialization Sequence Completed
2022-02-16 01:37:15 Outgoing Control Channel Authentication: Using 384 bit message hash 'SHA384' for HMAC authentication
2022-02-16 01:37:15 Incoming Control Channel Authentication: Using 384 bit message hash 'SHA384' for HMAC authentication
2022-02-16 01:37:15 TCP connection established with [AF_INET]10.11.1.35:15520
2022-02-16 01:37:15 10.11.1.35:15520 TLS: Initial packet from [AF_INET]10.11.1.35:15520, sid=27615aef 0df1177d
2022-02-16 01:37:15 10.11.1.35:15520 VERIFY OK: depth=1, CN=vpn.example.com
2022-02-16 01:37:15 10.11.1.35:15520 VERIFY OK: depth=0, CN=android
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_VER=3.git::d3f8b18b:Release
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_PLAT=android
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_NCP=2
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_TCPNL=1
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_PROTO=30
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_IPv6=0
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_AUTO_SESS=1
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_GUI_VER=net.openvpn.connect.android_3.2.6-7729
2022-02-16 01:37:15 10.11.1.35:15520 peer info: IV_SSO=webauth,openurl
2022-02-16 01:37:15 10.11.1.35:15520 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1588', remote='link-mtu 1587'
2022-02-16 01:37:15 10.11.1.35:15520 WARNING: 'comp-lzo' is present in local config but missing in remote config, local='comp-lzo'
2022-02-16 01:37:15 10.11.1.35:15520 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
2022-02-16 01:37:15 10.11.1.35:15520 [android] Peer Connection Initiated with [AF_INET]10.11.1.35:15520
2022-02-16 01:37:15 android/10.11.1.35:15520 MULTI_sva: pool returned IPv4=10.8.0.2, IPv6=(Not enabled)
2022-02-16 01:37:15 android/10.11.1.35:15520 MULTI: Learn: 10.8.0.2 -> android/10.11.1.35:15520
2022-02-16 01:37:15 android/10.11.1.35:15520 MULTI: primary virtual IP for android/10.11.1.35:15520: 10.8.0.2
2022-02-16 01:37:15 android/10.11.1.35:15520 Data Channel: using negotiated cipher 'AES-256-GCM'
2022-02-16 01:37:15 android/10.11.1.35:15520 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2022-02-16 01:37:15 android/10.11.1.35:15520 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2022-02-16 01:37:15 android/10.11.1.35:15520 SENT CONTROL [android]: 'PUSH_REPLY,route 10.11.1.0 255.255.255.0,dhcp-option DNS 10.11.1.2,dhcp-option DNS 10.11.1.3,comp-lzo no,route-gateway 10.8.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.8.0.2 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)
2022-02-16 01:37:15 android/10.11.1.35:15520 PUSH: Received control message: 'PUSH_REQUEST'
2022-02-16 01:38:04 android/10.11.1.35:15520 Connection reset, restarting [0]
2022-02-16 01:38:04 android/10.11.1.35:15520 SIGUSR1[soft,connection-reset] received, client-instance restarting

Credits

Docker image lisenet/openvpn:latest is based on kylemanna/docker-openvpn Dockerfile.

OpenVPN Kubernetes configuration is based on a Helm chart provided by suda/k8s-ovpn-chart.

References

https://github.com/kylemanna/docker-openvpn

https://github.com/suda/k8s-ovpn-chart

One thought on “OpenVPN Server on Kubernetes”

  1. Hello,

    thank you for article,if we want to scale open vpn pod,How it works? I mean is it possible to know which pod will host.Then pods has connection between of them

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *