There are many resources that can help you setting up a kubernetes cluster on a Raspberry Pi, but many of them only focus on some specific aspects. The idea of this repo is to try to collect all the aspects of a decent kubernetes setup for a Raspberry Pi, from dedicated considerations on the ARM architecture, to some basic networking aspects of a home made cluster.
This repo contains instructions for the setup of a kubernetes cluster on a raspberry pi(s) based on:
- k3s: a lightweight kubernetes distribution
- nginx ingress controller
- cert-manager to manage tls certificates generation
The plan is to extend this list to include tools for monitoring, logging and other functionalities.
To work with kubernetes we will need some cli tools to interact with our clusters. Some of them are:
- kubectl
- helm
A fantastic way to install different clis is provided by arkade, a tool created by Alex Ellis, the founder of OpenFaas.
To install arkade
simply run:
curl -sLS https://dl.get-arkade.dev | sudo sh
When arkade setup is done, to install kubectl and helm you simply need to run:
ark get kubectl
ark get helm
ark is a handy alias of arkade
The easiest way to install k3s is to run the following command:
curl -sfL https://get.k3s.io | sh -
This will create your master node, setup a systemd
service, create a kubectl configuration file and some shell scripts that can be used to stop it and/or uninstall it.
Nevertheless, this simple setup also deploys the default ingress controller (Traefik), but since I decided to go with nginx, it is possible to change the setup command in:
curl -sfL https://get.k3s.io | sh -s - --no-deploy traefik
full documentation for the setup options of a master node is available here
Now you can check that the cluster is up and running using kubectl, but first you have to retrieve the config file to connect to the cluster.
This file is automatically generated by the k3s setup and is located in /etc/rancher/k3s/k3s.yaml
; a simple way to use it is to copy the file in your local .kube
directory and define the KUBECONFIG
environment variable:
sudo cp /etc/rancher/k3s/k3s.yaml ${HOME}/.kube/k3s-config
export KUBECONFIG=${HOME}/.kube/k3s-config
To verify that your cluster is up and running, now you can run:
$ kubectl get node
NAME STATUS ROLES AGE VERSION
raspberrypi Ready master 12s v1.19.5+k3s2
The Nginx ingress controller provides a way to manage the incoming traffic using nginx as a reverse proxy.
To install the ingress controller the best reference to follow is available at the official website, in particular I will follow the helm approach described here.
The documentation describes some peculiarities of the bare metal setup that I suggest to read here; the main points are related to:
- having a service for the controller of type NodePort instead of LoadBalancer (usually automatically provided by the cloud provider for managed clusters like AKS, EKS, ...)
- making sure that the nginx controller is only deployed on a single node (the one that will be exposed to the internet traffic through the router port forwarding)
To ensure the second point first of all we need to tag the node in such a way that it will be the selected by the kubernetes scheduler for the nginx controller pods. Assuming a name raspberrypi
for the node (which coincide with the hostname), then you can execute the following command:
kubectl label nodes raspberrypi external-exposed=true
Now we can move on with the ingress controller setup.
The first step is to add the nginx helm repo and update helm:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
Now you are ready to install the helm chart with this set of additional values:
controller:
nodeSelector:
external-exposed: "true"
service:
nodePorts:
http: 30080
https: 30443
type: NodePort
externalTrafficPolicy: Local
defaultBackend:
enabled: true
image:
repository: k8s.gcr.io/defaultbackend-arm
The command to run (assuming the above file is called ingress.custom-values.yaml
) is simply:
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx -f ingress-custom-values.yaml
The approach with explicit --set
values doesn't work because of the way the selector for the node gets written.
With this set of values the ingress controller will be deployed on the kubernetes cluster, the controller pods will be scheduled on the node labeled with external-exposed=true
, will be exposed with a service of type NodePort on the ports 30080 (http) and 30443 (https), will preserve the source IP thanks to the externalTrafficPolicy
and will have a default backend with a dedicated arm image (the default backend image is not multi architecture, a specific tag is required) for all the requests that do not match any ingress definition.
To check the availability of the ingress controller you should see something similar to:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
ingress-nginx-defaultbackend-6b59ff499f-5dhjx 1/1 Running 0 15s
ingress-nginx-controller-f5b8f5b4-s6q6z 1/1 Running 0 15s
Now, since no ingress resources are defined for any backend, every http request (to the node port 30080 as defined in the helm deploy) to the ingress entry-point will return a 404 (except /healthz
). The request curl -i http://localhost:30080/whatever
will give something like:
HTTP/1.1 404 Not Found
Date: Thu, 07 Jan 2021 22:43:44 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 21
Connection: keep-alive
default backend - 404
The same is also true for https requests on the port 30443 (now we need to accept insecure connections with -k
because we did not provide any tls certificate) of the form curl -ik https://localhost:30443/whatever
Cert manager is a tool that simplifies the operations required to generate a tls certificate for a specific domain name. Clearly you need to have a domain name first, and for this you have some options/possibilities:
- if your ISP provider gives you a static ip you can register a domain name and bind it to your ip
- if you don't have a static ip, you can rely on a dynamic DNS service (see here for further details): most modern modems give you the possibility to automatically update the DNS resolution at every ip change (these services often offer a free plan to be manually renewed every month)
The cert manager setup is done directly following the approach suggested in the official documentation.
First we need a dedicated namespace:
kubectl create ns cert-manager
Then we need to add the cert manager helm repository:
helm repo add jetstack https://charts.jetstack.io
helm repo update
Finally we can deploy it:
helm upgrade --install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.1.0 \
--set installCRDs=true \
--wait
the
--wait
option helps having the resources ready for the next steps
To be able to generate ACME certificates with Let's Encrypt, we need to have a ClusterIssuer (or issuer, the difference is that an Issuer is bound to a namespace) resource on the cluster (see full documentation here).
Let's Encrypt offers, together with the production apis to request certificates (based on HTTP01 or DNS challenges) also staging apis to validate the workflow without having to worry too much about aggressive rate limits.
To simplify the delivery of both Let's Encrypt ClusterIssuers and additional certificate request resources (see here), you can deploy the two provided helm charts:
-
letsencrypt-cluster-issuers
:helm upgrade --install letsencrypt-cluster-issuers ./letsencrypt-cluster-issuers \ --set email=<email used for ACME registration> \ --wait
-
certificate-request:
helm upgrade --install certificate-request certificate-request \ --set tls.hostname=<DNS name for which you request a certificate> \ --set clusterIssuer.type=<prod|stag letsencrypt issuer> \ --set tls.secret.prefix=<prefix for the secret that will store the generated certificate> \ --wait
These last steps will create both the staging and production ClusterIssuers for Lest's Encrypt, together with the certificate request for the DNS name associated to the public ip of your cluster.
This setup is based on an HTTP-01 challenge (entirely managed by cert manager), but you need to ensure that you cluster can be reached using the dns on the standard port 80 (see the Let's Encrypt documentation)
All the instructions described here can be executed launching the setup-master.sh
script:
./setup-master.sh \
-e <your email for certificate requests> \
-n <your domain name> \
-p prod
the
-p
option refers to the prod or stag Let's Encrypt issuer
What has been described so far is the setup of a kubernetes cluster consisting of only 1 master node. k3s offers an easy way to add an additional worker node to the cluster, still based on the same installer.
First of all we need to retrieve the token created for the master node: this can be found by simply running, on the master node:
sudo cat /var/lib/rancher/k3s/server/node-token
This token will allow the worker node to join the cluster thanks to the command (to be executed on the worker node):
curl -sfL https://get.k3s.io | K3S_URL=<url of the cluster> K3S_TOKEN=<node token obtained from master node> sh -s -
By default, all the raspberry pis have the same hostname (raspberrypi
), and this can generate some confusion during the setup: you can follow one of these approaches:
- change the hostname of the worker node; the best approach in my opinion because it also allows for better identification of the pi in your network
- use the
--node-name
option for k3s setup - use the
--with-node-id
option for k3s setup
When the worker setup is complete, from the master node it is possible to verify that the worker node has been added with
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION
raspberrypi Ready master 151m v1.19.5+k3s2
raspberrypi2 Ready <none> 42s v1.19.5+k3s2
In some k3s versions the role of the worker node is not defined (""); to fix that it is possible to run:
kubectl label node <name of the worker node> node-role.kubernetes.io/worker=worker