Home Kubernetes cluster managed by GitOps

---

k3s cluster deployed with Ansible, backed by Flux and SOPS.

Components

• flux - GitOps operator for managing Kubernetes clusters from a Git repository
• kube-vip - Load balancer for the Kubernetes control plane nodes
• metallb - Load balancer for Kubernetes services
• cert-manager - Operator to request SSL certificates and store them as Kubernetes resources
• calico - Container networking interface for inter pod and service networking
• external-dns - Operator to publish DNS records to Cloudflare (and other providers) based on Kubernetes ingresses
• k8s_gateway - DNS resolver that provides local DNS to the Kubernetes ingresses
• traefik - Kubernetes ingress controller used for a HTTP reverse proxy of Kubernetes ingresses
• local-path-provisioner - provision persistent local storage with Kubernetes

Additional applications include hajimari, error-pages, echo-server, system-upgrade-controller, reflector, reloader, and kured

Provisioning is performed using Ansible.

📂 Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

📁 cluster      # k8s cluster defined as code
├─📁 flux       # flux, gitops operator, loaded before everything
├─📁 crds       # custom resources, loaded before 📁 core and 📁 apps
├─📁 charts     # helm repos, loaded before 📁 core and 📁 apps
├─📁 config     # cluster config, loaded before 📁 core and 📁 apps
├─📁 core       # crucial apps, namespaced dir tree, loaded before 📁 apps
└─📁 apps       # regular apps, namespaced dir tree, loaded last

🚀 Installation

🔐 Setting up Age

Create a Age Private and Public key. Using SOPS with Age allows us to encrypt secrets and use them in Ansible and Flux.

1. Create a Age Private / Public Key

   age-keygen -o age.agekey

2. Set up the directory for the Age key and move the Age file to it

   mkdir -p ~/.config/sops/age
   mv age.agekey ~/.config/sops/age/keys.txt

3. Export the SOPS_AGE_KEY_FILE variable in bashrc, zshrc or config.fish and source it, e.g.

   export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
   source ~/.bashrc

4. Fill out the Age public key in the .config.env under BOOTSTRAP_AGE_PUBLIC_KEY, note the public key should start with age

📄 Configuration

The .config.env file contains necessary configuration that is needed by Ansible and Flux.

1. Copy the .config.sample.env to .config.env and start filling out all the environment variables:

   All are required unless otherwise noted in the comments.

   cp .config.sample.env .config.env

2. Once that is done, verify the configuration is correct by running:

   task verify

3. If you do not encounter any errors, wire up the templated files and place them where they need to be.

   task configure

⚡ Preparing Ubuntu with Ansible

1. Ensure you are able to SSH into your nodes using a private SSH key without a passphrase.

2. Install the Ansible deps:

   task ansible:init

3. Verify Ansible can view the config:

   task ansible:list

4. Verify Ansible can ping the nodes:

   task ansible:ping

5. Run the Ubuntu Prepare Ansible playbook:

   task ansible:prepare

6. Reboot the nodes:

   task ansible:reboot

⛵ Installing k3s with Ansible

☢️ If you run into problems, you can run task ansible:nuke to destroy the k3s cluster and start over.

1. Verify Ansible can view your config:

   task ansible:list

2. Verify Ansible can ping your nodes:

   task ansible:ping

3. Install k3s with Ansible:

   task ansible:install

4. Verify the nodes are online:

   task cluster:nodes
   # NAME           STATUS   ROLES                       AGE     VERSION
   # k8s-0          Ready    control-plane,master      4d20h   v1.21.5+k3s1
   # k8s-1          Ready    worker                    4d20h   v1.21.5+k3s1

🔹 GitOps with Flux

1. Verify Flux can be installed:

   task cluster:verify
   # ► checking prerequisites
   # ✔ kubectl 1.21.5 >=1.18.0-0
   # ✔ Kubernetes 1.21.5+k3s1 >=1.16.0-0
   # ✔ prerequisites checks passed

2. Install Flux and sync the cluster to the Git repository:

   task cluster:install
   # namespace/flux-system configured
   # customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io created

3. Verify Flux components are running in the cluster:

   task cluster:pods -- -n flux-system
   # NAME                                       READY   STATUS    RESTARTS   AGE
   # helm-controller-5bbd94c75-89sb4            1/1     Running   0          1h
   # kustomize-controller-7b67b6b77d-nqc67      1/1     Running   0          1h
   # notification-controller-7c46575844-k4bvr   1/1     Running   0          1h
   # source-controller-7d6875bcb4-zqw9f         1/1     Running   0          1h

🎤 Verification Steps

1. View the Flux Git Repositories:

   task cluster:gitrepositories

2. View the Flux kustomizations:

   task cluster:kustomizations

3. View all the Flux Helm Releases:

   task cluster:helmreleases

4. View all the Flux Helm Repositories:

   task cluster:helmrepositories

5. View all the Pods:

   task cluster:pods

6. View all the certificates and certificate requests:

   task cluster:certificates

7. View all the ingresses:

   task cluster:ingresses

All the commands above can be run with one task:

task cluster:resources