Here, I perform DevOps best practices but at home. Check out the hardware section where I describe what sort of hardware I am using. Thanks to Ansible, it's very easy for me to manage my home infrastructure and the cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using tools like Kubernetes, Flux, Renovate and GitHub Actions.
There is a template over at onedr0p/cluster-template if you wanted to try and follow along with some of the practices I use here.
My cluster has been migrated from a k3s/longhorn combo to Talos and Rook Ceph. First of all, Talos is amazing, and I really recommend it to anyone who is looking for a lightweight k8s distribution. Currently, I still use a node with the e1000 driver, and the second node doesn't have a good primary disk, so I am currently running in single controller mode with two workers. In the future, I would like to change the setup to have three controllers. The reason I switched to rook-ceph is longhorn feels less stable it's still under development and I decied to finally give rook-ceph a try.
- cert-manager - SSL certificates - with Cloudflare DNS challenge
- cillium - CNI for k8s
- cloudflared: Enables Cloudflare secure access to my ingresses.
- external-dns: Automatically syncs ingress DNS records to a DNS provider.
- external-secrets: Managed Kubernetes secrets using 1Password Connect.
- flux - GitOps tool for deploying manifests from the
clusterdirectory - ingress-nginx: Kubernetes ingress controller using NGINX as a reverse proxy and load balancer.
- k8s_gateway - DNS resolver for all types of external Kubernetes resources
- kube-vip - layer 2 load balancer for the Kubernetes control plane
- rook-ceph - storage class provider for data persistence
- reflector - mirror configmaps or secrets to other Kubernetes namespaces
- reloader - restart pods when Kubernetes
configmaporsecretchanges - sops: Managed secrets for Kubernetes which are commited to Git.
- spegel: Stateless cluster local OCI registry mirror.
Flux watches my kubernetes folder (see Directories below) and makes the changes to my cluster based on the YAML manifests.
The way Flux works for me here is it will recursively search the kubernetes/apps folder until it finds the most top level kustomization.yaml per directory and then apply all the resources listed in it. That aforementioned kustomization.yaml will generally only have a namespace resource and one or many Flux kustomizations. Those Flux kustomizations will generally have a HelmRelease or other resources related to the application underneath it which will be applied.
Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.
This Git repository contains the following directories under kubernetes.
📁 kubernetes # Kubernetes cluster defined as code
├─📁 bootstrap # Flux installation
├─📁 flux # Main Flux configuration of repository
└─📁 apps # Apps deployed into my cluster grouped by namespace (see below)My homelab runs on the following hardware (all k8s nodes are running on ubuntu 20.04):
| Device | OS Disk Size | Data Disk Size | Ram | Purpose |
|---|---|---|---|---|
| k8s-2 (Intel NUC) | 1TB SSD SATA | 250GB NVMe | 32GB | Talos node |
| k8s-1 (Udoo Bolt V8 AMD Ryzen) | eMMC 30GB | 250GB NVMe | 32GB | Talos node |
| k8s-0 (VM) | 250GB NVMe SCSi | 250GB NVMe | 32GB | Talos node with Nvidia GPU and NVMe Disk |
| TRUENAS | ZFS raidz 1 40TB | 4x10TB HDD | 64GB | Storage |
| Unifi UDM Pro | SSD 14GB | HDD 1TB | 4GB | Router and security Gateway |
| Unifi Switch 16 PoE | N/A | N/A | N/A | Switch with 802.3at PoE+ ports |
| Database Server | 20GB | N/A | 2GB | Database |
| Offsite Machine | 60 GB | 8TB | 8GB | Backup offsite vm |
Feel free to checkout my blog axell.dev which is also open source! I also have made a blog post about HW, what were my choices... which ones were good and which ones were bad. Click here.
I am proud to be a member of the home operations (previously k8s-at-home) community! I received a lot of help and inspiration for my Kubernetes cluster from this community which helped a lot. Thanks! ❤️
If you are interested in running your own k8s cluster at home, I highly recommend you to check out the k8s-at-home website.
Be sure to check out kubesearch.dev for ideas on how to deploy applications or get ideas on what you may deploy.
See LINCENSE.