k8s cluster setting

This repository is a guide on how to build a Kubernetes cluster using Oracle virtual machines. Detailed virtual machine specifications and setup methods will be explained below.

Contents

1. VM Spec
2. VM Setting
3. Container runtime interface install ( CRI-O )
4. Kubernetes install
5. Kubernetes Cluster
6. ERROR FIX

1. VM Spec

We will create three virtual machines.

master node, Worker node Spec
    - Processor : 2
    - System memory : 4096 MB
    - Vedio memory : 16 MB

The virtual IPs and ports we will use when building a cluster are as follows.

Node name	Node IP	Node Port
k8s-master	192.168.1.10	1000
k8s-node1	192.168.1.10	1000
k8s-node2	192.168.1.10	1000
k8s-node3	192.168.1.10	1000

2. VM Setting

First, we will create a master virtual machine and name it ‘k8s-master’. Then install Ubuntu Live Server on this virtual machine.

Second, modify the /etc/netplan/00-installer-config.yaml file as follows.

sudo vi /etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
    ethernets:
        enp0s3:
            addresses:
                - 192.168.1.10/24
            routes:
                - to: default
                  via: 192.168.1.1
            nameservers:
                addresses:
                    - 8.8.8.8
                search:
                    - 8.8.4.4
    version: 2

And then apply netplan.

sudo netplan apply

If you want to check the changed results, try using the ifconfig command.

Third, set up the hosts.

sudo vi /etc/hosts

127.0.0.1 localhost
192.168.1.10 k8s-master
192.168.1.11 k8s-node1
192.168.1.12 k8s-node2
192.168.1.12 k8s-node3

Then, shut down the virtual machine and create two copies of the master virtual machine.

Finally, change the IP and hostname of the newly created virtual machine.

master@k8s-node1: ~$ sudo vi /etc/netplan/00installer-config.yaml

master@k8s-node1: ~$ sudo vi /etc/hostname

hostname

k8s-node1

master@k8s-node1: ~$ sudo hostname -F /etc/hostname
master@k8s-node1: ~$ sudo reboot

Applies to node2, node3 as well.

3. Container runtime interface install ( CRI-O )

If you are using k8s version 1.20.x or later, you will need to install Container runtime interface (CRI).

CRI is a standard interface called CRI that allows communication with multiple container runtimes.

We will install and use CRI-O among several CRIs.

Change to root privileges

master@k8s-master:~$ sudo -i

First, set up the network settings.

root@k8s-master:~$ cat <<EOF | sudo tee /etc/modules-load.d/crio.conf
overlay
br_netfilter
EOF

root@k8s-master:~$ sudo modprobe overlay
root@k8s-master:~$ sudo modprobe br_netfilter

root@k8s-master:~$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

root@k8s-master:~$ sudo sysctl --system

CRI-O install.

root@k8s-master:~$ export OS=xUbuntu_22.04 # OS version
root@k8s-master:~$ export VERSION=1.24 # CRI-O version

root@k8s-master:~$ echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
root@k8s-master:~$ echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

root@k8s-master:~$ curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -
root@k8s-master:~$ curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -

root@k8s-master:~$ sudo apt-get update

root@k8s-master:~$ sudo apt-get -y install cri-o cri-o-runc cri-tools

root@k8s-master:~$ sudo systemctl daemon-reload
root@k8s-master:~$ sudo systemctl enable crio --now

4. Kubernetes Install

For more information, please visit the Kubernetes homepage.

Kubernetes install

$ sudo apt-get update
$ sudo apt-get install -y apt-transport-https ca-certificates curl gpg

$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt-get update
$ sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl

$ sudo systemctl enable --now kubelet

5. Kubernetes Cluster

root@k8s-master:~$ kubeadm init

Running the kubeadm command generates relevant setup commands and a cluster configuration token.

root@k8s-master:~$ mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

root@k8s-node{1~3}:~$ kubeadm join 192.168.1.10:6443 --token [token..]

Grant kubectl commands to regular users as well

master@k8s-master:~$ mkdir -p $HOME/.kube
master@k8s-master:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
master@k8s-master:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

6. ERROR FIX

6-1. You must perform this command if your Kubernetes API server is unstable.

$ containerd config default | tee /etc/containerd/config.toml
$ sed -i 's/SystemdCgroup = False/SystemdCgroup = true/g' /etc/containerd/config.toml
$ service containerd restart
$ service kubelet restart

I believe this issue is caused by Docker and Containerd crashing while running.

6-2. If node is not ready state.

$ systemctl restart kubelet
$ systemctl restart containerd