Setup yt cluster in k8s with porto

[ernado]

Blockers:

• no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config (fix)
• RunPodSandbox: DockerImageNotFound: registry.k8s.io/pause:3.7 (workaround, portoctl docker-pull registry.k8s.io/pause:3.7) DockerImageNotFound: registry.k8s.io/pause:3.7 #2
• Unknown manifest mediaType: application/vnd.oci.image.index.v1+json (not a blocker Support OCI images ten-nancy/porto#7, not using OCI for now)
• CreateContainer: InvalidPath: Storage path does not exist
• PullImage: Unknown: Resource temporarily unavailable (fix)
• .CreateContainer: LayerNotFound: Layer not found
• memory_guarantee is not supported (fix), issue: Do not use memory_guarantee for vanilla kernels ten-nancy/portoshim#1
• volume link removes symlinks in parent directories  ten-nancy/porto#6

Progress:

• kubeadm init works
• basic CNI works
• basic 1-node k8s cluster operational
• basic CRI works (openebs, localpath)
• ytsaurus cluster is starting
• ytsaurus is working using porto integration (usePorto: true)
• cilium works
• ytsaurus+porto works with cilium

Current setup instructions:

1. Use Ubuntu 20.04 for systemd hybrid mode cgroup setup

2. Install porto, use fork:

Additional build deps

sudo apt install -y libtool autoconf

Configuration

cat >/etc/portod.conf.d/k8s.conf << EOF
log {
  verbose: true
  debug: true
}
daemon {
  docker_images_support: true
}
container {
  enable_systemd: true
  detect_systemd: true
  propagate_cpu_guarantee: true
  enable_blkio: true
  enable_cgroup2: true
  use_os_mode_cgroupns: true
  enable_docker_mode: true
  enable_rw_cgroupfs: true
  enable_numa_migration: true
  enable_rw_net_cgroups: true
  cpu_limit_scale: 1
  proportional_cpu_shares: false
  memory_high_limit_proportion: 0
  enable_sched_idle: true
}
EOF

3. Install portoshim

Also add crictl config:

cat >/etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/portoshim.sock
EOF

Setup binaries:

make
sudo make install

4. Install kubeadm and tools
5. Setup bridge CNI

mkdir -p /etc/cni/net.d /etc/cni/net.d/

Install cni plugins:

git clone https://github.com/containernetworking/plugins
cd plugins
./build_linux.sh
sudo mkdir -p /opt/cni/bin
sudo cp ./bin/* /opt/cni/bin/

Configure CNI:

cat >/etc/cni/net.d/10-porto.conflist << EOF
{
  "cniVersion": "1.0.0",
  "name": "porto",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ],
        "ranges": [
            [{ "subnet": "10.85.0.0/16" }]
        ]
      }
    }
  ]
}
EOF

6. Setup network options

modprobe br_netfilter
echo br_netfilter > /etc/modules-load.d/br_netfilter.conf

cat >/etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward=1
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl -p

7. Disable swap
8. Init cluster

portoctl docker-pull registry.k8s.io/pause:3.7

export CRI_ENDPOINT=unix:///run/portoshim.sock
kubeadm init --cri-socket=$CRI_ENDPOINT
kubectl taint nodes $(hostname) node-role.kubernetes.io/control-plane:NoSchedule-

9. Setup cilium

Install cli:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Install Cilium:

kubectl create ns cilium
cilium install --version 1.14.5 -n cilium --set bpf.autoMount.enabled=false

Check:

kubectl -n cilium get pods
kubectl -n cilium exec -i -t -c cilium-agent daemonset/cilium -- cilium endpoint list

10. Install CRI (openebs)

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm install openebs --namespace openebs openebs/openebs --create-namespace

Add default storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-hostpath
  annotations:
    openebs.io/cas-type: local
    storageclass.kubernetes.io/is-default-class: "true"
    cas.openebs.io/config: |
      - name: StorageType
        value: hostpath
      - name: BasePath
        value: /var/openebs
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: openebs-hostpath-pvc
spec:
  storageClassName: openebs-hostpath
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G

Check:

---
apiVersion: v1
kind: Pod
metadata:
  name: hello-local-hostpath-pod
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: openebs-hostpath-pvc
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage

kubectl describe pod hello-local-hostpath-pod 
kubectl get pv

11. Install ytsaurus operator

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl -n cert-manager rollout status --timeout=1m deployment cert-manager-webhook

helm pull oci://docker.io/ytsaurus/ytop-chart --version 0.4.1 --untar
helm upgrade --install --namespace ytop --create-namespace  ytsaurus ytop-chart/
kubectl -n ytop rollout status --timeout=1m deployment ytsaurus-ytop-chart-controller-manager

12. Setup ytsaurus cluster

As per minikube instructions:

wget https://github.com/ytsaurus/yt-k8s-operator/blob/main/config/samples/0.4.0/cluster_v1_minikube.yaml
kubectl create ns yt
kubectl apply -n yt -f cluster_v1_minikube.yaml

Minikube

git clone https://github.com/go-faster/minikube.git
cd minikube
make out/minikube
wget -O /tmp/minikube.iso https://github.com/go-faster/minikube/releases/download/v1.32.1-alpha.0/minikube-amd64.iso
./out/minikube start --iso-url=file:///tmp/minikube.iso --cni=cilium --container-runtime=porto --cache-images=false 

[ernado]

Should be mostly done.

Deploy and auxiliary scripts are moving to https://github.com/go-faster/ytst.

Issues extracted to #2, #3, #4.