Skip to content

Latest commit

 

History

History
208 lines (170 loc) · 6.47 KB

INSTALL.md

File metadata and controls

208 lines (170 loc) · 6.47 KB
Raw
  1. Install Dependencies
  2. Deployment
    1. Create a single-node Kubernetes cluster with Minikube
    2. Deploy KNoC as a Kubernetes node
  3. Testing
  4. Uninstall KNoC and Minikube

Install Dependencies

Docker

Install docker following the guide here.

Next start docker service and enable the permissions needed for the user

systemctl start docker
sudo groupadd docker
sudo usermod -aG docker $(whoami)

Log out and log back in so that your group membership is re-evaluated. OR (if in linux) run newgrp docker

  • you can check that docker is installed and running by running docker ps
    • you should see NO container running and NO message related to "permission denied"

Minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

Helm

In case you have curl installed...

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

or alternatively..

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Deployment

Start Minikube

In order to create a minikube single-node-cluster, you need to specify the minikube profile and the ip where the minikube's Kubernetes will listen to, for external requests.

First we need to set the environment:

export MINIKUBE_PROFILE=knoc
export ADVERTISED_HOST=139.91.92.71
export API_SERVER_PORT=8443
export PROXY_API_SERVER_PORT=38080
export KUBE_PROXY=${ADVERTISED_HOST}:${PROXY_API_SERVER_PORT}
  • optionally,do the following steps, in case this is not your first kubernetes configuration..
# save previous kubeconfig and prepare the path for the new kubeconfig generated by minikube
export LAST_KUBECONFIG="${KUBECONFIG:=/home/$(whoami)/.kube/config}" 
export KUBECONFIG="${HOME}/.kube/configs/${MINIKUBE_PROFILE}.yaml"

Now we re are ready to start the minikube:

minikube start -p ${MINIKUBE_PROFILE} --kubernetes-version=v1.19.13 --apiserver-ips=${ADVERTISED_HOST}
# To point your shell to minikube's docker-daemon, run
eval $(minikube -p $MINIKUBE_PROFILE docker-env) 

# Expose Kubernetes API server from minikube, using socat. 
# This is required for the argo executor that needs connection to the K8s Api server
# Then socat forwards traffic from the <system_that_hosts_minikube> to the ip of minikube
# redirect web traffic from one server to the local
socat TCP-LISTEN:${PROXY_API_SERVER_PORT},fork TCP:$(minikube -p $MINIKUBE_PROFILE ip):${API_SERVER_PORT} &

We can use the kubernetes cli that minikube is providing:

alias kubectl='minikube -p knoc kubectl --'

if you want to have the kubectl attached to minikube's specific version you can save the command bellow to your ~/.bashrc

echo "alias kubectl='minikube -p knoc kubectl --'" >> /home/$(whoami)/.bashrc

#refresh your terminal session
# i.e. in bash
# . ~/.bashrc

Deploy KNOC

Before using kubectl or docker commands, you have to first configure the terminal session you are in, with the command below:

export HELM_RELEASE=knoc
export SLURM_CLUSTER_IP=139.91.92.100
export SLURM_CLUSTER_USER=$(whoami)
export SLURM_CLUSTER_SSH_PRIV=/home/${SLURM_CLUSTER_USER}/.ssh/id_rsa

#################################################################

Make sure that you can login on the remote side using your private ssh key

#################################################################

# Download the source code
git clone git@github.com:CARV-ICS-FORTH/KNoC.git
cd KNoC

#build the container
docker build -t malvag/knoc:latest .

# setup vanilla argo in our cluster (Argo 3.0.2) that uses a slightly modified version of k8sapi-executor
kubectl create ns argo
kubectl apply -n argo -f deploy/argo-install.yaml


# And now you can run this

helm upgrade --install --debug --wait $HELM_RELEASE chart/knoc --namespace default \
    --set knoc.k8sApiServer=https://${KUBE_PROXY} \
    --set knoc.remoteSecret.address=${SLURM_CLUSTER_IP} \
    --set knoc.remoteSecret.user=${SLURM_CLUSTER_USER}  \
    --set knoc.remoteSecret.kubeContext=$(kubectl config current-context) \
    --set knoc.remoteSecret.privkey="$(cat ${SLURM_CLUSTER_SSH_PRIV})"

Testing our deployment

You can test our deployment is working by submiting a sample workflow to argo:

kubectl create -f examples/argo-workflow-sample.yaml

 -- example output: 
 workflow.argoproj.io/steps-pzvmd created

You can check that eveything works fine by executing the follow command after a minute or two:

kubectl get pods  # List all pods in the namespace

NAME                    READY     STATUS      RESTARTS   AGE
sample-workflow-bws9f   0/2       Completed   0          6m

You can expect to see the final state is going to be "Completed".

Delete the sample workflow

kubectl delete workflow $(kubectl get workflow --no-headers | cut -f1 -d' ')

Tear down

Remove knoc

helm uninstall --wait $HELM_RELEASE

In case you want to clean everything from the remote side:

# Clean slurm outputs and door executable
rm -f slurm-*.out door
# now let's clean door logs, kubernetes associated files and generated scripts
rm -rf .knoc .tmp

Delete minikube's profile

This command deletes the whole minikube-vm that includes the Kubernetes and the Docker deployments inside the vm.

minikube stop -p $MINIKUBE_PROFILE
minikube delete -p $MINIKUBE_PROFILE
# revert back to the old kubeconfig if you need to..
export KUBECONFIG=${LAST_KUBECONFIG}

Remove minikube and its data

minikube stop -p $MINIKUBE_PROFILE; minikube delete -p $MINIKUBE_PROFILE

# ++ optionally: if you run minikube on docker
unset DOCKER_HOST
unset DOCKER_TLS_VERIFY
unset DOCKER_CERT_PATH
docker stop (docker ps -aq)
# ++

rm -r ~/.kube ~/.minikube
sudo rm /usr/local/bin/minikube
sudo rm /usr/local/bin/helm

systemctl stop '*kubelet*.mount'
sudo rm -rf /etc/kubernetes/

# ++ optionally: if you run minikube on docker
docker system prune -af --volumes
systemctl stop docker
# ++