This file explains how to setup Amazon EKS (Elastic Kubernetes Service) on AWS. Then, it describes how to deploy Polyaxon (platform to manage machine learning experiments at scale) to the Kubernetes cluster using Helm chart and establish connection from local machine.
Command prefixes:
$
- run on local machineec2-user$
- run on AWS machine
$ pacman -S aws-cli
$ aws configure
For detailed instructions look here. Optionally, you may also install aws-shell, an interactive shell for the AWS CLI.
To enable ssh connection with created VMs you need to use an existing Key Pair or create a new one.
To create new Key Pair with name eks-key-pair
(has to be unique) run:
$ aws ec2 create-key-pair --key-name eks-key-pair --query 'KeyMaterial' --output text > ~/.aws/eks-private-key.pem
$ chmod 400 ~/.aws/eks-private-key.pem
$ aws cloudformation deploy \
--template-file amazon-eks-master.template.yaml \
--stack-name eks-stack \
--parameter-overrides AvailabilityZones=eu-west-2a,eu-west-2b,eu-west-2c KeyPairName=eks-key-pair RemoteAccessCIDR=0.0.0.0/0
For detailed instructions look here.
Created stack contains bastion of Linux VMs, which have kubectl
to interact with Kubernetes cluster.
They also have helm
to manage Kubernetes packages, including Polyaxon.
Find the public IP of one of the VMs:
$ IP=$(
aws ec2 describe-instances \
--filter Name=tag:Name,Values=LinuxBastion \
--query "Reservations[].Instances[].PublicIpAddress" \
--output text
)
Connect to it using ssh:
$ ssh -i ~/.aws/eks-private-key.pem ec2-user@${IP}
Test kubectl and helm:
ec2-user$ kubectl version
ec2-user$ helm version
ec2-user$ helm repo add polyaxon https://charts.polyaxon.com
ec2-user$ helm repo update
ec2-user$ helm install --name=polyaxon --namespace=polyaxon --wait polyaxon/polyaxon
ec2-user$ POLYAXON_IP=$(
kubectl get svc \
--namespace polyaxon polyaxon-polyaxon-api \
-o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
)
ec2-user$ POLYAXON_ROOT_PASSWORD=$(
kubectl get secret \
--namespace polyaxon polyaxon-polyaxon-secret \
-o jsonpath="{.data.POLYAXON_ADMIN_PASSWORD}" \
| base64 --decode
)
ec2-user$ echo "Polyaxon dashboard is available at: ${POLYAXON_IP}"
ec2-user$ echo "To setup polyaxon from local machine run:
polyaxon config set --host=${POLYAXON_IP} --http_port=80 --ws_port=1337
polyaxon login
"
ec2-user$ echo "USER: root"
ec2-user$ echo "PASSWORD: ${POLYAXON_ROOT_PASSWORD}"
$ pip install -U polyaxon-cli
$ polyaxon config set --host=<as-printed-above> --http_port=80 --ws_port=1337
$ polyaxon login --username root --password rootpassword
$ polyaxon cluster
$ polyaxon project create --name=quick-start --description='Polyaxon quick start.'
ec2-user$ helm delete polyaxon --purge
$ aws cloudformation delete-stack --stack-name eks-stack