Skip to content

Running CDK in a restricted environment

Jump to bottom
Rye Terrell edited this page Dec 18, 2017 · 25 revisions

You'll need two pieces of infrastructure to run CDK in an egress-restricted environment: an apt mirror and a docker registry.

Set up an apt mirror

First, create a VM with sufficient disk space. You’ll want approximately 200GB free for use by the apt mirror.

Install apt-mirror and nginx:

sudo apt update && sudo apt install -y apt-mirror nginx

Configure apt-mirror:

sudo tee /etc/apt/mirror.list > /dev/null <<EOL
set nthreads     20
set _tilde 0

deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse

deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu
EOL

Run apt-mirror (this will download ~170GB of data and can take a while):

sudo apt-mirror

Configure nginx:

sudo tee /etc/nginx/sites-enabled/default > /dev/null <<EOL
server {
        listen 80 default_server;
        listen [::]:80 default_server;
        root /var/www;
}
EOL

Serve the mirror:

sudo ln -s /var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu /var/www/ubuntu

Set up a private docker registry

First, set up a VM and install docker on it:

sudo apt install -y docker.io

Then kick off the registry:

sudo docker run -d -p 5000:5000 --restart=always --name registry registry:2

Finally, add the images you'll need to run Kubernetes to your registry. You may need some or all of the following images, depending on which CNI plugin you're using:

quay.io/calico/node
quay.io/calico/kube-policy-controller
gcr.io/google_containers/k8s-dns-sidecar-amd64
gcr.io/google_containers/k8s-dns-kube-dns-amd64
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64
gcr.io/google_containers/heapster-amd64
gcr.io/google_containers/nginx-ingress-controller
gcr.io/google_containers/heapster-influxdb-amd64
gcr.io/google_containers/heapster-grafana-amd64
gcr.io/google_containers/addon-resizer
gcr.io/google_containers/pause-amd64

Bootstrap an offline Juju environment

Note: as of November 2017, the following instructions do not work with the snap installation of Juju. Please use the apt-installed version for now.

Note: if you're using cdk-offline, skip this step.

We'll need to gather some metadata to boostrap Juju to work in a restricted environment. Create a directory to store Juju's metadata:

mkdir juju-metadata

Next generate the metadata. The following example is specific to the ap-southeast-2 region of AWS. See juju metadata generate-image --help and this json data to generate appropriate metadata for your environment.

juju metadata generate-image \
	-d juju-metadata \
	-i ami-550c3c36 \
	-r ap-southeast-2 \
	-u https://ec2.ap-southeast-2.amazonaws.com \
	--virt-type hvm \
	--storage=ssd

Finally, bootstrap Juju with your apt mirror and metadata:

juju bootstrap
     --metadata-source /home/ubuntu/juju-metadata \
     --config apt-mirror=http://<your apt mirror IP>/ubuntu/ \
     --config agent-stream=release --debug

Install CDK using cdk-shrinkwrap

On a machine outside your restricted environment (but with identical architecture), clone the cdk-shrinkwrap repo:

git clone https://github.com/juju-solutions/cdk-shrinkwrap.git

Enter the repo directory and run the script for your target CDK bundle (using canonical-kubernetes as an example):

cd cdk-shrinkwrap && ./shrinkwrap.py canonical-kubernetes --channel stable

This will produce a versioned and timestamped tarball of CDK, e.g., canonical-kubernetes-1.8.1-stable-amd64-2017-10-26-08-17-24.tar.gz.

Copy this tarball into your restricted environment. Then (again, using the canonical-kubernetes bundle example):

tar -xf canonical-kubernetes.tar.gz && cd canonical-kubernetes && ./deploy.sh

Note: you may need to edit the deploy.sh script to more accurately fit your needs. For example. you may need to add machines with constraints particular to your environment.

The cdk-offline test harness

The cdk-offline test harness creates a VPC in AWS with two subnets - one private, one public. One machine is created on the public subnet, which serves as a host for a squid proxy and the juju client. We'll refer to this machine henceforth as the client machine. On the private subnet, a controller is bootstrapped.

A gateway is created and attached to the VPC. A route is associated with the private subnet that routes all traffic through the client machine. Iptables is used to redirect all traffic going to ports 80 and 443 to ports 3129 and 3130, respectively, which squid is listening to. All other traffic is simply dropped.

The squid proxy server will proxy your apt mirror (which you must provide - see instructions above).

Deploy an offline testbed with cdk-offline

Preparation:

  • Create an apt mirror
  • Create a private docker registry
  • Create a cdk-shrinkwrap tarball
  • Configure your aws cli tool for the ap-southeast-2 region. This will be made configurable later.

Clone cdk-offline:

git clone https://github.com/juju-solutions/cdk-offline.git

Enter the repo directory and run deploy-squid.sh:

cd cdk-offline && ./deploy-squid.sh <your apt mirror IP address>

Standby to enter your aws credentials. These are used to bootstrap the juju environment and are not persisted.

After a while, the deployment script will complete. At the end of the deployment, a number of data points will be printed out. One of these is the IP address of the client machine. Make note of this.

During the deployment, a private key will be created and stored in the file cdk-offline.pem so that you can log into the client machine. Use this key to copy your cdk-shrinkwrap tarball to the client machine, e.g.:

scp -i ./cdk-offline.pem canonical-kubernetes-1.8.1-stable-amd64-2017-10-26-08-17-24.tar.gz ubuntu@123.45.67.89:

Then ssh to the client machine:

ssh -i ./cdk-offline.pem ubuntu@123.45.67.89

On the client machine, unpack the cdk-shrinkwrap tarball:

tar -xf canonical-kubernetes-1.8.1-stable-amd64-2017-10-26-08-17-24.tar.gz

In the unpacked cdk-shrinkwrap directory, you'll find a deploy.sh script. Edit this file. Change:

juju add-machine -n <something>

To this:

juju add-space cdkoffline 172.32.0.0/24
juju add-machine -n <something> --constraints spaces=cdkoffline

Finally, run the deployment script:

./deploy.sh

If all went well, you should have a working CDK installation inside a completely offline environment.

Clone this wiki locally