This directory contains tooling for building base images for use as nodes in Kubernetes Clusters. Packer is used for building these images.
- Building Wardroom Images
- An AWS account
- The AWS CLI installed and configured
- A Google Cloud account
- The gcloud CLI installed and configured
- A precreated service account json file
The following variables can be overriden when building images using the -var option when calling packer build:
| Variable | Default | Description |
|---|---|---|
| build_version | unset | A unique build version for the image |
| kubernetes_version | 1.9.5-00 | Kubernetes Version to install |
| kubernetes_cni_version | 0.6.0-00 | CNI Version to install |
For exmaple, to build all images for use with Kubernetes 1.8.9 for build version 1:
packer build -var kubernetes_version=1.8.9-00 -var build_version=1There are additional variables that may be set that affect the behavior of specific builds or packer post-processors. packer inspect packer.json will list all available variables and their default values.
If packer build is run without specifying which images to build, then it will attempt to build all configured images. packer inspect packer.json will list the configured builders. The --only option can be specified when running packer build to limit the images built.
For example, to build only the AWS Ubuntu image:
packer build -var build_version=`git rev-parse HEAD` --only=ami-ubuntu packer.jsonBuilding AWS images requires setting additional variables not set by default. The aws-us-east-1.json file is provided as an example.
To build both the Ubuntu and CentOS AWS AMIs:
packer build -var-file aws-us-east-1.json -var build_version=`git rev-parse HEAD` --only=ami-centos,ami-ubuntu packer.jsonThe Packer documentation for the Amazon AMI builder supplies a suggested set of minimum permissions. However, Wardroom has been successfully tested with the following IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:Describe*",
"ec2:TerminateInstances",
"ec2:StartInstances",
"ec2:CreateSnapshot",
"ec2:CreateImage",
"ec2:RunInstances",
"ec2:StopInstances",
"ec2:CreateKeyPair",
"ec2:DeleteKeyPair",
"ec2:CreateSecurityGroup",
"ec2:DeleteSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateTags",
"ec2:DeleteVolume"
],
"Resource": "*"
}
]
}Building Google Cloud images requires setting the GOOGLE_APPLICATION_CREDENTIALS environment variable and providing the IDs of the source images. For the latter, the gcp-source-images.json file is provided as an example.
To build only the Ubuntu Google Cloud Image:
export GOOGLE_APPLICATION_CREDENTIALS=<YOUR CREDENTIAL FILE>
packer build -var-file=gcp-source-images.json -var build_version=`git rev-parse HEAD` -var project_id=<your-project-id-here> -only gcp-ubuntu packer.jsonThe account used by Wardroom (as specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable) must have the following permissions in order for Wardroom to function as expected:
compute.disks.create
compute.disks.delete
compute.disks.useReadOnly
compute.images.create
compute.images.delete
compute.images.get
compute.instances.create
compute.instances.delete
compute.instances.get
compute.instances.setMetadata
compute.instances.setServiceAccount
compute.instances.start
compute.instances.stop
compute.machineTypes.get
compute.subnetworks.use
compute.subnetworks.useExternalIp
compute.zones.get
Building Oracle Cloud Infrastructure (OCI) images requires a correct configuration for the Oracle CLI as outlined in the "CLI Configuration Information" section of this page, althoug the Oracle CLI does not need to be installed (Packer will use the values in the configuration file).
You will also need the following pieces of information:
- The Oracle Cloud ID (OCID) of the compartment where the build VM will be instantiated (you can use the root compartment, whose OCID is equal to the tenancy OCID)
- The name of the availability domain where the build VM will be instantiated
- The OCID for the subnet that corresponds to the availability domain where the build VM will be instantiated
To build an OCI image:
packer build -var-file oci-us-phoenix-1.json -var build_version=`git rev-parse HEAD` -var oci_availability_domain="<name of availability domain>" -var oci_compartment_ocid="<OCID of compartment>" -var oci_subnet_ocid="<OCID of subnet in specified availability domain>" -only=oci-ubuntu packer.jsonConnect remotely to an instance created from the image and run the Node Conformance tests using the following commands:
wget https://dl.k8s.io/$(< /etc/kubernetes_community_ami_version)/kubernetes-test.tar.gz
tar -zxvf kubernetes-test.tar.gz kubernetes/platforms/linux/amd64
cd kubernetes/platforms/linux/amd64
sudo ./ginkgo --nodes=8 --flakeAttempts=2 --focus="\[Conformance\]" --skip="\[Flaky\]|\[Serial\]|\[sig-network\]|Container Lifecycle Hook" ./e2e_node.test -- --k8s-bin-dir=/usr/binThere is a helper script to aid in seeding built AMI's to all other AWS regions. This script can be installed from the root of this repository by running python3 setup.py install.
wardroom aws copy-ami -r <SOURCE_REGION> <SOURCE_AMI>Unlike AWS, Google Cloud Images are not limited to specific regions, so no further steps are needed to use the create images.
-
Build the base image
packer build -var-file aws-us-east-1.json -var build_version=`git rev-parse HEAD` --only=ami-ubuntu packer.json -
Run Node Conformance against the built image
-
Deploy the image using copy-ami
-
Update the Quick Start to use the new images
You'll need to download the credential file after creating your account. Make sure you don't commit it, it contains secrets.
If you want to use a service account for use with Wardroom, you'll also need to grant the service account the ServiceAccountUser role in order for Wardroom to function properly.
You'll also need to make note of the "project ID" you wish to run the container in. It's a string, and you can find it at the top of the Google Cloud Console, or with gcloud projects list.