Welcome to ci in a box!
From an empty GCP project to a dual-DC HA kubernetes CI/CD stack in about 6 minutes.
IMPORTANT I am focusing my efforts on a newer, better version of this. Please go to https://github.com/Stono/kube-gocd
It is a docker container, that deploys, builds and configures Kubernetes and GoCD on GKE (GCP) in a configuration that allows for CI/CD of docker applications. It's also a front runner in the how many times can i say docker in a minute competition.
Jokes aside, it is an open sourced slightly simplified version of the docker continuous integration and delivery setup I use on a daily basis. When I say CI/CD, I really do mean entirely docker based. That is the go agents, despite being docker containers themselves, build applications as containers, use Google Container Registry as an artifact store to promote them up the pipeline.
A picture says 1000 words, so this is a rather abstract example of the CI/CD flow this stack is designed to support:
Are you one of those people that spends the first few weeks of any new project setting up your infrastructure (ip's, firewalls, networking), Kubernetes, then installing your CI server (in my case, GoCD)? I am, and I was tired of it. I want to be able to kick off a docker/kubernetes/gcp project with the least amount of effort - and that's what this project is. I want to start working on the application as quickly as possible.
Essentially a command line interface for automating a bunch of low value work. It will:
- Create a GCP bucket to store your terraform state in
- Reserve three static IP's:
gocd,preprodandprod - Create a named network for your stack
- Deploy two subnetworks:
- preprod:
10.34.96.0/24 - prod:
10.34.97.0/24
- preprod:
- Configure firewall rules for web apps
- Deploy a preprod kubernetes cluster, HA'd across two zones, into the preprod subnet, with a container range of
10.37.64.0/19 - Deploy a prod kuberneters cluster in the same way, to the prod subnet, with a container range of
10.35.96.0/19 - Configure
slowandfaststorageclasses for PersistentVolumeClaims - Create a separate application namespace, and config maps for each namespace
- Provision some persistent storage for GoCD server to persist outside of kubernetes clusters
- Generate SSH and GPG keys for GoCD to interact with GitHub etc
- Deploy GoCD Master to your preprod kubernetes cluster
- Deploy 2x Special GCP tweaked GoCD Agents, preloaded with
kubectl,gcloud,terraformetc - Deploy Stono's Docker Nginx, which fronts your GoCD with a LetsEncrypt SSL certificate
- Automatically clean up orphaned firewall rules, disks etc, as terraform doesn't clean up pod resources if you delete a cluster
- Make you a cup of tea.
... Just kidding about the last one, it won't make you a brew. But with all this free time on your hands, you can totally make your own!
There are a few niggles I still need to sort out. Check them out here on the issues page.
Sure.
This repo itself runs inside a docker container, so that you don't need terraform, gcloud, gsuil and so on on your machine - how cool is that? So long as you have docker and docker-compose installed, clone this repository and type docker-compose run --rm ciinabox.
However, if you really want, you can run it on your host too. Just make sure you have all the components required. To run it on your host, run ./start. The script will validate you have all the required bits.
This project is designed to be reusable, so everything is done with environment variables. You can either create a .env file like the one below, or specify all of those environment variables in your docker-compose or docker run commands.
If you miss any variables, you will be prompted for them.
# Behaviour
NO_PROMPT=true # Dont prompt for confirmation of variables
NO_CONFIRM=true # Dont prompt to continue
# GCP Settings
GCP_PROJECT_NAME=your-gcp-project-name
TARGET_REGION=europe-west1
TARGET_ZONE_A=europe-west1-c
TARGET_ZONE_B=europe-west1-d
STACK_NAME=testing
STATE_BUCKET=$STACK_NAME-terraform
# Kubernetes stuff
PREPROD_CLUSTER_PASSWORD=testingpassword
PROD_CLUSTER_PASSWORD=testingpasswordprod
NETWORK_NAME=$STACK_NAME-poc-network
# Application Stuff
LETSENCRYPT_EMAIL=testing@test.com
GOCD_USERNAME=user-name
GOCD_PASSWORD=password
GOCD_AGENT_KEY=some-super-secure-agent-key
Pretty simple really, as suggested - run it in docker. The first time you run the container you will be prompted to login to the gcloud cli, just follow the instructions. Future runs will have your credentials saved via the volume mounts in docker-compose.yml.
$ docker-compose run --rm ciinabox
Checking tool dependencies...
+ kubectl
+ terraform
+ gcloud
+ gsutil
+ git
+ gpg2
+ curl
+ pdata
Checking tool versions...
+ kubectl (1.5.3 >= 1.5.3)
+ terraform (0.8.7 >= 0.8.7)
+ gcloud sdk (146.0.0 >= 146.0.0)
Checking environment configuration...
+ Lets Encrypt email address (testing@test.com):
+ This application stack name (testing):
+ GCS Bucket to store state in (testing-terraform):
+ GCP Private Network Name (testing-poc-network):
+ GCP Project Name (your-gcp-project-name):
+ The Kubernetes cluster password (testingpassword):
+ Username to login to GOCD with (test-user):
+ Password to login to GOCD with (password):
+ Secure key that agents connect with (testingtesting):
Setting cloud project...
Updated property [core/project].
+ Setup Complete!
Welcome to the all in one GCP Kubernetes deployment script.
Terraform commands:
- plan [module] Perform a terraform plan
- apply-plan [module] Apply a terraform plan
- destroy [module] [terraform options] Destroy one of these, (please be aware of dependencies)
Available modules: gocd modules networking preprod prod
Application commands:
- configure [preprod | prod] Configure an empty kubernetes cluster with basic config
- deploy [gocd-master | gocd-agents] Deploy either the go master or agents
Utility commands:
- bootstrap Does all of the above, on a new GCP project
- nuke Destroy everything in one devastating blow
- clean Cleans up orphaned firewall rules, disks etc
If you're on a fresh setup, just run docker-compose run --rm ciinabox bootstrap, wait a few minutes and then crack on building the stuff that matters. You'll get something like this:
$ docker-compose run --rm ciinabox bootstrap
Checking for cluster...
Planning the build...
Remote configuration updated
Remote state configured and pulled.
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.
google_compute_network.network: Refreshing state... (ID: testing-poc-network)
google_compute_address.preprod: Refreshing state... (ID: testing-preprod)
google_compute_address.gocd: Refreshing state... (ID: testing-gocd)
google_compute_address.prod: Refreshing state... (ID: testing-prod)
google_compute_firewall.web-ports: Refreshing state... (ID: testing-web-ports)
google_compute_firewall.standard-ports: Refreshing state... (ID: testing-standard-ports)
+ google_compute_address.gocd
address: "<computed>"
name: "testing-gocd"
region: "europe-west1"
self_link: "<computed>"
+ google_compute_address.preprod
address: "<computed>"
name: "testing-preprod"
region: "europe-west1"
self_link: "<computed>"
+ google_compute_address.prod
address: "<computed>"
name: "testing-prod"
region: "europe-west1"
self_link: "<computed>"
+ google_compute_firewall.standard-ports
allow.#: "2"
allow.1367131964.ports.#: "0"
allow.1367131964.protocol: "icmp"
allow.803338340.ports.#: "1"
allow.803338340.ports.0: "22"
allow.803338340.protocol: "tcp"
name: "testing-standard-ports"
network: "testing-poc-network"
project: "<computed>"
self_link: "<computed>"
source_ranges.#: "1"
source_ranges.1080289494: "0.0.0.0/0"
+ google_compute_firewall.web-ports
allow.#: "1"
allow.1250112605.ports.#: "2"
allow.1250112605.ports.0: "80"
allow.1250112605.ports.1: "443"
allow.1250112605.protocol: "tcp"
name: "testing-web-ports"
network: "testing-poc-network"
project: "<computed>"
self_link: "<computed>"
source_tags.#: "2"
source_tags.1936433573: "https-server"
source_tags.988335155: "http-server"
+ google_compute_network.network
gateway_ipv4: "<computed>"
name: "testing-poc-network"
self_link: "<computed>"
Plan: 6 to add, 0 to change, 0 to destroy.
Applying terraform plan...
Remote configuration updated
Remote state configured and pulled.
google_compute_address.preprod: Creating...
address: "" => "<computed>"
name: "" => "testing-preprod"
region: "" => "europe-west1"
self_link: "" => "<computed>"
google_compute_address.prod: Creating...
address: "" => "<computed>"
name: "" => "testing-prod"
region: "" => "europe-west1"
self_link: "" => "<computed>"
google_compute_network.network: Creating...
gateway_ipv4: "" => "<computed>"
name: "" => "testing-poc-network"
self_link: "" => "<computed>"
google_compute_address.gocd: Creating...
address: "" => "<computed>"
name: "" => "testing-gocd"
region: "" => "europe-west1"
self_link: "" => "<computed>"
google_compute_address.preprod: Creation complete
google_compute_address.gocd: Creation complete
google_compute_address.prod: Creation complete
google_compute_network.network: Creation complete
google_compute_firewall.web-ports: Creating...
allow.#: "" => "1"
allow.1250112605.ports.#: "" => "2"
allow.1250112605.ports.0: "" => "80"
allow.1250112605.ports.1: "" => "443"
allow.1250112605.protocol: "" => "tcp"
name: "" => "testing-web-ports"
network: "" => "testing-poc-network"
project: "" => "<computed>"
self_link: "" => "<computed>"
source_tags.#: "" => "2"
source_tags.1936433573: "" => "https-server"
source_tags.988335155: "" => "http-server"
google_compute_firewall.standard-ports: Creating...
allow.#: "" => "2"
allow.1367131964.ports.#: "" => "0"
allow.1367131964.protocol: "" => "icmp"
allow.803338340.ports.#: "" => "1"
allow.803338340.ports.0: "" => "22"
allow.803338340.protocol: "" => "tcp"
name: "" => "testing-standard-ports"
network: "" => "testing-poc-network"
project: "" => "<computed>"
self_link: "" => "<computed>"
source_ranges.#: "" => "1"
source_ranges.1080289494: "" => "0.0.0.0/0"
google_compute_firewall.web-ports: Creation complete
google_compute_firewall.standard-ports: Creation complete
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
Planning the build...
Remote configuration updated
Remote state configured and pulled.
Get: file:///mnt/git/stono/ci-in-a-box/terraform/modules/container
Get: file:///mnt/git/stono/ci-in-a-box/terraform/modules/cluster-subnet
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.
+ module.container.subnet.google_compute_subnetwork.subnet_europe
gateway_address: "<computed>"
ip_cidr_range: "10.34.96.0/24"
name: "testing-preprod-eu-west"
network: "testing-poc-network"
region: "europe-west1"
self_link: "<computed>"
+ module.container.google_container_cluster.cluster
additional_zones.#: "1"
additional_zones.0: "europe-west1-d"
cluster_ipv4_cidr: "10.37.64.0/19"
endpoint: "<computed>"
initial_node_count: "1"
instance_group_urls.#: "<computed>"
logging_service: "<computed>"
master_auth.#: "1"
master_auth.0.client_certificate: "<computed>"
master_auth.0.client_key: "<computed>"
master_auth.0.cluster_ca_certificate: "<computed>"
master_auth.0.password: "testingpassword"
master_auth.0.username: "admin"
monitoring_service: "none"
name: "testing-preprod"
network: "testing-poc-network"
node_config.#: "1"
node_config.0.disk_size_gb: "100"
node_config.0.machine_type: "n1-standard-2"
node_config.0.oauth_scopes.#: "8"
node_config.0.oauth_scopes.0: "https://www.googleapis.com/auth/cloud-platform"
node_config.0.oauth_scopes.1: "https://www.googleapis.com/auth/compute"
node_config.0.oauth_scopes.2: "https://www.googleapis.com/auth/devstorage.read_write"
node_config.0.oauth_scopes.3: "https://www.googleapis.com/auth/logging.write"
node_config.0.oauth_scopes.4: "https://www.googleapis.com/auth/servicecontrol"
node_config.0.oauth_scopes.5: "https://www.googleapis.com/auth/service.management"
node_config.0.oauth_scopes.6: "https://www.googleapis.com/auth/datastore"
node_config.0.oauth_scopes.7: "https://www.googleapis.com/auth/pubsub"
node_version: "1.5.3"
subnetwork: "testing-preprod-eu-west"
zone: "europe-west1-c"
Plan: 2 to add, 0 to change, 0 to destroy.
Applying terraform plan...
Remote configuration updated
Remote state configured and pulled.
module.container.subnet.google_compute_subnetwork.subnet_europe: Creating...
gateway_address: "" => "<computed>"
ip_cidr_range: "" => "10.34.96.0/24"
name: "" => "testing-preprod-eu-west"
network: "" => "testing-poc-network"
region: "" => "europe-west1"
self_link: "" => "<computed>"
module.container.subnet.google_compute_subnetwork.subnet_europe: Creation complete
module.container.google_container_cluster.cluster: Creating...
additional_zones.#: "" => "1"
additional_zones.0: "" => "europe-west1-d"
cluster_ipv4_cidr: "" => "10.37.64.0/19"
endpoint: "" => "<computed>"
initial_node_count: "" => "1"
instance_group_urls.#: "" => "<computed>"
logging_service: "" => "<computed>"
master_auth.#: "" => "1"
master_auth.0.client_certificate: "" => "<computed>"
master_auth.0.client_key: "" => "<computed>"
master_auth.0.cluster_ca_certificate: "" => "<computed>"
master_auth.0.password: "" => "testingpassword"
master_auth.0.username: "" => "admin"
monitoring_service: "" => "none"
name: "" => "testing-preprod"
network: "" => "testing-poc-network"
node_config.#: "" => "1"
node_config.0.disk_size_gb: "" => "100"
node_config.0.machine_type: "" => "n1-standard-2"
node_config.0.oauth_scopes.#: "" => "8"
node_config.0.oauth_scopes.0: "" => "https://www.googleapis.com/auth/cloud-platform"
node_config.0.oauth_scopes.1: "" => "https://www.googleapis.com/auth/compute"
node_config.0.oauth_scopes.2: "" => "https://www.googleapis.com/auth/devstorage.read_write"
node_config.0.oauth_scopes.3: "" => "https://www.googleapis.com/auth/logging.write"
node_config.0.oauth_scopes.4: "" => "https://www.googleapis.com/auth/servicecontrol"
node_config.0.oauth_scopes.5: "" => "https://www.googleapis.com/auth/service.management"
node_config.0.oauth_scopes.6: "" => "https://www.googleapis.com/auth/datastore"
node_config.0.oauth_scopes.7: "" => "https://www.googleapis.com/auth/pubsub"
node_version: "" => "1.5.3"
subnetwork: "" => "testing-preprod-eu-west"
zone: "" => "europe-west1-c"
module.container.google_container_cluster.cluster: Creation complete
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Configuring env...
Acquiring credentials for cluster testing-preprod
Fetching cluster endpoint and auth data.
kubeconfig entry generated for testing-preprod.
Applying common configuration: /mnt/git/stono/ci-in-a-box/configuration/common/default.configmap.yml
[i] Overriding namespace to: default
-> kubectl --namespace default apply -f ../tmp/default.configmap.yml
<- configmap "namespace-config" created
Deployment complete
Applying common configuration: /mnt/git/stono/ci-in-a-box/configuration/common/gce.storageclass.fast.yml
-> kubectl --namespace default apply -f ../tmp/gce.storageclass.fast.yml
<- storageclass "fast" created
Deployment complete
Applying common configuration: /mnt/git/stono/ci-in-a-box/configuration/common/application.namespace.yml
-> kubectl --namespace default apply -f ../tmp/application.namespace.yml
<- namespace "testing" created
Deployment complete
Applying common configuration: /mnt/git/stono/ci-in-a-box/configuration/common/gce.storageclass.slow.yml
-> kubectl --namespace default apply -f ../tmp/gce.storageclass.slow.yml
<- storageclass "slow" created
Deployment complete
Applying environment configuration: /mnt/git/stono/ci-in-a-box/configuration/preprod/namespace.configmap.yml
[i] Overriding namespace to: testing
-> kubectl --namespace testing apply -f ../tmp/namespace.configmap.yml
<- configmap "namespace-config" created
Deployment complete
Acquiring credentials for cluster testing-preprod
Fetching cluster endpoint and auth data.
kubeconfig entry generated for testing-preprod.
Deploying gocd secrets...
secret "gocd.users" created
secret "gocd.goagent-key" created
secret "gocd.ssh" created
Ensuring disks...
+ testing-gocd-master
+ testing-gocd-master-config
Deploying gocd master...
-> kubectl --namespace default apply -f ../tmp/master.pod.yml
<- deployment "gocd-master" created
Deployment complete
Deploying gocd master service...
-> kubectl --namespace default apply -f ../tmp/master.service.yml
<- service "gocd-master" created
Deployment complete
Deploying gocd NGINX...
-> kubectl --namespace default apply -f ../tmp/nginx.pod.yml
<- deployment "gocd-nginx" created
Deployment complete
Deploying gocd NGINX service with static ip: 146.148.27.251...
-> kubectl --namespace default apply -f ../tmp/nginx.service.yml
<- service "gocd-nginx" created
Deployment complete
Waiting for https://146.148.27.251/go to be available........
GoCD is online at: https://146.148.27.251/go!
Acquiring credentials for cluster testing-preprod
Fetching cluster endpoint and auth data.
kubeconfig entry generated for testing-preprod.
Ensuring GPG key...
agent.asc already exists
secret "goagent.gpg-key" created
Deploying gocd preprod agents...
[i] Overriding namespace to: default
-> kubectl --namespace default apply -f ../tmp/agent.pod.yml
<- deployment "gocd-agent-preprod" created
Deployment complete
Deploying gocd prod agents...
[i] Overriding namespace to: default
-> kubectl --namespace default apply -f ../tmp/agent.pod.yml
<- deployment "gocd-agent-prod" created
Deployment complete
GoCD agents deployed!
Bootstrap complete, have fun on GoCD at https://146.148.27.251/go
As mentioned, SSH and GPG keys are generated for the agents to git clone, git-crypt etc. You will find them in .tmp/.ssh and .tmp/.gnupg
There are a few more improvements to make:
- More variables, to control things like # of GCP machines
- Add Image Cleaner daemon
Apache-2.0. In summary, use it however you like but don't blame me if you break stuff!
A few people have shared the rather painful journey of discovery of the past few months when it comes to GCP, Kubernetes, GKE etc.