[#130102001] Bootstrap bosh and concourse

[saliceti]

What

Story: Create custom release concourse

We need a new concourse that will be used to build bosh releases. It was decided to take the opportunity and deploy this concourse with bosh instead of bosh-init. The main reasons are:

• bosh-init can handle only 1 VM. bosh can handle complex deployments and we may need it to scale concourse
• bosh can cache compiled artifacts
• bosh manages the state automatically
• reduces the size of the CF pipeline so this should improve our development time

This is now a generic concourse that we will use to build bosh releases, but eventually paas-cf may be migrated to use it for its bootstrap. We created a new repository that uses a lot of the paas-cf code so most is already known. The main changes are:

• bosh is now created in the bootstrap pipeline instead of the CF pipeline
• concourse is deployed with bosh instead of bosh-init
• since concourse lite is in a different VPC than bosh, bosh is now accessed via its external IP
• the security groups have changed slightly because we moved things around and the dependencies have changed
• update CPI to latest (v60). This was required for the auto-assign feature introduced in v55. The pipeline point to the bosh-init container tagged for the new version.
• changed IAM policies (See PR in aws-account-wide-terraform)

We have made other smaller changes:

• improve naming
• parallelise and merge pipeline jobs
• improve management of bosh_fqdn variable
• other minor things

How to review

• Apply changes from the PR in aws-account-wide-terraform
• Start concourse lite:

DEPLOY_ENV=testabcd make dev bootstrap

• Connect to localhost:8080 and start create pipeline
• In the end you should be able to access concourse at: https://concourse.testabcd.dev.cloudpipeline.digital
• Login to concourse, upload pipelines
• Destroy the environment using the destroy pipeline
• Destroy concourse-lite:

DEPLOY_ENV=testabcd make dev bootstrap-destroy

• Set up the local environment with python, ruby, go and check that make test passes successfully

Create the build CI

• Make sure the previous concourse lite is destroyed
• Start concourse lite for CI:

DEPLOY_ENV=build make ci bootstrap

• Connect to localhost:8080 and start create pipeline
• In the end you should be able to access concourse at: https://concourse.build.ci.cloudpipeline.digital
• Destroy concourse-lite:

DEPLOY_ENV=build make ci bootstrap-destroy

Who can review

Anyone but @henrytk or myself

[keymon]

I'm in general happy with the PR, waiting to test it.

[keymon]

@saliceti can you hook this with travis?

Shall we include the make merge_pr workflow here?

[saliceti]

Travis enabled, thanks.

merge_pr is used to signed merge commits in paas-cf because we validate the commits in the pipeline. Here the pipeline is triggered manually and we don't validate the commits, so I'm not sure we need merge_pr.

[keymon]

This won't work:

$ git diff
diff --git a/concourse/pipelines/create.yml b/concourse/pipelines/create.yml
index 38d5385..3df7f97 100644
--- a/concourse/pipelines/create.yml
+++ b/concourse/pipelines/create.yml
@@ -284,7 +284,7 @@ jobs:
               - |
                 if  [ -z "$(tar -tvzf existing-bosh-CA/bosh-CA.tar.gz)" ] ; then
                   certstrap init --passphrase "" --common-name bosh-CA
-                  ( cd out && tar -cvzf generated-bosh-CA/bosh-CA.tar.gz bosh-CA.* )
+                  ( cd out && tar -cvzf ../generated-bosh-CA/bosh-CA.tar.gz bosh-CA.* )
                 else
                   echo "The CA cert already exists, skipping generation..."
                   cp existing-bosh-CA/bosh-CA.tar.gz generated-bosh-CA/bosh-CA.tar.gz

[keymon]

Looks good to me, waiting for https://github.gds/government-paas/aws-account-wide-terraform/pull/58 to merge

[henrytk]

After discussion with @saliceti and @bleach it was decided we would rename the buckets to have a prefix of gds-paas, not paas. This will be consistent with the prefixes used for buckets used by ELBs. Policies will assert on suffix only for clarity.

I have changed the state bucket to gds-paas-ENV-bootstrap and blobstore buckets to gds-paas-ENV-bosh-blobstore. The changes have been squashed into the commits that introduced the files (concourse/scripts/pipelines.sh and terraform/bosh/blobstore.tf).

[keymon]

Just to keep record of it, I got this error while deploying in concourse-terraform the first time:

1 error(s) occurred:

* aws_elb.concourse: [WARN] Error creating ELB Listener with SSL Cert, retrying: CertificateNotFound: Server Certificate not found for the key: arn:aws:iam::595665891067:server-certificate/bector-concourse-20160926150825601060871plr
    status code: 400, request id: 32e1133a-83fb-11e6-9178-4fbcd6bf1553

Rerunning solved the issue.

I guess it takes some time to get the Server Certificate ready to be used. It is fine, as rerunning solves the issue and it is one-off.

[saliceti]

We had this in the past, I hope it's not a regression in Terraform:

• AWS: Certificate not found even though it was just created hashicorp/terraform#2499
• Creating ELB with certificate fails hashicorp/terraform#3275

This is not directly related to this PR.

[keymon]

I merge this after https://github.gds/government-paas/aws-account-wide-terraform/pull/58 has been merged.