Use GCS as the Terraform state backend #2938

gongmax · 2023-01-27T23:22:55Z

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation

/kind feature

/kind hotfix

What this PR does / Why we need it:
Use GCS as the backend of the e2e test cluster Terraform state, so the maintainers can have a central store of the state. This will also benefit the management of the additional e2e test clusters we will introduced in near future.

As a side work, the PR also remove the consul from the Autopilot e2e test cluster.

Which issue(s) this PR fixes:

Closes #

Special notes for your reviewer:
Since we will have the central authority of the e2e test clusters state, we should discourage users to create e2e test cluster for their local dev purpose. Users should be able to just create one test-cluster and perform all the dev on it including the e2e test. Instruction changes will be in following PRs.

markmandel · 2023-01-27T23:29:37Z

build/terraform/e2e/module.tf

@@ -28,6 +28,10 @@ terraform {
      version = "~> 2.3"
    }
  }
+  backend "gcs" {
+   bucket  = "agones-e2e-infra-bucket-tfstate"


Given that a bucket name has to be glopally unique - this should probably be dynamic (use the project name as a prefix?) so it can be run against development projects without failing.

Yeah, that's doable. But I have doubt whether it makes sense to create an e2e test cluster for development. I was thinking we should push user to just create one test-cluster and use it for the development purpose even the e2e test.

With it the way it is, you can't test this Terraform script, except to test it in production.

Yeah, I was expecting the backend part would be relatively stable since it's only a thing for our maintainers who have access to the infra and won't have many change (if any). But it's still a fair point. I will make it not hardcoded.

Honestly just something like {project}-tfstate makes sense to me, that's what I was thinking of doing.

agones-bot · 2023-01-28T00:31:45Z

Build Succeeded 👏

Build Id: d206037d-40ed-4487-9e91-c0151e62a7a6

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-22ec2a7-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-22ec2a7-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-22ec2a7-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-22ec2a7-amd64
Linux C++ SDK (build): agonessdk-1.30.0-22ec2a7-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-22ec2a7-amd64.zip

A preview of the website (the last 30 builds are retained):

https://22ec2a7-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2938/head:pr_2938 && git checkout pr_2938
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-22ec2a7-amd64

agones-bot · 2023-01-28T01:52:56Z

Build Failed 😱

Build Id: c7584429-ea28-4b2b-8cb6-21cb9841c898

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-01-28T04:28:17Z

Build Succeeded 👏

Build Id: caab8303-f722-408e-8289-57d1b01e28c0

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-1b12efb-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-1b12efb-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-1b12efb-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-1b12efb-amd64
Linux C++ SDK (build): agonessdk-1.30.0-1b12efb-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-1b12efb-amd64.zip

A preview of the website (the last 30 builds are retained):

https://1b12efb-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2938/head:pr_2938 && git checkout pr_2938
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-1b12efb-amd64

zmerlynn · 2023-01-30T22:48:44Z

build/includes/google-cloud.mk

 	$(DOCKER_RUN) bash -c 'cd $(mount_path)/build/terraform/e2e && terraform destroy -var project=$(GCP_PROJECT) -auto-approve'

 # Creates a gcloud cluster for prow
 gcloud-prow-build-cluster: GCP_PROJECT ?= $(shell $(current_project))
 gcloud-prow-build-cluster: $(ensure-build-image)
 gcloud-prow-build-cluster:
-	$(MAKE) terraform-init DIRECTORY=prow
+	$(MAKE) terraform-init BUCKET=$(GCP_PROJECT)-prow-infra-bucket-tfstate PREFIX=terraform/state DIRECTORY=prow


maybe not strictly necessary here, but given that we are at an inflection point of changing the e2e infrastructure and I don't think any of us understands if the Prow configuration is even working, should we just instead turn down Prow? I am afraid that if we continue as is we'll give someone a false sense of "maybe it works" when I feel like it has reached the point of bitrot.

I have no objections to removing the prow cluster.

I have no objections to removing the prow cluster.

Who are you, and what have you done with the real @roberthbailey ? 😁

agones-bot · 2023-01-31T17:59:10Z

Build Failed 😱

Build Id: e15b8cf6-cc64-4685-b046-c484abfa4b46

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-01-31T18:38:00Z

Build Succeeded 👏

Build Id: 5b5272ad-2b07-41fa-8daa-171d1ae1a159

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-7d9aae3-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-7d9aae3-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-7d9aae3-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-7d9aae3-amd64
Linux C++ SDK (build): agonessdk-1.30.0-7d9aae3-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-7d9aae3-amd64.zip

A preview of the website (the last 30 builds are retained):

https://7d9aae3-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2938/head:pr_2938 && git checkout pr_2938
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-7d9aae3-amd64

zmerlynn · 2023-01-31T18:46:40Z

build/terraform/e2e/module.tf

@@ -14,6 +14,7 @@


 // Run:
+//  terraform init -backend-config="bucket=<YOUR_GCP_ProjectID>-e2e-infra-bucket-tfstate" -backend-config="prefix=terraform/state"


out of curiousity, what happens if you don't provide these variables? does terraform error out in an obvious way, or does it default to something bad?

(I would like if it failed brightly, vs just continued without GCS, if that's something easy enough to do.)

If starting with a clean state, i.e. there is no tfstate file for this module in your dev environment, terraform init without those variables will succeed but the local root configuration directory for this module will be used as backend. But the following terraform apply will fail because the state doesn't match with the existing resources, unless the related resources (clusters, firewalls) are all deleted, which is basically starting from a brand new state.

Good point. I think that's good enough, yeah.

zmerlynn

Left a minor question. It's non-blocking to merge.

google-oss-prow · 2023-01-31T19:01:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: gongmax, zmerlynn
Once this PR has been reviewed and has the lgtm label, please assign markmandel for approval by writing /assign @markmandel in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow · 2023-01-31T19:02:13Z

New changes are detected. LGTM label has been removed.

agones-bot · 2023-01-31T19:05:46Z

Build Succeeded 👏

Build Id: 604fc25e-0783-4bfe-805c-b0a109dc840b

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-7d9aae3-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-7d9aae3-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-7d9aae3-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-7d9aae3-amd64
Linux C++ SDK (build): agonessdk-1.30.0-7d9aae3-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-7d9aae3-amd64.zip

A preview of the website (the last 30 builds are retained):

https://7d9aae3-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2938/head:pr_2938 && git checkout pr_2938
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-7d9aae3-amd64

agones-bot · 2023-01-31T19:39:23Z

Build Succeeded 👏

Build Id: 522b5ed9-8b27-4788-af9d-bb80713737c0

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.30.0-1f36d83-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.30.0-1f36d83-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.30.0-1f36d83-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.30.0-1f36d83-amd64
Linux C++ SDK (build): agonessdk-1.30.0-1f36d83-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.30.0-1f36d83-amd64.zip

A preview of the website (the last 30 builds are retained):

https://1f36d83-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/2938/head:pr_2938 && git checkout pr_2938
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.30.0-1f36d83-amd64

* Use GCS to as the Terraform state backend * fix format * Make gcs bucket dynamic; Use gcs as terraform backend for prow cluster * remove prow cluster

google-oss-prow bot added do-not-merge/work-in-progress size/M labels Jan 27, 2023

google-oss-prow bot requested review from aLekSer and cyriltovena January 27, 2023 23:23

markmandel reviewed Jan 27, 2023

View reviewed changes

google-oss-prow bot added size/L and removed size/M labels Jan 28, 2023

gongmax marked this pull request as ready for review January 28, 2023 05:06

google-oss-prow bot removed the do-not-merge/work-in-progress label Jan 28, 2023

google-oss-prow bot requested a review from markmandel January 28, 2023 05:07

gongmax changed the title ~~Use GCS to as the Terraform state backend~~ Use GCS as the Terraform state backend Jan 28, 2023

zmerlynn reviewed Jan 30, 2023

View reviewed changes

gongmax requested review from zmerlynn and removed request for cyriltovena and aLekSer January 31, 2023 18:43

zmerlynn reviewed Jan 31, 2023

View reviewed changes

zmerlynn approved these changes Jan 31, 2023

View reviewed changes

google-oss-prow bot assigned zmerlynn Jan 31, 2023

google-oss-prow bot added the lgtm label Jan 31, 2023

gongmax added 4 commits January 31, 2023 11:02

Use GCS to as the Terraform state backend

2bfa0e3

fix format

decdda9

Make gcs bucket dynamic; Use gcs as terraform backend for prow cluster

717c06f

remove prow cluster

1f36d83

zmerlynn force-pushed the backend branch from 7d9aae3 to 1f36d83 Compare January 31, 2023 19:02

google-oss-prow bot removed the lgtm label Jan 31, 2023

gongmax merged commit 36997f2 into googleforgames:main Jan 31, 2023

Kalaiselvi84 added this to the 1.30.0 milestone Feb 28, 2023

Kalaiselvi84 added the kind/feature New features for Agones label Feb 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use GCS as the Terraform state backend #2938

Use GCS as the Terraform state backend #2938

gongmax commented Jan 27, 2023

markmandel Jan 27, 2023 •

edited

Loading

gongmax Jan 27, 2023

markmandel Jan 28, 2023

gongmax Jan 28, 2023

zmerlynn Jan 30, 2023

agones-bot commented Jan 28, 2023

agones-bot commented Jan 28, 2023

agones-bot commented Jan 28, 2023

zmerlynn Jan 30, 2023

roberthbailey Jan 30, 2023

markmandel Jan 31, 2023

agones-bot commented Jan 31, 2023

agones-bot commented Jan 31, 2023

zmerlynn Jan 31, 2023

zmerlynn Jan 31, 2023

gongmax Jan 31, 2023

zmerlynn Jan 31, 2023

zmerlynn left a comment

google-oss-prow bot commented Jan 31, 2023

google-oss-prow bot commented Jan 31, 2023

agones-bot commented Jan 31, 2023

agones-bot commented Jan 31, 2023

		@@ -14,6 +14,7 @@


		// Run:
		// terraform init -backend-config="bucket=<YOUR_GCP_ProjectID>-e2e-infra-bucket-tfstate" -backend-config="prefix=terraform/state"

Use GCS as the Terraform state backend #2938

Use GCS as the Terraform state backend #2938

Conversation

gongmax commented Jan 27, 2023

markmandel Jan 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agones-bot commented Jan 28, 2023

agones-bot commented Jan 28, 2023

agones-bot commented Jan 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agones-bot commented Jan 31, 2023

agones-bot commented Jan 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmerlynn left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Jan 31, 2023

google-oss-prow bot commented Jan 31, 2023

agones-bot commented Jan 31, 2023

agones-bot commented Jan 31, 2023

markmandel Jan 27, 2023 •

edited

Loading