Add Terraform GKE and Helm modules tests with Terratest #1483

aLekSer · 2020-04-20T16:39:17Z

Useful to check if Terraform configuration is valid.

What type of PR is this?
/kind feature

What this PR does / Why we need it:

Which issue(s) this PR fixes:

Closes #1227.

Special notes for your reviewer:
There is a need to make local run of the test in a way that root go.mod and go.sum does not get affected.
This Test framework also good at verification of different variables used by helm templates
https://github.com/gruntwork-io/terratest/blob/master/test/helm_basic_example_template_test.go and https://github.com/gruntwork-io/terratest-helm-testing-example .

agones-bot · 2020-04-20T16:56:40Z

Build Succeeded 👏

Build Id: 303de1e1-8da6-474e-93e7-2e248626d179

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.6.0-5e7e9b2
image: gcr.io/agones-images/agones-sdk:1.6.0-5e7e9b2
image: gcr.io/agones-images/agones-ping:1.6.0-5e7e9b2
Linux C++ SDK (build): agonessdk-1.6.0-5e7e9b2-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.6.0-5e7e9b2.zip

A preview of the website (the last 30 builds are retained):

https://5e7e9b2-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-5e7e9b2

aLekSer · 2020-04-20T18:18:31Z

Saving go test file from this PR in IDE, changes root go.mod file.
This is a side-effect of the IDE running go build on file save. I am using VS Code.
VS Code Issue name is Do not automatically update go.mod and go.sum files.
Using Vim does not have this side-effects.
It seems that changing go.mod persist only when gke_test.go file is opened at VS Code. So I assume this is a minor issue.
There is an option to change file extension from .go to avoid this, used this approach previously in ../test/sdk/restapi/http-api-test.go.nolint due to swagger-codegen issues.

agones-bot · 2020-04-20T19:06:55Z

Build Succeeded 👏

Build Id: f198327a-df67-4e8d-be5a-87dee0184fba

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.6.0-2d75daa
image: gcr.io/agones-images/agones-sdk:1.6.0-2d75daa
image: gcr.io/agones-images/agones-ping:1.6.0-2d75daa
Linux C++ SDK (build): agonessdk-1.6.0-2d75daa-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.6.0-2d75daa.zip

A preview of the website (the last 30 builds are retained):

https://2d75daa-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-2d75daa

aLekSer · 2020-04-20T19:32:33Z

Well it seems reasonable to split out terraform config files into a separate repo. This would speed up using all ./examples/terraform-submodules/ and would not clone the whole Agones repo. Terratest could also be moved there.

aLekSer · 2020-04-20T19:59:07Z

This Terratest output (head and tail):

/.gocache:/root/.cache/go-build -e "KUBECONFIG=/root/.kube/config" -e "GO111MODULE=on" -w /go/src/agones.dev/agones  agones-build:e501e27b72 bash -c 'cd /go/src/agones.dev/agones/test/terraform && go test \                                                                                                                                                                                                         
        -timeout 1h -project agones-alexander '                                                                                                                                                            
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z retry.go:72: terraform [init -upgrade=false]
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z command.go:87: Running command terraform with args [init -upgrade=false]                                                                                
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z command.go:158: Initializing modules...                                                                                                                 
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z command.go:158:                                                                                                                                         
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z command.go:158: Initializing the backend...                                                                                                             
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z command.go:158:                                                                                                                                         
TestTerraformGKEInstallConfig 2020-04-20T19:30:33Z command.go:158: Initializing provider plugins...
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158:
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158: Terraform has been successfully initialized!                                                                                            
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158:
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158: You may now begin working with Terraform. Try running "terraform plan" to see                                                           
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158: any changes that are required for your infrastructure. All Terraform commands                                                           
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158: should now work.
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:87: Running command terraform with args [get -update]                                                                                        
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158: - gke_cluster in ../../../install/terraform/modules/gke                                                                                 
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:158: - helm_agones in ../../../install/terraform/modules/helm                                                                                
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z retry.go:72: terraform [apply -input=false -auto-approve -var name=terratest-cluster -var values_file= -var project=agones-alexander -lock=false]       
TestTerraformGKEInstallConfig 2020-04-20T19:30:35Z command.go:87: Running command terraform with args [apply -input=false -auto-approve -var name=terratest-cluster -var values_file= -var project=agones-al
exander -lock=false]
TestTerraformGKEInstallConfig 2020-04-20T19:30:45Z command.go:158: module.gke_cluster.data.google_client_config.default: Refreshing state...                                                               
TestTerraformGKEInstallConfig 2020-04-20T19:30:49Z command.go:158: module.helm_agones.null_resource.helm_init: Creating...    
..
TestTerraformGKEInstallConfig 2020-04-20T19:39:45Z command.go:158:                                                                                                                                         
TestTerraformGKEInstallConfig 2020-04-20T19:39:45Z command.go:158: Destroy complete! Resources: 6 destroyed.                                                                                               
PASS                                                                                                                                                                                                       
ok      github.com/agones/agones/test/terraform 553.243s                                                                                                                                                   
by-it00060:build alexander.apalikov$

agones-bot · 2020-04-20T20:09:18Z

Build Succeeded 👏

Build Id: 67e0ad44-ab32-4805-8a14-8ab1acd06768

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.6.0-74e7506
image: gcr.io/agones-images/agones-sdk:1.6.0-74e7506
image: gcr.io/agones-images/agones-ping:1.6.0-74e7506
Linux C++ SDK (build): agonessdk-1.6.0-74e7506-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.6.0-74e7506.zip

A preview of the website (the last 30 builds are retained):

https://74e7506-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-74e7506

agones-bot · 2020-04-20T20:13:23Z

Build Succeeded 👏

Build Id: e70e5ce0-c911-4af7-b94a-aae5114b3bd0

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.6.0-7b317e6
image: gcr.io/agones-images/agones-sdk:1.6.0-7b317e6
image: gcr.io/agones-images/agones-ping:1.6.0-7b317e6
Linux C++ SDK (build): agonessdk-1.6.0-7b317e6-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.6.0-7b317e6.zip

A preview of the website (the last 30 builds are retained):

https://7b317e6-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-7b317e6

markmandel

This is really, really cool.

Do you envisage we should be doing this in our CI build pipeline?

(As per splitting out, maybe let's start a new ticket, work out a design of how we want to do that? Definitely not against it. Easy enough to configure a new Cloud Build setup, or maybe prow, also need to consider backward compatibility.)

markmandel · 2020-04-23T21:42:02Z

test/terraform/gke_test.go

+func destroy(t *testing.T, options *terraform.Options) {
+	options.Targets = []string{"module.helm_agones.helm_release.agones"}
+	terraform.Destroy(t, options)
+	time.Sleep(30 * time.Second)


Curious - is there a way we could poll to see if this is complete, rather than waiting for a period of time? Waiting is probably pretty fragile.

Gentle bump on this review.

Returning back to this as well

Added the wait of kubectl services -n agones-system count would be 0 using terratest K8S module. Some updates to development .tf file was needed - creating kubeconfig file.

There is no other options to poll the result or at least nothing else come to my mind. But previous approach build a "bridge" for creating more complex tests.

aLekSer · 2020-05-12T14:18:17Z

go.mod was automatically changed by IDE after adding this test. Trying the idea to add same dependencies as in root go.mod into test/terraform one.

diff --git a/go.mod b/go.mod
index 0e80fa99..268cd898 100644
--- a/go.mod
+++ b/go.mod
@@ -3,50 +3,42 @@ module agones.dev/agones
 go 1.13

 require (
-       cloud.google.com/go v0.34.0
+       cloud.google.com/go v0.51.0
        contrib.go.opencensus.io/exporter/prometheus v0.1.0
        contrib.go.opencensus.io/exporter/stackdriver v0.8.0
        fortio.org/fortio v1.3.1
+       git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999 // indirect
        github.com/ahmetb/gen-crd-api-reference-docs v0.1.1
-       github.com/aws/aws-sdk-go v1.16.20 // indirect
        github.com/evanphx/json-patch v4.5.0+incompatible // indirect
        github.com/fsnotify/fsnotify v1.4.7
-       github.com/go-openapi/spec v0.19.0
-       github.com/gogo/protobuf v1.2.1 // indirect
+       github.com/go-openapi/spec v0.19.3

Seems to be go list related issue:
microsoft/vscode-go#2730 (comment)

aLekSer · 2020-05-13T08:53:24Z

Trying the following solution to avoid updating root go.mod, add build tag// +build ignore: bazel-contrib/rules_go#2262 (comment)
Actually no luck with it.

agones-bot · 2020-05-13T14:07:34Z

Build Failed 😱

Build Id: e6ab66b7-591e-40dc-ae8d-2843676f3c82

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2020-05-13T15:30:33Z

Build Failed 😱

Build Id: 1b828e64-96be-4fd0-972a-1164ce6fa673

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

aLekSer · 2020-05-13T15:34:34Z

make terraform-test run was successful:

--- PASS: TestTerraformGKEInstallConfig (627.42s)
    gke_test.go:33: Using project <projectName>
PASS
ok      terraform       627.463s

This PR is ready to review now.

markmandel

Sorry, missed this on last review

markmandel · 2020-08-26T16:29:52Z

build/includes/terraform.mk

@@ -84,3 +86,18 @@ gcloud-terraform-destroy-cluster: GCP_PROJECT ?= $(current_project)
 gcloud-terraform-destroy-cluster:
 	$(DOCKER_RUN) bash -c 'cd $(mount_path)/build/terraform/gke && terraform destroy -var project=$(GCP_PROJECT) -auto-approve'

+terraform-test: $(ensure-build-image)
+ifndef GCP_PROJECT
+	$(eval GCP_PROJECT=$(shell sh -c "gcloud config get-value project 2> /dev/null"))


This shouldn't be necessary if we put current_project back in.

markmandel · 2020-08-26T16:29:53Z

build/includes/terraform.mk

@@ -15,17 +15,19 @@
 # The GKE development cluster name
 GCP_TF_CLUSTER_NAME ?= agones-tf-cluster

-# the current project
-current_project := $(shell $(DOCKER_RUN) bash -c "gcloud config get-value project 2> /dev/null")


Just realised we're deleting current_project - this will break stuff 😢

Oops, sorry for that, returning it back.

Applying PR comments. Add gitignore.

Update terraform.Destroy first target pass.

Running test with -project equals empty string is forbiden and nonsense.

This would get the necessary project name from docker run.

agones-bot · 2020-08-31T12:50:35Z

Build Succeeded 👏

Build Id: e5dc3a1d-4dde-41d9-912f-0309a5dae3bf

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.9.0-34bcd8c
image: gcr.io/agones-images/agones-sdk:1.9.0-34bcd8c
image: gcr.io/agones-images/agones-ping:1.9.0-34bcd8c
Linux C++ SDK (build): agonessdk-1.9.0-34bcd8c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.9.0-34bcd8c.zip

A preview of the website (the last 30 builds are retained):

https://34bcd8c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.9.0-34bcd8c

agones-bot · 2020-08-31T13:05:46Z

Build Succeeded 👏

Build Id: 53305c71-862c-4ed3-811d-cde99c077dcc

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.9.0-960800b
image: gcr.io/agones-images/agones-sdk:1.9.0-960800b
image: gcr.io/agones-images/agones-ping:1.9.0-960800b
Linux C++ SDK (build): agonessdk-1.9.0-960800b-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.9.0-960800b.zip

A preview of the website (the last 30 builds are retained):

https://960800b-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.9.0-960800b

aLekSer · 2020-09-03T07:43:09Z

Fixed all comments, this PR is good to go now.

agones-bot · 2020-09-03T08:00:32Z

Build Succeeded 👏

Build Id: 70b0f1f0-5776-4095-aed8-98c738f1be08

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.9.0-ffc85ad
image: gcr.io/agones-images/agones-sdk:1.9.0-ffc85ad
image: gcr.io/agones-images/agones-ping:1.9.0-ffc85ad
Linux C++ SDK (build): agonessdk-1.9.0-ffc85ad-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.9.0-ffc85ad.zip

A preview of the website (the last 30 builds are retained):

https://ffc85ad-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.9.0-ffc85ad

google-oss-robot · 2020-09-03T20:11:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aLekSer, markmandel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [aLekSer,markmandel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-robot · 2020-09-03T20:13:39Z

New changes are detected. LGTM label has been removed.

agones-bot · 2020-09-03T20:32:47Z

Build Succeeded 👏

Build Id: 33fe73e2-d74e-4149-bd59-a30ced240005

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.9.0-3c37f4d
image: gcr.io/agones-images/agones-sdk:1.9.0-3c37f4d
image: gcr.io/agones-images/agones-ping:1.9.0-3c37f4d
Linux C++ SDK (build): agonessdk-1.9.0-3c37f4d-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.9.0-3c37f4d.zip

A preview of the website (the last 30 builds are retained):

https://3c37f4d-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/1483/head:pr_1483 && git checkout pr_1483
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.9.0-3c37f4d

…es#1483) * Enhance tests, use kubectl module to verify svc Co-authored-by: Mark Mandel <markmandel@google.com>

google-oss-robot added the do-not-merge/work-in-progress label Apr 20, 2020

google-oss-robot requested review from EricFortin and pooneh-m April 20, 2020 16:39

google-oss-robot added the size/XL label Apr 20, 2020

aLekSer changed the title ~~Add Terraform and Helm tests with Terratest~~ Add Terraform GKE and Helm modules tests with Terratest Apr 20, 2020

aLekSer force-pushed the terraform/adding-tests branch 2 times, most recently from 823f136 to 2d75daa Compare April 20, 2020 18:49

aLekSer force-pushed the terraform/adding-tests branch 2 times, most recently from 74e7506 to 7b317e6 Compare April 20, 2020 19:53

aLekSer marked this pull request as ready for review April 20, 2020 19:57

google-oss-robot removed the do-not-merge/work-in-progress label Apr 20, 2020

markmandel added area/operations Installation, updating, metrics etc area/tests Unit tests, e2e tests, anything to make sure things don't break labels Apr 23, 2020

markmandel reviewed Apr 23, 2020

View reviewed changes

aLekSer force-pushed the terraform/adding-tests branch from 7b317e6 to b6dd0e5 Compare May 13, 2020 13:50

aLekSer mentioned this pull request May 13, 2020

CI build pipeline enhancement: add Terratests to check example yaml, helm templates and terraform configs #1539

Closed

3 tasks

aLekSer force-pushed the terraform/adding-tests branch 2 times, most recently from d9cb72e to fba1b05 Compare May 13, 2020 15:29

markmandel reviewed Aug 26, 2020

View reviewed changes

aLekSer added 7 commits August 31, 2020 15:26

Enhance tests, use kubectl module to verify svc

500e58e

Fix terraform-clean command

c456b25

Applying PR comments. Add gitignore.

Add terraform-test make target

bf9a228

Fixed file to use .go ext. Split terraform configs

a193176

Updating Readme file, add clean command

4456a6c

Update terraform.Destroy first target pass.

Fixing the timers, adding comments

b6624b2

Propagate GCP_PROJECT env variable further to test

e8c9206

Running test with -project equals empty string is forbiden and nonsense.

aLekSer force-pushed the terraform/adding-tests branch 2 times, most recently from f62841f to 34bcd8c Compare August 31, 2020 12:32

This comment has been minimized.

Sign in to view

Adding current_project back

960800b

This would get the necessary project name from docker run.

aLekSer force-pushed the terraform/adding-tests branch from 34bcd8c to 960800b Compare August 31, 2020 12:46

Merge branch 'master' into terraform/adding-tests

ffc85ad

markmandel approved these changes Sep 3, 2020

View reviewed changes

google-oss-robot assigned markmandel Sep 3, 2020

google-oss-robot added the lgtm label Sep 3, 2020

google-oss-robot added the approved label Sep 3, 2020

Merge branch 'master' into terraform/adding-tests

3c37f4d

google-oss-robot removed the lgtm label Sep 3, 2020

markmandel merged commit bbb8119 into googleforgames:master Sep 3, 2020

markmandel added this to the 1.9.0 milestone Sep 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Terraform GKE and Helm modules tests with Terratest #1483

Add Terraform GKE and Helm modules tests with Terratest #1483

aLekSer commented Apr 20, 2020 •

edited

Loading

agones-bot commented Apr 20, 2020

aLekSer commented Apr 20, 2020 •

edited

Loading

agones-bot commented Apr 20, 2020

aLekSer commented Apr 20, 2020

aLekSer commented Apr 20, 2020

agones-bot commented Apr 20, 2020

agones-bot commented Apr 20, 2020

markmandel left a comment

markmandel Apr 23, 2020

markmandel May 11, 2020

aLekSer May 12, 2020

aLekSer May 13, 2020

aLekSer May 13, 2020

aLekSer commented May 12, 2020 •

edited

Loading

aLekSer commented May 13, 2020 •

edited

Loading

agones-bot commented May 13, 2020

agones-bot commented May 13, 2020

aLekSer commented May 13, 2020

markmandel left a comment

markmandel Aug 26, 2020

markmandel Aug 26, 2020

aLekSer Aug 31, 2020

This comment has been minimized.

agones-bot commented Aug 31, 2020

agones-bot commented Aug 31, 2020

aLekSer commented Sep 3, 2020

agones-bot commented Sep 3, 2020

google-oss-robot commented Sep 3, 2020

google-oss-robot commented Sep 3, 2020

agones-bot commented Sep 3, 2020

Add Terraform GKE and Helm modules tests with Terratest #1483

Add Terraform GKE and Helm modules tests with Terratest #1483

Conversation

aLekSer commented Apr 20, 2020 • edited Loading

agones-bot commented Apr 20, 2020

aLekSer commented Apr 20, 2020 • edited Loading

agones-bot commented Apr 20, 2020

aLekSer commented Apr 20, 2020

aLekSer commented Apr 20, 2020

agones-bot commented Apr 20, 2020

agones-bot commented Apr 20, 2020

markmandel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aLekSer commented May 12, 2020 • edited Loading

aLekSer commented May 13, 2020 • edited Loading

agones-bot commented May 13, 2020

agones-bot commented May 13, 2020

aLekSer commented May 13, 2020

markmandel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

agones-bot commented Aug 31, 2020

agones-bot commented Aug 31, 2020

aLekSer commented Sep 3, 2020

agones-bot commented Sep 3, 2020

google-oss-robot commented Sep 3, 2020

google-oss-robot commented Sep 3, 2020

agones-bot commented Sep 3, 2020

aLekSer commented Apr 20, 2020 •

edited

Loading

aLekSer commented Apr 20, 2020 •

edited

Loading

aLekSer commented May 12, 2020 •

edited

Loading

aLekSer commented May 13, 2020 •

edited

Loading