Fix an issue with resuming a failed container cluster creation #7121

trodge · 2023-01-12T22:45:39Z

fixes b/228111747

fixes hashicorp/terraform-provider-google#11431

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

container: fixed an issue with resuming failed cluster creation

modular-magician · 2023-01-12T23:01:13Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 226 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 226 insertions(+), 3 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2023-01-12T23:47:42Z

Tests analytics

Total tests: 2408
Passed tests 2155
Skipped tests: 251
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerCluster_failedCreation

modular-magician · 2023-01-12T23:48:12Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]

Tests failed during RECORDING mode:
TestAccContainerCluster_failedCreation[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

trodge · 2023-01-13T00:14:26Z

/gcbrun

modular-magician · 2023-01-13T00:29:56Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 226 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 226 insertions(+), 3 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2023-01-13T01:18:58Z

Tests analytics

Total tests: 2409
Passed tests 2155
Skipped tests: 251
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerCluster_failedCreation|TestAccRegionInstanceGroupManager_stateful

modular-magician · 2023-01-13T01:37:18Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccContainerCluster_failedCreation[Debug log]

Tests failed during RECORDING mode:
TestAccRegionInstanceGroupManager_stateful[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

melinath · 2023-01-20T22:02:44Z

reassigning to a random team member.

roaks3

Would this also fix hashicorp/terraform-provider-google#11431?

I had a few questions but overall solution looks good.

roaks3 · 2023-01-23T16:52:29Z

mmv1/third_party/terraform/resources/resource_container_cluster.go.erb

@@ -2247,11 +2249,43 @@ func resourceContainerClusterRead(d *schema.ResourceData, meta interface{}) erro
 		}
 		waitErr := containerOperationWait(config, op, project, location, "resuming GKE cluster", userAgent, d.Timeout(schema.TimeoutRead))
 		if waitErr != nil {
+			// Check if the create operation failed because Terraform was prematurely terminated. If it was we can persist the


For anyone else reading, it looks like this block is an exact copy from what is done in the *Create function above.

roaks3 · 2023-01-23T16:55:39Z

mmv1/third_party/terraform/resources/resource_container_cluster.go.erb

+			select {
+			case <-config.context.Done():
+				log.Printf("[DEBUG] Persisting %s so this operation can be resumed \n", op.Name)
+				if err := d.Set("operation", op.Name); err != nil {


Could this result in an infinite loop situation? It seems like a user might be able to get stuck in some situation where the operation never terminates properly and they can't get a terraform plan to succeed. IMO, one retry probably makes sense, and I think that's why the operation field is unset immediately above.

It looks like the test still passes without this part.

roaks3 · 2023-01-23T18:50:26Z

mmv1/third_party/terraform/tests/resource_container_cluster_test.go.erb

+		Steps: []resource.TestStep{
+			{
+				Config:      testAccContainerCluster_failedCreation(clusterName, project.ProjectId),
+				ExpectError: regexp.MustCompile(".*timeout while waiting for state to become 'DONE'.*"),


Do you need the .*s for these regexes? My understanding was that matching a subsection was valid, not necessarily the whole string.

I have removed the .*s.

roaks3 · 2023-01-23T18:50:34Z

mmv1/third_party/terraform/utils/bootstrap_utils_test.go

@@ -358,6 +359,136 @@ func BootstrapServicePerimeterProjects(t *testing.T, desiredProjects int) []*clo
 	return projects
 }

+func removeContainerServiceAgentRoleFromContainerEngineRobot(t *testing.T, project *cloudresourcemanager.Project) {
+	config := BootstrapConfig(t)


Since you are using BootstrapConfig, I don't believe the requests being made in this function will get recorded or replayed by the VCR. That means the requests will be made for every replay (ie. multiple per PR). You should be able to test with multiple GCB runs in this PR.

If the requests aren't recorded, it might be worth checking the policy first and only calling "Set" if it needs to be updated. That would also make it more like our other bootstrap functions, so you could potentially use that pattern more explicitly, ie. BootstrapProjectWithInvalidContainerEngineRobot(). This assumes that you never need to reset this project to the original state.

I have made this function only set the policy when changes are required.

…d a test

modular-magician · 2023-01-25T17:29:18Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 239 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 239 insertions(+), 3 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2023-01-25T18:13:07Z

Tests analytics

Total tests: 2429
Passed tests 2173
Skipped tests: 254
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccContainerCluster_failedCreation|TestAccRegionInstanceGroupManager_stateful

modular-magician · 2023-01-25T18:28:10Z

Tests passed during RECORDING mode:
TestAccContainerCluster_failedCreation[Debug log]

Tests failed during RECORDING mode:
TestAccRegionInstanceGroupManager_stateful[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

roaks3

I would still add "Fixes hashicorp/terraform-provider-google#11431" to the PR description if it will be fixed to get the auto-close behavior, but otherwise LGTM

rileykarson · 2023-01-27T23:42:19Z

@trodge: I've been seeing this pop up on VCR runs through the day. Is it possible that the bootstrap functions mean we need to exclude this from VCR?

…eCloudPlatform#7121) * Fix an issue with resuming a failed container cluster creation and add a test * Do not persist operation when resuming during read.

trodge requested a review from melinath January 13, 2023 01:46

melinath requested review from a team and roaks3 and removed request for melinath and a team January 20, 2023 22:02

roaks3 reviewed Jan 23, 2023

View reviewed changes

trodge added 2 commits January 24, 2023 19:48

Fix an issue with resuming a failed container cluster creation and ad…

39577fe

…d a test

Do not persist operation when resuming during read.

0429be8

trodge force-pushed the container-cluster branch from 96c308a to 0429be8 Compare January 25, 2023 17:13

trodge requested a review from roaks3 January 25, 2023 21:26

roaks3 approved these changes Jan 26, 2023

View reviewed changes

trodge merged commit 944567c into GoogleCloudPlatform:main Jan 26, 2023

trodge deleted the container-cluster branch January 26, 2023 17:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix an issue with resuming a failed container cluster creation #7121

Fix an issue with resuming a failed container cluster creation #7121

trodge commented Jan 12, 2023 •

edited

Loading

modular-magician commented Jan 12, 2023

modular-magician commented Jan 12, 2023

modular-magician commented Jan 12, 2023

trodge commented Jan 13, 2023

modular-magician commented Jan 13, 2023

modular-magician commented Jan 13, 2023

modular-magician commented Jan 13, 2023

melinath commented Jan 20, 2023

roaks3 left a comment

roaks3 Jan 23, 2023

roaks3 Jan 23, 2023

trodge Jan 25, 2023

roaks3 Jan 23, 2023

trodge Jan 25, 2023

roaks3 Jan 23, 2023

trodge Jan 25, 2023

modular-magician commented Jan 25, 2023

modular-magician commented Jan 25, 2023

modular-magician commented Jan 25, 2023

roaks3 left a comment

rileykarson commented Jan 27, 2023

Fix an issue with resuming a failed container cluster creation #7121

Fix an issue with resuming a failed container cluster creation #7121

Conversation

trodge commented Jan 12, 2023 • edited Loading

modular-magician commented Jan 12, 2023

Diff report

modular-magician commented Jan 12, 2023

Tests analytics

Action taken

modular-magician commented Jan 12, 2023

trodge commented Jan 13, 2023

modular-magician commented Jan 13, 2023

Diff report

modular-magician commented Jan 13, 2023

Tests analytics

Action taken

modular-magician commented Jan 13, 2023

melinath commented Jan 20, 2023

roaks3 left a comment

Choose a reason for hiding this comment

roaks3 Jan 23, 2023

Choose a reason for hiding this comment

roaks3 Jan 23, 2023

Choose a reason for hiding this comment

trodge Jan 25, 2023

Choose a reason for hiding this comment

roaks3 Jan 23, 2023

Choose a reason for hiding this comment

trodge Jan 25, 2023

Choose a reason for hiding this comment

roaks3 Jan 23, 2023

Choose a reason for hiding this comment

trodge Jan 25, 2023

Choose a reason for hiding this comment

modular-magician commented Jan 25, 2023

Diff report

modular-magician commented Jan 25, 2023

Tests analytics

Action taken

modular-magician commented Jan 25, 2023

roaks3 left a comment

Choose a reason for hiding this comment

rileykarson commented Jan 27, 2023

trodge commented Jan 12, 2023 •

edited

Loading