Support for concurrent nodepool CRUD operations

[danielvegamyhre]

The purpose of these changes is to implement support in the beta provider for concurrent node pool CRUD operations on a single cluster.

The GA provider should be unchanged (the global mutex store changes described below are technically included in the GA provider but the GA provider behavior is unchanged - however, I am happy to make the global mutex store changes specific to the beta provider if that is preferred).

The changes to the beta provider include:

• Updating the global mutex store to use sync.RWMutex instead of sync.Mutex and adding the necessary methods to the MutexKV struct to support acquiring shared/read locks.
• Removing the polling for cluster "ready" status, since with support for concurrent operations on the same cluster we no longer need to wait for the cluster to have no operations running on it before proceeding.
• For NP CRUD operations, instead of acquiring an exclusive/write lock on the cluster, we acquire a read/shared lock on the cluster and an exclusive/write lock on the node pool. This ensures cluster-wide operations (e.g. UpdateCluster) still will block NP level operations, but NP level operations on different NPs won't block each other. A NP-level mutex uses the cluster hash + node pool name to guarantee lock key uniqueness.
• Add retry logic to NP CRUD operations to retry while it receives an "incompatible operation" error (which has the FAILED_PRECONDITION canonical code), to safely retry concurrent operations blocked by a lock conflict with another operation.

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

container: Added support for concurrent node pool mutations on a cluster. Previously, node pool mutations were restricted to run synchronously clientside. NOTE: While this feature is supported in Terraform from this release onwards, only a limited number of GCP projects will support this behavior initially. The provider will automatically process mutations concurrently as the feature rolls out generally.

Reviewer Notes

• Ran the set of 33 TestAccContainerNodePool acceptance tests with the beta provider and they passed, although TestAccContainerNodePool_withWorkloadIdentityConfig seems flaky (only passed when I ran it individually). So it seems to be fully backward compatible.
• I did manual testing using my own *.tf files to create/delete multiple NPs concurrently, and confirmed the concurrency works.

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @rileykarson, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

---

Your assigned reviewer will help review your code by:

• Ensuring it's backwards compatible, covers common error cases, etc.
• Summarizing the change into a user-facing changelog note.
• Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

---

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 58 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 107 insertions(+), 55 deletions(-))
TF Validator: Diff ( 4 files changed, 59 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2210
Passed tests 1934
Skipped tests: 239
Failed tests: 37

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerCluster_withBinaryAuthorizationEnabledBoolLegacy|TestAccContainerCluster_withBinaryAuthorizationEvaluationModeClassic|TestAccContainerCluster_withBinaryAuthorizationEvaluationModeAutopilot|TestAccContainerCluster_withReleaseChannelEnabled|TestAccContainerCluster_withBinaryAuthorizationEnabledBool|TestAccContainerCluster_withAutoscalingProfile|TestAccContainerCluster_nodeAutoprovisioningDefaultsMinCpuPlatform|TestAccContainerCluster_withILBSubsetting|TestAccContainerCluster_withAuthenticatorGroupsConfig|TestAccContainerNodePool_withWorkloadIdentityConfig|TestAccContainerCluster_withMonitoringConfig|TestAccContainerCluster_withIdentityServiceConfig|TestAccContainerCluster_withAddons|TestAccContainerCluster_withFilteredNotificationConfig|TestAccContainerCluster_withNodePoolCIA|TestAccContainerCluster_withNodePoolResize|TestAccContainerCluster_withNodePoolAutoscaling|TestAccContainerCluster_withNodePoolUpdateVersion|TestAccContainerCluster_withShieldedNodes|TestAccContainerCluster_withResourceUsageExportConfig|TestAccContainerCluster_withCostManagementConfig|TestAccContainerCluster_withExternalIpsConfig|TestAccContainerCluster_nodeAutoprovisioning|TestAccContainerCluster_withNodeConfig|TestAccContainerCluster_deleteExclusionWindow|TestAccContainerCluster_withNodePoolDefaults|TestAccContainerCluster_updateVersion|TestAccContainerCluster_withIntraNodeVisibility|TestAccContainerCluster_updateMaintenanceExclusionOptions|TestAccContainerCluster_deleteMaintenanceExclusionOptions|TestAccContainerCluster_withMaintenanceExclusionWindow|TestAccContainerCluster_withRecurringMaintenanceWindow|TestAccContainerCluster_regionalWithNodeLocations|TestAccContainerCluster_withMaintenanceWindow|TestAccContainerCluster_withTelemetryEnabled|TestAccContainerCluster_withReleaseChannelEnabledDefaultVersion

[modular-magician]

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccContainerCluster_withBinaryAuthorizationEnabledBoolLegacy[Debug log]
TestAccContainerCluster_withBinaryAuthorizationEnabledBool[Debug log]
TestAccContainerCluster_withIdentityServiceConfig[Debug log]
TestAccContainerCluster_withFilteredNotificationConfig[Debug log]
TestAccContainerCluster_withNodePoolAutoscaling[Debug log]
TestAccContainerCluster_withShieldedNodes[Debug log]
TestAccContainerCluster_withResourceUsageExportConfig[Debug log]
TestAccContainerCluster_withCostManagementConfig[Debug log]
TestAccContainerCluster_withExternalIpsConfig[Debug log]
TestAccContainerCluster_withNodePoolDefaults[Debug log]
TestAccContainerCluster_updateMaintenanceExclusionOptions[Debug log]
TestAccContainerCluster_withMaintenanceExclusionWindow[Debug log]
TestAccContainerCluster_withRecurringMaintenanceWindow[Debug log]
TestAccContainerCluster_withMaintenanceWindow[Debug log]
TestAccContainerCluster_withTelemetryEnabled[Debug log]

Tests failed during RECORDING mode:
TestAccContainerCluster_withILBSubsetting[Error message] [Debug log]
TestAccContainerCluster_withMonitoringConfig[Error message] [Debug log]
TestAccContainerCluster_withAddons[Error message] [Debug log]
TestAccContainerCluster_withNodePoolResize[Error message] [Debug log]
TestAccContainerCluster_withNodePoolUpdateVersion[Error message] [Debug log]
TestAccContainerCluster_updateVersion[Error message] [Debug log]
TestAccContainerCluster_regionalWithNodeLocations[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[danielvegamyhre]

The 7 TestAccContainerCluster_* VCR tests that failed appear unrelated to my changes, for a couple reasons:

• I did not modify resource_container_cluster.go at all.
• 5/7 test errors are simply due to insufficient GCP quota.
• 2/7 tests (TestAccContainerCluster_regionalWithNodeLocations, TestAccContainerCluster_withNodePoolResize) in which the create cluster operation completed but not all nodes were healthy, which indicates an issue on the GKE side, not in the beta provider code.

[danielvegamyhre]

Note: I reran the failed TestAccContainerCluster_* acceptance tests locally and they passed.

[rileykarson]

I've made a pass of this, following up w/ internal doc next.

If one doesn't exist already, can you add a test that makes simultaneous updates to node pools? I'd like to have one kicking around so we can view the logs- both for this review + if we end up needing to debug in the future.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 162 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 4 files changed, 211 insertions(+), 55 deletions(-))
TF Validator: Diff ( 4 files changed, 59 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 0
Passed tests 0
Skipped tests: 0
Failed tests: 0

Errors occurred during REPLAYING mode. Please fix them to complete your PR
View the build log

[danielvegamyhre]

I added a test for concurrent node pool create/update/delete operations, and updated the PR to include changes in both the GA + Beta providers. When I generate the providers and build them locally, it is successful, and the test passes.

However, the CI build system runs into the following error: /go/pkg/mod/github.com/modular-magician/terraform-provider-google@v1.20.1-0.20221121230641-c2402eef8376/google/resource_container_node_pool.go:1095:26: undefined: lockKey

When I do a search for lockKey in resource_container_node_pool.go.erb (the version of the file in this PR), I see this variable does not exist in the file (it is the old variable name for the cluster level lock that I have removed from the node pool file). I am confused how this build error is occurring, especially when I cannot reproduce it locally. @rileykarson any idea what might be occurring here?

[rileykarson]

Still waiting on a response from someone about fixing the tfv-head issue, I'll kick the robot as well and see if anything changes. Minor comment about the test configs inline.

Other than our CI acting up, I think the outstanding steps are:

• Resolving whether we want the new call for the mutex key
• Those minor test comments
• Running the test in CI & seeing the retries in the logs
• Running the test locally (or grabbing logs from you- sharing internally is fine)

[rileykarson]

/gcbrun

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 213 insertions(+), 67 deletions(-))
Terraform Beta: Diff ( 4 files changed, 215 insertions(+), 71 deletions(-))
TF Validator: Diff ( 4 files changed, 59 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2303
Passed tests 2027
Skipped tests: 244
Failed tests: 32

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerNodePool_concurrent|TestAccContainerNodePool_withWorkloadIdentityConfig|TestAccContainerCluster_withBinaryAuthorizationEvaluationModeAutopilot|TestAccContainerCluster_withAutoscalingProfile|TestAccContainerCluster_withNodePoolResize|TestAccContainerCluster_withNodePoolUpdateVersion|TestAccContainerCluster_withReleaseChannelEnabled|TestAccContainerCluster_withEnablePrivateEndpointToggle|TestAccContainerCluster_nodeAutoprovisioningDefaultsMinCpuPlatform|TestAccContainerCluster_withNodePoolDefaults|TestAccContainerCluster_withILBSubsetting|TestAccContainerCluster_withAuthenticatorGroupsConfig|TestAccContainerCluster_withAddons|TestAccContainerCluster_withMonitoringConfig|TestAccContainerCluster_withLoggingVariantUpdates|TestAccContainerCluster_updateVersion|TestAccContainerCluster_withIntraNodeVisibility|TestAccContainerCluster_withLoggingConfig|TestAccContainerCluster_nodeAutoprovisioning|TestAccContainerCluster_withNotificationConfig|TestAccContainerCluster_regionalWithNodeLocations|TestAccContainerCluster_deleteExclusionWindow|TestAccContainerCluster_deleteMaintenanceExclusionOptions|TestAccContainerCluster_withGcpPublicCidrsAccessEnabledToggle|TestAccContainerCluster_withReleaseChannelEnabledDefaultVersion|TestAccContainerCluster_withNodePoolCIA|TestAccContainerCluster_withMasterAuthorizedNetworksConfig|TestAccContainerCluster_autoprovisioningDefaultsManagement|TestAccContainerCluster_nodeAutoprovisioningDefaultsDiskSizeGb|TestAccContainerCluster_withBinaryAuthorizationEvaluationModeClassic|TestAccLoggingBucketConfigProject_cmekSettings

[modular-magician]

Tests passed during RECORDING mode:
TestAccContainerNodePool_withWorkloadIdentityConfig[Debug log]
TestAccContainerCluster_withAutoscalingProfile[Debug log]
TestAccContainerCluster_withNodePoolResize[Debug log]
TestAccContainerCluster_withEnablePrivateEndpointToggle[Debug log]
TestAccContainerCluster_nodeAutoprovisioningDefaultsMinCpuPlatform[Debug log]
TestAccContainerCluster_withNodePoolDefaults[Debug log]
TestAccContainerCluster_withILBSubsetting[Debug log]
TestAccContainerCluster_withAddons[Debug log]
TestAccContainerCluster_updateVersion[Debug log]
TestAccContainerCluster_withIntraNodeVisibility[Debug log]
TestAccContainerCluster_withNotificationConfig[Debug log]
TestAccContainerCluster_regionalWithNodeLocations[Debug log]
TestAccContainerCluster_deleteExclusionWindow[Debug log]
TestAccContainerCluster_deleteMaintenanceExclusionOptions[Debug log]
TestAccContainerCluster_withGcpPublicCidrsAccessEnabledToggle[Debug log]
TestAccContainerCluster_withReleaseChannelEnabledDefaultVersion[Debug log]
TestAccContainerCluster_withNodePoolCIA[Debug log]
TestAccContainerCluster_withMasterAuthorizedNetworksConfig[Debug log]
TestAccContainerCluster_autoprovisioningDefaultsManagement[Debug log]
TestAccContainerCluster_nodeAutoprovisioningDefaultsDiskSizeGb[Debug log]
TestAccContainerCluster_withBinaryAuthorizationEvaluationModeClassic[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]

Tests failed during RECORDING mode:
TestAccContainerNodePool_concurrent[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[modular-magician]

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccContainerCluster_withEnablePrivateEndpointToggle[Debug log]
TestAccContainerCluster_withCostManagementConfig[Debug log]
TestAccContainerCluster_withExternalIpsConfig[Debug log]
TestAccContainerCluster_withBinaryAuthorizationEvaluationModeClassic[Debug log]
TestAccContainerCluster_withBinaryAuthorizationEnabledBool[Debug log]
TestAccContainerCluster_withAutoscalingProfile[Debug log]
TestAccContainerCluster_autoprovisioningDefaultsManagement[Debug log]
TestAccContainerCluster_nodeAutoprovisioningDefaultsMinCpuPlatform[Debug log]
TestAccContainerNodePool_withNetworkConfig[Debug log]
TestAccContainerCluster_withMaintenanceExclusionWindow[Debug log]
TestAccContainerCluster_withMaintenanceWindow[Debug log]
TestAccContainerCluster_withNodePoolCIA[Debug log]
TestAccContainerCluster_withTelemetryEnabled[Debug log]
TestAccContainerCluster_withFilteredNotificationConfig[Debug log]

Tests failed during RECORDING mode:
TestAccContainerNodePool_concurrent[Error message] [Debug log]
TestAccContainerCluster_nodeAutoprovisioningDefaultsDiskSizeGb[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[rileykarson]

TestAccContainerNodePool_concurrent is failing, but it's because you found a bug introduced recently rather than an issue with your PR. I filed hashicorp/terraform-provider-google#13157 to get that fixed.

TestAccContainerCluster_nodeAutoprovisioningDefaultsDiskSizeGb seems like a flake.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 183 insertions(+), 68 deletions(-))
Terraform Beta: Diff ( 4 files changed, 185 insertions(+), 72 deletions(-))
TF Validator: Diff ( 4 files changed, 39 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2310
Passed tests 2031
Skipped tests: 244
Failed tests: 35

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccContainerNodePool_withReservationAffinitySpecific|TestAccContainerNodePool_withReservationAffinity|TestAccContainerNodePool_withKubeletConfig|TestAccContainerNodePool_withWorkloadIdentityConfig|TestAccContainerNodePool_withSandboxConfig|TestAccContainerCluster_withBinaryAuthorizationEnabledBoolLegacy|TestAccContainerCluster_withResourceUsageExportConfig|TestAccContainerCluster_withNodePoolDefaults|TestAccContainerCluster_updateVersion|TestAccContainerCluster_withILBSubsetting|TestAccContainerCluster_withIdentityServiceConfig|TestAccContainerCluster_withAddons|TestAccContainerCluster_withNotificationConfig|TestAccContainerCluster_withIntraNodeVisibility|TestAccContainerCluster_withShieldedNodes|TestAccContainerCluster_regionalWithNodeLocations|TestAccContainerCluster_nodeAutoprovisioningDefaultsDiskSizeGb|TestAccContainerCluster_withGcpPublicCidrsAccessEnabledToggle|TestAccContainerCluster_withReleaseChannelEnabledDefaultVersion|TestAccContainerCluster_withMasterAuthorizedNetworksConfig|TestAccContainerCluster_deleteExclusionWindow|TestAccContainerCluster_deleteMaintenanceExclusionOptions|TestAccContainerCluster_updateMaintenanceExclusionOptions|TestAccContainerCluster_withRecurringMaintenanceWindow|TestAccContainerCluster_withNodePoolResize|TestAccContainerCluster_withNodePoolAutoscaling|TestAccContainerNodePool_012_ConfigModeAttr|TestAccContainerNodePool_concurrent|TestAccContainerNodePool_EmptyGuestAccelerator|TestAccContainerNodePool_version|TestAccContainerNodePool_withEnablePrivateNodesToggle|TestAccContainerNodePool_withLinuxNodeConfig|TestAccContainerNodePool_resize|TestAccContainerNodePool_withUpgradeSettings|TestAccLoggingBucketConfigProject_cmekSettings

[modular-magician]

Tests passed during RECORDING mode:
TestAccContainerNodePool_withReservationAffinity[Debug log]
TestAccContainerNodePool_withKubeletConfig[Debug log]
TestAccContainerNodePool_withWorkloadIdentityConfig[Debug log]
TestAccContainerNodePool_withSandboxConfig[Debug log]
TestAccContainerCluster_withBinaryAuthorizationEnabledBoolLegacy[Debug log]
TestAccContainerCluster_withResourceUsageExportConfig[Debug log]
TestAccContainerCluster_withIdentityServiceConfig[Debug log]
TestAccContainerCluster_withAddons[Debug log]
TestAccContainerCluster_withIntraNodeVisibility[Debug log]
TestAccContainerCluster_regionalWithNodeLocations[Debug log]
TestAccContainerCluster_nodeAutoprovisioningDefaultsDiskSizeGb[Debug log]
TestAccContainerCluster_withReleaseChannelEnabledDefaultVersion[Debug log]
TestAccContainerCluster_withMasterAuthorizedNetworksConfig[Debug log]
TestAccContainerCluster_deleteExclusionWindow[Debug log]
TestAccContainerCluster_deleteMaintenanceExclusionOptions[Debug log]
TestAccContainerCluster_updateMaintenanceExclusionOptions[Debug log]
TestAccContainerCluster_withRecurringMaintenanceWindow[Debug log]
TestAccContainerCluster_withNodePoolResize[Debug log]
TestAccContainerNodePool_concurrent[Debug log]
TestAccContainerNodePool_EmptyGuestAccelerator[Debug log]
TestAccContainerNodePool_withEnablePrivateNodesToggle[Debug log]
TestAccContainerNodePool_resize[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]

Errors occurred during RECORDING mode. Please fix them to complete your PR
View the build log or the debug log for each test

[rileykarson]

Failure here's just noise- not really sure why we triggered a rerun of all those, but we hit a process limit when RECORDING. Rerun should resolve.

/gcbrun

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 183 insertions(+), 68 deletions(-))
Terraform Beta: Diff ( 4 files changed, 185 insertions(+), 72 deletions(-))
TF Validator: Diff ( 4 files changed, 39 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2310
Passed tests 2052
Skipped tests: 244
Failed tests: 14

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerNodePool_withLinuxNodeConfig|TestAccContainerNodePool_012_ConfigModeAttr|TestAccContainerNodePool_withReservationAffinitySpecific|TestAccContainerNodePool_version|TestAccContainerNodePool_withUpgradeSettings|TestAccContainerCluster_withNodePoolAutoscaling|TestAccContainerCluster_withGcpPublicCidrsAccessEnabledToggle|TestAccContainerCluster_withNodePoolDefaults|TestAccContainerCluster_withILBSubsetting|TestAccContainerCluster_updateVersion|TestAccContainerCluster_withNotificationConfig|TestAccContainerCluster_withShieldedNodes|TestAccLoggingBucketConfigProject_cmekSettings

[modular-magician]

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccContainerNodePool_withLinuxNodeConfig[Debug log]
TestAccContainerNodePool_012_ConfigModeAttr[Debug log]
TestAccContainerNodePool_withReservationAffinitySpecific[Debug log]
TestAccContainerNodePool_version[Debug log]
TestAccContainerNodePool_withUpgradeSettings[Debug log]
TestAccContainerCluster_withNodePoolAutoscaling[Debug log]
TestAccContainerCluster_withGcpPublicCidrsAccessEnabledToggle[Debug log]
TestAccContainerCluster_withNodePoolDefaults[Debug log]
TestAccContainerCluster_withILBSubsetting[Debug log]
TestAccContainerCluster_updateVersion[Debug log]
TestAccContainerCluster_withNotificationConfig[Debug log]
TestAccContainerCluster_withShieldedNodes[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]

All tests passed
View the build log or the debug log for each test

[rileykarson]

Code LGTM, and I've confirmed we're seeing the intended behaviour against prod by checking the debug logs of the recording. I'd like to see the logs for a successful run w/ parallel operations enabled myself, mostly for peace-of-mind, given we're shipping code that's gonna be picked up by a future behaviour of the API. I'll follow up with you offline about that.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 179 insertions(+), 68 deletions(-))
Terraform Beta: Diff ( 4 files changed, 181 insertions(+), 72 deletions(-))
TF Validator: Diff ( 4 files changed, 39 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2312
Passed tests 2065
Skipped tests: 244
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccFirebaserulesRelease_BasicRelease|TestAccContainerNodePool_concurrent

[modular-magician]

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]

Tests failed during RECORDING mode:
TestAccContainerNodePool_concurrent[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 177 insertions(+), 68 deletions(-))
Terraform Beta: Diff ( 4 files changed, 179 insertions(+), 72 deletions(-))
TF Validator: Diff ( 4 files changed, 39 insertions(+), 7 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2319
Passed tests 2072
Skipped tests: 244
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerNodePool_concurrent|TestAccLoggingBucketConfigProject_cmekSettings

[modular-magician]

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccContainerNodePool_concurrent[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]

All tests passed
View the build log or the debug log for each test