Failing test(s): gcp-sa-pubsub.iam.gserviceaccount.com CryptoKey Encrypter/Decrypter role

[melinath]

Failure rate: 100% since 2022-09-11

Impacted tests:

• TestAccCloudFunctions2Function_fullUpdate
• TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample
• TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample
• TestAccEventarcTrigger_channel

Nightly builds:

• https://ci-oss.hashicorp.engineering/buildConfiguration/GoogleCloud_ProviderGoogleCloudGoogleProject/348525?buildTab=tests&expandedTest=-7237469270685448170

Message:

Error: Error waiting to create function: Error waiting for Creating function: Error code 9, message: Creating trigger failed for projects/project-id/locations/us-central1/triggers/tf-test-gcf-function7paif6qej6-862718: generic::failed_precondition: Cloud Pub/Sub did not have the necessary permissions configured to support this operation. Please verify that the service account service-1234567@gcp-sa-pubsub.iam.gserviceaccount.com was granted the Cloud KMS CryptoKey Encrypter/Decrypter role for the project containing the CryptoKey resource projects/project-id/locations/us-central1/keyRings/tftest-shared-keyring-1/cryptoKeys/tf-bootstrap-key2/grants/c481b13e954e859d.

This was likely an API change.

Affected Resource(s)

• google_cloudfunctions2_function

[AarshDhokai]

• TestAccCloudFunctions2Function_fullUpdate
  Google Cloud - 86% failure
  Google Cloud Beta - 100% failure

• TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample
  Google Cloud - 86% failure
  Google Cloud Beta - 100% failure

• TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample
  Google Cloud - 85% failure
  Google Cloud Beta - 100% failure

[AarshDhokai]

b/261395490

[SarahFrench]

I think these failing tests might be influenced by this fix : GoogleCloudPlatform/magic-modules#7208
However the problem fixed by that PR wasn't around when this issue was opened - so there may be other factors at play

[roaks3]

Two notes:

• This is related to TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample failed #13461, which was opened because we were seeing test failures in PRs as well
• Another issue we've seen is a race condition, where multiple tests may use the same permission at once, but because it is a resource that is created and then destroyed, the permission can be destroyed by one test while it is still being used by another. We have not figured out a good solution to this so far.

[ScottSuarez]

We should manage these permissions in pantheon ui instead.. Unless we either implement a skip delete version of these resources or have an initialization config to manage all permissions prior to tests running (iam bootstrap)

https://github.com/GoogleCloudPlatform/magic-modules/pull/7293/files

[ScottSuarez]

I removed these permissions from the corresponding configurations and have set their values in pantheon UI. Will continue to monitor this test but it should pass from here on out.

[trodge]

We should manage these permissions in pantheon ui instead.. Unless we either implement a skip delete version of these resources or have an initialization config to manage all permissions prior to tests running (iam bootstrap)

https://github.com/GoogleCloudPlatform/magic-modules/pull/7293/files

I have limited IAM bootstrap working here:
https://github.com/GoogleCloudPlatform/magic-modules/pull/7376/files

@ScottSuarez, could you let me know the roles you granted so I can add a bootstrap call to these tests' test_vars_overrides?

Edit- Ah, I see them here: https://github.com/GoogleCloudPlatform/magic-modules/pull/7293/files

[roaks3]

Heads up that I think the solutions above have gotten these tests to mostly pass, but we are still seeing some failures. I've opened GoogleCloudPlatform/magic-modules#7479 which I expect to resolve.

• TestAccCloudFunctions2Function_fullUpdate
• TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample
• TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample

=== RUN   TestAccCloudFunctions2Function_fullUpdate
=== PAUSE TestAccCloudFunctions2Function_fullUpdate
=== CONT  TestAccCloudFunctions2Function_fullUpdate
    provider_test.go:311: Step 1/2 error: Error running apply: exit status 1
        
        Error: Error waiting to create function: Error waiting for Creating function: Error code 9, message: Creating trigger failed for projects/ci-test-project-188019/locations/us-central1/triggers/tf-test-gcf-function24g4tj29o8-370438: generic::failed_precondition: Cloud Pub/Sub did not have the necessary permissions configured to support this operation. Please verify that the service account service-1067888929963@gcp-sa-pubsub.iam.gserviceaccount.com was granted the Cloud KMS CryptoKey Encrypter/Decrypter role for the project containing the CryptoKey resource projects/ci-test-project-188019/locations/us-central1/keyRings/tftest-shared-keyring-1/cryptoKeys/tf-bootstrap-eventarc-google-channel-config-key2/grants/c481b13e954e859d.
        
          with google_cloudfunctions2_function.function,
          on terraform_plugin_test.tf line 56, in resource "google_cloudfunctions2_function" "function":
          56: resource "google_cloudfunctions2_function" "function" {
        
--- FAIL: TestAccCloudFunctions2Function_fullUpdate (123.39s)
FAIL

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.