Failing test(s): TestAccCloudbuildv2Connection* (beta)

[SarahFrench]

Failure rates

• 100% since 2023-04-07

Impacted tests

• TestAccCloudbuildv2ConnectionIamMemberGenerated
• TestAccCloudbuildv2ConnectionIamBindingGenerated
• TestAccCloudbuildv2ConnectionIamPolicyGenerated
• TestAccCloudbuildv2Connection_GheConnection
• TestAccCloudbuildv2Connection_GhePrivConnection
• TestAccCloudbuildv2Connection_GithubConnection
• TestAccCloudbuildv2Repository_GheRepository
• TestAccCloudbuildv2Repository_GithubRepository
• TestAccCloudbuildv2Connection_GhePrivUpdateConnection
• TestAccCloudBuildTrigger_cloudbuildTriggerRepoExample
• TestAccCloudBuildTrigger_cloudbuildTriggerPubsubWithRepoExample
• TestAccCloudbuildv2Connection_GheCompleteConnection

Affected Resource(s)

• google_cloudbuildv2_connection (Beta)

Nightly build test history

• TestAccCloudbuildv2ConnectionIamMemberGenerated
• TestAccCloudbuildv2ConnectionIamBindingGenerated
• TestAccCloudbuildv2ConnectionIamPolicyGenerated
• TestAccCloudbuildv2Connection_GheCompleteConnection

Message(s)

Error about not having permissions to access a secret from a different project:

------- Stdout: -------
=== RUN   TestAccCloudbuildv2ConnectionIamPolicyGenerated
=== PAUSE TestAccCloudbuildv2ConnectionIamPolicyGenerated
=== CONT  TestAccCloudbuildv2ConnectionIamPolicyGenerated
vcr_utils.go:146: Step 1/2 error: Error running apply: exit status 1
Error: Error creating Connection: operation received error: error code "9", message: could not access secret "projects/PROJECT_ID/secrets/github-pat/versions/1" with service account "service-123456789@gcp-sa-cloudbuild.iam.gserviceaccount.com": generic::permission_denied: Permission 'secretmanager.versions.access' denied for resource 'projects/PROJECT_ID/secrets/github-pat/versions/1' (or it may not exist)., details: []
details: map[]
with google_cloudbuildv2_connection.my-connection,
on terraform_plugin_test.tf line 2, in resource "google_cloudbuildv2_connection" "my-connection":
2: resource "google_cloudbuildv2_connection" "my-connection" {
--- FAIL: TestAccCloudbuildv2ConnectionIamPolicyGenerated (4.69s)
FAIL

A different missing permission (Permission 'servicedirectory.services.resolve' denied on resource 'projects/gcb-terraform-creds/locations/us-central1/namespaces/myns/services/serv'.):

------- Stdout: -------
=== RUN   TestAccCloudbuildv2Connection_GhePrivUpdateConnection
=== PAUSE TestAccCloudbuildv2Connection_GhePrivUpdateConnection
=== CONT  TestAccCloudbuildv2Connection_GhePrivUpdateConnection
vcr_utils.go:146: Step 3/4 error: Error running apply: exit status 1
Error: Error updating Connection: operation received error: error code "7", message: could not connect to GitHub Enterprise server; please make sure that the Cloud Build P4SA (service-123456789@gcp-sa-cloudbuild.iam.gserviceaccount.com) has permission to access the Service Directory resource and the network resource: Get "https://ghe.proctor-private-ca.com/api/v3/meta": generic::permission_denied: Permission 'servicedirectory.services.resolve' denied on resource 'projects/gcb-terraform-creds/locations/us-central1/namespaces/myns/services/serv'., details: []
details: map[]
with google_cloudbuildv2_connection.primary,
on terraform_plugin_test.tf line 2, in resource "google_cloudbuildv2_connection" "primary":
2: resource "google_cloudbuildv2_connection" "primary" {
--- FAIL: TestAccCloudbuildv2Connection_GhePrivUpdateConnection (15.17s)
FAIL

[SarahFrench]

@roaks3 Could this be due to the test project split? It looks like the Cloud Build Service Agent from the test projects need to be given permissions on this secret from a separate project : https://github.com/hashicorp/terraform-provider-google-beta/blob/f7997b603c082fdd6680e4ab1573d034c1559bf4/google-beta/iam_cloudbuildv2_connection_generated_test.go#L126

Edit: plus other permission issues from the tests

[rileykarson]

Note: A googler likely needs to grant permissions to these internally

[trodge]

b/280100395

[melinath]

Forwarding this as it's an issue with test infrastructure owned by the service team.

[roaks3]

@SarahFrench Yes, access to these secrets was granted for our original project, so we did the same for the new GA/Beta projects, but it appears that the process has become more restrictive since it was originally done, and we're in-progress with granting permission using the new process. The service team is owning this: b/278293277 and b/278547809.

I'm also updating the description with 2 more tests affected by this: TestAccCloudBuildTrigger_cloudbuildTriggerRepoExample and TestAccCloudbuildv2Connection_GheCompleteConnection.

[roaks3]

I spoke with the service team on this yesterday, and it looks like this could take a bit longer to resolve (and the team may need to use an alternate approach). So for now, we can skip these tests with GoogleCloudPlatform/magic-modules#7867, and revert that change once we get things working.

cc @vicpadilla

[SarahFrench]

I'm seeing 100% failure of TestAccCloudbuildv2Connection_GhePrivConnection since 2023-04-08 in the Beta TeamCity project - LINK

------- Stdout: -------
=== RUN   TestAccCloudbuildv2Connection_GhePrivConnection
=== PAUSE TestAccCloudbuildv2Connection_GhePrivConnection
=== CONT  TestAccCloudbuildv2Connection_GhePrivConnection
    vcr_utils.go:152: Step 1/2 error: Error running apply: exit status 1
        Error: Error creating Connection: operation received error: error code "7", message: could not connect to GitHub Enterprise server; please make sure that the Cloud Build P4SA (service-653407317329@gcp-sa-cloudbuild.iam.gserviceaccount.com) has permission to access the Service Directory resource and the network resource: Get "https://ghe.proctor-private-ca.com/api/v3/meta": generic::permission_denied: Permission 'servicedirectory.networks.access' denied on resource 'projects/1033762898806/locations/global/networks/ghe'., details: []
         details: map[]
          with google_cloudbuildv2_connection.primary,
          on terraform_plugin_test.tf line 2, in resource "google_cloudbuildv2_connection" "primary":
           2: resource "google_cloudbuildv2_connection" "primary" {
--- FAIL: TestAccCloudbuildv2Connection_GhePrivConnection (5.40s)
FAIL

Same for TestAccCloudbuildv2Connection_GhePrivUpdateConnection - in the Beta TeamCity project - LINK

[roaks3]

@vicpadilla I thought I saw some follow-up changes on your end, but I don't see them referenced here. Pinging you to make sure you're aware, error is the same as before. We can of course skip again if this requires some time.

[vicpadilla]

Could you retry those tests? We were missing an IAM permission (for networking) for the P4SA of the nightly-beta and nightly-ga projects. I've just added those permissions.

[roaks3]

👍 it already failed from last night's run, so probably easiest to wait for the next run tonight

[roaks3]

Permission issue appears resolved from last night's run. Thanks for resolving @vicpadilla !

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.