Add support for Dataproc Metastore CMEK config

[Noremac201]

Adds support for configuring CMEK during creation for Dataproc Metastore configurations.

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

metastore: Added support for encryption_config during service creation.

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@melinath, please review this PR or find an appropriate assignee.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 46 insertions(+))
Terraform Beta: Diff ( 3 files changed, 176 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 124 insertions(+))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 46 insertions(+))
Terraform Beta: Diff ( 3 files changed, 176 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 124 insertions(+))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 46 insertions(+))
Terraform Beta: Diff ( 3 files changed, 176 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 124 insertions(+))

[melinath]

/gcbrun

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 46 insertions(+))
Terraform Beta: Diff ( 3 files changed, 176 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 124 insertions(+))

[Noremac201]

I don't see anything useful in the logs for the TerraformVCRCommunity failure, is there somewhere I'm missing to see if its flakes or my change?

[melinath]

It looks like the test timed out. I'll rerun it. Are you able to run your new tests successfully locally? /gcbrun

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 46 insertions(+))
Terraform Beta: Diff ( 3 files changed, 176 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 124 insertions(+))

[modular-magician]

Tests rerun

[modular-magician]

Tests count:
Total tests: 1959
Passed tests 1712
Skipped tests: 233
Failed tests: 14

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccServiceNetworkingPeeredDNSDomain_basic|TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekExample|TestAccContainerCluster_withNodePoolNodeConfig|TestAccContainerCluster_withAuthenticatorGroupsConfig|TestAccComputeBackendService_backendServiceNetworkEndpointExample|TestAccCloudBuildTrigger_available_secrets_config|TestAccCloudBuildTrigger_cloudbuildTriggerManualExample|TestAccCloudBuildTrigger_disable|TestAccCloudBuildTrigger_basic|TestAccBigtableAppProfile_bigtableAppProfileAnyclusterExample|TestAccBigtableAppProfile_bigtableAppProfileMulticlusterExample|TestAccBigtableAppProfile_bigtableAppProfileSingleclusterExample|TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceAdvancedExample|TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic

[modular-magician]

Tests passed during RECORDING mode:
TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic
TestAccComputeBackendService_backendServiceNetworkEndpointExample
TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceAdvancedExample
TestAccServiceNetworkingPeeredDNSDomain_basic
TestAccContainerCluster_withNodePoolNodeConfig
TestAccCloudBuildTrigger_cloudbuildTriggerManualExample
TestAccCloudBuildTrigger_disable
TestAccCloudBuildTrigger_available_secrets_config
TestAccCloudBuildTrigger_basic
TestAccBigtableAppProfile_bigtableAppProfileSingleclusterExample
TestAccBigtableAppProfile_bigtableAppProfileMulticlusterExample
TestAccBigtableAppProfile_bigtableAppProfileAnyclusterExample

Tests failed during RECORDING mode:
TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekExample
TestAccContainerCluster_withAuthenticatorGroupsConfig

Please fix these to complete your PR
You can view the build log here: https://storage.cloud.google.com/vcr-test-logs/beta/refs/heads/auto-pr-5881/artifacts/0a051151-f1d0-437b-b366-a68f7fb8d036/build-log/recording_test.log and the debug log for each test here: https://console.cloud.google.com/storage/browser/vcr-test-logs/beta/refs/heads/auto-pr-5881/artifacts/0a051151-f1d0-437b-b366-a68f7fb8d036/recording

[melinath]

It looks like TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekExample failed with the following error:

Error: Error creating Service: googleapi: Error 400: Found 1 problem:
	1) The Cloud KMS key (projects/ci-test-project-188019/locations/us/keyRings/tf-test-example-keyringl2vjah08wi/cryptoKeys/tf-test-example-keyl2vjah08wi) could not be validated. Please ensure that the key's purpose is configured as `ENCRYPT_DECRYPT`, and the Dataproc Metastore service agent (service-1067888929963@gcp-sa-metastore.iam.gserviceaccount.com) has been granted `roles/cloudkms.cryptoKeyEncrypterDecrypter`.

The other test failure is unrelated to your change. Please make sure TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekExample passes for you locally and then re-request review.

[Noremac201]

Hey @melinath, ENCRYPT_DECRYPT is the default for the keys in the example, however, the SA has no permission in this project, am I allowed to try to set the SA to have permissions via the TF example? Or is there a best practice there?

[melinath]

You'll probably need to match what the tests for kms crypto key do. It looks like the examples are only for documentation:

magic-modules/mmv1/products/kms/terraform.yaml

Line 60 in 6fb57b6

examples:

The actual tests are handwritten:

magic-modules/mmv1/third_party/terraform/tests/resource_kms_crypto_key_test.go

Line 168 in 6fb57b6

func TestAccKmsCryptoKey_basic(t *testing.T) {

But that might not actually be required in your case. The main point is that you'll need to create a project to put the crypto key inside:

magic-modules/mmv1/third_party/terraform/tests/resource_kms_crypto_key_test.go

Line 438 in 6fb57b6

func testGoogleKmsCryptoKey_basic(projectId, projectOrg, projectBillingAccount, keyRingName, cryptoKeyName string) string {

There should be other test-only examples that use a similar pattern. (If you can make this work as an example, that would be preferred.)

[Noremac201]

Alright, I got it working as an example. Sorry for how long it took, took some iterations to get a working combination.

[melinath]

Cool, glad you were able to make it work. It looks like the tests are failing and I'm not sure why; as a first step, could you rebase your branch off of main? There's been some changes to the CI/CD pipeline recently that might have fixed this issue.

[melinath]

/gcbrun in case it helps.

[melinath]

Rerunning tests because the vcr pipeline should be stable now. /gcbrun

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 41 insertions(+))
Terraform Beta: Diff ( 3 files changed, 187 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 140 insertions(+))

[modular-magician]

Tests analytics

Total tests: 1972
Passed tests 1728
Skipped tests: 238
Failed tests: 6

Action taken

Triggering VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccAccessContextManager|TestAccOSConfigPatchDeployment_osConfigPatchDeploymentFullExample|TestAccServiceNetworkingPeeredDNSDomain_basic|TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekTestExample|TestAccFirebaserulesRelease_BasicRelease|TestAccContainerCluster_withAuthenticatorGroupsConfig

[modular-magician]

Tests passed during RECORDING mode:
TestAccContainerCluster_withAuthenticatorGroupsConfig[view]
TestAccServiceNetworkingPeeredDNSDomain_basic[view]
TestAccOSConfigPatchDeployment_osConfigPatchDeploymentFullExample[view]
TestAccFirebaserulesRelease_BasicRelease[view]

Tests failed during RECORDING mode:
TestAccAccessContextManager[view]
TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekTestExample[view]

Please fix these to complete your PR
View the build log or the debug log for each test

[Noremac201]

Looks like my test failed because the Cloud Storage SA didn't have correct permissions. Sorry, hard to reconcile what's different about my local project and how the test project will react. This should be the final change.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 41 insertions(+))
Terraform Beta: Diff ( 3 files changed, 193 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 146 insertions(+))

[modular-magician]

Tests analytics

Total tests: 1972
Passed tests 1732
Skipped tests: 238
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekTestExample|TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic

[modular-magician]

Tests failed during RECORDING mode:
TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic[view]
TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekTestExample[view]

Please fix these to complete your PR
View the build log or the debug log for each test

[Noremac201]

Unfortunately at this point I'm a little lost on what to do:

The test is failing because it successfully creates then runs a GET request that throws a 404, everything /seems/ to match up, but I'm not sure how my test could've introduced this issue. Could it be a dependency issue in my terraform infra?

[Noremac201]

OK, with a run with TF_LOG=debug, I saw that we actually have a bug in our Dataproc Metastore resource, if version wasn't specified it would force recreation during the test.

Should be fixed in #5921.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 45 insertions(+))
Terraform Beta: Diff ( 3 files changed, 201 insertions(+))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 4 files changed, 150 insertions(+))

[modular-magician]

Tests analytics

Total tests: 1972
Passed tests 1733
Skipped tests: 238
Failed tests: 1

Action taken

Triggering VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekTestExample

[modular-magician]

Tests passed during RECORDING mode:
TestAccDataprocMetastoreService_dataprocMetastoreServiceCmekTestExample[view]

All tests passed
View the build log or the debug log for each test

[melinath]

@Noremac201 do you want to tweak the test in this PR to supply a version, or do you want to wait for the other PR to be merged?

[melinath]

(I would recommend tweaking the test in this PR.)

[Noremac201]

@melinath Yes, I believe I tweaked it in this PR, both the example and actual test have the version set and it's passing now. This PR should be good to go from what I can tell.

[melinath]

This LGTM generally - just want to confirm that kms_key is an input field? That means that the instance would need to be destroyed and recreated if the value is changed. I don't see that in the documentation so I just wanted to double-check that it's intentional.

[Noremac201]

Yes that is correct, it is IMMUTABLE in our API definition, though I also don't see it anywhere in our docs, I'll have to get that update to make it clearer.

[melinath]

LGTM