Machine image updates

[upodroid]

Fixes: hashicorp/terraform-provider-google#7019
Fixes: hashicorp/terraform-provider-google#7055

I noticed something else. There is a new CMEK field called kmsKeyServiceAccount which allows you to set the Service account to use with KMS. All the resources that use CMEK need to be updated with this field, seen it both v1 and beta apis.

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

compute: added storage_locations & cmek fields to `google_compute_machine_image` resource

compute: updated `google_compute_machine_image` resource to complete once the Image is ready.

`google_compute_instance_from_machine_image`

`google_compute_machine_image_iam_member`

`google_compute_machine_image_iam_binding`

`google_compute_machine_image_iam_policy`

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@c2thorn, please review this PR or find an appropriate assignee.

[upodroid]

IAM Tests are passing.

REDACTED  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  26⬇  6✎  3+  $    make testacc TEST=./google-beta TESTARGS='-run=TestAccComputeMachineImageIam'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccComputeMachineImageIam -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccComputeMachineImageIamBindingGenerated
=== PAUSE TestAccComputeMachineImageIamBindingGenerated
=== RUN   TestAccComputeMachineImageIamMemberGenerated
=== PAUSE TestAccComputeMachineImageIamMemberGenerated
=== RUN   TestAccComputeMachineImageIamPolicyGenerated
=== PAUSE TestAccComputeMachineImageIamPolicyGenerated
=== RUN   TestAccComputeMachineImageIamBindingGenerated_withCondition
=== PAUSE TestAccComputeMachineImageIamBindingGenerated_withCondition
=== RUN   TestAccComputeMachineImageIamBindingGenerated_withAndWithoutCondition
=== PAUSE TestAccComputeMachineImageIamBindingGenerated_withAndWithoutCondition
=== RUN   TestAccComputeMachineImageIamMemberGenerated_withCondition
=== PAUSE TestAccComputeMachineImageIamMemberGenerated_withCondition
=== RUN   TestAccComputeMachineImageIamMemberGenerated_withAndWithoutCondition
=== PAUSE TestAccComputeMachineImageIamMemberGenerated_withAndWithoutCondition
=== RUN   TestAccComputeMachineImageIamPolicyGenerated_withCondition
=== PAUSE TestAccComputeMachineImageIamPolicyGenerated_withCondition
=== CONT  TestAccComputeMachineImageIamBindingGenerated
=== CONT  TestAccComputeMachineImageIamBindingGenerated_withAndWithoutCondition
=== CONT  TestAccComputeMachineImageIamMemberGenerated_withCondition
=== CONT  TestAccComputeMachineImageIamMemberGenerated_withAndWithoutCondition
=== CONT  TestAccComputeMachineImageIamPolicyGenerated_withCondition
=== CONT  TestAccComputeMachineImageIamPolicyGenerated
=== CONT  TestAccComputeMachineImageIamBindingGenerated_withCondition
=== CONT  TestAccComputeMachineImageIamMemberGenerated
--- PASS: TestAccComputeMachineImageIamBindingGenerated_withCondition (158.54s)
--- PASS: TestAccComputeMachineImageIamMemberGenerated_withCondition (159.43s)
--- PASS: TestAccComputeMachineImageIamPolicyGenerated (163.24s)
--- PASS: TestAccComputeMachineImageIamBindingGenerated (168.55s)
--- PASS: TestAccComputeMachineImageIamPolicyGenerated_withCondition (170.35s)
--- PASS: TestAccComputeMachineImageIamBindingGenerated_withAndWithoutCondition (171.29s)
--- PASS: TestAccComputeMachineImageIamMemberGenerated_withAndWithoutCondition (171.72s)
--- PASS: TestAccComputeMachineImageIamMemberGenerated (171.72s)
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/google-beta 173.983s

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 83 insertions(+))
Terraform Beta: Diff ( 11 files changed, 2160 insertions(+), 56 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[upodroid]

In commit c7820aa , the test are failing because the logic to override disks is broken.

REDACTED  MCW0CDP3YY  ~  go  …  github.com  hashicorp  terraform-provider-google-beta   master  26⬇  10✎  6+  USAGE  $    make testacc TEST=./google-beta TESTARGS='-run=TestAccComputeInstanceFromMachineImage'
==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta -v -run=TestAccComputeInstanceFromMachineImage -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccComputeInstanceFromMachineImage_basic
=== PAUSE TestAccComputeInstanceFromMachineImage_basic
=== RUN   TestAccComputeInstanceFromMachineImage_overrideBootDisk
=== PAUSE TestAccComputeInstanceFromMachineImage_overrideBootDisk
=== RUN   TestAccComputeInstanceFromMachineImage_overrideAttachedDisk
=== PAUSE TestAccComputeInstanceFromMachineImage_overrideAttachedDisk
=== RUN   TestAccComputeInstanceFromMachineImage_overrideScratchDisk
=== PAUSE TestAccComputeInstanceFromMachineImage_overrideScratchDisk
=== RUN   TestAccComputeInstanceFromMachineImage_overrideMetadataDotStartupScript
=== PAUSE TestAccComputeInstanceFromMachineImage_overrideMetadataDotStartupScript
=== CONT  TestAccComputeInstanceFromMachineImage_basic
=== CONT  TestAccComputeInstanceFromMachineImage_overrideScratchDisk
=== CONT  TestAccComputeInstanceFromMachineImage_overrideBootDisk
=== CONT  TestAccComputeInstanceFromMachineImage_overrideAttachedDisk
=== CONT  TestAccComputeInstanceFromMachineImage_overrideMetadataDotStartupScript
    TestAccComputeInstanceFromMachineImage_overrideBootDisk: testing.go:674: Step 0 error: Check failed: Check 3/3 error: google_compute_instance_from_machine_image.foobar: Attribute 'boot_disk.0.name' didn't match "terraform-test-0dymegdnmb", got ""
    TestAccComputeInstanceFromMachineImage_overrideScratchDisk: testing.go:674: Step 0 error: Check failed: Check 3/3 error: google_compute_instance_from_machine_image.foobar: Attribute 'scratch_disk.0.interface' expected "NVME", got "SCSI"
    TestAccComputeInstanceFromMachineImage_overrideAttachedDisk: testing.go:674: Step 0 error: Check failed: Check 1/3 error: Not found: google_compute_instance_from_machine_image.inst
--- PASS: TestAccComputeInstanceFromMachineImage_basic (287.22s)
--- FAIL: TestAccComputeInstanceFromMachineImage_overrideAttachedDisk (310.96s)
--- FAIL: TestAccComputeInstanceFromMachineImage_overrideBootDisk (332.47s)
--- FAIL: TestAccComputeInstanceFromMachineImage_overrideScratchDisk (389.99s)
--- PASS: TestAccComputeInstanceFromMachineImage_overrideMetadataDotStartupScript (403.27s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-google-beta/google-beta 405.797s
FAIL
make: *** [testacc] Error 1

[upodroid]

I applied a similar logic in adjustInstanceFromMachineImageDisks from the google_compute_instance_from_instance_template but i'm getting inconsistent state behaviour. I suspect it is the type changes in that function.

https://gist.github.com/upodroid/1721c4e4ee26930619813a182c1e1be2 Results of the acceptance test from e2453db commit.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 83 insertions(+))
Terraform Beta: Diff ( 11 files changed, 1963 insertions(+), 56 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 83 insertions(+))
Terraform Beta: Diff ( 11 files changed, 1995 insertions(+), 56 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[c2thorn]

Thanks @upodroid! Have some minor changes for you

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 82 insertions(+))
Terraform Beta: Diff ( 11 files changed, 1982 insertions(+), 56 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[upodroid]

Applied the minor changes, the override logic is still broken.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 89 insertions(+))
Terraform Beta: Diff ( 11 files changed, 2017 insertions(+), 55 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[upodroid]

@c2thorn ^

[c2thorn]

Hi @upodroid, sorry about that. I was under the impression that you still wanted to address the override logic. Do you have any ideas for that?

[upodroid]

Not really. I'm stuck on making the disk override work correctly.

[c2thorn]

Okay, I'll take a look at it!

[c2thorn]

@upodroid I found out internally that there is a Compute bug for overriding disks from machine images. Your logic for that should be correct. Allegedly a fix should have already rolled out, but I am still running into failures using the latest googleapi go client. I'll follow up internally.

Also, I don't mind adding a commit or two to this PR, but I think I need push access to your fork.

[upodroid]

Thanks for the update. You should be able to edit my fork.

[c2thorn]

Hmm still getting a 403 trying to add to this PR. @upodroid can you try checking the box on this page stating Allow edits and access to secrets by maintainers ?

[upodroid]

I can't see it.

I ticked it the first time I opened a PR from my fork.

[upodroid]

[upodroid]

Urgh, the fork belongs to my gh org and not my user. I added you directly to the repo with write perms.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 7 files changed, 353 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 11 files changed, 1794 insertions(+), 68 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[c2thorn]

/gcbrun

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=154395"

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 7 files changed, 353 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 11 files changed, 1794 insertions(+), 68 deletions(-))
TF Conversion: Diff ( 1 file changed, 4 insertions(+))

[upodroid]

I updated the release notes to remove the iam resources. They were implemented in a separate PR.

[c2thorn]

@upodroid The machine image IAM resources? can't seem to find them in any other PR/in TPGB currently

[c2thorn]

Also, I am planning to move forward with this PR with the minor changes I've made. I've basically updated the PR to be compatible with general changes to the repo, and then removed the ability to override disks (not yet supported by the API). Testing changes now

[upodroid]

IAM was done in #3887 . I remember it because the issue was duplicated.

[upodroid]

nvm, that was compute image not machine image :D

[upodroid]

Sounds great, I just need to fix the typo @drebes pointed out earlier

[c2thorn]

@upodroid already taken care of!

[c2thorn]

Tests pass! I know it has been a while on this one, thanks for your help and patience. I'll likely merge this in later today if you don't have any concerns.

[upodroid]

Thanks, merge it once you are ready.

[c2thorn]

Might as well do it soon.

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceSpannerInstance_basic|TestAccComputeMachineImageIamBindingGenerated|TestAccComputeMachineImageIamMemberGenerated|TestAccComputeMachineImageIamPolicyGenerated|TestAccComputeMachineImageIamBindingGenerated_withCondition|TestAccComputeMachineImageIamMemberGenerated_withCondition|TestAccComputeMachineImageIamPolicyGenerated_withCondition|TestAccComputeInstanceFromMachineImage_basic|TestAccComputeInstanceFromMachineImage_overrideMetadataDotStartupScript|TestAccComputeMachineImage_machineImageBasicExample|TestAccContainerCluster_withConfidentialNodes You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=154398"