Adding resilience_mode field support to Cloud Composer #8058

Khaledmohamedrefaat · 2023-06-01T12:23:42Z

Add Cloud Composer Support for high resilience field

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

composer: added `resilience_mode` field in `google_composer_environment`

Fixes hashicorp/terraform-provider-google/issues/14751

google-cla · 2023-06-01T12:23:46Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

modular-magician · 2023-06-01T12:23:57Z

Oops! It looks like no changelog entry is attached to this PR. Please include a release note block in the PR body, as described in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md:

```release-note:TYPE
Release note
```

modular-magician · 2023-06-01T12:24:28Z

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @roaks3, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

Your assigned reviewer will help review your code by:

Ensuring it's backwards compatible, covers common error cases, etc.
Summarizing the change into a user-facing changelog note.
Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

modular-magician · 2023-06-01T16:48:05Z

Oops! It looks like no changelog entry is attached to this PR. Please include a release note block in the PR body, as described in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md:

```release-note:TYPE
Release note
```

mmv1/third_party/terraform/resources/resource_composer_environment.go.erb

mmv1/third_party/terraform/tests/resource_composer_environment_test.go.erb

modular-magician · 2023-06-06T19:04:40Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 5 files changed, 267 insertions(+), 175 deletions(-))
Terraform Beta: Diff ( 5 files changed, 267 insertions(+), 176 deletions(-))
TF Conversion: Diff ( 2 files changed, 56 insertions(+), 322 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_composer_environment (1 total tests)
Untested fields: config.resilience_mode

Please add acceptance tests which include these fields.

modular-magician · 2023-06-06T19:07:07Z

Tests analytics

Total tests: 12
Passed tests 12
Skipped tests: 0
Affected tests: 0

Errors occurred during REPLAYING mode. Please fix them to complete your PR
View the build log

roaks3 · 2023-06-06T21:16:42Z

It looks like there's a build error:

sh -c "'/home/runner/work/magic-modules/magic-modules/provider/scripts/gofmtcheck.sh'"
==> Checking that code complies with gofmt requirements...
Error: ./google/resource_composer_environment_test.go:530:1: expected declaration, found '}'
Error: ./google/resource_composer_environment_test.go:23[8](https://github.com/GoogleCloudPlatform/magic-modules/actions/runs/5192309052/jobs/9361294667#step:8:9)4:1: raw string literal not terminated

modular-magician · 2023-06-15T19:05:15Z

Tests analytics

Total tests: 2789
Passed tests 2488
Skipped tests: 297
Affected tests: 4

Action taken

Found 4 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComputeFirewallPolicyRule_multipleRules|TestAccComposerEnvironment_ComposerV2HighResilience|TestAccComposerEnvironment_ComposerV2

Get to know how VCR tests work

modular-magician · 2023-06-15T19:32:41Z

Tests passed during RECORDING mode:
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]
TestAccComposerEnvironment_ComposerV2HighResilience[Debug log]
TestAccComposerEnvironment_ComposerV2[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

roaks3

LGTM and tests pass. I had a few small questions/suggestions, but nothing major.

mmv1/third_party/terraform/resources/resource_composer_environment.go.erb

mmv1/third_party/terraform/tests/resource_composer_environment_test.go.erb

roaks3 · 2023-06-15T20:40:30Z

mmv1/third_party/terraform/tests/resource_composer_environment_test.go.erb

+		resilience_mode = "HIGH_RESILIENCE"
+		private_environment_config {
+			enable_private_endpoint                  = true
+			cloud_composer_network_ipv4_cidr_block   = "10.3.192.0/24"


I'm a little out of my depth so want to make sure: is there any risk of these cidr ranges conflicting with other resources in our various tests environments? (for example, if this test runs at the same time as the test above, testAccComposerEnvironment_composerV2)

I'm not sure if there is a risk. Nonetheless, these IP ranges have been duplicated several times already in these tests. I expect that duplicating them once more shouldn't change much.

Oh alright, I hadn't noticed that, but agreed it shouldn't be an issue then.

mmv1/third_party/terraform/website/docs/r/composer_environment.html.markdown

PeWu · 2023-06-16T11:31:34Z

Thanks! I addressed the comments. PTAL.

roaks3 · 2023-06-16T19:28:29Z

This is ready to merge, but we'll need to resolve the conflicts first, and then just double check that tests still pass.

modular-magician · 2023-06-19T21:25:36Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 127 insertions(+))
Terraform Beta: Diff ( 3 files changed, 127 insertions(+))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2023-06-19T22:42:53Z

Tests analytics

Total tests: 2797
Passed tests 2493
Skipped tests: 301
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComposerEnvironment_ComposerV2HighResilience|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-06-19T23:36:18Z

Tests passed during RECORDING mode:
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]

Tests failed during RECORDING mode:
TestAccComposerEnvironment_ComposerV2HighResilience[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

PeWu · 2023-06-20T09:07:47Z

VCR-test is marked as failed but when I click Details, it shows as successful in Cloud Build. What can be the issue here?

Khaledmohamedrefaat · 2023-06-20T10:10:59Z

@roaks3 I've tried running TestAccComposerEnvironment_ComposerV2HighResilience locally without any errors so it seems this error was just a flake, Let's retrigger the checks again.

modular-magician · 2023-06-20T16:38:18Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 127 insertions(+))
Terraform Beta: Diff ( 3 files changed, 127 insertions(+))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

roaks3 · 2023-06-20T16:41:32Z

This is the error from the failed test. Per https://cloud.google.com/build/docs/iam-roles-permissions, it looks like this role should be there by default when the service is enabled. It's possible some other test or PR is affecting that permission, but I don't think it is a concern of this test. In other words, we should be able to safely assume that role exists:

    vcr_utils.go:148: Step 1/3 error: Error running apply: exit status 1
        
        Error: Error waiting to create Environment: Error waiting for Creating Environment: Error code 9, message: Failed to create environment, but no error was surfaced. This can be caused by a lack of proper permissions. Check if this environment's service account 1067888929963-compute@developer.gserviceaccount.com has the 'roles/composer.worker' role and there is no firewall inhibiting internal communications set. In case, it is a Compute default service account it should have Editor permissions. For the required permissions information, please, refer to https://cloud.google.com/composer/docs/how-to/access-control  https://cloud.google.com/composer/docs/troubleshooting-environment-creation page contains more troubleshooting instructions
        
        The issue may be caused by missing IAM roles in the following Service Accounts:
         - 1067888929963@cloudbuild.gserviceaccount.com in project 1067888929963 is missing role roles/cloudbuild.builds.builder
        
        The list of missing roles is generated without checking individual permissions in IAM custom roles. If any of the Service Accounts above uses custom IAM roles, its permissions may be sufficient and a corresponding warning may be ignored.
        
          with google_composer_environment.test,
          on terraform_plugin_test.tf line 2, in resource "google_composer_environment" "test":
           2: resource "google_composer_environment" "test" {

--- FAIL: TestAccComposerEnvironment_ComposerV2HighResilience (3180.00s)
FAIL

modular-magician · 2023-06-20T18:20:21Z

Tests analytics

Total tests: 2797
Passed tests 2493
Skipped tests: 301
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComposerEnvironment_ComposerV2HighResilience|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-06-20T18:47:05Z

Tests passed during RECORDING mode:
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]
TestAccComposerEnvironment_ComposerV2HighResilience[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

roaks3 · 2023-06-20T19:51:56Z

Merging because the tests passed this time, and the missing role that caused the failure should not be an issue under normal circumstances.

…atform#8058) * Adding High Resilience Fields Support To Terraform * Bumping composer api version to 0.125.0 * Adding Test for composerV2HighResilience * Fixing function name * retrigger checks * retrigger checks * Replacing go.sum with the fresh version * Replacing go.mod.erb with the fresh version * Removing high resilience test and adding field to composerv2 test * Fixing Indentation * Remove HighResilience TEST * Removing HR test * Fixing Removing ComposerV2 test * Fixing go mod and go sum * Added Test for HighResilience * Fixed Indentation * Revert go.mod.erb and go.sum changes * Apply review comments * retrigger checks --------- Co-authored-by: Khaled Hassan <hkhaled@google.com> Co-authored-by: Ryan Oaks <ryanoaks@google.com> Co-authored-by: Przemek Wiech <pwiech@google.com>

modular-magician requested a review from roaks3 June 1, 2023 12:24

modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 1, 2023

kubasieron suggested changes Jun 2, 2023

View reviewed changes

Khaledmohamedrefaat requested a review from kubasieron June 6, 2023 08:34

kubasieron approved these changes Jun 6, 2023

View reviewed changes

modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 6, 2023

Khaledmohamedrefaat force-pushed the hr-changes branch from 29d1154 to 4799cd1 Compare June 7, 2023 09:37

modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 7, 2023

Khaled Hassan added 8 commits June 7, 2023 09:39

Adding High Resilience Fields Support To Terraform

cd02d7c

Bumping composer api version to 0.125.0

031bdd0

Adding Test for composerV2HighResilience

7d5c703

Fixing function name

9013c81

retrigger checks

2633f7f

retrigger checks

59a3436

Replacing go.sum with the fresh version

4bfa668

Replacing go.mod.erb with the fresh version

3be2396

Khaledmohamedrefaat force-pushed the hr-changes branch from 4799cd1 to 3be2396 Compare June 7, 2023 09:43

Khaled Hassan added 4 commits June 7, 2023 09:58

Removing high resilience test and adding field to composerv2 test

50b83b6

Fixing Indentation

23e7034

Remove HighResilience TEST

6f1f488

Removing HR test

cb003e8

roaks3 reviewed Jun 15, 2023

View reviewed changes

Apply review comments

4648fa7

modular-magician requested a review from roaks3 June 16, 2023 11:23

modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 16, 2023

Merge remote-tracking branch 'upstream/main' into hr-changes

88fdec0

modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 19, 2023

retrigger checks

8072b1a

modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 20, 2023

modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Jun 20, 2023

roaks3 approved these changes Jun 20, 2023

View reviewed changes

roaks3 merged commit 4c17e70 into GoogleCloudPlatform:main Jun 20, 2023
8 checks passed

This was referenced Jun 20, 2023

Adding resilience_mode field support to Cloud Composer hashicorp/terraform-provider-google-beta#5790

Merged

Adding resilience_mode field support to Cloud Composer hashicorp/terraform-provider-google#14939

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding resilience_mode field support to Cloud Composer #8058

Adding resilience_mode field support to Cloud Composer #8058

Khaledmohamedrefaat commented Jun 1, 2023 •

edited

Loading

google-cla bot commented Jun 1, 2023

modular-magician commented Jun 1, 2023

modular-magician commented Jun 1, 2023

modular-magician commented Jun 1, 2023

modular-magician commented Jun 6, 2023

modular-magician commented Jun 6, 2023

roaks3 commented Jun 6, 2023

modular-magician commented Jun 15, 2023

modular-magician commented Jun 15, 2023

roaks3 left a comment

roaks3 Jun 15, 2023

PeWu Jun 16, 2023

roaks3 Jun 16, 2023

PeWu commented Jun 16, 2023

roaks3 commented Jun 16, 2023

modular-magician commented Jun 19, 2023

modular-magician commented Jun 19, 2023

modular-magician commented Jun 19, 2023

PeWu commented Jun 20, 2023

Khaledmohamedrefaat commented Jun 20, 2023

modular-magician commented Jun 20, 2023

roaks3 commented Jun 20, 2023

modular-magician commented Jun 20, 2023

modular-magician commented Jun 20, 2023

roaks3 commented Jun 20, 2023

Adding resilience_mode field support to Cloud Composer #8058

Adding resilience_mode field support to Cloud Composer #8058

Conversation

Khaledmohamedrefaat commented Jun 1, 2023 • edited Loading

google-cla bot commented Jun 1, 2023

modular-magician commented Jun 1, 2023

modular-magician commented Jun 1, 2023

modular-magician commented Jun 1, 2023

modular-magician commented Jun 6, 2023

Diff report

Missing test report

modular-magician commented Jun 6, 2023

Tests analytics

roaks3 commented Jun 6, 2023

modular-magician commented Jun 15, 2023

Tests analytics

Action taken

modular-magician commented Jun 15, 2023

roaks3 left a comment

Choose a reason for hiding this comment

roaks3 Jun 15, 2023

Choose a reason for hiding this comment

PeWu Jun 16, 2023

Choose a reason for hiding this comment

roaks3 Jun 16, 2023

Choose a reason for hiding this comment

PeWu commented Jun 16, 2023

roaks3 commented Jun 16, 2023

modular-magician commented Jun 19, 2023

Diff report

modular-magician commented Jun 19, 2023

Tests analytics

Action taken

modular-magician commented Jun 19, 2023

PeWu commented Jun 20, 2023

Khaledmohamedrefaat commented Jun 20, 2023

modular-magician commented Jun 20, 2023

Diff report

roaks3 commented Jun 20, 2023

modular-magician commented Jun 20, 2023

Tests analytics

Action taken

modular-magician commented Jun 20, 2023

roaks3 commented Jun 20, 2023

Khaledmohamedrefaat commented Jun 1, 2023 •

edited

Loading