Add temp_bucket parameter for resource google_dataproc_cluster (#7927) #4375

melinath · 2021-01-07T19:38:42Z

Upstreamed from hashicorp/terraform-provider-google#8131. According to the docs the missing CLA is not an issue since this only changes third_party?

Allow using temp_bucket when creating a Dataproc cluster as addressed in hashicorp/terraform-provider-google#7927.

Closes hashicorp/terraform-provider-google#7927.

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dataproc: Added field `temp_bucket` to `google_dataproc_cluster` cluster config.

google-cla · 2021-01-07T19:38:46Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

modular-magician · 2021-01-07T19:46:36Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 99 insertions(+))
Terraform Beta: Diff ( 2 files changed, 99 insertions(+))

modular-magician · 2021-01-07T19:46:55Z

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=165840"

modular-magician · 2021-01-07T20:57:29Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccInstanceTemplateDatasource_name|TestAccInstanceTemplateDatasource_filter|TestAccInstanceTemplateDatasource_filter_mostRecent|TestAccDataSourceGoogleSubnetwork|TestAccDataSourceStorageBucketObjectContent_Basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccContainerCluster_withConfidentialNodes|TestAccContainerCluster_withPrivateClusterConfigMissingCidrBlock|TestAccDataprocAutoscalingPolicy_dataprocAutoscalingPolicyExample|TestAccDataprocClusterIamBinding|TestAccDataprocCluster_withTempBucket|TestAccDataprocClusterIamPolicy|TestAccDataprocClusterIamMember|TestAccDataprocCluster_withMetadataAndTags|TestAccDataprocCluster_withAccelerators|TestAccDataprocCluster_withInternalIpOnlyTrue|TestAccDataprocCluster_singleNodeCluster|TestAccDataprocCluster_updatable|TestAccDataprocCluster_withStagingBucket|TestAccDataprocCluster_withInitAction|TestAccDataprocCluster_withImageVersion|TestAccDataprocCluster_withConfigOverrides|TestAccDataprocCluster_withOptionalComponents|TestAccDataprocCluster_withLifecycleConfigIdleDeleteTtl|TestAccDataprocCluster_withEndpointConfig|TestAccDataprocCluster_withAutoscalingPolicy|TestAccDataprocCluster_withKerberos|TestAccDataprocJobIamBinding|TestAccDataprocJob_Spark|TestAccDataprocCluster_withServiceAcc|TestAccDataprocJob_updatable|TestAccDataprocJobIamPolicy|TestAccDataprocJob_PySpark|TestAccDataprocJobIamMember|TestAccDataprocJob_Hadoop|TestAccDataprocJob_Hive|TestAccDataprocCluster_KMS|TestAccDataprocJob_Pig|TestAccDataprocJob_SparkSql You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=165854"

google-cla · 2021-01-08T18:38:20Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

modular-magician · 2021-01-08T18:46:33Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 100 insertions(+))
Terraform Beta: Diff ( 2 files changed, 100 insertions(+))

modular-magician · 2021-01-08T18:46:49Z

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=166040"

modular-magician · 2021-01-08T20:11:51Z

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccInstanceTemplateDatasource_filter|TestAccInstanceTemplateDatasource_filter_mostRecent|TestAccDataSourceGoogleSubnetwork|TestAccDataSourceGoogleServiceAccountIdToken_impersonation|TestAccDataSourceStorageBucketObjectContent_Basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccContainerCluster_withConfidentialNodes|TestAccContainerCluster_withPrivateClusterConfigMissingCidrBlock|TestAccDataprocJob_Spark You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=166056"

melinath · 2021-01-08T21:40:18Z

dataproc tests failing on this branch are also failing on master; I believe they are unrelated to these changes.

ScottSuarez · 2021-01-08T22:17:04Z

you should link the issue like closes 7927

since the issue exists on a separate repo you can't do the hash. Use a quantified link so it will auto close on completion

ScottSuarez

I don't know much about ... so adding @slevenick for a second pass

slevenick

We will need the field to be added to the docs so that it will show up on the website

slevenick · 2021-01-08T22:48:31Z

third_party/terraform/tests/resource_dataproc_cluster_test.go.erb

+			{
+				// Simulate destroy of cluster by removing it from definition,
+				// but leaving the temp bucket (should not be auto deleted)
+				Config: testAccDataprocCluster_withTempBucketOnly(bucketName),


I'm not really sure why this step is here. It seems to be testing that deleting the cluster does not delete the storage bucket, but I'm not sure why we would be concerned about that. Declaring the storage bucket in a TF config and ensuring that it exists should be covered by storage bucket tests

@slevenick I believe the contributor copy-pasted this from the storage bucket tests directly above. My assumption was that there was a concern that GCP might auto-delete the related storage (or temp) bucket resource without terraform being able to control it.

If you think this is unnecessary, I can definitely ask them to remove it & open a ticket to update the storage_bucket tests as well.

talked with Sam - it's okay to leave this in place since it is the same as the other test.

google-cla · 2021-01-21T21:50:29Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

modular-magician · 2021-01-21T21:58:26Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 104 insertions(+))
Terraform Beta: Diff ( 3 files changed, 104 insertions(+))

modular-magician · 2021-01-21T21:58:42Z

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=168384"

melinath · 2021-01-21T22:02:54Z

@slevenick documentation added!

Add temp_bucket parameter for resource google_dataproc_cluster (#7927)

5c85814

google-cla bot added the cla: no label Jan 7, 2021

melinath requested review from a team and ScottSuarez and removed request for a team January 7, 2021 19:42

melinath removed the request for review from ScottSuarez January 7, 2021 21:45

Fix temp_bucket field, which should be a computed field.

37f0d28

melinath requested review from a team and ScottSuarez and removed request for a team January 8, 2021 21:39

ScottSuarez approved these changes Jan 8, 2021

View reviewed changes

ScottSuarez requested a review from slevenick January 8, 2021 22:21

slevenick requested changes Jan 8, 2021

View reviewed changes

Add documentation on temp_bucket.

c3f45d0

melinath requested a review from slevenick January 21, 2021 22:02

slevenick approved these changes Jan 21, 2021

View reviewed changes

melinath merged commit 81b5bb5 into master Jan 21, 2021

melinath deleted the upstream-tpg-8131 branch January 21, 2021 22:29

This was referenced Jan 21, 2021

Add temp_bucket parameter for resource google_dataproc_cluster (#7927) hashicorp/terraform-provider-google-beta#2893

Merged

Add temp_bucket parameter for resource google_dataproc_cluster (#7927) hashicorp/terraform-provider-google#8274

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add temp_bucket parameter for resource google_dataproc_cluster (#7927) #4375

Add temp_bucket parameter for resource google_dataproc_cluster (#7927) #4375

melinath commented Jan 7, 2021 •

edited

Loading

google-cla bot commented Jan 7, 2021

modular-magician commented Jan 7, 2021

modular-magician commented Jan 7, 2021

modular-magician commented Jan 7, 2021

google-cla bot commented Jan 8, 2021

modular-magician commented Jan 8, 2021

modular-magician commented Jan 8, 2021

modular-magician commented Jan 8, 2021

melinath commented Jan 8, 2021 •

edited

Loading

ScottSuarez commented Jan 8, 2021

ScottSuarez left a comment •

edited

Loading

slevenick left a comment

slevenick Jan 8, 2021

melinath Jan 11, 2021

melinath Jan 21, 2021

google-cla bot commented Jan 21, 2021

modular-magician commented Jan 21, 2021

modular-magician commented Jan 21, 2021

melinath commented Jan 21, 2021

Add temp_bucket parameter for resource google_dataproc_cluster (#7927) #4375

Add temp_bucket parameter for resource google_dataproc_cluster (#7927) #4375

Conversation

melinath commented Jan 7, 2021 • edited Loading

google-cla bot commented Jan 7, 2021

modular-magician commented Jan 7, 2021

Diff report:

modular-magician commented Jan 7, 2021

modular-magician commented Jan 7, 2021

google-cla bot commented Jan 8, 2021

modular-magician commented Jan 8, 2021

Diff report:

modular-magician commented Jan 8, 2021

modular-magician commented Jan 8, 2021

melinath commented Jan 8, 2021 • edited Loading

ScottSuarez commented Jan 8, 2021

ScottSuarez left a comment • edited Loading

Choose a reason for hiding this comment

slevenick left a comment

Choose a reason for hiding this comment

slevenick Jan 8, 2021

Choose a reason for hiding this comment

melinath Jan 11, 2021

Choose a reason for hiding this comment

melinath Jan 21, 2021

Choose a reason for hiding this comment

google-cla bot commented Jan 21, 2021

modular-magician commented Jan 21, 2021

Diff report:

modular-magician commented Jan 21, 2021

melinath commented Jan 21, 2021

melinath commented Jan 7, 2021 •

edited

Loading

melinath commented Jan 8, 2021 •

edited

Loading

ScottSuarez left a comment •

edited

Loading