Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Dataplex Asset resource #6310

Merged
merged 17 commits into from
Jul 29, 2022
Merged

Add Dataplex Asset resource #6310

merged 17 commits into from
Jul 29, 2022

Conversation

saurabh-net
Copy link
Contributor

part of hashicorp/terraform-provider-google#11648

If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read the Release Notes Guide before writing my release note below.
`google_dataplex_asset`

@modular-magician
Copy link
Collaborator

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @megan07, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

Your assigned reviewer will help review your code by:

  • Ensuring it's backwards compatible, covers common error cases, etc.
  • Summarizing the change into a user-facing changelog note.
  • Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.


@modular-magician modular-magician requested a review from megan07 July 21, 2022 00:08
@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 1350 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 5 files changed, 1350 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@saurabh-net saurabh-net changed the title Initial commit, verify asset.yaml Add Dataplex Asset resource Jul 22, 2022
@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1351 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1351 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2115
Passed tests 1883
Skipped tests: 226
Failed tests: 6

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccFirebaserulesRelease_BasicRelease|TestAccDataplexAsset_BasicAsset|TestAccComputeInstance_soleTenantNodeAffinities|TestAccSqlUser_mysqlDisabled

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccSqlUser_mysqlDisabled[view]
TestAccFirebaserulesRelease_BasicRelease[view]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccDataplexAsset_BasicAsset[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@megan07
Copy link
Contributor

megan07 commented Jul 25, 2022

/gcbrun

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1351 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1351 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2115
Passed tests 1883
Skipped tests: 226
Failed tests: 6

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccFirebaserulesRelease_BasicRelease|TestAccDataplexAsset_BasicAsset|TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccSqlUser_mysqlDisabled

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccSqlUser_mysqlDisabled[view]
TestAccFirebaserulesRelease_BasicRelease[view]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccDataplexAsset_BasicAsset[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@saurabh-net
Copy link
Contributor Author

It looks like I'll have to manually create a GCS bucket for the Dataplex asset to attach to. I'll mark this review in draft until I complete that step.

@saurabh-net saurabh-net marked this pull request as draft July 25, 2022 17:12
@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1356 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1356 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2119
Passed tests 1889
Skipped tests: 226
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccDataplexAsset_BasicAssetHandWritten|TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample

@modular-magician
Copy link
Collaborator

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccDataplexAsset_BasicAssetHandWritten[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1350 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1350 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2120
Passed tests 1890
Skipped tests: 226
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccDataplexAsset_BasicAssetHandWritten|TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample

@modular-magician
Copy link
Collaborator

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccDataplexAsset_BasicAssetHandWritten[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1365 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1365 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2120
Passed tests 1887
Skipped tests: 226
Failed tests: 7

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccFirebaserulesRelease_BasicRelease|TestAccDataplexAsset_BasicAssetHandWritten|TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccBillingSubaccount_basic|TestAccBillingSubaccount_renameOnDestroy

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccBillingSubaccount_basic[view]
TestAccFirebaserulesRelease_BasicRelease[view]
TestAccBillingSubaccount_renameOnDestroy[view]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccDataplexAsset_BasicAssetHandWritten[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1365 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1365 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2121
Passed tests 1889
Skipped tests: 226
Failed tests: 6

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccDataplexAsset_BasicAssetHandWritten|TestAccFirebaserulesRelease_BasicRelease|TestAccComputeInstance_soleTenantNodeAffinities|TestAccCGCSnippet_eventarcWorkflowsExample|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccCGCSnippet_eventarcWorkflowsExample[view]
TestAccFirebaserulesRelease_BasicRelease[view]

Tests failed during RECORDING mode:
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccDataplexAsset_BasicAssetHandWritten[view]
TestAccComputeInstance_soleTenantNodeAffinities[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1367 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1367 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2123
Passed tests 1892
Skipped tests: 226
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccFirebaserulesRelease_BasicRelease|TestAccDataplexAsset_BasicAssetHandWritten|TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[view]
TestAccDataplexAsset_BasicAssetHandWritten[view]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@saurabh-net saurabh-net marked this pull request as ready for review July 26, 2022 20:43
@saurabh-net
Copy link
Contributor Author

I've added an ignore_read for Asset.resource_spec.name.

This is an immutable field, and the Dataplex API accepts input in the format "projects/{projectID}/buckets/{bucketID}", however it returns it as "{bucketID}".

The API behavior is because of the need to maintain backwards compatibility but return results in line with other GCP APIs.

Since it's an immutable field, it's safe to skip the verify import-state-step for this field. I've discussed the above with @trodge, and all the tests are passing now.

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 6 files changed, 1367 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 6 files changed, 1367 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

Copy link
Contributor

@slevenick slevenick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nevermind, it's just really long! No issue

@slevenick
Copy link
Contributor

Couple notes:

resource_type should not document that it allows TYPE_UNSPECIFIED, as it is not a valid value to send to the API.

The discovery_spec.enabled field's documentation does not match its behavior

zone is a bit weird, as zone is a magic field in the provider where we set it as Optional and inherit from the provider's default zone. Is it possible to rename this field?

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2123
Passed tests 1893
Skipped tests: 226
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccFirebaserulesRelease_BasicRelease|TestAccComputeInstance_soleTenantNodeAffinities|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[view]

Tests failed during RECORDING mode:
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 9 files changed, 1356 insertions(+), 7 deletions(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 0
Passed tests 0
Skipped tests: 0
Failed tests: 0

Errors occurred during REPLAYING mode. Please fix them to complete your PR
View the build log

@slevenick
Copy link
Contributor

/gcbrun to fix build

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 9 files changed, 1356 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 9 files changed, 1356 insertions(+), 7 deletions(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2127
Passed tests 1893
Skipped tests: 226
Failed tests: 8

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccContainerCluster_withMeshCertificatesConfig|TestAccComputeInstance_soleTenantNodeAffinities|TestAccCGCSnippet_eventarcWorkflowsExample|TestAccActiveDirectoryDomain_update|TestAccActiveDirectoryDomain_activeDirectoryDomainBasicExample|TestAccActiveDirectoryPeering_activeDirectoryPeeringBasicExample

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccContainerCluster_withMeshCertificatesConfig[view]
TestAccCGCSnippet_eventarcWorkflowsExample[view]

Tests failed during RECORDING mode:
TestAccActiveDirectoryDomain_activeDirectoryDomainBasicExample[view]
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccActiveDirectoryPeering_activeDirectoryPeeringBasicExample[view]
TestAccActiveDirectoryDomain_update[view]

Please fix these to complete your PR
View the build log or the debug log for each test

@slevenick
Copy link
Contributor

/gcbrun there was a CI hiccup

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 9 files changed, 1356 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 9 files changed, 1356 insertions(+), 7 deletions(-))
TF Validator: Diff ( 2 files changed, 5 insertions(+), 5 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2127
Passed tests 1894
Skipped tests: 226
Failed tests: 7

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccComputeInstance_soleTenantNodeAffinities|TestAccActiveDirectoryDomain_update|TestAccActiveDirectoryDomain_activeDirectoryDomainBasicExample|TestAccCGCSnippet_eventarcWorkflowsExample|TestAccActiveDirectoryPeering_activeDirectoryPeeringBasicExample

@modular-magician
Copy link
Collaborator

Tests passed during RECORDING mode:
TestAccCGCSnippet_eventarcWorkflowsExample[view]

Tests failed during RECORDING mode:
TestAccActiveDirectoryDomain_activeDirectoryDomainBasicExample[view]
TestAccComputeInstance_soleTenantNodeAffinities[view]
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]
TestAccActiveDirectoryPeering_activeDirectoryPeeringBasicExample[view]
TestAccActiveDirectoryDomain_update[view]

Please fix these to complete your PR
View the build log or the debug log for each test

Copy link
Contributor

@slevenick slevenick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

One note but not urgent, you can fix as you have time:

This line mentions what happens when discovery_spec is unset, but the field has been marked required so it can never be unset: https://github.com/modular-magician/terraform-provider-google-beta/compare/auto-pr-6310-old..auto-pr-6310#diff-5d6d84399bf306af362c41c7ee924744379a41d52bb6d4754206961bd89c681aR100

@slevenick slevenick merged commit 38532a3 into GoogleCloudPlatform:main Jul 29, 2022
hao-nan-li pushed a commit to hao-nan-li/magic-modules that referenced this pull request Sep 6, 2022
* Initial commit, verify asset.yaml

* Use an existing public bucket

* Add Dataplex Asset into provider.go

* Use a sample bucket instead of a hardcoded one

* Fix asset YAML to have asset field

* Fix asset YAML to have bucket field

* Try again with a hardcoded bucket

* minor syntax fix

* Using handwritten tests since GCS buckets are not in DCL

* Fix bucket name reference in Dataplex Asset

* Ignore updates to bucket after creation

* Change dataplex terraform resource name to primary for the tests

* Add override for resource_spec.name. This is an immutable field and is normalized by the API.

* upgrade dcl to 1.16

* Update test to use dataplexZone instead of zone

* Add back comment accidentally removed in merge
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants