Add two outputs for dlp job trigger: outputting to SCC and DataCatalog #6855

Wallace99 · 2022-11-21T19:57:47Z

Add in outputting results of DLP job to SCC and DataCatalog. Also fix description of saveFindings block and remove unnecessary restriction on only one output.

Fixes hashicorp/terraform-provider-google#11861

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dlp: added `publish_findings_to_cloud_data_catalog` and `publish_summary_to_cscc` to `google_data_loss_prevention_job_trigger` resource

modular-magician · 2022-11-21T19:58:01Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

enhancement New outputs for DLP inspection trigger

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2022-11-21T19:58:50Z

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @shuyama1, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

Your assigned reviewer will help review your code by:

Ensuring it's backwards compatible, covers common error cases, etc.
Summarizing the change into a user-facing changelog note.
Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

shuyama1

@Wallace99 Thanks for making the change! The build failed due to

bundler: failed to load command: compiler (compiler)
RuntimeError: Field 'publish_summary_to_cscc' properties are nil!

Looks like publishSummaryToCscc and publishFindingsToCloudDataCatalog are added as nested object without subfields which will cause the build fail. Maybe consider adding some virtual subfields such as boolean field enable to make sure nested object are non-empty?

Wallace99 · 2022-11-28T19:51:50Z

@Wallace99 Thanks for making the change! The build failed due to
bundler: failed to load command: compiler (compiler)
RuntimeError: Field 'publish_summary_to_cscc' properties are nil!
Looks like publishSummaryToCscc and publishFindingsToCloudDataCatalog are added as nested object without subfields which will cause the build fail. Maybe consider adding some virtual subfields such as boolean field enable to make sure nested object are non-empty?

Ah thanks, will update that shortly!

Wallace99 · 2022-12-10T07:24:53Z

Hi @shuyama1, sorry for the delay. Do you know of elsewhere in the repo I could reference for adding virtual subfields? Would some custom logic also be needed? To configure a DLP inspection job to output to SCC an empty JSON object called publishSummaryToCscc is added to the actions array, and if it's not there it doesn't send to SCC (likewise for Data Catalog). If we add a virtual field called enabled for example, will the empty JSON object still be added if enabled is set to false by the user? Perhaps I'm misunderstanding what virtual subfields are?

shuyama1 · 2022-12-12T03:42:24Z

Hi @Wallace99, it seems that we may be able handle empty block schema by adding properties: [] to the nested field in the api.yaml, for example https://github.com/GoogleCloudPlatform/magic-modules/blame/main/mmv1/products/vertexai/api.yaml#L938. Would you mind trying that first and see if it works?

…g outputs

shuyama1 · 2022-12-13T05:28:03Z

/gcbrun

modular-magician · 2022-12-13T05:43:55Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 100 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 3 files changed, 100 insertions(+), 6 deletions(-))
TF Validator: Diff ( 3 files changed, 47 insertions(+), 3 deletions(-))

modular-magician · 2022-12-13T06:32:55Z

Tests analytics

Total tests: 2340
Passed tests 2091
Skipped tests: 244
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccDataLossPreventionJobTrigger_dlpJobTriggerUpdateExample|TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitPercentageExample|TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample|TestAccDataLossPreventionJobTrigger_dlpJobTriggerBasicExample

modular-magician · 2022-12-13T06:34:11Z

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerUpdateExample[Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBasicExample[Debug log]

Tests failed during RECORDING mode:
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitPercentageExample[Error message] [Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

mmv1/templates/terraform/examples/dlp_job_trigger_bigquery_row_limit.tf.erb

mmv1/templates/terraform/examples/dlp_job_trigger_bigquery_row_limit_percentage.tf.erb

shuyama1 · 2022-12-13T08:48:44Z

Hi @Wallace99! Would you mind also modifying the changelog a bit based on https://github.com/GoogleCloudPlatform/magic-modules/blob/main/.ci/RELEASE_NOTES_GUIDE.md? Thanks!

Co-authored-by: Shuya Ma <87669292+shuyama1@users.noreply.github.com>

modular-magician · 2022-12-13T20:44:13Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 100 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 3 files changed, 100 insertions(+), 6 deletions(-))
TF Validator: Diff ( 3 files changed, 47 insertions(+), 3 deletions(-))

modular-magician · 2022-12-13T21:32:49Z

Tests analytics

Total tests: 2344
Passed tests 2094
Skipped tests: 246
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample|TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitPercentageExample|TestAccLoggingBucketConfigProject_cmekSettings|TestAccFirebaserulesRelease_BasicRelease

modular-magician · 2022-12-13T21:34:11Z

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]

Tests failed during RECORDING mode:
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample[Error message] [Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitPercentageExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

shuyama1

Looks like only one of the fields in actions can be specified. Therefore, I propose we add back exactly_one_of and also include the newly added two fields. Plus, we need to use indexed path for exactly_one_of (save_findings -> inspect_job.0.actions.0.save_findings).

Would you mind testing it locally first to make sure the test passed? It will be faster to see tests result running locally as our CI need to run the PR against the whole test suite.

Wallace99 · 2022-12-13T22:31:35Z

The docs for the DLP rest api do say only one action can be specified, but I believe they're incorrect as I can create an inspection trigger with multiple actions both through the rest api and through the console. That's why I removed exactly_one_of as it's unnecessarily restrictive - there's no reason I can think of why someone shouldn't be able to output their results to SCC as well as to BigQuery for example. I left feedback on the DLP docs page, but I might try and contact them directly to get them to correct the docs or clarify whether it's actually intended to be able to only output to one place.

But yes, sorry I will locally test first! I've been having issues getting the tests to run locally so hoped this would be a trivial change that passed first try 😄

shuyama1 · 2022-12-15T19:29:31Z

@Wallace99 Thanks for the explanation! Please let me know when you hear back from the API team, as based on the current error for the failed test, it seems that it still does not allow multiple actions to be set at the same time:

    provider_test.go:315: Step 1/2 error: Error running apply: exit status 1
        
        Error: Error creating JobTrigger: googleapi: Error 400: Invalid value at 'job_trigger.inspect_job.actions[0]' (oneof), oneof field 'action' is already set. Cannot set 'saveFindings'
        Details:
        [
          {
            "@type": "type.googleapis.com/google.rpc.BadRequest",
            "fieldViolations": [
              {
                "description": "Invalid value at 'job_trigger.inspect_job.actions[0]' (oneof), oneof field 'action' is already set. Cannot set 'saveFindings'",
                "field": "job_trigger.inspect_job.actions[0]"
              }
            ]
          }
        ]
        
          with google_data_loss_prevention_job_trigger.bigquery_row_limit,
          on terraform_plugin_test.tf line 3, in resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit":
           3: resource "google_data_loss_prevention_job_trigger" "bigquery_row_limit" {
        
--- FAIL: TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample (2.12s)

Wallace99 · 2022-12-20T20:38:05Z

Hi @shuyama1, my mistake - I meant to say the DLP Python client rather than the REST api. Just now I verified that I can create an inspection job trigger with two actions using the Python client which from a quick look appears to be calling the RPCs here. I'm confused as to why then the test would be failing with the message above. I opened a support case seeking clarification, but they said that it's intentional that the REST api (and RPCs) only allow one action and that there's an internal request for this to be changed, but there's no timeline for when it will be done.

I'm happy in the meantime to put back in exactly_one_of for actions to allow this PR and revisit it at a future point.

shuyama1 · 2022-12-27T22:41:22Z

@Wallace99 Thanks for the explanation. I'd propose to bring back exactly_one_of since the REST api only allows one action at this moment. We can drop the restriction when the API behavior is changed in the future. Thanks!

modular-magician · 2023-01-15T19:53:29Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 94 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 2 files changed, 94 insertions(+), 4 deletions(-))
TF Validator: Diff ( 3 files changed, 47 insertions(+), 3 deletions(-))

modular-magician · 2023-01-15T20:44:42Z

Tests analytics

Total tests: 2419
Passed tests 2163
Skipped tests: 252
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccDataLossPreventionJobTrigger_dlpJobTriggerUpdateExample2|TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitPercentageExample|TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample

modular-magician · 2023-01-15T20:45:57Z

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerUpdateExample2[Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitPercentageExample[Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerBigqueryRowLimitExample[Debug log]

All tests passed
View the build log or the debug log for each test

shuyama1

LGTM! Only some nit-picks. Thank you for the work!

shuyama1 · 2023-01-17T20:28:03Z

mmv1/templates/terraform/examples/dlp_job_trigger_data_catalog_output.tf.erb

+	parent = "projects/<%= ctx[:test_env_vars]['project'] %>"
+	description = "Description"
+	display_name = "Displayname"
+
+	triggers {
+		schedule {
+			recurrence_period_duration = "86400s"
+		}
+	}
+
+	inspect_job {
+		inspect_template_name = "fake"
+		actions {
+			publish_findings_to_cloud_data_catalog {
+			}
+		}
+		storage_config {
+			big_query_options {
+				table_reference {
+				    project_id = "project"
+				    dataset_id = "dataset"
+				    table_id = "table_to_scan"
+				}
+
+				rows_limit_percent = 50
+				sample_method = "RANDOM_START"
+			}
+		}
+	}
+}


nit: spacing is a bit off here - we use spaces instead of tabs for resource configuration for tests

shuyama1 · 2023-01-17T20:28:17Z

mmv1/templates/terraform/examples/dlp_job_trigger_scc_output.tf.erb

+	parent = "projects/<%= ctx[:test_env_vars]['project'] %>"
+	description = "Description"
+	display_name = "Displayname"
+
+	triggers {
+		schedule {
+			recurrence_period_duration = "86400s"
+		}
+	}
+
+	inspect_job {
+		inspect_template_name = "fake"
+		actions {
+			publish_summary_to_cscc {
+            }
+		}
+		storage_config {
+			big_query_options {
+				table_reference {
+				    project_id = "project"
+				    dataset_id = "dataset"
+				    table_id = "table_to_scan"
+				}
+
+				rows_limit_percent = 50
+				sample_method = "RANDOM_START"
+			}
+		}
+	}
+}


nit: spacing here

modular-magician · 2023-01-19T19:57:33Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 94 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 2 files changed, 94 insertions(+), 4 deletions(-))
TF Validator: Diff ( 3 files changed, 47 insertions(+), 3 deletions(-))

modular-magician · 2023-01-19T20:46:54Z

Tests analytics

Total tests: 2425
Passed tests 2170
Skipped tests: 253
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccRegionInstanceGroupManager_stateful|TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample

modular-magician · 2023-01-19T20:51:06Z

Tests failed during RECORDING mode:
TestAccRegionInstanceGroupManager_stateful[Error message] [Debug log]
TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

shuyama1

Failing tests are unrelated! Thank you for the work!

shuyama1

Whoops, I just realize that we didn't include the newly added examples to the terraform.yaml file, so no tests or doc examples is generated. Sorry for missing this earlier! I thought I saw related tests passed at some point but I guess those were only existing ones. Some examples of how to refer test/doc examples in the terraform.yaml file at https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/products/dlp/terraform.yaml#L76
Plus, I think we can fairly skip generating these two examples in the web docs by adding skip_docs: true in the terraform.yaml file. Please let me know if you have any question generating the tests. Thanks!

modular-magician · 2023-01-22T20:48:54Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 216 insertions(+), 4 deletions(-))
Terraform Beta: Diff ( 3 files changed, 216 insertions(+), 4 deletions(-))
TF Validator: Diff ( 3 files changed, 47 insertions(+), 3 deletions(-))

modular-magician · 2023-01-22T21:38:57Z

Tests analytics

Total tests: 2425
Passed tests 2169
Skipped tests: 253
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccDataLossPreventionJobTrigger_dlpJobTriggerSccOutputExample|TestAccDataLossPreventionJobTrigger_dlpJobTriggerDataCatalogOutputExample|TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample

modular-magician · 2023-01-22T21:41:53Z

Tests passed during RECORDING mode:
TestAccDataLossPreventionJobTrigger_dlpJobTriggerSccOutputExample[Debug log]
TestAccDataLossPreventionJobTrigger_dlpJobTriggerDataCatalogOutputExample[Debug log]

Tests failed during RECORDING mode:
TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

shuyama1

Thank you!

GoogleCloudPlatform#6855) Co-authored-by: Shuya Ma <87669292+shuyama1@users.noreply.github.com> Fixes hashicorp/terraform-provider-google#11861

Add two outputs for dlp job trigger: outputting to SCC and DataCatalog

c46b52b

modular-magician requested a review from shuyama1 November 21, 2022 19:58

shuyama1 reviewed Nov 28, 2022

View reviewed changes

Add extra fields to allow empty nested object for SCC and Data Catalo…

1d2f89f

…g outputs

modular-magician requested a review from shuyama1 December 12, 2022 20:13

shuyama1 requested changes Dec 13, 2022

View reviewed changes

mmv1/templates/terraform/examples/dlp_job_trigger_bigquery_row_limit.tf.erb Outdated Show resolved Hide resolved

mmv1/templates/terraform/examples/dlp_job_trigger_bigquery_row_limit_percentage.tf.erb Outdated Show resolved Hide resolved

Make fields in test snake case

3dbd952

Co-authored-by: Shuya Ma <87669292+shuyama1@users.noreply.github.com>

modular-magician requested a review from shuyama1 December 13, 2022 20:28

shuyama1 requested changes Dec 13, 2022

View reviewed changes

Add back in exactly_one_of to dlp outputs

18cd4a3

modular-magician requested a review from shuyama1 January 15, 2023 19:38

shuyama1 approved these changes Jan 17, 2023

View reviewed changes

modular-magician requested a review from shuyama1 January 19, 2023 19:35

Fix spaces

a777a12

Wallace99 force-pushed the dlp-more-outputs branch from 623a74e to a777a12 Compare January 19, 2023 19:41

shuyama1 approved these changes Jan 19, 2023

View reviewed changes

shuyama1 requested changes Jan 19, 2023

View reviewed changes

Add tests to terraform.yaml

646208b

modular-magician requested a review from shuyama1 January 22, 2023 20:32

shuyama1 approved these changes Jan 24, 2023

View reviewed changes

shuyama1 merged commit c98f227 into GoogleCloudPlatform:main Jan 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add two outputs for dlp job trigger: outputting to SCC and DataCatalog #6855

Add two outputs for dlp job trigger: outputting to SCC and DataCatalog #6855

Wallace99 commented Nov 21, 2022 •

edited by shuyama1

Loading

modular-magician commented Nov 21, 2022

modular-magician commented Nov 21, 2022

shuyama1 left a comment

Wallace99 commented Nov 28, 2022

Wallace99 commented Dec 10, 2022

shuyama1 commented Dec 12, 2022

shuyama1 commented Dec 13, 2022

modular-magician commented Dec 13, 2022

modular-magician commented Dec 13, 2022

modular-magician commented Dec 13, 2022

shuyama1 commented Dec 13, 2022

modular-magician commented Dec 13, 2022

modular-magician commented Dec 13, 2022

modular-magician commented Dec 13, 2022

shuyama1 left a comment

Wallace99 commented Dec 13, 2022

shuyama1 commented Dec 15, 2022

Wallace99 commented Dec 20, 2022

shuyama1 commented Dec 27, 2022

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

shuyama1 left a comment •

edited

Loading

shuyama1 Jan 17, 2023

shuyama1 Jan 17, 2023

modular-magician commented Jan 19, 2023

modular-magician commented Jan 19, 2023

modular-magician commented Jan 19, 2023

shuyama1 left a comment

shuyama1 left a comment •

edited

Loading

modular-magician commented Jan 22, 2023

modular-magician commented Jan 22, 2023

modular-magician commented Jan 22, 2023

shuyama1 left a comment

Add two outputs for dlp job trigger: outputting to SCC and DataCatalog #6855

Add two outputs for dlp job trigger: outputting to SCC and DataCatalog #6855

Conversation

Wallace99 commented Nov 21, 2022 • edited by shuyama1 Loading

modular-magician commented Nov 21, 2022

modular-magician commented Nov 21, 2022

shuyama1 left a comment

Choose a reason for hiding this comment

Wallace99 commented Nov 28, 2022

Wallace99 commented Dec 10, 2022

shuyama1 commented Dec 12, 2022

shuyama1 commented Dec 13, 2022

modular-magician commented Dec 13, 2022

Diff report

modular-magician commented Dec 13, 2022

Tests analytics

Action taken

modular-magician commented Dec 13, 2022

shuyama1 commented Dec 13, 2022

modular-magician commented Dec 13, 2022

Diff report

modular-magician commented Dec 13, 2022

Tests analytics

Action taken

modular-magician commented Dec 13, 2022

shuyama1 left a comment

Choose a reason for hiding this comment

Wallace99 commented Dec 13, 2022

shuyama1 commented Dec 15, 2022

Wallace99 commented Dec 20, 2022

shuyama1 commented Dec 27, 2022

modular-magician commented Jan 15, 2023

Diff report

modular-magician commented Jan 15, 2023

Tests analytics

Action taken

modular-magician commented Jan 15, 2023

shuyama1 left a comment • edited Loading

Choose a reason for hiding this comment

shuyama1 Jan 17, 2023

Choose a reason for hiding this comment

shuyama1 Jan 17, 2023

Choose a reason for hiding this comment

modular-magician commented Jan 19, 2023

Diff report

modular-magician commented Jan 19, 2023

Tests analytics

Action taken

modular-magician commented Jan 19, 2023

shuyama1 left a comment

Choose a reason for hiding this comment

shuyama1 left a comment • edited Loading

Choose a reason for hiding this comment

modular-magician commented Jan 22, 2023

Diff report

modular-magician commented Jan 22, 2023

Tests analytics

Action taken

modular-magician commented Jan 22, 2023

shuyama1 left a comment

Choose a reason for hiding this comment

Wallace99 commented Nov 21, 2022 •

edited by shuyama1

Loading

shuyama1 left a comment •

edited

Loading

shuyama1 left a comment •

edited

Loading