Dataflow flex-templates resource

[geojaz]

Release Note Template for Downstream PRs (will be copied)

`google_dataflow_flex_template_job`

We'd like to get the beta provider for terraform to support Dataflow Flex-Templates. This is based on the hand-written dataflow_job resource and appears to work for flex-template CRUD activities.

I will continue to iterate on this next week with some tests and to finish off the docs. I'll ping whn it's good to be reviewed.

FYI @pupamanyu @WillBeebe @cxhercules

[drive-by edit by danawillow:] Fixes hashicorp/terraform-provider-google#6656

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@rileykarson, please review this PR or find an appropriate assignee.

[geojaz]

Upon further thinking, I believe it makes most sense to combine resource_dataflow_job and resource_dataflow_flex_template_job, since they hit the same GCP endpoint (with different params).... I'll refactor the new logic to pull it into resource_dataflow_job.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 144 insertions(+), 19 deletions(-))
Terraform Beta: Diff ( 4 files changed, 144 insertions(+), 19 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 163 insertions(+), 19 deletions(-))
Terraform Beta: Diff ( 4 files changed, 163 insertions(+), 19 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 163 insertions(+), 19 deletions(-))
Terraform Beta: Diff ( 4 files changed, 163 insertions(+), 19 deletions(-))

[rileykarson]

Do you mind showing me how Flex Templates end up using the same endpoint? It seems they've got a different gcloud resource (which are normally strongly linked to endpoints) as well as this CURL example which appears to hit a flex-specific endpoint: https://dataflow.googleapis.com/v1b3/projects/$PROJECT/locations/$LOCATION/flexTemplates

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@emilymye, please review this PR or find an appropriate assignee.

[geojaz]

@rileykarson You're right, I was having a brain fart. I've retracted my proposal to combine these resources and will finish off resource_dataflex_flex_template_job.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 163 insertions(+), 19 deletions(-))
Terraform Beta: Diff ( 4 files changed, 163 insertions(+), 19 deletions(-))

[geojaz]

@rileykarson @emilymye I'd love to get some feedback on the PR now. I can see that I have a failure in the cloudbuild trigger, but I'm unable to see what it is.

[rileykarson]

Overall looking strong- I left some nitpicks and comments about where we'll need version guards. I think we need to pass generation for the downstream to update and for me to run the tests.

[rileykarson]

just curious- is the id of the job expected to change here?

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 64 insertions(+))
Terraform Beta: Diff ( 5 files changed, 322 insertions(+))

[modular-magician]

Oops! It looks like you're using an unknown release-note type in your changelog entries:

• REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 65 insertions(+))
Terraform Beta: Diff ( 5 files changed, 323 insertions(+))

[modular-magician]

Oops! It looks like no changelog entry is attached to this PR. Please include a release note block in the PR body, as described in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md:

```release-note:TYPE
Release note
```

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 65 insertions(+))
Terraform Beta: Diff ( 5 files changed, 294 insertions(+))

[aaltay]

Thank you for adding me. Do you want me to review something specific?

Adding, people who directly work on templates: @azurezyq @arvindram03

[rileykarson]

Just a couple schema issues right now. They're the reason for the failed trigger.

--- FAIL: TestProvider (0.04s)
    provider_test.go:377: err: 1 error occurred:
        	* resource google_dataflow_flex_template_job: No Update defined, must set ForceNew on: []string{"on_delete", "labels"}

[rileykarson]

I believe we want a blank update method defined. That will make Terraform persist the value for on_delete into state. If there's no method at all, it requires this field to recreate the resource.

[rileykarson]

Do you mind rebasing on top of master? Something changed between the commit this was created at and now that causes our CI to fail, probably a setting that's encoded in a file inside the repo. I've had it happen on a few of my PRs as well.

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@danawillow, please review this PR or find an appropriate assignee.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 4 files changed, 65 insertions(+))
Terraform Beta: Diff ( 5 files changed, 295 insertions(+))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=122436"

[rileykarson]

Is this a complete flex template? I get the following error after making the changes noted above: Error: googleapi: Error 400: (165707124cd2c54b): Docker image not specified in the template file., badRequest

[aaltay]

You could link to : https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline for effects of cancel vs drain.

I do not know if destroyed is TF terminology. Dataflow term for this is stopping.

[rileykarson]

See above.

[aaltay]

Dataflow terminology for these two types of jobs are streaming and batch jobs respectively.

[rileykarson]

I'm going to keep this message as-is for consistency with the existing resource at https://www.terraform.io/docs/providers/google/r/dataflow_job.html. If you're able to PR a fix (or just file an issue that would be a big help!

[aaltay]

Suggestion: Replace for other kinds of Dataflow jobs. with for Dataflow batch jobs.

[rileykarson]

See above.

[aaltay]

Doc below says the default is cancel. (The default is "cancelled",)

[rileykarson]

Changing to cancel.

[aaltay]

Why is this line needed L127, read this info from this field already?

[rileykarson]

Doesn't hurt, but doesn't do anything- removed.

[rileykarson]

I copied the changes from this PR into #3772, superseding this one. @geojaz do you mind responding to the googlebot comment when you have the chance there? I squashed your commits into a single one, for ease of reviewability.