tutorial to trigger dataflow jobs using cloud scheduler #1396

zhongchen · 2020-08-11T17:42:12Z

Port the Medium tutorial to the community.

https://jira.gcp.solutions/browse/PUB-2764

ToddKopriva · 2020-08-11T17:44:31Z

The CircleCI tests are failing because the document doesn't use the frontmatter that it needs.

To see what you need to include at the top, see a published document, like this one:
https://github.com/GoogleCloudPlatform/community/edit/master/tutorials/embedded-c-getting-started/index.md

jpatokal

A few suggestions for adding detail. In step-by-step tutorials like this, it's good to be as explicit as possible and list out even steps that might be "obvious".

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/index.md

...s/schedule-dataflow-jobs-with-cloud-scheduler/schedule-dataflow-jobs-with-cloud-scheduler.md

jpatokal · 2020-08-12T10:12:52Z

...s/schedule-dataflow-jobs-with-cloud-scheduler/schedule-dataflow-jobs-with-cloud-scheduler.md

+![Set up your cloud scheduler](set_up_the_cloud_scheduler.png)
+
+
+If you use Terraform, here is one example to define a scheduler.


Tutorials usually avoid giving multiple paths. Should the user use the Console (above), or should they use Terraform? If TF, please give the command they would use to deploy this into their project.

sure. I will stick with the TF solution here.

I'm OK with the Terraform approach, but there's a lot of setup involved which I think we need to cover.

I've forked your branch, templatized the TF a bit more and added the basic steps here:
https://github.com/jpatokal/community/tree/zhong-cloud-scheduler-dataflow-tutorial

Specific commits:
jpatokal@488f91e
5136fd2 d268bba

I don't have the right to commit directly into this PR, but maybe you can pull the commit above from my repo?

thanks! I have merged your changes into your branch.

...s/schedule-dataflow-jobs-with-cloud-scheduler/schedule-dataflow-jobs-with-cloud-scheduler.md

ToddKopriva · 2020-08-12T13:54:28Z

Thanks for the review @jpatokal .

@zhongchen , I'll wait until you have resolved these review comments before I begin the editorial and production review.

ToddKopriva · 2020-08-14T21:30:18Z

@jpatokal , let me know when the review comments are resolved to your satisfaction.

zhongchen · 2020-08-19T17:48:30Z

@ToddKopriva Can you start providing some feedback as well?

jpatokal · 2020-08-20T13:41:58Z

@zhongchen Apologies for the delay, the notifications for this went into my personal acct (oops). See comments above.

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/scheduler-dataflow-demo/cloudbuild.yaml

jpatokal · 2020-08-20T14:51:01Z

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/scheduler-dataflow-demo/terraform/main.tf

+
+  http_target {
+    http_method = "POST"
+    uri = "https://dataflow.googleapis.com/v1b3/projects/${var.project_id}/locations/${var.region}/templates:launch?gcsPath=gs://zhong-gcp/templates/dataflow-demo-template"


I'm getting 403 PERMISSION_DENIED from Scheduler when invoking this? (URL updated to my own bucket, of course.) There's a Dataflow Step INFO log for "dataflow.jobs.create" with authorizationInfo: granted: true, which would imply that DF is OK, but no further logs.

I updated the doc to use Terraform to create the SA and set the permission. I tried to run the step in a cloud console and it worked for me.

Tested, and the Cloud Build SA cannot create/modify/act as other SAs or schedule jobs by default. Please direct the user to https://cloud.google.com/cloud-build/docs/securing-builds/configure-access-for-cloud-build-service-account#granting_a_role_using_the_iam_page and ask them to add these three roles:

Service Accounts > Service Account Admin

Service Accounts > Service Account User

Cloud Scheduler > Cloud Scheduler Admin

But even after granting all these, assigning the extra roles to the successfully created new SA fails:

Step #0 - "Terraform init": Error: Batch "iam-project-hybrid-prom modifyIamPolicy" for request "Create IAM Members roles/dataflow.admin serviceAccount:scheduler-dataflow-demo@hybrid-prom.iam.gserviceaccount.com for "project \"hybrid-prom\""" returned error: Error retrieving IAM policy for project "hybrid-prom": googleapi: Error 403: The caller does not have permission, forbidden

...which makes no sense to me since roles/iam.serviceAccountAdmin includes iam.serviceAccounts.getIamPolicy. terraform graph also shows that the dependency is correctly recognized and the mods only made after the SA exists:

"[root] google_project_iam_member.cloud-scheduler-dataflow" -> "[root] google_service_account.cloud-scheduler-demo"
"[root] google_project_iam_member.cloud-scheduler-gcs" -> "[root] google_service_account.cloud-scheduler-demo"

Step #0 - "Terraform init": google_service_account.cloud-scheduler-demo: Creating...
Step #0 - "Terraform init": google_service_account.cloud-scheduler-demo: Creation complete after 2s [id=projects/hybrid-prom/serviceAccounts/scheduler-dataflow-demo@hybrid-prom.iam.gserviceaccount.com]
Step #0 - "Terraform init": google_project_iam_member.cloud-scheduler-dataflow: Creating...
Step #0 - "Terraform init": google_project_iam_member.cloud-scheduler-gcs: Creating...
[errors]

What am I missing? Any ideas?

Cloud Scheduler Admin Dataflow Admin Service Account User Project IAM Admin

Maybe you are missing Project IAM Admin Role to manage IAM binding? I reduced the roles to the above four and it worked for me.

Can you try it as well?

A-ha! Project IAM Admin was the missing role, it works now!

Two more tweaks:

You need Service Account Admin to create the new SA

You don't need Dataflow Admin (because it's the new SA that kicks off the job)

Also, in the gcloud builds command you need to use $BUCKET_NAME (not $BUCKET), otherwise it tries to access gs://gs://bucket.

I actually removed the bucket_name completely. Currently bucket represents the bucket name. In the scripts, I have added back the gs:// prefix for all the bucket paths.

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/index.md

…zhongchen/community into zhong-cloud-scheduler-dataflow-tutorial

jpatokal · 2020-08-25T07:00:06Z

Working now! Zhong, please add in the two last tweaks, and @ToddKopriva , you'll be good to go.

ToddKopriva · 2020-08-25T15:16:40Z

Thanks, @jpatokal .

@zhongchen , I'll wait until you have made the suggested change, and then I'll begin the editorial and production process.

…zhongchen/community into zhong-cloud-scheduler-dataflow-tutorial

zhongchen · 2020-08-25T16:52:53Z

@ToddKopriva I have addressed all the comments.

ToddKopriva · 2020-09-08T22:27:22Z

https://cloud.google.com/community/tutorials/schedule-dataflow-jobs-with-cloud-scheduler

…latform#1396) * tutorial to trigger dataflow jobs using cloud scheduler * format the tutorial to fix circle ci checks * change the title format * address comments * Templating and step-by-step instructions * Enable APIs * Add template compilation * add the architecture diagram * rename the build script * address comments * minor fixes * address comments * add cloudbuild sa setup * add project iam admin role * add dummy logic for dataflow job * update sa setup * first edit pass during readthrough * second edit pass Co-authored-by: zhong <zhongchen@google.com> Co-authored-by: Todd Kopriva <43478937+ToddKopriva@users.noreply.github.com> Co-authored-by: Jani Patokallio <jani@google.com>

tutorial to trigger dataflow jobs using cloud scheduler

68852e2

google-cla bot added the cla: yes label Aug 11, 2020

ToddKopriva self-requested a review August 11, 2020 17:44

ToddKopriva self-assigned this Aug 11, 2020

zhong added 2 commits August 11, 2020 12:16

format the tutorial to fix circle ci checks

903e5c3

change the title format

f6bd1bb

jpatokal reviewed Aug 12, 2020

View reviewed changes

zhong and others added 2 commits August 12, 2020 10:13

address comments

126b76e

Merge branch 'master' into zhong-cloud-scheduler-dataflow-tutorial

5407372

Merge branch 'master' into zhong-cloud-scheduler-dataflow-tutorial

e62b1f7

Templating and step-by-step instructions

488f91e

Enable APIs

5136fd2

jpatokal reviewed Aug 20, 2020

View reviewed changes

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/scheduler-dataflow-demo/cloudbuild.yaml Show resolved Hide resolved

jpatokal reviewed Aug 20, 2020

View reviewed changes

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/scheduler-dataflow-demo/cloudbuild.yaml Outdated Show resolved Hide resolved

Add template compilation

d268bba

jpatokal reviewed Aug 20, 2020

View reviewed changes

zhong added 4 commits August 23, 2020 10:52

add the architecture diagram

c46d0b2

rename the build script

168295c

address comments

e174675

minor fixes

3752e16

jpatokal reviewed Aug 24, 2020

View reviewed changes

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/index.md Outdated Show resolved Hide resolved

jpatokal reviewed Aug 24, 2020

View reviewed changes

tutorials/schedule-dataflow-jobs-with-cloud-scheduler/index.md Outdated Show resolved Hide resolved

zhong added 3 commits August 24, 2020 10:20

address comments

ff0ec4f

add cloudbuild sa setup

144c1cf

add project iam admin role

eaa5d01

ToddKopriva and others added 3 commits August 24, 2020 13:21

Merge branch 'master' into zhong-cloud-scheduler-dataflow-tutorial

7090f9b

add dummy logic for dataflow job

f28ec2b

Merge branch 'zhong-cloud-scheduler-dataflow-tutorial' of github.com:…

c210975

…zhongchen/community into zhong-cloud-scheduler-dataflow-tutorial

Merge branch 'master' into zhong-cloud-scheduler-dataflow-tutorial

429e060

zhong added 2 commits August 25, 2020 09:51

update sa setup

3135df0

Merge branch 'zhong-cloud-scheduler-dataflow-tutorial' of github.com:…

0c784f6

…zhongchen/community into zhong-cloud-scheduler-dataflow-tutorial

ToddKopriva added 2 commits August 31, 2020 11:13

first edit pass during readthrough

af1b6d6

second edit pass

ebddd2d

ToddKopriva approved these changes Aug 31, 2020

View reviewed changes

Merge branch 'master' into zhong-cloud-scheduler-dataflow-tutorial

52c8cf1

ToddKopriva merged commit 7249510 into GoogleCloudPlatform:master Aug 31, 2020

zhongchen deleted the zhong-cloud-scheduler-dataflow-tutorial branch August 31, 2020 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tutorial to trigger dataflow jobs using cloud scheduler #1396

tutorial to trigger dataflow jobs using cloud scheduler #1396

zhongchen commented Aug 11, 2020

ToddKopriva commented Aug 11, 2020

jpatokal left a comment

jpatokal Aug 12, 2020

zhongchen Aug 12, 2020

jpatokal Aug 20, 2020 •

edited

Loading

zhongchen Aug 23, 2020

ToddKopriva commented Aug 12, 2020

ToddKopriva commented Aug 14, 2020

zhongchen commented Aug 19, 2020

jpatokal commented Aug 20, 2020

jpatokal Aug 20, 2020 •

edited

Loading

zhongchen Aug 24, 2020

jpatokal Aug 24, 2020

zhongchen Aug 24, 2020

jpatokal Aug 25, 2020

zhongchen Aug 25, 2020

jpatokal commented Aug 25, 2020

ToddKopriva commented Aug 25, 2020

zhongchen commented Aug 25, 2020

ToddKopriva commented Sep 8, 2020

		![Set up your cloud scheduler](set_up_the_cloud_scheduler.png)


		If you use Terraform, here is one example to define a scheduler.

tutorial to trigger dataflow jobs using cloud scheduler #1396

tutorial to trigger dataflow jobs using cloud scheduler #1396

Conversation

zhongchen commented Aug 11, 2020

ToddKopriva commented Aug 11, 2020

jpatokal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpatokal Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ToddKopriva commented Aug 12, 2020

ToddKopriva commented Aug 14, 2020

zhongchen commented Aug 19, 2020

jpatokal commented Aug 20, 2020

jpatokal Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpatokal commented Aug 25, 2020

ToddKopriva commented Aug 25, 2020

zhongchen commented Aug 25, 2020

ToddKopriva commented Sep 8, 2020

jpatokal Aug 20, 2020 •

edited

Loading

jpatokal Aug 20, 2020 •

edited

Loading