feat(glue): Job construct #12506

humanzz · 2021-01-14T12:24:56Z

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2021-01-14T12:24:59Z

humanzz · 2021-01-14T12:26:43Z

This is still a WIP since it depends on #12444 and #12450 being merged first. It can be reviewed for comments and below is a summary of what's introduced in this change

Add GlueVersion, WorkerType, JobCommandName enum-like classes for future-proofing these values
Add PythonVersion as a normal enum as it's less likely to change
Add Job construct
- CloudFormation base resource is a complex resource with almost all properties being optional and allowed in specific combinations based on command, glue version and worker types
- It supports multiple job types e.g. ETL, Streaming and Python Shell and languages e.g. Scala and Python. This requires different combinations of props
- One possibility is to try to create different constructs for the different types but I've opted to do just one base on resembling CloudFormation which does not prevent us from introducing more specialized types later
- Documentation is not accurate, based on integ test deployments which led me to not having any assertions on valid props combinations
  - https://docs.aws.amazon.com/glue/latest/dg/add-job.html
  - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html
- Not exposing some CloudFormation props
  - allocatedCapacity as it's deprecated in favour of maxCapacity
  - logUri which is not used and is reserved for future use
support job's event rules and rule-based metrics
- glue job does not emit success or failure metrics to cloudwatch metrics. Instead, it emits events to cloudwatch events
- add JobEventState enum for known job states in https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#glue-event-types
- add utility methods to create event rules and cloudwatch metrics based on those roles supporting the most common use cases; success, failure and timeout
add metric helper method https://docs.aws.amazon.com/glue/latest/dg/monitoring-awsglue-with-cloudwatch-metrics.html
add JobSpecialArgumentNames for glue special parameters
- https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
- https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html

humanzz · 2021-05-27T13:27:20Z

Hi @iliapolo, any hopes of actioning this any time soon?

packages/@aws-cdk/aws-glue/lib/job.ts

closes #12443 - Add GlueVersion, WorkerType, JobCommandName enum-like classes for future-proofing these values - Add PythonVersion as a normal enum as it's less likely to change - Add Job construct - CloudFormation base resource is a complex resource with almost all properties being optional and allowed in specific combinations based on command, glue version and worker types - It supports multiple job types e.g. ETL, Streaming and Python Shell and languages e.g. Scala and Python. This requires different combinations of props - One possibility is to try to create different constructs for the different types but I've opted to do just one base on resembling CloudFormation which does not prevent us from introducing more specialized types later - Documentation is not accurate, based on integ test deployments which led me to not having any assertions on valid props combinations - https://docs.aws.amazon.com/glue/latest/dg/add-job.html - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html - Not exposing some CloudFormation props - allocatedCapacity as it's deprecated in favour of maxCapacity - logUri which is not used and is reserved for future use

- glue job does not emit success or failure metrics to cloudwatch metrics. Instead, it emits events to cloudwatch events - add JobEventState enum for known job states in https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#glue-event-types - add utility methods to create event rules and cloudwatch metrics based on those roles supporting the most common use cases; success, failure and timeout

https://docs.aws.amazon.com/glue/latest/dg/monitoring-awsglue-with-cloudwatch-metrics.html

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html

humanzz · 2021-07-12T13:54:40Z

Apologies for not noticing the review earlier. I am OOTO at the moment but will attempt to progress this soon.

- introduce JobBase to contain common logic between Job and Import - JobBase now handles CloudWatch Event Rules and Rule-based metrics methods - address the rename comments - use the private constructor + static of() pattern where relevant - Make JobProps.glueVersion required

gitpod-io · 2021-07-27T11:54:20Z

Pull request has been modified.

humanzz · 2021-07-27T12:14:39Z

I've addressed the majority of the comments, @iliapolo
There still remains a couple which I've left comments above about

JobCommand and expanding its responsibility to take on the defaultArguments
Some @defaults which I'm not really sure about
anything else I might have missed :)

humanzz · 2021-09-07T21:29:38Z

Hi @BenChaimberg, I feel like we're getting close, one final push (of effort) to get this PR over the line :)

BenChaimberg

love these tests. I think we're at the finish line now

packages/@aws-cdk/aws-glue/README.md

packages/@aws-cdk/aws-glue/test/code.test.ts

packages/@aws-cdk/aws-glue/lib/job.ts

packages/@aws-cdk/aws-glue/test/job.test.ts

packages/@aws-cdk/aws-glue/test/integ.job.ts

- update README - use const instead of let and beforeEach where relevant - assert on stack assets metadata - add test case of 2 jobs reusing asset - other minor things

Pull request has been modified.

packages/@aws-cdk/aws-glue/lib/job.ts

packages/@aws-cdk/aws-glue/test/integ.job.ts

- fix bug with code bucket permissions - Glue role did not have permissions to read from the code bucket which manifested as DeniedAccess exceptions when attempting to run jobs - Update Code.bind to take a Job instead of Construct so it can grant it the necessary read permissions - Python shell failed saying Glue 2.0 is not supported - Console UI for adding a new Python shell jobs shows Glue 1.0 as the only supported version - Modified tests to reject using Glue 0.9 but didn't exclude Glue 2.0 or 3.0 for any future support by the new versions

humanzz · 2021-09-08T17:32:31Z

And now, ready for another review, @BenChaimberg

packages/@aws-cdk/aws-glue/lib/code.ts

packages/@aws-cdk/aws-glue/lib/job-executable.ts

Pull request has been modified.

BenChaimberg

one outstanding comment above

Pull request has been modified.

humanzz · 2021-09-08T22:14:36Z

one outstanding comment above

🏁 ? 😄

BenChaimberg

YAY! A fabulous job and thank you for putting up with my high standards!

aws-cdk-automation · 2021-09-08T22:54:59Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: 6bb18eb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2021-09-08T22:55:27Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions bot added the @aws-cdk/aws-glue Related to AWS Glue label Jan 14, 2021

github-actions bot assigned iliapolo Jan 14, 2021

humanzz mentioned this pull request Jan 15, 2021

(aws-glue): Job construct #12443

Closed

2 tasks

iliapolo mentioned this pull request Jan 15, 2021

feat(glue): add Job L2 construct #11128

Closed

iliapolo changed the title ~~feat(aws-glue): add Job construct (#12443)~~ feat(glue): add Job construct (#12443) Jan 15, 2021

humanzz changed the title ~~feat(glue): add Job construct (#12443)~~ feat(glue): Job construct (#12443) Feb 15, 2021

humanzz changed the title ~~feat(glue): Job construct (#12443)~~ feat(glue): Job construct Feb 15, 2021

iliapolo added p1 effort/medium Medium work item – several days of effort labels Mar 20, 2021

iliapolo previously requested changes Jun 27, 2021

View reviewed changes

iliapolo added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jun 27, 2021

humanzz added 5 commits July 11, 2021 11:15

add metric helper method

638129c

https://docs.aws.amazon.com/glue/latest/dg/monitoring-awsglue-with-cloudwatch-metrics.html

add JobSpecialArgumentNames for glue special parameters

b99dbce

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html

rebase to use Connection and SecurityConfiguration

06b4d55

BenChaimberg removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 12, 2021

Merge branch 'master' into glue-job

e0eb941

BenChaimberg assigned BenChaimberg and unassigned iliapolo Jul 27, 2021

humanzz added 2 commits July 27, 2021 21:52

improve docs

b2d866f

rename JobCommandName constants and JobCommand methods

29ff157

humanzz added 2 commits September 3, 2021 15:32

tweak tests #2

70b3e24

Merge branch 'master' into glue-job

a500cf2

remove role from IJob

32ba2ae

BenChaimberg previously requested changes Sep 8, 2021

View reviewed changes

address some comments

babd3ec

- update README - use const instead of let and beforeEach where relevant - assert on stack assets metadata - add test case of 2 jobs reusing asset - other minor things

humanzz added 2 commits September 8, 2021 13:25

Merge branch 'master' into glue-job

7379066

simplify job.test.ts

b62a868

BenChaimberg reviewed Sep 8, 2021

View reviewed changes

packages/@aws-cdk/aws-glue/lib/job.ts Outdated Show resolved Hide resolved

humanzz added 3 commits September 8, 2021 14:21

simplify testing success/failure/timeout rules and metrics

80c3f15

better handling for extraPythonFiles with non-Python jobs

094929c

update integ.job.ts

f01c0be

humanzz commented Sep 8, 2021

View reviewed changes

packages/@aws-cdk/aws-glue/test/integ.job.ts Outdated Show resolved Hide resolved

humanzz added 2 commits September 8, 2021 18:17

update integ test verification documentation

ea32eab

BenChaimberg previously requested changes Sep 8, 2021

View reviewed changes

packages/@aws-cdk/aws-glue/lib/code.ts Outdated Show resolved Hide resolved

packages/@aws-cdk/aws-glue/lib/code.ts Outdated Show resolved Hide resolved

packages/@aws-cdk/aws-glue/lib/job-executable.ts Outdated Show resolved Hide resolved

update Code.bind signature and PythonShell supported glue versions

98cc575

humanzz requested a review from BenChaimberg September 8, 2021 18:59

BenChaimberg previously requested changes Sep 8, 2021

View reviewed changes

narrow the permissions granted by S3Code

0328615

Merge branch 'master' into glue-job

6bb18eb

BenChaimberg approved these changes Sep 8, 2021

View reviewed changes

mergify bot merged commit fc74110 into aws:master Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(glue): Job construct #12506

feat(glue): Job construct #12506

humanzz commented Jan 14, 2021 •

edited by iliapolo

Loading

gitpod-io bot commented Jan 14, 2021 •

edited

Loading

humanzz commented Jan 14, 2021 •

edited

Loading

humanzz commented May 27, 2021

humanzz commented Jul 12, 2021

gitpod-io bot commented Jul 27, 2021

humanzz commented Jul 27, 2021

humanzz commented Sep 7, 2021

BenChaimberg left a comment

humanzz commented Sep 8, 2021

BenChaimberg left a comment

humanzz commented Sep 8, 2021

BenChaimberg left a comment

aws-cdk-automation commented Sep 8, 2021

mergify bot commented Sep 8, 2021

feat(glue): Job construct #12506

feat(glue): Job construct #12506

Conversation

humanzz commented Jan 14, 2021 • edited by iliapolo Loading

gitpod-io bot commented Jan 14, 2021 • edited Loading

humanzz commented Jan 14, 2021 • edited Loading

humanzz commented May 27, 2021

humanzz commented Jul 12, 2021

gitpod-io bot commented Jul 27, 2021

humanzz commented Jul 27, 2021

humanzz commented Sep 7, 2021

BenChaimberg left a comment

Choose a reason for hiding this comment

humanzz commented Sep 8, 2021

BenChaimberg left a comment

Choose a reason for hiding this comment

humanzz commented Sep 8, 2021

BenChaimberg left a comment

Choose a reason for hiding this comment

aws-cdk-automation commented Sep 8, 2021

AWS CodeBuild CI Report

mergify bot commented Sep 8, 2021

humanzz commented Jan 14, 2021 •

edited by iliapolo

Loading

gitpod-io bot commented Jan 14, 2021 •

edited

Loading

humanzz commented Jan 14, 2021 •

edited

Loading