-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(glue): Job construct #12506
feat(glue): Job construct #12506
Commits on Jul 11, 2021
-
feat(aws-glue): add Job construct (#12443)
closes #12443 - Add GlueVersion, WorkerType, JobCommandName enum-like classes for future-proofing these values - Add PythonVersion as a normal enum as it's less likely to change - Add Job construct - CloudFormation base resource is a complex resource with almost all properties being optional and allowed in specific combinations based on command, glue version and worker types - It supports multiple job types e.g. ETL, Streaming and Python Shell and languages e.g. Scala and Python. This requires different combinations of props - One possibility is to try to create different constructs for the different types but I've opted to do just one base on resembling CloudFormation which does not prevent us from introducing more specialized types later - Documentation is not accurate, based on integ test deployments which led me to not having any assertions on valid props combinations - https://docs.aws.amazon.com/glue/latest/dg/add-job.html - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html - Not exposing some CloudFormation props - allocatedCapacity as it's deprecated in favour of maxCapacity - logUri which is not used and is reserved for future use
Configuration menu - View commit details
-
Copy full SHA for c158a05 - Browse repository at this point
Copy the full SHA c158a05View commit details -
support job's event rules and rule-based metrics
- glue job does not emit success or failure metrics to cloudwatch metrics. Instead, it emits events to cloudwatch events - add JobEventState enum for known job states in https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#glue-event-types - add utility methods to create event rules and cloudwatch metrics based on those roles supporting the most common use cases; success, failure and timeout
Configuration menu - View commit details
-
Copy full SHA for 9ea48b6 - Browse repository at this point
Copy the full SHA 9ea48b6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 638129c - Browse repository at this point
Copy the full SHA 638129cView commit details -
Configuration menu - View commit details
-
Copy full SHA for b99dbce - Browse repository at this point
Copy the full SHA b99dbceView commit details -
Configuration menu - View commit details
-
Copy full SHA for 06b4d55 - Browse repository at this point
Copy the full SHA 06b4d55View commit details
Commits on Jul 27, 2021
-
- introduce JobBase to contain common logic between Job and Import - JobBase now handles CloudWatch Event Rules and Rule-based metrics methods - address the rename comments - use the private constructor + static of() pattern where relevant - Make JobProps.glueVersion required
Configuration menu - View commit details
-
Copy full SHA for 6bcdad8 - Browse repository at this point
Copy the full SHA 6bcdad8View commit details -
Configuration menu - View commit details
-
Copy full SHA for e0eb941 - Browse repository at this point
Copy the full SHA e0eb941View commit details -
Configuration menu - View commit details
-
Copy full SHA for b2d866f - Browse repository at this point
Copy the full SHA b2d866fView commit details
Commits on Jul 28, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 29ff157 - Browse repository at this point
Copy the full SHA 29ff157View commit details
Commits on Jul 29, 2021
-
Configuration menu - View commit details
-
Copy full SHA for c615c8e - Browse repository at this point
Copy the full SHA c615c8eView commit details
Commits on Jul 30, 2021
-
Configuration menu - View commit details
-
Copy full SHA for c31d7e4 - Browse repository at this point
Copy the full SHA c31d7e4View commit details -
Configuration menu - View commit details
-
Copy full SHA for f05afa7 - Browse repository at this point
Copy the full SHA f05afa7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a46dc5 - Browse repository at this point
Copy the full SHA 5a46dc5View commit details -
Configuration menu - View commit details
-
Copy full SHA for f59d688 - Browse repository at this point
Copy the full SHA f59d688View commit details
Commits on Aug 2, 2021
-
Configuration menu - View commit details
-
Copy full SHA for fba208d - Browse repository at this point
Copy the full SHA fba208dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6032807 - Browse repository at this point
Copy the full SHA 6032807View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b74200 - Browse repository at this point
Copy the full SHA 8b74200View commit details -
Configuration menu - View commit details
-
Copy full SHA for 98c19df - Browse repository at this point
Copy the full SHA 98c19dfView commit details
Commits on Aug 11, 2021
-
introduce JobExecutable and refactor accordingly
- introduce JobExecutable - drop JobCommand - rename JobCommandName to JobType
Configuration menu - View commit details
-
Copy full SHA for 933ebcd - Browse repository at this point
Copy the full SHA 933ebcdView commit details
Commits on Aug 12, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 6b3d31e - Browse repository at this point
Copy the full SHA 6b3d31eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f02798 - Browse repository at this point
Copy the full SHA 2f02798View commit details -
Configuration menu - View commit details
-
Copy full SHA for 52ae192 - Browse repository at this point
Copy the full SHA 52ae192View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe08089 - Browse repository at this point
Copy the full SHA fe08089View commit details -
Configuration menu - View commit details
-
Copy full SHA for 55b5ee4 - Browse repository at this point
Copy the full SHA 55b5ee4View commit details
Commits on Aug 13, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 3235ae7 - Browse repository at this point
Copy the full SHA 3235ae7View commit details -
Configuration menu - View commit details
-
Copy full SHA for ee74a8a - Browse repository at this point
Copy the full SHA ee74a8aView commit details
Commits on Aug 20, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 1367813 - Browse repository at this point
Copy the full SHA 1367813View commit details
Commits on Aug 24, 2021
-
Configuration menu - View commit details
-
Copy full SHA for c3c9e80 - Browse repository at this point
Copy the full SHA c3c9e80View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ff9d01 - Browse repository at this point
Copy the full SHA 0ff9d01View commit details -
Configuration menu - View commit details
-
Copy full SHA for cd27ef7 - Browse repository at this point
Copy the full SHA cd27ef7View commit details
Commits on Aug 30, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 4353f22 - Browse repository at this point
Copy the full SHA 4353f22View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb7e376 - Browse repository at this point
Copy the full SHA fb7e376View commit details -
Configuration menu - View commit details
-
Copy full SHA for 818be70 - Browse repository at this point
Copy the full SHA 818be70View commit details
Commits on Aug 31, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 7072a6f - Browse repository at this point
Copy the full SHA 7072a6fView commit details -
Configuration menu - View commit details
-
Copy full SHA for e860f4d - Browse repository at this point
Copy the full SHA e860f4dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1068229 - Browse repository at this point
Copy the full SHA 1068229View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf896fa - Browse repository at this point
Copy the full SHA bf896faView commit details -
Co-authored-by: Ben Chaimberg <youppi3@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 9f5f85a - Browse repository at this point
Copy the full SHA 9f5f85aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f587c9 - Browse repository at this point
Copy the full SHA 0f587c9View commit details -
- move AssetCode directory check to constructor - rename SparkUIConfig to SparkUILoggingLocation - rename filter to quiet - rename jobStateRule to onStateChange and make it public - check no reserved arguments used - rename JobProps.numberOfWorkers to JobProps.workerCount - move JobExecutableConfig to the bottom of the file - rename PythonJobExecutableProps to PythonSparkJobExecutableProps
Configuration menu - View commit details
-
Copy full SHA for 87dee59 - Browse repository at this point
Copy the full SHA 87dee59View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ded0f2 - Browse repository at this point
Copy the full SHA 0ded0f2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 82c1d98 - Browse repository at this point
Copy the full SHA 82c1d98View commit details -
Configuration menu - View commit details
-
Copy full SHA for c81e736 - Browse repository at this point
Copy the full SHA c81e736View commit details
Commits on Sep 1, 2021
-
apply suggestions from comments
Co-authored-by: Ben Chaimberg <youppi3@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 8ce0fe8 - Browse repository at this point
Copy the full SHA 8ce0fe8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2ed79e1 - Browse repository at this point
Copy the full SHA 2ed79e1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1eb9c20 - Browse repository at this point
Copy the full SHA 1eb9c20View commit details -
Configuration menu - View commit details
-
Copy full SHA for 16e3265 - Browse repository at this point
Copy the full SHA 16e3265View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7df0c1f - Browse repository at this point
Copy the full SHA 7df0c1fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8884ad7 - Browse repository at this point
Copy the full SHA 8884ad7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 13b03e8 - Browse repository at this point
Copy the full SHA 13b03e8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2b5f47e - Browse repository at this point
Copy the full SHA 2b5f47eView commit details
Commits on Sep 3, 2021
-
Configuration menu - View commit details
-
Copy full SHA for bc82d60 - Browse repository at this point
Copy the full SHA bc82d60View commit details -
Configuration menu - View commit details
-
Copy full SHA for 550b919 - Browse repository at this point
Copy the full SHA 550b919View commit details -
Configuration menu - View commit details
-
Copy full SHA for 70b3e24 - Browse repository at this point
Copy the full SHA 70b3e24View commit details -
Configuration menu - View commit details
-
Copy full SHA for a500cf2 - Browse repository at this point
Copy the full SHA a500cf2View commit details
Commits on Sep 8, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 32ba2ae - Browse repository at this point
Copy the full SHA 32ba2aeView commit details -
- update README - use const instead of let and beforeEach where relevant - assert on stack assets metadata - add test case of 2 jobs reusing asset - other minor things
Configuration menu - View commit details
-
Copy full SHA for babd3ec - Browse repository at this point
Copy the full SHA babd3ecView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7379066 - Browse repository at this point
Copy the full SHA 7379066View commit details -
Configuration menu - View commit details
-
Copy full SHA for b62a868 - Browse repository at this point
Copy the full SHA b62a868View commit details -
Configuration menu - View commit details
-
Copy full SHA for 80c3f15 - Browse repository at this point
Copy the full SHA 80c3f15View commit details -
Configuration menu - View commit details
-
Copy full SHA for 094929c - Browse repository at this point
Copy the full SHA 094929cView commit details -
Configuration menu - View commit details
-
Copy full SHA for f01c0be - Browse repository at this point
Copy the full SHA f01c0beView commit details -
fix issues identified trying to run jobs from integ tests
- fix bug with code bucket permissions - Glue role did not have permissions to read from the code bucket which manifested as DeniedAccess exceptions when attempting to run jobs - Update Code.bind to take a Job instead of Construct so it can grant it the necessary read permissions - Python shell failed saying Glue 2.0 is not supported - Console UI for adding a new Python shell jobs shows Glue 1.0 as the only supported version - Modified tests to reject using Glue 0.9 but didn't exclude Glue 2.0 or 3.0 for any future support by the new versions
Configuration menu - View commit details
-
Copy full SHA for cd2d2ee - Browse repository at this point
Copy the full SHA cd2d2eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for ea32eab - Browse repository at this point
Copy the full SHA ea32eabView commit details -
Configuration menu - View commit details
-
Copy full SHA for 98cc575 - Browse repository at this point
Copy the full SHA 98cc575View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0328615 - Browse repository at this point
Copy the full SHA 0328615View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6bb18eb - Browse repository at this point
Copy the full SHA 6bb18ebView commit details