feat(glue-alpha): add job run queuing to Glue job #31830

jdebuseamazon · 2024-10-21T09:36:58Z

Issue # (if applicable)

Reason for this change

Job within @aws-cdk/aws-glue-alpha does not currently include the jobRunQueuingEnabled property of the CfnJob within aws-cdk-lib/aws-glue. Setting this property currently requires a raw override.

Description of changes

Added jobRunQueuingEnabled to construction properties for Job, along with validation that this is not enabled when execution class is flexible and/or maxRetries exceeds zero (see).

Description of how you validated changes

Unit tests and an integration test.

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

jdebuseamazon · 2024-10-21T09:52:08Z

Exemption Request (for ❌ Features must contain a change to a README file): the change only introduced an optional construction property, and for consistency with existing similar construction properties such as maxCapacity and maxRetries no details were included in the README file

badmintoncryer · 2024-10-29T09:06:50Z

@jdebuseamazon Thank you for your contribution! I think it would be helpful to add a README description for this feature.

The descriptions for maxCapacity and minCapacity are necessary, but this should not be addressed in this PR.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

badmintoncryer

Thank you for your contribution!

I would appreciate it if you could provide links regarding the need for a max retry of 0 and the fact that queueing cannot be enabled with the FLEX ExecutionClass.

badmintoncryer · 2024-10-29T22:32:20Z

packages/@aws-cdk/aws-glue-alpha/test/job.test.ts

+    describe('enabling job run queuing', () => {
+      beforeEach(() => {
+        job = new glue.Job(stack, 'Job', {
+          ...defaultProps,
+          jobRunQueuingEnabled: true,
+        });
+      });
+
+      test('should set JobRunQueuingEnabled', () => {
+        Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', {
+          JobRunQueuingEnabled: true,
+        });
+      });
+    });


This description may be sufficient.

Suggested change

describe('enabling job run queuing', () => {

beforeEach(() => {

job = new glue.Job(stack, 'Job', {

...defaultProps,

jobRunQueuingEnabled: true,

});

});

test('should set JobRunQueuingEnabled', () => {

Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', {

JobRunQueuingEnabled: true,

});

});

});

test('enabling job run queuing', () => {

job = new glue.Job(stack, 'Job', {

...defaultProps,

jobRunQueuingEnabled: true,

});

Template.fromStack(stack).hasResourceProperties('AWS::Glue::Job', {

JobRunQueuingEnabled: true,

});

});

badmintoncryer · 2024-10-29T22:34:03Z

packages/@aws-cdk/aws-glue-alpha/test/job.test.ts

+    describe('validation for jobRunQueuingEnabled and maxRetries', () => {
+      test('jobRunQueuingEnabled with maxRetries > 0 should throw', () => {


If there is only one test within a describe block, I think a single test block is sufficient.

badmintoncryer · 2024-10-29T22:39:01Z

packages/@aws-cdk/aws-glue-alpha/lib/job.ts

@@ -743,6 +756,10 @@ export class Job extends JobBase {
      throw new Error('Both workerType and workerCount must be set');
    }

+    if (props.jobRunQueuingEnabled === true && props.maxRetries !== undefined && props.maxRetries > 0) {


Please add condition !Token.isUnresolved(props.maxRetries) to the if statement.

This will enable support when maxRetries is token.

Suggested change

if (props.jobRunQueuingEnabled === true && props.maxRetries !== undefined && props.maxRetries > 0) {

if (props.jobRunQueuingEnabled === true && props.maxRetries !== undefined && Token.isUnresolved(props.maxRetries) && props.maxRetries > 0) {

jdebuseamazon · 2024-10-30T09:36:07Z

Thank you for your contribution!

I would appreciate it if you could provide links regarding the need for a max retry of 0 and the fact that queueing cannot be enabled with the FLEX ExecutionClass.

No problem at all, very happy to contribute, and many thanks for reviewing. The restrictions on max retry and FLEX execution class are documented here (in the Considerations section)

badmintoncryer

Thank you for the fix! The documentation you provided was also very helpful.

By merging updated base branch as needed, the "needs-maintainer-review" label will be applied. After that, please wait for the maintainer's review😁

paulhcsun

Thanks for the contribution @jdebuseamazon! And thank you for helping to review @badmintoncryer!

paulhcsun · 2024-11-04T18:30:43Z

@Mergifyio update

mergify · 2024-11-04T18:30:47Z

update

☑️ Nothing to do

#commits-behind > 0 [📌 update requirement]
-closed [📌 update requirement]
-conflict [📌 update requirement]
queue-position = -1 [📌 update requirement]

aws-cdk-automation · 2024-11-04T19:00:38Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 55a91a1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2024-11-04T19:01:07Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions · 2024-11-04T19:01:28Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

jdebuseamazon added 2 commits October 18, 2024 16:32

Added job run queuing to Glue job

26adac9

Merge branch 'aws:main' into jdebuseamazon/glue-job-run-queuing-enabled

62e31a4

github-actions bot added the beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK label Oct 21, 2024

aws-cdk-automation requested a review from a team October 21, 2024 09:37

github-actions bot added feature-request A feature should be added or improved. p2 labels Oct 21, 2024

aws-cdk-automation previously requested changes Oct 21, 2024

View reviewed changes

jdebuseamazon changed the title ~~feat(aws-glue-alpha): add job run queuing to Glue job~~ feat(glue-alpha): add job run queuing to Glue job Oct 21, 2024

aws-cdk-automation added pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Oct 21, 2024

github-actions bot mentioned this pull request Oct 28, 2024

Weekly PR metrics report #31915

Closed

Merge branch 'main' into jdebuseamazon/glue-job-run-queuing-enabled

4e86a1e

Added job run queuing to README.md

0438cb6

badmintoncryer suggested changes Oct 29, 2024

View reviewed changes

Test simplification, added support for maxRetries as token

e38801e

aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Oct 30, 2024

badmintoncryer approved these changes Oct 30, 2024

View reviewed changes

Merge branch 'main' into jdebuseamazon/glue-job-run-queuing-enabled

ba7e719

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Oct 30, 2024

github-actions bot mentioned this pull request Nov 1, 2024

Monthly PR metrics report #31972

Closed

Merge branch 'main' into jdebuseamazon/glue-job-run-queuing-enabled

55a91a1

paulhcsun approved these changes Nov 4, 2024

View reviewed changes

mergify bot merged commit 5fca268 into aws:main Nov 4, 2024
14 checks passed

github-actions bot locked as resolved and limited conversation to collaborators Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(glue-alpha): add job run queuing to Glue job #31830

feat(glue-alpha): add job run queuing to Glue job #31830

jdebuseamazon commented Oct 21, 2024

aws-cdk-automation left a comment

jdebuseamazon commented Oct 21, 2024

badmintoncryer commented Oct 29, 2024

badmintoncryer left a comment

badmintoncryer Oct 29, 2024

badmintoncryer Oct 29, 2024

badmintoncryer Oct 29, 2024

jdebuseamazon commented Oct 30, 2024

badmintoncryer left a comment •

edited

Loading

paulhcsun left a comment

paulhcsun commented Nov 4, 2024

mergify bot commented Nov 4, 2024

aws-cdk-automation commented Nov 4, 2024

mergify bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

		describe('validation for jobRunQueuingEnabled and maxRetries', () => {
		test('jobRunQueuingEnabled with maxRetries > 0 should throw', () => {

	if (props.jobRunQueuingEnabled === true && props.maxRetries !== undefined && props.maxRetries > 0) {
	if (props.jobRunQueuingEnabled === true && props.maxRetries !== undefined && Token.isUnresolved(props.maxRetries) && props.maxRetries > 0) {

feat(glue-alpha): add job run queuing to Glue job #31830

feat(glue-alpha): add job run queuing to Glue job #31830

Conversation

jdebuseamazon commented Oct 21, 2024

Issue # (if applicable)

Reason for this change

Description of changes

Description of how you validated changes

Checklist

aws-cdk-automation left a comment

Choose a reason for hiding this comment

jdebuseamazon commented Oct 21, 2024

badmintoncryer commented Oct 29, 2024

badmintoncryer left a comment

Choose a reason for hiding this comment

badmintoncryer Oct 29, 2024

Choose a reason for hiding this comment

badmintoncryer Oct 29, 2024

Choose a reason for hiding this comment

badmintoncryer Oct 29, 2024

Choose a reason for hiding this comment

jdebuseamazon commented Oct 30, 2024

badmintoncryer left a comment • edited Loading

Choose a reason for hiding this comment

paulhcsun left a comment

Choose a reason for hiding this comment

paulhcsun commented Nov 4, 2024

mergify bot commented Nov 4, 2024

☑️ Nothing to do

aws-cdk-automation commented Nov 4, 2024

AWS CodeBuild CI Report

mergify bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

badmintoncryer left a comment •

edited

Loading