Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(glue): PythonRayExecutableProps has innaccurate properties #28625

Merged
merged 4 commits into from
Mar 8, 2024

Conversation

moomindani
Copy link
Contributor

@moomindani moomindani commented Jan 9, 2024

Closes #28570.

  • Added RayExecutableProps which supports s3PythonModules
  • Added check to block extraPythonFiles usage for Ray jobs
  • Added unit tests and integ tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions bot added bug This issue is a bug. repeat-contributor [Pilot] contributed between 3-5 PRs to the CDK effort/medium Medium work item – several days of effort p2 labels Jan 9, 2024
@aws-cdk-automation aws-cdk-automation requested a review from a team January 9, 2024 09:17
@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Jan 9, 2024
Copy link
Contributor

@lpizzinidev lpizzinidev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍

@moomindani
Copy link
Contributor Author

@lpizzinidev Thank you for looking! Could you please approve and merge this if it is okay?

@lpizzinidev
Copy link
Contributor

@moomindani Merging requires a core team member, I'm just a community reviewer 🙂

@aws-cdk-automation
Copy link
Collaborator

This PR has been in the MERGE CONFLICTS state for 3 weeks, and looks abandoned. To keep this PR from being closed, please continue work on it. If not, it will automatically be closed in a week.

@aws-cdk-automation aws-cdk-automation added pr/needs-maintainer-review This PR needs a review from a Core Team Member and removed pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Feb 5, 2024
@kaizencc kaizencc changed the title fix(glue): added RayExecutableProps which supports s3PythonModules fix(glue): PythonRayExecutableProps has innaccurate properties Feb 12, 2024
Copy link
Contributor

@kaizencc kaizencc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @moomindani, I have a few concerns before we merge this PR

*
* @default - no extra python files specified.
*
* @see `--s3-py-modules` in https://docs.aws.amazon.com/glue/latest/dg/author-job-ray-job-parameters.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@see expects a url after the tag, and nothing else

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's zipped utils.py which is used for this integration test. Since s3PythonModules accepts only Zip file, we need to have this.

I have verified that other packages in AWS CDK has similar zip files so I do not think this can be blocker.

@@ -253,7 +269,7 @@ export interface PythonShellExecutableProps extends SharedJobExecutableProps, Py
/**
* Props for creating a Python Ray job executable
*/
export interface PythonRayExecutableProps extends SharedJobExecutableProps, PythonExecutableProps {}
export interface PythonRayExecutableProps extends SharedJobExecutableProps, RayExecutableProps {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens to PythonExecutableProps? a) is it used elsewhere and b) we are not dropping support for any properties right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are dropping support for properties (as the attached issue suggests) it should have been called out in PR description as a BREAKING CHANGE.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a) is it used elsewhere
Yes. There are three job types; Spark, Python shell, and Ray. PythonExecutableProps is used for the first two.

b) we are not dropping support for any properties right?
Right. We are not dropping any or working properties.
Only the property we are dropping is extraPythonFiles which was not working for Ray job. This is not breaking change.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 12, 2024
@mergify mergify bot dismissed kaizencc’s stale review February 14, 2024 07:11

Pull request has been modified.

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 14, 2024
@moomindani
Copy link
Contributor Author

@kaizencc Can you take a look and merge it if there are no further update needed?

Copy link
Contributor

@paulhcsun paulhcsun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me, nice work @moomindani!

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 8b3a29a
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Copy link
Contributor

mergify bot commented Mar 8, 2024

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 7994733 into aws:main Mar 8, 2024
10 checks passed
moomindani added a commit to moomindani/aws-cdk that referenced this pull request Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. effort/medium Medium work item – several days of effort p2 pr/needs-maintainer-review This PR needs a review from a Core Team Member repeat-contributor [Pilot] contributed between 3-5 PRs to the CDK
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(aws-glue-alpha): PythonRayExecutableProps are inaccurate
5 participants