Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(sdk): enable dependency-free runtime install of kfp #9886

Merged
merged 14 commits into from
Aug 24, 2023
Merged

feat(sdk): enable dependency-free runtime install of kfp #9886

merged 14 commits into from
Aug 24, 2023

Conversation

connor-mccarthy
Copy link
Member

@connor-mccarthy connor-mccarthy commented Aug 16, 2023 •
edited
Loading

Description of your changes:
Compiles Python components and structures the KFP SDK such that the KFP SDK will require no additional runtime dependencies (excluding the standard library backport of typing via typing-extensions for components running Python 3.7 and Python 3.8).

This:

  • improves task startup latency
  • reduces errors associated with dependency conflicts between the KFP SDK and user code
  • reduces vulnerability to runtime bugs introduced by transitive dependencies

Checklist:

@google-oss-prow
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@connor-mccarthy connor-mccarthy force-pushed the dep-free-kfp-runtime branch 3 times, most recently from 89a1c33 to 4e705b7 Compare August 16, 2023 20:05
@connor-mccarthy
Copy link
Member Author

/test all

@connor-mccarthy connor-mccarthy force-pushed the dep-free-kfp-runtime branch 2 times, most recently from f89e170 to 63cf8c7 Compare August 16, 2023 23:21
@connor-mccarthy connor-mccarthy marked this pull request as ready for review August 16, 2023 23:41
chensun
chensun reviewed Aug 22, 2023
View reviewed changes
Copy link
Member

@chensun chensun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

sdk/python/kfp/dsl/__init__.py Outdated Show resolved Hide resolved
sdk/runtime_tests/__init__.py Outdated Show resolved Hide resolved
@google-oss-prow google-oss-prow bot added lgtm and removed lgtm labels Aug 22, 2023
@connor-mccarthy
Copy link
Member Author

/hold

@connor-mccarthy
Copy link
Member Author

/hold

@connor-mccarthy
Copy link
Member Author

/unhold

@connor-mccarthy
Copy link
Member Author

/test all

@google-oss-prow
Copy link

@connor-mccarthy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
kubeflow-pipelines-sdk-execution-tests d0027c8 link false /test kubeflow-pipelines-sdk-execution-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

chensun
chensun approved these changes Aug 24, 2023
View reviewed changes
Copy link
Member

@chensun chensun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

Thanks!

@google-oss-prow google-oss-prow bot added the lgtm label Aug 24, 2023
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit cf0e0cf into kubeflow:master Aug 24, 2023
1 check passed
stijntratsaertit pushed a commit to stijntratsaertit/kfp that referenced this pull request Feb 16, 2024
@connor-mccarthy @stijntratsaertit
feat(sdk): enable dependency-free runtime install of kfp (kubeflow#9886)
0c448b4 
* handle problematic imports

* separate runtime and compile-time symbols

* wrap kfp/__init__.py imports unavailable at runtime

* update component factory + tests

* add runtime tests

* add --no-deps flag to component factory

* update release notes

* clean up

* handle containerized python components

* update golden snapshots

* update component_factory unit tests

* respond to review feedback

* fix runtime test and compilation logic

* update tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chensun chensun chensun approved these changes

@gkcalat gkcalat Awaiting requested review from gkcalat

@jlyaoyuli jlyaoyuli Awaiting requested review from jlyaoyuli

Assignees

@chensun chensun

Labels
approved lgtm size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@connor-mccarthy @chensun