Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KFPv2 support step 1 #226

Merged
merged 68 commits into from
Jun 8, 2024
Merged

KFPv2 support step 1 #226

merged 68 commits into from
Jun 8, 2024

Conversation

roytman
Copy link
Member

@roytman roytman commented Jun 4, 2024 •
edited
Loading

Why are these changes needed?

This is the first step to support KFPv2.
Here we provide shared libraries and pipelines, which can work with KFPv1 and KFPv2.
We have tested the LFPv1 option, KFPv2 was only partially tested.
Next steps are:

  • tests ALL KFPv2 pipelines
  • update documentation
  • work on KFPv2 super pipelines.

Related issue number (if any).

revit13 and others added 25 commits June 2, 2024 10:40
@revit13 @roytman
Initial version.
6108300 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13 @roytman
Comment compute_exec_params_op.
0d4b3f7 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
separate lib for compile only dependencies
b2ad17f 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
merge with dev
cbe07d2 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
add missing files
ea8e8c4 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
remove kfp/kfp_support_lib_v2/src/kfp_support/workflow_support_v2/
b6e280e 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@revit13 @roytman
Update kfp_support_lib_v2.
997a1ed 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13 @roytman
Fixes.
df5ff64 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13 @roytman
More fixes.
228a8a5 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
revert 3 latest commits
77713d0
@roytman
add KFPv2 installation setting
433dcae 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
before rebase
815f0ca 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
add kfp_support_lib/Makefile
eeadb25 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
kfpv1 intermediate impelmentation
e343bdb 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
lib fixes
9fc3070 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
fix import
d67d8a9
@roytman
fix python_appserver_client tests
0417995
@roytman
some fixes
d280247
@revit13
Fixes after testing.
2ae0984 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
update noop kfpv2
96f8dd4
@revit13
Fixes after testing.
a0028b2 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13
Fix compute_exec_params
47c4c68 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13
Minor fix.
8ff9434 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
Merge pull request #223 from revit13/test1
de204fa 
Fixes after testing.
@roytman
add input paramters to the pipeline
97e51c1 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman roytman marked this pull request as draft June 4, 2024 09:06
revit13 and others added 4 commits June 4, 2024 13:58
@revit13
Change kfp_v1_workflow_support.
e5127bd 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
Merge pull request #227 from revit13/lib1
efe3ab1 
Change kfp_v1_workflow_support.
@roytman
remove Dockerfile_v2, update .make.transforms_workflows
8627a46 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
first universal pipeline
245610d 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
revit13 and others added 19 commits June 5, 2024 22:39
@revit13
Adjust tokenization and filter workflows.
5da574b 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
fix tests
2edfd67
@revit13
Fixes after testing.
d4fb2d5 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13
More fixes.
c3930e7 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
fix doc_id and tests
09ea1cc
@revit13
Minor fixes.
d0d2f59 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
fix .make.workflows
5e15040
@revit13
Fixes after testing.
5db777a 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
fix ededup
33a3d97
@roytman
fix ededup2
b31b04c
@roytman
fix ededup3
6e76bd8
@revit13
Adjuest code workflows.
3a07e68 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13
Fixes after testing.
09ba6bc 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13
More fixes.
5a17ebf 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@roytman
fix ededup
a1f759f 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
fix ededup2
36f6d92 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
fix ededup3
fc21bb8 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
fix fdedup
3892937 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
fix fdedup2
bb834ec 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman roytman changed the title KFPv2 support KFPv2 support step 1 Jun 7, 2024
@roytman roytman marked this pull request as ready for review June 7, 2024 05:00
@roytman roytman requested a review from blublinsky June 7, 2024 05:00
blublinsky
blublinsky reviewed Jun 7, 2024
View reviewed changes
Copy link
Collaborator

@blublinsky blublinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good, but I think it needs more work

kfp/kfp_support_lib/Makefile Show resolved Hide resolved
kfp/kfp_support_lib/kfp_v1_workflow_support/Makefile Show resolved Hide resolved
kfp/kfp_support_lib/kfp_v1_workflow_support/Makefile Show resolved Hide resolved
kfp/kfp_support_lib/kfp_v1_workflow_support/Makefile Show resolved Hide resolved
kfp/kfp_support_lib/kfp_v1_workflow_support/pyproject.toml Show resolved Hide resolved
..._support_lib/kfp_v2_workflow_support/src/workflow_support/runtime_utils/remote_jobs_utils.py
logger.warning("All executions failed")
sys.exit(1)
else:
logger.info(f"{n_launches} ot of {len(config_value)} succeeded")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. It does not depend on KFP version

kfp/kfp_support_lib/kfp_v2_workflow_support/test/pipeline_utils_test.py
# wait for completion
status, error = utils.wait_pipeline_completion(run_id=run, wait=10)
assert status.lower() == "succeeded"
assert error == ""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

transforms/code/proglang_select/kfp_ray/proglang_select_wf.py Show resolved Hide resolved
transforms/code/proglang_select/kfp_ray/proglang_select_wf.py Show resolved Hide resolved
transforms/universal/filter/src/local_pipeline.py Outdated Show resolved Hide resolved
@blublinsky
Copy link
Collaborator

It looks like there is more make stuff then the actual code

@roytman
Copy link
Member Author

roytman commented Jun 7, 2024

It looks like there is more make stuff then the actual code

This is the beauty, we can reuse the same pipelines for both versions.

@roytman
fix comments
f8ead56 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
add comments in Makefiles
53f3d3b 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
@roytman
add the non-unique ray cluster ID warning for KFPv2
74647f8 
Signed-off-by: Alexey Roytman <roytman@il.ibm.com>
blublinsky
blublinsky approved these changes Jun 7, 2024
View reviewed changes
Copy link
Collaborator

@blublinsky blublinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good, but I think it needs more work

@roytman roytman merged commit 36d4982 into dev Jun 8, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daw3rd daw3rd daw3rd left review comments

@blublinsky blublinsky blublinsky approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@roytman @blublinsky @daw3rd @revit13