- Fix PyPI/README description #9668
KFP SDK 2.0.0 release notes are distilled to emphasize high-level improvements in major version 2. See preceding 2.x.x pre-release notes for a more comprehensive list.
Also see the KFP SDK v1 to v2 migration guide.
The KFP SDK 2.0.0 release contains features present in the KFP SDK v1's v2 namespace along with considerable additional functionality. A selection of these features include:
- An improved and unified Python-based authoring experience for components and pipelines
- Support for using pipelines as components (pipeline in pipeline)
- Various additional configurations for tasks
- Compilation to an Argo-independent pipeline definition that enables pipelines to be compiled once and run anywhere
- Additonal SDK client functionality
- An improved KFP CLI
- Refreshed user documentation and reference documentation
Selected contributions from pre-releases:
- Support for
@componentdecorator for Python components #6825 - Support for loading v1 and v2 components using
load_component_from_*#6822 - Support importer in KFP v2 #6917
- Add metadata field for
importer#7112 - Add in filter to
list_pipeline_versionsclient method #7223 - Support getting pipeline status in exit handler via
PipelineTaskFinalStatus#7309 - Support v2 KFP API in SDK client
- Enable pip installation from custom PyPI repository #7453, #8871
- Add additional methods to
kfp.client.Client#7239, #7563, #7562, #7463, #7835 - CLI improvements #7547, #7558, #7559, #7560, , #7569, #7567, #7603, #7606, #7607, #7628, #7618
- Add Registry Client #7597, #7763
- Add support for authoring Custom Container Components #8066
- Support setting parallelism in
ParallelFor#8146 - Support using pipelines as components (pipeline in pipeline) #8179, #8204, #8209
- Support using pipeline in exit handlers #8220
- Support
google.-namespaced artifact types #8191, #8232, #8233, #8279 - Support passing upstream outputs to
importermetadata #7660 - Add ability to skip building image when using
kfp component build#8387 - Support single element
thenandelse_arguments toIfPresentPlaceholder#8414 - Enable use of input and output placeholders in f-strings #8494
- Add comments to IR YAML file #8467
- Support fanning-in parameters #8631 and artifacts #8808 from tasks in a
dsl.ParellelForcontext usingdsl.Collected - Support
.ignore_upstream_failure()onPipelineTask#8838 - Support setting cpu/memory requests #9121
- Support additional pipeline placeholders
- Support for platform-specific features via extension libraries #8940, #9140
- Support
display_nameanddescriptionin@dsl.pipelinedecorator #9153 - Extract component input and output descriptions from docstring #9156
- Allow user to specify platform when building container components #9212
See the KFP SDK v1 to v2 migration guide.
- Deprecate compiling pipelines to JSON in favor of compiling to YAML #8179
- Deprecate ability to select
--enginewhen building components #7559 - Deprecate
@dsl.component'soutput_component_fileparameter in favor of compilation via the mainCompiler#7554 - Deprecate client's
*_jobmethods in favor of*_recurring_runmethods #9112 - Deprecate pipeline task
.set_gpu_limitin favor of.set_accelerator_limit#8836 - Deprecate
.add_node_selector_constraintin favor of.set_accelerator_type#8980
- Various changes to dependency versions relative to v1
- Various dependencies removed relative to v1
- Various bug fixes applied in KFP SDK 2.x.x pre-release versions
- Various bug fixes associated with providing backward compatibility for KFP SDK v1
- Use YAML as default serialization format for pipeline IR #7431
- Support Python 3.10 #8186 and 3.11 #8907
- Enable overriding caching options at submission time #7912
- Format file when compiling to JSON #7712
- Allow artifact inputs in pipeline definition #8044
- Support task-level retry policy #7867
- Support multiple exit handlers per pipeline #8088
- Migrate Out-Of-Band (OOB) authentication flow #8262
- CLI
kfp component buildgenerates runtime-requirements.txt #8372 - Throw exception for component parameter named Output #8367
- Block illegal
IfPresentPlaceholderandConcatPlaceholderauthoring #8414 - Fix boolean default value compilation bug #8444
- Fix bug when writing to same file using gcsfuse and distributed training strategy in lightweight/containerized Python components #8544 [alternative fix after #8455 in
kfp==2.0.0b8], #8607 - Pipeline compilation is now triggered from
@pipelinedecorator instead ofCompiler.compile()method. Technically no breaking changes but compilation error could be exposed in a different (and earlier) stage #8179 - Fully support optional parameter inputs by writing
isOptionalfield to IR #8612 - Add support for optional artifact inputs (toward feature parity with KFP SDK v1) #8623
- Fix upload_pipeline method on client when no name is provided #8695
- Enables output definitions when compiling components as pipelines #8848
- Add experiment_id parameter to create run methods #9004
- Refresh KFP SDK v2 user documentation
- Refresh KFP SDK v2 reference documentation
- Fix integer value not allowed as float-typed return #9481
- Change
kubernetesversion requirement fromkubernetes>=8.0.0,<24tokubernetes>=8.0.0,<27#9545 - Fix bug when iterating over upstream task output in nested
dsl.ParallelForloop #9580
- Fix compilation of boolean constant passed to component #9390
- Depends on
kfp-server-api==2.0.0b2#9355
- Allow user to specify platform when building container components #9212
- Fix appengine import error #9323
- Support
display_nameanddescriptionin@dsl.pipelinedecorator #9153 - Extract component input and output descriptions from docstring #9156
- Fix module not found error for containerized python components #9157
pip_index_urlsis now considered also for containerized python component - the urls will be used for Dockerfile generation #8871- Support direct indexing into top-level of artifact metadata struct in Container Components #9131
- Support compiling platform specific features #8940
- Support setting cpu/memory requests. #9121
- Support PIPELINE_ROOT_PLACEHOLDER #9134
- SDK client v2beta1 API integration #9112
- Support submitting pipeline with platform config. #9140
- New SDK client only works with Kubeflow Pipeline v2.0.0-beta.1 and later version #9112
- Deprecate .add_node_selector_constraint in favor of .set_accelerator_type #8980
- Support python 3.11 #8907
- Fix loading non-canonical generic type strings from v1 component YAML (e.g.,
List[str],typing.List[str],Dict[str],typing.Dict[str, str]#9041 - Add experiment_id parameter to create run methods #9004
- Support setting task dependencies via kfp.kubernetes.mount_pvc #8999
- cpu_limit and memory_limit can be optional #8992
- Support fanning-in artifact outputs from a task in a
dsl.ParellelForcontext usingdsl.Collected#8808 - Introduces a new syntax for pipeline tasks to consume outputs from the upstream task while at the same time ignoring if the upstream tasks succeeds or not. #8838
- Deprecate pipeline task
.set_gpu_limitin favor of.set_accelerator_limit#8836
- Enables output definitions when compiling components as pipelines. #8848
- Fix bug when passing data between tasks using f-strings #8879
- Fix environment variable set in component yaml lost during compilation #8885
- Fix attribute error when running Containerized Python Components #8887
- Support fanning-in parameter outputs from a task in a
dsl.ParellelForcontext usingdsl.Collected#8631
- Fix upload_pipeline method on client when no name is provided #8695
- Fully support optional parameter inputs by witing
isOptionalfield to IR #8612 - Add support for optional artifact inputs (toward feature parity with KFP SDK v1) #8623
- Fix bug deserializing v1 component YAML with boolean defaults, struct defaults, and array defaults #8639
- Add comments to IR YAML file #8467
- Unblock valid topologies #8416
- Fix bug when writing to same file using gcsfuse and distributed training strategy in lightweight/containerized Python components #8544 [alternative fix after #8455 in
kfp==2.0.0b8], #8607
- Fix client methods #8507
- Add ability to skip building image when using
kfp component build#8387 - Support single element
thenandelse_arguments toIfPresentPlaceholder#8414 - Enable use of input and output placeholders in f-strings #8494
- Block illegal
IfPresentPlaceholderandConcatPlaceholderauthoring #8414 - Fix boolean default value compilation bug #8444
- Fix bug when writing to same file using gcsfuse and distributed training strategy in lightweight/containerized Python components #8455
- Clarify
PipelineTask.set_gpu_limitreference docs #8477
- Fix NamedTuple output with Dict/List bug #8316
- Fix PyPI typo in cli/component docstring #8361
- Fix "No KFP components found in file" error #8359
- CLI
kfp component buildgenerates runtime-requirements.txt #8372 - Throw exception for component parameter named Output #8367
- Improve KFP SDK reference documentation #8337
- Support
google.-namespaced artifact types #8191, #8232, #8233, #8279 - Support dynamic importer metadata #7660
- Migrate Out-Of-Band (OOB) authentication flow #8262
- Release KFP SDK v2 user documentation draft
- Support parallelism setting in ParallelFor #8146
- Support for Python v3.10 #8186
- Support pipeline as a component #8179, #8204, #8209
- Support using pipeline in exit handlers #8220
- Pipeline compilation is now triggered from
@pipelinedecorator instead ofCompiler.compile()method. Technically no breaking changes but compilation error could be exposed in a different (and earlier) stage. #8179
- Extend upper bound for Kubernetes to <24 in KFP SDK #8173
- Add support for ConcatPlaceholder and IfPresentPlaceholder in containerized component #8145
- Add support for containerized component #8066
- Enable overriding caching options at submission time #7912
- Allow artifact inputs in pipeline definition. #8044
- Support task-level retry policy #7867
- Support multiple exit handlers per pipeline #8088
- Include default registry context JSON in package distribution #7987
- Implement Registry Client #7597, #7763
- Write compiled JSON with formatting (multiline with indentation) #7712
- Add function to sdk client for terminating run #7835
- Re-enable component compilation via @component decorator (deprecated) #7554
- Make CLI output consistent, readable, and usable #7739
- Fix CLI upload pipeline version #7722
- feat(sdk): add
.list_pipeline_versionsand.unarchive_experimentmethods to Client #7563 - Add additional methods to
kfp.client.Client#7562, #7463 - Migrate V1 CLI to V2, with improvements #7547, #7558, #7559, #7560, , #7569, #7567, #7603, #7606, #7607, #7628, #7618
- Accepts
typing-extensions>=4,<5in addition totyping-extensions>=3.7.4,<4#7632 - Remove dependency on
pydantic#7639
- Enable pip installation from custom PyPI repository #7453
- Fix wrong kfp import causes wrong sdk_version being set in pipeline_spec. #7433
- Use YAML as default serialization format for package IR #7431
- Support submitting pipeline IR in yaml format via
kfp.client. #7458 - Add pipeline_task_name to PipelineTaskFinalStatus #7464
- Depends on
kfp-pipeline-spec>=0.1.14,<0.2.0#7464 - Depends on
google-cloud-storage>=2.2.1,<3#7493
- Enable pip installation from custom PyPI repository #7470
- Support getting pipeline status in exit handler. #7483
- No longer require KFP client for kfp components build #7410
- Require google-api-core>=1.31.5, >=2.3.2 #7377
- Depends on
kfp-server-api>=2.0.0a0, <3#7427
- Support passing parameters in v2 using google.protobuf.Value #6804.
- Implement experimental v2
@componentcomponent #6825 - Add load_component_from_* for v2 #6822
- Merge v2 experimental change back to v2 namespace #6890
- Add ImporterSpec v2 #6917
- Add add set_env_variable for Pipeline task #6919
- Add metadata field for importer #7112
- Add in filter to list_pipeline_versions SDK method #7223
- Add
enable_jobmethod to client #7239 - Support getting pipeline status in exit handler. #7309
- Support KFP v2 API in kfp.client #7411
- Remove sdk/python/kfp/v2/google directory for v2, including google client and custom job #6886
- APIs imported from the v1 namespace are no longer supported by the v2 compiler. #6890
- Deprecate v2 compatible mode in v1 compiler. #6958
- Drop support for python 3.6 #7303
- Deprecate v1 code to deprecated folder #7291
- Disable output_component_file temporarily for v2 early release #7390
- Fix importer ignoring reimport setting, and switch to Protobuf.Value for import uri #6827
- Fix display name support for groups #6832
- Fix regression on optional inputs #6905 #6937
- Depends on
google-auth>=1.6.1,<3#6939 - Change otherwise to else in yaml #6952
- Avoid pydantic bug on Union type #6957
- Fix bug for if and concat placeholders #6978
- Fix bug for resourceSpec #6979
- Fix regression on nested loops #6990
- Fix bug for input/outputspec and positional arguments #6980
- Fix importer not using correct output artifact type #7235
- Add verify_ssl for Kubeflow client #7174
- Depends on
typing-extensions>=3.7.4,<5; python_version<"3.9"#7288 - Depends on
google-api-core>=1.31.5, >=2.3.2#7377 - Fix bug that required KFP API server for
kfp components buildcommand to work #7430 - Pass default value for inputs and remove deprecated items in v1 #7405
- kfp.Client uses namespace from initialization if set for the instance context #7056
- Add importer_spec metadata to v1 #7180
- Fix breaking change in Argo 3.0, to define TTL for workflows. Makes SDK incompatible with KFP pre-1.7 versions #7141
- Remove redundant check in set_gpu_limit #6866
- Fix create_runtime_artifact not covering all types. #7168
- Depend on
absl-py>=0.9,<2#7172
- Improve CLI experience for archiving experiments, managing recurring runs and listing resources #6934
- N/A
- N/A
- N/A
- N/A
- Visualizations and metrics do not work with data_passing_methods. #6882
- Fix a warning message. #6911
- Refresh access token only when it expires. #6941
- Fix bug in checking values in _param_values. #6965
- N/A
- N/A
- N/A
- N/A
- N/A
- N/A
- Make
Artifacttype be compatible with any sub-artifact types bidirectionally #6859
- N/A
- N/A
- N/A
- N/A
- N/A
- N/A
- N/A
- Add optional support to specify description for pipeline version #6472.
- New v2 experimental compiler #6803.
- N/A
- N/A
- N/A
- N/A
- Fix the the specified 'mlpipeline-ui-metadata','mlpipeline-metrics' path is overrided by default value #6796
- Fix placeholder mapping error in v2. #6794
- Add
OnTransientErrorto allowed retry policies #6808 - Add optional
filterargument to list methods of KFP client #6748 - Depends on
kfp-pipeline-spec>=0.1.13,<0.2.0#6803
- N/A
- Add functions to sdk client to delete and disable jobs #6754
- N/A
- N/A
- N/A
- N/A
- Require base and target images for components built using
kfp components buildCLI command to be unique #6731 - Try to use
apt-get python3-pipwhen pip does not exist in containers used by v2 lightweight components #6737 - Implement LoopArgument and LoopArgumentVariable v2. #6755
- Implement Pipeline task settings for v2 dsl. #6746
- N/A
- Add v2 placeholder variables #6693
- Add a new command in KFP's CLI,
components, that enables users to manage and build v2 components in a container with Docker #6417 - Add
load_component_from_specfor SDK v1 which brings back the ability to build components directly in python, usingComponentSpec#6690
- N/A
- N/A
- N/A
- N/A
- Fix executor getting None as value when float 0 is passed in. #6682
- Fix function-based components not preserving the namespace of GCPC artifact types. #6702
- Fix
dsl.prefix in component I/O type annotation breaking component at runtime. #6714 - Update v2 yaml format #6661
- Implement v2 PipelineTask #6713
- Fix type_utils #6719
- Depends on
typing-extensions>=3.7.4,<4; python_version<"3.9"#6683 - Depends on
click>=7.1.2,<9#6691 - Depends on
cloudpickle>=2.0.0,<3#6703 - Depends on
typer>=0.3.2,<1.0#6417
- N/A
- N/A
- N/A
- N/A
- N/A
- N/A
- Support artifact types under google namespace #6648
- Fix a couple of bugs that affect nested loops and conditions in v2. #6643
- Add IfPresentPlaceholder and ConcatPlaceholder for v2 ComponentSpec.#6639
- N/A
- Support URI templates with ComponentStore. #6515
- N/A
- N/A
- N/A
- N/A
- Fix duplicate function for
list_pipeline_versions(). #6594 - Support re-use of PVC with VolumeOp. #6582
- When namespace file is missing, remove stack trace so it doesn't look like an error #6590
- Local runner supports additional docker options. #6599
- Fix the error that kfp v1 and v2 compiler failed to provide unique name for ops of the same component. #6600
- N/A
- N/A
- N/A
- N/A
- N/A
- N/A
- Fix component decorator could result in invalid component if
install_kfp_package=False. #6527) - v2 compiler to throw no task defined error. #6545
- Improve output parameter type checking in V2 SDK. #6566
- Use
Annotatedrather thanUnionforInputandOutput. #6573 - Depends on
typing-extensions>=3.10.0.2,<4. #6573
- N/A
- Support container environment variable in v2. #6515
- N/A
- N/A
- N/A
- N/A
- Define PipelineParameterChannel and PipelineArtifactChannel in v2. #6470
- Remove dead code on importer check in v1. #6508
- Fix issue where dict, list, bool typed input parameters don't accept constant values or pipeline inputs. #6523
- Fix passing in "" to a str parameter causes the parameter to receive it as None instead. #6533
- Get short name of complex input/output types to ensure we can map to appropriate de|serializer. #6504
- Fix Optional type hint causing executor to ignore user inputs for parameters. #6541
- Depends on
kfp-pipeline-spec>=0.1.10,<0.2.0#6515 - Depends on
kubernetes>=8.0.0,<19. #6532
- N/A
- Revert: "Add description to upload_pipeline_version in kfp" #6468
- N/A
- N/A
- N/A
- Fix bug in PodSpec that overwrites nodeSelector #6512
- Add Alpha feature notice for local client #6462
- Import mock from stdlib and drop dependency. #6456
- Update yapf config and move it to sdk folder. #6467
- Fix typing issues. #6480
- Load v1 and v2 component yaml into v2 ComponentSpec and convert v1 component spec to v2 component spec #6497
- Format all Python files under SDK folder. #6501
- N/A
- Add support to specify description for pipeline version #6395.
- Add support for schema_version in pipeline #6366
- Add support for enabling service account for cloud scheduler in google client #6013
kfp.componentsno longer imports everything fromkfp.components. For instance,load_component_from_*methods are available only fromkfp.components, but not fromkfp.components.- No more '_path' suffix striping from v2 components.
- N/A
- N/A
- N/A
- Refactor and move v2 related code to under the v2 namespace #6358
- Fix importer not taking output from upstream #6439
- Clean up the unused arg in AIPlatformCient docstring #6406
- Add BaseModel data classes and pipeline saving #6372
- N/A
- Surfaces Kubernetes configuration in container builder #6095
- N/A
- N/A
- N/A
- N/A
- Relaxes the requirement that component inputs/outputs must appear on the command line. #6268
- Fixed the compiler bug for legacy outputs mlpipeline-ui-metadata and mlpipeline-metrics. #6325
- Raises error on using importer in v2 compatible mode. #6330
- Raises error on missing pipeline name in v2 compatible mode. #6332
- Raises warning on container component without command. #6335
- Fixed the issue that SlicedClassificationMetrics, HTML, and Markdown type are not exposed in dsl package. #6343
- Fixed the issue that pip may not be available in lightweight component base image. #6359
- N/A