[feature] Adding description to pipeline version

[casassg]

Feature Area

/area frontend
/area backend

What feature would you like to see?

While using KFP, we lack a way to add descriptions to the pipeline version. This would be useful for example to track metadata of the version uploaded (like for example what git sha was this version generated from).

What is the use case or pain point?

Easier integration into CD systems for continous compilation and deployment of pipeline versions.

Is there a workaround currently?

Hard to do, you can modify the description of the main pipeline and append new lines for each version, that said it may eventually overload that column and make it hard to read vs just having it on the version itself.

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[Bobgy]

Thank you for explaining the use case clearly!
This seems helpful to many.
So welcome contributions!

[casassg]

@Bobgy any pointers to how to add a field? Seems I can should add it

pipelines/backend/api/pipeline.proto

Line 324 in cc83e10

message PipelineVersion {

but then not sure how to propagate this across server api and such (or if it needs it somewhere else) or it needs some DB migration. (Im less familiar w the inner workings of KFP sorry)

[Bobgy]

Backend code: https://github.com/kubeflow/pipelines/tree/master/backend
Backend api pipeline version proto:

pipelines/backend/api/pipeline.proto

Line 324 in 7872bd4

message PipelineVersion {

Regenerate code after updating proto, https://github.com/kubeflow/pipelines/tree/master/backend/api#auto-generation-of-go-client-and-swagger-definitions
Pipeline server:

pipelines/backend/src/apiserver/server/pipeline_server.go

Line 332 in 7872bd4

version, err := s.resourceManager.CreatePipelineVersion(request.Version, pipelineFile, common.IsPipelineVersionUpdatedByDefault())

Resource manager:

pipelines/backend/src/apiserver/resource/resource_manager.go

Line 1205 in 7872bd4

func (r *ResourceManager) CreatePipelineVersion(apiVersion *api.PipelineVersion, pipelineFile []byte, updateDefaultVersion bool) (*model.PipelineVersion, error) {

Pipeline store (db layer):

pipelines/backend/src/apiserver/storage/pipeline_store.go

Line 67 in 7872bd4

CreatePipelineVersion(*model.PipelineVersion, bool) (*model.PipelineVersion, error)

UI pipeline details page:
https://github.com/kubeflow/pipelines/blob/master/frontend/src/pages/PipelineDetails.tsx

Here are most of related code you need to touch, feel free to ask for help if you have further questions!

[casassg]

Started backend changes in #6261 . I regenerated most code and seems backend is working (mostly just ran unit tests to ensure data was being written to DB and could be read back). Missing to run some integration tests (need to spend more time to add integration tests).

[Bobgy]

Thank you for implementing the backend API change! Welcome continuing the UI integration, but you can always leave that to a different contributor too.

[casassg]

I'm happy to contribute to UI as well. Using this as a learning opportunity.

However if someone else comes w this issue and wants to implement it, let me know! (I will add to my backlog once backend gets merged, but I dont know when I can focus on this)

[casassg]

Start frontend changes, found out that I missed an endpoint for backend (the HTTP upload one, as those dont use grpc)

[casassg]

Will also start a separate PR to update kfp sdk

[Bobgy]

There's a report the SDK change caused a regression:

    pipeline_id=pipeline_id)
  File "/tmpfs/BUILD_ENV/.venv/lib/python3.7/site-packages/kfp/_client.py", line 1143, in upload_pipeline_version
    description=description,
  File "/tmpfs/BUILD_ENV/.venv/lib/python3.7/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 209, in upload_pipeline_version
    return self.upload_pipeline_version_with_http_info(uploadfile, **kwargs)  # noqa: E501
  File "/tmpfs/BUILD_ENV/.venv/lib/python3.7/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 265, in upload_pipeline_version_with_http_info
    " to method upload_pipeline_version" % key
kfp_server_api.exceptions.ApiTypeError: Got an unexpected keyword argument 'description' to method upload_pipeline_version

[Bobgy]

I'll be taking a look

[ConverJens]

@Bobgy This mismatch is present in the latest released sdk package 1.7.2. Could you build a new patch version with this fixed?

[casassg]

btw, fixed it #6472

[casassg]

Merged. I believe now the only missing part is for this to be released on the next KFP release

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.