Handle Tekton pipeline level variables

[jinchihe]

/feature

Which issue is resolved by this Pull Request:

The PR is going to resolve the problem:

If we used the tekton pipeline level variables, such as context.pipelineRun.name or context.pipelineRun.namespace, see detail of tekton variable here .
Currently even if we defined those variables in inputs section of components, after compiled by dsl, the variables is not loaded, and only replace value in containerOps. In fact, that's pipeline level variables, we need to pass that by params, cannot use ``context.pipelineRun.namespace` in containerOps directly, see the example for details: https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/using_context_variables.yaml

Description of your changes:

If there is tekton pipeline level variables in container, then

1. Replease the $(context.pipeline/pipelineRun).*) to $(params.var_name), for example: context.pipelineRun.name -> $(params.pipelineRun-name)
2. Add $(context.pipeline/pipelineRun).*) to pipeline_run['spec']['pipelineSpec']['tasks']['params']
3. Add $(context.pipeline/pipelineRun).*) to pipeline_run['spec']['pipelineSpec']['tasks']['taskSpec']['params']

The the container can use the variables normally.

Environment tested:

• Python Version (use python --version):
• Tekton Version (use tkn version):
• Kubernetes Version (use kubectl version):
• OS (e.g. from /etc/os-release):

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

  PR titles examples:

  • fix(frontend): fixes empty page. Fixes #1234
    Use fix to indicate that this PR fixes a bug.
  • feat(backend): configurable service account. Fixes #1234, fixes #1235
    Use feat to indicate that this PR adds a new feature.
  • chore: set up changelog generation tools
    Use chore to indicate that this PR makes some changes that users don't need to know.
  • test: fix CI failure. Part of #1234
    Use part of to indicate that a PR is working on an issue, but shouldn't close the issue when merged.

• Do you want this pull request (PR) cherry-picked into the current release branch?

  If yes, use one of the following options:

  • (Recommended.) Ask the PR approver to add the cherrypick-approved label to this PR. The release manager adds this PR to the release branch in a batch update.
  • After this PR is merged, create a cherry-pick PR to add these changes to the release branch. (For more information about creating a cherry-pick PR, see the Kubeflow Pipelines release guide.)

[jinchihe]

/cc @fenglixa @Tomcli

Would you please review? Great thanks.

[Tomcli]

thanks @jinchihe, can you update this test to include the edge case you have in this PR? You can then run make unit_test GENERATE_GOLDEN_YAML=True to regenerate the tekton pipelinerun yaml. Thanks

https://github.com/kubeflow/kfp-tekton/blob/master/sdk/python/tests/compiler/testdata/parallel_join_with_argo_vars.py

[jinchihe]

@Tomcli Thanks for your reviewing, I would like to add a new test since that's new features, instead of argo variable replacing.

I already added a new tests for the change, and update sdk/FEATURES.md to describe that. Thanks.

[jinchihe]

@Tomcli @fenglixa This is ready for another look.

By the way, shall we release a new minor version (0.5.1?) for recently changes? Thanks.

[jinchihe]

/cc @vincent-pli

[k8s-ci-robot]

@jinchihe: GitHub didn't allow me to request PR reviews from the following users: vincent-pli.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @vincent-pli

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jinchihe]

Addressed @fenglixa comments, also updated below tests to verify that:
https://github.com/kubeflow/kfp-tekton/blob/master/sdk/python/tests/compiler/testdata/parallel_join_with_argo_vars.py

[fenglixa]

/lgtm

@Tomcli , is this OK to your side?

[Tomcli]

it looks good to me
@ckadner will this be conflicting with your merge to kfp 1.3?

[ckadner]

Thanks @jinchihe -- I just have a few small wording corrections. Otherwise this looks good.

[ckadner]

@ckadner will this be conflicting with your merge to kfp 1.3?

Nope.

[jinchihe]

Thanks @ckadner for your reviewing. Addressed your comments.

[fenglixa]

Thanks @jinchihe
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fenglixa, jinchihe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [fenglixa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fenglixa]

/lgtm