[TEP-0115] Support Artifact Hub in Hub Resolver #5666

QuanZhang-William · 2022-10-19T15:41:15Z

Changes

Part of issues/667.
This commit adds support to resolve catalog from the Artifact Hub and keep current functionality of fetching catalog from the Tekton Hub.

Change 1:

The commit adds a new field type to the hub resolver indicating the type of the Hub to pull the resource from. The value can be set to tekton or artifact. By default, the resolver fetches resources from https://artifacthub.io/ when setting type to artifact.

Change 2:

Prior to this change, the hub resolver only supports pulling resources from the Tekton Hub. This commit updates the default hub type to artifact since the Artifact Hub will be the main entrypoint for Tekton Catalogs in the future.

Change 3:

Prior to this change, the default Tekton Hub URL is: https://api.hub.tekton.dev. This commit removes the default value of the Tekton Hub URL and enforces users to configure their own instance of Tekton Hub since the public instance https://api.hub.tekton.dev will be deprecated after the migration to Artifact Hub is done.

/kind feature

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Has Docs included if any changes are user facing
Has Tests included if any functionality added or changed
Follows the commit message standard
Meets the Tekton contributor standards (including
functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

The Hub Resolver will have a new `type` field to indicate the type of Hub from where to pull the resource. The default hub type is updated from the Tekton Hub to the Artifact Hub. Please see more details in TEP-0115

tekton-robot · 2022-10-19T15:41:17Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

QuanZhang-William · 2022-10-19T15:43:46Z

/kind feature

QuanZhang-William · 2022-10-19T15:44:04Z

/test all

tekton-robot · 2022-10-19T15:49:31Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.8%	2.4

QuanZhang-William · 2022-10-19T17:38:39Z

/test all

tekton-robot · 2022-10-19T17:44:26Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	75.6%	3.2

QuanZhang-William · 2022-10-19T18:07:14Z

/test all

tekton-robot · 2022-10-19T18:12:53Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.8%	2.4

abayer · 2022-10-19T19:51:19Z

Nice!

QuanZhang-William · 2022-10-19T20:56:14Z

/retest

tekton-robot · 2022-10-19T21:02:13Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.8%	2.4

tekton-robot · 2022-10-20T13:18:06Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.8%	2.4

pkg/resolution/resolver/hub/resolver.go

tekton-robot · 2022-10-20T15:00:24Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.8%	2.4

QuanZhang-William · 2022-10-20T15:04:09Z

/retest

tekton-robot · 2022-10-21T16:31:50Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.5%	2.1

examples/v1beta1/taskruns/hub-resolver.yaml

tekton-robot · 2022-10-21T18:55:41Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	74.5%	2.1

jerop · 2022-10-24T14:43:09Z

/retest

jerop · 2022-10-24T15:12:52Z

known flake: #4643

/retest

chuangw6

Thank you @QuanZhang-William !! 👍

pkg/resolution/resolver/hub/resolver.go

pkg/resolution/resolver/hub/resolver_test.go

docs/hub-resolver.md

pkg/resolution/resolver/hub/resolver.go

wlynch

Looks good!

pkg/resolution/resolver/hub/resolver.go

wlynch · 2022-10-24T18:52:40Z

pkg/resolution/resolver/hub/resolver.go

+	resVer := version
+
+	if hubType == ArtifactHubType && len(semVer) == 2 {
+		resVer = version + ".0"


I'm a bit concerned about doing this - i.e. if someone takes this version and tries to fetch the config, will this guarantee to always work? i.e. if artifacthub publishes 0.1, will I always be able to fetch 0.1.0 and will this always map to 0.1? Or could a different 0.1.0 be added after?

There's an argument that maybe the resolver shouldn't try to interpret semver and just always handle versions as-is.

Hi @wlynch, thanks for your input and this is a good question 😀 . The short answer is yes. If artifacthub publishes 0.1, and I will always be able to fetch 0.1.0

The artifact hub will always convert the resource yaml version to semVer (i.e. 0.6 -> 0.6.0) and the Hub uses the semVer to index the corresponding resource.

So technically you can publish a resource with yaml version 0.6 first (which is converted to 0.6.0 in the Hub), and then publish another yaml version of 0.6.0, it just overrides the previous 0.6.0 version in the Hub. But the API will always return the latest 0.6.0

The main reason I put such conversion is for backward compatibility, so that current users can keep version 0.6 in the resolver, and the type is default to artifact

I'd lean towards dropping this for the new artifacthub impl:

semver with a missing minor/patch version generally uses the latest version matching the prefix (i.e. the golang 1.19 docker image refers to the latest patch version 1.19.2) - I'm not sure we want to have the resolver implement this kind of lookup behavior.

I'm worried about version skew for ArtifactHub URLs - e.g. https://artifacthub.io/packages/tekton-task/tekton-catalog-tasks/git-clone/0.8.0 (even though this points to the 0.8 git tag), but https://artifacthub.io/packages/tekton-task/tekton-catalog-tasks/git-clone/0.8 404s.

It'll probably be simpler long term for the resolver to interpret the user provided version tag as-is, so it always matches the upstream state (up to you if you want to preserve the old behavior for compatibility).

The Artifact Hub always converts the resource yaml version and git tag to semVer here (by appending 0 as minor and patch version if missing) and requires the exact match between the 2 after conversion.

For 2, I don't think there is version skew for the Artifact Hub URL since all the valid URLs will use semVer no matter how do you tag the repo, and URLs with simple semver will always return 404s.

Because the resource is indexed by the resource yaml version, we cannot use the latest version matching the prefix logic here (so we can only append 0s).

So it seems to me that the lookup here brings the extra backward compatibility benefit and does not cause any problems (i.e. version 0.6 and version 0.6.0 in the hub resolver always points to the same resource)?

Thoughts? @jerop @abayer @vdemeester?

pkg/resolution/resolver/hub/resolver.go

tekton-robot · 2022-10-24T22:47:53Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	77.6%	5.2

Part of [issues/667]. This commit adds support to resolve catalog resource from the [Artifact Hub] while keeping current functionality of fetching resources from Tekton Hub. - Change 1: The commit adds a new field `type` to the hub resolver indicating the type of the Hub to pull the resource from. The value can be set to `tekton` or `artifact`. By default, the resolver fetches resources from `https://artifacthub.io/` when setting `type` to `" artifact"`, and fetches resources from user's private instance of Tekton Hub when setting `type` to `"tekton"`. - Change 2: Prior to this change, the hub resolver only supports pulling resources from the Tekton Hub. This commit updates the default hub type to `artifact` since the [Artifact Hub][Artifact Hub] will be the main entrypoint for Tekton Catalogs in the future. - Change 3: Prior to this change, the default Tekton Hub URL is: `https://api.hub.tekton.dev`. This commit removes the default value of the Tekton Hub URL and enforces users to configure their own instance of Tekton Hub since the public instance `https://api.hub.tekton.dev` will be deprecated after the migration to Artifact Hub is done. /kind feature [Artifact Hub]: https://artifacthub.io/ [issues/667]: tektoncd/hub#667

tekton-robot · 2022-10-24T23:02:07Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/resolution/resolver/hub/resolver.go	72.4%	77.6%	5.2

QuanZhang-William · 2022-10-25T13:08:03Z

@jerop @abayer @wlynch @chuangw6 . Could you please review the code again. Thanks 😁

test/resolvers_test.go

jerop · 2022-10-25T13:54:27Z

/approve

cc @abayer

wlynch · 2022-10-25T18:33:21Z

Would it be worth adding docs around configuring the different hub impls and the differences between them?

QuanZhang-William · 2022-10-28T16:38:12Z

Would it be worth adding docs around configuring the different hub impls and the differences between them?

Hi @wlynch, I have some documentation here talking about the configuration and migration to the Artifact Hub. Is there thing else you suggest to add?

wlynch

My mistake!

/lgtm

cmd/resolvers/main.go

tekton-robot · 2022-10-28T16:48:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jerop, wlynch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jerop]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tekton-robot requested review from afrittoli and jerop October 19, 2022 15:41

tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 19, 2022

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from 4200449 to 9533601 Compare October 19, 2022 17:34

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from 9533601 to 153e4fa Compare October 19, 2022 18:06

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from 153e4fa to 8d566f0 Compare October 19, 2022 20:55

QuanZhang-William marked this pull request as ready for review October 20, 2022 13:12

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 20, 2022

tekton-robot requested review from lbernick and vdemeester October 20, 2022 13:12

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from 8d566f0 to ee29374 Compare October 20, 2022 14:50

QuanZhang-William commented Oct 20, 2022

View reviewed changes

pkg/resolution/resolver/hub/resolver.go Outdated Show resolved Hide resolved

jerop added this to the Pipelines v0.41 milestone Oct 20, 2022

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from 00d375d to f62bac5 Compare October 21, 2022 16:26

jerop reviewed Oct 21, 2022

View reviewed changes

examples/v1beta1/taskruns/hub-resolver.yaml Outdated Show resolved Hide resolved

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from f62bac5 to e368375 Compare October 21, 2022 18:50

chuangw6 reviewed Oct 24, 2022

View reviewed changes

wlynch requested changes Oct 24, 2022

View reviewed changes

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from e368375 to 4a3f5e4 Compare October 24, 2022 22:42

QuanZhang-William force-pushed the tep-0115-artifact-hub-resolver branch from 4a3f5e4 to dda9d0d Compare October 24, 2022 22:56

jerop reviewed Oct 25, 2022

View reviewed changes

test/resolvers_test.go Show resolved Hide resolved

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 25, 2022

wlynch approved these changes Oct 28, 2022

View reviewed changes

cmd/resolvers/main.go Show resolved Hide resolved

tekton-robot assigned wlynch Oct 28, 2022

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 28, 2022

tekton-robot merged commit 1b941d3 into tektoncd:main Oct 28, 2022

chuangw6 mentioned this pull request Nov 3, 2022

Set ConfigSource in clusterresolver #5687

Merged

7 tasks

michaelsauter mentioned this pull request Nov 4, 2022

Explore Artifact Hub opendevstack/ods-pipeline#620

Open

This was referenced Nov 10, 2022

Set Source information in ResolutionRequest #5522

Closed

Chains needs to be able to "Identify entry point" - SLSA L3 requirement. tektoncd/chains#521

Closed

QuanZhang-William mentioned this pull request Jun 13, 2024

Migrate from Tekton Hub to Artifact Hub tektoncd/hub#667

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TEP-0115] Support Artifact Hub in Hub Resolver #5666

[TEP-0115] Support Artifact Hub in Hub Resolver #5666

QuanZhang-William commented Oct 19, 2022 •

edited

Loading

tekton-robot commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

abayer commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

tekton-robot commented Oct 20, 2022

tekton-robot commented Oct 20, 2022

QuanZhang-William commented Oct 20, 2022

tekton-robot commented Oct 21, 2022

tekton-robot commented Oct 21, 2022

jerop commented Oct 24, 2022

jerop commented Oct 24, 2022

chuangw6 left a comment

wlynch left a comment •

edited

Loading

wlynch Oct 24, 2022

QuanZhang-William Oct 24, 2022

wlynch Oct 25, 2022

QuanZhang-William Oct 27, 2022 •

edited

Loading

QuanZhang-William Oct 27, 2022

tekton-robot commented Oct 24, 2022

tekton-robot commented Oct 24, 2022

QuanZhang-William commented Oct 25, 2022

jerop commented Oct 25, 2022

wlynch commented Oct 25, 2022

QuanZhang-William commented Oct 28, 2022

wlynch left a comment •

edited

Loading

tekton-robot commented Oct 28, 2022

[TEP-0115] Support Artifact Hub in Hub Resolver #5666

[TEP-0115] Support Artifact Hub in Hub Resolver #5666

Conversation

QuanZhang-William commented Oct 19, 2022 • edited Loading

Changes

Submitter Checklist

Release Notes

tekton-robot commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

abayer commented Oct 19, 2022

QuanZhang-William commented Oct 19, 2022

tekton-robot commented Oct 19, 2022

tekton-robot commented Oct 20, 2022

tekton-robot commented Oct 20, 2022

QuanZhang-William commented Oct 20, 2022

tekton-robot commented Oct 21, 2022

tekton-robot commented Oct 21, 2022

jerop commented Oct 24, 2022

jerop commented Oct 24, 2022

chuangw6 left a comment

Choose a reason for hiding this comment

wlynch left a comment • edited Loading

Choose a reason for hiding this comment

wlynch Oct 24, 2022

Choose a reason for hiding this comment

QuanZhang-William Oct 24, 2022

Choose a reason for hiding this comment

wlynch Oct 25, 2022

Choose a reason for hiding this comment

QuanZhang-William Oct 27, 2022 • edited Loading

Choose a reason for hiding this comment

QuanZhang-William Oct 27, 2022

Choose a reason for hiding this comment

tekton-robot commented Oct 24, 2022

tekton-robot commented Oct 24, 2022

QuanZhang-William commented Oct 25, 2022

jerop commented Oct 25, 2022

wlynch commented Oct 25, 2022

QuanZhang-William commented Oct 28, 2022

wlynch left a comment • edited Loading

Choose a reason for hiding this comment

tekton-robot commented Oct 28, 2022

QuanZhang-William commented Oct 19, 2022 •

edited

Loading

wlynch left a comment •

edited

Loading

QuanZhang-William Oct 27, 2022 •

edited

Loading

wlynch left a comment •

edited

Loading