Add pkg/authn/kubernetes #1234

imjasonh · 2022-01-06T19:35:44Z

This takes the Kubernetes client-go parts from k8schain, removes the
forked-upstream credentialprovider bits and rewrites them to include
only exactly what we need.

It also reimplements k8schain in terms of this new K8s helper, and of
wrapped cred helpers using authn.NewKeychainFromHelper.

cc @vdemeester

codecov-commenter · 2022-01-06T19:37:38Z

Codecov Report

Merging #1234 (e0c07cf) into main (c370fbd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1234   +/-   ##
=======================================
  Coverage   73.77%   73.77%           
=======================================
  Files         111      111           
  Lines        8183     8183           
=======================================
  Hits         6037     6037           
  Misses       1549     1549           
  Partials      597      597

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c370fbd...e0c07cf. Read the comment docs.

imjasonh · 2022-01-06T19:43:08Z

hack/presubmit.sh failed in CI:

# sigs.k8s.io/json/internal/golang/encoding/json
Error: /home/runner/go/pkg/mod/sigs.k8s.io/json@v0.0.0-20211020170558-c049b76a60c6/internal/golang/encoding/json/encode.go:1249:12: sf.IsExported undefined (type reflect.StructField has no field or method IsExported)
Error: /home/runner/go/pkg/mod/sigs.k8s.io/json@v0.0.0-20211020170558-c049b76a60c6/internal/golang/encoding/json/encode.go:1255:18: sf.IsExported undefined (type reflect.StructField has no field or method IsExported)
~/work/go-containerregistry/go-containerregistry ~/work/go-containerregistry/go-containerregistry
Error: Process completed with exit code 2.

This is because StructField.IsExported was added in Go 1.17 -- the new module specifies go 1.17 which presumably picks up a later compatible sigs.k8s.io/json than is supported by the Go version used in CI. I bumped the CI version used to Go 1.17, since it's not that recent.

The pkg/authn/k8schain module is still go 1.14, maybe we should bump that to 1.17 too while we're at it. Maybe we should bump everything. I don't think it needs to be done in this PR.

imjasonh · 2022-01-06T19:51:26Z

pkg/authn/kubernetes/keychain.go

+				return nil, err
+			}
+			for k, v := range cfg.Auths {
+				m[k] = v


This (and line 119 below) means that later-specified secrets with matching repo/registry keys will overwrite previous ones. New fetches explicitly listed imagePullSecrets before searching for those attached to the SA, so those later ones, if there's a conflict, will win.

We have some options:

don't overwrite -- first matching auth wins

fetch implicit SA secrets before explicitly listed secrets

In any case, we probably need to spell it out better and guard it with tests.

Just to chime in here (I meant to before). If this matches K8s' behavior, then it SGTM. It's unfortunate we don't track that anymore, but it felt like that was only a matter of time anyways.

Well, it turns out the previous behavior was first-one-wins, which I verified with a test on main that I ported to this PR.

mattmoor · 2022-01-06T21:00:18Z

pkg/authn/k8schain/k8schain.go

+//
+// Deprecated: Use pkg/authn/kubernetes.


It is probably worth separating the conversation about Deprecation from this PR, because I don't see the value in forcing all of the folks that are using k8schain to drop it so that we can stop maintaining what is now a very small shim. We can always add these later if we can agree on a path forward for folks.

To me the main win here is cutting out the need for Vincent's fork, which is HUGE, and I totally appreciate all of the work (and arguing!) to get us to this point. When the world stops ending, I owe you some beverages 🍻

Yeah, you're probably right. Done.

jonjohnsonjr · 2022-01-06T21:05:35Z

pkg/authn/kubernetes/go.mod

@@ -0,0 +1,47 @@
+module github.com/google/go-containerregistry/pkg/authn/kubernetes


Can this just live in k8schain rather than having yet another submodule?

I suspect (@imjasonh can correct) this was in an effort to deprecate k8schain, which I think warrants more discussion. So if we push out deprecation for another convo, this SGTM.

I think it's useful to be able to have a K8s-secret-backed Keychain that doesn't also pull in all of k8schain's dependencies, even though they're much smaller and more well behaved now.

vdemeester

@imjasonh just one question, because we remove the files that have //go:build !disable_gcp, are we sure the behavior of the new thing we use instead (some part of go-containerregistry) doesn't do what upstream k8s did (querying a url infinitely if it came to not answering) ?
I am mainly asking to not re-open a bug in tekton (and knative) on openshift 😝. Otherwise looks good and simple 😻

vdemeester · 2022-01-07T08:44:27Z

pkg/authn/k8schain/go.mod

@@ -3,11 +3,16 @@ module github.com/google/go-containerregistry/pkg/authn/k8schain
 go 1.14

 require (
-	github.com/google/go-containerregistry v0.5.2-0.20210609162550-f0ce2270b3b4
-	github.com/vdemeester/k8s-pkg-credentialprovider v1.21.0-1


imjasonh · 2022-01-07T14:31:44Z

@imjasonh just one question, because we remove the files that have //go:build !disable_gcp, are we sure the behavior of the new thing we use instead (some part of go-containerregistry) doesn't do what upstream k8s did (querying a url infinitely if it came to not answering) ? I am mainly asking to not re-open a bug in tekton (and knative) on openshift 😝. Otherwise looks good and simple 😻

Good question.

First, based on my testing with the AWS keychain imported but not used, it doesn't exhibit the 10-minute mystery wait that was observed in Tekton (tektoncd/pipeline#4087). This isn't a guaranteed fix, because to be fair I never saw the mystery wait anyway, even before Tekton snipped it out. Maybe I wasn't testing the right way? 🤷

Secondly, a nice thing about k8schain effectively becoming a shorthand wrapper for N different keychains is that consumers like Tekton/Knative can just choose their own keychains, to remove some, or reorder them, etc. So in this world, if it's a concern, Tekton/Knative or OpenShift Pipelines/Serverless, could replace k8schain.NewNoClient() with the equivalent authn.NewMultiKeychain(default, k8s, google) and remove problematic/unnecessary keychains.

Tekton could even jump through some hoops and retain the build tag logic, and have AWS disabled selectively at compile-time. We can discuss that in the PR where we bring this change into Tekton.

vdemeester · 2022-01-07T15:10:03Z

Secondly, a nice thing about k8schain effectively becoming a shorthand wrapper for N different keychains is that consumers like Tekton/Knative can just choose their own keychains, to remove some, or reorder them, etc. So in this world, if it's a concern, Tekton/Knative or OpenShift Pipelines/Serverless, could replace k8schain.NewNoClient() with the equivalent authn.NewMultiKeychain(default, k8s, google) and remove problematic/unnecessary keychains.

Ah that make complete sense 😬 👍🏼

mattmoor · 2022-01-07T16:08:06Z

Another alternative would be to simply go mod -replace k8schain downstream with a shim that includes the helpers you want. Personally, I have been toying with the idea of doing precisely that (for other reasons), so inlining this shim downstream is also (in some sense) a breaking change 😅

This takes the Kubernetes client-go parts from k8schain, removes the forked-upstream credentialprovider bits and rewrites them to include only exactly what we need. It also reimplements k8schain in terms of this new K8s helper, and of wrapped cred helpers using authn.NewKeychainFromHelper.

…to registry

… don't override

pkg/authn/k8schain/go.mod

pkg/authn/kubernetes/go.mod

hasheddan · 2022-01-13T16:35:47Z

@imjasonh just a heads up, we have previously had issues with the hanging requests in @crossplane and recently had them resurface when folks are using proxies, so definitely happy to see this land. I have some community members running tests of consuming this implementation across a variety of environments (pull through caches, pulling using IRSA, etc.) in crossplane/crossplane#2841. Happy to exercise any other scenarios of interest and report back as helpful. Appreciate the efforts here :)

imjasonh · 2022-01-13T16:45:57Z

@hasheddan that's great to hear! More testers on this will definitely be helpful. ❤️

imjasonh requested review from mattmoor and jonjohnsonjr January 6, 2022 19:35

imjasonh force-pushed the authn-kubernetes branch from 70e2d8c to 6f79f33 Compare January 6, 2022 19:36

imjasonh mentioned this pull request Jan 6, 2022

Add pkg/authn/{amazon,azure,google} #1231

Closed

imjasonh mentioned this pull request Jan 6, 2022

Long delays on first taskrun caused by aws-sdk-go tektoncd/pipeline#4087

Closed

imjasonh commented Jan 6, 2022

View reviewed changes

mattmoor reviewed Jan 6, 2022

View reviewed changes

jonjohnsonjr reviewed Jan 6, 2022

View reviewed changes

imjasonh mentioned this pull request Jan 6, 2022

Don't use k8schain, statically link cloud cred helpers in cosign sigstore/cosign#1279

Merged

vdemeester reviewed Jan 7, 2022

View reviewed changes

imjasonh added 5 commits January 7, 2022 12:37

run presubmit using go 1.17

19a55e9

streamline a bit, check for auths matching repo first, then fallback …

046d56f

…to registry

don't deprecate k8schain just yet, type alias Options

147d78f

Preserve ordering behavior in k8schain -- first matching secret wins,…

1352567

… don't override

imjasonh force-pushed the authn-kubernetes branch from 931cc7b to 1352567 Compare January 7, 2022 17:38

mattmoor approved these changes Jan 7, 2022

View reviewed changes

dprotaso reviewed Jan 7, 2022

View reviewed changes

pkg/authn/k8schain/go.mod Outdated Show resolved Hide resolved

dprotaso reviewed Jan 7, 2022

View reviewed changes

pkg/authn/kubernetes/go.mod Outdated Show resolved Hide resolved

imjasonh added 2 commits January 7, 2022 14:29

pin to k8s 0.22.5

f0bffef

k8schain -> k8s 0.22.5

0f26452

jonjohnsonjr approved these changes Jan 7, 2022

View reviewed changes

go mod download k8s.io/api

e0c07cf

mattmoor merged commit a5c1d03 into google:main Jan 8, 2022

hasheddan mentioned this pull request Jan 13, 2022

Consume the new k8schain implementation in ggcr crossplane/crossplane#2841

Merged

3 tasks

imjasonh mentioned this pull request Jan 14, 2022

Update k8schain tektoncd/pipeline#4488

Merged

5 tasks

justaugustus mentioned this pull request Jan 27, 2022

Bump deps, add script to make it easier #1260

Merged

vdemeester mentioned this pull request Mar 31, 2022

Do not hang on GCE if Metadata Server is disabled vdemeester/k8s-pkg-credentialprovider#18

Closed

halvards mentioned this pull request May 2, 2022

Update authentication documentation google/k8s-digester#8

Merged

natalieparellano mentioned this pull request Jul 26, 2022

Add GCP credential provider buildpacks/lifecycle#887

Draft

zuzzas mentioned this pull request Feb 8, 2023

k8schain doesn't work with multiple Secrets #723

Closed

realshuting mentioned this pull request Feb 8, 2023

[Bug] Kyverno considers only the first ImagePullSecret provided in arguments kyverno/kyverno#4536

Closed

2 tasks

febi881 mentioned this pull request Jun 25, 2024

[Bug] image verification with private registries is not working kyverno/kyverno#3538

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pkg/authn/kubernetes #1234

Add pkg/authn/kubernetes #1234

imjasonh commented Jan 6, 2022

codecov-commenter commented Jan 6, 2022 •

edited

Loading

imjasonh commented Jan 6, 2022

imjasonh Jan 6, 2022

mattmoor Jan 6, 2022

imjasonh Jan 7, 2022

mattmoor Jan 6, 2022

imjasonh Jan 6, 2022

jonjohnsonjr Jan 6, 2022

mattmoor Jan 6, 2022

imjasonh Jan 6, 2022

vdemeester left a comment •

edited

Loading

vdemeester Jan 7, 2022

imjasonh commented Jan 7, 2022

vdemeester commented Jan 7, 2022

mattmoor commented Jan 7, 2022

hasheddan commented Jan 13, 2022

imjasonh commented Jan 13, 2022

		@@ -0,0 +1,47 @@
		module github.com/google/go-containerregistry/pkg/authn/kubernetes

Add pkg/authn/kubernetes #1234

Add pkg/authn/kubernetes #1234

Conversation

imjasonh commented Jan 6, 2022

codecov-commenter commented Jan 6, 2022 • edited Loading

Codecov Report

imjasonh commented Jan 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdemeester left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imjasonh commented Jan 7, 2022

vdemeester commented Jan 7, 2022

mattmoor commented Jan 7, 2022

hasheddan commented Jan 13, 2022

imjasonh commented Jan 13, 2022

codecov-commenter commented Jan 6, 2022 •

edited

Loading

vdemeester left a comment •

edited

Loading