1.12 Running in-cluster e2es no longer respects service account RBAC rules.

[timothysc]

Is this a BUG REPORT:
Sonobuoy runs the kubernetes e2e's differently then test-infra. It constructs an in-cluster config vs. being handed a kubeconfig from the infra.

In 1.12 something has broken this behavior, I'm still digging into the details.

What happened:
Run tests from within the cluster error occurs -
https://paste.fedoraproject.org/paste/D559nyX6SlGhA3tLPtWt5A

What you expected to happen:
Works fine, just like it had in 1.11

How to reproduce it (as minimally and precisely as possible):
Run tests from within the cluster given proper RBAC rules.

Anything else we need to know?:
Discovered when updating sonobuoy for 1.12.

/sig testing
/kind bug
/cc @kubernetes/sig-cluster-lifecycle @dims @BenTheElder @kubernetes/cncf-conformance-wg

[dims]

One data point, the in-cluster-client-configuration example seems to work (but was missing RBAC step):
#69232

[neolit123]

is there a known time when it last worked?

[timothysc]

is there a known time when it last worked?

1.11 worked fine

One data point, the in-cluster-client-configuration example seems to work (but was missing RBAC step):
#69232

Sonobuoy's legacy in-cluster config works as well.

The e2e test suite relies on a single utility wrapper that seems to not be tested.
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go#L2051

I'm still digging through the e2e setup, I haven't touched it since 1.9-ish timeframe.

[neolit123]

1.11 worked fine

needs a binary search bisect between 1.11 and 1.12. :\

[akutz]

I'm definitely interested in this as VMware runs e2e tests in cluster. We're still handing the tests a kubeconfig from the outside, but the plan is to move to the same model as used by Sonobuoy.

[darxkies]

The token field in /tmp/kubeconfig-xxx is missing. BearerToken in client-go is apparently empty. Probably caused by this: #67359

[darxkies]

FYI, the following command can trick sonobuoy/e2e 0.12 into running the tests but beware it is a hack:

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

[dims]

@timothysc i don't have proof, but looking through swathes of stuff, this one stood out - #66684 wanna try a local test in your environment with that removed?

[neolit123]

@liztio shared a possible solution in the sig-cluster-life chat, @timothysc can confirm if that was a fix for the same thing or perhaps something else.

[dims]

i'll take back my comment about #66684. trying to recreate the scenario i was able to print out the tmp file with the in-cluster config info.

1.11 - https://paste.fedoraproject.org/paste/8vHwWGJhgzrUkSlccBmVMg/raw
1.12 - https://paste.fedoraproject.org/paste/83My3KRhnny8kvSV9CWslQ/raw

Looks like the token information is missing in 1.12 exactly as @darxkies mentioned. So i support the darxkies's theory that #67359 is causing it.

[timothysc]

Thanks @darxkies I verified that #69272 does fix.

[dims]

/reopen

[k8s-ci-robot]

@dims: Reopening this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[stealthybox]

@timothysc @chuckha What should we do with Sonobuoy now that this is merged and cherry-picked?
Do we need to wait for 1.12.1 to drop before we can use that test suite?
Will we use that test suite for 1.12.0 clusters?

[timothysc]

Closing, I've been verifying with release-1.12 and with latest patches is works now.

[liggitt]

/close
per #69234 (comment)

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

/close
per #69234 (comment)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.