Per-Provider Post-Up Setup

[zmerlynn]

In the kops testing, we're currently disabling NodePort testing, c.f. kubernetes/kops#775 . I just added the -slow tests as well, and there's more of these issue. Essentially, all of the providers do some tiny amount of post-deployment setup to "ready" the cluster for e2e testing, but because kops is outside the normal kube-up framework, all kops deployments are as a customer would have them.

I see a couple of options here:

1. We add a function in the kubetest kops deployment that handles the post-deployment configuration. This could call a kops function, but likely as not, we'd just thunk out to the AWS CLI, or pull in the AWS SDK to kubetest (yay?).
2. We find a better way to test these things that would actually work in an as-deployed environment
3. We tag anything that needs special cluster setup and filter the tag out for kops testing. This has obvious coverage issues, but lets us filter the tag without specially calling out NodePort.

Thoughts from @kubernetes/sig-testing-misc?

[zmerlynn]

cc @justinsb

[spiffxp]

In the abstract, post-up seems like a valid hook. But I question why this wouldn't just be a modification to the existing deployer's "up"

I am firm believer that kubetest -test should "just work" without any additional configuration save for credentials, at least for conformance tests, so as to support the testing of clusters deployed with tools other than kube-up/kops/kube-anywhere

[zmerlynn]

@spiffxp: Then you're seemingly opting for (2) above, which is to say "tests should work on the default setup"?

[zmerlynn]

To be fair, this is my general desire as well - and I believe everything tagged with [Conformance] actually obeys this. I'd almost rather flip this on its head and first implement (3) with a [SpecialSetup] tag, then rewrite the tests if possible (i.e. (3) then (2)).

[spiffxp]

Yeah, I guess I'm opting for 2. Or asking that we find a better way than [TestTags:ParsedViaRegex] to classify these things.

Varying levels of fidelity here:

• allow tests to introspect cluster for its capabilities, and test appropriately?
• allow tests to compare the cluster's cloud provider against a hardcoded list of capabilities? (this seems roughly where we're at re: tests that require ssh)
• same as above but s/cloud-provider/e2e-deployer so it's passed vs. introspected
• create a general purpose tag that indicate non-conformant/non-standard setup (we tried using Feature: for this, it's grown past this scope now)
• mumbles something about feature flags

I'm open to whatever solves kops's immediate need, just pointing out it's not unique.

[zmerlynn]

mumbles something about feature flags

Heh, c.f. kubernetes/kubernetes#2953 .. which we haven't implemented.

[bgrant0607]

FWIW, I made some comments about feature flags in the layers doc:
https://github.com/kubernetes/community/blob/master/contributors/devel/architectural-roadmap.md

[fejta]

/assign @zmerlynn

I believe you added this logic to kubetest? Can we close if so?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[spiffxp]

/area kubetest

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close