KEP-3352: Aggregated Discovery KEP to Alpha

[Jefftree]

• One-line PR description: Add KEP for publishing an aggregated discovery document. This will reduce the number of requests by clients when fetching all api resources from the server.

• Issue link: Aggregated Discovery #3352

• Other comments:

[deads2k]

@Jefftree forgot a push maybe? I see some comments resolved, but not an update on the PR.

[deads2k]

This will likely need to have an input to readiness and you'll need to filter the discovery results by applicable group/version.

[lavalamp]

Yeah it's real bad if the first few clients get an empty doc because this controller hasn't run yet, but it's also bad to delay the cluster startup.

[ardaguclu]

If api server signals it is ready even when discovery aggregation is not completed, this will not only affects clients but also causes flakiness of some e2e tests depends on discovery?

[deads2k]

If api server signals it is ready even when discovery aggregation is not completed, this will not only affects clients but also causes flakiness of some e2e tests depends on discovery?

More importantly, that causes bad behavior of the namespace lifecycle controller.

[lavalamp]

ah yeah that's another thing to consider, how does this affect that controller.

• will clients be able to know that e.g. a specific GV's backing apiserver hasn't been responsive for a while?
• if an apiserver restarts in that situation, how does it build the cache?

[Jefftree]

Added that this will be an input to readiness.

will clients be able to know that e.g. a specific GV's backing apiserver hasn't been responsive for a while?

Yes, added a lastContacted time for all group versions

[deads2k]

Is this still WIP or do you want it?

[lavalamp]

all my comments are minor except for the one about what do we do on apiserver startup.

[lavalamp]

Yeah it's real bad if the first few clients get an empty doc because this controller hasn't run yet, but it's also bad to delay the cluster startup.

[ardaguclu]

Does that mean that client will no longer cache documents as files?

[ardaguclu]

or caching just one file with TTL

[Jefftree]

Yes, we will skip the serverresources.json caching mechanism that is provided by client-go. Instead, we will rely on the httpcache library to automatically cache and use ETags. kubectl uses the diskcache provided by http cache so we should still be writing the cached document to a file.

[deads2k]

looks good in concept.

[deads2k]

There will be some details to work out either here or during implementation of the controller and APIs. @lavalamp is here next week, so I'll leave where that ends up.

PRR lgtm too.

I volunteer to review the implementation for this one. Please ping me on slack as PRs get opened up.

/approve
/assign @lavalamp

[lavalamp]

is there a reason to have the group level? Why not just have all group versions?

[Jefftree]

We need the separation so we can indicate the priority of group-versions within a group.

[lavalamp]

That field is commented out below, is that intentional?

[Jefftree]

Yes, per David's comment, we'll remove the preferredVersion field and sort the Versions list based on priority. The preferredVersion should be at the top of the list.

[Jefftree]

Oh, if you mean that the GroupVersion field is commented out, this was based on #3364 (comment)

[lavalamp]

Well, you can still keep it sorted even if the list was flat, but we can work this out in the code.

[lavalamp]

If it is nil, does that imply that APIResources above is an empty list?

[Jefftree]

Yes, added a note

[lavalamp]

/lgtm
/approve

thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, Jefftree, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [deads2k]
• keps/sig-api-machinery/OWNERS [deads2k,lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[smarterclayton]

Is this a typo? "have not been aggregated"? I expected to see a description of what client visible behavior we'd see for the discovery doc on initialization:

1. Built-ins - always visible?
2. CRDs - only visible once CRDs are loaded, so might be visible?
3. Aggregated APIs - could take materially longer to be visible?

I'd like to see some examples added to the KEP on the outcome for clients to clarify what they see during startup and how they know discovery is incomplete. Will an incomplete discovery doc return a non 200 error message? Will clients be expected to retry if the document is not complete?

[smarterclayton]

Asked in the #octant channel, poked the OpenShift UI folks for what their feedback on watch was (Sam said poll with etag would be enough).

I looked at a couple of places that do discovery:

• https://github.com/kubernetes-client/python/blob/master/kubernetes/base/dynamic/discovery.py is python client - Ansible uses this for kube declarative management
• https://github.com/kubernetes-client/java/blob/master/util/src/main/java/io/kubernetes/client/Discovery.java java, although i can't tell whether this is actually dynamic discovery or not (my java has atrophied)

I suspect that the aggregated api endpoint would be very valuable for removing some code, but latency and the need not to have a complex cache would be the big win. Will poke at others to see if we have any examples of dynamic discovery clients in the ecosystem that would benefit from better discovery as well who would potentially be impacted if we hit unrepresentable things.

[spadgett]

For OpenShift UI, we have the following wants:

1. Improve performance of initial API discovery. Right now, it's slow due to the number of requests, and discovery can block the UI loading. Browsers limit how many parallel requests we can make.
2. Efficiently detect new types so that the UI can react (for instance, when an operator is installed and adds CRDs).

The proposal definitely addresses (1). I think polling with ETags is sufficient for (2). Happy to see this 👍

[kikisdeliveryservice]

Kep.yaml needs to be updated to reflect 1.26.

[smarterclayton]

Some of the nuances we discussed in comments deserve to be in the KEP:

1. Which clients are we targeting primarily: clients that need to frequently request or maintain a discovery cache, like kubectl or web interfaces
2. We want all clients to benefit from this, but our alpha is targeted at solving these problems for those key consumers
3. We have some use cases that may require additional work to correctly support, we should identify those before entry to beta / reach consensus on their utility:
   • Namespace controller needs to ensure that discovery documents are refreshed after the namespace goes into the terminating phase (needs to guarantee that it sees the list of resources that could have been created within the namespace before the namespace is deleted). The controller needs to be able to guarantee discovery doc list happens-after that, which is best solved by letting the namespace controller "wait" for the next refresh. What we can't allow is for the namespace controller to not observe that resource
   • The use of server side caching may break some client side batch loops done with kubectl that need happens-before semantics, we should consider that usecase in kubectl and ensure we have a mitigation
4. Polling being sufficient over watch for most users
5. The API types in meta/v1 should be meta/v1alpha1 to start, not meta/v1