Allow informers for getting a stream of data instead of chunking

[p0lyn0mial]

Enhancement Description

• One-line enhancement description (can be used as a release note): informers can get a stream of data to prime the caches
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/3157-watch-list/README.md
• Discussion Link:
• Primary contact (assignee): @p0lyn0mial
• Responsible SIGs: sig-apimachinery, sig-scalability
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.32
  • Stable release target (x.y): TBD
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-3157: allow informers for getting a stream of data instead of chunking #3142
    • Update KEP 3157 for milestone 1.27 #3822
  • Code (k/k) update PR(s):
    • WIP: POC introduce API streaming kubernetes#106477
    • e2e/apimachinery: Add an e2e test for the API Streaming feature kubernetes#117277
    • cacher avoid double locking kubernetes#117410
  • Docs (k/website) update PR(s):
    • Streaming lists documentation website#39993
• Beta
  • KEP (k/enhancements) update PR(s):
    • 1.30 KEP 3157 (watch-list): add a new beta criterion #4453
    • 1.32 Update KEP 3157 (watch-list) for milestone 1.32 #4836
  • Code (k/k) update PR(s):
    • 1.30 storage/cacher: ensure the cache is at the Most Recent ResourceVersion when streaming was requested kubernetes#122830
    • 1.31 cleanup: replace ENABLE_CLIENT_GO_WATCH_LIST_ALPHA with WatchListClient gate kubernetes#122791
    • 1.32:
      • client-go/rest/request: decodes initialEventsListBlueprint for watchlist requests kubernetes#127812
      • apiserver/handlers/watch: encode initialEventsListBlueprint with watchEncoder kubernetes#127587
      • cacher: prevents sending events with ResourceVersion < RequiredResourceVersion kubernetes#127500
      • metadata client: add support for API streaming  kubernetes#127388
      • apimachinery/meta/types.go: define InitialEventsListBlueprintAnnotationKey const kubernetes#127319
      • cacher: apply key for initial events only if the call is not recursive kubernetes#127265
      • adds watchListEndpointRestrictions for watchlist requests kubernetes#126996
      • Promote WatchList feature to Beta  kubernetes#128053
  • Docs (k/website) update(s):
    • WIP: Document KEP 3157 (watch-list) for milestone 1.30 website#45283
    • WIP: Add a blog post about using the API streaming (KEP 3157) website#45285
    • WIP: Add a blog post introducing feature gates for client-go website#45286
    • Add feature gate WatchListClient for v1.30 website#47261
    • Document KEP 3157 (watch-list) for milestone 1.32 website#48512
    • Blog post: Enhancing Kubernetes API Server Efficiency with API Streaming website#48513

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig api-machinery
/sig scalability

[gracenng]

Hi @p0lyn0mial ! 1.24 Enhancements team here. Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd. This enhancements is targeting alpha for 1.24
Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo. KEP-3157: allow informers for getting a stream of data instead of chunking #3142
• KEP status is marked as implementable for this release with latest-milestone: 1.24
• KEP has a test plan section filled out.
  - [ ] KEP has up to date graduation criteria. -- not needed for alpha
• KEP has a production readiness review that has been completed and merged into k/enhancements -- PRR not found

The status of this enhancement is track as at risk
Thanks!

[p0lyn0mial]

Hi @gracenng! Many thanks for the heads-up. Could you please double-check the last point? The KEP has the Production Readiness Review Questionnaire. In addition to that, I have filled the required bits for the alpha version - it just hasn't been reviewed yet.

[gracenng]

Hi @p0lyn0mial,
Thanks for the reply. Your PRR Questionaire in the KEP needs to be approved by the PRR team, which will result in a file in
https://github.com/kubernetes/enhancements/tree/master/keps/prod-readiness/
Your sig has not made a folder there yet, please check out the format.

Regarding your second point, I am tracking your KEP files and the first bullet will be checked when the PR is merged. Does that answer your question?

[gracenng]

Hi @p0lyn0mial , 1.24 Enhancements Team here.

Reaching out as we're less than a week away from Enhancement Freeze on Thursday, February 3rd.
There's no update for this enhancement since last checkin, let me know if I missed anything.
Current status is at risk

[p0lyn0mial]

Hi @gracenng, it looks like it won't make it to 1.24. Please leave me some instructions on how to update the KEP so it won't show up on your query.

[gracenng]

No worries I’ll take care of it

…

On Mon, Jan 31, 2022 at 12:42 AM Lukasz Szaszkiewicz < ***@***.***> wrote: Hi @gracenng <https://github.com/gracenng>, it looks like it won't make it to 1.24. Please leave me some instructions on how to update the KEP so it won't show up on your query. — Reply to this email directly, view it on GitHub <#3157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKCRLOZTCK54ONCDRDEXZCTUYZDP3ANCNFSM5MHDCLRQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dingshujie]

@p0lyn0mial how is feature going? we met a lot of server overloading scene(kube-apiserver oom or master oom), looking forward to this feature. if you need help，you can assign some tasks to me.

[JohnRusk]

@Dingshujie BTW, if you have more than a few hundred nodes, then this fix will probably help to reduce API server load kubernetes/kubernetes#109381 . (If you have more than a few thousand nodes it will definitely help). It solves a bug where the kubelets would keep doing the same listing calls over and over again (typically immediately after a restart of the API server). Obvoiusly, other list issues are possible too, so that fix won't solve them all for you. But it might help.

[Dingshujie]

@Dingshujie BTW, if you have more than a few hundred nodes, then this fix will probably help to reduce API server load kubernetes/kubernetes#109381 . (If you have more than a few thousand nodes it will definitely help). It solves a bug where the kubelets would keep doing the same listing calls over and over again (typically immediately after a restart of the API server). Obvoiusly, other list issues are possible too, so that fix won't solve them all for you. But it might help.

@JohnRusk thanks, i already cherry-pick this to 1.23 branch. but this fixs can't slove the situation that custom-defined operator list issues.
and finally expect all list from watch cache

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[wojtek-t]

/remove-lifecycle rotten

[p0lyn0mial]

/milestone v1.26

[rashansmith]

Hello @p0lyn0mial 👋 from the v1.32 Communications Team!
We'd love for you to consider writing a feature blog about your enhancement! Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt-in, let us know and open a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

[AnshumanTripathi]

Hello @p0lyn0mial 👋, 1.32 Docs Shadow here.
Does this enhancement work planned for 1.32 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday October 24th 2024 18:00 PDT.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[AnshumanTripathi]

Hi @p0lyn0mial 👋,

Just a reminder to open a placeholder PR against dev-1.32 branch in the k/website repo for this (steps available here). The deadline for this is Thursday Oct 24 at 18:00 PDT.

Thanks,

Anshuman

[p0lyn0mial]

@rashansmith I have opened a Feature Blog placeholder PR at kubernetes/website#48513

@AnshumanTripathi I have opened a placeholder PR against dev-1.32 at kubernetes/website#48512

[jenshu]

Hello @p0lyn0mial 👋, v1.32 Enhancements team here.

Are there any code changes planned for this KEP in 1.32? If so, can you please link the PRs in the issue description above? In the meantime, I will mark this enhancement as at risk for code freeze. Thanks!

[jenshu]

Hello @p0lyn0mial 👋, v1.32 Enhancements team here,

Unfortunately, the implementation (code related) PR(s) associated with this enhancement are not in the merge-ready state by code-freeze and hence this enhancement is now removed from the v1.32 milestone.

If you still wish to progress this enhancement in v1.32, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

/milestone clear

[p0lyn0mial]

@jenshu Hey, I'm not sure which PRs you're referring to, but the feature was promoted to Beta in version 1.32 (kubernetes/kubernetes#128053). Could you please reapply the milestone tag?

[jenshu]

Hi @p0lyn0mial, ah thanks. The PR description above doesn't show any 1.32 PRs and I didn't see any responses to my previous question so wasn't sure if there were any PRs associated with 1.32

Can you just confirm if kubernetes/kubernetes#128053 is the only code-related PR needed for 1.32?
If so, I'm happy to add it back to the milestone. Thanks!

[p0lyn0mial]

@jenshu I had more PRs, but the one I linked is important because it enabled the feature for version 1.32. Also, I apologize for not responding to your previous questions.

[jenshu]

Hi @p0lyn0mial, no problem! If possible, can you find the list of PRs associated with this KEP for 1.32 so I can list them in the issue description above? Have all the PRs been merged already?

[jenshu]

Hi @p0lyn0mial just following up again - can you confirm if there were other PRs merged for 1.32 besides kubernetes/kubernetes#128053, and if so, can you list them here? Thanks!

[p0lyn0mial]

Hey @jenshu, done ^.

[jenshu]

/milestone v1.32

[rashansmith]

Hello @p0lyn0mial 👋, v1.32 Comms team here,

Thank you for creating a placeholder PR for a feature blog!

The deadline for completing blog content to be Ready for Review is Monday 25th November 2024 . For more information about writing a blog see the blog contribution guidelines.

Please feel free to reach out if you have any questions!