Scalability: Increase resync period or make parameterizable

[msau42]

The default resync period of 15 seconds is IMO too frequent. Once there are many PVs in the system, and multiple CSI plugins, there will be a lot of unnecessary churn. API watch is supposed to be better now and shouldn't drop events that frequently.

[msau42]

I guess the problem is the external-provisioner uses resync as its retry mechanism. So fixing this probably requires substantial changes to external-provisioner.

[wongma7]

@msau42 Any ideas what the standard is? How high can we go? 12 hrs? It's 15 seconds for no other reasons than: copy the PV controller (which is so low only because it needs it to do its "transactions" in a timely manner) and I guess paranoia over imaginary scenarios wherein missing events leads to data deletion (???)

[wongma7]

Anyway I can definitely try to decouple resync from retries, cleaning up the retry logic may solve #101 at the same time

[msau42]

I'm not sure if there's a standard. Searching here, I see kubelet uses 15 minutes, component config default seems to be 12 hours. I think once we decouple resync from retries, then we could at least increase it to something on the order of minutes.

[wongma7]

Thanks, I've opened a PR here if you/anyone is interested kubernetes-retired/external-storage#825 . Did some basic testing and 15 minutes seems fine. Retries are now handled by the ratelimiting workqueue , i.e. we AddRateLimited if Provision returns err, Forget if succeed, & never retry if retry threshold is crossed

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[msau42]

/close

Will reconsider after we have better scale testing in place

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

/close

Will reconsider after we have better scale testing in place

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.