feat: use rate limited queue

[gdsoumya]

Fixes #15233

This PR adds a custom rate limiter that is a combination of a BucketLitmer and an ItemExponentialRateLimiter just like the previously used default limiter. However, the custom limiter's exponential backoff auto resets for an item if the coolDown period is over for that item. This is needed because the default exponential limiter needs a manual Forget() call to reset the backoff time but in issue #15233 which is caused due to external factors we cannot determine when Forget() should be called so we delegate it to the limiter to auto forget once the coolDown period is reached.

ENV to configure rate limits:

1. WORKQUEUE_BUCKET_SIZE : default 500
2. WORKQUEUE_BUCKET_QPS : default 50
3. WORKQUEUE_FAILURE_COOLDOWN_NS: default 0s, if duration set to 0 individual rate limiting is disabled(default)
4. WORKQUEUE_BASE_DELAY_NS: default 1000 (1μs)
5. WORKQUEUE_MAX_DELAY_NS: default 3s (3 * 10^9)
6. WORKQUEUE_BACKOFF_FACTOR: default 1.5

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.

[codecov]

Codecov Report

Attention: 22 lines in your changes are missing coverage. Please review.

Comparison is base (c9aa373) 49.68% compared to head (4403308) 49.67%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #15480      +/-   ##
==========================================
- Coverage   49.68%   49.67%   -0.02%     
==========================================
  Files         267      267              
  Lines       46362    46387      +25     
==========================================
+ Hits        23036    23043       +7     
- Misses      21065    21084      +19     
+ Partials     2261     2260       -1     

Files	Coverage Δ
controller/appcontroller.go	54.18% <82.35%> (+0.04%)	⬆️
util/env/env.go	80.99% <0.00%> (-15.09%)	⬇️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alexmt]

UI has a notable difference:

Without rate limiting:

Screen.Recording.2023-09-18.at.8.25.11.AM.mov

With rate limiting:

Screen.Recording.2023-09-18.at.8.26.19.AM.mov

Can you please check if we can tweak rate limiting params to avoid it?

[gdsoumya]

@alexmt we change WORKQUEUE_MAX_DELAY_NS to 10^9 (1 second) or even lower down to milliseconds if we want.

[jessesuen]

In addition to reducing the max delay, I think the backoff can also less aggressive so that it is only throttling apps that have continuous, and sustained reconciles, as opposed to apps that sudden spikes in activity.

[gdsoumya]

@jessesuen I don't think that can be configured directly, one field we can tune for similar results is the WORKQUEUE_BASE_DELAY_NS, the formula uses float64(r.baseDelay.Nanoseconds()) * math.Pow(2, float64(exp.failures)) so if the base delay is smaller then for small spike in syncs it will not throttle too much but even with 1ns as the base we will hit the max delay(1s) in about 30 quick re-queues.

[gdsoumya]

@jannfis your concerns are very valid, but one thing to note - the bucket limiter is there by default which is big enough (default 500 bucket size) to handle almost all existing argocd users without changing their existing experience. The per-item limiter(exponential one) is disabled by default and can be enabled as per user requirements.

[jessesuen]

LGTM!

To summarize this PR, what is being introduced are new controller tuning options to have a per-item rate-limited workqueue. This is disabled by default (WORKQUEUE_FAILURE_COOLDOWN_NS=0) and unless enabled, will behave exactly the same as before. But at least now, we have the ability to protect the controller when we hit situations like in #15233 where a misbehaving 3rd party controller could negatively impact the argo-cd application controller due to unthrottled reconciliations.

Now that we will have this in place, the way I see this moving forward, is for people to experiment with values in real-world setups in order to come up with a good default. Once we are comfortable with that number, in future releases we will turn this on by default with some proven values.

@jannfis are your concerns met?

[crenshaw-dev]

imo we could really use a paragraph or two in docs explaining these options and how to experiment with them. If we'd like to solicit feedback from the community, we should make it super easy for them to understand how to tune these values so they can report back what they found to be effective.

[gdsoumya]

Where do you think would be a good place to put the doc? A separate file under operator manual or in some exisitng file?

[jessesuen]

I think the best place would be along where we document other controller tuning variables:

argo-cd/docs/operator-manual/high_availability.md

Line 41 in ad49623

### argocd-application-controller

I know the file is called "high availability," but there are already performance-oriented tuning variables documented and so it wouldn't be totally out of place.

[gdsoumya]

Makes sense, will add the docs for rate limiting there.

[gdsoumya]

Updated PR to include the docs PTAL.

[jannfis]

Sorry coming back so late to this. Change overall looks good to me.

[jannfis]

@jessesuen Thanks, that makes sense!