fix(repo-server): excess git requests, add shared cache lock on revisions (Issue #14725) #17109

nromriell · 2024-02-07T02:36:22Z

Additional Fixes #14725

Third part of a split of the #16309 and follow-up to #16410

Another piece of the fixes for #14725. This resolves the excess ls-remote requests that still occur in high application count environments due to the fact that all active processes get their cache invalidated at the same time. When this happens they all try to re-fetch the remotes at the same time and all miss the cache.

The solution is to add a soft cache level lock on the key, the first process to try and access that key on a miss will obtain the lock and attempt to refresh the cache.

This changed required adding the ability to use "set only if not exists" on the util/cache (redis NX or memcached add) interface. Without this option all of the processes could still end up in a position where they think they own the lock, so there are some minor changes to the util/cache interface:

Adds a DisableOverwrite option to represent the NX/add during set option
The Item struct now also contains options for the cache set actions. This allows passing Delete and DisableOverwrite in a shared options value and replaces the position of the current delete option in the SetItem call

This is the final major logic portion of the changes to remove all the excess git calls, but we can get some good performance improvements by also adding the two layer cache to the caching layer for revisions. So I'll be following up with another PR to add that. Most of the abstraction to prepare for that was handled as part of this PR.

Baseline without changes with 200 multi-source applications with a ref only source:

With all the changes up to and including this PR under the same conditions:

Checklist:

Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
The title of the PR states what changed and the related issues number (used for the release note).
The title of the PR conforms to the Toolchain Guide
I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
I've updated documentation as required by this PR.
I have signed off all my commits as required by DCO
I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
My build is green (troubleshooting builds).
I have added a brief description of why this PR is necessary and/or what this PR solves.

ishitasequeira

@nromriell Thanks for following up with this improvement PRs.

Took an initial quick view on the PR. Left some minor comments. Will test this PR and share feedback.

reposerver/cache/cache.go

util/git/client.go

ishitasequeira

Overall PR looks great. @nromriell just minor comments .

@leoluz @crenshaw-dev Could you provide an additional set of eyes to review in case I overlooked anything?

util/cache/client.go

util/git/client.go

reposerver/cache/cache.go

util/git/client.go

reposerver/cache/cache.go

util/git/client.go

nromriell · 2024-02-28T15:38:44Z

With the refactor I need to re-run the tests with the changes live, I'll try and take a look at that tonight

reposerver/cache/cache.go

reposerver/repository/repository_test.go

util/cache/cache.go

ishitasequeira

@nromriell left some minor nits.

@agaudreault do you have any other concerns on the PR?

agaudreault · 2024-03-15T15:13:08Z

@nromriell left some minor nits.

@agaudreault do you have any other concerns on the PR?

No new comments, LGTM :) Good work and good refactor 👍

Signed-off-by: nromriell <nateromriell@gmail.com>

nromriell · 2024-03-23T18:29:24Z

Thanks @ishitasequeira @agaudreault for all your time on these great reviews!

Fixed that PR feedback @ishitasequeira and I was able to finally test this out and it's looking good to me after fixing one more issue I found where the cmd.Flags for revisionCacheLockTimeout was writing to the wrong variable.

I had to swap the rollup period to actually show the difference between 2.10.4 and this change since it includes the previous work, but here's the metrics between 2.10.4 and this PR. The swap is at about the 10:37 mark.

ishitasequeira

Thanks @nromriell for addressing the comments. I missed this one last check earlier

ishitasequeira · 2024-03-23T19:56:54Z

reposerver/cache/cache.go

+	case err != nil && err != ErrCacheMiss:
+		return "", err


Would it make sense to log the err if it's not ErrCacheMiss ?

As is this will always log in util/git/client.go#getRefs for debug but this is treated as a critical failure so definitely seems like a good idea to have here to make sure it gets logged. Added in an error log for it

Signed-off-by: nromriell <nateromriell@gmail.com>

ishitasequeira

Thanks @nromriell, LGTM!!

agaudreault

LGTM! Good job 💪

agaudreault

LGTM! Good job 💪

…ions (Issue argoproj#14725) (argoproj#17109) * fix(repo-server): excess git requests, cache lock on revisions Signed-off-by: nromriell <nateromriell@gmail.com> * fix: pr feedback, simplify, add configurable variable Signed-off-by: nromriell <nateromriell@gmail.com> * fix: codegen, lint Signed-off-by: nromriell <nateromriell@gmail.com> * fix: test print, no opts set, var type nit Signed-off-by: nromriell <nateromriell@gmail.com> * chore: add additional logging for unexpected cache error Signed-off-by: nromriell <nateromriell@gmail.com> --------- Signed-off-by: nromriell <nateromriell@gmail.com> Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com> Signed-off-by: Kevin Lyda <kevin@lyda.ie>

…ions (Issue argoproj#14725) (argoproj#17109) * fix(repo-server): excess git requests, cache lock on revisions Signed-off-by: nromriell <nateromriell@gmail.com> * fix: pr feedback, simplify, add configurable variable Signed-off-by: nromriell <nateromriell@gmail.com> * fix: codegen, lint Signed-off-by: nromriell <nateromriell@gmail.com> * fix: test print, no opts set, var type nit Signed-off-by: nromriell <nateromriell@gmail.com> * chore: add additional logging for unexpected cache error Signed-off-by: nromriell <nateromriell@gmail.com> --------- Signed-off-by: nromriell <nateromriell@gmail.com> Co-authored-by: Ishita Sequeira <46771830+ishitasequeira@users.noreply.github.com>

nromriell requested a review from a team as a code owner February 7, 2024 02:36

nromriell force-pushed the lsremote-part3 branch from 18eb32c to 821ce1e Compare February 7, 2024 04:47

ishitasequeira requested changes Feb 16, 2024

View reviewed changes

reposerver/cache/cache.go Outdated Show resolved Hide resolved

util/git/client.go Outdated Show resolved Hide resolved

ishitasequeira requested changes Feb 20, 2024

View reviewed changes

util/cache/client.go Outdated Show resolved Hide resolved

util/git/client.go Outdated Show resolved Hide resolved

agaudreault reviewed Feb 21, 2024

View reviewed changes

nromriell force-pushed the lsremote-part3 branch from 821ce1e to fd0fa44 Compare February 22, 2024 00:59

ishitasequeira reviewed Feb 27, 2024

View reviewed changes

reposerver/cache/cache.go Show resolved Hide resolved

util/git/client.go Outdated Show resolved Hide resolved

nromriell force-pushed the lsremote-part3 branch from 8ee25e3 to cdddedd Compare February 28, 2024 15:33

nromriell requested a review from a team as a code owner February 28, 2024 15:33

ishitasequeira reviewed Mar 14, 2024

View reviewed changes

reposerver/cache/cache.go Outdated Show resolved Hide resolved

reposerver/repository/repository_test.go Outdated Show resolved Hide resolved

util/cache/cache.go Outdated Show resolved Hide resolved

ishitasequeira reviewed Mar 14, 2024

View reviewed changes

nromriell added 4 commits March 23, 2024 06:42

fix(repo-server): excess git requests, cache lock on revisions

8d3fdfb

Signed-off-by: nromriell <nateromriell@gmail.com>

fix: pr feedback, simplify, add configurable variable

481f6ee

Signed-off-by: nromriell <nateromriell@gmail.com>

fix: codegen, lint

f069756

Signed-off-by: nromriell <nateromriell@gmail.com>

fix: test print, no opts set, var type nit

d8c2313

Signed-off-by: nromriell <nateromriell@gmail.com>

nromriell force-pushed the lsremote-part3 branch from cdddedd to d8c2313 Compare March 23, 2024 18:03

ishitasequeira reviewed Mar 23, 2024

View reviewed changes

chore: add additional logging for unexpected cache error

c95ec36

Signed-off-by: nromriell <nateromriell@gmail.com>

ishitasequeira approved these changes Mar 23, 2024

View reviewed changes

Merge branch 'master' into lsremote-part3

847ed77

agaudreault approved these changes Mar 25, 2024

View reviewed changes

ishitasequeira merged commit c4fdc54 into argoproj:master Mar 25, 2024
27 checks passed

agaudreault mentioned this pull request Jul 29, 2024

REQUEST: Promotion to Approver for @agaudreault argoproj/argoproj#310

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(repo-server): excess git requests, add shared cache lock on revisions (Issue #14725) #17109

fix(repo-server): excess git requests, add shared cache lock on revisions (Issue #14725) #17109

nromriell commented Feb 7, 2024 •

edited

Loading

ishitasequeira left a comment

ishitasequeira left a comment

nromriell commented Feb 28, 2024

ishitasequeira left a comment

agaudreault commented Mar 15, 2024

nromriell commented Mar 23, 2024

ishitasequeira left a comment

ishitasequeira Mar 23, 2024

nromriell Mar 23, 2024

ishitasequeira left a comment

agaudreault left a comment

agaudreault left a comment

fix(repo-server): excess git requests, add shared cache lock on revisions (Issue #14725) #17109

fix(repo-server): excess git requests, add shared cache lock on revisions (Issue #14725) #17109

Conversation

nromriell commented Feb 7, 2024 • edited Loading

ishitasequeira left a comment

Choose a reason for hiding this comment

ishitasequeira left a comment

Choose a reason for hiding this comment

nromriell commented Feb 28, 2024

ishitasequeira left a comment

Choose a reason for hiding this comment

agaudreault commented Mar 15, 2024

nromriell commented Mar 23, 2024

ishitasequeira left a comment

Choose a reason for hiding this comment

ishitasequeira Mar 23, 2024

Choose a reason for hiding this comment

nromriell Mar 23, 2024

Choose a reason for hiding this comment

ishitasequeira left a comment

Choose a reason for hiding this comment

agaudreault left a comment

Choose a reason for hiding this comment

agaudreault left a comment

Choose a reason for hiding this comment

nromriell commented Feb 7, 2024 •

edited

Loading