fix(controller): cache deadlock on delete and re-add cluster

[nromriell]

Fixes an issue with a deadlock in the application controller on cluster delete.

How to reproduce: Quickly add, remove, and re-add a cluster using the argo cli to the same application controller shard. Although the fast timing isn't required it is re-produced more reliably that way.

Problem: controller.cache.handleDeleteEvent and gitops-engine.pkg.cache.cluster.EnsureSync have competing locks under the conditions:

• controller.cache.handleDeleteEvent calls cluster.Invalidate()
• EnsureSync calls gitops-engine.pkg.cache.cluster.sync which processes items in the RunAllAsync calling c.setNode(c.newResource(un)) the issue lies in the c.newResource call because of the populateResourceInfoHandler

Since controller.cache sets clustercache's populateResourceInfoHandler whose function requires locking the liveStateCache in the controller handleDeleteEvent ends up waiting on the lock at the top of cluster.Invalidate while clustercache.sync is waiting to get the lock on the liveStateCache

Fix: have handleDeleteEvent release the liveStateCache lock before calling invalidate. Then lock again before deleting the cluster. Also swapped initial lock to a read lock since we are not modifying.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• I have added a brief description of why this PR is necessary and/or what this PR solves.

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.02% 🎉

Comparison is base (58138ad) 49.88% compared to head (43dd77d) 49.90%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #14780      +/-   ##
==========================================
+ Coverage   49.88%   49.90%   +0.02%     
==========================================
  Files         261      261              
  Lines       44935    44937       +2     
==========================================
+ Hits        22417    22427      +10     
+ Misses      20311    20303       -8     
  Partials     2207     2207              

Files Changed	Coverage Δ
controller/cache/cache.go	27.01% <100.00%> (+1.91%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alexmt]

Thank you! LGTM

[alexmt]

/cherry-pick release-2.5

[gcp-cherry-pick-bot]

Cherry-pick failed with Merge error 2f84dacc035772859574bc393eb55d5fe8284793 into temp-cherry-pick-a53fee-release-2.5

[alexmt]

/cherry-pick release-2.6

[gcp-cherry-pick-bot]

Cherry-pick failed with Merge error 2f84dacc035772859574bc393eb55d5fe8284793 into temp-cherry-pick-a53fee-release-2.6

[alexmt]

/cherry-pick release-2.7

[crenshaw-dev]

/cherry-pick release-2.8

[alexmt]

Manually created cherry-pick PRs for 2.5 and 2.6