fix: Add ENV variable to configure GRPC Keep Alive Time (#15656)

[BhavikaSharma]

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.

Description:
When creating an application that runs a custom plugin that runs for over ~45 seconds, the user is returned an ENHANCE_YOUR_CALM error. The CMP timeout is a configurable value with a default of 60s. Multiple argocd app commands run the CMP (e.g. as validation). Therefore, the GRPCKeepAliveEnforcementMinimum should be configurable.

Closes #15656

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (9e0e8d5) 49.62% compared to head (f54e034) 49.56%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #15806      +/-   ##
==========================================
- Coverage   49.62%   49.56%   -0.06%     
==========================================
  Files         267      269       +2     
  Lines       46479    46558      +79     
==========================================
+ Hits        23063    23075      +12     
- Misses      21144    21211      +67     
  Partials     2272     2272              

Files	Coverage Δ
common/common.go	31.57% <100.00%> (ø)
server/server.go	56.23% <100.00%> (ø)
pkg/apiclient/grpcproxy.go	0.00% <0.00%> (ø)
util/grpc/grpc.go	0.00% <0.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[BhavikaSharma]

There seems to be flakiness in the e2e tests. A different test seems to be failing in each run despite no changes, e.g.

Passing: https://github.com/argoproj/argo-cd/actions/runs/6401735616/job/17377379600
Failing: https://github.com/argoproj/argo-cd/actions/runs/6409855823/job/17403620905

Passing: https://github.com/argoproj/argo-cd/actions/runs/6401032458/job/17375509894
Failing: https://github.com/argoproj/argo-cd/actions/runs/6401735616/job/17377289561

Rerunning the pipeline as described here.

[renperez-cpi]

@BhavikaSharma any update on this?

[jcogilvie]

The most recent failure seems to indicate that upstream redis has changed. Is this a problem for this PR?

[crenshaw-dev]

@jcogilvie merged upstream, should fix the CI failure.

[BhavikaSharma]

@crenshaw-dev , thanks for merging in the upstream. We seem to have a green build.

[renperez-cpi]

@BhavikaSharma is this going to be merge soon? argo seems to be getting worst.

2023/10/16 19:29:00 ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".
2023/10/16 19:41:12 ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".
2023/10/16 19:50:21 ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".
2023/10/16 19:54:25 ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".
2023/10/16 20:06:37 ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".
2023/10/16 20:16:47 ERROR: [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".

[ishitasequeira]

As #15708 was merged, can you add pkg/apiclient/grpcproxy.go to the change as well?

[BhavikaSharma]

I have added pkg/apiclient/grpcproxy.go to the change as well.

[ishitasequeira]

One minor suggestion. Otherwise LGTM!!

[BhavikaSharma]

@ishitasequeira thanks for the approval! I have added the change suggested above and merged in the latest changes from main with a passing pipeline. If it looks good, would you be able to help merge? Thanks 😄 .