feat: Allow connecting from the Notification Controller to the Repo Server without TLS (#19629)

[m-wynn]

Unsure if this should be a fix or a feature, so I created an enhancement proposal #19629.

All of the Argo CD components I have worked with have a way to disable TLS between each other via the argocd-cmd-params-cm ConfigMap. One issue I've run into is, when calling .repo.GetCommitMetadata in notifications templates, it tries to reach out to the Repo Server over TLS. There does not seem to be an easy way to disable it, short of overlaying the notifications template deployment itself and adding a command line flag.

This is necessary for environments where Istio is managing mutual TLS between each component (#2784).

Closes #19629

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[m-wynn]

Tested this on my test cluster.

After deploying this version (replacing the cluster-install line of my kustomize resources with github.com/m-wynn/argo-cd//manifests/cluster-install?ref=c631a6824a6b80715aa515c89cda0fb05256631b and overriding the argocd-notifications-controller-deployment image tag to one built off this branch), but without changing the configmap, I get the same error as before:

time="2024-08-22T00:27:50Z" level=error msg="Failed to notify recipient {slack test_channel} defined in resource argocd/argocd: template: app-deployed:22:63: executing \"app-deployed\" at <call .repo.GetCommitMetadata .app.status.sync.revision>: error calling call: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\" using the configuration in namespace argocd" resource=argocd/argocd

And then after adding notificationscontroller.repo.server.plaintext: "true" to argocd-cmd-params-cm and restarting the notifications controller deployment, it works and I get a slack notification with the commit metadata.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Please upload report for BASE (master@d1c74ac). Learn more about missing BASE report.
Report is 33 commits behind head on master.

Files	Patch %	Lines
cmd/argocd-notification/commands/controller.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #19630   +/-   ##
=========================================
  Coverage          ?   55.90%           
=========================================
  Files             ?      316           
  Lines             ?    43790           
  Branches          ?        0           
=========================================
  Hits              ?    24482           
  Misses            ?    16759           
  Partials          ?     2549           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pasha-codefresh]

LGTM, thank you for your contribution