feat: self service notification

[mayzhang2000]

This is to enable argocd to have self-service notification.
This PR needs #15702 as pre-requisite.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.

[codecov]

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (b40d8cb) 49.97% compared to head (0c1ea2b) 49.92%.
Report is 28 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #15704      +/-   ##
==========================================
- Coverage   49.97%   49.92%   -0.05%     
==========================================
  Files         266      266              
  Lines       46027    46092      +65     
==========================================
+ Hits        23002    23012      +10     
- Misses      20769    20819      +50     
- Partials     2256     2261       +5     

Files	Coverage Δ
notification_controller/controller/controller.go	63.97% <48.27%> (-1.55%)	⬇️

... and 11 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[crenshaw-dev]

I think we really oughta have some unit tests for when apps-in-any-namespace is enabled.

[crenshaw-dev]

Can you add a description of what happens when there's a "base configuration" defined in the argocd namespace and also a cm/secret in the App namespace? What happens if keys in the ConfigMaps/Secrets conflict? Which takes precedence? An example or two might help.

[crenshaw-dev]

After reading the docs, I have questions about self-service behavior. I think the answers should be documented and, ideally, unit- or e2e-tested:

• What happens if I define the same trigger in both the app namespace and the argocd namespace?
  • Suggestion: allow access to namespaced and global triggers, with namespaced taking precedence.
• What happens if I define defaultTriggers in both?
  • Suggestion: both are available, namespaced takes precedence.
• What if I define the same defaultTriggers.<some-name> in both?
  • Suggestion: both are available, namespaced takes precedence.
• What happens if I define the same template in both the app namespace and the argocd namespace?
  • Suggestion: both are available, namespaced takes precedence.
• What happens if I set the context field in both? Are the contexts merged (and if so, who takes precedence?). Is the context of one used and the other ignored? Do I get an error?
  • Suggestion: merge the contexts, with namespaced taking precedence.
• What happens if I set the subscriptions field in both ("default" or "global" subscriptions)? Are the subscriptions merged (and if so, who takes precedence?). Are the subscriptions of one used and the other ignored? Do I get an error?
  • Suggestion: both are available, namespaced takes precedence.
• How are secrets handled? Are secrets merged (if so, what's the precedence rule)? Are secrets of one used and the other ignored? Do I get an error?
  • Suggestion: I'm not sure... looking into this more.

I've opted to give the namespaced config the most control. That does take control out of the hands of the admin, making it difficult to "override" user-defined behavior. But I think that, for notifications, that is probably fine.

[crenshaw-dev]

After chatting with the team, sounds like this is the conclusion: everything is separate. It's as if a completely different notifications controller were running for the user-defined config, completely unaware of the argocd namespace config.

I think that's a good design for now. Simple, safe, easy to understand. I do think we should make that design explicit in the docs and add tests for the three basic scenarios: 1) config only in argocd, 2) config only in the user namespace, and 3) config in both.