Secrets within notifications-controller service.webhook are not replaced

[ivanivicrho]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD and Notification Controller v2.10.4

Secrets defined in argocd-notifications-secret are not accessible via {{ secrets.xx }} in notifications template. A feature to replace secrets inside a template was implemented here.

Given this template configuration inside argocd-notifications-cm configmap:

  service.webhook.gh-poc: |
    url: https://api.github.com/repos/ORG_REDACTED/REPO_REDACTED/dispatches
    headers:
    - name: Accept
      value: application/vnd.github.v3+json
    - name: Authorization
      value: 'token {{ .secrets.ghPocToken }}'
  template.gh-poc: |
    webhook:
      gh-poc:
        method: POST
        body: |
          { "event_type": "poc" }
  trigger.on-deployed-send-poc: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
      description: Github Actions Trigger PoC
      send: [gh-poc]
      oncePer: app.status.operationState.syncResult.revision

and a argocd-notifications-secret secret:

apiVersion: v1
data:
  ghPocToken: REDACTED
  grafana-api-key: REDACTED
  slack-token: REDACTED
immutable: false
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
type: Opaque

This results in:

Sending request: POST /repos/REDACTED/REDACTED/dispatches HTTP/1.1\r\nHost: api.github.com\r\nAccept: 
application/vnd.github.v3+json\r\nAuthorization: token {{ .secrets.ghPocToken }}\r\n\r\n{ \"event_type\": \"poc\" }\n" 
service=gh-poc

Failed to notify recipient {gh-poc } defined in resource argocd/REDACTED: 
request to {{ \"event_type\": \"poc\" }\n POST https://api.github.com/repos/REDACTED/REDACTED/dispatches gh-poc} 
has failed with error code 401
: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://docs.github.com/rest\"} using the configuration in 
namespace argocd" resource=argocd/REDACTED

If secret value is hardcoded in configmap's template, webhook request obviously passes:

Sending request: POST /repos/REDACTED/REDACTED/dispatches 
HTTP/1.1\r\nHost: api.github.com\r\nAccept: application/vnd.github.v3+json\r\n
Authorization: token github_pat_REDACTED\r\n\r\n{ \"event_type\": \"poc\" }\n" 
service=gh-poc

The only differences that I can see between docs and this example:

• example in this issue uses data in Kubernetes Secret, while docs uses stringData. IMHO both data and stringData should work equally here.
• docs use secret inside a template, while example in this issue uses secret inside service.webhook configuration. IMHO both should be supported

To Reproduce

Apply example from above verbatim, into argocd v2.10.4

Expected behavior

Expect secret value defined in service.webhook to be propagated and properly replaced. Since token .secrets.ghPocToken is not replaced at all - I would expect either an empty string "" or an error being logged/thrown if value is not found - I hypothesize the placeholder is not replaced inside the template.

Screenshots

Version

v2.10.4

Logs

Paste any relevant application logs here.

[ivanivicrho]

Closing, as services are part of notifications-engine https://github.com/argoproj/notifications-engine , not argocd's core.

And this is actually already implemented in notifications-engine, but a different way to reference secrets should be used:

  service.webhook.gh-poc: |
    url: https://api.github.com/repos/REDACTED/REDACTED/dispatches
    headers:
    - name: Accept
      value: application/vnd.github.v3+json
    - name: Authorization
      value: 'token $ghPocToken'

Leaving the comment here, as different ways the separate part of argocd NS system (notifications-engine, and templating) are configured to fetch secrets causes quite a bit of friction.

In templates, it is:

{{ .secrets.SOME_VALUE }}

In services, it is:

$SOME_VALUE

Hopefully this comment will help someone else who is stuck on a similar issue.