Ambassador isn't loading a secret in a separate namespace

[djsly]

Describe the bug

We are running multiple Ambassador in different namespaces with the Single namespace flag set to true. We created a shared certificate (secret) namespace to host all the secrets that our ambassador can use.

To Reproduce
Steps to reproduce the behavior:

1. Install Ambassador 0.86.1 with a secret in a different namespace
2. use TLSContext

apiVersion: ambassador/v1
      kind:  TLSContext
      name:  tls
      secret: my-secret.shared-certs
      secret_namespacing: true
      hosts: ["*"]
      alpn_protocols: h2[, http/1.1]
      min_tls_version: v1.2
      redirect_cleartext_from: 8080

3. Watch the error in the logs and UI, the resolve_secret does it three time

2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: Found Ambassador service: ambassador-mix-stage
2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: YAML: using C parser
2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: YAML: using C dumper
2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs, namespace mix-stage: namespacing is True
2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs: asking handler to load
2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: SecretRecorder: Trying to load secret cd4-crt-nuance-com-cert in namespace shared-certs from TLSContext <IRTLSContext ambassador-mix-stage.mix-stage.2>
2020/01/15 05:12:32 aggregator: watch hook stderr: 2020-01-15 05:12:32 watch-hook INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs: asking handler to cache
2020/01/15 05:12:32 aggregator: watch hook stderr: 

2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: Found Ambassador service: ambassador-mix-stage
2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: YAML: using C parser
2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: YAML: using C dumper
2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs, namespace mix-stage: namespacing is True
2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs: asking handler to load
2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: SecretRecorder: Trying to load secret cd4-crt-nuance-com-cert in namespace shared-certs from TLSContext <IRTLSContext ambassador-mix-stage.mix-stage.2>
2020/01/15 05:12:33 aggregator: watch hook stderr: 2020-01-15 05:12:33 watch-hook INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs: asking handler to cache
2020/01/15 05:12:33 aggregator: watch hook stderr: 

The third time its the AmbassadorEventWatcher who is performing the check

2020/01/15 05:12:33 notify: python /ambassador/post_update.py --watt  http://localhost:8002/snapshots/1
2020-01-15 05:12:33 diagd 0.86.1 [P492TThreadPoolExecutor-0_0] INFO: Update requested: watt, http://localhost:8002/snapshots/1
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: copying configuration: watt, http://localhost:8002/snapshots/1 to /ambassador/snapshots/snapshot-tmp.yaml
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: Found pod labels: ['app.kubernetes.io/instance="ambassador"\n', 'app.kubernetes.io/name="ambassador"\n', 'app.kubernetes.io/part-of="ambassador"\n', 'harness.io/release-name="ambassador"\n', 'pod-template-hash="69977998bc"']
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: Found Ambassador service: ambassador-mix-stage
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: YAML: using C parser
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: YAML: using C dumper
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs, namespace mix-stage: namespacing is True
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: resolve_secret cd4-crt-nuance-com-cert.shared-certs: asking handler to load
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] ERROR: Secret cd4-crt-nuance-com-cert.shared-certs unknown
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] ERROR: ambassador-mix-stage.mix-stage.2: <RichStatus BAD error=TLSContext tls found no certificate in secret cd4-crt-nuance-com-cert in namespace shared-certs, ignoring... hostname=ambassador-mix-stage-69977998bc-w5vzn version=0.86.1>
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: re_type safe
2020-01-15 05:12:33 diagd 0.86.1 [P492TAmbassadorEventWatcher] INFO: V2Listener: Using log_format 'ACCESS [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"

Expected behavior
CERT / Secret should be loaded

Versions (please complete the following information):

• Ambassador: 0.86.1]
• Kubernetes environment: AKS
• Version 1.14.8