Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

linkerd-destination Failed to connect error=Connection refused (os error 111) #8943

Closed
ghost opened this issue Jul 21, 2022 · 15 comments
Closed

linkerd-destination Failed to connect error=Connection refused (os error 111) #8943

ghost opened this issue Jul 21, 2022 · 15 comments
Assignees
@kleimkuhler
Labels
needs/repro priority/triage support

Comments

@ghost
Copy link

ghost commented Jul 21, 2022 •
edited by kleimkuhler
Loading

What is the issue?

linkerd-destination

Failed to connect error=Connection refused (os error 111) linkerd-proxy-injector [ 35.151685s] WARN ThreadId(01) outbound:server{orig_dst=10.100.0.1:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN

How can it be reproduced?

Installing Linkerd on EKS (see versions below).

Logs, error output, etc

Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
[     0.000552s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.000961s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.000970s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.000972s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.000974s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.000976s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.000982s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.000985s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.001294s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.002596s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.111886s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.111925s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.315208s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.334095s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.728772s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.739373s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.230447s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.241135s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.731135s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.742475s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.231776s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.244449s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.733423s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.746656s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.234091s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.248590s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.734746s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.750740s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.236419s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.251923s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.738047s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.753136s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.239700s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.741339s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.784321s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     6.241973s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.743626s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.245289s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.745783s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.247417s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.748071s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.249737s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.751411s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.252087s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.753703s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.255348s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.755991s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.257644s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.759336s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    13.260010s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    13.761640s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    14.263279s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)```

k logs -n linkerd linkerd-destination-69479dd76-sg4qr 
Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
[     0.000552s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.000961s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.000970s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.000972s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.000974s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.000976s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.000982s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.000985s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.001294s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.002596s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.111886s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.111925s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.315208s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.334095s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.728772s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.739373s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.230447s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.241135s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.731135s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.742475s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.231776s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.244449s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.733423s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.746656s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.234091s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.248590s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.734746s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.750740s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.236419s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.251923s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.738047s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.753136s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.239700s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.741339s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.784321s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     6.241973s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.743626s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.245289s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.745783s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.247417s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.748071s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.249737s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.751411s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.252087s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.753703s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.255348s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.755991s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.257644s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.759336s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    13.260010s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    13.761640s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    14.263279s]  WARN ThreadId(01) policy:watch{port=8090}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)

output of linkerd check -o short

linkerd check -o short
Linkerd core checks
===================

linkerd-existence
-----------------
‼ cluster networks can be verified
    the following nodes do not expose a podCIDR:
	ip-172-31-27-xx.us-west-2.compute.internal
	ip-172-31-39-xx.us-west-2.compute.internal
    see https://linkerd.io/2.11/checks/#l5d-cluster-networks-verified for hints

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2022-07-23T03:47:52Z
    see https://linkerd.io/2.11/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

Status check results are √

Environment

  • Kubernetes Version: v1.21.12-eks-5308cf7
  • LinkerD Version: stable-2.10.2

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

No response
@ghost ghost added the bug label Jul 21, 2022
@kleimkuhler kleimkuhler added support and removed bug labels Jul 21, 2022
@kleimkuhler
Copy link
Contributor

I'd recommend upgrading to the latest stable 2.11.4. While I can't be sure, this does look like an issue that has already been fixed by linkerd/linkerd2-proxy#1670. If you continue to run into issues after upgrading we can work from there.

@ghost
Copy link
Author

ghost commented Jul 21, 2022 •
edited by ghost
Loading

Hi @kleimkuhler!

Upgraded to 2.11.4. Still getting the issue (see below).

Also, there are no controller.yaml and controller-rbac.yaml files in the 2.11.4 helm chart (comparing to v2.10.0). Are those separate?

linkerd check
Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
‼ cluster networks can be verified
    the following nodes do not expose a podCIDR:
        ip-10-228-129-76.us-west-2.compute.internal
        ip-10-228-156-176.us-west-2.compute.internal
        ip-10-228-168-248.us-west-2.compute.internal
        ip-10-228-183-58.us-west-2.compute.internal
    see https://linkerd.io/2.11/checks/#l5d-cluster-networks-verified for hints

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2022-07-23T19:28:07Z
    see https://linkerd.io/2.11/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.11.2 but the latest stable version is 2.11.4
    see https://linkerd.io/2.11/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
‼ control plane and cli versions match
    control plane running stable-2.11.4 but cli running stable-2.11.2
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
‼ control plane proxies and cli versions match
    linkerd-destination-7cffc7d5-52zmq running stable-2.11.4 but cli running stable-2.11.2
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-cli-version for hints

linkerd-ha-checks
-----------------
√ pod injection disabled on kube-system

Status check results are √

k logs linkerd-destination-685d589ff-pgjzq -n linkerd 
Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
[     0.000564s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.000948s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.000959s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.000961s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.000963s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.000965s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.000970s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.000972s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.001277s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.004275s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.110574s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.113399s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.321359s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.322972s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.754570s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     0.754765s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.255256s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.258925s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.755972s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     1.762226s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.257667s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.266056s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.759390s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     2.769277s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.260085s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.273099s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.761784s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     3.776129s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.263474s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.279346s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.764189s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     4.783465s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.264829s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.287688s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.766561s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     5.790754s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     6.267241s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.293872s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     6.767926s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     6.797062s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     7.269607s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.300075s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     7.771323s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     7.803070s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     8.272058s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.306351s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     8.773796s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     8.810368s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     9.275464s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[     9.313601s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     9.776119s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.276771s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    10.331736s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[    10.778422s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.278917s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    11.780614s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.282280s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)
[    12.783045s]  WARN ThreadId(01) policy:watch{port=8086}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=Connection refused (os error 111)

k logs linkerd-proxy-injector-c66568955-j5h49  -n linkerd                                   
Defaulted container "linkerd-proxy" out of: linkerd-proxy, proxy-injector, linkerd-init (init)
[     0.000603s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.000953s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.000965s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.000967s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.000969s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.000971s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.000977s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.000979s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003318s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.003357s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.114741s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.115002s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.322457s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.328010s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.740550s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     0.760019s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.242948s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.263299s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.746159s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     1.766823s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.248980s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.269280s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.752406s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     2.772779s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.254630s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.275797s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.757388s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     3.778533s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.260427s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.281742s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.762614s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     4.784492s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.265301s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.287995s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.767947s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     5.792509s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     6.271626s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     6.295259s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     6.774210s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     6.799058s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     7.276797s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     7.301561s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     7.780498s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     7.805211s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     8.283148s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     8.308855s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     8.785681s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     8.811594s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-identity-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[     9.289565s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[     9.770984s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local
[     9.792178s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.170220s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.274693s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.294849s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.484021s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.798487s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.917784s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    10.988684s]  INFO ThreadId(01) inbound:server{port=9995}:rescue{client.addr=10.228.168.248:47188}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[    11.301008s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    11.421500s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    11.804044s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    11.924151s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    12.306962s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    12.427193s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    12.810785s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    12.929919s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    13.169235s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}: linkerd_stack::failfast: TCP Server entering failfast after 3s
[    13.169368s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.136.109:443}: linkerd_stack::failfast: TCP Server entering failfast after 3s
[    13.169433s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:33150
[    13.169461s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:33156
[    13.169471s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:33154
[    13.169477s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:33164
[    13.169484s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:33166
[    13.169490s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:39102
[    13.169496s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:39108
[    13.169503s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:39110
[    13.169510s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:39112
[    13.314915s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    13.433043s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    13.818642s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    13.935697s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    14.170305s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42474
[    14.170325s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42476
[    14.170330s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42478
[    14.170334s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42482
[    14.170338s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42484
[    14.170342s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42486
[    14.170348s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49454
[    14.170352s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49456
[    14.170356s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49466
[    14.322035s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    14.439426s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    14.824715s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    14.942065s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    15.171103s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42490
[    15.171125s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42492
[    15.171129s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42502
[    15.171134s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42500
[    15.171138s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42498
[    15.171142s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42504
[    15.171146s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49468
[    15.171150s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49484
[    15.171193s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42508
[    15.328734s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    15.445109s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    15.832689s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    15.947754s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    16.172086s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49500
[    16.172113s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49494
[    16.172123s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49492
[    16.172129s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49498
[    16.172135s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49504
[    16.172141s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49508
[    16.172145s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42524
[    16.172149s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42526
[    16.172154s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42528
[    16.336296s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    16.451816s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    16.839304s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    16.955678s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    17.172887s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49520
[    17.172913s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49522
[    17.172919s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49530
[    17.172924s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49532
[    17.172928s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49536
[    17.172932s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49538
[    17.172936s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42544
[    17.172940s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42542
[    17.172944s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42556
[    17.342298s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    17.370397s]  INFO ThreadId(01) inbound:server{port=9995}:rescue{client.addr=10.228.168.248:38920}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[    17.459619s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    17.845158s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    17.962265s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    18.173662s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42572
[    18.173685s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42574
[    18.173690s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42578
[    18.173694s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49556
[    18.173698s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49560
[    18.173702s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49562
[    18.173710s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49564
[    18.173713s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49566
[    18.173717s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49568
[    18.347818s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    18.465294s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    18.850560s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    18.967982s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    19.174790s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42608
[    19.174812s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42604
[    19.174817s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42610
[    19.174821s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42614
[    19.174825s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42616
[    19.174829s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49582
[    19.174833s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49590
[    19.174838s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49596
[    19.174842s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49598
[    19.353232s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    19.471400s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    19.855698s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    19.974185s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    20.175762s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42624
[    20.175790s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42626
[    20.175795s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42630
[    20.175799s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42622
[    20.175803s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42636
[    20.175808s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42638
[    20.175811s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49602
[    20.175818s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49608
[    20.175822s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49612
[    20.359276s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    20.477390s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    20.861739s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    20.980098s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    21.176436s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49622
[    21.176457s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49624
[    21.176462s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49630
[    21.176466s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49636
[    21.176471s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49638
[    21.176476s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49640
[    21.176480s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42656
[    21.176484s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42644
[    21.176488s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42652
[    21.366020s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    21.483230s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    21.869863s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    21.986020s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    22.177240s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49670
[    22.177262s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49660
[    22.177267s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49656
[    22.177271s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49658
[    22.177275s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:49676
[    22.177279s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42678
[    22.177283s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42690
[    22.177287s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42694
[    22.177291s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42696
[    22.373533s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    22.488900s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    22.876194s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    22.992594s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    23.177831s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52416
[    23.177860s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52418
[    23.177867s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52432
[    23.177872s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52436
[    23.177878s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42342
[    23.177884s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42332
[    23.177891s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42330
[    23.177897s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42344
[    23.177902s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42348
[    23.379404s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    23.495586s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    23.882208s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    23.998392s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    24.131077s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42354
[    24.157545s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52442
[    24.232192s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42358
[    24.303851s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42360
[    24.372525s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42362
[    24.385445s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    24.487826s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42364
[    24.500891s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    24.508988s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42366
[    24.528858s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42368
[    24.607301s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42370
[    24.888607s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    25.004076s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    25.131989s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52458
[    25.158485s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42374
[    25.233111s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52462
[    25.304753s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52464
[    25.373378s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42380
[    25.391272s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    25.489014s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42384
[    25.507271s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    25.510431s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52472
[    25.529811s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42390
[    25.608689s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42392
[    25.894305s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    26.010630s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    26.133293s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52480
[    26.159678s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52482
[    26.234283s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52484
[    26.305886s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42400
[    26.374495s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52488
[    26.397481s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    26.490179s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42404
[    26.511565s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42406
[    26.513325s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    26.531031s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42408
[    26.609672s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52496
[    26.899987s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    27.016428s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    27.134279s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52522
[    27.160716s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42438
[    27.235379s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42440
[    27.307015s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52528
[    27.370307s]  INFO ThreadId(01) inbound:server{port=9995}:rescue{client.addr=10.228.168.248:54462}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[    27.370479s]  INFO ThreadId(01) inbound:server{port=9995}:rescue{client.addr=10.228.168.248:54460}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: Connection refused (os error 111)
[    27.375523s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52542
[    27.403791s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    27.491275s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52544
[    27.512700s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42460
[    27.518675s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    27.532117s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42462
[    27.610740s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52550
[    27.907115s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    28.022234s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    28.135267s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52552
[    28.161944s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42468
[    28.236822s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42470
[    28.308662s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52558
[    28.376236s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42474
[    28.410174s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    28.491984s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42476
[    28.513408s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42478
[    28.525799s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    28.532891s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52566
[    28.611503s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52568
[    28.912883s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    29.029274s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    29.136053s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52576
[    29.162542s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52578
[    29.238288s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52586
[    29.309853s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42502
[    29.377524s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42504
[    29.416770s]  WARN ThreadId(01) policy:watch{port=9995}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-policy.linkerd.svc.cluster.local. type: AAAA class: IN
[    29.493282s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42506
[    29.514728s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52594
[    29.531741s]  WARN ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: SRV class: IN; failed to resolve A record: no record found for name: linkerd-dst-headless.linkerd.svc.cluster.local. type: AAAA class: IN
[    29.534150s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:42510
[    29.612759s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=TCP Server service in fail-fast client.addr=10.228.164.154:52598
[    30.137348s]  INFO ThreadId(01) outbound:server{orig_dst=10.228.163.209:443}: linkerd_stack::failfast: TCP Server service has recovered

Also, how can we fix the "the following nodes do not expose a podCIDR" issue?

Thank you!

@ghost
Copy link
Author

ghost commented Jul 24, 2022

@kleimkuhler v2.10.2 runs with no issues.

@alan-kea
Copy link

alan-kea commented Aug 1, 2022 •
edited
Loading

@kleimkuhler
@jesseadev
i'm getting this error too, so the first thing to do to resolve this is installing v2.10.2?

@ghost
Copy link
Author

ghost commented Aug 6, 2022 •
edited by ghost
Loading

@kleimkuhler We rolled back to v2.10.2. I haven't had a chance to do a deep dive. I was hoping someone from the linkerd team would have insights. @alan-kea

@Tizull
Copy link

Tizull commented Aug 11, 2022

This looks like a permissions issue with EKS. I had the same problem when trying to have the CP come up.

@ghost
Copy link
Author

ghost commented Aug 12, 2022

@Tizull Could you please give more details on the permissions fix?

@h3nry-1
Copy link

h3nry-1 commented Aug 16, 2022

@jesseadev i'm also having trouble with the "os error 111". In relation to your question: "Also, how can we fix the "the following nodes do not expose a podCIDR" issue?"
The entry on the linkerd website didn't help me.
What i did was:

  1. Find out the CIDR of your Kubernetes Network (CNI). These are the ranges used by calico, weavenet etc.
  2. Edit your node information (for every node): kubectl edit node <YOUR_NODE_NAME>
    There should be a field "spec: {}", where you can add information like: spec: {"podCIDR":"YOUR_POD_CIDR"} (example: spec: {"podCIDR":"10.0.10.0/24"}.
    When entries exist on that field you can add your information in the following format:
spec:
  podCIDR: 10.0.10.0/24
  podCIDRs:
  - 10.0.10.0/24

Thats how i got the green checkmark with "linkerd check".

@h3nry-1
Copy link

h3nry-1 commented Aug 24, 2022

The fix that worked for me was to delete the "linkerd-destination-XXX" pod.
I installed linkerd with the command "linkerd install > install-linkerd.yaml" and then used "kubectl apply -f install-linkerd.yaml". Maybe there was a problem with the starting order from the three pods in the linkerd namespace.

@kleimkuhler
Copy link
Contributor

@jesseadev sorry for the lack of updates on this issue. A reproduction would be helpful with narrowing down what's going on here; if you have manifests and a k8s environment that reproduces this easily I'd definitely be interested.

Separately, the "the following nodes do not expose a podCIDR" warning was removed in favor or a less noisy check. You'll find this has changed on the most recent edges as well as the recently released stable-2.12.0.

It would be helpful to know if stable-2.12.0 fixes this for you if that is an easy thing to test. I understand 2.11 seems to be causing the issue as you don't see it on stable-2.10.2, but it's good to at least know if this got fixed without a deliberate change on Linkerd's end.

@kleimkuhler
Copy link
Contributor

It would also be helpful to know if anyone on this thread has seen this result in control plane components failing to start properly. Or, does all the control plane Pods get up and running and linkerd check doesn't show any significant errors?

While these logs can make it seem like there is a larger issue going on, if you're just seeing it at startup it very well could be that the DNS records have not been created yet for the other component Pods that are starting up.

@h3nry-1
Copy link

h3nry-1 commented Aug 29, 2022

In my case all pods can start, are up and linkerd check doesn't show any errors. The Issues remain just at the startup. After deleting the destination pod one time everything works fine.

@kleimkuhler
Copy link
Contributor

@h3nry-1 I'm a little confused by your comment; all Pods can start and linkerd check does not show any errors... yet some issue remained at startup? Did you have to delete a destination Pod because it was stuck starting up, or because it initially showed some errors and then was healthy?

In the case that this error is being returned at Pod startup, but eventually everything gets up and running, I'm thinking it could just be a race between other control plane Pods being created and therefore DNS entries don't exist yet.

I'm not sure right now if there is any actual issue here if all the control plane Pods become healthy (without manual restarts) and linkerd check has no errors.

@h3nry-1
Copy link

h3nry-1 commented Aug 31, 2022

@kleimkuhler Im sorry for my confusing comment but im still not quite sure why my errors occured or why they are gone. All pods can start up and 'kubectl get pod' shows 4/4 linkerd-destination, 2/2 linkerd-identity and 2/2 linkerd-proxy-injector pods. With 'kubectl logs linkerd-destination' the OS-Error 111 occured. After i manually deleted the linkerd-destination pod, the OS-Errors were gone from 'kubectl logs linkerd-destination'. I had some problems with reattaching my persistent Volumes in my K8s Cluster, maybe that was the problem in my case. I wish i could give you more insight in this regard.

@kleimkuhler
Copy link
Contributor

No problem h3nry-1. I'm going to close this out for now. I have some unanswered questions on here and it seems to me that this happens at startup but eventually the control plane components become healthy and linkerd check has no errors. As I stated in an above comment, it's likely due to the fact that the Pod is not yet created and therefore the DNS records are not created immediately leading to the DNS errors in the logs.

If anyone on this issue does continue to see this problem and it leads to control plane components being stuck or the application not working correctly, I'd ask that you open a new issue with as much detail as you can provide. It feels better to start from a fresh issue on the latest stable rather than trying to piece together the comments on here.

@nathanmcgarvey nathanmcgarvey mentioned this issue Sep 10, 2022
Closed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kleimkuhler kleimkuhler

Labels
needs/repro priority/triage support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@adleong @kleimkuhler @Tizull @alan-kea @h3nry-1