Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connect sidecar proxy does not honor removal of upstream #10422

Closed
shoenig opened this issue Jun 17, 2021 · 4 comments
Closed

Connect sidecar proxy does not honor removal of upstream #10422

shoenig opened this issue Jun 17, 2021 · 4 comments

Comments

@shoenig
Copy link
Member

shoenig commented Jun 17, 2021

While working on hashicorp/nomad#10776 which fixes Nomad to update a Connect proxy's upstreams without restarting the proxy or parent task, I noticed this behavior where my tasks would still be able to make use of proxy upstreams even after updating the sidecar proxy to no longer have upstreams. Seems like the envoy should stop the listener when the upstream is removed. Not using ACLs or intentions.

$ nomad version 
Nomad v1.1.2-dev (7ba60b4e33fe6f6f62902e61a974c4a152b1cc90)
$ consul version 
Consul v1.9.5
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
$ consul agent -dev
$ sudo nomad agent -dev-connect
# test.nomad
job "test" {
  datacenters = ["dc1"]

  group "server" {
    network {
      mode = "bridge"
    }

    service {
      name = "destination"
      port = 8999

      connect {
        sidecar_service {}
      }
    }

    task "server" {
      driver = "docker"
      config {
        image = "shoenig/simple-http:v1"
        args  = ["server"]
      }
    }
  }


  group "client" {
    network {
      mode = "bridge"
    }

    service {
      name = "client"
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "destination"
              local_bind_port  = 8999
            }
          }
        }
      }
    }

    task "client" {
      driver = "docker"
      config {
        image = "shoenig/simple-http:v1"
        args  = ["client"]
      }
    }
  }
}
$ nomad job run test.nomad

Checking our service registrations has upstream as expected

$ curl -s localhost:8500/v1/catalog/service/client-sidecar-proxy | jq .[].ServiceProxy
{
  "DestinationServiceName": "client",
  "DestinationServiceID": "_nomad-task-8db34479-3402-9437-de6e-15016f1e4fe8-group-client-client-",
  "LocalServiceAddress": "127.0.0.1",
  "Config": {
    "bind_address": "0.0.0.0",
    "bind_port": 23131
  },
  "Upstreams": [
    {
      "DestinationType": "service",
      "DestinationName": "destination",
      "Datacenter": "",
      "LocalBindPort": 8999,
      "MeshGateway": {}
    }
  ],
  "MeshGateway": {},
  "Expose": {}
}

Connection working as expected

$ nomad alloc logs 8db client | tail -n 3
2021/06/17 13:46:42 INFO  [client] sending request to http://127.0.0.1:8999 ...
2021/06/17 13:46:42 INFO  [client]  -> GET response code: (200)
2021/06/17 13:46:42 INFO  [client]  -> GET response: the time is 1:46PM

Modify client service to remove upstreams

    service {                                                                                                                                                       
      name = "client"                                                                                                                                               
      connect {                                                                                                                                                     
        sidecar_service {                                                                                                                                           
          # proxy {                                                                                                                                                 
          #   upstreams {                                                                                                                                           
          #     destination_name = "destination"                                                                                                                    
          #     local_bind_port = 8999                                                                                                                              
          #   }                                                                                                                                                     
          # }                                                                                                                                                       
        }                                                                                                                                                           
      }                                                                                                                                                             
    }

Run modified job

$ nomad job run test.nomad

Consul synced service

    2021-06-17T08:47:36.703-0500 [INFO]  agent: Synced service: service=_nomad-task-8db34479-3402-9437-de6e-15016f1e4fe8-group-client-client--sidecar-proxy

Checking upstream removed from service definition

$ curl -s localhost:8500/v1/catalog/service/client-sidecar-proxy | jq .[].ServiceProxy
{
  "DestinationServiceName": "client",
  "DestinationServiceID": "_nomad-task-8db34479-3402-9437-de6e-15016f1e4fe8-group-client-client-",
  "LocalServiceAddress": "127.0.0.1",
  "Config": {
    "bind_address": "0.0.0.0",
    "bind_port": 23131
  },
  "MeshGateway": {},
  "Expose": {}
}

Notice connections still working (!)

$ nomad alloc logs 8db client | tail -n 3
2021/06/17 13:49:20 INFO  [client] sending request to http://127.0.0.1:8999 ...
2021/06/17 13:49:20 INFO  [client]  -> GET response code: (200)
2021/06/17 13:49:20 INFO  [client]  -> GET response: the time is 1:49PM

Exec into container, lets look around

$ nomad alloc exec -task connect-proxy-client 8db /bin/bash

root@1016a9f7a781:/# apt update && apt install curl && apt install net-tools

root@1016a9f7a781:/# curl localhost:19001/config_dump
config dump response 
{
 "configs": [
  {
   "@type": "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump",
   "bootstrap": {
    "node": {
     "id": "_nomad-task-8db34479-3402-9437-de6e-15016f1e4fe8-group-client-client--sidecar-proxy",
     "cluster": "client",
     "metadata": {
      "namespace": "default",
      "envoy_version": "1.16.2"
     },
     "hidden_envoy_deprecated_build_version": "e98e41a8e168af7acae8079fc0cd68155f699aa3/1.16.2/Clean/RELEASE/BoringSSL",
     "user_agent_name": "envoy",
     "user_agent_build_version": {
      "version": {
       "major_number": 1,
       "minor_number": 16,
       "patch": 2
      },
      "metadata": {
       "revision.status": "Clean",
       "revision.sha": "e98e41a8e168af7acae8079fc0cd68155f699aa3",
       "build.type": "RELEASE",
       "ssl.version": "BoringSSL"
      }
     },
     "extensions": [
      {
       "name": "envoy.transport_sockets.alts",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "envoy.transport_sockets.quic",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "envoy.transport_sockets.raw_buffer",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "envoy.transport_sockets.tap",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "envoy.transport_sockets.tls",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "raw_buffer",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "tls",
       "category": "envoy.transport_sockets.downstream"
      },
      {
       "name": "quiche",
       "category": "envoy.quic_server_codec"
      },
      {
       "name": "envoy.extensions.http.cache.simple",
       "category": "envoy.http.cache"
      },
      {
       "name": "envoy.ip",
       "category": "envoy.resolvers"
      },
      {
       "name": "dubbo.hessian2",
       "category": "envoy.dubbo_proxy.serializers"
      },
      {
       "name": "envoy.filters.connection_pools.http.generic",
       "category": "envoy.upstreams"
      },
      {
       "name": "envoy.filters.connection_pools.http.http",
       "category": "envoy.upstreams"
      },
      {
       "name": "envoy.filters.connection_pools.http.tcp",
       "category": "envoy.upstreams"
      },
      {
       "name": "default",
       "category": "envoy.dubbo_proxy.route_matchers"
      },
      {
       "name": "envoy.compression.gzip.compressor",
       "category": "envoy.compression.compressor"
      },
      {
       "name": "udp_default_writer",
       "category": "envoy.udp_packet_writers"
      },
      {
       "name": "udp_gso_batch_writer",
       "category": "envoy.udp_packet_writers"
      },
      {
       "name": "envoy.access_loggers.file",
       "category": "envoy.access_loggers"
      },
      {
       "name": "envoy.access_loggers.http_grpc",
       "category": "envoy.access_loggers"
      },
      {
       "name": "envoy.access_loggers.tcp_grpc",
       "category": "envoy.access_loggers"
      },
      {
       "name": "envoy.file_access_log",
       "category": "envoy.access_loggers"
      },
      {
       "name": "envoy.http_grpc_access_log",
       "category": "envoy.access_loggers"
      },
      {
       "name": "envoy.tcp_grpc_access_log",
       "category": "envoy.access_loggers"
      },
      {
       "name": "quiche_quic_listener",
       "category": "envoy.udp_listeners"
      },
      {
       "name": "raw_udp_listener",
       "category": "envoy.udp_listeners"
      },
      {
       "name": "envoy.internal_redirect_predicates.allow_listed_routes",
       "category": "envoy.internal_redirect_predicates"
      },
      {
       "name": "envoy.internal_redirect_predicates.previous_routes",
       "category": "envoy.internal_redirect_predicates"
      },
      {
       "name": "envoy.internal_redirect_predicates.safe_cross_scheme",
       "category": "envoy.internal_redirect_predicates"
      },
      {
       "name": "dubbo",
       "category": "envoy.dubbo_proxy.protocols"
      },
      {
       "name": "auto",
       "category": "envoy.thrift_proxy.transports"
      },
      {
       "name": "framed",
       "category": "envoy.thrift_proxy.transports"
      },
      {
       "name": "header",
       "category": "envoy.thrift_proxy.transports"
      },
      {
       "name": "unframed",
       "category": "envoy.thrift_proxy.transports"
      },
      {
       "name": "envoy.retry_priorities.previous_priorities",
       "category": "envoy.retry_priorities"
      },
      {
       "name": "envoy.dog_statsd",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.metrics_service",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.stat_sinks.dog_statsd",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.stat_sinks.hystrix",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.stat_sinks.metrics_service",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.stat_sinks.statsd",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.statsd",
       "category": "envoy.stats_sinks"
      },
      {
       "name": "envoy.dynamic.ot",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.lightstep",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.tracers.datadog",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.tracers.dynamic_ot",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.tracers.lightstep",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.tracers.opencensus",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.tracers.xray",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.tracers.zipkin",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.zipkin",
       "category": "envoy.tracers"
      },
      {
       "name": "envoy.transport_sockets.alts",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "envoy.transport_sockets.quic",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "envoy.transport_sockets.raw_buffer",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "envoy.transport_sockets.tap",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "envoy.transport_sockets.tls",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "envoy.transport_sockets.upstream_proxy_protocol",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "raw_buffer",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "tls",
       "category": "envoy.transport_sockets.upstream"
      },
      {
       "name": "envoy.resource_monitors.fixed_heap",
       "category": "envoy.resource_monitors"
      },
      {
       "name": "envoy.resource_monitors.injected_resource",
       "category": "envoy.resource_monitors"
      },
      {
       "name": "envoy.watchdog.abort_action",
       "category": "envoy.guarddog_actions"
      },
      {
       "name": "envoy.watchdog.profile_action",
       "category": "envoy.guarddog_actions"
      },
      {
       "name": "envoy.filters.listener.http_inspector",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.filters.listener.original_dst",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.filters.listener.original_src",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.filters.listener.proxy_protocol",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.filters.listener.tls_inspector",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.listener.http_inspector",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.listener.original_dst",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.listener.original_src",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.listener.proxy_protocol",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.listener.tls_inspector",
       "category": "envoy.filters.listener"
      },
      {
       "name": "envoy.retry_host_predicates.omit_canary_hosts",
       "category": "envoy.retry_host_predicates"
      },
      {
       "name": "envoy.retry_host_predicates.omit_host_metadata",
       "category": "envoy.retry_host_predicates"
      },
      {
       "name": "envoy.retry_host_predicates.previous_hosts",
       "category": "envoy.retry_host_predicates"
      },
      {
       "name": "envoy.extensions.network.socket_interface.default_socket_interface",
       "category": "envoy.bootstrap"
      },
      {
       "name": "envoy.cluster.eds",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.cluster.logical_dns",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.cluster.original_dst",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.cluster.static",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.cluster.strict_dns",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.clusters.aggregate",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.clusters.dynamic_forward_proxy",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.clusters.redis",
       "category": "envoy.clusters"
      },
      {
       "name": "envoy.grpc_credentials.aws_iam",
       "category": "envoy.grpc_credentials"
      },
      {
       "name": "envoy.grpc_credentials.default",
       "category": "envoy.grpc_credentials"
      },
      {
       "name": "envoy.grpc_credentials.file_based_metadata",
       "category": "envoy.grpc_credentials"
      },
      {
       "name": "envoy.filters.thrift.rate_limit",
       "category": "envoy.thrift_proxy.filters"
      },
      {
       "name": "envoy.filters.thrift.router",
       "category": "envoy.thrift_proxy.filters"
      },
      {
       "name": "envoy.client_ssl_auth",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.echo",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.ext_authz",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.client_ssl_auth",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.direct_response",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.dubbo_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.echo",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.ext_authz",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.http_connection_manager",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.kafka_broker",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.local_ratelimit",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.mongo_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.mysql_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.postgres_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.ratelimit",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.rbac",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.redis_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.rocketmq_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.sni_cluster",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.sni_dynamic_forward_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.tcp_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.thrift_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.filters.network.zookeeper_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.http_connection_manager",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.mongo_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.ratelimit",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.redis_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.tcp_proxy",
       "category": "envoy.filters.network"
      },
      {
       "name": "envoy.buffer",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.cors",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.csrf",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.ext_authz",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.fault",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.adaptive_concurrency",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.admission_control",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.aws_lambda",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.aws_request_signing",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.buffer",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.cache",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.cdn_loop",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.compressor",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.cors",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.csrf",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.decompressor",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.dynamic_forward_proxy",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.dynamo",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.ext_authz",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.fault",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.grpc_http1_bridge",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.grpc_http1_reverse_bridge",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.grpc_json_transcoder",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.grpc_stats",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.grpc_web",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.gzip",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.header_to_metadata",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.health_check",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.ip_tagging",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.jwt_authn",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.local_ratelimit",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.lua",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.oauth",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.on_demand",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.original_src",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.ratelimit",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.rbac",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.router",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.squash",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.http.tap",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.grpc_http1_bridge",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.grpc_json_transcoder",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.grpc_web",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.gzip",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.health_check",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.http_dynamo_filter",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.ip_tagging",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.local_rate_limit",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.lua",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.rate_limit",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.router",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.squash",
       "category": "envoy.filters.http"
      },
      {
       "name": "envoy.filters.udp.dns_filter",
       "category": "envoy.filters.udp_listener"
      },
      {
       "name": "envoy.filters.udp_listener.udp_proxy",
       "category": "envoy.filters.udp_listener"
      },
      {
       "name": "quiche",
       "category": "envoy.quic_client_codec"
      },
      {
       "name": "envoy.filters.dubbo.router",
       "category": "envoy.dubbo_proxy.filters"
      },
      {
       "name": "envoy.health_checkers.redis",
       "category": "envoy.health_checkers"
      },
      {
       "name": "envoy.compression.gzip.decompressor",
       "category": "envoy.compression.decompressor"
      },
      {
       "name": "auto",
       "category": "envoy.thrift_proxy.protocols"
      },
      {
       "name": "binary",
       "category": "envoy.thrift_proxy.protocols"
      },
      {
       "name": "binary/non-strict",
       "category": "envoy.thrift_proxy.protocols"
      },
      {
       "name": "compact",
       "category": "envoy.thrift_proxy.protocols"
      },
      {
       "name": "twitter",
       "category": "envoy.thrift_proxy.protocols"
      }
     ]
    },
    "static_resources": {
     "clusters": [
      {
       "name": "local_agent",
       "type": "STATIC",
       "connect_timeout": "1s",
       "hidden_envoy_deprecated_hosts": [
        {
         "pipe": {
          "path": "alloc/tmp/consul_grpc.sock"
         }
        }
       ],
       "http2_protocol_options": {}
      }
     ]
    },
    "dynamic_resources": {
     "lds_config": {
      "ads": {}
     },
     "cds_config": {
      "ads": {}
     },
     "ads_config": {
      "api_type": "GRPC",
      "grpc_services": [
       {
        "envoy_grpc": {
         "cluster_name": "local_agent"
        },
        "initial_metadata": [
         {
          "key": "x-consul-token"
         }
        ]
       }
      ]
     }
    },
    "admin": {
     "access_log_path": "/dev/null",
     "address": {
      "socket_address": {
       "address": "127.0.0.1",
       "port_value": 19001
      }
     }
    },
    "stats_config": {
     "stats_tags": [
      {
       "tag_name": "consul.destination.custom_hash",
       "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.service_subset",
       "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.service",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.namespace",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.datacenter",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.routing_type",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.trust_domain",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.target",
       "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.destination.full_target",
       "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)"
      },
      {
       "tag_name": "consul.upstream.service",
       "regex": "^(?:tcp|http)\\.upstream\\.(([^.]+)(?:\\.[^.]+)?\\.[^.]+\\.)"
      },
      {
       "tag_name": "consul.upstream.datacenter",
       "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)"
      },
      {
       "tag_name": "consul.upstream.namespace",
       "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.([^.]+))?\\.[^.]+\\.)"
      },
      {
       "tag_name": "consul.custom_hash",
       "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.service_subset",
       "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.service",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.namespace",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.datacenter",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.routing_type",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.trust_domain",
       "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)"
      },
      {
       "tag_name": "consul.target",
       "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)"
      },
      {
       "tag_name": "consul.full_target",
       "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)"
      },
      {
       "tag_name": "local_cluster",
       "fixed_value": "client"
      },
      {
       "tag_name": "consul.source.service",
       "fixed_value": "client"
      },
      {
       "tag_name": "consul.source.namespace",
       "fixed_value": "default"
      },
      {
       "tag_name": "consul.source.datacenter",
       "fixed_value": "dc1"
      }
     ],
     "use_all_default_tags": true
    },
    "layered_runtime": {
     "layers": [
      {
       "name": "static_layer",
       "static_layer": {
        "envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1": true,
        "envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name": true,
        "envoy.deprecated_features:envoy.api.v2.Cluster.tls_context": true
       }
      }
     ]
    }
   },
   "last_updated": "2021-06-17T13:44:44.786Z"
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v3.ClustersConfigDump",
   "version_info": "00000003",
   "static_clusters": [
    {
     "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "local_agent",
      "type": "STATIC",
      "connect_timeout": "1s",
      "hosts": [
       {
        "pipe": {
         "path": "alloc/tmp/consul_grpc.sock"
        }
       }
      ],
      "http2_protocol_options": {}
     },
     "last_updated": "2021-06-17T13:44:44.814Z"
    }
   ],
   "dynamic_active_clusters": [
    {
     "version_info": "00000001",
     "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "local_app",
      "type": "STATIC",
      "connect_timeout": "5s",
      "load_assignment": {
       "cluster_name": "local_app",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "127.0.0.1",
              "port_value": 0
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2021-06-17T13:44:44.843Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
   "version_info": "00000003",
   "dynamic_listeners": [
    {
     "name": "public_listener:0.0.0.0:23131",
     "active_state": {
      "version_info": "00000001",
      "listener": {
       "@type": "type.googleapis.com/envoy.api.v2.Listener",
       "name": "public_listener:0.0.0.0:23131",
       "address": {
        "socket_address": {
         "address": "0.0.0.0",
         "port_value": 23131
        }
       },
       "filter_chains": [
        {
         "tls_context": {
          "common_tls_context": {
           "tls_params": {},
           "tls_certificates": [
            {
             "certificate_chain": {
              "inline_string": "-----BEGIN CERTIFICATE-----\nMIICRDCCAeqgAwIBAgIBCDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktZXRk\nczJzZXUuY29uc3VsLmNhLjZlZGQ4YjIyLmNvbnN1bDAeFw0yMTA2MTcxMzQzNDRa\nFw0yMTA2MjAxMzQzNDRaMC0xKzApBgNVBAMTImNsaWVudC5zdmMuZGVmYXVsdC42\nZWRkOGIyMi5jb25zdWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARY6HS3/2EG\nRC6HdHoX7bNwXNTt732sHCh5zvJhaDLarm90JIJUzhOE8CVI3Y0La89BwJAFWo2v\nsrwbA55lQ9iTo4H2MIHzMA4GA1UdDwEB/wQEAwIDuDAdBgNVHSUEFjAUBggrBgEF\nBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHQ4EIgQgyWnTAJsqz//+\nNpneBCHzJEsyh0pcqBcvvDXqYEX41xAwKwYDVR0jBCQwIoAgYgiKwcOlnGeU7I2w\n/419CQn2tkjOOplcaP327xIraiQwXAYDVR0RBFUwU4ZRc3BpZmZlOi8vNmVkZDhi\nMjItYzJlZS0wNjhmLTJlYzgtMDYwOWM3NTVlOGRhLmNvbnN1bC9ucy9kZWZhdWx0\nL2RjL2RjMS9zdmMvY2xpZW50MAoGCCqGSM49BAMCA0gAMEUCIQDoFT/TYNzsjgn4\ng1KK+Qd/2KCjd7Q2JcZEtOqkQuSYPQIgBCqVg+XoXGIp1WfjiOzxKAVI/3FdMBxU\naMTpAPgJbEY=\n-----END CERTIFICATE-----\n"
             },
             "private_key": {
              "inline_string": "[redacted]"
             }
            }
           ],
           "validation_context": {
            "trusted_ca": {
             "inline_string": "-----BEGIN CERTIFICATE-----\nMIICEDCCAbWgAwIBAgIBBzAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktZXRk\nczJzZXUuY29uc3VsLmNhLjZlZGQ4YjIyLmNvbnN1bDAeFw0yMTA2MTcxMzM1MjRa\nFw0zMTA2MTcxMzM1MjRaMDExLzAtBgNVBAMTJnByaS1ldGRzMnNldS5jb25zdWwu\nY2EuNmVkZDhiMjIuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEJdM/\n6VmktlNI81/vMRFNyvR9XKOTzmoCrFBU4DGbxmE+pIhvX0WkpOfKEpcNHdm52fCd\nfoN5bMlzkt0JLoEj5qOBvTCBujAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw\nAwEB/zApBgNVHQ4EIgQgYgiKwcOlnGeU7I2w/419CQn2tkjOOplcaP327xIraiQw\nKwYDVR0jBCQwIoAgYgiKwcOlnGeU7I2w/419CQn2tkjOOplcaP327xIraiQwPwYD\nVR0RBDgwNoY0c3BpZmZlOi8vNmVkZDhiMjItYzJlZS0wNjhmLTJlYzgtMDYwOWM3\nNTVlOGRhLmNvbnN1bDAKBggqhkjOPQQDAgNJADBGAiEAmxo8lpl5M0/hMXslbAX9\nTDeRUEXuPF+noYA9d1RcdWYCIQDyBFnVcofowba6bKOw3oGZqaC3tU0qXjRQK12J\ncle2OQ==\n-----END CERTIFICATE-----\n"
            }
           }
          },
          "require_client_certificate": true
         },
         "filters": [
          {
           "name": "envoy.filters.network.rbac",
           "config": {
            "stat_prefix": "connect_authz",
            "rules": {
             "action": "DENY"
            }
           }
          },
          {
           "name": "envoy.tcp_proxy",
           "config": {
            "stat_prefix": "public_listener",
            "cluster": "local_app"
           }
          }
         ]
        }
       ]
      },
      "last_updated": "2021-06-17T13:44:44.883Z"
     }
    },
    {
     "name": "destination:127.0.0.1:8999",
     "draining_state": {
      "version_info": "00000001",
      "listener": {
       "@type": "type.googleapis.com/envoy.api.v2.Listener",
       "name": "destination:127.0.0.1:8999",
       "address": {
        "socket_address": {
         "address": "127.0.0.1",
         "port_value": 8999
        }
       },
       "filter_chains": [
        {
         "filters": [
          {
           "name": "envoy.tcp_proxy",
           "config": {
            "stat_prefix": "upstream.destination.default.dc1",
            "cluster": "destination.default.dc1.internal.6edd8b22-c2ee-068f-2ec8-0609c755e8da.consul"
           }
          }
         ]
        }
       ]
      },
      "last_updated": "2021-06-17T13:44:44.883Z"
     }
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump"
  }
 ]
}

At this point I don't know what I'm looking at, but it seems like the envoy config is still holding on to my destination destination? LMK if I'm just holding it wrong.
@rboyer
Copy link
Member

rboyer commented Jun 17, 2021 •
edited

After the reconfig the upstream listener is listed as draining_state meaning envoy keeps it around for a period of time after change/removal. I mostly ever encountered this in the context of modifying the listener in which case the listening socket is transferred to the updated listener and the old one is orphaned to drain existing connections. I have yet to be in a scenario where there was no need for transfer like this.

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining

If you wait long enough do you see this envoy log on the downstream?

[2021-06-17 09:16:43.970][2409938][info][upstream] [source/server/lds_api.cc:60] lds: remove listener 'destination:127.0.0.1:8999'

I locally reproduced the scenario and by the time I was executing the curl it didn't work anymore.

It may have also been one of those fun connection pooling issues if you weren't explicitly switching the service to an L7 protocol. Connections would have been established when the now-draining listener was actively listening, but after the listening socket was destroyed if you were using a pooled long-lived connection from curl it wouldn't have re-dialed, instead preferring to use the established socket.

If you set this config entry out of band before redoing your test scenario does the errant behavior still manifest?

kind = "service-defaults"
name = "destination"
protocol = "http"

@shoenig
Copy link
Member Author

shoenig commented Jun 17, 2021

Annnnnd it did start finally failing, after about 12 minutes

2021/06/17 14:31:59 INFO  [client]  -> GET response code: (200)

2021/06/17 14:43:41 ERROR [client]  -> GET error: Get "http://127.0.0.1:8999": dial tcp 127.0.0.1:8999: connect: connection refused

Feel free to close this out if that's working as intended, but this long of a lingering listener seems like a lot?

@rboyer
Copy link
Member

rboyer commented Jun 17, 2021

The envoy default value is 10m:
https://www.envoyproxy.io/docs/envoy/latest/operations/cli.html?highlight=drain#cmdoption-drain-time-s

@shoenig
Copy link
Member Author

shoenig commented Jun 17, 2021

Awesome

@shoenig shoenig closed this as completed Jun 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rboyer @shoenig