Consul Connect: Multiple services with the same upstream

[varsanojidan]

Hi, we recently ran into an issue when trying to build services around Consul Connect.

Setup
Deploy a Nomad job with multiple Services (all in the same container), each having their own sidecar proxy, and the same upstream destination/LocalBindPort.

Issue
Since each service has the same upstream, requests made within the container to that upstream could potentially end up routing through any of the services' sidecars, as a result we are seeing intermittent: "Connection Refused"

Our question is: Is there a way around this issue (or a better design practice) that would avoid us having to individualize the upstreams to the individual services in the container?

Any help would be greatly appreciated.

[schmichael]

Would you mind submitting the jobspec (or a minimal reproduction)?

...the same upstream destination/LocalBindPort.

I'm not sure why that works at all! If I'm understanding correctly you end up with multiple sidecars - each one trying to bind to the same port inside the group's network namespace! Maybe I'm misunderstanding something in which case the jobspec would clear things up.

Can you only specify the upstream on one service? If not I'd love to hear more about your use case as it may not be one we've fully considered.

[shoenig]

A small reproduction (correct me if I'm on the wrong track @varsanojidan!)

job "upstreams" {
  datacenters = ["dc1"]

  group "server" {
    network {
      mode = "bridge"
    }

    service {
      name = "server"
      port = 8999
      connect {
        sidecar_service {}
      }
    }

    task "server" {
      driver = "docker"
      config {
	image = "shoenig/simple-http:v1"
	args = ["server"]
      }
    }
  }

  group "clients" {
    network {
      mode = "bridge"
    }

    service {
      name = "client-a"
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "server"
              local_bind_port  = 8999
            }
          }
        }
      }
    }

    service {
      name = "client-b"
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "server"
              local_bind_port  = 8999
            }
          }
        }
      }
    }    

    task "client-a" {
      driver = "docker"
      config {
	image = "shoenig/simple-http:v1"
	args = ["client"]
      }
    }

    task "client-b" {
      driver = "docker"
      config {
	image = "shoenig/simple-http:v1"
	args = ["client"]
      }
    }    
  }
}

The client requests in this case work because one envoy is able to bind the listener while the other errors with,

[2021-06-18 19:05:59.888][8][warning][config] [source/common/config/grpc_subscription_impl.cc:107] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) server:127.0.0.1:8999: cannot bind '127.0.0.1:8999': Address already in use

This isn't about connecting to the same upstream, but rather both of the client services are trying to use the same bind address and port inside the group's network namespace. We could add some validation and reject such a job on job submission, since we know it is not going to work.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.