Tags in sidecar-proxy service break Fabio

[liujingchen]

Nomad version

Nomad v0.10.1 (0d4e5d9)

Operating system and Environment details

Ubuntu 18.04
Consul v1.6.1
Fabio 1.5.11

Issue

We are using Fabio to proxy traffic to applications running in Nomad. But the new Consul Connect makes it not working anymore due to the tags in sidecar-proxy services.
Use the official example but some changes to the count-dashboard service:

job "countdash" {
  datacenters = ["dc1"]

  group "api" {
     ......
  }

  group "dashboard" {
    network {
      mode = "bridge"

      port "http" {
        # no need for static port, Fabio can get the dynamic port and proxy to it
        to     = 9002
      }
    }

    service {
      name = "count-dashboard"
      port = "http" #  advertise the dynamic port
      #  This tag tells Fabio to proxy HTTP requests with Host "dashboard.com" to this service.
      tags = ["urlprefix-dashboard.com/"]
      #  Add a health check to make Fabio proxy traffic to it.
      check {
          name = "count-dashboard-check"
          type = "http"
          port = "http"
          path = "/"
          interval = "5s"
          timeout  = "2s"
      }

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "count-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "dashboard" {
      ......
    }
  }
}

But this doesn't work now because the Envoy sidecar will also register itself to Consul and copying all the tags from count-dashboard, like this:

~$ curl -s http://localhost:8500/v1/catalog/service/count-dashboard-sidecar-proxy | jq
[
  {
    ......
    "ServiceName": "count-dashboard-sidecar-proxy",
    "ServiceTags": [
      "urlprefix-dashboard.com/"
    ],
    ........
]

This confused Fabio. Fabio will misunderstand and send HTTP request of "dashboard.com" to this sidecar proxy too, and all these request will fail because the sidecar only accept TLS connection.

I don't think it is necessary for the sidecar proxy to copy the tags from the application service at all. Please consider changing this behavior, or provide an optional parameter to stop copying the tags.

[tgross]

Hi @liujingchen and thanks for reporting this! This sort of thing was reported in #6415 but I really appreciate that you provided a specific production use case here. We've fixed this in #6448, which is slated for the 0.10.2 release very soon.

[wadel86]

@liujingchen thanks for raising this issue !

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.