Nomad should not set Consul ServiceAddress / setting ServiceAddress should be configurable

[mimato]

Nomad version

Nomad v0.8.4 (dbee1d7d051619e90a809c23cf7e55750900742a)

Operating system and Environment details

Ubuntu 16.04

Consul v0.7.5
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Issue

When Nomad registers a service in Consul, it always sets the ServiceAddress for that service. This means that the address translation that works with the node address for WAN/LAN addresses will not work for these registered services. There are workarounds when using Consul Template (just use the .NodeAddress), but when the services are accessed through DNS, Consul will only provide the ServiceAddress (which is the LAN address), even when the service is being queried from another datacenter.

We have consul clusters in multiple regions, and nodes have different WAN and LAN addresses, and sometimes need to be addressed cross-region through DNS. It would be ideal if we could ask Nomad to not set the service address (perhaps in the job file or in the server or client config), so that the node address will be used and translated between WAN/LAN properly.

Reproduction steps

Create two Consul clusters in separate regions (region1 and region2), create a Nomad cluster in one of those regions (region1).

If you query a service in region1 from region2, via DNS, you will get the LAN IP of that service, rather than the translated WAN node address.

Possibly Related Issues

#3629
#2770

[joshuaclausen]

Agreed. This is essentially the same as the "running Nomad behind NAT" scenario. Issue #2770 does seem to be the same. I'd like to be able to run a "microservice behind NAT" scenario. This would enable Fabio, for instance, to correctly route to the public IP of a service when the host of that service is behind NAT.

[roman-vynar]

We also need to register Consul services using FQDN by Nomad because we want to use HTTPS with TLS verify for discovered services.
Also a service healthcheck address should be also configurable and allow us to set FQDN to do an internal consul http check using https://hostname address w/o any InsecureVerify True because it's insecure :)

Currently, we patched Nomad like this Quiq@a081ee2

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.