address_mode=host uses consul address instead of nomad one

[urusha]

Nomad version

Nomad v0.12.5 (514b0d6)

Operating system and Environment details

debian 10 amd64

Issue

When using address_mode="host" in service stanza, and nomad bind_addr/advertise_addr is not the same as consul address, consul service is registered with consul's bind_addr/advertise_addr but not nomad one.
Don't know if this is a bug or a feature, but for now it seems there is no way to register service with nomad's ip address using address_mode="host" if consul address differs. Our case is registering 2 services for a task: with address_mode="driver" (direct access to docker's IP) and with address_mode="host" (access through proxy listening on the nomad IP; DNS name of the service is used for setting the right Host header)

Reproduction steps

consul:

bind_addr = 0.0.0.0
advertise_addr = ipC

nomad:

bind_addr = ipN
advertise {
    rpc = "ipN"
    http = "ipN"
    serf = "ipN"
}

job:

      service {
        name = "service1"
        address_mode = "host"
      }

service1 will be registered using ipC ip address.

[tgross]

The advertise/bind addresses for Consul and Nomad are for advertising and binding for Consul and Nomad's own HTTP, RPC, etc. so you're right that those won't change the advertised address of a Nomad task in Consul. That address is coming from fingerprinting the host. However, what you should be able to do is to use a host_network on the client and bind to that on the job, and have the address respected.

Unfortunately it appears that the service stanza won't advertise the address allocated if the network.port[n].host_network is configured. Which makes this another case of this bug: #8615

Example client configuration on a host with two IPs (10.0.2.15 and 10.199.0.200)

client {
  host_network "private" {
    cidr = "10.199.0.200/24"
    reserved_ports = "22"
  }
}

Example job:

job "example" {
  datacenters = ["dc1"]

  group "cache" {
    network {
      port "db" {
        to = 6379
        host_network = "private"
      }
    }

    service {
      name = "redis"
      # (unfortunately this doesn't work either way)
      # address_mode = "host"
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"

        ports = ["db"]
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

The results:

$ nomad alloc status
..
Allocation Addresses
Label  Dynamic  Address
*db    yes      10.199.0.200:23385 -> 6379

$ curl -s localhost:8500/v1/catalog/service/redis | jq '.[0].ServiceAddress'
"10.0.2.15"

[tgross]

@urusha I've dropped a link to this issue in #8615 but seeing as how that's already got someone assigned to it, I'm going to close this as a duplicate. But thank you for reporting this! Please keep an eye on #8615.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.