Please support multiple IP / interfaces in bind_addr and friends

[ieugen]

Hi,

Nomad docs suggest it can listen to only 1 IP address.
A lot of servers usually have 2 interfaces (1 private and 1 public) but it's not uncommon to have more IP addresses.
This is more true when we talk about IPV6 which has a huge address space and you have a link local and a public IP address by default.

I believe nomad has a limitation by supporting only 1 IP address.
For example PostgreSQL can listen to multiple IP addresses since a long time ago https://www.postgresql.org/docs/9.5/runtime-config-connection.html .

The issue I've personally hit was that leader election failed because I was binding to 0.0.0.0 leader election was done over the Docker IP address (172.17.x.x).

Binding to public IP created issues with mutual TLS auth - I could not connect to that since my certificates did not include the public IP.

All in all I'm sure there are workarounds to fixing my issues however, having the ability to listen to the interfaces I want will make setup easier and some use cases possible.

Thanks,
Eugen

[shomodj]

bind_addr supports go-sockaddr format

or you can separate them with spaces:

bind_addr = "127.0.0.1 10.10.10.10"

you also might look into advertise

or I'm missing something?

[ieugen]

I get this:

Bind address resolution failed: Multiple addresses found ("127.0.0.1 10.10.0.4"), please configure one.

Nomad v0.12.7 (6147cb5)

[shomodj]

Sorry @ieugen you are right, consul client_addr can do that, I assumed nomad does the same thing.

You can do this for now:

/tmp/nomad.hcl

data_dir = "/tmp/nomad"
bind_addr = "0.0.0.0"

advertise {
  http = "172.27.40.194"
}

nomad agent -server -config=/tmp/nomad.hcl

nomad agent -server -config=/tmp/nomad.hcl
==> Loaded configuration from /tmp/nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 172.27.40.194:4646; RPC: 192.168.86.30:4647; Serf: 192.168.86.30:4648
            Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
                Client: false
             Log Level: INFO
                Region: global (DC: dc1)
                Server: true
               Version: 0.12.1

and if you want jobs/tasks to listen on specific interface you can use network_interface

HTH

[ieugen]

Thanks,

I managed to fix it but the issue is that I still have an open port on a network that I have to firewall.

Also, I've checked with consul (latest release) and it seems to have a similar issue:

bind_addr = "0.0.0.0"

consul[572647]: ==> Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.

With bind_addr = "127.0.0.1 10.10.0.4"

consul[572881]: ==> bind_addr cannot contain multiple addresses. Use 'addresses.{dns,http,https}' instead.

[ieugen]

To give some feedback:

As a user I should be able to specify which network interfaces (IP addresses) my applications should bind to.
I should be able to choose the interface I would like my clients to prefer.

Right now it seems I can only bind to one address IP or all.
I can use advertise to choose one.

[shomodj]

@ieugen you are right, at the moment it looks like only consul client_addr supports this:

In Consul 1.0 and later this can be set to a space-separated list of addresses to bind to, 
or a go-sockaddr template that can potentially resolve to multiple addresses.

somehow I got the expression that all *_addr config options do this :(

[angrycub]

We talked about this a little in Gitter, but I wanted to capture this here for other folks as well.

Nomad currently supports two different schemes to use multiple IP addresses for your workload in Nomad.

• Multi Interface Networking - You can define multiple host_networks on a client and then refer to those networks in the port definition of your job specification. There's more information about this solution here.

• CNI plugins - CNI Plugins allow you to define fairly complex configurations and then have tasks create networks based on the CNI configuration. This is an advanced configuration and requires non-trivial background information about the plugins and configuration. You can read more about this here.

[ieugen]

@angrycub : My issue is with the client / server bind addresses.
I did get into workload networking.
Also, IPV6 would be very nice.

I have a static /56 IPV6 allocation from my provider.
For me it's not an issue to move almost all of my workload to IPV6.
Static IPV4 allocations are scarce.

[vnikolov88]

This is still an issue, @angrycub's comment is not really relevant, the issue here is you want to have the Nomad Server accessible only from the private network and loopback, binding to all interfaces and advertising only the private one works but you need to rely on the firewall to block the publicly bound interface. Can't the guys from the Consul team and Nomad team just sit in a room and standardized this?

[marco-m]

Hello @tgross, any news about the roadmap for this ticket? I think that @vnikolov88 comment above explains perfectly why this feature is important, better than the initial description, and might help in deciding the priority. Thanks!

[marco-m]

Partial support just shipped in v1.2.4
🎉