retry_join (inside server_join) does not support go-sockaddr syntax

[shantanugadgil]

Nomad version

Nomad v0.11.3 (8918fc8)

Operating system and Environment details

CentOS 8/Ubuntu 18.04/et.al

Issue

Nomad's retry_join directive only supports the go-discover syntax, but does not support the go-sockaddr syntax.
https://www.nomadproject.io/docs/configuration/server_join/#retry_join

Consul's retry_join supports both
https://www.consul.io/docs/agent/options#_retry_join

Reproduction steps

define a client section in the server config and define the client section as:

client {
  enabled    = "true"
  node_class = "nomad-server"

  server_join {
    retry_join = ["{{ GetInterfaceIP \"eth0\" }}"]
  }
}

Upon restarting the service, this will keep emitting error messages in the logs about how the retry_join parameter is invalid

* Failed to resolve {{ GetInterfaceIP "eth0" }}: lookup {{ GetInterfaceIP "eth0" }}: no such host

Job file (if appropriate)

n/a

Nomad Client logs (if appropriate)

n/a

Nomad Server logs (if appropriate)

* Failed to resolve {{ GetInterfaceIP "eth0" }}: lookup {{ GetInterfaceIP "eth0" }}: no such host

The use case here is of a typical quick one node server/client with minimal tinkering needed to the config file.

[shantanugadgil]

Any reason why the syntax has not been supported (originally) in Nomad's retry_join?

Having this greatly helps to setup a single node server + client mode (no, I do not want to use "dev" mode) without indulging in ip addr jugglery! 😄

[angrycub]

@shantanugadgil I saw your comment in #10404 and it raised questions for me. I've never seen a case where a Nomad agent running in client and server mode did not join itself properly in the absence of a join configuration. Could you perhaps share more about your configuration? I'm unclear as to why you'd need a join configuration at all.

[shantanugadgil]

@angrycub following are my configuration files:

==> client.hcl <==
# client settings

client {
  enabled = true

  node_class = "nomad_server"

  network_interface = "eth0"

  server_join {
    retry_join = ["192.168.0.4"]
    retry_max  = 0
  }
}

plugin "raw_exec" {
  config {
    enabled = true
  }
}

plugin "docker" {
  config {
    auth {
      config = "/root/.docker/config.json"
      # Nomad will prepend "docker-credential-" to the helper value and call
      # that script name.
      helper = "ecr-login"
    }
    
    gc {
      image_delay = "24h"
    }
    
    volumes {
      enabled = true
    }
  }
}

==> common.hcl <==
# common settings

bind_addr = "0.0.0.0"

datacenter = "dc1"

data_dir = "/var/lib/nomad"

name = "myservername"

disable_update_check = true

leave_on_interrupt = true

leave_on_terminate = true

enable_syslog = true

log_file = "/var/log/nomad.log"

log_rotate_bytes = 1048576

log_rotate_max_files = 3

addresses {
  http = "0.0.0.0"
}

advertise {
  http = "{{ GetInterfaceIP \"eth0\" }}"
  rpc  = "{{ GetInterfaceIP \"eth0\" }}"
  serf = "{{ GetInterfaceIP \"eth0\" }}"
}

==> server.hcl <==
# server settings

server {
  enabled = true

  # count should be equal to the number of servers
  bootstrap_expect = 1
  encrypt          = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

  server_join {
    retry_join = ["192.168.0.4"]
    retry_max = 0
  }
}

Are you suggesting that the server_join stanza itself is not needed in single node client + server mode?

Let me try commenting it out in the client.hcl and see if it works.

EDIT: Please note, the above configuration is actually adapted from the multi node setup, where servers and clients are different, so I am all ears on how to optimize this to remove hard coded IP addresses for single node setups :)