nomad client errors when server bootstrap value is set but server enabled is false

[vvitayau]

Nomad version

Nomad v0.9.3 (c5e8b66c3789e4e7f9a83b4e188e9a937eea43ce)

Operating system and Environment details

Red Hat Enterprise Linux Server release 7.6 (Maipo)

Issue

Nomad-client fails to start with https://www.nomadproject.io/docs/configuration/server.html#enabled is set to false.

It should startup properly and ignore the entire server section if this flag is set false.
I can circumvent the issue but was surprised to encounter this.

$ journalctl -fu nomad
-- Logs begin at Tue 2019-07-30 18:30:12 UTC. --
Jul 30 18:34:29  nomad[25608]: ==> Bootstrap requires server mode to be enabled
Jul 30 18:34:29  systemd[1]: nomad.service: main process exited, code=exited, status=1/FAILURE

Reproduction steps

Nomad works after removal the server section from the /etc/nomad.d/client.hcl configuration. It appears enabled=false is ignored by the agent to warrant this removal.

server {
  authoritative_region = "global"
  bootstrap_expect     = "3"
  enabled              = false
  encrypt              = "xxx"
}

[vvitayau]

logs of client failure on startup and working after removal of the server section from the configuration

$ sudo vi /etc/nomad.d/client.hcl
$ sudo service nomad restart; journalctl -fu nomad
Redirecting to /bin/systemctl restart nomad.service
-- Logs begin at Tue 2019-07-30 18:30:18 UTC. --
Jul 30 18:34:43 x systemd[1]: nomad.service: main process exited, code=exited, status=1/FAILURE
Jul 30 18:34:43 x systemd[1]: Unit nomad.service entered failed state.
Jul 30 18:34:43 x systemd[1]: nomad.service failed.
Jul 30 18:34:44 x systemd[1]: nomad.service holdoff time over, scheduling restart.
Jul 30 18:34:44 x systemd[1]: Stopped Nomad Agent.
Jul 30 18:34:44 x systemd[1]: start request repeated too quickly for nomad.service
Jul 30 18:34:44 x systemd[1]: Failed to start Nomad Agent.
Jul 30 18:34:44 x systemd[1]: Unit nomad.service entered failed state.
Jul 30 18:34:44 x systemd[1]: nomad.service failed.
Jul 30 19:54:16 x systemd[1]: Started Nomad Agent.
Jul 30 19:54:16 x nomad[23255]: ==> Loaded configuration from /etc/nomad.d/client.hcl
Jul 30 19:54:16 x nomad[23255]: ==> Starting Nomad agent...
Jul 30 19:54:16 x nomad[23255]: ==> Nomad agent configuration:
Jul 30 19:54:16 x nomad[23255]: Advertise Addrs: HTTP: xxx:4646
Jul 30 19:54:16 x nomad[23255]: Bind Addrs: HTTP: xxx:4646
Jul 30 19:54:16 x nomad[23255]: Client: true
Jul 30 19:54:16 x nomad[23255]: Log Level: INFO
Jul 30 19:54:16 x nomad[23255]: Region: global (DC: dc1)
Jul 30 19:54:16 x nomad[23255]: Server: false
Jul 30 19:54:16 x nomad[23255]: Version: 0.9.3
Jul 30 19:54:16 x nomad[23255]: ==> Nomad agent started! Log data will stream in below:
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.234Z [INFO ] client: using state directory: state_dir=/opt/nomad/data/client
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.241Z [INFO ] client: using alloc directory: alloc_dir=/opt/nomad/data/alloc
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.243Z [INFO ] client.fingerprint_mgr.cgroup: cgroups are available
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.245Z [INFO ] client.fingerprint_mgr.consul: consul agent is available
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.259Z [INFO ] client.fingerprint_mgr.vault: Vault is available
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.266Z [INFO ] client.plugin: starting plugin manager: plugin-type=driver
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.266Z [INFO ] client.plugin: starting plugin manager: plugin-type=device
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.337Z [INFO ] client: started client: node_id=xxx
Jul 30 19:54:16 x nomad[23255]: 2019-07-30T19:54:16.343Z [INFO ] client: node registration complete
Jul 30 19:54:23 x nomad[23255]: 2019-07-30T19:54:23.954Z [INFO ] client: node registration complete

[stale]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[tgross]

I'm going to tag @notnoop on this issue because he has PR #6047 open for it already and I don't want stalebot to close it.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.