Problem deploying system task/Consul auto-join questions

[cetex]

I'm trying to deploy nginx through nomad as a system task and i'm failing to understand why it doesn't work, maybe i'm just blind or maybe i've seriously misunderstood something..

job "nginx" {
    datacenters = ["dc1", "dc2"]
    type = "system"
    priority = 50
    constraint {
        attribute = "${attr.kernel.name}"
        value = "linux"
    }
    constraint {
        attribute = "${consul.datacenter}"
        value = "dc1"
    }
    # Configure the job to do rolling updates
    update {
        stagger = "10s"
        max_parallel = 1
    }
    group "nginx" {
        restart {
            attempts = 10
            interval = "5m"
            delay = "25s"
            mode = "delay"
        }
        task "nginx" {
            driver = "docker"
            config {
                image = "docker-repo.service.consul:5000/trusty/nginx:latest"
                network_mode = "host"
                interactive = true
                command = "/launch-nginx.sh"
                args = []
            }
            env {
                ROLE = "${ROLE}"
                HOSTNAME = "$(hostname)"
                DATACENTER = "${DATACENTER}"
                RACK = "${RACK}"
                CLUSTER = "${CLUSTER}"
                local_ipv4 = "${local_ipv4}"
            }
            service {
                name = "${TASKGROUP}-nginx"
                tags = ["global", "nginx"]
                port = "http"
                check {
                    name = "alive"
                    type = "tcp"
                    interval = "10s"
                    timeout = "2s"
                }
            }
            resources {
                cpu = 19200 # 20% of 40cores at 2.4Ghz.
                memory = 16000 # 16GB
                network {
                    mbits = 2000
                    port "http" {
                        static = "80"
                    }
                    port "https" {
                        static = "443"
                    }
                }
            }
        }
    }
}

When trying to run it:

==> Monitoring evaluation "2bbb98d3"
    Evaluation triggered by job "nginx"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "2bbb98d3" finished with status "complete" but failed to place all allocations:
    Task Group "nginx" (failed to place 22 allocations):
      * Class "slave" filtered 1 nodes
      * Constraint "${consul.datacenter} = dc1" filtered 1 nodes

I'm trying to deploy this to s01 and s02 in dc1

# nomad node-status
ID        DC   Name         Class   Drain  Status
ea233f64  dc1  s01.dc1      slave   false  ready
e3073141  dc1  s02.dc1      master  false  ready
ce565d96  dc2  s17.r13.dc2  slave   false  ready
10a3a35c  dc2  s18.r13.dc2  slave   false  ready
22def55f  dc2  s19.r13.dc2  slave   false  ready
b21fcfae  dc2  s10.r13.dc2  slave   false  ready
92cba516  dc2  s09.r13.dc2  slave   false  ready
d96b3acd  dc2  s12.r13.dc2  slave   false  ready
e55af94a  dc2  s20.r13.dc2  slave   false  ready
ae6e3ba5  dc2  s08.r13.dc2  slave   false  ready
4e59d2d4  dc2  s07.r13.dc2  slave   false  ready
09c2f638  dc2  s11.r13.dc2  slave   false  ready
520e6d42  dc2  s06.r13.dc2  slave   false  ready
2ccaa2f3  dc2  s16.r13.dc2  slave   false  ready
973e2db0  dc2  s05.r13.dc2  slave   false  ready
9442eabb  dc2  s04.r13.dc2  slave   false  ready
f968b71c  dc2  s03.r13.dc2  slave   false  ready
9c429e1f  dc2  s02.r13.dc2  slave   false  ready
4fd545f9  dc2  s01.r13.dc2  slave   false  ready
ae938452  dc2  s13.r13.dc2  master  false  ready
7ee97540  dc2  s15.r13.dc2  master  false  ready
9b6e4c7b  dc2  s14.r13.dc2  master  false  ready

As you can see i'm basically trying to run jobs on the nomad masters and failing pretty badly.

nomad version
Nomad v0.4.1

Commandline we use to run nomad:

nomad agent -server -bootstrap-expect=3 -client -config /tmp/nomad.cfg
cat /tmp/nomad.cfg
datacenter = "dc1"
client {
  enabled = true
  node_class = "master"
}
data_dir = "/data/1/nomad"
bind_addr = "0.0.0.0"
telemetry {
  "statsd_address" = "127.0.0.1:8125"
}
advertise {
  http = "x.x.x.x:4646"
  rpc = "x.x.x.x:4647"
  serf = "x.x.x.x:4648"
}

[jippi]

does it place 0 allocations in your cluster?

e.g. nomad status nginx shows nothing?

[cetex]

sadly no.

# nomad status nginx
ID          = nginx     
Name        = nginx     
Type        = system
Priority    = 50
Datacenters = dc1,dc2
Status      = dead
Periodic    = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
nginx       21      0         0        0       0         0

Allocations
No allocations placed

[jippi]

can you try and check the evaluation from the API and see if there is any additional information?

also, debug wise, do nomad detect >1gbit network interface and the resources you require?

e.g. nomad node-status -verbose 22def55f - which will also output if consul.datacenter is as you expect :)

[cetex]

Probably on to something there. (This is one of the two nodes in dc1 i want to run the job in)
I only want to deploy to this DC, not the other one..
How do i match properly on DC?

And, where does this filter come from?
* Class "slave" filtered 1 nodes

nomad node-status -verbose ea233f64
ID     = ea233f64-146d-2d42-12e5-d840bb7ad197
Name   = s01
Class  = slave
DC     = dc1
Drain  = false
Status = ready
Uptime = 50h27m40s

Allocated Resources
CPU          Memory       Disk         IOPS
0/57600 MHz  0 B/252 GiB  0 B/121 GiB  0/0

Allocation Resource Utilization
CPU          Memory
0/57600 MHz  0 B/252 GiB

Host Resource Utilization
CPU           Memory           Disk
78/57600 MHz  2.5 GiB/252 GiB  805 MiB/122 GiB

Attributes
arch                      = amd64
cpu.frequency             = 1200
cpu.modelname             = Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
cpu.numcores              = 48
cpu.totalcompute          = 57600
driver.docker             = 1
driver.docker.version     = 1.11.0
driver.exec               = 1
kernel.name               = linux
kernel.version            = 4.0.9-040009-generic
memory.totalbytes         = 270375706624
nomad.revision            = '8fdc55e16b54f176a711c115966ba234e8bb7879+CHANGES'
nomad.version             = 0.4.1
os.name                   = ubuntu
os.version                = 14.04
unique.cgroup.mountpoint  = /sys/fs/cgroup
unique.hostname           = s01
unique.network.ip-address = x.x.x.x
unique.storage.bytesfree  = 130096115712
unique.storage.bytestotal = 130951184384
unique.storage.volume     = /dev/zram0

[dadgar]

Believe they are getting filtered by this constraint:

constraint {
        attribute = "${consul.datacenter}"
        value = "dc1"
    }

[cetex]

Yeah.. I've done some tests and cut it down a bit, seems like you're right.

Works once i removed those constraints and just set datacenter = ["dc1"]
One problem i have though is that i might want to deploy this nginx service separately to "dc1" and "dc2", (for example with different parameters, testing one image / version of the job in one dc while not changing anything in the other), how would i do that without naming the job differently for each datacenter now when i have tied everything together through consul? (They're different consul clusters, but tied together with "consul join -wan" and somehow nomad finds and lists all datacenters automagically..)

When i had datacenter = ["dc1", "dc2"] it tried to place the job on all nodes in both dc's which is not what i had in mind.. (which is why i had that consul.datacenter constraint there, which didn't work at all it seems)

[dadgar]

Hey,

If your Consul clusters are federated Nomad will find other Nomad servers by searching through the various Consul datacenters!

To do that type of deployment I see two ways:

1. Have a single job with two task groups. Inside each task group have a constraint:

constraint {
   attribute = "${node.datacenter}"
   # The other task group would have "dc2"   
   value = "dc1" 
}

And then you can run the same job and each task group would have the parameters for the DC it is targeting.

2. Have two job files with different names. The name of a job in Nomad is its unique identifier so they would have to be unique. This is probably what I would recommend but both will work so whatever is simpler for you.

[cetex]

Ok. Is there any way to disable federation but use consul to find local neighbours?

We use "consul join -wan" to get dns lookups to work from our jumpstation to the different datacenters, (to find mesos, aurora and similar..) and have production, integration and testing tied together like this so we can reach all of them from our jumpstations.
We only use this to find datacenters to be able to deploy / use it for management of them, no internal communication between the services we run in the different dc's, and in fact most dc's can't talk to each other since it's blocked in the network (although these two mentioned here can since they're both in testing, although different testing environments..)

Maybe this is what "region" in nomad is about? (set region to dc1 or dc2 instead of default maybe?)

Would be pretty nice to also tell consul not to build a full mesh between all dc's and just setup point-to-point connectivity between some dc's (think "consul join -p2pwan" to build a hub & spoke setup where one dc knows all other dc's and each other dc only knows about the hub dc/cluster)

[dadgar]

Since Nomad can manage many datacenters in a single region we search through federated Consul datacenters to search for the relevant Nomad Servers since it is a common scenario to have many Consul DCs in a single Nomad region.

Is there a reason you would want Nomad to not federate?

[cetex]

Yes, an important issue is that I want to limit the failure domain in case someone screws up.
Also, in case nomad screws up, i don't want to risk bringing down multiple dc's at once.
And what happens when we have a 1Mbit VPN across the world we only plan to use for management, 500ms roundtrip and nomad decides to elect the node on the other side of the world as leader for some reason just because we've wan-joined that dc in consul to get dns lookups working from our jumpstation?

And stuff like this makes me worried, wiped out the nodes and restarted nomad with a new region set to try to isolate it entirely from the other clusters.
Immediately tries to talk to other nomad clusters it shouldn't even touch (from my perspective) anyways which is a quite big issue for us. (This is with v0.5.0-rc2 though compared to v0.4.1 above)

    2016/11/15 08:46:20 [INFO] snapshot: Creating new snapshot at /data/1/nomad/server/raft/snapshots/508-114701-1479199580668.tmp
    2016/11/15 08:46:20 [INFO] raft: Copied 29831 bytes to local snapshot
    2016/11/15 08:46:20 [INFO] raft: Installed remote snapshot

I expect a setting which only uses local consul cluster to find local nomad neighbours, and which doesn't try to / refuses to talk to any other cluster unless being explicitly told to talk to someone else. What if that snapshot is broken / compromised in the other datacenter or there's a bug that gets triggered in a couple of months when we upgrade to 0.6.0 or similar which brings down "*"?, that simply can't happen.

Consul has local leaders in each datacenter and doesn't try to build a global cluster unless being told to specifically, that global cluster (to my knowledge) isn't used for much else than allowing certain queries to be passed between the datacenters directly, each datacenter still has it's own leaders and those leaders are autonomous, if connectivity to central consul is lost I know that the datacenter won't loose quorum, die or something else. I have to go out of my way quite a bit to accidently write / delete in all key-value stores globally, or change the registered services in each datacenter. The only downside currently with consul is that it expects to have full connectivity between all datacenters and then complains when we drop traffic between all datacenters that's not the "hub" datacenter, but that's about it.
With nomad it seems like if I give people access to the test environment they can still do changes to production if consul in those two datacenters happens to know about each other, which is definitely not something we want.

And then we have stuff like this which just started occuring, not sure why. But nomad simply won't start for some reason. I have no idea why, but it seems like it's trying to connect to the other datacenter and expect some leadership election to happen which simply doesn't happen for some reason. If it didn't try to connect to the other datacenter (if i block all traffic between the datacenters) it just works as i expect it to.

==> Starting Nomad agent...
==> Nomad agent configuration:

                 Atlas: <disabled>
                Client: true
             Log Level: INFO
                Region: dc1 (DC: dc1)
                Server: true
               Version: 0.5.0rc2

==> Nomad agent started! Log data will stream in below:

    2016/11/15 09:43:07 [INFO] raft: Node at xx.xx.xx.149:4647 [Follower] entering Follower state (Leader: "")
    2016/11/15 09:43:07 [WARN] memberlist: Binding to public address without encryption!
    2016/11/15 09:43:07 [INFO] serf: EventMemberJoin: s01.dc1 xx.xx.xx.149
    2016/11/15 09:43:07.016247 [INFO] nomad: starting 48 scheduling worker(s) for [service batch system _core]
    2016/11/15 09:43:07.016473 [INFO] client: using state directory /data/1/nomad/client
    2016/11/15 09:43:07.016517 [INFO] client: using alloc directory /data/1/nomad/alloc
    2016/11/15 09:43:07.016951 [INFO] fingerprint.cgroups: cgroups are available
    2016/11/15 09:43:07.018743 [INFO] nomad: adding server s01.dc1 (Addr: xx.xx.xx.149:4647) (DC: dc1)
    2016/11/15 09:43:07.019659 [INFO] fingerprint.consul: consul agent is available
    2016/11/15 09:43:07.044787 [INFO] client: Node ID "e6dff786-ea52-7e51-7a8d-11d5c9799a65"
    2016/11/15 09:43:07 [WARN] raft: Failed to get previous log: 118153 log not found (last: 0)
    2016/11/15 09:43:07 [INFO] snapshot: Creating new snapshot at /data/1/nomad/server/raft/snapshots/508-114697-1479202987857.tmp
    2016/11/15 09:43:07 [INFO] raft: Copied 30392 bytes to local snapshot
    2016/11/15 09:43:07 [INFO] raft: Installed remote snapshot
    2016/11/15 09:43:10.066595 [WARN] server.consul: failed to query service "nomad" in Consul datacenter "test-XXXXXX": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.205.0.239:8300: getsockopt: no route to host)
    2016/11/15 09:43:11.066421 [WARN] server.consul: failed to query service "nomad" in Consul datacenter "com1_XXXXXX": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.221.64.52:8300: getsockopt: no route to host)
    2016/11/15 09:43:12.066378 [WARN] server.consul: failed to query service "nomad" in Consul datacenter "core": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.0.0.10:8300: getsockopt: no route to host)
    2016/11/15 09:43:12.118468 [ERR] client: registration failure: No cluster leader
    2016/11/15 09:43:13.114567 [WARN] server.consul: failed to query service "nomad" in Consul datacenter "com1_XXXXXXX": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.207.64.120:8300: getsockopt: no route to host)
    2016/11/15 09:43:14.114243 [WARN] server.consul: failed to query service "nomad" in Consul datacenter "build": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.202.201.123:8300: getsockopt: no route to host)
    2016/11/15 09:43:14 [INFO] serf: EventMemberJoin: s15.rxx.dc2.global xx.xx.xx.82
    2016/11/15 09:43:14 [INFO] serf: EventMemberJoin: s14.rxx.dc2.global xx.xx.xx.81
    2016/11/15 09:43:14 [INFO] serf: EventMemberJoin: s13.rxx.dc2.global xx.xx.xx.80
    2016/11/15 09:43:14.162054 [INFO] nomad: adding server s15.rxx.dc2.global (Addr: xx.xx.xx.82:4647) (DC: dc2)
    2016/11/15 09:43:14.162113 [INFO] nomad: adding server s14.rxx.dc2.global (Addr: xx.xx.xx.81:4647) (DC: dc2)
    2016/11/15 09:43:14.162142 [INFO] nomad: adding server s13.rxx.dc2.global (Addr: xx.xx.xx.80:4647) (DC: dc2)
    2016/11/15 09:43:14.256927 [INFO] server.consul: successfully contacted 3 Nomad Servers
    2016/11/15 09:43:16.234659 [ERR] client.consul: error discovering nomad servers: 8 error(s) occurred:

* unable to query service "nomad" from Consul datacenter "com1_XXXXXX": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.221.128.181:8300: getsockopt: no route to host)
* rpc error: No path to region
* rpc error: No path to region
* rpc error: No path to region
* unable to query service "nomad" from Consul datacenter "build": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.202.201.123:8300: getsockopt: no route to host)
* unable to query service "nomad" from Consul datacenter "com1_XXXXXX": Get http://127.0.0.1:8500/v1/catalog/service/nomad?dc=com1_XXXXXX&near=_agent&stale=&tag=rpc&wait=2000ms: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
* unable to query service "nomad" from Consul datacenter "test-XXXXXX": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.205.128.155:8300: getsockopt: no route to host)
* unable to query service "nomad" from Consul datacenter "core": Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.0.0.10:8300: getsockopt: no route to host)
    2016/11/15 09:43:17.056206 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.092430 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.095460 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.108925 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.108929 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.153295 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.156878 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.159981 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.166179 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.168129 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.177177 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.179162 [ERR] worker: failed to dequeue evaluation: No cluster leader
    2016/11/15 09:43:17.181742 [ERR] worker: failed to dequeue evaluation: No cluster leader
<lots and lots of these>

I wipe these new nodes again, stop nomad in the other datacenter, restart it and it "just works" to launch it properly.
Also, wiped them yet again, nomad running in the other datacenter but added iptables rules that blocks communication between the datacenters and it "just works" again.

I'm not sure why it breaks right now actually, could it be new version?, some state about this datacenter that the other datacenter believes it has?

For us things like this is a dealbreaker, I thought we could get this deployed and tested relatively soon (I really like the idea behind nomad and that we could drop our legacy way of deploying the platform-level jobs) but at the current state we either can't run nomad at all, or we can't use consul as we use it today (Which means we can't "federate" consul, which in turn means we'd have to redesign large parts of our internal dns infrastructure to be able to run nomad, which probably won't happen)

[cetex]

@dadgar any thoughts on this?

[dadgar]

@cetex Sorry I didn't see your response. Nomad has two concepts that are important to understand what we are doing with Consul.

1. A region: A region is a scheduling domain that consists of Nomad Servers and Clients. The clients/servers may be spread across many datacenter but the set of servers manage all scheduling for that region and can not make scheduling decisions in others.

2. Federation: Federation allows one region to be aware of the existence of another. This allows a job to be submitted in region A that targets region B and the Nomad servers will gracefully forward to the correct scheduler and do nothing more.

We use Consul so that servers within a region discover each other and so that clients can discover their servers.

As we do this scan through Consul, we can also detect servers that are part of separate Nomad regions and we automatically federate them. This does not mean we compromise the configuration of the servers (we won't elect something across the world unless you have placed servers with the same region across the world which will lead to problems regardless).

As for not being able to start that looks like a configuration problem and would be more than happy to help if you want to file a new issue with the relevant configs and nomad/consul network topology!

Thanks,
Alex

[cetex]

Alright!

So in our case we should set region=datacenter_name to have nomad only elect leaders from the local datacenter?
Does this also mean that names for jobs within multiple regions can be the same but treated entirely separately?
API in region1 would be handled entirely separately from API in region2? (this is a requirement for us)

It would be nice to be able to disable federation entirely but still retain the functionality to lookup other nomad nodes within the local consul datacenter though, since if nomad in Dc1 (region1) doesn't even know about nomad in Dc2 (region2) accidental screw-ups doesn't happen as easily, but nomad agents and servers would still find each other in the local datacenter.
In our case this could mean that a user is deploying something with breaking changes to dev-environment but if the job is defined with a production datacenter nomad would happily try to deploy it to production instead which is a pretty bad scenario as it could bring production down.

When it comes to deploying a job defined for Dc1 by contacting nomad in Dc2 I consider the intent of that deploy unclear and the deploy should be denied instead of trying to work around the error within nomad.

The only way to do what we want today seems to be to disable consul integration entirely in nomad, which means we'd loose a lot of functionality, or remove all federation between consul datacenters which would also remove a lot of functionality.

Another option would be to make consul support a hub-and-spoke design, where the hub (our jumpstation) can reach and know of all spokes (datacenters: prod, dev) and all spokes know of the hub, but where two spokes don't know about each other.

Regarding starting nomad i'm willing to try to recreate it later on if we can resolve these issues first.

[dadgar]

So in our case we should set region=datacenter_name to have nomad only elect leaders from the local datacenter?
Does this also mean that names for jobs within multiple regions can be the same but treated entirely separately?
API in region1 would be handled entirely separately from API in region2? (this is a requirement for us)

This is correct

It would be nice to be able to disable federation entirely but still retain the functionality to lookup other nomad nodes within the local consul datacenter though, since if nomad in Dc1 (region1) doesn't even know about nomad in Dc2 (region2) accidental screw-ups doesn't happen as easily, but nomad agents and servers would still find each other in the local datacenter.

Can you please file a separate issue for this.

When it comes to deploying a job defined for Dc1 by contacting nomad in Dc2 I consider the intent of that deploy unclear and the deploy should be denied instead of trying to work around the error within nomad.

I think there may be some confusion between Consul's and Nomad's topology. Please take a look at our architecture page. At a high level, a set of Nomad servers manage a Region. A region can consist of multiple data centers. Jobs can span datacenter but not regions. This is useful if you have a job that should be resilient to a single datacenter failure as Nomad can detect and reschedule onto a separate DC.

There is no state shared across regions. The Nomad servers can communicate via Serf (a gossip protocol) which allows federated regions to forward jobs to the appropriate region. This is useful if you have a service creating jobs, it can submit jobs to its local server and have the server forward to the appropriate regional servers.

[dadgar]

Going to close this as the relevant issue has been filed separately

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.