drain mode clogging

[devennorton]

The setup
We're evaluating using nomad and have set up an environment in ec2 for the purposes.

what we have is a cluster of 3 hosts running consul server, 1 host running nomad server, and 3 app hosts running nomad agents (as well as consul agents)

on the nomad server if we attempt to put a host into drain mode

nomad node-drain -enable 45906f8c-6cc9-4471-29d9-002ebb62fdd6z

we can see its set in drain mode:

[root@ip-172-16-0-30 nomad]# nomad node-status
ID                                    DC   Name   Class  Drain  Status
303dabe6-93e0-0d8c-5f2f-ca2c625dfa5f  dc1  app02  app    false  ready
45906f8c-6cc9-4471-29d9-002ebb62fdd6  dc1  app03  app    true   ready
92b4053b-375f-9c48-75a6-1ef311ff826e  dc1  app01  app    false  ready

but from the log on the app host it only appears to stop 1 service:

    2016/01/13 23:15:31 [ERR] client: dropping update to alloc '4d7a8c7c-b90c-f088-391f-74cadfb50d78'
    2016/01/13 23:15:36 [INFO] driver.docker: stopped container 4bccb0b1cf6834752afb8bf3928cb60190405a6c94d6f67710c842a7d2f6b46b
    2016/01/13 23:15:44 [INFO] driver.docker: removed container 4bccb0b1cf6834752afb8bf3928cb60190405a6c94d6f67710c842a7d2f6b46b
    2016/01/13 23:15:44 [INFO] driver.docker: image 41ae09ebc0d4c9fd3c06504aee906b76081a62607855997f881b77a80449c959 is still in use by 12 container(s)
    2016/01/13 23:15:44 [INFO] consul: deregistering service pyserver-pyserver-group-pyserver-task with consul
    2016/01/13 23:15:44 [ERR] client: dropping update to alloc '4d7a8c7c-b90c-f088-391f-74cadfb50d78'

and if we check the status for that node with nomad node-status 45906f8c-6cc9-4471-29d9-002ebb62fdd6 we get

### Allocations
ID                                    EvalID                                JobID     TaskGroup       DesiredStatus  ClientStatus
30304ec3-059d-5c33-b689-4a622052f941  b84bcd11-7152-a9ce-23b6-56dc05b205f9  pyserver  pyserver-group  run            running
ae7c9fb3-e4ae-69be-68d3-7d9ba174c0f7  8639a632-2c2c-556d-154d-c48258f4622a  pyserver  pyserver-group  run            running
d7681473-aafc-de94-4c12-b3d06e53c98e  8639a632-2c2c-556d-154d-c48258f4622a  pyserver  pyserver-group  stop           dead
33f146f6-fb44-95a2-759f-e711638e60fe  8639a632-2c2c-556d-154d-c48258f4622a  pyserver  pyserver-group  run            running

if we run nomad node-drain -enable 45906f8c-6cc9-4471-29d9-002ebb62fdd6z again it will stop 1 more, and we see that reflected in the nodes status:

### Allocations
ID                                    EvalID                                JobID     TaskGroup       DesiredStatus  ClientStatus
30304ec3-059d-5c33-b689-4a622052f941  b84bcd11-7152-a9ce-23b6-56dc05b205f9  pyserver  pyserver-group  stop           dead
ae7c9fb3-e4ae-69be-68d3-7d9ba174c0f7  8639a632-2c2c-556d-154d-c48258f4622a  pyserver  pyserver-group  run            running
d7681473-aafc-de94-4c12-b3d06e53c98e  8639a632-2c2c-556d-154d-c48258f4622a  pyserver  pyserver-group  stop           dead
33f146f6-fb44-95a2-759f-e711638e60fe  8639a632-2c2c-556d-154d-c48258f4622a  pyserver  pyserver-group  run            running

nomad, and consul versions across the board are:

# nomad -version
Nomad v0.2.3
# consul version
Consul v0.6.1
Consul Protocol: 3 (Understands back to: 1)

[bastiaanb]

Same behavior with Nomad v0.3.0-dev (9e7e214).

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.