clusterf-docker: Set weight=0 for stopping containers

[SpComb]

Containers that are in the process of being stopped, but not yet down, should still be present as backends with their weight set to zero. Once the container is torn down (the IP address no longer exists), it is safe to remove the backend.

Ideally this would allow clean shutdown of service backends, with the docker stop ... command triggering a a weight=0 update and SIGKILL to the service to allow existing connections to finish processing, and then once the service exits, the backend is removed.

Needs some more awareness of docker container events/lifecycles in github.com/qmsk/clusterf/docker to provide both Running and Stopping flags in ContainerEvent.

clusterf-docker[3091]: 2015/12/08 13:17:21 containerEvent kill:1d26f748ddcda964f2d8bf0aaafe073325d79f1a74bf0d6797fd37018d520609: teardown
clusterf-docker[3091]: 2015/12/08 13:17:21 teardownContainer 1d26f748ddcda964f2d8bf0aaafe073325d79f1a74bf0d6797fd37018d520609: retract #{elasticsearch 1d26f748ddcda964f2d8bf0aaafe073325d79f1a74bf0d6797fd37018d520609 {10.5.107.4  9200 0 0}}
clusterf-docker[3091]: 2015/12/08 13:17:30 containerEvent kill:1d26f748ddcda964f2d8bf0aaafe073325d79f1a74bf0d6797fd37018d520609: skip
clusterf-docker[3091]: 2015/12/08 13:17:31 containerEvent die:1d26f748ddcda964f2d8bf0aaafe073325d79f1a74bf0d6797fd37018d520609: skip
clusterf-docker[3091]: 2015/12/08 13:17:31 containerEvent stop:1d26f748ddcda964f2d8bf0aaafe073325d79f1a74bf0d6797fd37018d520609: skip

[SpComb]

Implemented in the new clusterf-docker model:

2016/05/09 16:43:55 docker:Networks.update connect network 37334d3c1c5ca7239f9fafce6937912913342997efceaaa5089b8196d1f7d95a -> {false}: remove
2016/05/09 16:43:55 Update config...
2016/05/09 16:43:55 docker:Client.updateState: mount volume 45f2687de092cf7b84b438995e291660b83845dd6794a570f800ca511568d054: skip
2016/05/09 16:43:55 Update config...
2016/05/09 16:43:56 docker:Containers.update start container e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b -> running: down
2016/05/09 16:43:56 Update config...
2016/05/09 16:43:56 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:43:56 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:43:56 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:43:57 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: new services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:05 docker:Containers.update kill container e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b -> stopping: up
2016/05/09 16:44:05 Update config...
2016/05/09 16:44:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: set services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:05 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: set services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:06 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: set services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:06 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: set services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:15 docker:Containers.update kill container e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b -> stopping: up
2016/05/09 16:44:15 Update config...
2016/05/09 16:44:15 docker:Containers.update die container e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b -> stopping: up
2016/05/09 16:44:15 Update config...
2016/05/09 16:44:15 docker:Networks.update disconnect network 37334d3c1c5ca7239f9fafce6937912913342997efceaaa5089b8196d1f7d95a -> {false}: remove
2016/05/09 16:44:15 docker:Client.updateState: unmount volume 45f2687de092cf7b84b438995e291660b83845dd6794a570f800ca511568d054: skip
2016/05/09 16:44:16 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: remove services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:16 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: remove services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:16 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: remove services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:16 config:EtcdSource etcd+http://localhost:2379/clusterf-dev: writer: remove services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
2016/05/09 16:44:16 Update config...
2016/05/09 16:44:16 Update config...
2016/05/09 16:44:16 docker:Containers.update stop container e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b -> stopped: up
2016/05/09 16:44:16 Update config...

With etcd getting updated appropriately:

$ etcdctl watch --recursive --forever /clusterf-dev
[set] /clusterf-dev/services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","tcp":8083,"weight":10}
[set] /clusterf-dev/services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","tcp":8086,"weight":10}
[set] /clusterf-dev/services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","tcp":2003,"weight":10}
[set] /clusterf-dev/services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","udp":25826,"weight":10}




[set] /clusterf-dev/services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","udp":25826,"weight":0}
[set] /clusterf-dev/services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","tcp":8083,"weight":0}
[set] /clusterf-dev/services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","tcp":8086,"weight":0}
[set] /clusterf-dev/services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
{"ipv4":"10.6.107.2","tcp":2003,"weight":0}



[delete] /clusterf-dev/services/influxdb/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
PrevNode.Value: {"ipv4":"10.6.107.2","tcp":8086,"weight":0}
[delete] /clusterf-dev/services/influxdb-graphite/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
PrevNode.Value: {"ipv4":"10.6.107.2","tcp":2003,"weight":0}
[delete] /clusterf-dev/services/influxdb-collectd/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
PrevNode.Value: {"ipv4":"10.6.107.2","udp":25826,"weight":0}
[delete] /clusterf-dev/services/influxdb-admin/backends/e722f282c0120f3eda3aebc7ae72722266b92290fdc91a6c93e4d60b1881e40b
PrevNode.Value: {"ipv4":"10.6.107.2","tcp":8083,"weight":0}