[issue] Unable to collect nomad client metrics

[jtuthehien]

Nomad version

Output from nomad version: Nomad v0.4.1
docker version
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64

Operating system and Environment details

Unbuntu14

Issue

Nomad doesnt fire docker stats
Received: [WARN] client: error fetching stats of task redis: stats collection hasn't started yet

docker stats shows normally

Reproduction steps

Set up nomad client + server in 1 host
nomad agent -config /etc/nomad.d/

Config files:
common.hcl:
'
data_dir = "/var/lib/nomad"
bind_addr = "0.0.0.0"
log_level = "DEBUG"
enable_debug = true
enable_syslog = true

advertise {

We need to specify our host's IP because we can't

advertise 0.0.0.0 to other nodes in our cluster.

rpc = "10.128.0.2:4647"
serf = "10.128.0.2:4648"
}

telemetry {
statsd_address = "127.0.0.1:8125"
publish_allocation_metrics = true
publish_node_metrics = true
}
'
server.hcl
'server {
enabled = true
bootstrap_expect = 1
}'

client.hcl
'client {
enabled = true
servers = ["10.128.0.2:4647"]
stats {
data_points = 35
collection_interval = "5s"
}
}
'

Nomad logs (if appropriate)

2016/09/04 23:26:34.358980 [DEBUG] worker: dequeued evaluation b98ef677-9f64-b2b1-247f-b90926bfba17
2016/09/04 23:26:34.359308 [DEBUG] sched: <Eval 'b98ef677-9f64-b2b1-247f-b90926bfba17' JobID: 'redis'>: allocs: (place 1) (update 0) (migrate 0) (stop 0) (ignore 0) (lost 0)
2016/09/04 23:26:34.360041 [DEBUG] http: Request /v1/jobs?region=global (12.470889ms)
2016/09/04 23:26:34.361171 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17?region=global (118.093µs)
2016/09/04 23:26:34.362151 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17/allocations?region=global (126.885µs)
2016/09/04 23:26:34.365018 [DEBUG] worker: submitted plan for evaluation b98ef677-9f64-b2b1-247f-b90926bfba17
2016/09/04 23:26:34.365143 [DEBUG] sched: <Eval 'b98ef677-9f64-b2b1-247f-b90926bfba17' JobID: 'redis'>: setting status to complete
2016/09/04 23:26:34.365540 [DEBUG] client: updated allocations at index 8 (pulled 1) (filtered 0)
2016/09/04 23:26:34.365700 [DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 0)
2016/09/04 23:26:34.366944 [DEBUG] client: starting task runners for alloc '7cc14753-c9a5-f388-cd03-dc032950920e'
2016/09/04 23:26:34.367153 [DEBUG] client: starting task context for 'redis' (alloc '7cc14753-c9a5-f388-cd03-dc032950920e')
2016/09/04 23:26:34.369982 [DEBUG] worker: updated evaluation <Eval 'b98ef677-9f64-b2b1-247f-b90926bfba17' JobID: 'redis'>
2016/09/04 23:26:34.370149 [DEBUG] worker: ack for evaluation b98ef677-9f64-b2b1-247f-b90926bfba17
2016/09/04 23:26:34.479407 [DEBUG] client: updated allocations at index 10 (pulled 0) (filtered 1)
2016/09/04 23:26:34.479583 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2016/09/04 23:26:35.363452 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17?region=global (194.916µs)
2016/09/04 23:26:35.364572 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17/allocations?region=global (223.447µs)
2016/09/04 23:26:44.801067 [DEBUG] driver.docker: docker pull redis:latest succeeded
2016/09/04 23:26:44.804048 [DEBUG] driver.docker: identified image redis:latest as 6fc557a78dedfa06327896fdf69bb49ca512779336f540da80c7e78b2a9ded9b
2016/09/04 23:26:44 [DEBUG] plugin: starting plugin: /usr/bin/nomad []string{"/usr/bin/nomad", "executor", "/var/lib/nomad/alloc/7cc14753-c9a5-f388-cd03-dc032950920e/redis/redis-executor.out"}
2016/09/04 23:26:44 [DEBUG] plugin: waiting for RPC address for: /usr/bin/nomad
2016/09/04 23:26:44 [DEBUG] plugin: nomad: 2016/09/04 23:26:44 [DEBUG] plugin: plugin address: unix /tmp/plugin155931495
2016/09/04 23:26:44.817534 [DEBUG] driver.docker: using 268435456 bytes memory for redis:latest
2016/09/04 23:26:44.817559 [DEBUG] driver.docker: using 500 cpu shares for redis:latest
2016/09/04 23:26:44.817574 [DEBUG] driver.docker: binding directories []string{"/var/lib/nomad/alloc/7cc14753-c9a5-f388-cd03-dc032950920e/alloc:/alloc", "/var/lib/nomad/alloc/7cc14753-c9a5-f388-cd03-dc032950920e/redis:/local"} for redis:latest
2016/09/04 23:26:44.817586 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
2016/09/04 23:26:44.817606 [DEBUG] driver.docker: allocated port 10.128.0.2:31234 -> 6379 (mapped)
2016/09/04 23:26:44.817618 [DEBUG] driver.docker: exposed port 6379
2016/09/04 23:26:44.817669 [DEBUG] driver.docker: setting container name to: redis-7cc14753-c9a5-f388-cd03-dc032950920e
2016/09/04 23:26:44.845135 [INFO] driver.docker: created container 435f99308b421969d43261db06ae7c9edda6f9c4cca3325c9abeaf7b79021a3f
2016/09/04 23:26:45.037656 [INFO] driver.docker: started container 435f99308b421969d43261db06ae7c9edda6f9c4cca3325c9abeaf7b79021a3f
2016/09/04 23:26:45.040070 [WARN] client: error fetching stats of task redis: stats collection hasn't started yet
2016/09/04 23:26:45.281671 [DEBUG] client: updated allocations at index 11 (pulled 0) (filtered 1)
2016/09/04 23:26:45.281988 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)

Job file (if appropriate)

Standard example.nomad generated with nomad init

[diptanu]

@jtuthehien Do you see the stats if you run the nomad alloc-status <alloc-id> and nomad node-status <node-id> command?

There is a bug in 0.4.1 which prevents the stats to be pushed to statsd, that's been fixed and is going to go out with 0.5

[jtuthehien]

Hi @diptanu : yes.
nomad node-status -self
ID = d2855bb4
Name = nomad-client1
Class =
DC = dc1
Drain = false
Status = ready
Uptime = 34h13m27s

Allocated Resources
CPU Memory Disk IOPS
1000/2300 MHz 512 MiB/3.6 GiB 600 MiB/16 GiB 0/0

Allocation Resource Utilization
CPU Memory
3/2300 MHz 13 MiB/3.6 GiB

Host Resource Utilization
CPU Memory Disk
23/2300 MHz 451 MiB/3.6 GiB 2.3 GiB/20 GiB

Allocations
ID Eval ID Job ID Task Group Desired Status Client Status
bb0b64a4 6b4a20ed example cache run running
7cc14753 b98ef677 redis cache run running

nomad alloc-status bb0b64a4
ID = bb0b64a4
Eval ID = 6b4a20ed
Name = example.cache[0]
Node ID = d2855bb4
Job ID = example
Client Status = running

Task "redis" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
1/500 MHz 6.3 MiB/256 MiB 300 MiB 0 db: 10.128.0.2:44208

By the way, is nomad compatible with Docker version > 1.9 ( like 1.11)

[diptanu]

@jtuthehien Thanks for the information, so yeah publishing would work when we release 0.5. Fixed via #1670 I am closing this issue, since the fix has landed on master.

And yes, Nomad should be compatible with the current version of Docker. Please open an issue if it's not.

[vynjo]

@jtuthehien We had a similar issue, does work on the latest 0.5 build, but because of another issue we built an 0.4.1 binary with the appropriate patch. It's here if you want to use...

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.