-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[issue] Unable to collect nomad client metrics #1684
Comments
@jtuthehien Do you see the stats if you run the There is a bug in 0.4.1 which prevents the stats to be pushed to statsd, that's been fixed and is going to go out with 0.5 |
Hi @diptanu : yes. Allocated Resources Allocation Resource Utilization Host Resource Utilization Allocations nomad alloc-status bb0b64a4 Task "redis" is "running" By the way, is nomad compatible with Docker version > 1.9 ( like 1.11) |
@jtuthehien Thanks for the information, so yeah publishing would work when we release 0.5. Fixed via #1670 I am closing this issue, since the fix has landed on master. And yes, Nomad should be compatible with the current version of Docker. Please open an issue if it's not. |
@jtuthehien We had a similar issue, does work on the latest 0.5 build, but because of another issue we built an 0.4.1 binary with the appropriate patch. It's here if you want to use... |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Output from
nomad version
: Nomad v0.4.1docker version
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64
Operating system and Environment details
Unbuntu14
Issue
Nomad doesnt fire docker stats
Received: [WARN] client: error fetching stats of task redis: stats collection hasn't started yet
docker stats shows normally
Reproduction steps
Set up nomad client + server in 1 host
nomad agent -config /etc/nomad.d/
Config files:
common.hcl:
'
data_dir = "/var/lib/nomad"
bind_addr = "0.0.0.0"
log_level = "DEBUG"
enable_debug = true
enable_syslog = true
advertise {
We need to specify our host's IP because we can't
advertise 0.0.0.0 to other nodes in our cluster.
rpc = "10.128.0.2:4647"
serf = "10.128.0.2:4648"
}
telemetry {
statsd_address = "127.0.0.1:8125"
publish_allocation_metrics = true
publish_node_metrics = true
}
'
server.hcl
'server {
enabled = true
bootstrap_expect = 1
}'
client.hcl
'client {
enabled = true
servers = ["10.128.0.2:4647"]
stats {
data_points = 35
collection_interval = "5s"
}
}
'
Nomad logs (if appropriate)
2016/09/04 23:26:34.358980 [DEBUG] worker: dequeued evaluation b98ef677-9f64-b2b1-247f-b90926bfba17
2016/09/04 23:26:34.359308 [DEBUG] sched: <Eval 'b98ef677-9f64-b2b1-247f-b90926bfba17' JobID: 'redis'>: allocs: (place 1) (update 0) (migrate 0) (stop 0) (ignore 0) (lost 0)
2016/09/04 23:26:34.360041 [DEBUG] http: Request /v1/jobs?region=global (12.470889ms)
2016/09/04 23:26:34.361171 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17?region=global (118.093µs)
2016/09/04 23:26:34.362151 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17/allocations?region=global (126.885µs)
2016/09/04 23:26:34.365018 [DEBUG] worker: submitted plan for evaluation b98ef677-9f64-b2b1-247f-b90926bfba17
2016/09/04 23:26:34.365143 [DEBUG] sched: <Eval 'b98ef677-9f64-b2b1-247f-b90926bfba17' JobID: 'redis'>: setting status to complete
2016/09/04 23:26:34.365540 [DEBUG] client: updated allocations at index 8 (pulled 1) (filtered 0)
2016/09/04 23:26:34.365700 [DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 0)
2016/09/04 23:26:34.366944 [DEBUG] client: starting task runners for alloc '7cc14753-c9a5-f388-cd03-dc032950920e'
2016/09/04 23:26:34.367153 [DEBUG] client: starting task context for 'redis' (alloc '7cc14753-c9a5-f388-cd03-dc032950920e')
2016/09/04 23:26:34.369982 [DEBUG] worker: updated evaluation <Eval 'b98ef677-9f64-b2b1-247f-b90926bfba17' JobID: 'redis'>
2016/09/04 23:26:34.370149 [DEBUG] worker: ack for evaluation b98ef677-9f64-b2b1-247f-b90926bfba17
2016/09/04 23:26:34.479407 [DEBUG] client: updated allocations at index 10 (pulled 0) (filtered 1)
2016/09/04 23:26:34.479583 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
2016/09/04 23:26:35.363452 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17?region=global (194.916µs)
2016/09/04 23:26:35.364572 [DEBUG] http: Request /v1/evaluation/b98ef677-9f64-b2b1-247f-b90926bfba17/allocations?region=global (223.447µs)
2016/09/04 23:26:44.801067 [DEBUG] driver.docker: docker pull redis:latest succeeded
2016/09/04 23:26:44.804048 [DEBUG] driver.docker: identified image redis:latest as 6fc557a78dedfa06327896fdf69bb49ca512779336f540da80c7e78b2a9ded9b
2016/09/04 23:26:44 [DEBUG] plugin: starting plugin: /usr/bin/nomad []string{"/usr/bin/nomad", "executor", "/var/lib/nomad/alloc/7cc14753-c9a5-f388-cd03-dc032950920e/redis/redis-executor.out"}
2016/09/04 23:26:44 [DEBUG] plugin: waiting for RPC address for: /usr/bin/nomad
2016/09/04 23:26:44 [DEBUG] plugin: nomad: 2016/09/04 23:26:44 [DEBUG] plugin: plugin address: unix /tmp/plugin155931495
2016/09/04 23:26:44.817534 [DEBUG] driver.docker: using 268435456 bytes memory for redis:latest
2016/09/04 23:26:44.817559 [DEBUG] driver.docker: using 500 cpu shares for redis:latest
2016/09/04 23:26:44.817574 [DEBUG] driver.docker: binding directories []string{"/var/lib/nomad/alloc/7cc14753-c9a5-f388-cd03-dc032950920e/alloc:/alloc", "/var/lib/nomad/alloc/7cc14753-c9a5-f388-cd03-dc032950920e/redis:/local"} for redis:latest
2016/09/04 23:26:44.817586 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
2016/09/04 23:26:44.817606 [DEBUG] driver.docker: allocated port 10.128.0.2:31234 -> 6379 (mapped)
2016/09/04 23:26:44.817618 [DEBUG] driver.docker: exposed port 6379
2016/09/04 23:26:44.817669 [DEBUG] driver.docker: setting container name to: redis-7cc14753-c9a5-f388-cd03-dc032950920e
2016/09/04 23:26:44.845135 [INFO] driver.docker: created container 435f99308b421969d43261db06ae7c9edda6f9c4cca3325c9abeaf7b79021a3f
2016/09/04 23:26:45.037656 [INFO] driver.docker: started container 435f99308b421969d43261db06ae7c9edda6f9c4cca3325c9abeaf7b79021a3f
2016/09/04 23:26:45.040070 [WARN] client: error fetching stats of task redis: stats collection hasn't started yet
2016/09/04 23:26:45.281671 [DEBUG] client: updated allocations at index 11 (pulled 0) (filtered 1)
2016/09/04 23:26:45.281988 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)
Job file (if appropriate)
Standard example.nomad generated with nomad init
The text was updated successfully, but these errors were encountered: