does nomad must honor collection_interval when collect telemetry by allocations? #6961
Comments
tantra35
changed the title
|
Hi @tantra35! It looks like client stats are collected at the specified interval but task resource usage stats collection is a tight loop around the collection here: stats_hook.go#L87-L112. Based on the more detailed treatment of the configuration in the telemetry metrics docs I'm not sure whether or not that's intentional and just a documentation gap. |
|
My colleague @notnoop has pointed out to me that we do pass the interval to the task driver's |
tgross
added
type/bug
stage/needs-investigation
and removed
stage/needs-discussion
labels
|
@tgross on test stand for simplicity purposes we use |
|
@tgross thanks for clarify as code works. After some debugging i found that value of interval passed to so problem is in |
|
Thanks @tantra35! We'll look into this! |
|
@tgross ah sorry, just found and this is our misconfiguration in provisioning, so bug doesn't realy present |
|
@tgross after some investigations i mast reopen this issue, due problem actualy exist at least in |
|
No problem. 😀 We'll take a look. |
|
Repro from our testing for 0.10.4's upcoming release candidate. Run an exec job and find its allocs: ssh into the node ▶ diff e2e/terraform/shared/nomad/base.hcl base.hcl
14c14,15
< collection_interval = "1s"
---
> collection_interval = "60s"
> statsd_address = "localhost:8125"Create a statsd config: {
graphitePort: 2003
, port: 8125
, backends: [ "./backends/console" ]
, dumpMessages: true
}Run a statsd collector on the node, using that configuration file: Restart the Nomad client, wait a while, and then check that statsd is getting logs at the expected rate. We'll receive debug messages every 60 -- these are Nomad's collection intervals. The lines between (every 10 seconds) are statsd's own flushing of all the metrics it knows to console. |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.10.2 (40edb4d+CHANGES)
Is it normal that nomad doesn't honor
collection_interval(https://www.nomadproject.io/docs/configuration/telemetry.html#collection_interval) when collecting statistic by allocationsFor example on nomad client we have flow telemetry configuration:
And i think that telemetry for allocation will be collected every 60 seconds, but this is not true, and statistics for allocation collected every 1 second, at
localhost:8125we have custom agregated statsd proxy and can intercept all metrics, so from logs from this daemon we have follow frequency on metrics:for example metric
nomad.client.allocs.memory.kernel_max_usage.vault_debug-00.test.5f9fb00c-93da-2a3c-80ba-21322f989a6a.vault_debug_task.defaultcollected evry 1 secondThe text was updated successfully, but these errors were encountered: