-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
does nomad must honor collection_interval when collect telemetry by allocations? #6961
Comments
Hi @tantra35! It looks like client stats are collected at the specified interval but task resource usage stats collection is a tight loop around the collection here: stats_hook.go#L87-L112. Based on the more detailed treatment of the configuration in the telemetry metrics docs I'm not sure whether or not that's intentional and just a documentation gap. |
My colleague @notnoop has pointed out to me that we do pass the interval to the task driver's |
@tgross on test stand for simplicity purposes we use |
@tgross thanks for clarify as code works. After some debugging i found that value of interval passed to
so problem is in |
Thanks @tantra35! We'll look into this! |
@tgross ah sorry, just found and this is our misconfiguration in provisioning, so bug doesn't realy present |
@tgross after some investigations i mast reopen this issue, due problem actualy exist at least in |
No problem. 😀 We'll take a look. |
Repro from our testing for 0.10.4's upcoming release candidate. Run an exec job and find its allocs:
ssh into the node ▶ diff e2e/terraform/shared/nomad/base.hcl base.hcl
14c14,15
< collection_interval = "1s"
---
> collection_interval = "60s"
> statsd_address = "localhost:8125" Create a statsd config: {
graphitePort: 2003
, port: 8125
, backends: [ "./backends/console" ]
, dumpMessages: true
} Run a statsd collector on the node, using that configuration file:
Restart the Nomad client, wait a while, and then check that statsd is getting logs at the expected rate. We'll receive debug messages every 60 -- these are Nomad's collection intervals. The lines between (every 10 seconds) are statsd's own flushing of all the metrics it knows to console.
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.10.2 (40edb4d+CHANGES)
Is it normal that nomad doesn't honor
collection_interval
(https://www.nomadproject.io/docs/configuration/telemetry.html#collection_interval) when collecting statistic by allocationsFor example on nomad client we have flow telemetry configuration:
And i think that telemetry for allocation will be collected every 60 seconds, but this is not true, and statistics for allocation collected every 1 second, at
localhost:8125
we have custom agregated statsd proxy and can intercept all metrics, so from logs from this daemon we have follow frequency on metrics:for example metric
nomad.client.allocs.memory.kernel_max_usage.vault_debug-00.test.5f9fb00c-93da-2a3c-80ba-21322f989a6a.vault_debug_task.default
collected evry 1 secondThe text was updated successfully, but these errors were encountered: