Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cached metric isn't reexposed and thus disappear from Prometheus #273

Open
cenon-digitaleo opened this issue Jun 10, 2024 · 0 comments
Open

Comments

@cenon-digitaleo
Copy link

Describe the change

Make a cached metric available at all times even when not reevaluated.

Current situation

With this configuration:

  • Metric is configured to be cached for 30mins
  • Prometheus scrape_interval is 5mins
  • Aggregation returns no document
  • overrideEmpty and emptyValue are configured accordingly to set value to 0

I only have 5mins of data in a 30min time window. It seems in this case that the metric is only exposed when cache expires and so when the aggregation is run again. I'm pretty sure these 5mins correspond to Prometheus query.loopback-delta configuration which means I only have one data point every 30mins and that Prometheus can only find it for 5mins.

Worth mentioning that if the aggregation do returns document I don't encounter the issue, it only happens when the override mechanism is triggered.

Should

Something is not working correctly in the scenario detailed previously and I believe it comes from the aggregate function where the metric creation differs when we don't have any document in the aggregation result. Metric should be available at all times even in the "overrideEmpty" case.

By the way it seems the DefaultCache is not acting as expected:

if c.config.DefaultCache > 0 && aggregation.Cache != 0 {
    aggregation.Cache = c.config.DefaultCache
}

In this code I would expect that if you explicitly specify a different cache value from the default one on an aggregation it should not be overridden by the default one. Then, it should be == and not != so that an aggregation with no cache configured gets the cache value from the default one if it is configured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant