Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timestamp to exported metrics #29

Open
ahus1 opened this issue May 18, 2017 · 11 comments
Open

Add timestamp to exported metrics #29

ahus1 opened this issue May 18, 2017 · 11 comments

Comments

@ahus1
Copy link

ahus1 commented May 18, 2017

The graphite exporter keeps the metrics for up to 5 minutes. It even records the time in Timestamp. But it doesn't export the time it received the metrics to Prometheus.

I would like the graphite_exporter to export the time when it received a metrics like the push gateway.

This would help a setup where Prometheus scrapes the graphite_exporter every 10 seconds, but new data is pushed to graphite in a different timing (i.e. 30 or 60 seconds).

@brian-brazil
Copy link
Contributor

This has intricate dependencies elsewhere.

I would like the graphite_exporter to export the time when it received a metrics like the push gateway.

The push_gateway does not do this.

@ahus1
Copy link
Author

ahus1 commented May 18, 2017

Hello @brian-brazil - thank you for the fast reply. Apparently I've been mistaken about the push_gateway.

But isn't the timestamp intended to be used for something like this? Can you elaborate on the intricate dependencies?

@ahus1
Copy link
Author

ahus1 commented May 18, 2017

... I've read the "about timestamps" for the push gateway.

The only way to do it without a timestamp would be to align the the pushing to graphite with the scraping of prometheus, and to set the expiry time in the graphite_exporter to i.e. 2-3x the push/pull interval?!

@matthiasr
Copy link
Contributor

I think adding timestamps is not wrong in this case – we are bridging from graphite's understanding of time to prometheus'. This is different from the Pushgateway use case. We just need to be clear that you can't use this to backfill like you can in graphite.

Could you please be more specific about the "intricate dependencies"?

@brian-brazil
Copy link
Contributor

That was a reference to staleness handling. Lack of timestamp support in client_golang would be the main blocker now.

@matthiasr
Copy link
Contributor

Another thought – with timestamps, what use is the staleness delta? I mean, it is still nice to remove metrics from memory eventually but would it make a difference logically, if the last sample just stays at the last timestamp?

@brian-brazil
Copy link
Contributor

brian-brazil commented Jan 18, 2018

I assume you meant the sample expiry. You still want to put a bound on memory growth, and having data that's too old won't work well on the Prometheus end.

@ahus1
Copy link
Author

ahus1 commented Jan 18, 2018

@matthiasr - I liked the fact that the data of this exporter removed data eventually from memory. This makes it absolutely maintenance-free. Otherwise getting rid of obsolete metrics would require a manual purge or restart.

Regarding timestamps - yes, I would still like that.

@matthiasr
Copy link
Contributor

matthiasr commented Jan 18, 2018 via email

@naveenyallane
Copy link

We are facing similar issue with Azure exporter (https://github.com/RobustPerception/azure_metrics_exporter).
We observed 5 minutes gap from the metric that is exported to prometheus and actual metric in the Azure monitor.

@brian-brazil
Copy link
Contributor

Please keep on topic, this issue is about the Graphite exporter. The relevant support has been in client_golang for a while.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants