Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix infinit suffix add to counter metrics #3491

Closed
wants to merge 1 commit into from

Conversation

ehsannm
Copy link

@ehsannm ehsannm commented Nov 23, 2022

No description provided.

@ehsannm
Copy link
Author

ehsannm commented Nov 23, 2022

There is critical bug in Prometheus exporter, counters are growing indefinitely by adding _total to the end of the metric.

Copy link
Contributor

@MrAlias MrAlias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will move our implementation out of compliance with the specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#sums

Monotonic Sum metric points MUST have _total added as a suffix to the metric name.

If someone already named their metric with a suffix of _total an additional _total suffix must be added still.

@ehsannm
Copy link
Author

ehsannm commented Nov 24, 2022

Thanks @MrAlias , but where should we check this infinit loop. The problem is with each new variableLabel change we get another _total added to the counter name.

HELP qlub_pos_requests_ratio_total

TYPE qlub_pos_requests_ratio_total counter

qlub_pos_requests_ratio_total{code="200",country_code="ae",method="get_order",vendor="dummy"} 6

HELP qlub_pos_requests_ratio_total_total

TYPE qlub_pos_requests_ratio_total_total counter

qlub_pos_requests_ratio_total_total{code="200",country_code="ae",method="get_table",vendor="dummy"} 1

@ehsannm
Copy link
Author

ehsannm commented Nov 24, 2022

ok i realized this bug is in current release tag v0.33.0 and is fixed in main branch. I close this PR. It should be fixed in next release.

@ehsannm ehsannm closed this Nov 24, 2022
@ehsannm
Copy link
Author

ehsannm commented Nov 24, 2022

BTW, i think it is critical bug, why not release a hotfix v0.33.1 for it ? @MrAlias

@jmacd
Copy link
Contributor

jmacd commented Dec 2, 2022

For the next reader, the fix went in here: #3369

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants