Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ExternalLabels() for Prometheus v3.0 #7893

Merged

Conversation

simonpasquier
Copy link
Contributor

@simonpasquier simonpasquier commented Nov 8, 2024

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Prometheus v3.0.0-rc.0 introduces a new scrape protocol (PrometheusText1.0.0) which is present by default in the global configuration. It breaks the Thanos sidecar when it wants to retrieve the external labels.

This change replaces the use of the Prometheus GlobalConfig struct by a minimal struct which unmarshals only the external_labels key.

See also prometheus-operator/prometheus-operator#7078

Verification

Unit test added.

Prometheus v3.0.0-rc.0 introduces a new scrape protocol
(`PrometheusText1.0.0`) which is present by default in the global
configuration. It breaks the Thanos sidecar when it wants to retrieve
the external labels.

This change replaces the use of the Prometheus `GlobalConfig` struct by
a minimal struct which unmarshals only the `external_labels` key.

See also prometheus-operator/prometheus-operator#7078

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
@simonpasquier
Copy link
Contributor Author

cc @saswatamcode

Copy link
Member

@saswatamcode saswatamcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for fixing!

I wonder, if bumping prometheus version in our e2e tests also makes sense (probably good to do once v3 is fully released)?

@simonpasquier
Copy link
Contributor Author

I wonder, if bumping prometheus version in our e2e tests also makes sense (probably good to do once v3 is fully released)?

What we've done for Prometheus operator (and this is how we found out the issue) is to configure a periodic workflow running all our e2e tests with the latest v3.0.0 release (currently rc.0). It gives an early signal on potentially breaking changes.

@saswatamcode
Copy link
Member

Yup, I think we might need similar! In any case, merging this fix

@saswatamcode saswatamcode merged commit bfbabbb into thanos-io:main Nov 8, 2024
22 checks passed
@simonpasquier simonpasquier deleted the fix-external-labels-with-prom3 branch November 8, 2024 10:12
@gebn
Copy link

gebn commented Nov 16, 2024

Heads up - Prometheus v3.0.0 is now out, so folks using the latest Thanos release v0.36.1 will start to see Sidecar breaking.

A workaround is to remove the new PrometheusText1.0.0 from the default list of scrape protocols:

global:
  ...
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants