-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(operator): Allow structured metadata only if V13 schema provided #13463
fix(operator): Allow structured metadata only if V13 schema provided #13463
Conversation
I don't think this is the correct fix for us, because we want the users to be able to use structured metadata at some point. The validation code seems to only check the "currently active schema", which means that this validation still triggers even when the "future schemas" contain a valid configuration. I think how the content of the error is to be interpreted is:
I think for us the correct handling is the second case. We already produce a warning that the user should update to v13, so they know that we want them to use For me using |
8d00c5d
to
87c38a2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I'm confused... I don’t get why we set both the loki-config.yaml
+ CLI Arg when we want to allow structured metadata enabled 🤔
I'm similarly confused. I was more thinking of something like this when I wrote the comment yesterday. |
@JoaoBraveCoding @xperimental I agree with both of you that disabling the validation is enough, but since we allow only valid schemas via the LokiStack CRD, I though we should set |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still very confused... isn't allow_structured_metadata
on the config and the CLI arg -validation.allow-structured-metadata=false
the same thing? So currently:
v11 || v12
we don't set a CLI but we setallow_structured_metadata: false
v13
we set both the CLI-validation.allow-structured-metadata=false
&&allow_structured_metadata: true
which seems contradictory.
Also by looking at the distributor code, it seems allow_structured_metadata: true
is required to ingest structured metadata no? Otherwise, we will always be landing here
Do we have steps/instructions for sending structured metadata for Loki to test this PR?
Nope because the validation code is doing this over the "active schemas". we can define v13 into the future and still want to have allow to true and validation to false, else the pods crash. |
Tested this with a client actually trying to send structured metadata and, in the end, it turns out that I did not understand what For me it looks as if we need to duplicate the validation logic that Loki does during startup in the operator to set the correct @JoaoBraveCoding suggested that there is probably an interval after which the operator will re-reconcile the LokiStack again (I remember this being a thing called "ResyncPeriod" in the informers). If this is the case then we probably would not need to add extra logic to the operator to requeue the LokiStack after a timeout, because it might happen often enough for our "at most daily" configuration change. |
I was trying to dig this out and found that maybe it makes more sense to use |
Seemed to work fine in our "overnight test", so 👍 |
…rafana#13463) Co-authored-by: Robert Jacob <rojacob@redhat.com>
[release-5.9] Backport PR grafana#13463
What this PR does / why we need it:
With #13422 the Loki configuration defaults to
allow_structured_metadata: true
including runtime validation to have at least one V13 schema defined. However, existing LokiStack resource might not include a V13 schema yet running into a crashloop situation.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Checklist
CONTRIBUTING.md
guide (required)feat
PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.md
production/helm/loki/Chart.yaml
and updateproduction/helm/loki/CHANGELOG.md
andproduction/helm/loki/README.md
. Example PRdeprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR