Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(kafka): Enable querier to optionally query partition ingesters #14418

Merged
merged 7 commits into from
Oct 9, 2024
Merged

feat(kafka): Enable querier to optionally query partition ingesters #14418

merged 7 commits into from
Oct 9, 2024

Conversation

benclive
Copy link
Contributor

@benclive benclive commented Oct 8, 2024

What this PR does / why we need it:

Implements logic to enable queriers to query the right ingesters based on active partitions.

  • When the flag is active, old ingesters will no longer be queried at all.
  • Includes shuffle sharding support
  • Includes request minimization to only query 1 ingester instance per partition instead of all/both of them.

Which issue(s) this PR fixes:
Fixes https://github.com/grafana/loki-private/issues/1132

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@benclive benclive requested a review from a team as a code owner October 8, 2024 10:16
benclive
benclive commented Oct 8, 2024
View reviewed changes
pkg/querier/ingester_querier.go Outdated
return nil, err
}
tenantShards := q.getShardCountForTenant(tenantID)
// TODO: Confirm the shuffle-shard lookback period == query-ingesters-within
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find an answer to this: is the lookback period the same as the query-ingesters-within parameter or something else?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is query-ingesters-within can we get confirmation from @pracucci ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shuffle sharding lookback period in Mimir is configured here:
https://github.com/grafana/mimir/blob/ab3e627b1da134f33c06dd09ee4402a2eaad68f1/pkg/mimir/modules.go#L448-L450

It's set equal to TSDB retention, but that's a recent change. In the past it was set to the query-ingesters-within parameter. The reason why we changed to TSDB retention is because now we let to customize the query-ingesters-within parameter on a per-tenant basis (via the live-reloaded runtime config) and so it's not the same setting for the entire cluster. The worst case scenario we want to cover is the TSDB retention period in ingesters (which is also set to 13h).

I think for your use case setting it to query-ingesters-within parameter is fine.

@benclive benclive force-pushed the enable-partition-ingester-query-path branch from 52ce46e to 67b98b0 Compare October 8, 2024 10:46
grobinson-grafana
grobinson-grafana reviewed Oct 8, 2024
View reviewed changes
pkg/loki/modules.go Outdated Show resolved Hide resolved
pkg/querier/ingester_querier.go Show resolved Hide resolved
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Oct 8, 2024
cyriltovena
cyriltovena approved these changes Oct 9, 2024
View reviewed changes
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

There's just a lint problem :)

@benclive benclive merged commit 633bb5e into main Oct 9, 2024
61 checks passed
@benclive benclive deleted the enable-partition-ingester-query-path branch October 9, 2024 10:04
trevorwhitney pushed a commit that referenced this pull request Oct 10, 2024
@benclive @trevorwhitney
feat(kafka): Enable querier to optionally query partition ingesters (#…
0ee464f 
…14418)
@loki-gh-app loki-gh-app bot mentioned this pull request Nov 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@grobinson-grafana grobinson-grafana grobinson-grafana left review comments

@pracucci pracucci pracucci left review comments

@cyriltovena cyriltovena cyriltovena approved these changes

Assignees
No one assigned
Labels
size/L type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@benclive @cyriltovena @pracucci @grobinson-grafana