Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add initial support for a kafka-based ingest path #13992

Merged
merged 12 commits into from
Aug 30, 2024

Conversation

benclive
Copy link
Contributor

What this PR does / why we need it:
Adds write path using Kafka. There are two components: the partition ring which is updated by the ingesters and read by the kafka tee, and the kafka tee itself which is instantiated and managed in the distributor.

  • Kafka supports use of the Partition ring to assign partitions based on the number of active ingesters
    • I've attached the ring to the rf1 ingesters for now - we can easily move it to the regular ingester if preferred
  • Two new configs: One for the partition ring params, one for enabling this kafka ingest path. We can add more kafka params here.

I put the code under two packages in the ingester-rf1 package to keep it separate, but definitely happy to reorganise!
Most of the config setup is copied from Mimir, including injecting the config at runtime into the ingester config so it can start up successfully.

image

  • Ring management page shows the active status and metrics report success.
  • I had to add an small override to enable this to run locally when the hostname does not match the ingester name regex - not sure what the best option is here!

@benclive benclive requested a review from a team as a code owner August 28, 2024 16:45
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Aug 29, 2024
@benclive benclive force-pushed the benclive/rf1-kafka-tee branch from f8ddda0 to 9332b5f Compare August 29, 2024 13:57
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cyriltovena cyriltovena merged commit 33c26f4 into main Aug 30, 2024
61 checks passed
@cyriltovena cyriltovena deleted the benclive/rf1-kafka-tee branch August 30, 2024 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XXL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants