Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(blooms): Apply retention in planner #13484

Merged
merged 4 commits into from
Jul 11, 2024

Conversation

salvacorts
Copy link
Contributor

@salvacorts salvacorts commented Jul 11, 2024

What this PR does / why we need it:
This PR implements retention in the bloom planner.

Notes to reviewers

The retention manager is copied from the bloom compactor, the only difference is that we have removed the logic to check if the compactor owns the retention by looking at the ring.

Therefore I'd focus on reviewing:

  • Docs
  • Config.go
  • Metrics.go
  • Planner.go
  • Skim over retention.go and retention_test.go

Tested out in dev by setting the retention_period: 10d for tenant 145265.

Before retention: Both tenants have blocks up to the earliest day built.
Screenshot 2024-07-11 at 11 53 34

After retention: Tenant 29 still has data for older days but 145265 don't.
Screenshot 2024-07-11 at 12 07 18

Here are the retention logs:

2024-07-11 12:06:09.624	level=info ts=2024-07-11T10:06:09.546039726Z caller=retention.go:179 component=bloom-planner subcomponent=retention-manager msg="finished applying retention" daysProcessed=6 tenants=1
2024-07-11 12:06:02.124	level=info ts=2024-07-11T10:06:02.073912794Z caller=retention.go:156 component=bloom-planner subcomponent=retention-manager day=2024-06-25 tenant=145265 msg="applying retention to tenant" keys=259
2024-07-11 12:05:54.525	level=info ts=2024-07-11T10:05:54.455099907Z caller=retention.go:156 component=bloom-planner subcomponent=retention-manager day=2024-06-26 tenant=145265 msg="applying retention to tenant" keys=259
2024-07-11 12:05:46.724	level=info ts=2024-07-11T10:05:46.6384181Z caller=retention.go:156 component=bloom-planner subcomponent=retention-manager day=2024-06-27 tenant=145265 msg="applying retention to tenant" keys=259
2024-07-11 12:05:38.624	level=info ts=2024-07-11T10:05:38.618366406Z caller=retention.go:156 component=bloom-planner subcomponent=retention-manager day=2024-06-28 tenant=145265 msg="applying retention to tenant" keys=259
2024-07-11 12:05:30.424	level=info ts=2024-07-11T10:05:30.420296936Z caller=retention.go:156 component=bloom-planner subcomponent=retention-manager day=2024-06-29 tenant=145265 msg="applying retention to tenant" keys=259
2024-07-11 12:05:21.824	level=info ts=2024-07-11T10:05:21.749845286Z caller=retention.go:156 component=bloom-planner subcomponent=retention-manager day=2024-06-30 tenant=145265 msg="applying retention to tenant" keys=257
2024-07-11 12:05:21.325	level=info ts=2024-07-11T10:05:21.270021915Z caller=retention.go:92 component=bloom-planner subcomponent=retention-manager msg="Applying retention" today=2024-07-11 lastDayRun=1970-01-01

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@salvacorts salvacorts force-pushed the salvacorts/bloom-refactor/retention branch from b63aeb7 to 3d1889c Compare July 11, 2024 08:53
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jul 11, 2024
@salvacorts salvacorts marked this pull request as ready for review July 11, 2024 10:21
@salvacorts salvacorts requested a review from a team as a code owner July 11, 2024 10:21

# Max lookback days for retention.
# CLI flag: -bloom-build.planner.retention.max-lookback-days
[max_lookback_days: <int> | default = 365]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we hide this setting from the docs? Usually you don't need to set this, only in very special edge cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it can cause confusion and could be mixed up with retention period of blooms, which it isn't right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have an strong opinion on this. I'll go ahead and hide it.

@salvacorts salvacorts merged commit 2cc901a into main Jul 11, 2024
61 checks passed
@salvacorts salvacorts deleted the salvacorts/bloom-refactor/retention branch July 11, 2024 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XXL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants