Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segment files #3261

Merged
merged 9 commits into from
Oct 2, 2020
Merged

Segment files #3261

merged 9 commits into from
Oct 2, 2020

Conversation

pstibrany
Copy link
Contributor

@pstibrany pstibrany commented Sep 30, 2020

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

This PR adds segment_files field into meta.json file (Thanos section), which lists all segment files for a block. This helps to avoid call to storage to get this list. This small optimization helps to reduce amount of LIST operations in clusters with many created blocks.

Verification

  • Added unit test for shipper, added check in compactor test
  • Verified that new code path in store/bucket.go is hit by existing tests.

Filled by components that write meta.json (shipper, compactor,
repair, ...)

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Don't panic if segment index is incorrect.
Added comment to Iter about sorted results.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
…lling block.GetSegmentFiles creates an import cycle.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dear Peter.

Amazing. LGTM. 💪

pkg/block/block.go Outdated Show resolved Hide resolved
pkg/block/metadata/meta.go Outdated Show resolved Hide resolved
pkg/store/bucket.go Outdated Show resolved Hide resolved
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
@pstibrany
Copy link
Contributor Author

Dear Bartek,

thanks for your review. I've updated the PR with your suggestions. Please take a look at your convenience.

With love,
-Peter

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I double checked if the new property is set in every place and I think so. This is apparently a small optimisation, but according to our calculations it would have a big impact to reduce bucket listing operations. Thanks @pstibrany !

@pracucci pracucci merged commit 358e526 into thanos-io:master Oct 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants