Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tsdb/index sampling #6329

Merged
merged 7 commits into from
Jun 7, 2022
Merged

Tsdb/index sampling #6329

merged 7 commits into from
Jun 7, 2022

Conversation

owen-d
Copy link
Member

@owen-d owen-d commented Jun 7, 2022

This PR lays the groundwork for index sampling in TSDB. To do so, it creates a new struct,

type Stats struct {
	Streams uint64
	Chunks  uint64
	Bytes   uint64
	Entries uint64
}

A set of (user, from, through, matchers) will resolve to the approximation of underlying streams, chunks, bytes, and entries.

In the short term, this will enable dynamic sharding in the query planner. In the long term, we can expose this as an endpoint and use it to power a host of new features such as cardinality analysis and informing users how much data a query will touch before running it.

There will be a followup PR which wires this into the index-gateway grpc layer.

ref #5428

@owen-d owen-d requested a review from a team as a code owner June 7, 2022 16:36
@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
-               loki	-0.7%

Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants