Loki: Per-tenant stream sharding #7311

DylanGuedes · 2022-09-30T18:16:46Z

What this PR does / why we need it:

Move stream sharding configuration to its own package to avoid cyclic imports
Change stream sharding to be a per-tenant configuration
Change ingesters to reject whole streams due to rate-limit based on per-tenant stream sharding
Change stream sharding flags prefix from distributor.shard-stream to shard-stream

Special notes for your reviewer:
This is part of the stream sharding work

grafanabot · 2022-10-01T14:32:02Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.1%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

MasslessParticle

Looks good, overall. We've got to fix a bug and improve the tests

MasslessParticle · 2022-10-03T17:03:46Z

pkg/ingester/stream.go

@@ -346,7 +348,7 @@ func (s *stream) validateEntries(entries []logproto.Entry, isReplay bool) ([]log
 		}

 		now := time.Now()
-		if !s.cfg.RateLimitWholeStream && !s.limiter.AllowN(now, len(entries[i].Line)) {
+		if rateLimitWholeStream && !s.limiter.AllowN(now, len(entries[i].Line)) {


This ought to be !rateLimitWholeStream.

I see why our test missed this: because whole-stream rate limiting happens at the end, the result looks the same. It's important that we not call AllowN when rateLimitWholeStream==true because a successful call changes the limiter's state and affects the rate limit. It there a way we can tell how the limiter was called in the test to help us in the future?

great catch, thank you. I'll investigate how to do that.

Pushed a fix and a test covering it, wdyt?

MasslessParticle · 2022-10-03T17:08:31Z

pkg/ingester/stream_test.go

 	require.Contains(t, err.Error(), (&validation.ErrStreamRateLimit{RateLimit: l.PerStreamRateLimit, Labels: s.labelsString, Bytes: flagext.ByteSize(len(entries[1].Line))}).Error())
 }

 func TestPushRateLimitAllOrNothing(t *testing.T) {
 	l := validation.Limits{
 		PerStreamRateLimit:      10,
 		PerStreamRateLimitBurst: 10,
+		ShardStreams: &shardstreams.Config{


I don't think we need this setup because the limits for this are used in instance.go to determine rateLimitWholeStream

We should probably have a test in instance.go to make sure this stays wired together the same way in the future.

Good call. I added a test covering it, WDYT?

pkg/distributor/distributor_test.go

grafanabot · 2022-10-03T18:51:56Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.1%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-04T13:09:41Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
-        distributor	-0.1%
+            querier	0%
- querier/queryrange	-0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
-               loki	-0.1%

MasslessParticle · 2022-10-05T16:03:18Z

pkg/ingester/stream_test.go

@@ -312,6 +313,9 @@ func TestPushRateLimit(t *testing.T) {
 	l := validation.Limits{
 		PerStreamRateLimit:      10,
 		PerStreamRateLimitBurst: 10,
+		ShardStreams: &shardstreams.Config{


nit: Because whole stream rate limiting is controlled by the flag we pass to Push, there's no need to set this up here, either.

ops, good catch. Fixed!

MasslessParticle

Left a nit but lgtm

…-per-tenant

grafanabot · 2022-10-05T16:46:11Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
-        distributor	-0.1%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
-               loki	-0.1%

…-per-tenant

grafanabot · 2022-10-05T18:15:33Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.1%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-05T18:53:45Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.1%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-05T19:42:23Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.1%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

**What this PR does / why we need it**: - Move stream sharding configuration to its own package to avoid cyclic imports - Change stream sharding to be a per-tenant configuration - Change ingesters to reject whole streams due to rate-limit based on per-tenant stream sharding - Change stream sharding flags prefix from `distributor.shard-stream` to `shard-stream`

pull-request-size bot added the size/L label Sep 30, 2022

DylanGuedes force-pushed the make-stream-shard-per-tenant branch 8 times, most recently from c6d3b1e to fb6e575 Compare October 1, 2022 13:49

Reuse stream-sharding as a parameter to ignore pushes.

f191f5e

DylanGuedes force-pushed the make-stream-shard-per-tenant branch from fb6e575 to f191f5e Compare October 1, 2022 14:27

DylanGuedes changed the title ~~Loki: Reuse stream-sharding as a parameter to ignore pushes.~~ Loki: Per-tenant stream sharding Oct 3, 2022

DylanGuedes marked this pull request as ready for review October 3, 2022 13:43

DylanGuedes requested a review from a team as a code owner October 3, 2022 13:43

MasslessParticle requested changes Oct 3, 2022

View reviewed changes

DylanGuedes added 3 commits October 3, 2022 15:25

Add JSON tags to variables.

6b71d0a

Remove commented code.

9c22291

Make linter happy.

51916e7

DylanGuedes added 5 commits October 3, 2022 16:26

Add streamsharding configuration test.

f821f1a

Push test scenario that checks bursts.

e6d08d4

Fix rateLimitWholeStream flag bug.

027388c

Make linter happy.

1db06af

Fix import order.

d288665

DylanGuedes mentioned this pull request Oct 5, 2022

Loki: Add sharding support for negative/zeroed desired rate #7342

Merged

MasslessParticle reviewed Oct 5, 2022

View reviewed changes

MasslessParticle approved these changes Oct 5, 2022

View reviewed changes

Remove unnecessary ShardStreamConfig assignment.

bf363a5

Merge branch 'main' of github.com:grafana/loki into make-stream-shard…

5ba4d76

…-per-tenant

Merge branch 'main' of github.com:grafana/loki into make-stream-shard…

7e82c48

…-per-tenant

Update flag and docs.

ed3b884

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Oct 5, 2022

DylanGuedes merged commit b1d4efa into grafana:main Oct 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Loki: Per-tenant stream sharding #7311

Loki: Per-tenant stream sharding #7311

DylanGuedes commented Sep 30, 2022 •

edited

Loading

grafanabot commented Oct 1, 2022

MasslessParticle left a comment

MasslessParticle Oct 3, 2022

DylanGuedes Oct 3, 2022

DylanGuedes Oct 3, 2022

MasslessParticle Oct 3, 2022

DylanGuedes Oct 3, 2022

grafanabot commented Oct 3, 2022

grafanabot commented Oct 4, 2022

MasslessParticle Oct 5, 2022

DylanGuedes Oct 5, 2022

MasslessParticle left a comment

grafanabot commented Oct 5, 2022

grafanabot commented Oct 5, 2022

grafanabot commented Oct 5, 2022

grafanabot commented Oct 5, 2022

Loki: Per-tenant stream sharding #7311

Loki: Per-tenant stream sharding #7311

Conversation

DylanGuedes commented Sep 30, 2022 • edited Loading

grafanabot commented Oct 1, 2022

MasslessParticle left a comment

Choose a reason for hiding this comment

MasslessParticle Oct 3, 2022

Choose a reason for hiding this comment

DylanGuedes Oct 3, 2022

Choose a reason for hiding this comment

DylanGuedes Oct 3, 2022

Choose a reason for hiding this comment

MasslessParticle Oct 3, 2022

Choose a reason for hiding this comment

DylanGuedes Oct 3, 2022

Choose a reason for hiding this comment

grafanabot commented Oct 3, 2022

grafanabot commented Oct 4, 2022

MasslessParticle Oct 5, 2022

Choose a reason for hiding this comment

DylanGuedes Oct 5, 2022

Choose a reason for hiding this comment

MasslessParticle left a comment

Choose a reason for hiding this comment

grafanabot commented Oct 5, 2022

grafanabot commented Oct 5, 2022

grafanabot commented Oct 5, 2022

grafanabot commented Oct 5, 2022

DylanGuedes commented Sep 30, 2022 •

edited

Loading