doc: Add shuffle sharding doc #6173

kavirajk · 2022-05-17T08:27:19Z

Signed-off-by: Kaviraj kavirajkanagaraj@gmail.com

What this PR does / why we need it:
Add shuffle sharding doc for Loki.

What is it?
Why we need it on Loki?
Key Configs
Key metrics

And some operational advice.

Which issue(s) this PR fixes:
Fixes #NA

Special notes for your reviewer:
Markdown preview: https://github.com/grafana/loki/blob/0dcc6034ea927b16ff9a79eb71100b98a736da78/docs/sources/operations/shuffle-sharding.md

Checklist

Documentation added
Is this an important fix or new feature? Add an entry in the CHANGELOG.md.

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

grafanabot · 2022-05-17T08:32:16Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

grafanabot · 2022-05-17T10:07:34Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

docs/sources/operations/shuffle-sharding.md

sandeepsukhani

overall it looks great on technical explanation. There are some suggestions by Dan and I will leave it to Karen for suggesting any corrections in writing.

sandeepsukhani · 2022-05-26T09:27:36Z

docs/sources/operations/shuffle-sharding.md

+The maximum number of queriers can be overridden on a per-tenant basis in the limits overrides configuration (`max_queriers_per_tenant`).
+
+### The impact of “query of death”
+In the event a tenant is repeatedly sending a “query of death” which leads the querier to crash or getting killed because of out-of-memory, the crashed querier will get disconnected from the query-frontend and a new querier will be immediately assigned to the tenant’s shard. This practically invalidates the assumption that shuffle-sharding can be used to contain the blast radius in case of a query of death.


You can add a line explaining a bit more why that is a problem by mentioning that the tenant would get assigned to other running queries to match the max_queriers_per_tenant and might eventually affect other tenant shards which beats the purpose of this feature.

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

grafanabot · 2022-05-26T13:38:39Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
-               loki	-0.6%

grafanabot · 2022-05-26T13:38:48Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-26T13:40:45Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-05-26T13:40:45Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

grafanabot · 2022-06-02T08:04:45Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-06-02T08:04:59Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-06-02T08:14:02Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

sandeepsukhani

LGTM

kavirajk · 2022-06-02T08:47:03Z

@KMiller-Grafana can you take a look at the doc when you get a chance?

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

jeschkies

Thanks for the write up. It help me understand shuffle sharding better 🙂

KMiller-Grafana

One small change please!

docs/sources/operations/shuffle-sharding.md

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>

grafanabot · 2022-07-21T17:45:49Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

KMiller-Grafana

Thanks so much.

grafanabot · 2022-07-21T20:22:08Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

* doc: Add shuffle sharding doc Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add query-scheduler config as well Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Add line explaining what is "query-of-death" exactly means * PR remarsk from Karen Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix image URL Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> (cherry picked from commit a1e75da)

* doc: Add shuffle sharding doc Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add query-scheduler config as well Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Add line explaining what is "query-of-death" exactly means * PR remarsk from Karen Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix image URL Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> (cherry picked from commit a1e75da) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>

* doc: Add shuffle sharding doc Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add query-scheduler config as well Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> * Add line explaining what is "query-of-death" exactly means * PR remarsk from Karen Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix image URL Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update docs/sources/operations/shuffle-sharding.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>

doc: Add shuffle sharding doc

0dcc603

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

kavirajk requested a review from KMiller-Grafana as a code owner May 17, 2022 08:27

pull-request-size bot added the size/M label May 17, 2022

github-actions bot added the area/docs label May 17, 2022

kavirajk mentioned this pull request May 17, 2022

query-scheduler: Now proper support for querier_forget_delay #6174

Merged

4 tasks

Add query-scheduler config as well

b0197f1

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

kavirajk requested a review from trevorwhitney May 17, 2022 15:35

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

danmdinu reviewed May 26, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

sandeepsukhani reviewed May 26, 2022

View reviewed changes

kavirajk and others added 2 commits May 26, 2022 15:33

Update docs/sources/operations/shuffle-sharding.md

c122214

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Update docs/sources/operations/shuffle-sharding.md

0417b2a

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

github-actions bot added the area/docs label May 26, 2022

kavirajk and others added 4 commits May 26, 2022 15:33

Update docs/sources/operations/shuffle-sharding.md

864e626

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Update docs/sources/operations/shuffle-sharding.md

1f0ab5c

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Update docs/sources/operations/shuffle-sharding.md

d9a85cc

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Update docs/sources/operations/shuffle-sharding.md

f595d6f

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Update docs/sources/operations/shuffle-sharding.md

7afb75d

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Update docs/sources/operations/shuffle-sharding.md

8142171

Co-authored-by: Dan Mihai Dinu <33793329+danmdinu@users.noreply.github.com>

Add line explaining what is "query-of-death" exactly means

7791792

sandeepsukhani approved these changes Jun 2, 2022

View reviewed changes

chaudum added area/docs and removed area/docs labels Jun 13, 2022

KMiller-Grafana added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jul 20, 2022

PR remarsk from Karen

d516df9

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

pull-request-size bot added size/L and removed size/M labels Jul 21, 2022

Fix image URL

f19703b

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

jeschkies approved these changes Jul 21, 2022

View reviewed changes

KMiller-Grafana suggested changes Jul 21, 2022

View reviewed changes

docs/sources/operations/shuffle-sharding.md Outdated Show resolved Hide resolved

Update docs/sources/operations/shuffle-sharding.md

652838a

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>

KMiller-Grafana approved these changes Jul 21, 2022

View reviewed changes

KMiller-Grafana added the backport release-2.6.x Tag a PR with this label to create a PR which cherry pics it into the release-2.6.x branch label Jul 21, 2022

Merge branch 'main' into kavirajk/doc-shuffle-sharding

24f0d3b

kavirajk merged commit a1e75da into main Jul 21, 2022

kavirajk deleted the kavirajk/doc-shuffle-sharding branch July 21, 2022 21:05

grafanabot mentioned this pull request Jul 21, 2022

[release-2.6.x] doc: Add shuffle sharding doc #6742

Merged

osg-grafana removed the area/docs label Oct 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: Add shuffle sharding doc #6173

doc: Add shuffle sharding doc #6173

kavirajk commented May 17, 2022 •

edited

Loading

grafanabot commented May 17, 2022

grafanabot commented May 17, 2022

sandeepsukhani left a comment

sandeepsukhani May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented Jun 2, 2022

grafanabot commented Jun 2, 2022

grafanabot commented Jun 2, 2022

sandeepsukhani left a comment

kavirajk commented Jun 2, 2022

jeschkies left a comment

KMiller-Grafana left a comment

grafanabot commented Jul 21, 2022

KMiller-Grafana left a comment

grafanabot commented Jul 21, 2022

doc: Add shuffle sharding doc #6173

doc: Add shuffle sharding doc #6173

Conversation

kavirajk commented May 17, 2022 • edited Loading

grafanabot commented May 17, 2022

grafanabot commented May 17, 2022

sandeepsukhani left a comment

Choose a reason for hiding this comment

sandeepsukhani May 26, 2022

Choose a reason for hiding this comment

grafanabot commented May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented May 26, 2022

grafanabot commented Jun 2, 2022

grafanabot commented Jun 2, 2022

grafanabot commented Jun 2, 2022

sandeepsukhani left a comment

Choose a reason for hiding this comment

kavirajk commented Jun 2, 2022

jeschkies left a comment

Choose a reason for hiding this comment

KMiller-Grafana left a comment

Choose a reason for hiding this comment

grafanabot commented Jul 21, 2022

KMiller-Grafana left a comment

Choose a reason for hiding this comment

grafanabot commented Jul 21, 2022

kavirajk commented May 17, 2022 •

edited

Loading