[RFC] Feature/ingester stream backpressure

[splitice]

What this PR does / why we need it:

This PR add backpressure to the response to ingester Query(). A similar aproach could also be used for QuerySample()

This is done through the addition of an Ack(id) method and a limit to the max in-flight responses in the stream (currently 20)

This is built upon my branch for #6216

Which issue(s) this PR fixes:
I suspect would fix #5804

Special notes for your reviewer:

This needs a cleanup and represents a RFC on the concept which it will get if positively reviewed.

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0.4%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
-               iter	-0.4%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0.4%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
-               iter	-0.4%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0.4%
+        distributor	0%
+            querier	0.1%
+ querier/queryrange	0%
-               iter	-0.8%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0.4%
+        distributor	0%
+            querier	0.1%
+ querier/queryrange	0%
-               iter	-0.8%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0.4%
+        distributor	0%
+            querier	0.1%
+ querier/queryrange	0%
-               iter	-0.8%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[DylanGuedes]

a few (not so important) things. Just for curiosity, what would the backpressure improve? and since you used the term RFC, do you have a doc or anything explaining the reasoning behind the idea?

[DylanGuedes]

(same error from line 573)

Suggested change

	if err != nil {
	return &logproto.AckResponse{}, err
	}

[DylanGuedes]

hmm nitpicking maybe you should call it SendAck() instead of ReleaseAck()? (since SendAck will send an empty struct message to the channel)

[splitice]

I dare say between #6241 and this that these are the two main causes of ingester memory peaks. Whether or not this is the direction loki team want to go is worthy of comment.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[splitice]

Go away

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[splitice]

Go away

[MasslessParticle]

Hey @splitice, thanks for your contribution!

We've added a new way to get visibility on issues like this. Please check out LIDs. I'm going to close this for now. If you'd still like to discuss it, please open a lid.