Reduce chunk write queue memory usage 2

[pstibrany]

This PR reimplements chan chunkWriteJob with custom buffered queue that should use less memory, because it doesn't preallocate entire buffer for maximum queue size at once. Instead it allocates individual "segments" with smaller size.

As elements are added to the queue, they fill individual segments. When elements are removed from the queue (and segments), empty segments can be thrown away. This doesn't change memory usage of the queue when it's full, but decreases its memory footprint when queue is empty (queue will keep max 1 segment in such case).

This PR comes from grafana/mimir-prometheus#247. We use it in Grafana Mimir to reduce memory usage in Mimir clusters with thousands of open TSDBs in single process.

Related to #10873.

[pstibrany]

See #10873 (comment) for first part of the story.

After applying fix from #10873 and then fix from this PR at ~14:00 to ingester-zone-b, we've got:

Legend:

• zone-a (green) using new chunk mapper without fix Reduce chunk write queue memory usage 1 #10873 and Reduce chunk write queue memory usage 2 #10874
• zone-b (yellow) is using new chunk mapper with fix Reduce chunk write queue memory usage 1 #10873 and at about 14:00 got updated with fix from this PR as well
• zone-c using old chunk mapper from before Write chunks via queue, predicting the refs #10051

We can see that applying both fixes (#10873 and from this PR) reduced memory usage of new chunk mapper to the same values as old chunk mapper.

(In this case, each zone had 20 TSDBs open altogether)

[pstibrany]

Here is a graph with much larger Mimir deployment with 5200 open TSDBs in each zone, after enabling new chunk mapper with both fixes in ingester-zone-a, while zones B and C still use old chunk mapper. Graph starts with rollout in zone-a, and shows cumulative memory usage for each zone (in GB):

[bwplotka]

This looks good, thanks!

However, I don't see the mentioned improvement in the second graph: #10874 (comment) - do you mean those small steps down when the load was potentially smaller?

Anyway, it's great to reduce the allocation when the queue is smaller, so LGTM. Thanks 💪🏽

Just FYI the only negative I see is that generally linked lists can trash L-caches when iterating. And we iterate jobs to process them. However, given the write chunk job requires disk write so higher I/O - linked list overhead probably does not matter. Did you maybe check the impact of this on latency for writing those chunks (queue feeling up quicker)? I wonder if we have even a means to measure this currently. 🤔

[pstibrany]

This looks good, thanks!

Thank you for review!

However, I don't see the mentioned improvement in the second graph: #10874 (comment) - do you mean those small steps down when the load was potentially smaller?

The improvement is in the yellow line. At first you can see it being better (lower) than green (zone-a), but worse than blue (zone-c). After applying fix from this PR (roughtly at 14 UTC on the graph), it got to the same level as blue (zone-c, running old chunk mapper).

Anyway, it's great to reduce the allocation when the queue is smaller, so LGTM. Thanks 💪🏽

Just FYI the only negative I see is that generally linked lists can trash L-caches when iterating. And we iterate jobs to process them. However, given the write chunk job requires disk write so higher I/O - linked list overhead probably does not matter. Did you maybe check the impact of this on latency for writing those chunks (queue feeling up quicker)? I wonder if we have even a means to measure this currently. 🤔

We iterate full segment first (slice of 8192 jobs), before jumping (using linked list) to the next segment. As you point out each chunk job performs IO. I haven't checked the impact on latency of writing the chunks though.

[bwplotka]

Restarted windows test flake

[bwplotka]

Happy to merge, unless you want to check it @codesome

[codesome]

1 less PR to review for me? Yes please!