Stream chunks from ingesters to queriers (#4886)

* Initial PoC implementation of ingester logic.

* Rename field.

* Retain only iterators, rather than entire ChunkSeries.

* Add note about enforcing limits on the number of series returned by a single query.

* Add initial implementation of series.Series implementation that streams chunks from ingesters.

* Signal when initial streaming of series is complete.

* Sort imports.

* Fix broken test.

* Refactor Distributor.QueryStream() method signature to make it easier to introduce streaming.

* Fix typo in file name.

* Implement querier side of QueryStream RPC.

Still need to use returned SeriesStreamers to create
streamingChunkSeries and pass these through to the PromQL engine.

* Pass chunks to PromQL engine.

* Fix bugs.

TestReadWriteModeQueryingIngester now passes :tada:

* Add hack fix for issue where streams are cancelled before query evaluation has completed.

This however means that requests for unnecessary zones are retained
for longer.

* Deduplicate identical chunks received while streaming.

* Add note about possible goroutine leak.

* Remove unnecessary reference to series labels on SeriesStreamer.

* Partially fix issue where goroutine in SeriesStreamer.StartBuffering() could be leaked.

This fix only applies to requests from query-scheduler, and does not
apply to requests direct to the Prometheus HTTP query and query_range
API endpoints.

* Use correct field name in comment.

* Don't report context cancellation if there are series available in the buffer.

* Remove unused method.

* Add configuration flag to enable / disable streaming chunks from ingesters.

* Fix issue where remote read integration tests fail due to conflicting field types.

* Add CLI flag to experimental features list.

* Fix linting warnings.

* Make per-ingester series buffer size configurable and increase default value to 128.

* Hack: replace `container_cpu_usage_seconds_total` with `process_cpu_seconds_total` in dashboard panels

* Increase load generated.

* Pool slices of chunks when streaming chunks from ingesters.

* Disable gzip compression for requests to ingesters.

* Add benchmark and test for merging and sorting series.

* Rename function in preparation for alternative implementations

* Change method signatures to make other implementations easier.

* Add extra test case.

* Initial implementation of heap-based series set merging.

* Preallocate sources slice.

* Sort ingesters with no series remaining last, so that we can avoid heap.Pop calls.

* Correctly reset input data for benchmark :facepalm:

Benchmark results:

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/querier
                                                                                      │ original.txt  │              heap.txt               │
                                                                                      │    sec/op     │    sec/op     vs base               │
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          170.8n ± 17%   114.3n ±  4%  -33.10% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        2021.5n ±  0%   959.1n ±  2%  -52.55% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       29.883µ ±  0%   7.215µ ±  4%  -75.86% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      390.81µ ±  1%   72.05µ ±  3%  -81.56% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     4292.6µ ±  1%   807.3µ ±  3%  -81.19% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          227.8n ±  3%   155.2n ±  2%  -31.89% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         2.715µ ±  0%   1.336µ ±  6%  -50.78% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        38.07µ ±  0%   10.82µ ±  4%  -71.56% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       472.0µ ±  0%   110.0µ ±  2%  -76.69% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      5.177m ±  1%   1.168m ±  3%  -77.44% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          291.8n ±  1%   195.6n ±  2%  -32.96% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         3.391µ ±  2%   1.895µ ±  2%  -44.11% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        45.96µ ±  3%   16.69µ ±  2%  -63.69% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       578.7µ ±  5%   163.6µ ±  4%  -71.73% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      6.285m ±  5%   1.691m ±  3%  -73.09% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          293.9n ±  2%   235.5n ±  3%  -19.88% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         5.111µ ±  1%   2.146µ ±  3%  -58.01% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        66.90µ ±  0%   18.28µ ±  3%  -72.67% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       824.5µ ±  0%   177.2µ ±  2%  -78.50% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      9.031m ±  1%   2.207m ±  3%  -75.56% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          413.6n ±  5%   352.8n ±  3%  -14.70% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         7.003µ ±  3%   3.318µ ±  6%  -52.62% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        84.98µ ±  1%   31.11µ ±  3%  -63.39% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      1020.8µ ±  3%   301.4µ ±  4%  -70.47% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     11.257m ±  2%   3.385m ±  0%  -69.93% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          539.2n ±  2%   472.2n ±  0%  -12.42% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         8.416µ ±  1%   4.633µ ±  0%  -44.95% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       100.90µ ±  0%   44.26µ ±  0%  -56.14% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      1154.0µ ±  0%   431.0µ ±  0%  -62.65% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     12.835m ±  2%   4.663m ±  1%  -63.67% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          577.4n ±  1%   485.3n ±  0%  -15.95% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        11.559µ ±  2%   4.702µ ±  1%  -59.32% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       138.92µ ±  1%   46.40µ ±  0%  -66.60% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      1691.6µ ±  0%   481.0µ ±  1%  -71.56% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     19.775m ±  1%   5.412m ±  1%  -72.63% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          823.1n ±  2%   705.8n ±  1%  -14.25% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        14.915µ ±  1%   8.085µ ±  0%  -45.80% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       177.34µ ±  3%   84.14µ ±  2%  -52.56% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      2036.9µ ±  4%   851.3µ ± 27%  -58.21% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      24.12m ±  2%   10.29m ± 20%  -57.33% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          1.070µ ±  1%   1.190µ ±  6%  +11.22% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         18.03µ ±  1%   13.15µ ±  1%  -27.09% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        207.2µ ±  4%   140.6µ ±  0%  -32.13% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       2.396m ±  1%   1.417m ±  1%  -40.85% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      27.16m ±  2%   14.79m ±  1%  -45.53% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1_series_per_ingester-10         2.130µ ±  3%   1.461µ ±  0%  -31.41% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        30.32µ ±  2%   14.87µ ±  3%  -50.96% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       392.4µ ±  1%   161.8µ ±  0%  -58.77% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      4.314m ±  2%   1.575m ± 12%  -63.49% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     53.20m ±  1%   18.38m ±  2%  -65.45% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1_series_per_ingester-10         2.821µ ±  0%   2.420µ ±  0%  -14.20% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        38.05µ ±  3%   28.46µ ±  1%  -25.22% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       474.2µ ±  0%   317.6µ ±  2%  -33.03% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      5.210m ±  3%   2.954m ±  4%  -43.29% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     68.68m ±  3%   33.43m ±  0%  -51.33% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1_series_per_ingester-10         3.504µ ±  1%   3.672µ ±  0%   +4.81% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10_series_per_ingester-10        45.91µ ±  0%   42.41µ ±  2%   -7.64% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       551.5µ ±  0%   487.2µ ±  1%  -11.66% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      6.000m ±  1%   4.511m ±  1%  -24.82% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     77.14m ±  1%   50.72m ±  1%  -34.25% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1_series_per_ingester-10        30.24µ ±  0%   20.30µ ±  2%  -32.85% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10_series_per_ingester-10       392.7µ ±  3%   251.2µ ±  1%  -36.03% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_100_series_per_ingester-10      4.354m ±  0%   2.529m ±  2%  -41.93% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10     54.83m ±  2%   27.80m ±  5%  -49.31% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10    969.5m ±  4%   291.5m ±  1%  -69.94% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1_series_per_ingester-10        38.36µ ±  2%   39.75µ ±  0%   +3.61% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10_series_per_ingester-10       476.8µ ±  1%   523.0µ ±  1%   +9.70% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_100_series_per_ingester-10      5.258m ±  2%   4.925m ±  0%   -6.34% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10     69.28m ±  2%   53.59m ±  2%  -22.65% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10   1377.1m ±  5%   571.5m ±  4%  -58.50% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1_series_per_ingester-10        47.22µ ±  4%   61.78µ ±  1%  +30.84% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10_series_per_ingester-10       569.7µ ±  3%   804.0µ ±  2%  +41.13% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_100_series_per_ingester-10      6.073m ±  1%   7.634m ±  1%  +25.70% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10     82.84m ±  6%   82.56m ±  1%        ~ (p=0.485 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10   1723.3m ±  5%   881.3m ±  1%  -48.86% (p=0.002 n=6)
geomean                                                                                  256.8µ         132.0µ        -48.60%

                                                                                      │ original.txt  │               heap.txt                │
                                                                                      │     B/op      │     B/op      vs base                 │
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1_series_per_ingester-10            168.0 ± 0%     160.0 ± 0%   -4.76% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         2.565Ki ± 0%   2.266Ki ± 0%  -11.69% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        33.90Ki ± 0%   19.80Ki ± 0%  -41.60% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       446.8Ki ± 0%   230.1Ki ± 0%  -48.50% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      3.732Mi ± 0%   2.747Mi ± 0%  -26.39% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1_series_per_ingester-10            184.0 ± 0%     168.0 ± 0%   -8.70% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         2.722Ki ± 0%   2.273Ki ± 0%  -16.47% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        35.45Ki ± 0%   19.80Ki ± 0%  -44.14% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       462.5Ki ± 0%   230.1Ki ± 0%  -50.24% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      3.885Mi ± 0%   2.747Mi ± 0%  -29.29% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1_series_per_ingester-10            200.0 ± 0%     176.0 ± 0%  -12.00% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         2.878Ki ± 0%   2.281Ki ± 0%  -20.73% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        37.02Ki ± 0%   19.81Ki ± 0%  -46.49% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       478.1Ki ± 0%   230.1Ki ± 0%  -51.86% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      4.038Mi ± 0%   2.747Mi ± 0%  -31.96% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1_series_per_ingester-10            312.0 ± 0%     344.0 ± 0%  +10.26% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         6.505Ki ± 0%   4.555Ki ± 0%  -29.98% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        68.25Ki ± 0%   39.62Ki ± 0%  -41.95% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       892.9Ki ± 0%   404.2Ki ± 0%  -54.73% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      7.465Mi ± 0%   5.995Mi ± 0%  -19.70% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1_series_per_ingester-10            344.0 ± 0%     360.0 ± 0%   +4.65% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         6.817Ki ± 0%   4.570Ki ± 0%  -32.96% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        71.38Ki ± 0%   39.63Ki ± 0%  -44.48% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       924.1Ki ± 0%   404.3Ki ± 0%  -56.25% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      7.769Mi ± 0%   5.995Mi ± 0%  -22.84% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1_series_per_ingester-10            376.0 ± 0%     376.0 ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         7.130Ki ± 0%   4.586Ki ± 0%  -35.68% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        74.48Ki ± 0%   39.65Ki ± 0%  -46.77% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       955.3Ki ± 0%   404.3Ki ± 0%  -57.68% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      8.075Mi ± 0%   5.995Mi ± 0%  -25.76% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1_series_per_ingester-10            600.0 ± 0%     712.0 ± 0%  +18.67% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        14.677Ki ± 0%   9.133Ki ± 0%  -37.78% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       138.21Ki ± 0%   79.26Ki ± 0%  -42.66% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       1.737Mi ± 0%   1.094Mi ± 0%  -36.99% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      14.92Mi ± 0%   12.53Mi ± 0%  -16.03% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1_series_per_ingester-10            664.0 ± 0%     744.0 ± 0%  +12.05% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        15.301Ki ± 0%   9.164Ki ± 0%  -40.11% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       144.47Ki ± 0%   79.29Ki ± 0%  -45.12% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       1.798Mi ± 0%   1.094Mi ± 0%  -39.12% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      15.53Mi ± 0%   12.53Mi ± 0%  -19.32% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1_series_per_ingester-10            728.0 ± 0%     776.0 ± 0%   +6.59% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10_series_per_ingester-10        15.924Ki ± 0%   9.195Ki ± 0%  -42.25% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       150.72Ki ± 0%   79.32Ki ± 0%  -47.37% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       1.858Mi ± 0%   1.094Mi ± 0%  -41.12% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      16.14Mi ± 0%   12.53Mi ± 0%  -22.38% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1_series_per_ingester-10         2.565Ki ± 0%   2.336Ki ± 0%   -8.95% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        33.91Ki ± 0%   19.87Ki ± 0%  -41.41% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       447.1Ki ± 0%   230.2Ki ± 0%  -48.52% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      3.736Mi ± 0%   2.747Mi ± 0%  -26.47% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     33.48Mi ± 0%   32.16Mi ± 0%   -3.94% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1_series_per_ingester-10         2.722Ki ± 0%   2.414Ki ± 0%  -11.30% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        35.47Ki ± 0%   19.95Ki ± 0%  -43.77% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       462.8Ki ± 0%   230.3Ki ± 0%  -50.24% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      3.889Mi ± 0%   2.747Mi ± 0%  -29.36% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     35.02Mi ± 0%   32.16Mi ± 0%   -8.16% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1_series_per_ingester-10         2.878Ki ± 0%   2.492Ki ± 0%  -13.40% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10_series_per_ingester-10        37.04Ki ± 0%   20.02Ki ± 0%  -45.94% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       478.4Ki ± 0%   230.3Ki ± 0%  -51.86% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      4.041Mi ± 0%   2.748Mi ± 0%  -32.01% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     36.53Mi ± 0%   32.16Mi ± 0%  -11.97% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1_series_per_ingester-10        33.91Ki ± 0%   20.66Ki ± 0%  -39.06% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10_series_per_ingester-10       447.1Ki ± 0%   231.0Ki ± 0%  -48.34% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_100_series_per_ingester-10      3.736Mi ± 0%   2.748Mi ± 0%  -26.44% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10     33.49Mi ± 0%   32.16Mi ± 0%   -3.95% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10    445.8Mi ± 0%   315.1Mi ± 0%  -29.31% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1_series_per_ingester-10        35.47Ki ± 0%   21.54Ki ± 0%  -39.27% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10_series_per_ingester-10       462.8Ki ± 0%   231.9Ki ± 0%  -49.90% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_100_series_per_ingester-10      3.889Mi ± 0%   2.749Mi ± 0%  -29.30% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10     35.02Mi ± 0%   32.16Mi ± 0%   -8.15% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10    468.0Mi ± 0%   315.1Mi ± 0%  -32.66% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1_series_per_ingester-10        37.03Ki ± 0%   22.41Ki ± 0%  -39.47% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10_series_per_ingester-10       478.4Ki ± 0%   232.7Ki ± 0%  -51.36% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_100_series_per_ingester-10      4.041Mi ± 0%   2.750Mi ± 0%  -31.96% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10     36.55Mi ± 0%   32.16Mi ± 0%  -11.99% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10    490.0Mi ± 0%   315.1Mi ± 0%  -35.68% (p=0.002 n=6)
geomean                                                                                  209.9Ki        143.8Ki       -31.51%
¹ all samples are equal

                                                                                      │ original.txt │               heap.txt               │
                                                                                      │  allocs/op   │  allocs/op   vs base                 │
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1_series_per_ingester-10           4.000 ± 0%    4.000 ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10_series_per_ingester-10          23.00 ± 0%    17.00 ± 0%  -26.09% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_100_series_per_ingester-10         211.0 ± 0%    110.0 ± 0%  -47.87% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       2.035k ± 0%   1.014k ± 0%  -50.17% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      20.26k ± 0%   10.02k ± 0%  -50.53% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1_series_per_ingester-10           5.000 ± 0%    4.000 ± 0%  -20.00% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10_series_per_ingester-10          33.00 ± 0%    17.00 ± 0%  -48.48% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_100_series_per_ingester-10         311.0 ± 0%    110.0 ± 0%  -64.63% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       3.035k ± 0%   1.014k ± 0%  -66.59% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      30.26k ± 0%   10.02k ± 0%  -66.88% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1_series_per_ingester-10           6.000 ± 0%    4.000 ± 0%  -33.33% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10_series_per_ingester-10          43.00 ± 0%    17.00 ± 0%  -60.47% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_100_series_per_ingester-10         411.0 ± 0%    110.0 ± 0%  -73.24% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       4.035k ± 0%   1.014k ± 0%  -74.87% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      40.26k ± 0%   10.02k ± 0%  -75.11% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1_series_per_ingester-10           6.000 ± 0%    6.000 ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10_series_per_ingester-10          44.00 ± 0%    28.00 ± 0%  -36.36% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_100_series_per_ingester-10         416.0 ± 0%    211.0 ± 0%  -49.28% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       4.065k ± 0%   2.015k ± 0%  -50.43% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      40.55k ± 0%   20.02k ± 0%  -50.62% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1_series_per_ingester-10           8.000 ± 0%    6.000 ± 0%  -25.00% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10_series_per_ingester-10          64.00 ± 0%    28.00 ± 0%  -56.25% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_100_series_per_ingester-10         616.0 ± 0%    211.0 ± 0%  -65.75% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       6.065k ± 0%   2.015k ± 0%  -66.78% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      60.55k ± 0%   20.02k ± 0%  -66.93% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          10.000 ± 0%    6.000 ± 0%  -40.00% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10_series_per_ingester-10          84.00 ± 0%    28.00 ± 0%  -66.67% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_100_series_per_ingester-10         816.0 ± 0%    211.0 ± 0%  -74.14% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       8.065k ± 0%   2.015k ± 0%  -75.02% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      80.55k ± 0%   20.02k ± 0%  -75.14% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          10.000 ± 0%    9.000 ± 0%  -10.00% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10_series_per_ingester-10          86.00 ± 0%    49.00 ± 0%  -43.02% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_100_series_per_ingester-10         818.0 ± 0%    412.0 ± 0%  -49.63% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       8.126k ± 0%   4.018k ± 0%  -50.55% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      81.15k ± 0%   40.03k ± 0%  -50.67% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          14.000 ± 0%    9.000 ± 0%  -35.71% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         126.00 ± 0%    49.00 ± 0%  -61.11% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        1218.0 ± 0%    412.0 ± 0%  -66.17% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      12.126k ± 0%   4.018k ± 0%  -66.86% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     121.14k ± 0%   40.03k ± 0%  -66.96% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          18.000 ± 0%    9.000 ± 0%  -50.00% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         166.00 ± 0%    49.00 ± 0%  -70.48% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        1618.0 ± 0%    412.0 ± 0%  -74.54% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      16.126k ± 0%   4.018k ± 0%  -75.08% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     161.15k ± 0%   40.03k ± 0%  -75.16% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          23.00 ± 0%    17.00 ± 0%  -26.09% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         211.0 ± 0%    110.0 ± 0%  -47.87% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       2.035k ± 0%   1.014k ± 0%  -50.17% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      20.26k ± 0%   10.02k ± 0%  -50.55% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     204.0k ± 0%   100.0k ± 0%  -50.96% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          33.00 ± 0%    17.00 ± 0%  -48.48% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         311.0 ± 0%    110.0 ± 0%  -64.63% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       3.035k ± 0%   1.014k ± 0%  -66.59% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      30.26k ± 0%   10.02k ± 0%  -66.89% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     304.0k ± 0%   100.0k ± 0%  -67.10% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          43.00 ± 0%    17.00 ± 0%  -60.47% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         411.0 ± 0%    110.0 ± 0%  -73.24% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       4.035k ± 0%   1.014k ± 0%  -74.87% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      40.26k ± 0%   10.02k ± 0%  -75.11% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     404.0k ± 0%   100.0k ± 0%  -75.24% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1_series_per_ingester-10         211.0 ± 0%    110.0 ± 0%  -47.87% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10_series_per_ingester-10       2.035k ± 0%   1.014k ± 0%  -50.17% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_100_series_per_ingester-10      20.26k ± 0%   10.02k ± 0%  -50.55% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10     204.0k ± 0%   100.0k ± 0%  -50.96% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10    2.038M ± 0%   1.000M ± 0%  -50.94% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1_series_per_ingester-10         311.0 ± 0%    110.0 ± 0%  -64.63% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10_series_per_ingester-10       3.035k ± 0%   1.014k ± 0%  -66.59% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_100_series_per_ingester-10      30.26k ± 0%   10.02k ± 0%  -66.89% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10     304.0k ± 0%   100.0k ± 0%  -67.10% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10    3.038M ± 0%   1.000M ± 0%  -67.09% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1_series_per_ingester-10         411.0 ± 0%    110.0 ± 0%  -73.24% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10_series_per_ingester-10       4.035k ± 0%   1.014k ± 0%  -74.87% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_100_series_per_ingester-10      40.26k ± 0%   10.02k ± 0%  -75.11% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10     404.0k ± 0%   100.0k ± 0%  -75.24% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10    4.038M ± 0%   1.000M ± 0%  -75.24% (p=0.002 n=6)
geomean                                                                                  1.859k         746.8       -59.83%
¹ all samples are equal

* Reduce heap allocations by assuming we'll have at most one instance of each series per zone.

Benchmark results:

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/querier
                                                                                      │   heap.txt    │         heap-zone-count.txt         │
                                                                                      │    sec/op     │    sec/op     vs base               │
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          114.3n ±  4%   100.2n ±  9%  -12.30% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         959.1n ±  2%   883.0n ±  2%   -7.94% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        7.215µ ±  4%   6.272µ ±  0%  -13.08% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       72.05µ ±  3%   63.52µ ±  1%  -11.83% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      807.3µ ±  3%   701.6µ ±  1%  -13.10% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          155.2n ±  2%   147.6n ±  0%   -4.90% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         1.336µ ±  6%   1.288µ ±  0%   -3.63% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        10.82µ ±  4%   10.38µ ±  1%   -4.07% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       110.0µ ±  2%   104.7µ ±  1%   -4.86% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      1.168m ±  3%   1.107m ±  0%   -5.24% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          195.6n ±  2%   193.6n ±  0%        ~ (p=0.366 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         1.895µ ±  2%   1.893µ ±  0%        ~ (p=0.623 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        16.69µ ±  2%   16.70µ ±  0%        ~ (p=1.000 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       163.6µ ±  4%   166.2µ ±  1%        ~ (p=0.065 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      1.691m ±  3%   1.744m ± 13%   +3.09% (p=0.026 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          235.5n ±  3%   221.0n ±  1%   -6.16% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         2.146µ ±  3%   1.930µ ±  0%  -10.07% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        18.28µ ±  3%   17.32µ ±  0%   -5.24% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       177.2µ ±  2%   161.6µ ±  0%   -8.83% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      2.207m ±  3%   1.969m ±  2%  -10.80% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          352.8n ±  3%   314.6n ±  1%  -10.83% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         3.318µ ±  6%   3.220µ ±  1%   -2.95% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        31.11µ ±  3%   30.29µ ±  1%   -2.65% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       301.4µ ±  4%   291.4µ ±  1%   -3.31% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      3.385m ±  0%   3.273m ±  1%   -3.29% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          472.2n ±  0%   447.4n ±  1%   -5.26% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         4.633µ ±  0%   4.627µ ±  0%        ~ (p=0.093 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        44.26µ ±  0%   45.26µ ±  0%   +2.26% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       431.0µ ±  0%   434.6µ ±  1%   +0.84% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      4.663m ±  1%   4.711m ±  2%   +1.02% (p=0.015 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          485.3n ±  0%   475.2n ±  1%   -2.08% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         4.702µ ±  1%   4.646µ ±  3%        ~ (p=0.065 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        46.40µ ±  0%   46.63µ ±  4%        ~ (p=1.000 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       481.0µ ±  1%   502.5µ ±  7%        ~ (p=0.394 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      5.412m ±  1%   5.268m ±  5%        ~ (p=0.394 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          705.8n ±  1%   722.3n ±  0%   +2.34% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         8.085µ ±  0%   8.197µ ±  0%   +1.39% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        84.14µ ±  2%   85.44µ ±  0%   +1.55% (p=0.041 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       851.3µ ± 27%   859.6µ ±  2%        ~ (p=0.589 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     10.292m ± 20%   9.149m ±  9%  -11.11% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          1.190µ ±  6%   1.164µ ±  3%        ~ (p=0.240 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         13.15µ ±  1%   13.34µ ±  5%        ~ (p=0.394 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        140.6µ ±  0%   140.5µ ±  0%        ~ (p=1.000 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       1.417m ±  1%   1.404m ±  0%   -0.91% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      14.79m ±  1%   14.62m ±  0%   -1.15% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1_series_per_ingester-10         1.461µ ±  0%   1.468µ ±  0%   +0.48% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        14.87µ ±  3%   13.93µ ±  0%   -6.30% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       161.8µ ±  0%   158.6µ ±  0%   -1.98% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      1.575m ± 12%   1.527m ±  1%   -3.04% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     18.38m ±  2%   18.13m ±  1%   -1.38% (p=0.004 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1_series_per_ingester-10         2.420µ ±  0%   2.420µ ±  0%        ~ (p=0.781 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        28.46µ ±  1%   28.49µ ±  0%        ~ (p=0.394 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       317.6µ ±  2%   318.0µ ±  0%        ~ (p=0.485 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      2.954m ±  4%   2.993m ±  1%        ~ (p=0.394 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     33.43m ±  0%   33.48m ±  0%        ~ (p=0.180 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1_series_per_ingester-10         3.672µ ±  0%   3.706µ ±  0%   +0.91% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10_series_per_ingester-10        42.41µ ±  2%   42.07µ ±  0%        ~ (p=0.065 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       487.2µ ±  1%   496.4µ ±  0%   +1.88% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      4.511m ±  1%   4.610m ±  0%   +2.20% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     50.72m ±  1%   51.45m ±  0%   +1.44% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1_series_per_ingester-10        20.30µ ±  2%   20.88µ ±  0%   +2.84% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10_series_per_ingester-10       251.2µ ±  1%   254.4µ ±  0%   +1.25% (p=0.009 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_100_series_per_ingester-10      2.529m ±  2%   2.504m ±  0%   -0.98% (p=0.015 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10     27.80m ±  5%   27.74m ±  0%        ~ (p=1.000 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10    291.5m ±  1%   289.8m ±  1%   -0.56% (p=0.041 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1_series_per_ingester-10        39.75µ ±  0%   39.94µ ±  0%   +0.49% (p=0.009 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10_series_per_ingester-10       523.0µ ±  1%   516.4µ ±  0%   -1.25% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_100_series_per_ingester-10      4.925m ±  0%   4.988m ±  0%   +1.28% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10     53.59m ±  2%   53.65m ±  0%        ~ (p=0.818 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10    571.5m ±  4%   565.3m ±  0%   -1.09% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1_series_per_ingester-10        61.78µ ±  1%   62.70µ ±  0%   +1.49% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10_series_per_ingester-10       804.0µ ±  2%   811.1µ ±  0%        ~ (p=0.065 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_100_series_per_ingester-10      7.634m ±  1%   7.687m ±  0%   +0.70% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10     82.56m ±  1%   83.31m ±  0%   +0.91% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10    881.3m ±  1%   886.5m ±  0%   +0.59% (p=0.026 n=6)
geomean                                                                                  132.0µ         129.2µ         -2.09%

                                                                                      │   heap.txt    │          heap-zone-count.txt          │
                                                                                      │     B/op      │     B/op      vs base                 │
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1_series_per_ingester-10            160.0 ± 0%     104.0 ± 0%  -35.00% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         2.266Ki ± 0%   1.719Ki ± 0%  -24.14% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        19.80Ki ± 0%   14.33Ki ± 0%  -27.62% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       230.1Ki ± 0%   175.4Ki ± 0%  -23.77% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      2.747Mi ± 0%   2.213Mi ± 0%  -19.44% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1_series_per_ingester-10            168.0 ± 0%     136.0 ± 0%  -19.05% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         2.273Ki ± 0%   1.961Ki ± 0%  -13.75% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        19.80Ki ± 0%   16.68Ki ± 0%  -15.78% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       230.1Ki ± 0%   198.9Ki ± 0%  -13.58% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      2.747Mi ± 0%   2.442Mi ± 0%  -11.11% (p=0.002 n=6)
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1_series_per_ingester-10            176.0 ± 0%     176.0 ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         2.281Ki ± 0%   2.281Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        19.81Ki ± 0%   19.81Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       230.1Ki ± 0%   230.1Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      2.747Mi ± 0%   2.747Mi ± 0%        ~ (p=0.794 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1_series_per_ingester-10            344.0 ± 0%     232.0 ± 0%  -32.56% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         4.555Ki ± 0%   3.461Ki ± 0%  -24.01% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        39.62Ki ± 0%   28.68Ki ± 0%  -27.61% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10       404.2Ki ± 0%   294.9Ki ± 0%  -27.06% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      5.995Mi ± 0%   4.927Mi ± 0%  -17.82% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1_series_per_ingester-10            360.0 ± 0%     296.0 ± 0%  -17.78% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         4.570Ki ± 0%   3.945Ki ± 0%  -13.68% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        39.63Ki ± 0%   33.38Ki ± 0%  -15.77% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10       404.3Ki ± 0%   341.8Ki ± 0%  -15.46% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      5.995Mi ± 0%   5.384Mi ± 0%  -10.18% (p=0.002 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1_series_per_ingester-10            376.0 ± 0%     376.0 ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         4.586Ki ± 0%   4.586Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        39.65Ki ± 0%   39.65Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       404.3Ki ± 0%   404.3Ki ± 0%        ~ (p=1.000 n=6)
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      5.995Mi ± 0%   5.995Mi ± 0%        ~ (p=0.801 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1_series_per_ingester-10            712.0 ± 0%     488.0 ± 0%  -31.46% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         9.133Ki ± 0%   6.945Ki ± 0%  -23.95% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        79.26Ki ± 0%   57.38Ki ± 0%  -27.60% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      1120.5Ki ± 0%   901.8Ki ± 0%  -19.52% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10      12.53Mi ± 0%   10.39Mi ± 0%  -17.05% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1_series_per_ingester-10            744.0 ± 0%     616.0 ± 0%  -17.20% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         9.164Ki ± 0%   7.914Ki ± 0%  -13.64% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        79.29Ki ± 0%   66.79Ki ± 0%  -15.77% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      1120.5Ki ± 0%   995.5Ki ± 0%  -11.16% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10      12.53Mi ± 0%   11.31Mi ± 0%   -9.74% (p=0.002 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1_series_per_ingester-10            776.0 ± 0%     776.0 ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         9.195Ki ± 0%   9.195Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        79.32Ki ± 0%   79.32Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10       1.094Mi ± 0%   1.094Mi ± 0%        ~ (p=0.494 n=6)
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10      12.53Mi ± 0%   12.53Mi ± 0%        ~ (p=0.234 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1_series_per_ingester-10         2.336Ki ± 0%   1.789Ki ± 0%  -23.41% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        19.87Ki ± 0%   14.40Ki ± 0%  -27.53% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_100_series_per_ingester-10       230.2Ki ± 0%   175.5Ki ± 0%  -23.76% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      2.747Mi ± 0%   2.213Mi ± 0%  -19.44% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     32.16Mi ± 0%   26.82Mi ± 0%  -16.61% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1_series_per_ingester-10         2.414Ki ± 0%   2.102Ki ± 0%  -12.94% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        19.95Ki ± 0%   16.82Ki ± 0%  -15.67% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_100_series_per_ingester-10       230.3Ki ± 0%   199.0Ki ± 0%  -13.57% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      2.747Mi ± 0%   2.442Mi ± 0%  -11.11% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     32.16Mi ± 0%   29.11Mi ± 0%   -9.49% (p=0.002 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1_series_per_ingester-10         2.492Ki ± 0%   2.492Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10_series_per_ingester-10        20.02Ki ± 0%   20.02Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_100_series_per_ingester-10       230.3Ki ± 0%   230.3Ki ± 0%        ~ (p=1.000 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      2.748Mi ± 0%   2.748Mi ± 0%        ~ (p=0.976 n=6)
MergingAndSortingSeries/10_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     32.16Mi ± 0%   32.16Mi ± 0%        ~ (p=0.229 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1_series_per_ingester-10        20.66Ki ± 0%   15.20Ki ± 0%  -26.47% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10_series_per_ingester-10       231.0Ki ± 0%   176.3Ki ± 0%  -23.68% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_100_series_per_ingester-10      2.748Mi ± 0%   2.214Mi ± 0%  -19.43% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10     32.16Mi ± 0%   26.82Mi ± 0%  -16.61% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10    315.1Mi ± 0%   261.7Mi ± 0%  -16.95% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1_series_per_ingester-10        21.54Ki ± 0%   18.41Ki ± 0%  -14.51% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10_series_per_ingester-10       231.9Ki ± 0%   200.6Ki ± 0%  -13.48% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_100_series_per_ingester-10      2.749Mi ± 0%   2.444Mi ± 0%  -11.10% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10     32.16Mi ± 0%   29.11Mi ± 0%   -9.49% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10    315.1Mi ± 0%   284.6Mi ± 0%   -9.68% (p=0.002 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1_series_per_ingester-10        22.41Ki ± 0%   22.41Ki ± 0%        ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10_series_per_ingester-10       232.7Ki ± 0%   232.7Ki ± 0%        ~ (p=0.455 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_100_series_per_ingester-10      2.750Mi ± 0%   2.750Mi ± 0%        ~ (p=0.294 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10     32.16Mi ± 0%   32.16Mi ± 0%        ~ (p=0.476 n=6)
MergingAndSortingSeries/100_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10    315.1Mi ± 0%   315.1Mi ± 0%        ~ (p=1.000 n=6)
geomean                                                                                  143.8Ki        125.1Ki       -12.98%
¹ all samples are equal

                                                                                      │  heap.txt   │         heap-zone-count.txt         │
                                                                                      │  allocs/op  │  allocs/op   vs base                │
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         17.00 ± 0%    17.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        110.0 ± 0%    110.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      1.014k ± 0%   1.014k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     10.02k ± 0%   10.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         17.00 ± 0%    17.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        110.0 ± 0%    110.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      1.014k ± 0%   1.014k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     10.02k ± 0%   10.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         17.00 ± 0%    17.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        110.0 ± 0%    110.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      1.014k ± 0%   1.014k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/1_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     10.02k ± 0%   10.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          6.000 ± 0%    6.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         28.00 ± 0%    28.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        211.0 ± 0%    211.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      2.015k ± 0%   2.015k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     20.02k ± 0%   20.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          6.000 ± 0%    6.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         28.00 ± 0%    28.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        211.0 ± 0%    211.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      2.015k ± 0%   2.015k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     20.02k ± 0%   20.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          6.000 ± 0%    6.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         28.00 ± 0%    28.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        211.0 ± 0%    211.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      2.015k ± 0%   2.015k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/2_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     20.02k ± 0%   20.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1_series_per_ingester-10          9.000 ± 0%    9.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10_series_per_ingester-10         49.00 ± 0%    49.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_100_series_per_ingester-10        412.0 ± 0%    412.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10      4.018k ± 0%   4.018k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10     40.03k ± 0%   40.03k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1_series_per_ingester-10          9.000 ± 0%    9.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10_series_per_ingester-10         49.00 ± 0%    49.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_100_series_per_ingester-10        412.0 ± 0%    412.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_1000_series_per_ingester-10      4.018k ± 0%   4.018k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_2_zones,_10000_series_per_ingester-10     40.03k ± 0%   40.03k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1_series_per_ingester-10          9.000 ± 0%    9.000 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10_series_per_ingester-10         49.00 ± 0%    49.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_100_series_per_ingester-10        412.0 ± 0%    412.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_1000_series_per_ingester-10      4.018k ± 0%   4.018k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/4_ingesters_per_zone,_3_zones,_10000_series_per_ingester-10     40.03k ± 0%   40.03k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1_series_per_ingester-10         17.00 ± 0%    17.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10_series_per_ingester-10        110.0 ± 0%    110.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_100_series_per_ingester-10      1.014k ± 0%   1.014k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_1000_series_per_ingester-10     10.02k ± 0%   10.02k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_1_zones,_10000_series_per_ingester-10    100.0k ± 0%   100.0k ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_1_series_per_ingester-10         17.00 ± 0%    17.00 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_10_series_per_ingester-10        110.0 ± 0%    110.0 ± 0%       ~ (p=1.000 n=6) ¹
MergingAndSortingSeries/10_ingesters_per_zone,_2_zones,_100_series_per_ingester-10      1.014k ± 0%   1.014k ± 0%       ~ (p=1.000 n=6) ¹
Mer…

CHANGELOG.md

@@ -50,6 +50,7 @@
 * [ENHANCEMENT] Ruler: trigger a synchronization of tenant's rule groups as soon as they change the rules configuration via API. This synchronization is in addition of the periodic syncing done every `-ruler.poll-interval`. The new behavior is enabled by default, but can be disabled with `-ruler.sync-rules-on-changes-enabled=false` (configurable on a per-tenant basis too). #4975 #5053
 * [ENHANCEMENT] Distributor: Improve invalid tenant shard size error message. #5024
 * [ENHANCEMENT] Store-gateway: record index header loading time separately in `cortex_bucket_store_series_request_stage_duration_seconds{stage="load_index_header"}`. Now index header loading will be visible in the "Mimir / Queries" dashboard in the "Series request p99/average latency" panels. #5011 #5062
+* [ENHANCEMENT] Querier and ingester: add experimental support for streaming chunks from ingesters to queriers while evaluating queries. This can be enabled with `-querier.prefer-streaming-chunks=true`. #4886
 * [ENHANCEMENT] Update Docker base images from `alpine:3.17.3` to `alpine:3.18.0`. #5065
 * [BUGFIX] Metadata API: Mimir will now return an empty object when no metadata is available, matching Prometheus. #4782
 * [BUGFIX] Store-gateway: add collision detection on expanded postings and individual postings cache keys. #4770

cmd/mimir/config-descriptor.json

@@ -1640,6 +1640,28 @@
           "fieldType": "boolean",
           "fieldCategory": "advanced"
         },
+        {
+          "kind": "field",
+          "name": "prefer_streaming_chunks",
+          "required": false,
+          "desc": "Request ingesters stream chunks. Ingesters will only respond with a stream of chunks if the target ingester supports this, and this preference will be ignored by ingesters that do not support this.",
+          "fieldValue": null,
+          "fieldDefaultValue": false,
+          "fieldFlag": "querier.prefer-streaming-chunks",
+          "fieldType": "boolean",
+          "fieldCategory": "experimental"
+        },
+        {
+          "kind": "field",
+          "name": "streaming_chunks_per_ingester_series_buffer_size",
+          "required": false,
+          "desc": "Number of series to buffer per ingester when streaming chunks from ingesters.",
+          "fieldValue": null,
+          "fieldDefaultValue": 512,
+          "fieldFlag": "querier.streaming-chunks-per-ingester-buffer-size",
+          "fieldType": "int",
+          "fieldCategory": "experimental"
+        },
         {
           "kind": "field",
           "name": "max_concurrent",

cmd/mimir/help-all.txt.tmpl

@@ -1555,6 +1555,8 @@ Usage of ./cmd/mimir/mimir:
     	Maximum number of split (by time) or partial (by shard) queries that will be scheduled in parallel by the query-frontend for a single input query. This limit is introduced to have a fairer query scheduling and avoid a single query over a large time range saturating all available queriers. (default 14)
   -querier.max-samples int
     	Maximum number of samples a single query can load into memory. This config option should be set on query-frontend too when query sharding is enabled. (default 50000000)
+  -querier.prefer-streaming-chunks
+    	[experimental] Request ingesters stream chunks. Ingesters will only respond with a stream of chunks if the target ingester supports this, and this preference will be ignored by ingesters that do not support this.
   -querier.query-ingesters-within duration
     	Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester. (default 13h)
   -querier.query-store-after duration
@@ -1579,6 +1581,8 @@ Usage of ./cmd/mimir/mimir:
     	Override the default minimum TLS version. Allowed values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
   -querier.store-gateway-client.tls-server-name string
     	Override the expected name on the server certificate.
+  -querier.streaming-chunks-per-ingester-buffer-size uint
+    	[experimental] Number of series to buffer per ingester when streaming chunks from ingesters. (default 512)
   -querier.timeout duration
     	The timeout for a query. This config option should be set on query-frontend too when query sharding is enabled. This also applies to queries evaluated by the ruler (internally or remotely). (default 2m0s)
   -query-frontend.align-queries-with-step

docs/sources/mimir/configure/about-versioning.md

@@ -89,6 +89,7 @@ The following features are currently experimental:
     - `-blocks-storage.tsdb.block-postings-for-matchers-cache-force`
 - Querier
   - Use of Redis cache backend (`-blocks-storage.bucket-store.metadata-cache.backend=redis`)
+  - Streaming chunks from ingester to querier (`-querier.prefer-streaming-chunks`, `-querier.streaming-chunks-per-ingester-buffer-size`)
 - Query-frontend
   - `-query-frontend.querier-forget-delay`
   - Instant query splitting (`-query-frontend.split-instant-queries-by-interval`)

docs/sources/mimir/references/configuration-parameters/index.md

@@ -1064,6 +1064,17 @@ store_gateway_client:
 # CLI flag: -querier.shuffle-sharding-ingesters-enabled
 [shuffle_sharding_ingesters_enabled: <boolean> | default = true]
 
+# (experimental) Request ingesters stream chunks. Ingesters will only respond
+# with a stream of chunks if the target ingester supports this, and this
+# preference will be ignored by ingesters that do not support this.
+# CLI flag: -querier.prefer-streaming-chunks
+[prefer_streaming_chunks: <boolean> | default = false]
+
+# (experimental) Number of series to buffer per ingester when streaming chunks
+# from ingesters.
+# CLI flag: -querier.streaming-chunks-per-ingester-buffer-size
+[streaming_chunks_per_ingester_series_buffer_size: <int> | default = 512]
+
 # The number of workers running in each querier process. This setting limits the
 # maximum number of concurrent queries in each querier.
 # CLI flag: -querier.max-concurrent

integration/ingester_test.go

@@ -4,7 +4,9 @@
 package integration
 
 import (
+	"fmt"
 	"net/http"
+	"strconv"
 	"testing"
 	"time"
 
@@ -20,7 +22,7 @@ import (
 	"github.com/grafana/mimir/pkg/util/test"
 )
 
-func TestIngesterMixedFloatHistogramSeries(t *testing.T) {
+func TestIngesterQuerying(t *testing.T) {
 	query := "foobar"
 	queryEnd := time.Now().Round(time.Second)
 	queryStart := queryEnd.Add(-1 * time.Hour)
@@ -332,49 +334,58 @@ func TestIngesterMixedFloatHistogramSeries(t *testing.T) {
 
 	for testName, tc := range testCases {
 		t.Run(testName, func(t *testing.T) {
-			s, err := e2e.NewScenario(networkName)
-			require.NoError(t, err)
-			defer s.Close()
+			for _, streamingEnabled := range []bool{true, false} {
+				t.Run(fmt.Sprintf("streaming enabled: %v", streamingEnabled), func(t *testing.T) {
+					s, err := e2e.NewScenario(networkName)
+					require.NoError(t, err)
+					defer s.Close()
 
-			flags := mergeFlags(
-				BlocksStorageFlags(),
-				BlocksStorageS3Flags(),
-			)
-			flags["-distributor.ingestion-tenant-shard-size"] = "0"
-			flags["-ingester.ring.heartbeat-period"] = "1s"
+					baseFlags := map[string]string{
+						"-distributor.ingestion-tenant-shard-size": "0",
+						"-ingester.ring.heartbeat-period":          "1s",
+						"-querier.prefer-streaming-chunks":         strconv.FormatBool(streamingEnabled),
+					}
 
-			// Start dependencies.
-			consul := e2edb.NewConsul()
-			minio := e2edb.NewMinio(9000, flags["-blocks-storage.s3.bucket-name"])
-			require.NoError(t, s.StartAndWaitReady(consul, minio))
+					flags := mergeFlags(
+						BlocksStorageFlags(),
+						BlocksStorageS3Flags(),
+						baseFlags,
+					)
 
-			// Start Mimir components.
-			distributor := e2emimir.NewDistributor("distributor", consul.NetworkHTTPEndpoint(), flags)
-			ingester := e2emimir.NewIngester("ingester", consul.NetworkHTTPEndpoint(), flags)
-			querier := e2emimir.NewQuerier("querier", consul.NetworkHTTPEndpoint(), flags)
-			require.NoError(t, s.StartAndWaitReady(distributor, ingester, querier))
+					// Start dependencies.
+					consul := e2edb.NewConsul()
+					minio := e2edb.NewMinio(9000, flags["-blocks-storage.s3.bucket-name"])
+					require.NoError(t, s.StartAndWaitReady(consul, minio))
 
-			// Wait until distributor has updated the ring.
-			require.NoError(t, distributor.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_ring_members"}, e2e.WithLabelMatchers(
-				labels.MustNewMatcher(labels.MatchEqual, "name", "ingester"),
-				labels.MustNewMatcher(labels.MatchEqual, "state", "ACTIVE"))))
+					// Start Mimir components.
+					distributor := e2emimir.NewDistributor("distributor", consul.NetworkHTTPEndpoint(), flags)
+					ingester := e2emimir.NewIngester("ingester", consul.NetworkHTTPEndpoint(), flags)
+					querier := e2emimir.NewQuerier("querier", consul.NetworkHTTPEndpoint(), flags)
+					require.NoError(t, s.StartAndWaitReady(distributor, ingester, querier))
 
-			// Wait until querier has updated the ring.
-			require.NoError(t, querier.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_ring_members"}, e2e.WithLabelMatchers(
-				labels.MustNewMatcher(labels.MatchEqual, "name", "ingester"),
-				labels.MustNewMatcher(labels.MatchEqual, "state", "ACTIVE"))))
+					// Wait until distributor has updated the ring.
+					require.NoError(t, distributor.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_ring_members"}, e2e.WithLabelMatchers(
+						labels.MustNewMatcher(labels.MatchEqual, "name", "ingester"),
+						labels.MustNewMatcher(labels.MatchEqual, "state", "ACTIVE"))))
 
-			client, err := e2emimir.NewClient(distributor.HTTPEndpoint(), querier.HTTPEndpoint(), "", "", userID)
-			require.NoError(t, err)
+					// Wait until querier has updated the ring.
+					require.NoError(t, querier.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_ring_members"}, e2e.WithLabelMatchers(
+						labels.MustNewMatcher(labels.MatchEqual, "name", "ingester"),
+						labels.MustNewMatcher(labels.MatchEqual, "state", "ACTIVE"))))
 
-			res, err := client.Push(tc.inSeries)
-			require.NoError(t, err)
-			require.Equal(t, http.StatusOK, res.StatusCode)
+					client, err := e2emimir.NewClient(distributor.HTTPEndpoint(), querier.HTTPEndpoint(), "", "", userID)
+					require.NoError(t, err)
 
-			result, err := client.QueryRange(query, queryStart, queryEnd, queryStep)
-			require.NoError(t, err)
+					res, err := client.Push(tc.inSeries)
+					require.NoError(t, err)
+					require.Equal(t, http.StatusOK, res.StatusCode)
 
-			require.Equal(t, tc.expected.String(), result.String())
+					result, err := client.QueryRange(query, queryStart, queryEnd, queryStep)
+					require.NoError(t, err)
+
+					require.Equal(t, tc.expected.String(), result.String())
+				})
+			}
 		})
 	}
 }

integration/querier_test.go

@@ -868,77 +868,82 @@ func TestQuerierWithBlocksStorageOnMissingBlocksFromStorage(t *testing.T) {
 func TestQueryLimitsWithBlocksStorageRunningInMicroServices(t *testing.T) {
 	const blockRangePeriod = 5 * time.Second
 
-	s, err := e2e.NewScenario(networkName)
-	require.NoError(t, err)
-	defer s.Close()
+	for _, streamingEnabled := range []bool{true, false} {
+		t.Run(fmt.Sprintf("streaming enabled: %v", streamingEnabled), func(t *testing.T) {
+			s, err := e2e.NewScenario(networkName)
+			require.NoError(t, err)
+			defer s.Close()
 
-	// Configure the blocks storage to frequently compact TSDB head
-	// and ship blocks to the storage.
-	flags := mergeFlags(BlocksStorageFlags(), BlocksStorageS3Flags(), map[string]string{
-		"-blocks-storage.tsdb.block-ranges-period":   blockRangePeriod.String(),
-		"-blocks-storage.tsdb.ship-interval":         "1s",
-		"-blocks-storage.bucket-store.sync-interval": "1s",
-		"-blocks-storage.tsdb.retention-period":      ((blockRangePeriod * 2) - 1).String(),
-		"-querier.max-fetched-series-per-query":      "3",
-	})
+			// Configure the blocks storage to frequently compact TSDB head
+			// and ship blocks to the storage.
+			flags := mergeFlags(BlocksStorageFlags(), BlocksStorageS3Flags(), map[string]string{
+				"-blocks-storage.tsdb.block-ranges-period":   blockRangePeriod.String(),
+				"-blocks-storage.tsdb.ship-interval":         "1s",
+				"-blocks-storage.bucket-store.sync-interval": "1s",
+				"-blocks-storage.tsdb.retention-period":      ((blockRangePeriod * 2) - 1).String(),
+				"-querier.max-fetched-series-per-query":      "3",
+				"-querier.prefer-streaming-chunks":           strconv.FormatBool(streamingEnabled),
+			})
 
-	// Start dependencies.
-	consul := e2edb.NewConsul()
-	minio := e2edb.NewMinio(9000, flags["-blocks-storage.s3.bucket-name"])
-	memcached := e2ecache.NewMemcached()
-	require.NoError(t, s.StartAndWaitReady(consul, minio, memcached))
+			// Start dependencies.
+			consul := e2edb.NewConsul()
+			minio := e2edb.NewMinio(9000, flags["-blocks-storage.s3.bucket-name"])
+			memcached := e2ecache.NewMemcached()
+			require.NoError(t, s.StartAndWaitReady(consul, minio, memcached))
 
-	// Add the memcached address to the flags.
-	flags["-blocks-storage.bucket-store.index-cache.memcached.addresses"] = "dns+" + memcached.NetworkEndpoint(e2ecache.MemcachedPort)
+			// Add the memcached address to the flags.
+			flags["-blocks-storage.bucket-store.index-cache.memcached.addresses"] = "dns+" + memcached.NetworkEndpoint(e2ecache.MemcachedPort)
 
-	// Start Mimir components.
-	distributor := e2emimir.NewDistributor("distributor", consul.NetworkHTTPEndpoint(), flags)
-	ingester := e2emimir.NewIngester("ingester", consul.NetworkHTTPEndpoint(), flags)
-	storeGateway := e2emimir.NewStoreGateway("store-gateway", consul.NetworkHTTPEndpoint(), flags)
-	require.NoError(t, s.StartAndWaitReady(distributor, ingester, storeGateway))
+			// Start Mimir components.
+			distributor := e2emimir.NewDistributor("distributor", consul.NetworkHTTPEndpoint(), flags)
+			ingester := e2emimir.NewIngester("ingester", consul.NetworkHTTPEndpoint(), flags)
+			storeGateway := e2emimir.NewStoreGateway("store-gateway", consul.NetworkHTTPEndpoint(), flags)
+			require.NoError(t, s.StartAndWaitReady(distributor, ingester, storeGateway))
 
-	querier := e2emimir.NewQuerier("querier", consul.NetworkHTTPEndpoint(), flags)
-	require.NoError(t, s.StartAndWaitReady(querier))
+			querier := e2emimir.NewQuerier("querier", consul.NetworkHTTPEndpoint(), flags)
+			require.NoError(t, s.StartAndWaitReady(querier))
 
-	c, err := e2emimir.NewClient(distributor.HTTPEndpoint(), querier.HTTPEndpoint(), "", "", "user-1")
-	require.NoError(t, err)
+			c, err := e2emimir.NewClient(distributor.HTTPEndpoint(), querier.HTTPEndpoint(), "", "", "user-1")
+			require.NoError(t, err)
 
-	// Push some series to Mimir.
-	series1Name := "series_1"
-	series2Name := "series_2"
-	series3Name := "series_3"
-	series4Name := "series_4"
-	series1Timestamp := time.Now()
-	series2Timestamp := series1Timestamp.Add(blockRangePeriod * 2)
-	series3Timestamp := series1Timestamp.Add(blockRangePeriod * 2)
-	series4Timestamp := series1Timestamp.Add(blockRangePeriod * 3)
+			// Push some series to Mimir.
+			series1Name := "series_1"
+			series2Name := "series_2"
+			series3Name := "series_3"
+			series4Name := "series_4"
+			series1Timestamp := time.Now()
+			series2Timestamp := series1Timestamp.Add(blockRangePeriod * 2)
+			series3Timestamp := series1Timestamp.Add(blockRangePeriod * 2)
+			series4Timestamp := series1Timestamp.Add(blockRangePeriod * 3)
 
-	series1, _, _ := generateFloatSeries(series1Name, series1Timestamp, prompb.Label{Name: series1Name, Value: series1Name})
-	series2, _, _ := generateHistogramSeries(series2Name, series2Timestamp, prompb.Label{Name: series2Name, Value: series2Name})
-	series3, _, _ := generateFloatSeries(series3Name, series3Timestamp, prompb.Label{Name: series3Name, Value: series3Name})
-	series4, _, _ := generateHistogramSeries(series4Name, series4Timestamp, prompb.Label{Name: series4Name, Value: series4Name})
+			series1, _, _ := generateFloatSeries(series1Name, series1Timestamp, prompb.Label{Name: series1Name, Value: series1Name})
+			series2, _, _ := generateHistogramSeries(series2Name, series2Timestamp, prompb.Label{Name: series2Name, Value: series2Name})
+			series3, _, _ := generateFloatSeries(series3Name, series3Timestamp, prompb.Label{Name: series3Name, Value: series3Name})
+			series4, _, _ := generateHistogramSeries(series4Name, series4Timestamp, prompb.Label{Name: series4Name, Value: series4Name})
 
-	res, err := c.Push(series1)
-	require.NoError(t, err)
-	require.Equal(t, 200, res.StatusCode)
-	res, err = c.Push(series2)
-	require.NoError(t, err)
-	require.Equal(t, 200, res.StatusCode)
+			res, err := c.Push(series1)
+			require.NoError(t, err)
+			require.Equal(t, 200, res.StatusCode)
+			res, err = c.Push(series2)
+			require.NoError(t, err)
+			require.Equal(t, 200, res.StatusCode)
 
-	result, err := c.QueryRange("{__name__=~\"series_.+\"}", series1Timestamp, series2Timestamp.Add(1*time.Hour), blockRangePeriod)
-	require.NoError(t, err)
-	require.Equal(t, model.ValMatrix, result.Type())
+			result, err := c.QueryRange("{__name__=~\"series_.+\"}", series1Timestamp, series2Timestamp.Add(1*time.Hour), blockRangePeriod)
+			require.NoError(t, err)
+			require.Equal(t, model.ValMatrix, result.Type())
 
-	res, err = c.Push(series3)
-	require.NoError(t, err)
-	require.Equal(t, 200, res.StatusCode)
-	res, err = c.Push(series4)
-	require.NoError(t, err)
-	require.Equal(t, 200, res.StatusCode)
+			res, err = c.Push(series3)
+			require.NoError(t, err)
+			require.Equal(t, 200, res.StatusCode)
+			res, err = c.Push(series4)
+			require.NoError(t, err)
+			require.Equal(t, 200, res.StatusCode)
 
-	_, err = c.QueryRange("{__name__=~\"series_.+\"}", series1Timestamp, series4Timestamp.Add(1*time.Hour), blockRangePeriod)
-	require.Error(t, err)
-	assert.ErrorContains(t, err, "the query exceeded the maximum number of series")
+			_, err = c.QueryRange("{__name__=~\"series_.+\"}", series1Timestamp, series4Timestamp.Add(1*time.Hour), blockRangePeriod)
+			require.Error(t, err)
+			assert.ErrorContains(t, err, "the query exceeded the maximum number of series")
+		})
+	}
 }
 
 func TestHashCollisionHandling(t *testing.T) {