chunks: decode varints directly from byte buffer; stop panicing on some corrupt inputs #7264

bboreham · 2022-09-27T13:33:11Z

What this PR does / why we need it:

This is much faster, since we don't pay for byte-at-a-time reads from the buffer.

For best performance with gzip and lz4, we still want a bufio.Reader. We don't need the pool of buffered readers now.

Other compression formats have their own internal buffers so don't need extra buffering.

Checklist

Reviewed the CONTRIBUTING.md guide
NA Documentation added
NA Tests updated
CHANGELOG.md updated
NA Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

(using changes from #7246, ignoring lines that didn't change much)

name                           old time/op      new time/op      delta
Write/ordered-lz4-256k-4           24.1ms ± 7%      22.1ms ± 2%    -8.52%  (p=0.008 n=5+5)
Write/unordered-lz4-4              54.6ms ±29%      49.1ms ± 1%   -10.04%  (p=0.016 n=5+4)
Read/none_66_kB-4                  4.16ms ± 6%      2.54ms ± 2%   -38.87%  (p=0.008 n=5+5)
Read/gzip_66_kB-4                   177ms ± 4%       161ms ± 4%    -9.09%  (p=0.008 n=5+5)
Read/lz4-64k_66_kB-4               59.5ms ± 2%      49.5ms ± 3%   -16.80%  (p=0.008 n=5+5)
Read/lz4-256k_66_kB-4              62.9ms ± 0%      51.2ms ± 1%   -18.58%  (p=0.016 n=4+5)
Read/lz4-1M_66_kB-4                62.9ms ± 2%      51.5ms ± 1%   -18.03%  (p=0.008 n=5+5)
Read/lz4_66_kB-4                   63.7ms ± 4%      51.2ms ± 1%   -19.68%  (p=0.008 n=5+5)
Read/snappy_66_kB-4                51.1ms ± 2%      41.9ms ± 1%   -17.95%  (p=0.016 n=5+4)
Read/flate_66_kB-4                  168ms ± 4%       150ms ± 2%   -10.49%  (p=0.008 n=5+5)
Read/zstd_66_kB-4                   334ms ±16%       272ms ± 6%   -18.52%  (p=0.008 n=5+5)
Read/none_262_kB-4                 3.93ms ± 1%      2.48ms ± 2%   -36.91%  (p=0.008 n=5+5)
Read/gzip_262_kB-4                  159ms ± 5%       143ms ± 2%   -10.12%  (p=0.008 n=5+5)
Read/lz4-64k_262_kB-4              53.5ms ± 1%      43.2ms ± 3%   -19.35%  (p=0.008 n=5+5)
Read/lz4-256k_262_kB-4             55.8ms ± 1%      45.3ms ± 2%   -18.75%  (p=0.008 n=5+5)
Read/lz4-1M_262_kB-4               57.5ms ± 2%      45.5ms ± 2%   -20.91%  (p=0.008 n=5+5)
Read/lz4_262_kB-4                  56.8ms ± 2%      45.3ms ± 2%   -20.26%  (p=0.008 n=5+5)
Read/snappy_262_kB-4               46.0ms ± 2%      37.5ms ± 1%   -18.57%  (p=0.008 n=5+5)
Read/flate_262_kB-4                 153ms ± 3%       134ms ± 2%   -12.33%  (p=0.008 n=5+5)
Read/none_524_kB-4                 3.85ms ± 2%      2.46ms ± 2%   -36.04%  (p=0.008 n=5+5)
Read/gzip_524_kB-4                  128ms ± 2%       119ms ± 2%    -7.64%  (p=0.008 n=5+5)
Read/lz4-64k_524_kB-4              44.6ms ± 3%      34.7ms ± 1%   -22.19%  (p=0.008 n=5+5)
Read/lz4-256k_524_kB-4             45.4ms ± 1%      36.7ms ± 5%   -19.07%  (p=0.008 n=5+5)
Read/lz4-1M_524_kB-4               47.3ms ± 1%      38.6ms ± 2%   -18.36%  (p=0.008 n=5+5)
Read/lz4_524_kB-4                  47.4ms ± 1%      38.0ms ± 1%   -19.70%  (p=0.008 n=5+5)
Read/snappy_524_kB-4               37.3ms ± 1%      31.1ms ± 3%   -16.47%  (p=0.008 n=5+5)
Read/flate_524_kB-4                 122ms ± 1%       111ms ± 3%    -9.04%  (p=0.008 n=5+5)
Read/sample_none_66_kB-4           7.81ms ± 1%      6.62ms ± 6%   -15.24%  (p=0.008 n=5+5)
Read/sample_gzip_66_kB-4            237ms ± 2%       219ms ± 1%    -7.57%  (p=0.008 n=5+5)
Read/sample_lz4-64k_66_kB-4         105ms ± 2%        92ms ± 1%   -12.03%  (p=0.008 n=5+5)
Read/sample_lz4-256k_66_kB-4        109ms ± 1%        98ms ± 2%    -9.65%  (p=0.008 n=5+5)
Read/sample_lz4-1M_66_kB-4          113ms ± 8%        98ms ± 2%   -13.66%  (p=0.008 n=5+5)
Read/sample_lz4_66_kB-4             109ms ± 1%        99ms ± 2%    -9.29%  (p=0.008 n=5+5)
Read/sample_snappy_66_kB-4         86.6ms ± 1%      77.2ms ± 6%   -10.92%  (p=0.008 n=5+5)
Read/sample_flate_66_kB-4           236ms ± 8%       214ms ± 3%    -9.11%  (p=0.008 n=5+5)
Read/sample_none_262_kB-4          7.70ms ± 2%      6.34ms ± 2%   -17.67%  (p=0.008 n=5+5)
Read/sample_lz4-64k_262_kB-4       94.7ms ± 1%      84.4ms ± 2%   -10.90%  (p=0.008 n=5+5)
Read/sample_lz4-256k_262_kB-4      98.4ms ± 1%      87.7ms ± 2%   -10.93%  (p=0.008 n=5+5)
Read/sample_lz4-1M_262_kB-4         101ms ± 2%        90ms ± 2%   -11.19%  (p=0.008 n=5+5)
Read/sample_lz4_262_kB-4            100ms ± 2%        90ms ± 3%   -10.78%  (p=0.008 n=5+5)
Read/sample_snappy_262_kB-4        77.9ms ± 2%      68.7ms ± 2%   -11.84%  (p=0.008 n=5+5)
Read/sample_flate_262_kB-4          211ms ± 2%       190ms ± 2%    -9.86%  (p=0.008 n=5+5)
Read/sample_none_524_kB-4          7.82ms ± 2%      6.43ms ± 2%   -17.80%  (p=0.008 n=5+5)
Read/sample_gzip_524_kB-4           176ms ± 1%       164ms ± 2%    -6.80%  (p=0.008 n=5+5)
Read/sample_lz4-64k_524_kB-4       77.7ms ± 1%      69.8ms ± 4%   -10.10%  (p=0.008 n=5+5)
Read/sample_lz4-256k_524_kB-4      81.7ms ± 2%      72.1ms ± 3%   -11.79%  (p=0.008 n=5+5)
Read/sample_lz4-1M_524_kB-4        83.8ms ± 1%      74.4ms ± 1%   -11.21%  (p=0.008 n=5+5)
Read/sample_lz4_524_kB-4           83.2ms ± 0%      76.1ms ± 8%    -8.53%  (p=0.016 n=4+5)
Read/sample_snappy_524_kB-4        63.6ms ± 2%      56.5ms ± 2%   -11.23%  (p=0.008 n=5+5)
Read/sample_flate_524_kB-4          172ms ± 2%       157ms ± 1%    -8.88%  (p=0.008 n=5+5)

name                           old speed        new speed        delta
Write/ordered-lz4-256k-4          687MB/s ± 6%     749MB/s ± 2%    +9.09%  (p=0.008 n=5+5)
Write/unordered-lz4-4             313MB/s ±24%     341MB/s ± 1%    +9.23%  (p=0.016 n=5+4)
Read/none_66_kB-4                1.82GB/s ± 6%    2.97GB/s ± 2%   +63.42%  (p=0.008 n=5+5)
Read/gzip_66_kB-4                 669MB/s ± 4%     736MB/s ± 4%   +10.00%  (p=0.008 n=5+5)
Read/lz4-64k_66_kB-4             1.42GB/s ± 2%    1.71GB/s ± 3%   +20.21%  (p=0.008 n=5+5)
Read/lz4-256k_66_kB-4            1.43GB/s ± 0%    1.76GB/s ± 1%   +22.83%  (p=0.016 n=4+5)
Read/lz4-1M_66_kB-4              1.43GB/s ± 2%    1.74GB/s ± 1%   +21.99%  (p=0.008 n=5+5)
Read/lz4_66_kB-4                 1.41GB/s ± 4%    1.76GB/s ± 1%   +24.44%  (p=0.008 n=5+5)
Read/snappy_66_kB-4              1.29GB/s ± 2%    1.57GB/s ± 1%   +21.86%  (p=0.016 n=5+4)
Read/flate_66_kB-4                710MB/s ± 4%     793MB/s ± 2%   +11.70%  (p=0.008 n=5+5)
Read/zstd_66_kB-4                 425MB/s ±14%     517MB/s ± 6%   +21.74%  (p=0.008 n=5+5)
Read/none_262_kB-4               1.93GB/s ± 1%    3.06GB/s ± 2%   +58.54%  (p=0.008 n=5+5)
Read/gzip_262_kB-4                701MB/s ± 5%     779MB/s ± 2%   +11.20%  (p=0.008 n=5+5)
Read/lz4-64k_262_kB-4            1.47GB/s ± 1%    1.82GB/s ± 3%   +24.03%  (p=0.008 n=5+5)
Read/lz4-256k_262_kB-4           1.48GB/s ± 1%    1.82GB/s ± 2%   +23.09%  (p=0.008 n=5+5)
Read/lz4-1M_262_kB-4             1.46GB/s ± 2%    1.84GB/s ± 2%   +26.44%  (p=0.008 n=5+5)
Read/lz4_262_kB-4                1.48GB/s ± 2%    1.85GB/s ± 2%   +25.42%  (p=0.008 n=5+5)
Read/snappy_262_kB-4             1.31GB/s ± 2%    1.61GB/s ± 1%   +22.80%  (p=0.008 n=5+5)
Read/flate_262_kB-4               729MB/s ± 3%     831MB/s ± 2%   +14.05%  (p=0.008 n=5+5)
Read/none_524_kB-4               1.97GB/s ± 2%    3.08GB/s ± 2%   +56.33%  (p=0.008 n=5+5)
Read/gzip_524_kB-4                714MB/s ± 2%     773MB/s ± 2%    +8.27%  (p=0.008 n=5+5)
Read/lz4-64k_524_kB-4            1.47GB/s ± 3%    1.88GB/s ± 1%   +28.48%  (p=0.008 n=5+5)
Read/lz4-256k_524_kB-4           1.50GB/s ± 1%    1.86GB/s ± 5%   +23.63%  (p=0.008 n=5+5)
Read/lz4-1M_524_kB-4             1.49GB/s ± 1%    1.83GB/s ± 2%   +22.51%  (p=0.008 n=5+5)
Read/lz4_524_kB-4                1.49GB/s ± 1%    1.86GB/s ± 1%   +24.53%  (p=0.008 n=5+5)
Read/snappy_524_kB-4             1.34GB/s ± 1%    1.60GB/s ± 3%   +19.75%  (p=0.008 n=5+5)
Read/flate_524_kB-4               750MB/s ± 1%     824MB/s ± 3%    +9.97%  (p=0.008 n=5+5)
Read/sample_none_66_kB-4          967MB/s ± 1%    1142MB/s ± 6%   +18.13%  (p=0.008 n=5+5)
Read/sample_gzip_66_kB-4          501MB/s ± 2%     542MB/s ± 1%    +8.19%  (p=0.008 n=5+5)
Read/sample_lz4-64k_66_kB-4       811MB/s ± 2%     922MB/s ± 1%   +13.66%  (p=0.008 n=5+5)
Read/sample_lz4-256k_66_kB-4      825MB/s ± 1%     913MB/s ± 2%   +10.68%  (p=0.008 n=5+5)
Read/sample_lz4-1M_66_kB-4        795MB/s ± 7%     920MB/s ± 1%   +15.67%  (p=0.008 n=5+5)
Read/sample_lz4_66_kB-4           824MB/s ± 1%     909MB/s ± 2%   +10.25%  (p=0.008 n=5+5)
Read/sample_snappy_66_kB-4        759MB/s ± 1%     852MB/s ± 6%   +12.37%  (p=0.008 n=5+5)
Read/sample_flate_66_kB-4         506MB/s ± 7%     556MB/s ± 4%    +9.88%  (p=0.008 n=5+5)
Read/sample_none_262_kB-4         983MB/s ± 2%    1194MB/s ± 2%   +21.47%  (p=0.008 n=5+5)
Read/sample_lz4-64k_262_kB-4      830MB/s ± 1%     932MB/s ± 2%   +12.24%  (p=0.008 n=5+5)
Read/sample_lz4-256k_262_kB-4     839MB/s ± 1%     942MB/s ± 2%   +12.28%  (p=0.008 n=5+5)
Read/sample_lz4-1M_262_kB-4       829MB/s ± 2%     934MB/s ± 2%   +12.60%  (p=0.008 n=5+5)
Read/sample_lz4_262_kB-4          836MB/s ± 2%     937MB/s ± 3%   +12.11%  (p=0.008 n=5+5)
Read/sample_snappy_262_kB-4       774MB/s ± 2%     878MB/s ± 2%   +13.44%  (p=0.008 n=5+5)
Read/sample_flate_262_kB-4        529MB/s ± 2%     587MB/s ± 2%   +10.94%  (p=0.008 n=5+5)
Read/sample_none_524_kB-4         970MB/s ± 2%    1180MB/s ± 2%   +21.64%  (p=0.008 n=5+5)
Read/sample_gzip_524_kB-4         521MB/s ± 1%     559MB/s ± 2%    +7.31%  (p=0.008 n=5+5)
Read/sample_lz4-64k_524_kB-4      842MB/s ± 1%     937MB/s ± 4%   +11.30%  (p=0.008 n=5+5)
Read/sample_lz4-256k_524_kB-4     834MB/s ± 2%     945MB/s ± 3%   +13.37%  (p=0.008 n=5+5)
Read/sample_lz4-1M_524_kB-4       842MB/s ± 1%     949MB/s ± 1%   +12.63%  (p=0.008 n=5+5)
Read/sample_lz4_524_kB-4          849MB/s ± 0%     929MB/s ± 7%    +9.53%  (p=0.016 n=4+5)
Read/sample_snappy_524_kB-4       783MB/s ± 2%     882MB/s ± 2%   +12.65%  (p=0.008 n=5+5)
Read/sample_flate_524_kB-4        534MB/s ± 2%     585MB/s ± 1%    +9.73%  (p=0.008 n=5+5)

name                           old alloc/op     new alloc/op     delta
Write/unordered-lz4-4              18.8MB ± 4%      17.2MB ± 7%    -8.28%  (p=0.008 n=5+5)
Read/none_66_kB-4                  41.9kB ± 0%      43.7kB ± 0%    +4.41%  (p=0.008 n=5+5)
Read/gzip_66_kB-4                   713kB ± 0%       741kB ± 0%    +3.85%  (p=0.016 n=5+4)
Read/lz4-64k_66_kB-4                730kB ± 1%       748kB ± 0%    +2.53%  (p=0.016 n=5+4)
Read/lz4-256k_66_kB-4               770kB ± 0%       806kB ± 3%    +4.70%  (p=0.016 n=4+5)
Read/lz4-1M_66_kB-4                 770kB ± 0%       812kB ± 4%    +5.43%  (p=0.016 n=4+5)
Read/lz4_66_kB-4                    770kB ± 0%       792kB ± 0%    +2.83%  (p=0.029 n=4+4)
Read/snappy_66_kB-4                 361kB ± 0%       376kB ± 0%    +4.02%  (p=0.008 n=5+5)
Read/flate_66_kB-4                  715kB ± 0%       743kB ± 0%    +3.88%  (p=0.016 n=5+4)
Read/none_262_kB-4                 11.2kB ± 0%      11.7kB ± 0%    +4.12%  (p=0.016 n=5+4)
Read/gzip_262_kB-4                  198kB ± 0%       203kB ± 0%    +2.93%  (p=0.016 n=4+5)
Read/lz4-64k_262_kB-4               169kB ± 0%       174kB ± 0%    +2.75%  (p=0.008 n=5+5)
Read/lz4-256k_262_kB-4              177kB ± 0%       182kB ± 0%    +2.79%  (p=0.008 n=5+5)
Read/lz4-1M_262_kB-4                181kB ± 0%       186kB ± 0%    +2.77%  (p=0.008 n=5+5)
Read/lz4_262_kB-4                   181kB ± 0%       186kB ± 0%    +2.73%  (p=0.016 n=5+4)
Read/snappy_262_kB-4               87.8kB ± 0%      90.1kB ± 0%    +2.58%  (p=0.008 n=5+5)
Read/flate_262_kB-4                 197kB ± 0%       203kB ± 0%    +2.63%  (p=0.016 n=5+4)
Read/zstd_262_kB-4                  553MB ± 0%       552MB ± 0%    -0.15%  (p=0.032 n=5+5)
Read/none_524_kB-4                 5.94kB ± 0%      6.16kB ± 0%    +3.76%  (p=0.016 n=5+4)
Read/gzip_524_kB-4                 96.2kB ± 0%      98.3kB ± 0%    +2.16%  (p=0.016 n=4+5)
Read/lz4-64k_524_kB-4              70.9kB ± 0%      72.8kB ± 0%    +2.71%  (p=0.016 n=4+5)
Read/lz4-256k_524_kB-4             73.8kB ± 0%      75.8kB ± 0%    +2.68%  (p=0.008 n=5+5)
Read/lz4-1M_524_kB-4               76.5kB ± 0%      78.7kB ± 0%    +2.77%  (p=0.016 n=4+5)
Read/lz4_524_kB-4                  76.5kB ± 0%      78.6kB ± 0%    +2.74%  (p=0.016 n=5+4)
Read/snappy_524_kB-4               39.1kB ± 0%      39.5kB ± 0%    +1.19%  (p=0.008 n=5+5)
Read/flate_524_kB-4                95.4kB ± 0%      97.3kB ± 0%    +2.06%  (p=0.008 n=5+5)
Read/sample_none_66_kB-4           39.9kB ± 0%      41.7kB ± 0%    +4.62%  (p=0.008 n=5+5)
Read/sample_gzip_66_kB-4            686kB ± 0%       715kB ± 0%    +4.23%  (p=0.016 n=4+5)
Read/sample_lz4-64k_66_kB-4         707kB ± 0%       728kB ± 0%    +2.88%  (p=0.016 n=5+4)
Read/sample_lz4-256k_66_kB-4        749kB ± 0%       783kB ± 4%    +4.65%  (p=0.008 n=5+5)
Read/sample_lz4_66_kB-4             748kB ± 0%       770kB ± 0%    +2.91%  (p=0.016 n=5+4)
Read/sample_snappy_66_kB-4          350kB ± 0%       364kB ± 0%    +4.04%  (p=0.029 n=4+4)
Read/sample_flate_66_kB-4           688kB ± 0%       716kB ± 0%    +4.08%  (p=0.029 n=4+4)
Read/sample_none_262_kB-4          10.6kB ± 0%      11.0kB ± 0%    +4.23%  (p=0.016 n=5+4)
Read/sample_gzip_262_kB-4           194kB ± 0%       199kB ± 0%    +2.48%  (p=0.016 n=4+5)
Read/sample_lz4-64k_262_kB-4        165kB ± 0%       169kB ± 0%    +2.86%  (p=0.016 n=4+5)
Read/sample_lz4-256k_262_kB-4       172kB ± 0%       177kB ± 0%    +2.92%  (p=0.016 n=4+5)
Read/sample_lz4-1M_262_kB-4         176kB ± 0%       181kB ± 0%    +2.91%  (p=0.029 n=4+4)
Read/sample_lz4_262_kB-4            176kB ± 0%       181kB ± 0%    +2.81%  (p=0.016 n=5+4)
Read/sample_snappy_262_kB-4        88.1kB ± 0%      90.8kB ± 0%    +3.11%  (p=0.029 n=4+4)
Read/sample_flate_262_kB-4          194kB ± 0%       198kB ± 0%    +2.13%  (p=0.029 n=4+4)
Read/sample_none_524_kB-4          5.56kB ± 0%      5.76kB ± 0%    +3.65%  (p=0.016 n=5+4)
Read/sample_gzip_524_kB-4          95.4kB ± 0%      96.9kB ± 0%    +1.56%  (p=0.008 n=5+5)
Read/sample_lz4-64k_524_kB-4       69.0kB ± 0%      71.0kB ± 0%    +2.87%  (p=0.008 n=5+5)
Read/sample_lz4-256k_524_kB-4      71.8kB ± 0%      73.8kB ± 0%    +2.79%  (p=0.016 n=4+5)
Read/sample_lz4-1M_524_kB-4        74.5kB ± 0%      76.6kB ± 0%    +2.75%  (p=0.008 n=5+5)
Read/sample_lz4_524_kB-4           74.5kB ± 0%     188.4kB ±89%  +152.98%  (p=0.008 n=5+5)
Read/sample_snappy_524_kB-4        40.7kB ± 1%      41.3kB ± 1%    +1.58%  (p=0.008 n=5+5)
Read/sample_flate_524_kB-4         95.2kB ± 0%      96.2kB ± 0%    +0.95%  (p=0.008 n=5+5)

name                           old allocs/op    new allocs/op    delta
Read/gzip_66_kB-4                   13.2k ± 0%       13.2k ± 0%    -0.04%  (p=0.000 n=5+4)
Read/zstd_66_kB-4                   18.0k ± 1%       17.8k ± 1%    -1.06%  (p=0.032 n=5+5)
Read/gzip_262_kB-4                  4.61k ± 0%       4.61k ± 0%    -0.11%  (p=0.029 n=4+4)
Read/flate_262_kB-4                 4.61k ± 0%       4.60k ± 0%    -0.11%  (p=0.008 n=5+5)
Read/zstd_262_kB-4                  7.10k ± 2%       6.88k ± 3%    -3.01%  (p=0.008 n=5+5)
Read/gzip_524_kB-4                  2.62k ± 0%       2.62k ± 0%    -0.06%  (p=0.000 n=4+5)
Read/zstd_524_kB-4                  6.13k ± 2%       5.94k ± 2%    -3.03%  (p=0.024 n=5+5)
Read/sample_flate_66_kB-4           13.2k ± 0%       13.2k ± 0%    -0.01%  (p=0.029 n=4+4)
Read/sample_flate_262_kB-4          4.62k ± 0%       4.61k ± 0%    -0.22%  (p=0.029 n=4+4)
Read/sample_zstd_262_kB-4           7.12k ± 1%       6.97k ± 1%    -2.11%  (p=0.008 n=5+5)
Read/sample_gzip_524_kB-4           2.63k ± 0%       2.63k ± 0%    -0.11%  (p=0.008 n=5+5)
Read/sample_flate_524_kB-4          2.63k ± 0%       2.62k ± 0%    -0.15%  (p=0.008 n=5+5)
Read/sample_zstd_524_kB-4           6.11k ± 2%       5.89k ± 1%    -3.68%  (p=0.016 n=5+4)

This is much faster, since we don't pay for byte-at-a-time reads from the buffer. For best performance with gzip and lz4, we still want a bufio.Reader. We don't need the pool of buffered readers now. Other compression formats have their own internal buffers so don't need extra buffering.

grafanabot · 2022-09-27T13:38:38Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0.4%
+              logql	0%
+               loki	0%

cyriltovena · 2022-09-27T15:19:07Z

pkg/chunkenc/memchunk.go

 		}
+		var l uint64
+		ts, tWidth = binary.Varint(si.readBuf[:si.readBufValid])


cyriltovena

Nice work !

owen-d

Just one question, but looks great! Thanks.

owen-d · 2022-09-27T15:37:32Z

pkg/chunkenc/memchunk.go

+	// Shift down what is still left in the fixed-size read buffer, if any.
+	si.readBufValid = copy(si.readBuf[:], si.readBuf[tWidth+lWidth+n:si.readBufValid])


I think this is impossible, since si.buf will be long enough to hold the line (logic above) and

Copy returns the number of elements copied, which will be the minimum of len(src) and len(dst).

Adjusting readBuf & readBufValid could further complicate the next moveNext call if there's not enough space remaining for the varints (timestamp, linesize).

Suggested change

// Shift down what is still left in the fixed-size read buffer, if any.

si.readBufValid = copy(si.readBuf[:], si.readBuf[tWidth+lWidth+n:si.readBufValid])

si.readBufValid = 0

Again, I think this is impossible, but this will make the code a little clearer.

Suppose the line was of length 0 bytes, or anything up to about 10. Then tWidth is maybe 4, lWidth is 1 and n is <=10, but readBufValid can be 20. So we need to copy.

Ah, yes this makes sense. I definitely missed it on my first pass; thanks for the help and great PR!

sandeepsukhani

Nice one 🤩

bboreham · 2022-09-28T06:59:17Z

pkg/chunkenc/memchunk.go

-			return 0, nil, false
+	var ts int64
+	var tWidth, lWidth, lineSize int
+	for lWidth == 0 { // Read until both varints have enough bytes.


I realise this can loop forever on some corrupt inputs - if there are some bytes in the buffer but not enough to decode two varints. Need to detect this somehow and error.

bboreham · 2022-09-29T21:16:11Z

I added a test for the infinite loop that I foresaw, and that made the gzip decoder panic, so I fixed both issues.

grafanabot · 2022-09-29T21:21:29Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
- querier/queryrange	-0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0.4%
+              logql	0%
+               loki	0%

owen-d

LGTM

owen-d · 2022-09-30T14:58:33Z

pkg/chunkenc/memchunk.go

+	// Shift down what is still left in the fixed-size read buffer, if any.
+	si.readBufValid = copy(si.readBuf[:], si.readBuf[tWidth+lWidth+n:si.readBufValid])


Ah, yes this makes sense. I definitely missed it on my first pass; thanks for the help and great PR!

grafanabot · 2022-09-30T15:07:11Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0.4%
+              logql	0%
+               loki	0%

…me corrupt inputs (grafana#7264) **What this PR does / why we need it**: This is much faster, since we don't pay for byte-at-a-time reads from the buffer. For best performance with gzip and lz4, we still want a bufio.Reader. We don't need the pool of buffered readers now. Other compression formats have their own internal buffers so don't need extra buffering. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` (using changes from grafana#7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-lz4-256k-4 24.1ms ± 7% 22.1ms ± 2% -8.52% (p=0.008 n=5+5) Write/unordered-lz4-4 54.6ms ±29% 49.1ms ± 1% -10.04% (p=0.016 n=5+4) Read/none_66_kB-4 4.16ms ± 6% 2.54ms ± 2% -38.87% (p=0.008 n=5+5) Read/gzip_66_kB-4 177ms ± 4% 161ms ± 4% -9.09% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 59.5ms ± 2% 49.5ms ± 3% -16.80% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 62.9ms ± 0% 51.2ms ± 1% -18.58% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 62.9ms ± 2% 51.5ms ± 1% -18.03% (p=0.008 n=5+5) Read/lz4_66_kB-4 63.7ms ± 4% 51.2ms ± 1% -19.68% (p=0.008 n=5+5) Read/snappy_66_kB-4 51.1ms ± 2% 41.9ms ± 1% -17.95% (p=0.016 n=5+4) Read/flate_66_kB-4 168ms ± 4% 150ms ± 2% -10.49% (p=0.008 n=5+5) Read/zstd_66_kB-4 334ms ±16% 272ms ± 6% -18.52% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 2.48ms ± 2% -36.91% (p=0.008 n=5+5) Read/gzip_262_kB-4 159ms ± 5% 143ms ± 2% -10.12% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 53.5ms ± 1% 43.2ms ± 3% -19.35% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 55.8ms ± 1% 45.3ms ± 2% -18.75% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 57.5ms ± 2% 45.5ms ± 2% -20.91% (p=0.008 n=5+5) Read/lz4_262_kB-4 56.8ms ± 2% 45.3ms ± 2% -20.26% (p=0.008 n=5+5) Read/snappy_262_kB-4 46.0ms ± 2% 37.5ms ± 1% -18.57% (p=0.008 n=5+5) Read/flate_262_kB-4 153ms ± 3% 134ms ± 2% -12.33% (p=0.008 n=5+5) Read/none_524_kB-4 3.85ms ± 2% 2.46ms ± 2% -36.04% (p=0.008 n=5+5) Read/gzip_524_kB-4 128ms ± 2% 119ms ± 2% -7.64% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 44.6ms ± 3% 34.7ms ± 1% -22.19% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 45.4ms ± 1% 36.7ms ± 5% -19.07% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 47.3ms ± 1% 38.6ms ± 2% -18.36% (p=0.008 n=5+5) Read/lz4_524_kB-4 47.4ms ± 1% 38.0ms ± 1% -19.70% (p=0.008 n=5+5) Read/snappy_524_kB-4 37.3ms ± 1% 31.1ms ± 3% -16.47% (p=0.008 n=5+5) Read/flate_524_kB-4 122ms ± 1% 111ms ± 3% -9.04% (p=0.008 n=5+5) Read/sample_none_66_kB-4 7.81ms ± 1% 6.62ms ± 6% -15.24% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 237ms ± 2% 219ms ± 1% -7.57% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 105ms ± 2% 92ms ± 1% -12.03% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 109ms ± 1% 98ms ± 2% -9.65% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 113ms ± 8% 98ms ± 2% -13.66% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 109ms ± 1% 99ms ± 2% -9.29% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 86.6ms ± 1% 77.2ms ± 6% -10.92% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 236ms ± 8% 214ms ± 3% -9.11% (p=0.008 n=5+5) Read/sample_none_262_kB-4 7.70ms ± 2% 6.34ms ± 2% -17.67% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 94.7ms ± 1% 84.4ms ± 2% -10.90% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 98.4ms ± 1% 87.7ms ± 2% -10.93% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 101ms ± 2% 90ms ± 2% -11.19% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 100ms ± 2% 90ms ± 3% -10.78% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 77.9ms ± 2% 68.7ms ± 2% -11.84% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 211ms ± 2% 190ms ± 2% -9.86% (p=0.008 n=5+5) Read/sample_none_524_kB-4 7.82ms ± 2% 6.43ms ± 2% -17.80% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 176ms ± 1% 164ms ± 2% -6.80% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 77.7ms ± 1% 69.8ms ± 4% -10.10% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 81.7ms ± 2% 72.1ms ± 3% -11.79% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 83.8ms ± 1% 74.4ms ± 1% -11.21% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 83.2ms ± 0% 76.1ms ± 8% -8.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 63.6ms ± 2% 56.5ms ± 2% -11.23% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 172ms ± 2% 157ms ± 1% -8.88% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-lz4-256k-4 687MB/s ± 6% 749MB/s ± 2% +9.09% (p=0.008 n=5+5) Write/unordered-lz4-4 313MB/s ±24% 341MB/s ± 1% +9.23% (p=0.016 n=5+4) Read/none_66_kB-4 1.82GB/s ± 6% 2.97GB/s ± 2% +63.42% (p=0.008 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 736MB/s ± 4% +10.00% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 1.42GB/s ± 2% 1.71GB/s ± 3% +20.21% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 1.43GB/s ± 0% 1.76GB/s ± 1% +22.83% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 1.43GB/s ± 2% 1.74GB/s ± 1% +21.99% (p=0.008 n=5+5) Read/lz4_66_kB-4 1.41GB/s ± 4% 1.76GB/s ± 1% +24.44% (p=0.008 n=5+5) Read/snappy_66_kB-4 1.29GB/s ± 2% 1.57GB/s ± 1% +21.86% (p=0.016 n=5+4) Read/flate_66_kB-4 710MB/s ± 4% 793MB/s ± 2% +11.70% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 517MB/s ± 6% +21.74% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 3.06GB/s ± 2% +58.54% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 779MB/s ± 2% +11.20% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 1.47GB/s ± 1% 1.82GB/s ± 3% +24.03% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 1.48GB/s ± 1% 1.82GB/s ± 2% +23.09% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 1.46GB/s ± 2% 1.84GB/s ± 2% +26.44% (p=0.008 n=5+5) Read/lz4_262_kB-4 1.48GB/s ± 2% 1.85GB/s ± 2% +25.42% (p=0.008 n=5+5) Read/snappy_262_kB-4 1.31GB/s ± 2% 1.61GB/s ± 1% +22.80% (p=0.008 n=5+5) Read/flate_262_kB-4 729MB/s ± 3% 831MB/s ± 2% +14.05% (p=0.008 n=5+5) Read/none_524_kB-4 1.97GB/s ± 2% 3.08GB/s ± 2% +56.33% (p=0.008 n=5+5) Read/gzip_524_kB-4 714MB/s ± 2% 773MB/s ± 2% +8.27% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 1.47GB/s ± 3% 1.88GB/s ± 1% +28.48% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 1.50GB/s ± 1% 1.86GB/s ± 5% +23.63% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 1.49GB/s ± 1% 1.83GB/s ± 2% +22.51% (p=0.008 n=5+5) Read/lz4_524_kB-4 1.49GB/s ± 1% 1.86GB/s ± 1% +24.53% (p=0.008 n=5+5) Read/snappy_524_kB-4 1.34GB/s ± 1% 1.60GB/s ± 3% +19.75% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 824MB/s ± 3% +9.97% (p=0.008 n=5+5) Read/sample_none_66_kB-4 967MB/s ± 1% 1142MB/s ± 6% +18.13% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 501MB/s ± 2% 542MB/s ± 1% +8.19% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 811MB/s ± 2% 922MB/s ± 1% +13.66% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 825MB/s ± 1% 913MB/s ± 2% +10.68% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 795MB/s ± 7% 920MB/s ± 1% +15.67% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 824MB/s ± 1% 909MB/s ± 2% +10.25% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 759MB/s ± 1% 852MB/s ± 6% +12.37% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 506MB/s ± 7% 556MB/s ± 4% +9.88% (p=0.008 n=5+5) Read/sample_none_262_kB-4 983MB/s ± 2% 1194MB/s ± 2% +21.47% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 830MB/s ± 1% 932MB/s ± 2% +12.24% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 839MB/s ± 1% 942MB/s ± 2% +12.28% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 829MB/s ± 2% 934MB/s ± 2% +12.60% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 836MB/s ± 2% 937MB/s ± 3% +12.11% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 774MB/s ± 2% 878MB/s ± 2% +13.44% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 529MB/s ± 2% 587MB/s ± 2% +10.94% (p=0.008 n=5+5) Read/sample_none_524_kB-4 970MB/s ± 2% 1180MB/s ± 2% +21.64% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 521MB/s ± 1% 559MB/s ± 2% +7.31% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 842MB/s ± 1% 937MB/s ± 4% +11.30% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 834MB/s ± 2% 945MB/s ± 3% +13.37% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 842MB/s ± 1% 949MB/s ± 1% +12.63% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 849MB/s ± 0% 929MB/s ± 7% +9.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 783MB/s ± 2% 882MB/s ± 2% +12.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 534MB/s ± 2% 585MB/s ± 1% +9.73% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.2MB ± 7% -8.28% (p=0.008 n=5+5) Read/none_66_kB-4 41.9kB ± 0% 43.7kB ± 0% +4.41% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 741kB ± 0% +3.85% (p=0.016 n=5+4) Read/lz4-64k_66_kB-4 730kB ± 1% 748kB ± 0% +2.53% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 770kB ± 0% 806kB ± 3% +4.70% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 770kB ± 0% 812kB ± 4% +5.43% (p=0.016 n=4+5) Read/lz4_66_kB-4 770kB ± 0% 792kB ± 0% +2.83% (p=0.029 n=4+4) Read/snappy_66_kB-4 361kB ± 0% 376kB ± 0% +4.02% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 743kB ± 0% +3.88% (p=0.016 n=5+4) Read/none_262_kB-4 11.2kB ± 0% 11.7kB ± 0% +4.12% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 203kB ± 0% +2.93% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 174kB ± 0% +2.75% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 177kB ± 0% 182kB ± 0% +2.79% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 181kB ± 0% 186kB ± 0% +2.77% (p=0.008 n=5+5) Read/lz4_262_kB-4 181kB ± 0% 186kB ± 0% +2.73% (p=0.016 n=5+4) Read/snappy_262_kB-4 87.8kB ± 0% 90.1kB ± 0% +2.58% (p=0.008 n=5+5) Read/flate_262_kB-4 197kB ± 0% 203kB ± 0% +2.63% (p=0.016 n=5+4) Read/zstd_262_kB-4 553MB ± 0% 552MB ± 0% -0.15% (p=0.032 n=5+5) Read/none_524_kB-4 5.94kB ± 0% 6.16kB ± 0% +3.76% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.3kB ± 0% +2.16% (p=0.016 n=4+5) Read/lz4-64k_524_kB-4 70.9kB ± 0% 72.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/lz4-256k_524_kB-4 73.8kB ± 0% 75.8kB ± 0% +2.68% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 76.5kB ± 0% 78.7kB ± 0% +2.77% (p=0.016 n=4+5) Read/lz4_524_kB-4 76.5kB ± 0% 78.6kB ± 0% +2.74% (p=0.016 n=5+4) Read/snappy_524_kB-4 39.1kB ± 0% 39.5kB ± 0% +1.19% (p=0.008 n=5+5) Read/flate_524_kB-4 95.4kB ± 0% 97.3kB ± 0% +2.06% (p=0.008 n=5+5) Read/sample_none_66_kB-4 39.9kB ± 0% 41.7kB ± 0% +4.62% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 715kB ± 0% +4.23% (p=0.016 n=4+5) Read/sample_lz4-64k_66_kB-4 707kB ± 0% 728kB ± 0% +2.88% (p=0.016 n=5+4) Read/sample_lz4-256k_66_kB-4 749kB ± 0% 783kB ± 4% +4.65% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 748kB ± 0% 770kB ± 0% +2.91% (p=0.016 n=5+4) Read/sample_snappy_66_kB-4 350kB ± 0% 364kB ± 0% +4.04% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 716kB ± 0% +4.08% (p=0.029 n=4+4) Read/sample_none_262_kB-4 10.6kB ± 0% 11.0kB ± 0% +4.23% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 199kB ± 0% +2.48% (p=0.016 n=4+5) Read/sample_lz4-64k_262_kB-4 165kB ± 0% 169kB ± 0% +2.86% (p=0.016 n=4+5) Read/sample_lz4-256k_262_kB-4 172kB ± 0% 177kB ± 0% +2.92% (p=0.016 n=4+5) Read/sample_lz4-1M_262_kB-4 176kB ± 0% 181kB ± 0% +2.91% (p=0.029 n=4+4) Read/sample_lz4_262_kB-4 176kB ± 0% 181kB ± 0% +2.81% (p=0.016 n=5+4) Read/sample_snappy_262_kB-4 88.1kB ± 0% 90.8kB ± 0% +3.11% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 194kB ± 0% 198kB ± 0% +2.13% (p=0.029 n=4+4) Read/sample_none_524_kB-4 5.56kB ± 0% 5.76kB ± 0% +3.65% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 96.9kB ± 0% +1.56% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 69.0kB ± 0% 71.0kB ± 0% +2.87% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 71.8kB ± 0% 73.8kB ± 0% +2.79% (p=0.016 n=4+5) Read/sample_lz4-1M_524_kB-4 74.5kB ± 0% 76.6kB ± 0% +2.75% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 74.5kB ± 0% 188.4kB ±89% +152.98% (p=0.008 n=5+5) Read/sample_snappy_524_kB-4 40.7kB ± 1% 41.3kB ± 1% +1.58% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 96.2kB ± 0% +0.95% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Read/gzip_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.04% (p=0.000 n=5+4) Read/zstd_66_kB-4 18.0k ± 1% 17.8k ± 1% -1.06% (p=0.032 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.61k ± 0% -0.11% (p=0.029 n=4+4) Read/flate_262_kB-4 4.61k ± 0% 4.60k ± 0% -0.11% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 6.88k ± 3% -3.01% (p=0.008 n=5+5) Read/gzip_524_kB-4 2.62k ± 0% 2.62k ± 0% -0.06% (p=0.000 n=4+5) Read/zstd_524_kB-4 6.13k ± 2% 5.94k ± 2% -3.03% (p=0.024 n=5+5) Read/sample_flate_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.01% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 4.62k ± 0% 4.61k ± 0% -0.22% (p=0.029 n=4+4) Read/sample_zstd_262_kB-4 7.12k ± 1% 6.97k ± 1% -2.11% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.63k ± 0% -0.11% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.62k ± 0% -0.15% (p=0.008 n=5+5) Read/sample_zstd_524_kB-4 6.11k ± 2% 5.89k ± 1% -3.68% (p=0.016 n=5+4) ``` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>

bboreham requested a review from a team as a code owner September 27, 2022 13:33

pull-request-size bot added the size/L label Sep 27, 2022

cyriltovena reviewed Sep 27, 2022

View reviewed changes

pkg/chunkenc/memchunk.go

}

var l uint64

ts, tWidth = binary.Varint(si.readBuf[:si.readBufValid])

Copy link

Contributor

cyriltovena Sep 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

cyriltovena approved these changes Sep 27, 2022

View reviewed changes

owen-d reviewed Sep 27, 2022

View reviewed changes

sandeepsukhani approved these changes Sep 28, 2022

View reviewed changes

bboreham commented Sep 28, 2022

View reviewed changes

bboreham added 3 commits September 29, 2022 15:00

Add test for corrupt chunk contents

dc4c78d

Avoid infinite loop on corrupt chunk contents

4c84b0b

Don't panic on unexpected error from chunk decoding

3168c85

bboreham changed the title ~~chunks: decode varints directly from byte buffer~~ chunks: decode varints directly from byte buffer; stop panicing on some corrupt inputs Sep 29, 2022

owen-d approved these changes Sep 30, 2022

View reviewed changes

Merge branch 'main' into chunk-buffers

9ce2380

owen-d enabled auto-merge (squash) September 30, 2022 15:02

owen-d merged commit 9498ea2 into main Sep 30, 2022

owen-d deleted the chunk-buffers branch September 30, 2022 15:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chunks: decode varints directly from byte buffer; stop panicing on some corrupt inputs #7264

chunks: decode varints directly from byte buffer; stop panicing on some corrupt inputs #7264

bboreham commented Sep 27, 2022 •

edited

Loading

grafanabot commented Sep 27, 2022

cyriltovena Sep 27, 2022

cyriltovena left a comment

owen-d left a comment

owen-d Sep 27, 2022

bboreham Sep 27, 2022

owen-d Sep 30, 2022

sandeepsukhani left a comment

bboreham Sep 28, 2022

bboreham commented Sep 29, 2022

grafanabot commented Sep 29, 2022

owen-d left a comment

owen-d Sep 30, 2022

grafanabot commented Sep 30, 2022

		// Shift down what is still left in the fixed-size read buffer, if any.
		si.readBufValid = copy(si.readBuf[:], si.readBuf[tWidth+lWidth+n:si.readBufValid])

	// Shift down what is still left in the fixed-size read buffer, if any.
	si.readBufValid = copy(si.readBuf[:], si.readBuf[tWidth+lWidth+n:si.readBufValid])
	si.readBufValid = 0

chunks: decode varints directly from byte buffer; stop panicing on some corrupt inputs #7264

chunks: decode varints directly from byte buffer; stop panicing on some corrupt inputs #7264

Conversation

bboreham commented Sep 27, 2022 • edited Loading

grafanabot commented Sep 27, 2022

cyriltovena Sep 27, 2022

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

owen-d left a comment

Choose a reason for hiding this comment

owen-d Sep 27, 2022

Choose a reason for hiding this comment

bboreham Sep 27, 2022

Choose a reason for hiding this comment

owen-d Sep 30, 2022

Choose a reason for hiding this comment

sandeepsukhani left a comment

Choose a reason for hiding this comment

bboreham Sep 28, 2022

Choose a reason for hiding this comment

bboreham commented Sep 29, 2022

grafanabot commented Sep 29, 2022

owen-d left a comment

Choose a reason for hiding this comment

owen-d Sep 30, 2022

Choose a reason for hiding this comment

grafanabot commented Sep 30, 2022

bboreham commented Sep 27, 2022 •

edited

Loading