Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(blooms): Resolve bloom blocks on index gateway and shard by block address #12720

Merged
merged 28 commits into from
Apr 23, 2024
Merged

perf(blooms): Resolve bloom blocks on index gateway and shard by block address #12720

merged 28 commits into from
Apr 23, 2024

Commits on Apr 22, 2024

  1. Add metric that observes time spent resolving blocks 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    6cf356a View commit details
    Browse the repository at this point in the history

  2. Remove deprecated field from FilterChunkRefsRequest payload 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    a4c5b7e View commit details
    Browse the repository at this point in the history

  3. Add Blocks field to FilterChunkRefRequest message 

    When preset, the bloom gateway should use the given blocks
instead of resolving them from the request parameters.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    1042f31 View commit details
    Browse the repository at this point in the history

  4. Partition filter requests by day on client side 

    later we want to resolve bloom blocks for a filter single request, and
because blocks are built per day, we need to split the original interval
into separate days

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    882694e View commit details
    Browse the repository at this point in the history

  5. Reject queries on bloom gw when they span across multiple days 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    533eb09 View commit details
    Browse the repository at this point in the history

  6. Do not split request by day in bloom gateway 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d511035 View commit details
    Browse the repository at this point in the history

  7. Pass down blocks from requests to tasks 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    9f98240 View commit details
    Browse the repository at this point in the history

  8. Add processor test case with providing blocks 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    be1fa4f View commit details
    Browse the repository at this point in the history

  9. Resolve blocks and shard by block address on index gateways 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    4e6f11d View commit details
    Browse the repository at this point in the history

  10. Add BloomStore as dependency for IndexGateway 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    1c0393b View commit details
    Browse the repository at this point in the history

  11. Use interval for block resolver 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    43b046d View commit details
    Browse the repository at this point in the history

  12. Cleanup 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    16fe790 View commit details
    Browse the repository at this point in the history

  13. Test and benchmark for chunk grouping 

    ```console
$ benchstat old.txt new.txt
goos: linux
goarch: amd64
pkg: github.com/grafana/loki/v3/pkg/bloomgateway
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
                 │   old.txt    │              new.txt               │
                 │    sec/op    │   sec/op     vs base               │
GroupChunkRefs-8   12.42µ ± 18%   11.43µ ± 3%  -8.01% (p=0.000 n=10)

                 │   old.txt    │             new.txt              │
                 │     B/op     │     B/op      vs base            │
GroupChunkRefs-8   7.148Ki ± 0%   7.148Ki ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

                 │  old.txt   │            new.txt             │
                 │ allocs/op  │ allocs/op   vs base            │
GroupChunkRefs-8   100.0 ± 0%   100.0 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal
```

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    b103e27 View commit details
    Browse the repository at this point in the history

  14. Additional test for partitioning request 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    15d69aa View commit details
    Browse the repository at this point in the history

  15. Add span to bloom querier 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    ec54d21 View commit details
    Browse the repository at this point in the history

  16. Debugging 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    6971505 View commit details
    Browse the repository at this point in the history

  17. fixup! Debugging 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    ba56649 View commit details
    Browse the repository at this point in the history

  18. Sort input series before sending to bloom gateway 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    fa8b97d View commit details
    Browse the repository at this point in the history

  19. Fix linter error 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    8938f97 View commit details
    Browse the repository at this point in the history

  20. Remove unused test 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    4d70f88 View commit details
    Browse the repository at this point in the history

  21. Revert some gateway changes to keep PR minimal 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    529c553 View commit details
    Browse the repository at this point in the history

  22. Make linter happy 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    8f977d3 View commit details
    Browse the repository at this point in the history

  23. fixup! Make linter happy 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    736a77c View commit details
    Browse the repository at this point in the history

  24. Return unfiltered list if no block are given 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    5ae6596 View commit details
    Browse the repository at this point in the history

  25. Remove old code path for resolving blocks on bloom gateways 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    e53ee29 View commit details
    Browse the repository at this point in the history

  26. Assume sorted chunk when grouping by fingerprint 

    Chunk refs are already sorted by their fingerprint when they are read
from the TSDB index.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    4724cca View commit details
    Browse the repository at this point in the history

  27. Define index gateway service dependencies statically 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    b8dd578 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. fixup! Define index gateway service dependencies statically 

    Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
    @chaudum
    chaudum committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    a04b558 View commit details
    Browse the repository at this point in the history