Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reload search blocks and replay search WAL #1000

Merged
merged 23 commits into from
Oct 6, 2021

Conversation

annanay25
Copy link
Contributor

@annanay25 annanay25 commented Sep 30, 2021

What this PR does:

  • Adds versioned encoding support to StreamingSearchBlock. See benchmark numbers below for each encoding/compression format.
  • Adds ability to reload search WAL files and BackendSearchBlocks
  • Clubs WAL filename parsing logic between AppendBlock and StreamingSearchBlock

also

  • PR 1000!
Annanays-Mac:tempo annanay$ go test -bench=. ./tempodb/search -test.bench BenchmarkStreamingSearchBlockSearch
StreamingSearchBlock search throughput: 148µs elapsed 0.03 MB = 207.49 MiB/s throughput
goos: darwin
goarch: amd64
pkg: github.com/grafana/tempo/tempodb/search
cpu: Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
BenchmarkStreamingSearchBlockSearch/none-8         	StreamingSearchBlock search throughput: 7.165565ms elapsed 3.07 MB = 428.55 MiB/s throughput
StreamingSearchBlock search throughput: 556.788432ms elapsed 307.08 MB = 551.53 MiB/s throughput
StreamingSearchBlock search throughput: 1.241136492s elapsed 661.76 MB = 533.19 MiB/s throughput
   21550	     57596 ns/op
StreamingSearchBlock search throughput: 1.081183ms elapsed 0.01 MB = 13.58 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/gzip-8         	StreamingSearchBlock search throughput: 19.094169ms elapsed 1.47 MB = 76.92 MiB/s throughput
StreamingSearchBlock search throughput: 1.011387332s elapsed 91.91 MB = 90.87 MiB/s throughput
    6258	    161620 ns/op
StreamingSearchBlock search throughput: 1.059104ms elapsed 0.02 MB = 18.37 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/lz4-64k-8      	StreamingSearchBlock search throughput: 10.095353ms elapsed 1.95 MB = 192.71 MiB/s throughput
StreamingSearchBlock search throughput: 966.274862ms elapsed 194.55 MB = 201.34 MiB/s throughput
StreamingSearchBlock search throughput: 1.276660241s elapsed 241.57 MB = 189.22 MiB/s throughput
   12417	    102818 ns/op
StreamingSearchBlock search throughput: 2.327903ms elapsed 0.02 MB = 8.36 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/lz4-256k-8     	StreamingSearchBlock search throughput: 15.296878ms elapsed 1.95 MB = 127.18 MiB/s throughput
StreamingSearchBlock search throughput: 850.358104ms elapsed 152.27 MB = 179.07 MiB/s throughput
StreamingSearchBlock search throughput: 1.189829468s elapsed 214.86 MB = 180.58 MiB/s throughput
   11044	    107738 ns/op
StreamingSearchBlock search throughput: 7.191181ms elapsed 0.02 MB = 2.71 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/lz4-1M-8       	StreamingSearchBlock search throughput: 16.926936ms elapsed 1.95 MB = 114.93 MiB/s throughput
StreamingSearchBlock search throughput: 636.008588ms elapsed 137.68 MB = 216.48 MiB/s throughput
StreamingSearchBlock search throughput: 1.329665501s elapsed 259.74 MB = 195.34 MiB/s throughput
   13351	     99595 ns/op
StreamingSearchBlock search throughput: 47.276275ms elapsed 0.02 MB = 0.41 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/lz4-8          	StreamingSearchBlock search throughput: 51.30639ms elapsed 0.49 MB = 9.48 MiB/s throughput
StreamingSearchBlock search throughput: 77.626341ms elapsed 11.36 MB = 146.36 MiB/s throughput
StreamingSearchBlock search throughput: 794.61228ms elapsed 175.56 MB = 220.94 MiB/s throughput
StreamingSearchBlock search throughput: 1.279594939s elapsed 265.11 MB = 207.18 MiB/s throughput
   13627	     93904 ns/op
StreamingSearchBlock search throughput: 695.265µs elapsed 0.02 MB = 26.61 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/snappy-8       	StreamingSearchBlock search throughput: 12.04733ms elapsed 1.85 MB = 153.57 MiB/s throughput
StreamingSearchBlock search throughput: 807.816236ms elapsed 183.64 MB = 227.33 MiB/s throughput
StreamingSearchBlock search throughput: 1.137160722s elapsed 272.76 MB = 239.86 MiB/s throughput
   14743	     77135 ns/op
StreamingSearchBlock search throughput: 2.389645ms elapsed 0.01 MB = 5.67 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/zstd-8         	StreamingSearchBlock search throughput: 19.784036ms elapsed 1.35 MB = 68.45 MiB/s throughput
StreamingSearchBlock search throughput: 886.004558ms elapsed 82.00 MB = 92.55 MiB/s throughput
StreamingSearchBlock search throughput: 1.245373764s elapsed 111.03 MB = 89.16 MiB/s throughput
    8199	    151898 ns/op
StreamingSearchBlock search throughput: 12.581351ms elapsed 0.02 MB = 1.59 MiB/s throughput
BenchmarkStreamingSearchBlockSearch/s2-8           	StreamingSearchBlock search throughput: 17.628801ms elapsed 1.88 MB = 106.79 MiB/s throughput
StreamingSearchBlock search throughput: 516.341665ms elapsed 127.85 MB = 247.61 MiB/s throughput
StreamingSearchBlock search throughput: 1.147026964s elapsed 297.08 MB = 259.00 MiB/s throughput
   14834	     77327 ns/op
PASS

Which issue(s) this PR fixes:
Fixes part of #932

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@joe-elliott
Copy link
Member

PR #1000!

tempodb/wal/wal.go Outdated Show resolved Hide resolved
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@annanay25 annanay25 marked this pull request as ready for review October 1, 2021 13:24
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
@mdisibio mdisibio mentioned this pull request Oct 1, 2021
12 tasks
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Copy link
Contributor

@mdisibio mdisibio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very close to LGTM. Left a few comments.

annanay25 and others added 5 commits October 1, 2021 23:57
Signed-off-by: Annanay <annanayagarwal@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
…lock encoding

Signed-off-by: Martin Disibio <mdisibio@gmail.com>
…lock encoding

Signed-off-by: Martin Disibio <mdisibio@gmail.com>
…essary

Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
modules/ingester/ingester.go Show resolved Hide resolved
modules/ingester/ingester_test.go Outdated Show resolved Hide resolved
modules/ingester/ingester_test.go Outdated Show resolved Hide resolved
tempodb/search/streaming_search_block.go Show resolved Hide resolved
tempodb/wal/wal.go Outdated Show resolved Hide resolved
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
@mdisibio mdisibio merged commit 98b6f73 into grafana:main Oct 6, 2021
@@ -269,6 +270,53 @@ func testAppendReplayFind(t *testing.T, e backend.Encoding) {
require.NoError(t, err)
}

func TestParseFileName(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a test for ParseFileName in append_block_test.go. But no worries, we can club the two in a later PR

@annanay25 annanay25 deleted the reload-search-blocks branch October 11, 2021 08:22

}

func BenchmarkBackendSearchBlockSearch(b *testing.B) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mdisibio did you intend to relocate this test as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants