Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Versioned WAL #638

Merged
merged 22 commits into from
Apr 12, 2021
Merged

Versioned WAL #638

merged 22 commits into from
Apr 12, 2021

Conversation

joe-elliott
Copy link
Member

What this PR does:
Reuses all of the existing versioned code to enabled versioned WALs. This includes the ability to compress the WAL which addresses #418.

Benchmarks:

goos: linux
goarch: amd64
pkg: github.com/grafana/tempo/tempodb/wal
cpu: Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
BenchmarkWALNone-8     	     132	   8162877 ns/op	 2543406 B/op	   38124 allocs/op
BenchmarkWALSnappy-8   	     135	   8712571 ns/op	 2657195 B/op	   36621 allocs/op
BenchmarkWALLZ4-8      	      84	  13025362 ns/op	 5149115 B/op	   49653 allocs/op
BenchmarkWALGZIP-8     	      36	  34852004 ns/op	 3893758 B/op	   40209 allocs/op
BenchmarkWALZSTD-8     	      21	  47785689 ns/op	49718721 B/op	  112807 allocs/op
PASS
ok  	github.com/grafana/tempo/tempodb/wal	7.972s

The benchmarks do not reveal a clear winner here so I'm leaving None as the default and documenting the ability to turn this on. We will experiment with snappy internally.

Which issue(s) this PR fixes:
Fixes #418

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@joe-elliott
Exposed versioned encoding
ba4bf66 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Exposed versioned encoding methods
0cefb0d 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
use versioned encoding
17da830 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Handle v1 encoding
12c1747 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
working with v2 - kind of
bd27012 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added filename tests
6567f68 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added block tests
8258eb8 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added block tests
7c8838d 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Improved parsefilename
c119b19 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
add backend/encoding support for wal iteration. get v0 working
0362400 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
comment
17120ff 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
v2 wal
a83e2f0 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Added bench
753aa41 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
made encoding configurable
c7e0b6b 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
improved testing and benchmarking
78dae82 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Merge branch 'master' into versioned-wal
43ae631
@joe-elliott
Test additional functionality
de1639a 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
Updated docs
5138bbd 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott mentioned this pull request Apr 9, 2021
Closed
@joe-elliott
changelog
86ec2cb 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
lint
55c0073 
Signed-off-by: Joe Elliott <number101010@gmail.com>
mdisibio
mdisibio reviewed Apr 9, 2021
View reviewed changes
Copy link
Contributor

@mdisibio mdisibio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few thoughts

example/docker-compose/etc/tempo-s3-minio.yaml Outdated Show resolved Hide resolved
tempodb/encoding/iterator_wal.go Outdated Show resolved Hide resolved
@joe-elliott
snappy => none
7482bcc 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott
walIterator => recordlessIterator
2891fc3 
Signed-off-by: Joe Elliott <number101010@gmail.com>
@mdisibio mdisibio self-requested a review April 12, 2021 16:00
@joe-elliott joe-elliott merged commit 034d971 into grafana:master Apr 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdisibio mdisibio mdisibio approved these changes

@achatterjee-grafana achatterjee-grafana Awaiting requested review from achatterjee-grafana

@annanay25 annanay25 Awaiting requested review from annanay25

@dgzlopes dgzlopes Awaiting requested review from dgzlopes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add WAL Checksums
2 participants
@joe-elliott @mdisibio