Compressed Blocks? Compressed Blocks!

[joe-elliott]

What this PR does:
This is a rather large PR that completes the versioned blocks refactor as well as adds the ability to configure compression for backend blocks. Compression is configured under a new block config like so:

storage:
  trace:
    block:
      encoding: gzip

The default is currently lz4-256k but other accepted values are none, gzip, lz4-64k, lz4-1M, lz4, snappy. There were obviously quite a few additional changes to get to this point which are detailed here.

Changes

• breaking change Moved config options related to creating a block from under wal to block

storage:
  trace:
    block:
      bloom_filter_false_positive: .05 # previously under wal
      index_downsample: 10 # previously under wal

Consequently the cli params also changed:
trace.wal.bloom-filter-false-positive => trace.block.bloom-filter-false-positive
trace.wal.index-downsample => trace.block.index-downsample

• Removed "FindMetrics". tempo_query_reads and tempo_query_bytes_read have been removed from code and the operational dashboard. These metrics provide some value but due to other conflicts were let go. They may be possible to restore in part down the road.
• Created a versionedEncoding interface through which the various blocks interact with the data in the backend. This shim is messy at the moment and could use some cleanup but I think is the right direction to support future block versions.
• Adds a new field to meta.json (encoding) that contains the current block encoding.

Benchmarks
Benchmarks are available in ./tempodb/encoding. The following output was generated using a 2.1GB block:

size:  418015343
BenchmarkWriteGzip-8     	       1	20011276448 ns/op	2405691104 B/op	 1069040 allocs/op
size:  562592236
BenchmarkWriteSnappy-8   	       1	5828081439 ns/op	2340102872 B/op	 1068429 allocs/op
size:  513621266
BenchmarkWriteLZ4256-8   	       1	7211287032 ns/op	2358356288 B/op	 1073532 allocs/op
size:  502683445
BenchmarkWriteLZ41M-8    	       1	7782626250 ns/op	2407883104 B/op	 1073505 allocs/op
size:  2277963784
BenchmarkWriteNone-8     	       1	4984978160 ns/op	2330022224 B/op	 1068867 allocs/op
size:  418015343
BenchmarkReadGzip-8      	       1	7532250071 ns/op	6754164296 B/op	  324485 allocs/op
size:  562592236
BenchmarkReadSnappy-8    	       1	2833897991 ns/op	6898954664 B/op	  314043 allocs/op
size:  513621266
BenchmarkReadLZ4256-8    	       1	2160460619 ns/op	6854046728 B/op	  318342 allocs/op
size:  502683445
BenchmarkReadLZ41M-8     	       1	2336932036 ns/op	6855424864 B/op	  318311 allocs/op
size:  2277963784
BenchmarkReadNone-8      	       1	2933628972 ns/op	8614623312 B/op	  314845 allocs/op

This PR still requires some additional testing and so is marked as a Draft, but is otherwise ready for review.

Which issue(s) this PR fixes:
Fixes #54

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mdisibio]

Getting some errors running locally:

level=info ts=2021-02-04T19:56:49.3021514Z caller=compactor.go:134 msg="compacting block" block="&{Version:v0 BlockID:fcf48fac-39d6-41a5-9707-c932a4ce625a MinID:[0 0 0 0 0 0 0 0 0 4 87 65 5 156 202 236] MaxID:[0 0 0 0 0 0 0 0 255 255 246 75 14 159 116 118] TenantID:single-tenant StartTime:2021-02-02 16:39:33.291675138 +0000 UTC EndTime:2021-02-02 16:41:03.702583232 +0000 UTC TotalObjects:18875 CompactionLevel:1 Encoding:none}"
level=info ts=2021-02-04T19:56:49.4665595Z caller=compactor.go:134 msg="compacting block" block="&{Version:v1 BlockID:0bdcf10a-cf31-452b-93f3-2ceebdadaf93 MinID:[0 0 0 0 0 0 0 0 0 0 91 23 45 40 208 164] MaxID:[0 0 0 0 0 0 0 0 255 255 255 100 152 56 227 222] TenantID:single-tenant StartTime:2021-02-02 16:46:08.1133679 +0000 UTC EndTime:2021-02-02 16:51:01.206751331 +0000 UTC TotalObjects:36932 CompactionLevel:2 Encoding:snappy}"
level=info ts=2021-02-04T19:58:18.1165904Z caller=compactor.go:126 msg="compaction complete"
level=error ts=2021-02-04T19:58:18.1166729Z caller=compactor.go:100 msg="error during compaction cycle" err="error iterating through object in backend: snappy: corrupt input"

It looks like the compactor was trying to compact an existing block (version 0/none) with a new block (version 1/snappy). The block looks ok at first glance, data file is 22MB compared to 50MB from this test workload. Going to test again with a blank slate.

Edit:
Reproduced with blank slate. This is using https://github.com/mdisibio/tempo-load-test but pointed at real AWS S3.

level=info ts=2021-02-04T20:16:25.8065162Z caller=compactor.go:83 msg="starting compaction cycle" tenantID=single-tenant offset=0
level=info ts=2021-02-04T20:16:25.8066325Z caller=compactor.go:94 msg="Compacting hash" hashString=single-tenant-1-447908
level=info ts=2021-02-04T20:16:25.8070468Z caller=compactor.go:134 msg="compacting block" block="&{Version:v1 BlockID:5f6fbcff-d60c-4c8e-82a5-781eb75a6206 MinID:[0 0 0 0 0 0 0 0 0 8 111 249 202 21 78 17] MaxID:[0 0 0 0 0 0 0 0 255 253 71 97 189 1 224 205] TenantID:single-tenant StartTime:2021-02-04 20:05:00.8816271 +0000 UTC EndTime:2021-02-04 20:07:23.1142084 +0000 UTC TotalObjects:18219 CompactionLevel:1 Encoding:snappy}"
level=info ts=2021-02-04T20:16:25.9202119Z caller=compactor.go:134 msg="compacting block" block="&{Version:v1 BlockID:5a8fbcbd-4eac-43bd-b326-58c206a38dcb MinID:[0 0 0 0 0 0 0 0 0 5 132 24 18 249 155 62] MaxID:[0 0 0 0 0 0 0 0 255 255 251 227 146 118 43 80] TenantID:single-tenant StartTime:2021-02-04 20:07:22.9653326 +0000 UTC EndTime:2021-02-04 20:12:22.802562 +0000 UTC TotalObjects:37565 CompactionLevel:1 Encoding:snappy}"
level=info ts=2021-02-04T20:16:31.3034375Z caller=compactor.go:126 msg="compaction complete"
level=error ts=2021-02-04T20:16:31.3035326Z caller=compactor.go:100 msg="error during compaction cycle" err="error iterating through object in backend: snappy: corrupt input"
level=info ts=2021-02-04T20:16:31.3036022Z caller=compactor.go:87 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant

[joe-elliott]

Getting some errors running locally ...

I have tested snappy, gzip and a mix of snappy and gzip blocks locally without errors. I'm using the local backend on my laptop. I have not yet tried a mix of v0 and v1 blocks. I will continue digging, but if you find some steps to repro, please share.

[mdisibio]

Getting some errors running locally ...

I have tested snappy, gzip and a mix of snappy and gzip blocks locally without errors. I'm using the local backend on my laptop. I have not yet tried a mix of v0 and v1 blocks. I will continue digging, but if you find some steps to repro, please share.

✔️ It is working now with both snappy and lz4-256K. I'm thinking this commit fixed it, since the input error happened towards the end of the compaction cycle, maybe the last flushes were bad: Pass tracker correctly. f853fb2

[mdisibio]

Could these branches reuse At(i) ?