Ingester block lifetime changes

[mdisibio]

What this PR does:
This PR redoes some of the block lifecycle on the ingester to improve stability and startup times.

Previously the ingester would convert the WAL into a CompleteBlock which persisted the data on disk but kept the index and bloom filters in memory. After a restart the blocks would have to be recreated from the WALs since memory was lost, which caused both heavy performance hit on startup, but also an inability to query data that had been flushed but no longer in the WAL.

Now the ingester uses a local backend mounted under /wal/blocks. The WAL is flushed to this local backend first, which fully persists the data/index/bloom, etc. Then it is flushed to the remote backend. On restart any existing blocks are simply reloaded and the ingester resumes where it was. The flushed state of a local block is persisted with a timestamp in a dedicated flushed file.

No configuration or hosting changes are needed. The local backend is mounted under /wal/blocks/ by reusing the existing wal storage location, and it also uses the existing block encoding/downsampling/flushing options.

Which issue(s) this PR fixes:
#633

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Have some initial thoughts. It makes sense to me to move the "LocalBlock" and related functionality inside the /tempodb/wal package and provide methods for the ingester/instance to use.

[joe-elliott]

Thank you for the changes. This is definitely getting closer.

Some mild refactoring comments, but other than that I'm really liking where this is.