Add WAL Checksums

[joe-elliott]

Is your feature request related to a problem? Please describe.
Currently our only indication that there is corruption in the WAL is that proto fails to Unmarshal. Add a checksum here so we can more robustly detect corruption issues.

Describe the solution you'd like

• Add per record checksums
• Verify checksums on replay. Research whether a single corrupt record can be skipped
• Version the WAL file for future changes

Additional context
Unfortunately this may require versioning and abandoning previous WAL files on upgrade. We need to determine if there's anyway to handle the old WAL file format and the new. Given that we're still in 0.x I'm ok with the upgrade including a dropped WAL as long as this is documented as well as ways to preserve data.

[fitzoh]

Add per record checksums

Just to clarify, this would be serialized as part of an Object, right?

And the layout here would maybe go from
| total length (uint32) | id length (uint32) | id (variable) | object bytes (variable) |
to something like
| total length (uint32) | id length (uint32) | crc checksum (uint32) | id (variable) | object bytes (variable) |

We need to determine if there's anyway to handle the old WAL file format and the new.

This feels like it might be a little gross, but would it make sense to maintain multiple WAL format parser?
Thinking something like this:

var LatestObjectBinaryFormat = objectBinaryFormatV1{}

type ObjectBinaryFormat interface {
	MarshalObjectToWriter(id ID, b []byte, w io.Writer) (int, error)
	UnmarshalObjectFromReader(r io.Reader) (ID, []byte, error)
	UnmarshalAndAdvanceBuffer(buffer []byte) ([]byte, ID, []byte, error)
}

// | total length (uint32) | id length (uint32) | id (variable) | object bytes (variable) |
type objectBinaryFormatV0 struct {}
// | total length (uint32) | id length (uint32) | crc checksum (uint32) | id (variable) | object bytes (variable) |
type objectBinaryFormatV1 struct {}

Finally, any thoughts on how to manage the WAL version? Encode in the filename? Magic bytes at the start of the file? Something else?

[joe-elliott]

Just to clarify, this would be serialized as part of an Object, right?

Yes, I believe per object is the right way to go.

This feels like it might be a little gross, but would it make sense to maintain multiple WAL format parser?

Yup, we are heading this way with the blocks as well: multiple parsers supporting multiple formats. These changes will be impacted by the refactoring done to support multiple backend blocks so this issue will probably be delayed until we get versioned blocks done.

Finally, any thoughts on how to manage the WAL version? Encode in the filename? Magic bytes at the start of the file? Something else?

I was thinking of a brief header would indicate the WAL version and other important information. If the header is absent then assume it's a "v0" block.

[joe-elliott]

#638 has given us the ability to have wal checksums by using compression. Considering this fixed with that PR.