(v2) indexed unchunked files

[wkalt]

It should be possible to produce an MCAP file with all the crash safety guarantees of an unchunked file, as well as all the indexing features of a chunked file.

A simple append-only way to do it would be to introduce a concept like a "batch", with "batch start" and "batch end" records. The difference between a batch and a chunk is the length of the batch isn't known by the writer ahead of time. All the other indexing features remain the same as for chunks.

[wkalt]

From a UX point of view, this makes "unchunked files" just as introspectable as chunked files. Currently if you used unchunked files you give up most of the indexing features and most of the output in "mcap info".

[james-rms]

Once this is in, do you feel like we could deprecate writing MCAP with no chunks in v2?

[wkalt]

I could imagine scenarios where you still want to record an unindexed bag if the overhead of the message indexes is not tolerable (e.g really small messages). In that case these records wouldn't make sense.

[james-rms]

in a future where using message indexes is not recommended (i would not recommend it now), does that change anything?

[wkalt]

it would definitely make the bloat less of a big deal. it would still seem weird to have these markers that don't really do anything, though.

One possible way to fix that problem would be to give these markers a bit of extra power, by combining them with a "sync marker" concept similar to what Avro has. You could actually replace the start/end markers with a single sync marker record.

Sync Record
data [16]byte

The sync data would be a random byte string by default, generated on instantiation of the writer. It must not occur anywhere in the file naturally. The user must also be able to supply it as config in case they require deterministic outputs. It would also be recorded in the header record.

Sync records are recorded before every chunk or in the locations of the start/end markers for the unchunked file. This allows indexes to be built on them, but also enable two further benefits:

• readers can split a file without parsing it, and then parallelize the processing
• recovery after a crash only needs to scan backward to the last intact sync record. It doesn't need to scan a whole file.

I think those benefits would be enough to make inclusion of this concept in unchunked files not seem weird.

[wkalt]

also worth noting - this isn't a breaking change so it's not really V2. It is just marked V2 because it's so next-gen.