Encode LogMsg using protobuf

[jprochazk]

Related

• Closes Replace our serde/MsgPack encoding of enum LogMsg with protobuf #7988
• Follow-up to Untangle re_protos dependencies #8381

What

This PR introduces Serializer::Protobuf to re_log_encoding, and inverts the dependency graph of re_protos, which no longer depends on other re_* crates. This meant the conversion impls from protobuf types to rerun types and back had to be moved into their respective crates. For example, From<StoreId> for re_protos::common::v0::StoreIdis now inre_log_types`.

When encoding a file using this serializer, the data is encoded using a combination of:

• A custom stream-level protocol
• Protocol buffers
• Arrow IPC

The stream-level protocol has changed only a bit, because compression is no longer done for all messages in the stream, and only the contents of ArrowMsg are ever compressed at all. This means the uncompressed_len and compressed_len could be unified to just len.

The actual layout of the messages has not changed, LogMsg is preserved and so are its semantics.

The stream of data stored in an example RRD file using this new encoding looks like:

FileHeader { b"RRIO", version, compression, serializer }     ;; 10 bytes
MessageHeader { kind, len }                                  ;; 8 bytes
SetStoreInfo { application_id, store_id, store_source, ... } ;; len bytes
MessageHeader { kind, len }                                  ;; 8 bytes
ArrowMsg { store_id, arrow_msg }                             ;; len bytes
MessageHeader { kind: End, len: 0 }                          ;; 8 bytes

Note that this stream-level protocol is only used for .rrd files. On the wire, we will use gRPC, which has its own protocol.

In the case of ArrowMsg, the schema+chunk is encoded using Arrow IPC into a byte payload, which may additionally be compressed. The compression setting is stored separately for every ArrowMsg, but per-message compression functionality is not yet exposed through re_log_encoding.

[github-actions]

Latest documentation preview deployed successfully.

Result	Commit	Link
✅	0d3ec40	https://landing-hx1w29tq8-rerun.vercel.app/docs

^{Note: This comment is updated whenever you push a commit.}

[github-actions]

Web viewer built successfully. If applicable, you should also test it:

• I have tested the web viewer

Result	Commit	Link	Manifest
✅	cd6ab2e	https://rerun.io/viewer/pr/8347	+nightly +main

^{Note: This comment is updated whenever you push a commit.}

[zehiko]

thanks for untangling these dependency issues!

generally looks ok to me, few comments and a question about more clear re-use of common arrow seriailzation logic to make re_log_encoding crate clearer.

[jprochazk]

could add a unit/integration test that uses new codec to write and read rrd stream using Protobuf? something similar-ish to test_loading_with_retryable_reader

This is now tested as part of the tests in re_log_encoding/src/decoder/mod.rs. I updated the fake log messages there to actually encode a more realistic scenario (blueprint that sets the background color, though I'm sure it's wrong as I haven't actually tested if it does that, it's just meant to be some data to encode/decode), to exercise encoding for all 3 LogMsg variants.

suggested adding a few basic unit tests in a few

Done for re_log_types and re_tuid. It doesn't exercise every possible path but all the major ones are hit in some way (e.g. the StoreSourceExtra).

[emilk]

I only had time for a partial review today - will take a closer look tomorrow!

[jprochazk]

I updated the benchmark to include protobuf variants of all the encode/decode parts. There are way too many benchmarks now, so I tried using ChatGPT to make sense of the results:

Note that it messed up units, and produced total gibberish when asked to fix it. They aren't all in milliseconds.

Results seem fairly mixed, some wins and some losses. The mono-points non-batched vs batched decode is interesting, as msgpack seems to be better at non-batched data? Encode seems to be a win across the board for protobuf, but only by a little.

[emilk]

👍

[emilk]

😬

Tip: add a // TODO suffix on code you plan to remove (the linter will stop you from merging it).

Or use dbg!(options); (again, this will not pass CI)

You can also just use re_log::trace and run with RUST_LOG=re_log_encoding=trace

[jprochazk]

I didn't plan to remove this, what's wrong with keeping a println in a test? It doesn't get printed unless the test fails, in which case it's useful information about which part of the test failed

[emilk]

Wait, what the hack is this @teh-cmc ??