Implement capnproto replication

[fpetkovski]

Our profiles from production show that a lot of CPU and memory in receivers is used for unmarshaling protobuf messages. Although it is not possible to change the remote-write format, we have the freedom to change the protocol used for replicating timeseries data.

This commit introduces a new feature in receivers where replication can be done using Cap'n Proto instead of gRPC + Protobuf. The advantage of the former protocol is that deserialization is far cheaper and fields can be accessed directly from the received message (byte slice) without allocating intermediate objects. There is an additional cost for serialization because we have to convert from Protobuf to the Cap'n proto format, but in our setup this still results in a net reduction in resource usage.

We have a split router-receiver setup and this is our resource usage in staging after enabling the new replication method:

Router usage did go up as well, but we still got an overall net reduction:

We can also experiment with other formats by having a more generic flag: receive.replication-format=...

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Verification

[GiedriusS]

Doesn't this include a whole new RPC framework? I'm very interested in this. Glad to finally see #7071 tackled. How do you think this compared to dRPC? And is it worth eschewing gRPC altogether?

[saswatamcode]

Really interested in this too! Our profiles show similar for unmarshaling

[fpetkovski]

Doesn't this include a whole new RPC framework? I'm very interested in this. Glad to finally see #7071 tackled. How do you think this compared to dRPC? And is it worth eschewing gRPC altogether?

Yes this comes with its own RPC framework which can be a double edge sword. It's one more thing to learn and troubleshoot if it goes wrong. I haven't looked into dRPC yet in practice, but maybe there is a way to abstract the RPC framework and not expose internals to end users, this way we can swap it out in future releases in a transparent manner. In this PR we expose the listen port as a parameter since we need separate connections for Cap'n Proto.

[squat]

DRPC allows serving both gRPC and DRPC on the same socket to allow for incrementally upgrading/migrating a fleet. We might consider doing something similar whether we go with capnproto or DRPC

[GiedriusS]

Mhm, we probably need to experiment more with the transparent approach before deciding what to do. For example, we could maybe deprecate --grpc* command line parameters and have a generic listener for RPCs. Perhaps let's just continue as is and mark this as experimental everywhere?

My only suggestion would be:

          {"address": "node-1:10901", "capnproto_address": "node-1:19391"},

Instead of having two addresses here, let's have "rpc_protocol": "grpc/capnproto" or something like that. I think it would be more future-proof with regard to transparent switching between RPC protocols on the receiving end.

[fpetkovski]

Perhaps we can use something like https://github.com/soheilhy/cmux to multiplex multiple protocols on the same port. This way we don't need to pollute arguments with flags for each protocol. Let me see if this is easily doable, and also add some tests for the new RPC.

[GiedriusS]

Adding e2e test: fpetkovski#7

[pedro-stanaka]

lg

[GiedriusS]

This gives me:

    /home/giedriusstatkevicius/dev/thanos/pkg/receive/writecapnp/marshal_bench_test.go:101: 
        	Error Trace:	/home/giedriusstatkevicius/dev/thanos/pkg/receive/writecapnp/marshal_bench_test.go:101
        	            				/usr/local/go/src/testing/benchmark.go:193
        	            				/usr/local/go/src/testing/benchmark.go:215
        	            				/usr/local/go/src/runtime/asm_amd64.s:1700
        	Error:      	Received unexpected error:
        	            	unmarshal: short header section

Does this benchmark work for you?

[fpetkovski]

Looks like we are unmarshaling proto bytes into a capnproto request. I've fixed the benchmark, all of them should be passing now.

[GiedriusS]

make docs failing 😬

[fpetkovski]

Rebased on main and regenerated docs. We should be good to go!

We can keep iterating over time, I'd like to see how we can leverage interning in the future.