make control-plane and data-plane wire formats portable

[rafaelcr]

The SIP-003 and SIP-005 control-plane and data-plane wire formats were manually implemented in the Stacks node using plain Rust code on top of the node's custom HTTP client.

While this works well for the purposes of the Stacks network, it makes it very hard for other developers to build apps that want to inspect, use, or build on top of these messages (for example, see the stacks-scout project). In order to do this, there's a lot of manual work involved to write extensive (and in many cases boilerplate) code for serialization and de-serialization, as well as manual debugging to make sure they match the current Stacks node implementation (which is not 100% reflected in SIP-005 and SIP-003).

I suggest a better approach would be to refactor this code to make the wire formats portable, preferably using a widely-available and open source standard such as:

• Protocol Buffers
• Cap'n Proto
• FlatBuffers
• ...or any other standard

This would make a few things much easier:

• Generating client code for these formats in a long list of popular programming languages
• Versioning the formats to keep track of new fields (new tx types, etc.)
• Maintaining this code in the Stacks node repo
• Building apps to inspect the mempool, the global Stacks network, etc.

[jcnelson]

I suggest a better approach would be to refactor this code to make the wire formats portable, preferably using a widely-available and open source standard such as:

Not happening. A canonical, parsimonious binary representation and codec for chain data that is 100% guaranteed to be both backwards and forwards compatible with all correct node implementations is a core requirement of a blockchain. This is not something you can afford to outsource -- one careless misstep in a 3rd party codec library would spell disaster for us.

Generating client code for these formats in a long list of popular programming languages

We already have tier-1 support in Rust and Javascript (TypeScript). Are there others for which there is a pressing need? Perhaps the Foundation can create a grant for them?

Versioning the formats to keep track of new fields (new tx types, etc.)

We seem to be doing this without difficulty.

Maintaining this code in the Stacks node repo

The ongoing maintenance burden for the codec is actually very minimal. In fact, once a codec is written and used in a consensus-critical code path, it cannot be modified (so no further work may be done on it). The bulk of ongoing codec work is in adding support for new messages, which would have to happen anyway regardless of library choice.

Building apps to inspect the mempool, the global Stacks network, etc.

I think it's a good idea to split the codec logic for Stacks data into its own crate, if that would unblock any of these tasks.

[kantai]

Overall, my personal rationale for using something like protobufs in particular is that a protobuf definition provides a declarative specification for a protocol which is machine-checkable and can be used for code and test generation. Our current serialization is procedural, specified in code (I am aware that there is a SIP with a spec, but it is out-of-date and not complete, which speaks to the maintenance difficulty here), and cannot be used for test generation.

This is not something you can afford to outsource -- one careless misstep in a 3rd party codec library would spell disaster for us.

I disagree with this on two levels. First, libraries like protobufs (which is less a library, and more a code generator) are specifically designed with this problem in mind: the best practice for using protobufs is to commit generated code. Second, accidental modification of consensus serialization paths in the current codebase are trivial, and I think that, in practice, the likelihood of a serialization bug in the stacks-blockchain codebase is in actuality much higher than a bug in a serialization library (at least one which would impact our serializations).

Versioning the formats to keep track of new fields (new tx types, etc.)
We seem to be doing this without difficulty.

I think this is actually enormously difficult in the current codebase -- you can look at the STXBalance serialization for an example. Part of the difficulty is that "consensus serialization" logic is peppered throughout the codebase and often tied to struct definitions. It's also hard to evaluate the downsides to the current serialization schemes -- maybe we do okay with versioning because we go to extreme lengths to avoid altering serializations.

Generating client code for these formats in a long list of popular programming languages
We already have tier-1 support in Rust and Javascript (TypeScript). Are there others for which there is a pressing need? Perhaps the Foundation can create a grant for them?

Yes -- hardware wallet "apps" like the stacks app on Ledger need these serializations. https://github.com/Zondax/ledger-stacks has its own rust implementation for deserialization. The fact that this could not use the stacks-blockchain repo's implementation as a library indicates that tier-1 support may not be in place.

The ongoing maintenance burden for the codec is actually very minimal. In fact, once a codec is written and used in a consensus-critical code path, it cannot be modified (so no further work may be done on it). The bulk of ongoing codec work is in adding support for new messages, which would have to happen anyway regardless of library choice.

I disagree, for many of the reasons I state above. Adding new types for serialization in the codebase is difficult and dangerous. The trait StacksMessageCodec is used for network messages (actually not consensus critical) but also the consensus critical serializations. This makes adding new StacksMessageCodec implementations very tempting because they're used by the network code. But now, the codebase has introduced something which is maybe consensus critical and maybe not. How is a later developer supposed to know which implementations are consensus critical and which are not? Using something like protobufs doesn't by itself solve this, but rethinking how the blockchain codebase employs serialization would help. For an example of this dilemma, you can look at the PR #3663

[jcnelson]

First, libraries like protobufs (which is less a library, and more a code generator) are specifically designed with this problem in mind: the best practice for using protobufs is to commit generated code

With respect to blocks and transactions, the problem here is that the byte-level representation is part of our protocol's interface. It's not enough that an auto-generated codec can encode and decode bytes in a forward-compatible manner; we specifically need the bytes themselves to be stable. Because the value-add of declarative protocol languages is that the developer does not see the underlying bytes and does not have to deal with them, their API contracts may permit the creation of different byte representations of the same structured data, so long as their auto-generated codecs are forward-compatible with it. This is acceptable for network messages, but unacceptable for consensus data. So I would be firmly against using a 3rd party library for encoding/decoding consensus data.

Second, accidental modification of consensus serialization paths in the current codebase are trivial, and I think that, in practice, the likelihood of a serialization bug in the stacks-blockchain codebase is in actuality much higher than a bug in a serialization library

While this is true in principle, we can substantially de-risk accidental modification by placing both the codec helper functions as well as all implementations of StacksMessageCodec into their own crate(s). Then, we can make it so changes to these crates are automatically flagged for an extra-careful set of reviews.

I think this is actually enormously difficult in the current codebase -- you can look at the STXBalance serialization for an example

I disagree. The reason STXBalance was challenging was because we never expected this struct to be versioned in the first place, and now we have to back-track and add it in. If we forgot to include versioning data in the protocol language definition, we'd be in a similar situation. Furthermore, even if auto-generating the codecs for a versioned piece of structured data is easy, it does not free you from implementing and testing all of the epoch-gating that will be required for it to be used (which I think is the harder of the two tasks by far).

I also think that it's a feature that it's similarly hard to change the versions of p2p messages. We don't want to make it too easy to accidentally introduce a network partition due to some peers not knowing how to talk to other peers. In fact, the p2p system has a separate services bitfield in the Preamble struct in order to help peers determine which message format(s) the remote peer will expect. Again, a protocol definition language does not help us here -- we'd have to do this work either way.

Yes -- hardware wallet "apps" like the stacks app on Ledger need these serializations

I'd love to know more about what difficulty they faced. They couldn't just crib the requisite code from our codebase? Do you know what stopped them?

Adding new types for serialization in the codebase is difficult and dangerous. The trait StacksMessageCodec is used for network messages (actually not consensus critical) but also the consensus critical serializations.

While it's not universally the case, the benefit of using StacksMessageCodec (or even serde) is that additional, message-specific validation can occur as part of deserialization. For example, consider the codec for StacksBlock -- its deserialization code checks a lot of bespoke things about the block in order to ensure that incorrectly-structured block data can never be decoded. Protocol buffers do not help us there; instead, we'd have to be sure to run these checks every time we ever decoded block data, which is extremely brittle and error-prone.

We do something similar for network messages -- we ensure that no more than _$BYTES_ data can be consumed from the read handle in order to plug DDoS vulnerabilities from badly-formed supersized messages.

[rafaelcr]

This is not something you can afford to outsource -- one careless misstep in a 3rd party codec library would spell disaster for us.

Hmm yeah, good point.

We already have tier-1 support in Rust and Javascript (TypeScript). Are there others for which there is a pressing need? Perhaps the Foundation can create a grant for them?

Ah, nice, perhaps we missed the TS support. Could you point me to where I can find it or learn more about it? A good idea could be to publish a client library outside of this repo that is easily accessible and maintainable so others can use it (e.g. published to npm). We could use the work we did for Stacks Scout, for example.

I'm not aware of active projects for other languages that might need them right now but it would be helpful to have a way to generate (or port) libraries for them in the future, if possible.

The ongoing maintenance burden for the codec is actually very minimal. In fact, once a codec is written and used in a consensus-critical code path, it cannot be modified (so no further work may be done on it). The bulk of ongoing codec work is in adding support for new messages, which would have to happen anyway regardless of library choice.

I see. The most difficult thing for us was to figure out the correct spec from Rust code and SIP text, but maybe a reasonable solution to that would be to maintain a document under this repo's docs/ that explains the message structures in detail, i.e. combine SIP-003 and SIP-005 + newer changes from the new tx types and use that moving forward.

[wileyj]

perhaps a good opportunity to improve documentation