chore: rewrite block payload

[ggutoski]

Closes #1076

This PR:

• Complete rewrite of everything in sequencer/src/block.
• Maintain serialization compatibility with current main branch, though it's janky. Search serialization compatibility in comment TODOs.

This PR does not:

• Only minimal tests for basic correctness in block/tests.rs. Deep testing for arbitrary block payloads is deferred to a future PR.

Key places to review:

• TODO in code comments. I've checked them all prior to posting this PR. Many are flags to request changes to upstream code such as TODO(QueryablePayload). Some are requests for feedback from reviewers.
• Anything outside module block. Especially NamespaceProofQueryData struct in endpoints.rs as discussed in zulip, plus knock-on changes in api.rs, nasty-client.rs.
• Proposal to move NamespaceId struct into ns_table.rs: chore: rewrite block payload #1499 (comment), plus a laundry list of complaints:

  espresso-sequencer/sequencer/src/transaction.rs

  Lines 9 to 28 in ab276e5

  	/// TODO [`NamespaceId`] has historical debt to repay:
  	/// - It must fit into 4 bytes in order to maintain serialization compatibility
  	/// for [`crate::block::NsTable`], yet it currently occupies a `u64`. Thus,
  	/// any code that creates a [`NamespaceId`] must ensure that it fits into 4
  	/// bytes, which is bug-prone. We should either (i) allow it to occupy 8
  	/// bytes, breaking serialization compatibility, or (ii) define it as a 4-byte
  	/// struct such as `u32`.
  	/// - We should move [`NamespaceId`] to `crate::block::full_payload::ns_table`
  	/// module because that's where it's byte length is dictated, so that's where
  	/// it makes the most sense to put serialization. See
  	/// <https://github.com/EspressoSystems/espresso-sequencer/pull/1499#issuecomment-2134065090>
  	/// - Don't expose a constructor that allows construction of an invalid
  	/// [`NamespaceId`]. (Example: caller could use [`From`] to get a
  	/// [`NamespaceId`] that occupies more than 4 bytes.) Instead I suggest we
  	/// allow construction only via serialization.
  	/// - It impls [`Namespace`] from [`jf_merkle_tree`], but this seems unneeded
  	/// now that we're not using jellyfish's namespace merkle tree.
  	/// - We derive lots of things that perhaps we shouldn't: `Into`, `From`,
  	/// `Default`, `Ord`. Perhaps derivations for [`NamespaceId`] should match
  	/// that of [`Transaction`].

• Should we allow duplicate namespace IDs in the namespace table? See ns_table.rs (search "duplicate"), especially NsIter struct. See also discussion in zulip
  • Current plan is to allow duplicate namespace IDs for now, but forbid it in the future.
• FYI detailed spec in rustdoc in ns_table.rs and types.rs. (Not that you need to read it!)

Top-down tour of code

Module: block

block.rs: Exposes an API to the rest of this workspace. Currently restricted only to

espresso-sequencer/sequencer/src/block.rs

Line 6 in 46bbe9a

pub use full_payload::{NsProof, NsTable, Payload};

I expect in the future we might also need to expose TxProof and possibly other items. The goal is to expose as little as possible outside block module.

block/full_payload.rs, block/namespace_payload.rs: An attempt to impose a hierarchy of isolation/abstraction within block so that it's not just a flat wall of types. (It's difficult to define a natural hierarchy; but this presentation seems not bad to me.)

• full_payload: Any code that needs to know the structure of a multi-namespace payload. Contains the all-important Payload struct (which impls BlockPayload, QueryablePayload traits) plus NsTable, NsProof, etc.
• namespace_payload: Self-contained implementation of an individual namespace. NsPayload plus TxProof, etc.

As usual, these modules expose as little as possible. Lots of use of pub(in crate::block) to restrict visibility within block module.

Isolation of low-level binary format

Low-level details of binary format is isolated as much as possible. All reads from a block payload must use the following 2 modules:

• block/full_payload/ns_table.rs. 3 constants define byte lengths of objects that appear in a namespace table:
  

  espresso-sequencer/sequencer/src/block/full_payload/ns_table.rs

  Lines 20 to 23 in 46bbe9a

  	/// Byte lengths for the different items that could appear in a namespace table.
  	const NUM_NSS_BYTE_LEN: usize = 4;
  	const NS_OFFSET_BYTE_LEN: usize = 4;
  	const NS_ID_BYTE_LEN: usize = 8;

• block/namespace_payload/types.rs (Open to suggestions for a better name.) Similar to the namespace table, 2 constants define byte lengths of objects that appear in a transaction table:
  

  espresso-sequencer/sequencer/src/block/namespace_payload/types.rs

  Lines 12 to 14 in 46bbe9a

  	/// Byte lengths for the different items that could appear in a tx table.
  	const NUM_TXS_BYTE_LEN: usize = 4;
  	const TX_OFFSET_BYTE_LEN: usize = 4;

Extensive use of newtypes to prevent bugs

There are many different items floating around: payload byte length, namespace byte length, number of transactions in a namespace, index of a transaction, index of a namespace, byte range for a namespace, byte range for a transaction, etc.

It's easy to accidentally confuse these quantities. Example: accidental use of payload byte length when you intend to use namespace byte length. I've made extensive use of newtypes to prevent nasty bugs that might arise from such accidental misuse.

Example: construct a TxIter

• TxIter::new has an arg of type NumTxs: a newtype wrapper for usize that indicates the number of transactions in a namespace.
• To get a NumTxs call NumTxs::new, which has args of type NumTxsUnchecked and NsPayloadByteLen.
• To get a NumTxsUnchecked call NsPayload::read_num_txs. This method extracts the declared number of transactions from the transaction table. A transaction table might contain corrupted data, so this quantity might cause a panic if used directly without checking. That's why you cannot construct a TxIter from a NumTxsUnchecked -> you must get a NumTxs.
• Similarly, you also need a NsPayloadByteLen to construct a NumTxs. You get that from NsPayload::byte_len or NsPayloadRange::byte_len. This newtype prevents you from accidentally using the payload byte len instead of the namespace byte len.
• Normally I would not bother to decompose this process into so many low-level steps because it adds complexity to the codebase. But TxProof demands many low-level pieces. Given that these pieces must exist for TxProof, it makes sense to create them, then build higher-level helpers from these low-level tools as needed.
  • Example: NsPayload::read_num_txs is actually a convenience helper that bundles several lower-level steps.

Modules full_payload, namespace_payload

Module full_payload is relatively simple because NsProof is not very demanding because the correctness of the namespace table is trusted as part of the block header.

Module namespace_payload is complex because TxProof is very demanding, so let's talk about the contents of namespace_payload:

• ns_payload.rs: Struct NsPayload for payload bytes for an individual namespace. No surprises here.
• ns_payload_range.rs: Newtype NsPayloadRange for Range<usize> that indicates a subslice of a Payload for an individual namespace. Lots of important metadata can be obtained from only this range with no need to have a full NsPayload. Indeed, in some contexts such as TxProof::verify a full NsPayload is not available so the only option is to rely on NsPayloadRange.
• types.rs: All low-level binary details of a namespace are isolated in this file. Most of the newtypes mentioned above depend on these low-level binary details, so most of those newtypes are also defined here.
• iter.rs: A public-facing iterator Iter over all transactions in a block payload. Under the hood it's a simple Cartesian product NsIter x TxIter.
  • NsIter is defined in ns_table.rs and can be accessed publicly via NsTable::iter.
  • TxIter is defined in types.rs and can be accessed via NsPayload::iter. But TxIter is not publicly accessible outside block.

Serialization for fixed-byte-length integers in uint_bytes.rs

The magic constants defined in ns_table.rs, types.rs indicate the byte length of various items in the payload.

Example: NUM_NSS_BYTE_LEN = 4 indicates that a namespace table shall use 4 bytes to encode the number of namespaces in the block. Perhaps in the future we'll decide that 3 bytes is sufficient to describe this quantity, as we expect the number of namespaces in a single block will never exceed 16 million. In that case, you can change NUM_NSS_BYTE_LEN from 4 to 3 and everything should work perfectly.

In order to make the code generic over these constants I had to write my own serialization functions in uint_bytes.rs. In pseudocode:

fn usize_to_bytes<const N: usize>(n: usize) -> [u8; N];
fn usize_from_bytes<const N: usize>(bytes: &[u8]) -> usize;

Most uses of this serialization are for a usize quantity, but NamespaceId serialization is u64. The only way to make these functions generic over primitive integer types usize, u64, etc is to put them in a macro uint_bytes_impl.

This file includes other helper functions, extensive rustdoc, and detailed tests. There's also a separate macro bytes_serde_impl for custom serde derivation to/from a fixed-length bytes array [u8; N].

Limitations

This code should work perfectly if all the magic constants are at most 8. Anything beyond 8 exceeds the byte length of usize, u64 and so you can expect build failure and/or startup panic if you try it.

If you want to accommodate sizes larger than 8 then you'll need to change the implementation of all affected types to use something other than usize, u64. I do not foresee any demand for sizes larger than 8 bytes: typical use is to serialize an integer quantity, and we do not expect any such quantity to exceed 64 bits.

A possible exception: NamespaceId is an identifier, not a quantity. In the future we might use 32 bytes for NamespaceId to allow a namespace ID to be a hash output. In that case NamespaceId will need modification and can no longer use the functions in uint_bytes.rs.

[ggutoski]

On further discussion with @jbearer , I need to investigate whether it's feasible to maintain serialization compatibility. Putting this PR back into draft mode for now.

[jbearer]

This is unbelievably well thought out. I applaud you.

Most of my comments are very minor and you should feel free to ignore them. The only really important one is the one on NamespaceId.

[jbearer]

Think we're good here 👍