BlockID: Consider ways to remove PartsHeader

[ebuchman]

When a proposer proposes a block, they serialize the block, split it into equal sized chunks (we call them "parts"), and gossip those parts. The BlockID is then defined as the normal block header hash, plus a PartsHeader, which includes the number of parts and their Merkle root. The PartsHeader is a commitment to this gossip mechanism, and is signed in both the proposal and the votes as part of the BlockID. See BlockID.

Outside the real-time consensus protocol (ie. for someone who just wants the blockchain like a light client), the PartsHeader is pretty useless, so it's inclusion is a bit of an unfortunate artifact. To remove it would require some change to the mechanism in the consensus reactor for gossipping block parts. Would be really nice to have this separation, and reduce the BlockID to just the block header hash, but need to be careful doing it.

One possible idea is to chunk the block using it's existing structure. So rather than serializing into bytes and chunking arbitrarily, we'd have the header be one chunk, the transactions be a set of chunks but chunked using their existing merkle tree structure, and so on. Then the main block header hash could be still be used for secure gossipping of chunks.

[ValarDragon]

There are significantly more efficient ways to gossip tx data then just plain chunking the txs. (Using erasure codes) It would be ideal if these changes can be done with only breaks to p2p version, not block version.

I believe this can be done. If we do something along the lines of https://github.com/tendermint/tendermint/issues/2530, you can just throw in into that MT, which the light clients don't need / don't pay for, but is available to the validators.

[musalbas]

Instead of have a consensus-critical Merkle root of transactions, why not just have a consensus-critical Merkle root of fixed-sized erasure coded data chunks? Light clients can still efficiently get proofs of transaction inclusion from them, but there's slightly more overhead. See section 4.3 in https://arxiv.org/pdf/1809.09044.pdf for an example of this.

[ebuchman]

Absolutely could consider that. The whole header needs a rework, see also https://github.com/tendermint/tendermint/issues/2530 and tendermint/tendermint#1007

[cmwaters]

I'm going to close this and link it to the more recent issue described here