Increase max_pov_size to 10MB

[sandreim]

Currently we run relay chain with a max PoV size of 5MB on Polkadot/Kusama. We've recently discovered during benchmarking that the storage proof overhead increases significantly with the number of keys in storage such that the parachain throughput drops to 50% with 1 mil accounts for example.

Based on the numbers in #4399 we should be able to double the max pov size and still only require 50% of hw spec bandwidth at worst case.
I'd expect the CPU cost to increase to 100% at worst for the erasure encoding/decoding, we should determine it using subsystem benchmarks and see how it fits with the upcoming new hw specs.

CC @eskimor @burdges

[burdges]

I do not have strong feelings on increasing PoV sizes near-term, but certianly if we put gluttons on kusama then it's nice to be able to pressure them properly.

We've recently discovered during benchmarking that the storage proof overhead increases significantly with the number of keys in storage such that the parachain throughput drops to 50% with 1 mil accounts for example.

Yes, we make storage proofs 4x larger than necessarily by inherreting the stupid radix 16 trie from ETH.

[s0me0ne-unkn0wn]

So, a rough roadmap could be like that. In the next SDK release, we bump MAX_POV_SIZE to 10 Mb. Runtimes stay with their old 5 Mb limit compiled in, and they will not allow to build a block longer than 5 Mb. We're starting to prepare runtimes based on that new release while people upgrade nodes. When the supermajority upgrades, we're ready to enact new runtimes. After the runtimes are enacted, we get parablocks with 5 Mb proofs (10 Mb / 2) limited by collators (so all the 5 Mb may be used by user transactions without 25% reservation). Then, we can lift that halving limitation pointwise where we need it.

The other option is to make the proof size configurable, but I'm not sure how much sense it makes to do that, given that it's a complication anyway and the event of changing this constant is so rare.

CC @dmitry-markin any concerns from the networking layer side?

CC @eskimor @bkchr @skunert

Feel free to CC other relevant people.

[bkchr]

Maximum code size is already controlled by the host configuration:

polkadot-sdk/polkadot/runtime/parachains/src/configuration.rs

Line 73 in 717bbb2

pub max_code_size: u32,

I strongly hope that we have not hard coded anywhere this 5MiB limit.

[s0me0ne-unkn0wn]

I strongly hope that we have not hard coded anywhere this 5MiB limit.

As I see from the code, the MAX_POV_SIZE constant in primitives rules everything. Do you think bringing it to the configuration makes sense? I mean, if we want to change it every six months, then it makes sense definitely, but now we're seeing it as a big move and a one-shot change.

[sandreim]

The MAX_POV_SIZE constant is also present in req/response protocols config.

I think we should remove all the constant references and use the configuration value.

In the future we might want to raise it above 10MB, I don't see any reason why not if the validators have enough bandwidth and CPU. We'd still be capped at 16MB on the parachain maximum response size of the block request protocol:

polkadot-sdk/substrate/client/network/sync/src/block_request_handler.rs

Line 92 in 717bbb2

16 * 1024 * 1024,

[bkchr]

Do you think bringing it to the configuration makes sense?

Otherwise there is no consensus on this number. Which leads to the situation that we need to please all validators to upgrade. This is not really how a decentralized network should work. Using some upper number on the node makes sense, but it should give some leeway to the on chain value.

[alexggh]

Do you think bringing it to the configuration makes sense?

Otherwise there is no consensus on this number. Which leads to the situation that we need to please all validators to upgrade. This is not really how a decentralized network should work. Using some upper number on the node makes sense, but it should give some leeway to the on chain value.

The runtime set code does something like this, so I guess that was the leeway which we now want to increase.

if self.max_pov_size > MAX_POV_SIZE {
     return Err(MaxPovSizeExceedHardLimit { max_pov_size: self.max_pov_size })
}

[eskimor]

There are two values here:

1. The limit on networking req/response protocols.
2. The actual runtime configuration.

(1) poses an upper limit on (2). Yes ideally we would derive the network limit from the runtime configuration, but that would require some refactoring as we currently configure that value on node startup, when setting up the protocol.

It is worth double checking that (2) is correctly used everywhere in validation and not the constant. Other than that the process is as follows:

1. Bump the networking limit ... this is backwards compatible and can be back ported.
2. Once enough validators have upgraded we can change the runtime configuration.

Assuming we use the runtime configuration correctly (and persisted validation data, which should be derived from it), there can be no consensus issue. We nevertheless must have the majority of validators upgraded to the higher networking limit, otherwise honest nodes would not be able to fetch a large PoV, which could cause a finality stall.

[eskimor]

Do you think bringing it to the configuration makes sense?

Otherwise there is no consensus on this number. Which leads to the situation that we need to please all validators to upgrade. This is not really how a decentralized network should work. Using some upper number on the node makes sense, but it should give some leeway to the on chain value.

The runtime set code does something like this, so I guess that was the leeway which we now want to increase.

if self.max_pov_size > MAX_POV_SIZE {
     return Err(MaxPovSizeExceedHardLimit { max_pov_size: self.max_pov_size })
}

Indeed. First the node side limit, then the runtime value.

[dmitry-markin]

One thing regarding the networking req/resp protocols limit to keep in mind, is that the block response limit was set for minimal supported network bandwidth. I.e., it should not time out even on the slowest supported connections.

I don't have the numbers at hand, but the 16 MB block response should not time out in 20 seconds, so the minimum bandwidth is presumably 1 MB/sec in the spec. If we raise this limit, we should also make sure we either increase the request-response protocol timeout (but this can introduce more latency with unresponsive peers) or raise the bandwidth requirements.

[alexggh]

From: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware

The minimum symmetric networking speed is set to 500 Mbit/s (= 62.5 MB/s). This is required to support a large number of parachains and allow for proper congestion control in busy network situations.

So at least from theoretical point of view we should have bandwidth for validators.

[dmitry-markin]

From: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware

The minimum symmetric networking speed is set to 500 Mbit/s (= 62.5 MB/s). This is required to support a large number of parachains and allow for proper congestion control in busy network situations.

So at least from theoretical point of view we should have bandwidth for validators.

Also, regular full nodes must be able to follow the chain.

[aaron2048]

+1 for this. Moonbeam has adopted Asynchronous backing and increased the EVM gas per block and we're running into difficulties due to this limitation.

[AndreiEres]

Moving comments from #5753

We should consider networking speed limitations if we want to increase the maximum PoV size to 10 MB. The current PoV request timeout is set to 1.2s to handle 5 consecutive requests during a 6s block. With the number of parallel requests set to 10, validators will need the following networking speeds:

• 5 MB PoV: at least 42 MB/s, ideally 50 MB/s.
• 10 MB PoV: at least 84 MB/s, ideally 100 MB/s.

The current required speed of 50 MB/s aligns with the 62.5 MB/s specified in the reference hardware requirements. Increasing the PoV size to 10 MB may require a higher networking speed. This is worst case scenario, when all blocks you need to recover are full.

[sandreim]

Moving comments from #5753

We should consider networking speed limitations if we want to increase the maximum PoV size to 10 MB. The current PoV request timeout is set to 1.2s to handle 5 consecutive requests during a 6s block.

With the number of parallel requests set to 10, validators will need the following networking speeds:
Where does this number come from ?

• 5 MB PoV: at least 42 MB/s, ideally 50 MB/s.

If we have around 6-7 approvals per validator, then we'd need to recover 5 * 7 = 35 MB per relay chain block, that is almost 6MB/s required to keep up with finality.

For backing, assuming there async backing limits (max_candidate_depth 3, max ancestry len 2), there should be at most 5 * 3 * 2 = 5 MB/s

Which means a total of lets say 11MB/s for all PoV fetches.

• 10 MB PoV: at least 84 MB/s, ideally 100 MB/s.

This would mean 22MB/s based on the above math which is well under the spec leaving room for all other gossip and relay chain block fetching.

[burdges]

We think 100MB/s = 1Gb/s has good odds of handling 15 approvals per valudators then? Or do we think there needs to be a lot more slack somehow?