feat: increase main_storage_proof_size_soft_limit

[pugachAG]

This PR changes storage proof soft size limit from 3mb to 4mb. Soft limits defines the value after which we stop executing receipts and move them to the delayed queue.
This change addresses the issues on mainnet where we have up to 12% of chunks hitting the size limit. It is worse on shard 5 where it reaches 25%. With this PR we decrease the number of such chunks by ~70%.
This change potentially increases the max possible size of state witness, but in practice compressed storage proof doesn't increase much with the current mainnet traffic (experiment is 4mb limit and update_shard is 3mb):

Also see zulip thread for more context.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.52%. Comparing base (0e5e7c7) to head (442e7d4).

Files with missing lines	Patch %	Lines
core/primitives-core/src/version.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11997      +/-   ##
==========================================
- Coverage   71.53%   71.52%   -0.02%     
==========================================
  Files         814      814              
  Lines      164424   164425       +1     
  Branches   164424   164425       +1     
==========================================
- Hits       117623   117597      -26     
- Misses      41640    41662      +22     
- Partials     5161     5166       +5     

Flag	Coverage Δ
backward-compatibility	0.17% <0.00%> (ø)
db-migration	0.17% <0.00%> (ø)
genesis-check	1.28% <0.00%> (-0.01%)	⬇️
integration-tests	38.58% <0.00%> (-0.01%)	⬇️
linux	71.21% <0.00%> (-0.02%)	⬇️
linux-nightly	71.11% <0.00%> (-0.01%)	⬇️
macos	54.07% <0.00%> (-0.01%)	⬇️
pytests	1.54% <0.00%> (-0.01%)	⬇️
sanity-checks	1.35% <0.00%> (-0.01%)	⬇️
unittests	65.41% <0.00%> (-0.02%)	⬇️
upgradability	0.21% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jancionear]

LGTM

There's an integration test for soft size limits, I was surprised that it didn't fail:

nearcore/integration-tests/src/tests/client/features/storage_proof_size_limit.rs

Line 131 in 51e4728

// Spawn 3 transactions, each reading 2MB of data. The first two should end up in the same chunk.

I see that it submits multiple receipts that read 2MB of state and expects the limit to be hit after 2 receipts are executed. That probably didn't change when the limit changed, with a 4MB limit it might also stop processing things after two receipts. It could be good to make the test more sensistive, maybe submit receipts that read 100kB of state instead of 2MB?

[pugachAG]

@jancionear thanks for pointing out the test, it actually helped me to discover that I've forgotten to update stable version: 9b94e67

[jancionear]

@jancionear thanks for pointing out the test, it actually helped me to discover that I've forgotten to update stable version: 9b94e67

Ah right, missed that too. You'll also probably need to update the hashes generated by cargo insta (?)

[walnut-the-cat]

Aug 30th status report

• Code complete, but faced a slight issue during merging the feature in. Investigation in progress with SRE team. ref