move paras inherent filtering to runtime #4028

drahnr · 2021-10-06T16:28:45Z

add filter methods for bitfields and candidates
provide randomness via inherent, and use it for candidate selection
add tests
~~remove duplicate provisioner checks~~ defered to a separate PR
Ref Refactor inherent filtering to be contained completely in the runtime #3989

Contains changes of #3924

follow up work:

runtime/parachains/src/inclusion.rs

primitives/src/v1/mod.rs

runtime/parachains/src/paras_inherent.rs

rphmeier

Approach so far looks good. A few comments:

This changes the runtime flow substantially, as it relates to bitfields. The implementers guide needs to be changed
These filtering processes should be run off-chain (fn create_inherent) to avoid providing useless data to the on-chain enter
I have the feeling that on-chain, we shouldn't silently drop data but should instead have the inherent fail. The block would be carrying around more data than it'd need to. However, that would mean that if we have a bug in create_inherent it could lead to validators being unable to author a block. So for now, something more relaxed seems better, although we should have some kind of warning if data does need to be skipped.
When providing randomness via an inherent, it needs to be backwards compatible. So nodes that don't provide the randomness should still be able to invoke create_inherent successfully.

This also relates heavily to #4020, so we should consider how to combine those. I suggest the following:

off-chain, an initial filter to remove data without heavy signature checks based on anticipated weights. In particular, this could limit disputes to a set amount, prioritizing disputes which have enough votes to conclude and have not already concluded. That can be done in Refactor runtime fn enter(...) to select disputes, bitfields and backing candidates to protect against over length block issues #4020, outside of this.
off-chain, a secondary filter to remove invalid data based on concluded disputes and the scheduler module

This will have the potential to limit more than necessary, but in general it shouldn't trigger as long as our weights are sensible.

I also filed #4055 because this runs off-chain and then enter is called immediately afterwards.

bkchr · 2021-10-12T10:36:07Z

3\. I have the feeling that on-chain, we shouldn't silently drop data but should instead have the inherent fail.

FYI, if an inherent fails, the entire block is invalid and the author will stop with producing this block.

rphmeier · 2021-10-12T12:16:27Z

Yeah, that is why I wrote the following:

However, that would mean that if we have a bug in create_inherent it could lead to validators being unable to author a block. So for now, something more relaxed seems better, although we should have some kind of warning if data does need to be skipped.

The warning would be on-chain and it would either alert us to an attack or a bug in create_inherent

primitives/src/v1/mod.rs

runtime/parachains/src/paras_inherent.rs

drahnr · 2021-11-16T09:14:26Z

runtime/parachains/src/paras_inherent.rs

 			let parent_hash = <frame_system::Pallet<T>>::parent_hash();
+			// Check that the submitted parent header indeed corresponds to the previous block hash.
 			ensure!(
 				parent_header.hash().as_ref() == parent_hash.as_ref(),


This is one of the basics and will remain checked in the provisioningsubsystem. This would indicate a bug rather anything else iiuc. So in reference to the above statement, this is fine.

runtime/parachains/src/paras_inherent.rs

…d-inherent-filtering

* comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest

emostov · 2021-11-16T18:39:29Z

bot merge

* master: (38 commits) Replicate Rob's PR (#4337) Companion for Taskmanager: Remove `clean_shutdown` (#4336) prefer code upgrades in inherent filtering (#4334) remove provisioner checks (#4254) Log para inherent inputs (#4331) Dispute spam protection (#4134) Dependabot: Ignore sub-tokens (#4328) export hrmp config (#4324) Add missing license header (#4321) Use non-empty validation code (#4322) fix pallet-xcm extrinsic doc comments (#4317) prepare worker: Catch unexpected unwinds (#4304) Enable BEEFY explicitly (#4320) Bump serde_json from 1.0.70 to 1.0.71 (#4316) Bump strum from 0.22.0 to 0.23.0 (#4308) Remove sort_unstable_by (#4314) Bump tokio from 1.13.0 to 1.14.0 (#4298) Substrate companion: Authority discovery multiple peer ids (#4295) Companion for substrate#9878 (#3949) move paras inherent filtering to runtime (#4028) ...

* move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com> Co-authored-by: Zeke Mostov <z.mostov@gmail.com> Co-authored-by: Bastian Köcher <info@kchr.de>

* move paras inherent filtering to runtime (#4028) * move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <rphmeier@gmail.com> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com> Co-authored-by: Zeke Mostov <z.mostov@gmail.com> Co-authored-by: Bastian Köcher <info@kchr.de> * prefer code upgrades in inherent filtering (#4334) * impl prefered items Closes #4330 * do not stop attempting to select, just because one did not fit * doc * prefered -> preferred * missing usage of the preferred indices * sigh * shuffle is not available for chacha * remove duplicate weight addition * ref vs no ref * add additional assurances to `create_inherent` (#4349) * minor: move checks into separate fn * add additional validity checks * simplify shuffling * Closes potential OOB weight * improve docs * fooo * remove obsolete comment * move filtering into the rollback-transaction Technically this is not necessary but avoids future footguns. * move check up and avoid duplicate checks * refactor: make sure backed candidates are sane, even more * doc wording Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * refactor: avoid const generics for sake of wasm size `true` -> `FullCheck::Skip`, `false` -> `FullCheck::Yes`. * chore: unify `CandidateCheckContext` instance names * refactor: introduce `IndexedRetain` for `Vec<T>` * chore: make tests prefix free * doc: re-introduce removed comment * refactor: remove another const generic to save some wasm size Co-authored-by: Zeke Mostov <z.mostov@gmail.com> * Inherent filtering follow up (#4305) * Add feature more feature gating for benchmarking + tests * New line * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs * Do not assume we use max validators per core * Use kusama weights for rococo (hopefully temp) * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras_inherent.rs * Add more validity votes when neccesary * Some fixes for the last commit * Restore westend weights * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras_inherent.rs * Revert bad westend weights write * Make sure to update val idx before skipping * Fix validity vote range to max at group size' * Temp setup for rococo * cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_paras_inherent.rs --header=./file_header.txt * Augment generated Rococo weights * Make it compile * Revert range for enter_backed_candidates_variable * Delete runtime/kusama/src/weights/runtime_paras_paras_inherent.rs Co-authored-by: Parity Bot <admin@parity.io> * prepare worker: Catch unexpected unwinds (#4304) * prepare worker: Catch unexpected unwinds * Use more specific wording for unknown panic payload * Treat non-deterministic prep errors as internal errors (#4364) Closes #4293 This PR changes the way how we treat a certain subset of PVF preparation errors. Specifically, now only the deterministic errors are treated as invalid candidates. That is, the errors that are easily attributable to either the the PVF contents or the wasmtime code, but not e.g. I/O errors that could be triggered by the OS (insufficient memory, disk failure, too much load, etc). The latter are treated as internal errors and thus do not trigger the disputes. Co-authored-by: Robert Habermeier <rphmeier@gmail.com> Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Lldenaurois <Ljdenaurois@gmail.com> Co-authored-by: Zeke Mostov <z.mostov@gmail.com> Co-authored-by: Bastian Köcher <info@kchr.de> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Sergei Shulepov <sergei@parity.io>

drahnr added A3-in_progress Pull request is in progress. No review needed at this stage. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. labels Oct 6, 2021

drahnr self-assigned this Oct 6, 2021

drahnr added the D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed. label Oct 6, 2021

drahnr changed the title ~~move inherent filtering to runtime~~ WIP move inherent filtering to runtime Oct 6, 2021

drahnr force-pushed the bernhard-inherent-filtering branch from 0ff0988 to 6abfab3 Compare October 6, 2021 20:14

move things around, add filter methods��

87b6cde

drahnr force-pushed the bernhard-inherent-filtering branch from e412104 to 87b6cde Compare October 8, 2021 12:16

validator keys, modify availability bitfields according to disputes

bf1f98b

rphmeier mentioned this pull request Oct 11, 2021

Refactor runtime fn enter(...) to select disputes, bitfields and backing candidates to protect against over length block issues #4020

Closed