pvf-precheck: Integrate PVF pre-checking into paras module (#4457)

* pvf-precheck: Integrate PVF pre-checking into paras module

Closes #4009

This is the most of the runtime-side change needed for #3211.

Here is how it works.

The PVF pre-checking can be triggered either by an upgrade or by
onboarding (i.e. calling `schedule_para_initialize`). The PVF
pre-checking process is identified by the PVF code hash that is being
voted on. If there is already PVF pre-checking process running, then no
new PVF pre-checking process will be started. Instead, we just subscribe
to the existing one.

If there is no PVF pre-checking process running but the PVF code hash
was already saved in the storage, that necessarily means (I invite the
reviewers to double-check this invariant) that the PVF already passed
pre-checking. This is equivalent to instant approving of the PVF.

The pre-checking process can be concluded either by obtaining a
supermajority or if it expires.

Each validator checks the list of PVFs available for voting. The vote is
binary, i.e. accept or reject a given PVF. As soon as the supermajority
of votes are collected for one of the sides of the vote, the voting is
concluded in that direction and the effects of the voting are enacted.

Only validators from the active set can participate in the vote. The set
of active validators can change each session. That's why we reset the
votes each session. A voting that observed a certain number of sessions
will be rejected.

The effects of the PVF accepting depend on the operations requested it:

1. All onboardings subscribed to the approved PVF pre-checking process will
get scheduled and after passing 2 session boundaries they will be onboarded.
2. All upgrades subscribed to the approved PVF pre-checking process will
get scheduled very similarly to the existing process. Upgrades with
pre-checking are really the same process that is just delayed by the
time required for pre-checking voting. In case of instant approval the
mechanism is exactly the same. This is important from parachains
compatibility standpoint since following the delayed upgrade requires
the parachain to implement
paritytech/cumulus#517.

In case, PVF pre-checking process was concluded with rejection, then all
the requesting operations get cancelled. For onboarding it means it gets
without movement: the lifecycle of such parachain is terminated on the
`Onboarding` state and after rejection the lifecycle is none. That in
turn means that the caller can attempt registering the parachain once
more. For upgrading it means that the upgrade process is aborted: that
flashes go-ahead signal with `Abort` flag.

Rejection leads to removing the allegedly bad validation code from the
chain storage. Among other things, this implies that the operation can
be re-requested. That allows for retrying an operation in case there was
some bug. At the same time it does not look as a DoS vector due to the
caching performed by the nodes.

PVF pre-checking can be enabled and disabled. Initially, according to
the changes in #4420, this mechanism is disabled. Triggering the PVF
pre-checking when it is disabled just means that we insta approve the
requesting operation. This should lead to the behavior being unchanged.

Follow-ups:

- expose runtime APIs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs

* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs

* Review fixes

Co-authored-by: Parity Bot <admin@parity.io>

runtime/common/src/assigned_slots.rs

@@ -555,6 +555,7 @@ mod tests {
 	use sp_core::H256;
 	use sp_runtime::{
 		traits::{BlakeTwo256, IdentityLookup},
+		transaction_validity::TransactionPriority,
 		DispatchError::BadOrigin,
 	};
 
@@ -577,6 +578,14 @@ mod tests {
 		}
 	);
 
+	impl<C> frame_system::offchain::SendTransactionTypes<C> for Test
+	where
+		Call: From<C>,
+	{
+		type Extrinsic = UncheckedExtrinsic;
+		type OverarchingCall = Call;
+	}
+
 	parameter_types! {
 		pub const BlockHashCount: u32 = 250;
 	}
@@ -627,9 +636,15 @@ mod tests {
 		type WeightInfo = parachains_configuration::TestWeightInfo;
 	}
 
+	parameter_types! {
+		pub const ParasUnsignedPriority: TransactionPriority = TransactionPriority::max_value();
+	}
+
 	impl parachains_paras::Config for Test {
 		type Event = Event;
 		type WeightInfo = parachains_paras::TestWeightInfo;
+		type UnsignedPriority = ParasUnsignedPriority;
+		type NextSessionRotation = crate::mock::TestNextSessionRotation;
 	}
 
 	impl parachains_shared::Config for Test {}

runtime/common/src/integration_tests.rs

@@ -38,7 +38,10 @@ use runtime_parachains::{
 use sp_core::{crypto::KeyTypeId, H256};
 use sp_io::TestExternalities;
 use sp_keystore::{testing::KeyStore, KeystoreExt};
-use sp_runtime::traits::{BlakeTwo256, IdentityLookup, One};
+use sp_runtime::{
+	traits::{BlakeTwo256, IdentityLookup, One},
+	transaction_validity::TransactionPriority,
+};
 use sp_std::sync::Arc;
 
 type UncheckedExtrinsic = frame_system::mocking::MockUncheckedExtrinsic<Test>;
@@ -73,6 +76,14 @@ frame_support::construct_runtime!(
 	}
 );
 
+impl<C> frame_system::offchain::SendTransactionTypes<C> for Test
+where
+	Call: From<C>,
+{
+	type Extrinsic = UncheckedExtrinsic;
+	type OverarchingCall = Call;
+}
+
 use crate::{auctions::Error as AuctionsError, crowdloan::Error as CrowdloanError};
 
 parameter_types! {
@@ -170,9 +181,15 @@ impl shared::Config for Test {}
 
 impl origin::Config for Test {}
 
+parameter_types! {
+	pub const ParasUnsignedPriority: TransactionPriority = TransactionPriority::max_value();
+}
+
 impl paras::Config for Test {
 	type Event = Event;
 	type WeightInfo = paras::TestWeightInfo;
+	type UnsignedPriority = ParasUnsignedPriority;
+	type NextSessionRotation = crate::mock::TestNextSessionRotation;
 }
 
 parameter_types! {

runtime/common/src/mock.rs

@@ -17,10 +17,13 @@
 //! Mocking utilities for testing.
 
 use crate::traits::Registrar;
-use frame_support::dispatch::{DispatchError, DispatchResult};
+use frame_support::{
+	dispatch::{DispatchError, DispatchResult},
+	weights::Weight,
+};
 use parity_scale_codec::{Decode, Encode};
 use primitives::v1::{HeadData, Id as ParaId, ValidationCode};
-use sp_runtime::traits::SaturatedConversion;
+use sp_runtime::{traits::SaturatedConversion, Permill};
 use std::{cell::RefCell, collections::HashMap};
 
 thread_local! {
@@ -210,3 +213,21 @@ impl<T: frame_system::Config> TestRegistrar<T> {
 		MANAGERS.with(|x| x.borrow_mut().clear());
 	}
 }
+
+/// A very dumb implementation of `EstimateNextSessionRotation`. At the moment of writing, this
+/// is more to satisfy type requirements rather than to test anything.
+pub struct TestNextSessionRotation;
+
+impl frame_support::traits::EstimateNextSessionRotation<u32> for TestNextSessionRotation {
+	fn average_session_length() -> u32 {
+		10
+	}
+
+	fn estimate_current_session_progress(_now: u32) -> (Option<Permill>, Weight) {
+		(None, 0)
+	}
+
+	fn estimate_next_session_rotation(_now: u32) -> (Option<u32>, Weight) {
+		(None, 0)
+	}
+}

runtime/common/src/paras_registrar.rs

@@ -583,6 +583,7 @@ mod tests {
 	use sp_io::TestExternalities;
 	use sp_runtime::{
 		traits::{BlakeTwo256, IdentityLookup},
+		transaction_validity::TransactionPriority,
 		Perbill,
 	};
 
@@ -605,6 +606,14 @@ mod tests {
 		}
 	);
 
+	impl<C> frame_system::offchain::SendTransactionTypes<C> for Test
+	where
+		Call: From<C>,
+	{
+		type Extrinsic = UncheckedExtrinsic;
+		type OverarchingCall = Call;
+	}
+
 	const NORMAL_RATIO: Perbill = Perbill::from_percent(75);
 	parameter_types! {
 		pub const BlockHashCount: u32 = 250;
@@ -661,9 +670,15 @@ mod tests {
 
 	impl origin::Config for Test {}
 
+	parameter_types! {
+		pub const ParasUnsignedPriority: TransactionPriority = TransactionPriority::max_value();
+	}
+
 	impl paras::Config for Test {
 		type Event = Event;
 		type WeightInfo = paras::TestWeightInfo;
+		type UnsignedPriority = ParasUnsignedPriority;
+		type NextSessionRotation = crate::mock::TestNextSessionRotation;
 	}
 
 	impl configuration::Config for Test {

runtime/kusama/src/lib.rs

@@ -1189,9 +1189,15 @@ impl parachains_inclusion::Config for Runtime {
 	type RewardValidators = parachains_reward_points::RewardValidatorsWithEraPoints<Runtime>;
 }
 
+parameter_types! {
+	pub const ParasUnsignedPriority: TransactionPriority = TransactionPriority::max_value();
+}
+
 impl parachains_paras::Config for Runtime {
 	type Event = Event;
 	type WeightInfo = weights::runtime_parachains_paras::WeightInfo<Runtime>;
+	type UnsignedPriority = ParasUnsignedPriority;
+	type NextSessionRotation = Babe;
 }
 
 parameter_types! {

runtime/kusama/src/weights/runtime_parachains_paras.rs

@@ -16,7 +16,7 @@
 //! Autogenerated weights for `runtime_parachains::paras`
 //!
 //! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION 4.0.0-dev
-//! DATE: 2021-09-21, STEPS: `50`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! DATE: 2021-12-14, STEPS: `50`, REPEAT: 20, LOW RANGE: `[]`, HIGH RANGE: `[]`
 //! EXECUTION: Some(Wasm), WASM-EXECUTION: Compiled, CHAIN: Some("kusama-dev"), DB CACHE: 128
 
 // Executed Command:
@@ -33,7 +33,6 @@
 // --header=./file_header.txt
 // --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs
 
-
 #![cfg_attr(rustfmt, rustfmt_skip)]
 #![allow(unused_parens)]
 #![allow(unused_imports)]
@@ -46,60 +45,56 @@ pub struct WeightInfo<T>(PhantomData<T>);
 impl<T: frame_system::Config> runtime_parachains::paras::WeightInfo for WeightInfo<T> {
 	// Storage: Paras CurrentCodeHash (r:1 w:1)
 	// Storage: Paras CodeByHashRefs (r:1 w:1)
-	// Storage: Paras PastCodeMeta (r:1 w:1)
-	// Storage: Paras PastCodePruning (r:1 w:1)
-	// Storage: Paras PastCodeHash (r:0 w:1)
 	// Storage: Paras CodeByHash (r:0 w:1)
 	fn force_set_current_code(c: u32, ) -> Weight {
 		(0 as Weight)
 			// Standard Error: 0
 			.saturating_add((3_000 as Weight).saturating_mul(c as Weight))
-			.saturating_add(T::DbWeight::get().reads(4 as Weight))
-			.saturating_add(T::DbWeight::get().writes(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(2 as Weight))
+			.saturating_add(T::DbWeight::get().writes(3 as Weight))
 	}
 	// Storage: Paras Heads (r:0 w:1)
 	fn force_set_current_head(s: u32, ) -> Weight {
-		(16_088_000 as Weight)
+		(11_818_000 as Weight)
 			// Standard Error: 0
 			.saturating_add((1_000 as Weight).saturating_mul(s as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}
 	// Storage: Configuration ActiveConfig (r:1 w:0)
-	// Storage: Paras FutureCodeUpgrades (r:1 w:1)
-	// Storage: Paras UpcomingUpgrades (r:1 w:1)
+	// Storage: Paras FutureCodeHash (r:1 w:1)
+	// Storage: Paras CurrentCodeHash (r:1 w:0)
 	// Storage: Paras UpgradeCooldowns (r:1 w:1)
+	// Storage: Paras PvfActiveVoteMap (r:1 w:0)
+	// Storage: Paras CodeByHash (r:1 w:1)
+	// Storage: Paras UpcomingUpgrades (r:1 w:1)
 	// Storage: System Digest (r:1 w:1)
 	// Storage: Paras CodeByHashRefs (r:1 w:1)
-	// Storage: Paras CodeByHash (r:0 w:1)
-	// Storage: Paras FutureCodeHash (r:0 w:1)
+	// Storage: Paras FutureCodeUpgrades (r:0 w:1)
 	// Storage: Paras UpgradeRestrictionSignal (r:0 w:1)
 	fn force_schedule_code_upgrade(c: u32, ) -> Weight {
 		(0 as Weight)
 			// Standard Error: 0
 			.saturating_add((3_000 as Weight).saturating_mul(c as Weight))
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
+			.saturating_add(T::DbWeight::get().reads(9 as Weight))
 			.saturating_add(T::DbWeight::get().writes(8 as Weight))
 	}
 	// Storage: Paras FutureCodeUpgrades (r:1 w:1)
 	// Storage: Paras FutureCodeHash (r:1 w:1)
 	// Storage: Paras CurrentCodeHash (r:1 w:1)
 	// Storage: System Digest (r:1 w:1)
-	// Storage: Paras PastCodeMeta (r:1 w:1)
-	// Storage: Paras PastCodePruning (r:1 w:1)
 	// Storage: Paras Heads (r:0 w:1)
-	// Storage: Paras PastCodeHash (r:0 w:1)
 	// Storage: Paras UpgradeGoAheadSignal (r:0 w:1)
 	fn force_note_new_head(s: u32, ) -> Weight {
-		(69_114_000 as Weight)
+		(37_343_000 as Weight)
 			// Standard Error: 0
 			.saturating_add((1_000 as Weight).saturating_mul(s as Weight))
-			.saturating_add(T::DbWeight::get().reads(6 as Weight))
-			.saturating_add(T::DbWeight::get().writes(9 as Weight))
+			.saturating_add(T::DbWeight::get().reads(4 as Weight))
+			.saturating_add(T::DbWeight::get().writes(6 as Weight))
 	}
 	// Storage: ParasShared CurrentSessionIndex (r:1 w:0)
 	// Storage: Paras ActionsQueue (r:1 w:1)
 	fn force_queue_action() -> Weight {
-		(26_752_000 as Weight)
+		(22_980_000 as Weight)
 			.saturating_add(T::DbWeight::get().reads(2 as Weight))
 			.saturating_add(T::DbWeight::get().writes(1 as Weight))
 	}

runtime/parachains/src/configuration.rs

@@ -73,6 +73,11 @@ pub struct HostConfiguration<BlockNumber> {
 	/// This parameter affects the upper bound of size of `CandidateCommitments`.
 	pub hrmp_max_message_num_per_candidate: u32,
 	/// The minimum period, in blocks, between which parachains can update their validation code.
+	///
+	/// If PVF pre-checking is enabled this should be greater than the maximum number of blocks
+	/// PVF pre-checking can take. Intuitively, this number should be greater than the duration
+	/// specified by [`pvf_voting_ttl`]. Unlike, [`pvf_voting_ttl`], this parameter uses blocks
+	/// as a unit.
 	pub validation_upgrade_frequency: BlockNumber,
 	/// The delay, in blocks, before a validation upgrade is applied.
 	pub validation_upgrade_delay: BlockNumber,

runtime/parachains/src/hrmp.rs

@@ -1370,6 +1370,7 @@ mod tests {
 			configuration: crate::configuration::GenesisConfig {
 				config: crate::configuration::HostConfiguration {
 					max_downward_message_size: 1024,
+					pvf_checking_enabled: false,
 					..Default::default()
 				},
 			},

runtime/parachains/src/mock.rs

@@ -38,7 +38,8 @@ use sp_core::H256;
 use sp_io::TestExternalities;
 use sp_runtime::{
 	traits::{BlakeTwo256, IdentityLookup},
-	KeyTypeId,
+	transaction_validity::TransactionPriority,
+	KeyTypeId, Permill,
 };
 use std::{cell::RefCell, collections::HashMap};
 
@@ -70,6 +71,14 @@ frame_support::construct_runtime!(
 	}
 );
 
+impl<C> frame_system::offchain::SendTransactionTypes<C> for Test
+where
+	Call: From<C>,
+{
+	type Extrinsic = UncheckedExtrinsic;
+	type OverarchingCall = Call;
+}
+
 parameter_types! {
 	pub const BlockHashCount: u32 = 250;
 	pub BlockWeights: frame_system::limits::BlockWeights =
@@ -181,9 +190,33 @@ impl crate::shared::Config for Test {}
 
 impl origin::Config for Test {}
 
+parameter_types! {
+	pub const ParasUnsignedPriority: TransactionPriority = TransactionPriority::max_value();
+}
+
+/// A very dumb implementation of `EstimateNextSessionRotation`. At the moment of writing, this
+/// is more to satisfy type requirements rather than to test anything.
+pub struct TestNextSessionRotation;
+
+impl frame_support::traits::EstimateNextSessionRotation<u32> for TestNextSessionRotation {
+	fn average_session_length() -> u32 {
+		10
+	}
+
+	fn estimate_current_session_progress(_now: u32) -> (Option<Permill>, Weight) {
+		(None, 0)
+	}
+
+	fn estimate_next_session_rotation(_now: u32) -> (Option<u32>, Weight) {
+		(None, 0)
+	}
+}
+
 impl crate::paras::Config for Test {
 	type Event = Event;
 	type WeightInfo = crate::paras::TestWeightInfo;
+	type UnsignedPriority = ParasUnsignedPriority;
+	type NextSessionRotation = TestNextSessionRotation;
 }
 
 impl crate::dmp::Config for Test {}