feat: upgrade to Polkadot v0.9.32 (#434)

## fixes KILTProtocol/ticket#2289 and KILTProtocol/ticket#2296 * Upgrades from Polkadot v0.9.29 to v0.9.32 * Adds missing feature implementations for all tomls (checked via `subalfred check features` in all crates) * Actually necessary for application of paritytech/substrate#10592 (see runtime changes in [dd81eac](dd81eac)) * Migrates Democracy, Preimage and Scheduler pallets to use bounded Calls, see below ## Summary of changes (Polkadot v0.9.30-0.9.32) * Weights v2 are not fully there yet, but the struct now includes the [second field for storage size](paritytech/substrate#12277) (tracking issue: https://github.com/paritytech/substrate/issues/12176) ### Breaking Changes * Breaking: Outer enums (paritytech/substrate#11981) * `Origin` --> `RuntimeOrigin` * `Call` --> `RuntimeCall` * `Event` --> `RuntimeEvent` * ~Convention seems to be to keep `Event`, `Call`, `Origin` for inner pallet usage, e.g. `Did::Origin`~ Update: We use `Runtime` prefix internally as well ### Noteworthy PRs * paritytech/substrate#12109 * paritytech/substrate#12328 * paritytech/cumulus#1585 * Following the effort of decoupling collators and full relay nodes, this PR adds the possibility of pointing the collator to an “external” (non in-process) relay node. This is still considered experimental, and the **relay node is suggested to run on the same machine than the collator for the moment**. * To specify the relay full node rpc: `polkadot-parachain --alice --collator --relay-chain-rpc-url <rpc-websocket-url>` * paritytech/substrate#12486 * Before this change only the interpreted WASM executor was included in per default compilations. Making the compiled executor opt-in, now, compiled WASM executor is set by default and an opt-out instead. This could lead to **big performance difference** between using these two, as more recent versions of the interpreter see a regression in performance. ### Scheduler, Preimage, Democracy Migration * paritytech/substrate#11649 * Referenda, Democracy, Scheduler and Preimage pallets are all now bounded in storage access footprint * Removed the concept of a "hard deadline" or weight-override priority and no longer guarantees that at least one scheduled item will be executed per block (since these are both dangerous to parachains which have a strict need of weight limits). This means you must ensure that scheduled items are below the MaximumWeight or they will not be executed. * Interesting comment: paritytech/substrate#11649 (comment) > There is migration code which moves existing proposals and referenda over to the new format. However IT DOES NOT MIGRATE EVERYTHING: > > * Preimages are **NOT** migrated. Any registered preimages in Democracy at the time of migration are dropped. Their balance is **NOT UNRESERVED**. > * The re-dispatcher used in the old Democracy implementation is removed. Any proposals scheduled for dispatch by Democracy **WILL NOT EXECUTE**. > > This means you SHOULD ensure that: > > * **the preimage for the runtime upgrade is placed as an imminent preimage, not with a deposit;** > * **no other preimages are in place at the time of upgrade;** > * **there are no other proposals scheduled for dispatch by Democracy at the time of upgrade.** > > The Democracy pallet will be marked as deprecated immediately once Referenda is considered production-ready. **ALL TEAMS ARE RECOMMENDED TO SWITCH SWAY FROM DEMOCRACY PALLET TO REFERENDA/CONVICTION-VOTING PALLETS ASAP** #### Result of `try-runtime` against Spiritnet on Friday Nov 18, 2022: ``` 2022-11-18 09:27:23.917 INFO main runtime::preimage::migration::v1: Migrating 0 images 2022-11-18 09:27:23.917 INFO main runtime::scheduler::migration: Trying to migrate 0 agendas... 2022-11-18 09:27:23.917 INFO main runtime::scheduler::migration: Migrated 0 agendas. 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: 0 public proposals will be migrated. 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: 25 referenda will be migrated. 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #7 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #20 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #13 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #5 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #8 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #1 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #19 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #9 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #16 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #14 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #21 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #15 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #24 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #22 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #2 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #10 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #0 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #6 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #11 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #3 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #17 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #18 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #23 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #4 2022-11-18 09:27:23.917 INFO main runtime::democracy::migration::v1: migrating referendum #25 2022-11-18 09:27:23.918 INFO main runtime::democracy::migration::v1: 0 public proposals migrated, 25 referenda migrated ``` ## Checklist: - [x] I have verified that the code works - [x] No panics! (checked arithmetic ops, no indexing `array[3]` use `get(3)`, ...) - [x] I have verified that the code is easy to understand - [ ] If not, I have left a well-balanced amount of inline comments - [x] I have [left the code in a better state](https://deviq.com/principles/boy-scout-rule) - [x] I have documented the changes (where applicable)
KILTprotocol · Dec 6, 2022 · 431c1e9 · 431c1e9
1 parent 81d881a
commit 431c1e9
Show file tree

Hide file tree

Showing 81 changed files with 3,298 additions and 3,285 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
diff --git a/nodes/parachain/Cargo.toml b/nodes/parachain/Cargo.toml
@@ -77,6 +77,7 @@ cumulus-primitives-core = {workspace = true, features = ["std"]}
 cumulus-primitives-parachain-inherent = {workspace = true, features = ["std"]}
 cumulus-relay-chain-inprocess-interface.workspace = true
 cumulus-relay-chain-interface.workspace = true
+cumulus-relay-chain-minimal-node.workspace = true
 cumulus-relay-chain-rpc-interface.workspace = true
 
 # Polkadot dependencies
@@ -99,12 +100,25 @@ fast-gov = [
   "runtime-common/fast-gov",
 ]
 runtime-benchmarks = [
+  "clone-runtime/runtime-benchmarks",
+  "frame-benchmarking/runtime-benchmarks",
+  "frame-benchmarking-cli/runtime-benchmarks",
+  "polkadot-cli/runtime-benchmarks",
+  "polkadot-primitives/runtime-benchmarks",
   "polkadot-service/runtime-benchmarks",
+  "runtime-common/runtime-benchmarks",
   "peregrine-runtime/runtime-benchmarks",
+  "sc-service/runtime-benchmarks",
   "spiritnet-runtime/runtime-benchmarks",
+  "sp-runtime/runtime-benchmarks",
+  "xcm/runtime-benchmarks",
 ]
 try-runtime = [
+  "clone-runtime/try-runtime",
   "peregrine-runtime/try-runtime",
+  "polkadot-cli/try-runtime",
+  "polkadot-service/try-runtime",
+  "runtime-common/try-runtime",
   "spiritnet-runtime/try-runtime",
   "try-runtime-cli",
 ]
diff --git a/nodes/parachain/src/cli.rs b/nodes/parachain/src/cli.rs
@@ -53,7 +53,7 @@ pub(crate) enum Subcommand {
 
 	/// Sub-commands concerned with benchmarking.
 	/// The pallet benchmarking moved to the `pallet` sub-command.
-	#[clap(subcommand)]
+	#[command(subcommand)]
 	Benchmark(frame_benchmarking_cli::BenchmarkCmd),
 
 	/// Try some command against runtime state.
@@ -69,11 +69,11 @@ pub(crate) enum Subcommand {
 /// Command for building the genesis state of the parachain
 #[derive(Debug, Parser)]
 pub(crate) struct BuildSpecCmd {
-	#[clap(flatten)]
+	#[command(flatten)]
 	pub(crate) inner_args: sc_cli::BuildSpecCmd,
 
 	/// The name of the runtime which should get executed.
-	#[clap(long, default_value = DEFAULT_RUNTIME)]
+	#[arg(long, default_value = DEFAULT_RUNTIME)]
 	pub(crate) runtime: String,
 }
 
@@ -86,16 +86,16 @@ impl Deref for BuildSpecCmd {
 }
 
 #[derive(Debug, clap::Parser)]
-#[clap(
+#[command(
 	propagate_version = true,
 	args_conflicts_with_subcommands = true,
 	subcommand_negates_reqs = true
 )]
 pub(crate) struct Cli {
-	#[clap(subcommand)]
+	#[command(subcommand)]
 	pub(crate) subcommand: Option<Subcommand>,
 
-	#[clap(flatten)]
+	#[command(flatten)]
 	pub(crate) run: cumulus_client_cli::RunCmd,
 
 	// Disable automatic hardware benchmarks.
@@ -105,16 +105,16 @@ pub(crate) struct Cli {
 	///
 	/// The results are then printed out in the logs, and also sent as part of
 	/// telemetry, if telemetry is enabled.
-	#[clap(long)]
+	#[arg(long)]
 	pub no_hardware_benchmarks: bool,
 
 	/// The name of the runtime which should get executed.
-	#[clap(long, default_value = DEFAULT_RUNTIME)]
+	#[arg(long, default_value = DEFAULT_RUNTIME)]
 	pub(crate) runtime: String,
 
 	/// Relaychain arguments
-	#[clap(raw = true)]
-	pub(crate) relaychain_args: Vec<String>,
+	#[arg(raw = true)]
+	pub(crate) relay_chain_args: Vec<String>,
 }
 
 #[derive(Debug)]

diff --git a/nodes/parachain/src/command.rs b/nodes/parachain/src/command.rs
@@ -25,7 +25,7 @@ use codec::Encode;
 use cumulus_client_cli::generate_genesis_block;
 use cumulus_primitives_core::ParaId;
 use frame_benchmarking_cli::{BenchmarkCmd, SUBSTRATE_REFERENCE_HARDWARE};
-use log::info;
+use log::{info, warn};
 #[cfg(feature = "try-runtime")]
 use polkadot_service::TaskManager;
 use runtime_common::Block;
@@ -35,7 +35,7 @@ use sc_cli::{
 };
 use sc_service::config::{BasePath, PrometheusConfig};
 use sp_core::hexdisplay::HexDisplay;
-use sp_runtime::traits::{AccountIdConversion, Block as BlockT};
+use sp_runtime::traits::{AccountIdConversion, Block as BlockT, Zero};
 use std::net::SocketAddr;
 
 trait IdentifyChain {
@@ -271,7 +271,7 @@ pub fn run() -> Result<()> {
 					&config,
 					[RelayChainCli::executable_name()]
 						.iter()
-						.chain(cli.relaychain_args.iter()),
+						.chain(cli.relay_chain_args.iter()),
 				);
 
 				let polkadot_config =
@@ -336,6 +336,14 @@ pub fn run() -> Result<()> {
 					)?;
 					cmd.run(partials.client)
 				}),
+				#[cfg(not(feature = "runtime-benchmarks"))]
+				(BenchmarkCmd::Storage(_), _) => Err(sc_cli::Error::Input(
+					"Compile with --features=runtime-benchmarks \
+						to enable storage benchmarks."
+						.into(),
+				)
+				.into()),
+				#[cfg(feature = "runtime-benchmarks")]
 				(BenchmarkCmd::Storage(cmd), "spiritnet") => runner.sync_run(|config| {
 					let partials = new_partial::<spiritnet_runtime::RuntimeApi, SpiritnetRuntimeExecutor, _>(
 						&config,
@@ -347,6 +355,7 @@ pub fn run() -> Result<()> {
 
 					cmd.run(config, partials.client.clone(), db, storage)
 				}),
+				#[cfg(feature = "runtime-benchmarks")]
 				(BenchmarkCmd::Storage(cmd), "peregrine") => runner.sync_run(|config| {
 					let partials = new_partial::<peregrine_runtime::RuntimeApi, PeregrineRuntimeExecutor, _>(
 						&config,
@@ -358,6 +367,7 @@ pub fn run() -> Result<()> {
 
 					cmd.run(config, partials.client.clone(), db, storage)
 				}),
+				#[cfg(feature = "runtime-benchmarks")]
 				(BenchmarkCmd::Storage(cmd), "clone") => runner.sync_run(|config| {
 					let partials = new_partial::<clone_runtime::RuntimeApi, CloneRuntimeExecutor, _>(
 						&config,
@@ -421,7 +431,7 @@ pub fn run() -> Result<()> {
 					&config,
 					[RelayChainCli::executable_name()]
 						.iter()
-						.chain(cli.relaychain_args.iter()),
+						.chain(cli.relay_chain_args.iter()),
 				);
 
 				let id = ParaId::from(para_id);
@@ -446,6 +456,10 @@ pub fn run() -> Result<()> {
 					if config.role.is_authority() { "yes" } else { "no" }
 				);
 
+				if collator_options.relay_chain_rpc_url.is_some() && !cli.relay_chain_args.len().is_zero() {
+					warn!("Detected relay chain node arguments together with --relay-chain-rpc-url. This command starts a minimal Polkadot node that only uses a network-related subset of all relay chain CLI options.");
+				}
+
 				if config.chain_spec.is_peregrine() {
 					crate::service::start_node::<PeregrineRuntimeExecutor, peregrine_runtime::RuntimeApi>(
 						config,

diff --git a/nodes/parachain/src/service.rs b/nodes/parachain/src/service.rs
@@ -29,9 +29,8 @@ use cumulus_client_service::{
 use cumulus_primitives_core::ParaId;
 use cumulus_relay_chain_inprocess_interface::build_inprocess_relay_chain;
 use cumulus_relay_chain_interface::{RelayChainError, RelayChainInterface, RelayChainResult};
-use cumulus_relay_chain_rpc_interface::{create_client_and_start_worker, RelayChainRpcInterface};
+use cumulus_relay_chain_minimal_node::build_minimal_relay_chain_node;
 use polkadot_service::{CollatorPair, NativeExecutionDispatch};
-use sc_client_api::ExecutorProvider;
 use sc_executor::NativeElseWasmExecutor;
 use sc_network::{NetworkBlock, NetworkService};
 use sc_service::{Configuration, TFullBackend, TFullClient, TaskManager};
@@ -181,7 +180,7 @@ where
 		&task_manager,
 	)?;
 
-	let params = PartialComponents {
+	Ok(PartialComponents {
 		backend,
 		client,
 		import_queue,
@@ -190,9 +189,7 @@ where
 		transaction_pool,
 		select_chain: (),
 		other: (telemetry, telemetry_worker_handle),
-	};
-
-	Ok(params)
+	})
 }
 
 async fn build_relay_chain_interface(
@@ -204,10 +201,7 @@ async fn build_relay_chain_interface(
 	hwbench: Option<sc_sysinfo::HwBench>,
 ) -> RelayChainResult<(Arc<(dyn RelayChainInterface + 'static)>, Option<CollatorPair>)> {
 	match collator_options.relay_chain_rpc_url {
-		Some(relay_chain_url) => {
-			let client = create_client_and_start_worker(relay_chain_url, task_manager).await?;
-			Ok((Arc::new(RelayChainRpcInterface::new(client)) as Arc<_>, None))
-		}
+		Some(relay_chain_url) => build_minimal_relay_chain_node(polkadot_config, task_manager, relay_chain_url).await,
 		None => build_inprocess_relay_chain(
 			polkadot_config,
 			parachain_config,
@@ -310,15 +304,16 @@ where
 	let prometheus_registry = parachain_config.prometheus_registry().cloned();
 	let transaction_pool = params.transaction_pool.clone();
 	let import_queue = cumulus_client_service::SharedImportQueue::new(params.import_queue);
-	let (network, system_rpc_tx, start_network) = sc_service::build_network(sc_service::BuildNetworkParams {
-		config: &parachain_config,
-		client: client.clone(),
-		transaction_pool: transaction_pool.clone(),
-		spawn_handle: task_manager.spawn_handle(),
-		import_queue: import_queue.clone(),
-		block_announce_validator_builder: Some(Box::new(|_| Box::new(block_announce_validator))),
-		warp_sync: None,
-	})?;
+	let (network, system_rpc_tx, tx_handler_controller, start_network) =
+		sc_service::build_network(sc_service::BuildNetworkParams {
+			config: &parachain_config,
+			client: client.clone(),
+			transaction_pool: transaction_pool.clone(),
+			spawn_handle: task_manager.spawn_handle(),
+			import_queue: import_queue.clone(),
+			block_announce_validator_builder: Some(Box::new(|_| Box::new(block_announce_validator))),
+			warp_sync: None,
+		})?;
 
 	let rpc_builder = {
 		let client = client.clone();
@@ -345,6 +340,7 @@ where
 		backend: backend.clone(),
 		network: network.clone(),
 		system_rpc_tx,
+		tx_handler_controller,
 		telemetry: telemetry.as_mut(),
 	})?;
 
@@ -382,7 +378,6 @@ where
 		)?;
 
 		let spawner = task_manager.spawn_handle();
-
 		let params = StartCollatorParams {
 			para_id: id,
 			block_status: client.clone(),
@@ -407,7 +402,6 @@ where
 			relay_chain_interface,
 			relay_chain_slot_duration,
 			import_queue,
-			collator_options,
 		};
 
 		start_full_node(params)?;
@@ -446,22 +440,21 @@ where
 {
 	let slot_duration = cumulus_client_consensus_aura::slot_duration(&*client)?;
 
-	cumulus_client_consensus_aura::import_queue::<sp_consensus_aura::sr25519::AuthorityPair, _, _, _, _, _, _>(
+	cumulus_client_consensus_aura::import_queue::<sp_consensus_aura::sr25519::AuthorityPair, _, _, _, _, _>(
 		cumulus_client_consensus_aura::ImportQueueParams {
 			block_import: client.clone(),
-			client: client.clone(),
+			client,
 			create_inherent_data_providers: move |_, _| async move {
-				let time = sp_timestamp::InherentDataProvider::from_system_time();
+				let timestamp = sp_timestamp::InherentDataProvider::from_system_time();
 
 				let slot = sp_consensus_aura::inherents::InherentDataProvider::from_timestamp_and_slot_duration(
-					*time,
+					*timestamp,
 					slot_duration,
 				);
 
-				Ok((time, slot))
+				Ok((slot, timestamp))
 			},
 			registry: config.prometheus_registry(),
-			can_author_with: sp_consensus::CanAuthorWithNativeVersion::new(client.executor().clone()),
 			spawner: &task_manager.spawn_essential_handle(),
 			telemetry,
 		},
@@ -539,17 +532,17 @@ where
 								id,
 							)
 							.await;
-						let time = sp_timestamp::InherentDataProvider::from_system_time();
+						let timestamp = sp_timestamp::InherentDataProvider::from_system_time();
 
 						let slot = sp_consensus_aura::inherents::InherentDataProvider::from_timestamp_and_slot_duration(
-							*time,
+							*timestamp,
 							slot_duration,
 						);
 
 						let parachain_inherent = parachain_inherent.ok_or_else(|| {
 							Box::<dyn std::error::Error + Send + Sync>::from("Failed to create parachain inherent")
 						})?;
-						Ok((time, slot, parachain_inherent))
+						Ok((slot, timestamp, parachain_inherent))
 					}
 				},
 				block_import: client.clone(),

diff --git a/nodes/standalone/Cargo.toml b/nodes/standalone/Cargo.toml
@@ -70,10 +70,18 @@ try-runtime-cli = {workspace = true, optional = true}
 [features]
 default = []
 runtime-benchmarks = [
+  "frame-benchmarking/runtime-benchmarks",
+  "frame-benchmarking-cli/runtime-benchmarks",
+  "frame-system/runtime-benchmarks",
   "mashnet-node-runtime/runtime-benchmarks",
   "runtime-common/runtime-benchmarks",
+  "sc-service/runtime-benchmarks",
+  "sp-runtime/runtime-benchmarks",
 ]
 try-runtime = [
+  "frame-system/try-runtime",
   "mashnet-node-runtime/try-runtime",
+  "pallet-transaction-payment/try-runtime",
+  "runtime-common/try-runtime",
   "try-runtime-cli",
 ]
diff --git a/nodes/standalone/src/benchmarking.rs b/nodes/standalone/src/benchmarking.rs
@@ -120,7 +120,7 @@ impl frame_benchmarking_cli::ExtrinsicBuilder for TransferKeepAliveBuilder {
 pub fn create_benchmark_extrinsic(
 	client: &FullClient,
 	sender: sp_core::sr25519::Pair,
-	call: runtime::Call,
+	call: runtime::RuntimeCall,
 	nonce: u32,
 ) -> runtime::UncheckedExtrinsic {
 	let genesis_hash = client.block_hash(0).ok().flatten().expect("Genesis block exists; qed");

diff --git a/nodes/standalone/src/cli.rs b/nodes/standalone/src/cli.rs
@@ -21,17 +21,17 @@ use sc_cli::RunCmd;
 
 #[derive(Debug, Parser)]
 pub struct Cli {
-	#[clap(subcommand)]
+	#[command(subcommand)]
 	pub subcommand: Option<Subcommand>,
 
-	#[clap(flatten)]
+	#[command(flatten)]
 	pub run: RunCmd,
 }
 
 #[derive(Debug, Parser)]
 pub enum Subcommand {
 	/// Key management cli utilities
-	#[clap(subcommand)]
+	#[command(subcommand)]
 	Key(sc_cli::KeySubcommand),
 
 	/// Build a chain specification.
@@ -56,7 +56,7 @@ pub enum Subcommand {
 	Revert(sc_cli::RevertCmd),
 
 	/// Sub-commands concerned with benchmarking.
-	#[clap(subcommand)]
+	#[command(subcommand)]
 	Benchmark(frame_benchmarking_cli::BenchmarkCmd),
 
 	/// Try some command against runtime state.

diff --git a/nodes/standalone/src/command.rs b/nodes/standalone/src/command.rs
@@ -143,7 +143,7 @@ pub fn run() -> sc_cli::Result<()> {
 			let runner = cli.create_runner(cmd)?;
 
 			runner.sync_run(|config| {
-				let PartialComponents { client, backend, .. } = service::new_partial(&config)?;
+				let PartialComponents { client, .. } = service::new_partial(&config)?;
 				// This switch needs to be in the client, since the client decides
 				// which sub-commands it wants to support.
 				match cmd {
@@ -157,7 +157,13 @@ pub fn run() -> sc_cli::Result<()> {
 						cmd.run::<Block, service::ExecutorDispatch>(config)
 					}
 					BenchmarkCmd::Block(cmd) => cmd.run(client),
+					#[cfg(not(feature = "runtime-benchmarks"))]
+					BenchmarkCmd::Storage(_) => {
+						Err("Storage benchmarking can be enabled with `--features runtime-benchmarks`.".into())
+					}
+					#[cfg(feature = "runtime-benchmarks")]
 					BenchmarkCmd::Storage(cmd) => {
+						let PartialComponents { client, backend, .. } = service::new_partial(&config)?;
 						let db = backend.expose_db();
 						let storage = backend.expose_storage();