XCM v3

[gavofyork]

Introduce XCM version 3 and remove XCM versions 0 and 1.

Cumulus companion: paritytech/cumulus#697

Closes #4069
Closes #4670

TODO

• Safe Dispatches
• ReportError & QueryHolding should use QueryResponseInfo
• Remove max_assets and ensure general asset match limitation
• Introduce & enforce maximum value for Holding.len().
• Tests
• XCM Benchmarks for Generic Instructions #3940
• XCM v3: Introduce querier field into QueryReponse #4732
• XCM v3: Bridge infrastructure #4681
• XCM v3: ExchangeAsset and Remote-locking #4945
• XCM v3: Support for non-fungibles #4950
• XCM v3: Context & ID hash #4756
• Consider removing XCM v1: Remove XCMv1 #5094
• Move ContainsPair to Substrate
• Asset namespacing primitive to allow the hosting of foreign assets
• Cumulus companion
• Clean up LocationInverter/UniversalLocation: It should just be a Get<InteriorMultiLocation>
• Extend PR description to include XCM v3: Context & ID hash #4756 and XCM v3: Support for non-fungibles #4950.
• Some late fixes for XCMv3 #5237
• Reuse sp_weights::Weight as xcm::v3::Weight: Use 2D weights in XCM v3 #6134
  • Introduce tests in xcm-builder where weight parts are not same to ensure any/all gt/lt comparison operations are correct.
• Ensure all message queues are restricted to to 10 messages per block. Introduce whitelist for Transact and limit UMP processing to 10 messages per block #6280
• Introduce a (configurable but highly restrictive whitelist on Transact. Introduce whitelist for Transact and limit UMP processing to 10 messages per block #6280
• Limit the number of overweight messages which can exist to something sensible (e.g. 1000). Add upper limit on the number of overweight messages in the queue #6298
• Limit reentrancy via a thread_local counter to limit stack depth which an Xcm Transact can create. Create thread_local in XCM executor to limit recursion depth #6304
• Double-check weights and fees in pallet-xcm - in particular, ensure that executing and sending is costly enough.
• Write tests for FixedRateOfFungible.
• Cumulus companion.

Features

Reverse Checking

Checking account book-keeping now works in bidirectionally. Previously only assets which were minted locally were checked (and the checking account stored assets which have been teleported out and not yet received back). Now this works for assets minted remotely. In this case, the checking account stores the assets which have been teleported in but not yet teleported back out, which should equal the total amount in all other accounts.

This is an initial implementation of this feature. A far better implementation which does not result in incorrect total issuance of any checked assets is planned in paritytech/polkadot-sdk#487.

UnpaidExecution instruction

A new instruction which makes clear when unpaid execution is to be allowed via the AllowUnpaidExecution barrier. This should always be used by a chain which expects free execution (e.g. the Relay-chain executing on a system parachain) since otherwise the barrier will not work. This ensure that the barrier can block unauthorised usages of the unpaid execution such as when an account sends a message from the Relay-chain to the parachain (which would have DescendOrigin forcibly prefixed and therefore could not prepend UnpaidExecution).

Minor Fixes to Barriers

The barrier AllowKnownQueryResponses requires its message to have only a single instruction. Any usage of ClearOrigin may also only be singular.

Two-stage SendXcm API supporting pricing and ID hash

SendXcm has been refactored to become a two-stage process, with the first stage yielding a ticket (which can be used to complete the operation) and a price for the operation. This allows the operation to be prepared and priced before optional completion at a later stage. This is utilised in order to allow for payment be determined and happen (or fail) prior to the delivery.

The second stage returns an identifying hash for the message which should be determinable at the destination side. Generally this is the hash of the bytes expressing the message sent "on the wire".

Topic

There is a new item in the XCVM state: topic: Option<[u8; 32]>. Context includes a topic: Option<[u8; 32]> field, representing this value. There should be two new instructions:

- `SetTopic([u8; 32])`: Sets the XCVM's topic to `Some`.
- `ResetTopic`: Sets the XCVM's topic to `None`.

Context

Functions in traits used by XcmExecutor now have knowledge of the Context of their instruction. This includes the Origin, message hash and Topic of the XCVM state:

pub struct XcmContext {
  origin: Option<MultiLocation>,
  message_hash: XcmHash,
  topic: [u8; 32],
}

Remote locking

Four new instructions added allowing for one chain to lock assets and notify another chain of this change as well as allowing this notified chain to request the assets be unlocked and for it to instruct the locking chain to unlock the assets.

The instructions are:

• LockAsset
• UnlockAsset
• NoteUnlockable
• RequestUnlock

A new API is introduced to allow integration of the on-chain locking mechanisms into these instructions and an example implementation of the API is done in XCM Pallet.

ExchangeAsset API

A new trait-based API is introduced to allow chains to implement the ExchangeAsset instruction.

Universal Location, Context

There is now a Universal Location under which all systems which generate their own consensus exist, and by extension, all possible locations within consensus. The concept of ancestry has been deprecated and instead there is the concept of Context, which is the location of something (e.g. generally the local chain) within the Universal Location. For chains which generate their own consensus like Polkadot and Kusama, it will generally just be a Junction::GlobalConsensus; but for other locations like parachains and smart contracts, it will be longer. Because contexts are locations relative to the Universal Location and because this is the top-level (though imaginary) consensus, all contexts can be expressed as a InteriorMultiLocations (i.e. they need not utilise any parents).

NetworkId no longer has Any

NetworkId is reworked to remove the Any which allows it to be used within the new Junction::GlobalConsensus. In the original usages of NetworkId, it is now an Option<NetworkId> where None may be placed instead to mean the equivalent of NetworkId::Any.

ExportXcm

A new API ExportXcm is introduced to allow for sending XCM to a system within a different global consensus. Like the SendXcm API this is two-stage and allows for price-discovery prior to initiation of final delivery. There are new SendXcm implementations (LocalUnpaidExporter, UnpaidRemoteExporter, SovereignPaidRemoteExporter) allowing use of ExportXcm to allow for messages to be routed via implementations of ExportXcm as well as a simple API (HaulBlobExporter) to allow easy integration of point-to-point inter-consensus systems (i.e. bridges).

New instruction UniversalOrigin

The UniversalOrigin instruction is introduced allowing messages to alter their origin to that of some alternative GlobalConsensus, effectively allowing a location to spoof another. This is a strictly permissioned instruction only available to select origins and only for particular changes. The specific use-case is for allowing bridges to forward a message and have it be executed as though it were from a location which is being bridged. It is utilised in the aforementioned SendXcm instructions.

New barrier

NOTE: For more information on this and other elements of XCM bridge integration, see the relevant PR.

Expectations

These are a set of instructions to allow errors to be thrown based on expectations of register contents. This effectively allows branching based on the contents of the Holding, Error and Origin registers.

Safe dispatches

This adds several facilities aiming to provide cross-chain pallet discovery and version-safety services. The features:

• A new register Transact Status Register, storing the status of the most recent attempt to use Transact instruction.
• A new instruction QueryPallet used to get a QueryRespose message informing of instances of a named pallet available on the chain.
• A new instruction ExpectPallet used to throw an error if an identified pallet does not match expected versions and other information.
• A new instruction ReportTransactStatus used to report the contents of the Transact Status Register to a destination.
• A new instruction ClearTransactStatus to clear the value in the Transact Status Register.

BurnAsset

A new instruction BurnAsset for explicitly discarding from Holding.

Complexity caps

New facilities to ensure that the weight of operations involing the Holding Register can be estimated sensibly in advance. This introduces two main features:

• The maximum size of Holding is capped with a new Config trait item for XcmExecutor.
• There are two new wildcard variants which limit the matches by an integer count specified in the wildcard. This obsoletes max_assets.

Introduce querier field into QueryReponse

Introduce an additional field into the QueryResponse XCM instruction querier indicating the original initiator which led to this response being sent (from the point of view of the receiver). Typically this will simply be Here, since the receiver of the response will be the one who originally made the query.

The XCM pallet's query response handler system now allows expected query responses to have have an expected initiator of the response. Generally this is set to Here when new queries are initiated.

Before any query response is executed using the XCM pallet's response-dispatch system, the querier field is matched against what is expected. If it differs, then the ExpectedResponse barrier doesn't pass and the response is ignored.

Support for Non-fungibles

Provides XCM/MultiAsset support for using any implementer of the assets::nonfungible traits including, notably, Uniques pallet.

Introduces an example utilizing the pallet_uniques::Config::CreateOrigin type param allowing a Uniques pallet instance to automatically hold foreign NFTs in a reserve mechanism. Here we can allow the location which hosts them to create the NFT collection and thus determine who pays the deposits for any assets minted therein. NFTs can then be transferred in using the usual ReserveAssetTransfer mechanism, or, if the chains trust each other, then beamed in with a Teleport.

Code migration considerations v2 -> v3

XCM Pallet

The configuration trait for the XCM pallet has several new type parameters which must be configured. Assuming you have a Balances pallet configured and a type LocationToAccountId which converts a MultiLocation to an AccountId, then to retain previous behaviour, it can be set like this:

Old code:

impl pallet_xcm::Config for Runtime {
	/* snip */
}

New code:

use frame_support::traits::ConstU32;
impl pallet_xcm::Config for Runtime {
	/* snip */
	type Currency = Balances;
	type CurrencyMatcher = ();
	type TrustedLockers = ();
	type SovereignAccountOf = LocationToAccountId;
	type MaxLockers = ConstU32<8>;
}

XCM Executor Config trait

Configuration trait xcm_executor::Config has seven new items type MaxAssetsIntoHolding: Get<u32> (to be set to the most distinct assets you expect your chain to have to deal with in a single XCM at once, e.g. 8) type PalletInstancesInfo: PalletsInfoAccess (which should generally be set to AllPalletsWithSystem), AssetLocker, AssetExchanger, FeeManager and MessageExporter which can be all be defined as () to get the same behaviour as previously. Finally UniversalAliases, which can be set to Nothing to give the previous behaviour. Example code change:

Old code:

impl xcm_executor::Config for XcmConfig {
	/* snip */
}

New code:

impl xcm_executor::Config for XcmConfig {
	/* snip */
	type PalletInstancesInfo = AllPalletsWithSystem;
	type MaxAssetsIntoHolding = ConstU32<8>;
	type AssetLocker = ();
	type AssetExchanger = ();
	type FeeManager = ();
	type MessageExporter = ();
	type UniversalAliases = Nothing;
}

xcm_builder::FungiblesAdapter's CheckAsset parameter

The CheckAsset type parameter in xcm_builder::FungiblesAdapter and xcm_builder::NonFungiblesAdapter has had its type constraint altered and is now a AssetChecking<Assets::AssetId> rather than a Contains<Assets::AssetId>. If you previously passed Nothing to it, then you can replace this with xcm_builder::NoChecking. If you implemented it as T, then it needs to be wrapped with a xcm_builder::LocalMint<T>. So:

pub type CheckAsset = /* snip */;
pub type FungiblesTransactor = FungiblesAdapter<
	Assets,
	Matcher,
	LocationToAccountId,
	AccountId,
	CheckAsset,
	CheckingAccount,
>;

would become:

pub type CheckAsset = /* snip */;
pub type FungiblesTransactor = FungiblesAdapter<
	Assets,
	Matcher,
	LocationToAccountId,
	AccountId,
	xcm_builder::LocalMint<CheckAsset>,
	CheckingAccount,
>;

max_assets and Counted wildcards

max_assets has now been removed as a field from DepositAsset and DepositReserveAsset. For the purposes of calculating weight, the number of assets deposited when using a wildcard for assets field will be limited through MaxAssetsIntoHolding. If this yields too great a weight (as it likely will for operations which touch storage for each asset), then two new wildcard matches are provided: AllCounted and AllOfCounted, which have the same effect of limiting the number of assets matched as max_assets did.

Example code change:

Old code:

let beneficiary = /* snip */;
let x = DepositAsset { assets: (Here, 100).into(), max_assets: 1, beneficiary };
let y = DepositAsset { assets: Wild(All), max_assets: 5, beneficiary },

New code:

let beneficiary = /* snip */;
let x = DepositAsset { assets: (Here, 100).into(), beneficiary };
let y = DepositAsset { assets: Wild(AllCounted(5)), beneficiary },

CheckAccount should now be specified as Local

The xcm_builder::CurrencyAdapter now accepts a Get<Option<(AccountId, MintLocation)>> instead of just Get<Option<AccountId>> as its fifth type parameter. Assuming you provide this through a parameter_types!, you will want to change it accordingly:

parameter_types! {
	pub CheckAccount: AccountId = XcmPallet::check_account();
}

becomes

parameter_types! {
	pub CheckAccount: (AccountId, MintLocation) = (XcmPallet::check_account(), MintLocation::Local);
}

If you have no checking accounts, then Nothing will still work.

Location Inversion Removed

The act of inverting a location is now largely contained within reanchoring functions and is no longer exposed in downstream APIs. InvertLocation trait and LocationInverter struct are removed. Those config type items now become UniversalLocation: Get<InteriorMultiLocation>.

The UniversalLocation is similar but different to the old Ancestry: it is the location of the chain within Universal Consensus, including the top-level consensus system. It should always begin with a GlobalConsensus junction.

Old code, for a Kusama parachain:

parameter_types! {
	pub Ancestry: MultiLocation = Parachain(ParachainInfo::parachain_id().into()).into();
}
impl xcm_executor::Config for XcmConfig {
	// snip
	type LocationInverter = LocationInverter<Ancestry>;
}

New code for that same parachain:

parameter_types! {
	pub UniversalLocation: InteriorMultiLocation = (
		GlobalConsensus(Kusama),
		Parachain(ParachainInfo::parachain_id().into(),
	).into();
}
impl xcm_executor::Config for XcmConfig {
	// snip
	type UniversalLocation = UniversalLocation;
}

Note that it is now a InteriorMultiLocation, and that the location includes the relay-chain consensus, since it is the parent consensus.

XcmContext

Several APIs now accept an additional context: &XcmContext parameter (except in the case of withdraw_asset, which accepts a maybe_context: Option<&XcmContext>).

This parameter must be added, but its value may be ignored.

XcmHash

SendXcm and ExportXcm must both provide an XcmHash value back from deliver function if it returns Ok. It is up to the trait implementation exactly how this hash is determined, however in general it should identify the message which was sent and should be identifiable on both sides of the transport. Generally this will mean hashing the data which is encoded and set on the wire (or the common part of it, if it changes over the transport).

XcmExecutor

XcmExecutor::execute has now been renamed to XcmExecutor::bench_process and is only enabled for benchmarks. Prior to this it has collided with the trait function ExecuteXcm::execute.

Delivery Pricing

xcm_sender::ChildParachainRouter now takes an extra generic parameter to indicate the price for sending a message using UMP. For previous behaviour, use () for this new parameter.

Old code:

pub type XcmRouter = (
	xcm_sender::ChildParachainRouter<Runtime, XcmPallet>,
);

New code:

pub type XcmRouter = (
	xcm_sender::ChildParachainRouter<Runtime, XcmPallet, ()>,
);

The struct xcm_sender::ConstantPrice can be used in place of () in order to allow for the price to be configured simply with a constant. Or, of course, you can create your own implementation for maximum flexibility over delivery pricing.

Cumulus

cumulus_primitives_utility::ParentAsUmp now takes an extra generic parameter to indicate the price for sending a message using UMP. For previous behaviour, use () for this new parameter.

Old code:

pub type XcmRouter = (
	cumulus_primitives_utility::ParentAsUmp<ParachainSystem, PolkadotXcm>,
);

New code:

pub type XcmRouter = (
	cumulus_primitives_utility::ParentAsUmp<ParachainSystem, PolkadotXcm, ()>,
);

Similarly, cumulus_pallet_xcmp_queue::Config has an additional type PriceForSiblingDelivery. This may also be set to () for previous behaviour:

Old code:

impl cumulus_pallet_xcmp_queue::Config for Runtime {
	// snip
}

New code:

impl cumulus_pallet_xcmp_queue::Config for Runtime {
	// snip
	type PriceForSiblingDelivery = ();
}

Renames & Minor Refactors

Renames:

• v2::Instruction::Transact::origin_type is renamed to become v3::Instruction::Transact::origin_kind.
• v2::Instruction::QueryHolding is renamed to become v3::Instruction::ReportHolding.
• xcm_builder::ConvertedConcreteAssetId is renamed to ConvertedConcreteId.
• xcm_builder::ConvertedAbstractAssetId is renamed to ConvertedAbstractId.

Refactors:

• query_id/dest/max_response_weight is now combined into a single struct QueryResponseInfo. Therefore:
  • v2::QueryHolding{query_id, dest, max_response_weight, assets} becomes v3::Instruction::ReportHolding{response_info, assets}.
  • v2::ReportError{query_id, dest, max_response_weight} becomes v3::Instruction::ReportError(response_info).
  • Junction::into is not a const fn any longer. Use Junction::into_location.
• FilterAssetLocation is now replaced by ContainsPair<MultiAsset, MultiLocation>.

[KiChjang]

Does the inclusion of a new ConversionError sound good to you? This will be an error enum that gets returned by TryFrom conversions upon failure. If so, do we retroactively introduce them to all versions, or only starting from v3?

[gavofyork]

No alterations to v2's standard are possible now.

[gavofyork]

Just weights to do now... cc @shawntabrizi @KiChjang

[KiChjang]

@gavofyork Weights are actually being worked on in #3940. OK to merge this as-is and let that PR handle it? Shawn has no problems with doing so.

[gavofyork]

Sure.

[gavofyork]

May not want to merge quite yet though... might be better sitting in its own branch until XCM v3 is ready.

[KiChjang]

While merging master, this is one of the things that I've changed in order to get the code to compile, and I'm not certain whether this line (and this file in general, especially the part where I added depositable_count to Config::worst_case_holding) is the correct fix.

May want to review this a little closer.

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/an-xcm3-experiment-kusama-treasury-account-taking-stake-in-the-kabocha-parachain/2127/1

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/substrate-developer-newsletter-february-2023-notable-updates-for-frame-devs-new-deep-dives-released-calling-builders-to-apply-to-rfps-and-bounties/2137/1