Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Enrich metadata with type information #8615

Merged
618 commits merged into from
Sep 15, 2021
Merged

Enrich metadata with type information #8615

618 commits merged into from
Sep 15, 2021

Conversation

ascjones
Copy link
Contributor

@ascjones ascjones commented Apr 13, 2021

⚠️ Breaking Metadata Change ⚠️

Closes #917 👀,

polkadot companion: paritytech/polkadot#3336

Migration guide: https://gist.github.com/ascjones/0d81a4c44e84cacd9f714cd34a6de823

Overview

  • Replaces in tree frame-metadata crate with https://github.com/paritytech/frame-metadata/
  • New v14 metadata structure defined here, which contains type information using scale-info
  • Annotates all SCALE encoded types with TypeInfo derive which will generate the static type information to be provided in the metadata.

See the latest generated metadata json for the default node-runtime here. This was generated by running the substrate node on this branch and then running https://github.com/ascjones/subsee to download and decode -> json serialize.

A prototype (Rust only for now) code generator exists here: https://github.com/ascjones/chameleon to verify whether there is enough information in the metadata to generate all types.

todo:

  • Generate errors metadata
  • Restore and fix all commented out tests
  • Figure out how to handle the existing DefaultByteGetter pattern for storage/constants for the new metadata.
  • scale-info 0.7 release containing all substrate compatibility features
    • update parity-common and frame-metadata to use this release
  • Decide how to handle custom (non derived) codec encoding (see todos).
  • Are the all the (required for scale-info type registration) vec! allocations okay which replace existing static array refs?
  • Clean up and unify namespacing of frame-metadata and scale-info types, e.g. remove all the v13s and add some imports to remove fully qualified types, which are there primarily for easy grepping during development.
  • Polkadot companion
  • Revisit handling of PhantomData in scale-info as it can in some cases require an unnecessary trait bound for the generic parameter. Can just skip those fields, or just revisit the approach, see https://github.com/paritytech/scale-info/blob/f0f689be810fc221a2aa9f015c0b0e1cd35fdd55/src/ty/mod.rs#L425
  • Benchmark:
    • Code size: by how much does this increase the size of the resulting wasm blob? If significantly we should consider storing the metadata somewhere else other than the runtime (see Rich type information for runtime metadata using `scale-info` #8370 (comment))
    • The size of the encoded metadata itself, the result of state_getMetadata, is there a benefit to oregenerating/compressing it? Relates to previous point.
    • Compilation times: how does having the extra type metadata affect the overall substrate compilation times? If a considerable slowdown then should we consider feature gating the metadata generation?
  • Replace existing frame-metadata crate with https://github.com/paritytech/frame-metadata
    • Remove existing crate and replace all usages
    • Publish frame-metadata to crates.io and replace git dependecies with that
  • Verify encoding and decoding of substrate calls/events/storage/constants using the metadata, e.g. using https://github.com/ascjones/chameleon
  • Note follow ups from review comments, possibly create an issue for them.

Follow ups:

@ascjones ascjones added the A3-in_progress Pull request is in progress. No review needed at this stage. label Apr 13, 2021
@olanod
Copy link
Contributor

olanod commented Apr 14, 2021

Would paritytech/frame-metadata#6 be to big/out of scope to be part of this new version?

@ascjones
Copy link
Contributor Author

Not sure, it certainly would be of enormous benefit in terms of the size of the metadata.

The consideration would be how to generate/distribute the version of the metadata which includes the docs, perhaps could be embedded with the clients which use it e.g. polkadot.js. So needs some thinking about which I've not done yet.

@bkchr
Copy link
Member

bkchr commented Apr 14, 2021

* Figure out how to handle the existing `DefaultByteGetter` pattern for storage/constants for the new metadata.

Just kill it. This never brought any benefit.

frame/balances/src/tests_composite.rs Outdated Show resolved Hide resolved
frame/balances/src/tests_composite.rs Outdated Show resolved Hide resolved
frame/balances/src/tests_composite.rs Outdated Show resolved Hide resolved
frame/balances/src/tests_composite.rs Outdated Show resolved Hide resolved
frame/contracts/src/lib.rs Outdated Show resolved Hide resolved
frame/support/procedural/src/pallet/expand/call.rs Outdated Show resolved Hide resolved
frame/support/procedural/src/pallet/expand/storage.rs Outdated Show resolved Hide resolved
frame/support/procedural/src/pallet/expand/event.rs Outdated Show resolved Hide resolved
frame/support/procedural/src/pallet/parse/event.rs Outdated Show resolved Hide resolved
frame/system/src/extensions/check_genesis.rs Outdated Show resolved Hide resolved
@olanod
Copy link
Contributor

olanod commented Apr 14, 2021

Not sure, it certainly would be of enormous benefit in terms of the size of the metadata.

The consideration would be how to generate/distribute the version of the metadata which includes the docs, perhaps could be embedded with the clients which use it e.g. polkadot.js. So needs some thinking about which I've not done yet.

How sensible is it to introduce new RPC calls, e.g. state_getMetadataCompact? and what about encoding of the results, is it set in stone somewhere that binary blobs have to be encoded in hex and no base64 for example? would be nicer if compact means is even more compact.

@real-felix
Copy link
Contributor

I've created a newtype as pub struct Set<T>(Vec<T>);, but when I try to use it in a storage, it is marked as Set, without the generic information. Thus I cannot use it with several different generic parameters in polkadot.js.

Is it related to this issue? Is it soon to be solved?

liuchengxu added a commit to canyon-network/canyon that referenced this pull request Oct 9, 2021
liuchengxu added a commit to canyon-network/canyon that referenced this pull request Oct 9, 2021
btwiuse pushed a commit to NFTT-studio/NFTMart-Node that referenced this pull request Dec 23, 2021
btwiuse pushed a commit to NFTT-studio/NFTMart-Node that referenced this pull request Dec 28, 2021
btwiuse added a commit to NFTT-studio/NFTMart-Node that referenced this pull request Jun 23, 2022
* WIP: Upgrade substrate

* Upgrade substrate to ce1746ddf6588c8f9ecfb77411c3cac7a2c09db3

* Update Cargo.lock

* Fix cargo check --features=runtime-benchmarks

* Upgrade substrate to e1ddbb3ffec3c8f05de6cb05d9f5f7a922788c3f

* Add `LightSyncStateExtension` to the chain spec as an extension

* Upgrade substrate to eb4de697ccd45948254db422abbcd735ae3c295a

* cargo fmt

* Upgrade substrate to ba153b9ae050eda022f002d74d76f98d1e339a81

Enrich metadata with type information paritytech/substrate#8615

* Upgrade substrate to 435f56edc14a3a7e895ff5370f6e5179dc547cc4

* Upgrade substrate to 88b4fc861129b63b445492e3088d7f12382f0128

* Upgrade substrate to ce3c31f2bfa7e10817a8a0833faddeaee818910d

* Upgrade substrate to 7fcebc86856164a346f6762d339e8cc3b6ec9d30

* Upgrade substrate to bf9683eee40f82cc4e01a05cd375b0e8bba3c8ef

* Upgrade substrate to 12b6441ef138303f50fa2fb20895fbf57c8e4735 (polkadot-v0.9.12-without-cache)

* Upgrade substrate to e09e70a3d14b3757de0be427c6ac22dfc6fcd498 (purestake/substrate-polkadot-0.9.12)

* bump to edition 2021; fix compile errors

* Upgrade substrate to 3a5aa8c1a48b015b12e2419fa76b2e2530b3ac75 (purestake/substrate-polkadot-0.9.13)

* Fix node compile error

* Downgrade substrate to c881b9c86df5d0e4e3607921598768a3ab8ccdc8 (PureStake/frontier#moonbeam-polkadot-v0.9.13)

* init pallet-evm from purestake

* Use purestake/substrate branch moonbeam-polkadot-v0.9.13 for consistency

* Add evm to genesis config

* CI: bump rust toolchain version to nightly-2021-12-13

* add pallet-ethereum: init

* install pallet ethereum and self contained unchecked extrinsic

* impl fp_rpc::EthereumRuntimeRPCApi<Block> for Runtime

* disable node/testing

* Add NetApi RPC endpoint

* debug1

* fix compile error

* add transaction converter

* use opaque::Block

* Add Eth RPC

* add pallet-deposit for transferring fund from substrate to evm

* prefund accounts

* set eth chainid to 12191

* set ss58prefix to 42

* use forked frontier

* bump frontier: add evm decimal converter

* patch ss58-registry

* Revert "set ss58prefix to 42"

This reverts commit c7d6c0d.

* bump frontier: add TokenDecimals support in pallet-evm

* pallet-evm: set CallOrigin/WithdrawOrigin to EnsureAddressTruncated

* drop FrontierBlockImport consensus support to maintain backward compatibility

* add pallet-template

* upgrade frontier(pallet-evm): add fn token_decimals() -> u8

* add standard precompiles

* add PalletTemplatePrecompile

* create pallet-template-precompiles under /precompiles

* deposit: add benchmarking

* add PalletTemplateInterface.sol

* pallet-template-precompiles: fix do_something() dispatch

* add pallet-evm-precompile-balances-erc20 from moonbeam

* pallet-template-precompiles: fix deps repo

* init withdraw-balance-precompiles

* add withdraw-balance-precompiles to runtime

* pallet-evm: impl custom FeeCalculator and GasWeightMapping

* withdraw balance debug

* withdraw-balance: fix compile errors

* add nftmart-nft-precompiles

* init nftmart-{auction,order}-precompiles

* update nftmart-* precompiles contract interface

* add pallet-nop and precompiles for debugging

* add frame-system-precompiles

* initialize precompile opcode placeholder to allow calling from other contracts

* nftmart-nft-precompiles: add arg parsing logic

* add account type conversion

* nftmart-nft-precompiles: dispatch_call: burn

* nftmart-nft-precompiles: dispatch_call: mint

* nftmart-nft-precompiles: dispatch_call: destroy_class

* nftmart-nft-precompiles: dispatch_call: create_class

* nftmart-nft-precompiles: dispatch_call: proxy_mint

* sp-arithmetic: turn off default-features to make compiler happy

* precompiles: add write methods for all nft pallets

* rewrite precompiles with BTreeMap

* Add return value to createClass()

* impl EvmData for NftItem

* add pallet-ethereum-chain-id to make chain_id configurable

* fix token decimals

* fix withdraw-balance precompile gasometer

* init pallet-staking-precompiles

* pallet-staking-precompiles: add chill()

* init pallet-identity

* pallet-staking-precompiles: add more methods

* nftmart-nft-precompiles: add read methods

* debug: disable non essential precompiles

* precompiles/utils: reduce warning

* pallet-identity-precompiles: add set_name()

* ethereum-chain-id: add setter method

* runtime: expose EthereumChainId's methods

* bump runtime version to 280

* housekeeping

* fix cargo bench --features runtime-benchmarks

* bump frontier

* fix readme

* add cargo doc workflow build step

* frontier: use associated const for TOKEN_DECIMAL

git grep github.com/purestake/substrate | cut -d : -f 1 | sort -u | xargs sed -i -e s,github.com/purestake/substrate,github.com/nftt-studio/substrate,g

Co-authored-by: navigaid <navigaid@gmail.com>
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A3-in_progress Pull request is in progress. No review needed at this stage. C1-low PR touches the given topic and has a low impact on builders. D2-breaksapi D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fully self-descriptive metadata types