Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

genesis-builder: implemented for all runtimes #1492

Merged
merged 5 commits into from
Sep 27, 2023
Merged

genesis-builder: implemented for all runtimes #1492

merged 5 commits into from
Sep 27, 2023

Conversation

michalkucharczyk
Copy link
Contributor

This PR implements GenesisBuilder API for all the runtimes in polkadot repo.

Step towards: #25

@michalkucharczyk
Copy link
Contributor Author

It is extracted part of #1256

@michalkucharczyk michalkucharczyk requested a review from a team September 11, 2023 12:00
@paritytech-ci paritytech-ci requested review from a team September 11, 2023 12:01
@michalkucharczyk michalkucharczyk added the T4-runtime_API This PR/Issue is related to runtime APIs. label Sep 11, 2023
ordian
ordian approved these changes Sep 11, 2023
View reviewed changes
polkadot/runtime/polkadot/Cargo.toml Outdated Show resolved Hide resolved
polkadot/runtime/kusama/Cargo.toml Outdated Show resolved Hide resolved
@michalkucharczyk @ordian
Apply suggestions from code review
05e2184 
Co-authored-by: ordian <write@reusable.software>
bkchr
bkchr approved these changes Sep 26, 2023
View reviewed changes
Copy link
Member

@bkchr bkchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove everywhere the disable-genesis-builder. Otherwise looks good.

polkadot/runtime/polkadot/Cargo.toml Outdated
Comment on lines 323 to 325
# When enabled, the GenesisBuilder API will not be supported, GenesisConfig shall be
# stripped from the final binary
disable-genesis-builder = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we need them. Please remove them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the size is not a concern, then we can keep serde + genesis builder within binary. On the other hand it is not used on-chain. Less code, maybe better - I mean some audits, attack vectors, etc...

Anyway, I am also not sure, so I'll remove it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7d2b3f1

@michalkucharczyk michalkucharczyk merged commit 5a2833c into master Sep 27, 2023
6 checks passed
@michalkucharczyk michalkucharczyk deleted the mku-genesis-builder-for-runtimes branch September 27, 2023 09:59
ordian added a commit that referenced this pull request Sep 27, 2023
@ordian
merge master and resolve conflicts (v8)
6fb4b7f 
* master: (61 commits)
  OpenGov in Westend and Rococo (#1177)
  Associated type Hasher for `QueryPreimage`, `StorePreimage` and `Bounded` (#1720)
  Migrate polkadot-primitives to v6 (#1543)
  genesis-builder: implemented for all runtimes (#1492)
  `BlockId` removal: `tx-pool` refactor (#1678)
  Bump directories from 4.0.1 to 5.0.1 (#1656)
  Allow debug_assertions in short-benchmarks CI job (#1711)
  chainHead/storage: Fix storage iteration using the query key (#1665)
  Implement more useful traits in `Slot` type (#1595)
  Make downloads in parallel and give more time to complete (#1699)
  Bump actions/checkout from 4.0.0 to 4.1.0 (#1688)
  contracts: Fix incorrect storage alias in mirgration (#1687)
  Fix documentation about justification and `finalized == true` requirement (#1607)
  tweak pallet macro (genesis_config etc) to cater for RA users as well. (#1689)
  Uncoupling pallet-xcm from frame-system's RuntimeCall (#1684)
  Bump aes-gcm from 0.10.2 to 0.10.3 (#1681)
  docs / Update PR template to reflect monorepo (#1674)
  update contributing guide and ui-tests scripts (#1668)
  pallet epm: add `TrimmingStatus` to the mined solution (#1659)
  Update HRMP pallet benchmarking to use benchmarks v2 (#1676)
  ...
@michalkucharczyk michalkucharczyk mentioned this pull request Sep 29, 2023
Closed
ordian added a commit that referenced this pull request Oct 10, 2023
@ordian
Merge branch 'tsv-disabling-node-side' into tsv-disabling-backing
26866fa 
* tsv-disabling-node-side: (69 commits)
  runtime-api: cleanup after v7 stabilization (#1729)
  Move requests-responses and polling from `ChainSync` to `SyncingEngine` (#1650)
  Add custom error message for `StorageNoopGuard` (#1727)
  Clarify docs
  cargo fmt
  add a CAVEAT comment
  implement disabled_validators correctly
  remove unnecessary hash string (#1722)
  OpenGov in Westend and Rococo (#1177)
  Associated type Hasher for `QueryPreimage`, `StorePreimage` and `Bounded` (#1720)
  Migrate polkadot-primitives to v6 (#1543)
  genesis-builder: implemented for all runtimes (#1492)
  `BlockId` removal: `tx-pool` refactor (#1678)
  Bump directories from 4.0.1 to 5.0.1 (#1656)
  Allow debug_assertions in short-benchmarks CI job (#1711)
  chainHead/storage: Fix storage iteration using the query key (#1665)
  Implement more useful traits in `Slot` type (#1595)
  Make downloads in parallel and give more time to complete (#1699)
  Bump actions/checkout from 4.0.0 to 4.1.0 (#1688)
  contracts: Fix incorrect storage alias in mirgration (#1687)
  ...
@kiltbot kiltbot mentioned this pull request Oct 19, 2023
Closed
@Polkadot-Forum
Copy link

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-release-analysis-v1-2-0/4451/1

svyatonik added a commit to svyatonik/runtimes that referenced this pull request Nov 7, 2023
@svyatonik
adapt paritytech/polkadot-sdk#1492
7104fcd
svyatonik added a commit to svyatonik/runtimes that referenced this pull request Nov 7, 2023
@svyatonik
adapt paritytech/polkadot-sdk#1492
0ad007f
@radkomih radkomih mentioned this pull request Nov 9, 2023
Closed
@ahmadkaouk ahmadkaouk mentioned this pull request Nov 16, 2023
Closed
@dnjscksdn98 dnjscksdn98 mentioned this pull request Jan 12, 2024
Merged
13 tasks
This was referenced Mar 4, 2024
Closed
Merged
bgallois pushed a commit to duniter/duniter-polkadot-sdk that referenced this pull request Mar 25, 2024
@michalkucharczyk @ordian
genesis-builder: implemented for all runtimes (paritytech#1492)
7f9c125 
This PR implements [`GenesisBuilder`
API](https://github.com/paritytech/polkadot-sdk/blob/a414ea7515c9cdc81f1d12410e646afc148250e8/substrate/primitives/genesis-builder/src/lib.rs#L38)
for all the runtimes in polkadot repo.

Step towards: paritytech#25

---------

Co-authored-by: ordian <write@reusable.software>
@ggwpez ggwpez mentioned this pull request Mar 26, 2024
Merged
@Dinonard Dinonard mentioned this pull request Mar 28, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Apr 8, 2024
@ggwpez @liamaharon
[FRAME] Runtime Omni Bencher (#3512)
9543d31 
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes #3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
#1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes #2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
#3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Ank4n pushed a commit that referenced this pull request Apr 9, 2024
@ggwpez @liamaharon @Ank4n
[FRAME] Runtime Omni Bencher (#3512)
65a5d7b 
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes #3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
#1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes #2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
#3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
dharjeezy pushed a commit to dharjeezy/polkadot-sdk that referenced this pull request Apr 9, 2024
@ggwpez @liamaharon @dharjeezy
[FRAME] Runtime Omni Bencher (paritytech#3512)
63a20c3 
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes paritytech#3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
paritytech#1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes paritytech#2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
paritytech#3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
AurevoirXavier added a commit to darwinia-network/darwinia that referenced this pull request Jun 21, 2024
@AurevoirXavier
paritytech/polkadot-sdk#1492
720fd47
AurevoirXavier added a commit to darwinia-network/darwinia that referenced this pull request Jun 28, 2024
@AurevoirXavier @boundless-forest
To polkadot-v1.7.2 (#1516)
61e127a 
* Setup deps

* Remove Koi from account migration test

* paritytech/polkadot-sdk#1495

* Bump

* paritytech/polkadot-sdk#1524

* !! paritytech/polkadot-sdk#1363

* paritytech/polkadot-sdk#1492

* paritytech/polkadot-sdk#1911

* paritytech/polkadot-sdk#1900

Signed-off-by: Xavier Lau <xavier@inv.cafe>

* paritytech/polkadot-sdk#1661

* paritytech/polkadot-sdk#2144

* paritytech/polkadot-sdk#2048

* paritytech/polkadot-sdk#1672

* paritytech/polkadot-sdk#2303

* paritytech/polkadot-sdk#1256

* Remove identity and vesting

* Fixes

* paritytech/polkadot-sdk#2657

* paritytech/polkadot-sdk#1313

* paritytech/polkadot-sdk#2331

* paritytech/polkadot-sdk#2409 part.1

* paritytech/polkadot-sdk#2767

* paritytech/polkadot-sdk#2521

Signed-off-by: Xavier Lau <xavier@inv.cafe>

* paritytech/polkadot-sdk#1222

* paritytech/polkadot-sdk#1234 part.1

* Satisfy compiler

* XCM V4 part.1

* paritytech/polkadot-sdk#1246

* Remove pallet-democracy part.1

* paritytech/polkadot-sdk#2142

* paritytech/polkadot-sdk#2428

* paritytech/polkadot-sdk#3228

* XCM V4 part.2

* Bump

* Build all runtimes

* Build node

* Remove pallet-democracy

Signed-off-by: Xavier Lau <xavier@inv.cafe>

* Format

* Fix pallet tests

* Fix precompile tests

* Format

* Fixes

* Async, remove council, common pallet config

* Fix `ethtx-forward` test case (#1519)

* Fix ethtx-forward tests

* Format

* Fix following the review

* Fixes

* Fixes

* Use default impl

* Benchmark helper

* Bench part.1

* Bench part.2

* Bench part.3

* Fix all tests

* Typo

* Feat

* Fix EVM tracing build

* Reuse upstream `proof_size_base_cost()` (#1521)

* Format issue

* Fixes

* Fix CI

---------

Signed-off-by: Xavier Lau <xavier@inv.cafe>
Co-authored-by: Bear Wang <boundless.forest@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lexnv lexnv lexnv approved these changes

@davxy davxy davxy approved these changes

@bkchr bkchr bkchr approved these changes

@skunert skunert skunert approved these changes

@ordian ordian ordian approved these changes

Assignees
No one assigned
Labels
T4-runtime_API This PR/Issue is related to runtime APIs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@michalkucharczyk @Polkadot-Forum @ordian @skunert @bkchr @davxy @lexnv