CI short benchmarking with `frame-omni-bencher` #4405

bkontur · 2024-05-07T22:18:11Z

TODO

setup and fix get_preset for SP testnets - Moved presets to the testnet runtimes #5327
- BridgeHubRococo is not working: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6195778
- collectives-westend: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6193730
- contracts-rococo: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6193731
investigate for all short-benchmark-* replacing ./artifacts/polkadot and ./artifacts/polkadot-parachain with ./artifacts/frame-omni-bencher
- than we won't need build-short-benchmark and build-short-benchmark-cumulus, but we need to build all runtimes with --features runtime-benchmarks as for quick-benchmarks-omni
remove from github:
- continuous-integration/gitlab-build-short-benchmark
- continuous-integration/gitlab-build-short-benchmark-cumulus

Follow-up TODO:

remove gitlab pipelines (where everything works)
enable: frame-omni-bencher short checks #5268

Possible follow-ups

remove benchmarking code from polkadot / polkadot-parachain binaries
simplify gitlab short-benchmarks with matrix-like syntax

.github/workflows/release-30_publish_release_draft.yml

bkontur · 2024-05-22T06:46:54Z

@ggwpez
Currently, we can use frame-omni-bencher either with pre-generated chain-spec with runtime code included or with pre-compiled runtime wasm blob which has GenesisBuilder::get_preset defined.

I am thinking about another possibility that could be maybe very useful,
what if we add support for frame-omni-bencher to run with both combined:

pre-compiled runtime wasm blob which has GenesisBuilder::get_preset, which could generate default genesis spec
and then merge/override default genesis spec with provided json chain-spec as argument (without runtime code attribute)

./target/release/frame-omni-bencher v1 benchmark pallet \ 
    --runtime ./target/release/wbuild/bridge-hub-rococo-runtime/bridge_hub_rococo_runtime.compact.compressed.wasm
    --chain some-chain-spec-with-overrides.json
    --all --steps 2 --repeat 1

This would add flexibility, e.g. when you want to tune benchmarking with different genesis, you just change json and not need to compile wasm again and so on.

I checked the code, and it should be that hard, maybe just to add new GenesisBuilder type e.g. RuntimeWithSpecOverride or something like that.

ggwpez · 2024-05-22T13:51:40Z

This would add flexibility, e.g. when you want to tune benchmarking with different genesis, you just change json and not need to compile wasm again and so on.

Yea it could be useful. But should probably rename the argument from --chain to --genesis-spec since i want to deprecate that. Its just not a good name in general 🙈

bkontur · 2024-05-29T08:01:01Z

sync with: #4617
cc: @alvicsam

marking it as release-able, attaching the same version number that is attached to other binaries such as `polkadot` and `polkadot-parachain`. I have more thoughts about the version number, though. The chain-spec builder is mainly a user of the `sp-genesis-builder` api. So the versioning should be such that it helps users know give a version of `sp-genesis-builder` in their runtime, which version of `chain-spec-builder` should they use? With this, we can possibly alter the version number to always match `sp-genesis-builder`. Fixes #4352 - [x] Add to release artifacts ~~similar to #4405 done here: #4557 --------- Co-authored-by: Branislav Kontur <bkontur@gmail.com>

marking it as release-able, attaching the same version number that is attached to other binaries such as `polkadot` and `polkadot-parachain`. I have more thoughts about the version number, though. The chain-spec builder is mainly a user of the `sp-genesis-builder` api. So the versioning should be such that it helps users know give a version of `sp-genesis-builder` in their runtime, which version of `chain-spec-builder` should they use? With this, we can possibly alter the version number to always match `sp-genesis-builder`. Fixes paritytech#4352 - [x] Add to release artifacts ~~similar to paritytech#4405 done here: paritytech#4557 --------- Co-authored-by: Branislav Kontur <bkontur@gmail.com>

- Part of paritytech/ci_cd#1006 - Closes: paritytech/ci_cd#1010 - Related: #4405 - Possibly affecting how frame-omni-bencher works on different runtimes: #5083 Currently works in parallel with gitlab short benchmarks. Triggered only by adding `GHA-migration` label to assure smooth transition (kind of feature-flag). Later when tested on random PRs we'll remove the gitlab and turn on by default these tests --------- Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

- Part of paritytech/ci_cd#1006 - Closes: paritytech/ci_cd#1010 - Related: paritytech#4405 - Possibly affecting how frame-omni-bencher works on different runtimes: paritytech#5083 Currently works in parallel with gitlab short benchmarks. Triggered only by adding `GHA-migration` label to assure smooth transition (kind of feature-flag). Later when tested on random PRs we'll remove the gitlab and turn on by default these tests --------- Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

- Part of https://github.com/paritytech/ci_cd/issues/1006 - Closes: https://github.com/paritytech/ci_cd/issues/1010 - Related: paritytech#4405 - Possibly affecting how frame-omni-bencher works on different runtimes: paritytech#5083 Currently works in parallel with gitlab short benchmarks. Triggered only by adding `GHA-migration` label to assure smooth transition (kind of feature-flag). Later when tested on random PRs we'll remove the gitlab and turn on by default these tests --------- Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

paritytech-cicd-pr · 2024-09-01T21:33:55Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 3/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7209519

clippy clippy clippy Fix

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Fix changes_to_storage Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Better error messages Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> genesis-preset flag Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Bench runner fix genesis storage Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> clippy

bkontur · 2024-09-13T12:40:44Z

closing in favor of:
#5327
#5655
#5706

bkontur added the R0-silent Changes should not be mentioned in any release notes label May 14, 2024

kianenigma mentioned this pull request May 20, 2024

Publish chain-spec-builder #4518

Merged

1 task

kianenigma reviewed May 21, 2024

View reviewed changes

.github/workflows/release-30_publish_release_draft.yml Outdated Show resolved Hide resolved

bkontur mentioned this pull request May 22, 2024

Update to SDK 1.13 polkadot-fellows/runtimes#322

Merged

4 tasks

bkontur force-pushed the bko-bencher branch from ee4f181 to 0beec71 Compare May 29, 2024 07:49

bkontur changed the title ~~Add frame-omni-bencher to the CI artifacts and release binary~~ CI short benchmarking with frame-omni-bencher May 29, 2024

alvicsam mentioned this pull request May 29, 2024

[WIP][ci] Move short benchmarks to GHA #4617

Closed

bkontur mentioned this pull request Jun 10, 2024

asset-hub-rococo: genesis config presets added #3996

Merged

serban300 mentioned this pull request Jul 23, 2024

Omni-node: TODOs #4966

Closed

serban300 mentioned this pull request Jul 31, 2024

Omni-node: add solochain with manual consensus logic #5027

Closed

This was referenced Aug 6, 2024

frame-omni-bencher short checks #5267

Closed

frame-omni-bencher short checks #5268

Merged

bkontur force-pushed the bko-bencher branch from 995fd2f to a47aff4 Compare August 12, 2024 10:35

serban300 mentioned this pull request Aug 12, 2024

polkadot-parachain: Add omni-node variant with u64 block number #5269

Merged

bkontur mentioned this pull request Aug 12, 2024

Moved presets to the testnet runtimes #5327

Open

8 tasks

bkontur force-pushed the bko-bencher branch from a47aff4 to 769a244 Compare August 13, 2024 08:54

bkontur mentioned this pull request Aug 16, 2024

frame-omni-bencher fix for --repeat > 1 #5083

Draft

bkontur force-pushed the bko-bencher branch 2 times, most recently from e1ff127 to 6135102 Compare September 1, 2024 19:33

bkontur force-pushed the bko-bencher branch 2 times, most recently from 4670223 to 425fd15 Compare September 1, 2024 21:01

bkontur force-pushed the bko-bencher branch 3 times, most recently from caf2442 to 5856dca Compare September 2, 2024 19:36

kianenigma mentioned this pull request Sep 3, 2024

polkadot-omni-node: Fully remove benchmark sub-command #5564

Open

bkontur and others added 8 commits September 9, 2024 16:12

Set get_preset for AssetHubRococo

80fb7ab

Set get_preset for AssetHubWestend

5a39b4d

Set get_preset for BridgeHubRococo

0201876

clippy clippy clippy Fix

".git/.scripts/commands/fmt/fmt.sh"

8fb4dfd

Re-use sp_genesis_builder::DEV_RUNTIME_PRESET

bc435cd

Enable also new GHA for short-benchmarks

cf41627

WIP: std::path::absolute to std::fs::canonicalize

eac9676

bkontur force-pushed the bko-bencher branch from 5856dca to eac9676 Compare September 9, 2024 14:13

bkontur closed this Sep 13, 2024

bkontur deleted the bko-bencher branch September 13, 2024 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI short benchmarking with `frame-omni-bencher` #4405

CI short benchmarking with `frame-omni-bencher` #4405

bkontur commented May 7, 2024 •

edited

Loading

bkontur commented May 22, 2024

ggwpez commented May 22, 2024

bkontur commented May 29, 2024

paritytech-cicd-pr commented Sep 1, 2024

bkontur commented Sep 13, 2024

CI short benchmarking with frame-omni-bencher #4405

CI short benchmarking with frame-omni-bencher #4405

Conversation

bkontur commented May 7, 2024 • edited Loading

TODO

Follow-up TODO:

Possible follow-ups

bkontur commented May 22, 2024

ggwpez commented May 22, 2024

bkontur commented May 29, 2024

paritytech-cicd-pr commented Sep 1, 2024

bkontur commented Sep 13, 2024

CI short benchmarking with `frame-omni-bencher` #4405

CI short benchmarking with `frame-omni-bencher` #4405

bkontur commented May 7, 2024 •

edited

Loading