Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Benchmarks script improvements #2214

Merged
merged 3 commits into from
Feb 21, 2023
Merged

Benchmarks script improvements #2214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c97d049
Benchmarks script improvements
muharem Feb 20, 2023
d3bf076
EOF
muharem Feb 20, 2023
a58daef
Merge remote-tracking branch 'origin/master' into muharem-benchmarks-…
Feb 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 15 additions & 60 deletions scripts/benchmarks-ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,70 +1,25 @@
#!/usr/bin/env bash

steps=50
repeat=20
category=$1
runtimeName=$2
artifactsDir=$3
steps=${4:-50}
repeat=${5:-20}

benchmarkOutput=./parachains/runtimes/$category/$runtimeName/src/weights
benchmarkRuntimeName="$runtimeName-dev"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muharem
what about renaming benchmarkRuntimeName -> benchmarkRuntimeChain ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runtimes have names. chains are run by nodes and run runtimes.


if [[ $runtimeName == "statemint" ]] || [[ $runtimeName == "statemine" ]] || [[ $runtimeName == "westmint" ]]; then
pallets=(
pallet_assets
pallet_balances
pallet_collator_selection
pallet_multisig
pallet_proxy
pallet_session
pallet_timestamp
pallet_utility
pallet_uniques
cumulus_pallet_xcmp_queue
frame_system
pallet_xcm_benchmarks::generic
pallet_xcm_benchmarks::fungible
)
elif [[ $runtimeName == "collectives-polkadot" ]]; then
pallets=(
pallet_alliance
pallet_balances
pallet_collator_selection
pallet_collective
pallet_multisig
pallet_proxy
pallet_session
pallet_timestamp
pallet_utility
cumulus_pallet_xcmp_queue
frame_system
)
elif [[ $runtimeName == "bridge-hub-kusama" ]] || [[ $runtimeName == "bridge-hub-polkadot" ]]; then
pallets=(
frame_system
pallet_balances
pallet_collator_selection
pallet_multisig
pallet_session
pallet_timestamp
pallet_utility
cumulus_pallet_xcmp_queue
pallet_xcm_benchmarks::generic
pallet_xcm_benchmarks::fungible
)
elif [[ $runtimeName == "bridge-hub-rococo" ]]; then
pallets=(
frame_system
pallet_balances
pallet_collator_selection
pallet_multisig
pallet_session
pallet_timestamp
pallet_utility
cumulus_pallet_xcmp_queue
pallet_xcm_benchmarks::generic
pallet_xcm_benchmarks::fungible
)
# Load all pallet names in an array.
pallets=($(
${artifactsDir}/polkadot-parachain benchmark pallet --list --chain="${benchmarkRuntimeName}" |\
tail -n+2 |\
cut -d',' -f1 |\
sort |\
uniq
))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is assuming that all pallets will get benchmarked. Just wondering if there's any pallets left that we currently still use hardcoded substrate weights from? I guess if you've run this for low repetitions and had no trouble then this should be fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it does not. if the pallet not listed in define_benchmarks macro, it wont be returned by the command


if [ ${#pallets[@]} -ne 0 ]; then
echo "[+] Benchmarking ${#pallets[@]} pallets for runtime $runtime"
else
echo "$runtimeName pallet list not found in benchmarks-ci.sh"
exit 1
Expand All @@ -75,10 +30,10 @@ do
output_file="${pallet//::/_}"
extra_args=""
# a little hack for pallet_xcm_benchmarks - we want to force custom implementation for XcmWeightInfo
if [[ "$pallet" == "pallet_xcm_benchmarks::generic" ]] || [[ "$pallet" == "pallet_xcm_benchmarks::fungible" ]]; then
if [[ "$pallet" == "pallet_xcm_benchmarks::generic" ]] || [[ "$pallet" == "pallet_xcm_benchmarks::fungible" ]]; then
output_file="xcm/$output_file"
extra_args="--template=./templates/xcm-bench-template.hbs"
fi
fi
$artifactsDir/polkadot-parachain benchmark pallet \
$extra_args \
--chain=$benchmarkRuntimeName \
Expand Down
18 changes: 11 additions & 7 deletions scripts/benchmarks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
#!/usr/bin/env bash

target=${1:-production}
steps=${2:-50}
repeat=${3:-20}

__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

${__dir}/benchmarks-ci.sh collectives collectives-polkadot target/production
${__dir}/benchmarks-ci.sh collectives collectives-polkadot target/$target $steps $repeat

${__dir}/benchmarks-ci.sh assets statemine target/production
${__dir}/benchmarks-ci.sh assets statemint target/production
${__dir}/benchmarks-ci.sh assets westmint target/production
${__dir}/benchmarks-ci.sh assets statemine target/$target $steps $repeat
${__dir}/benchmarks-ci.sh assets statemint target/$target $steps $repeat
${__dir}/benchmarks-ci.sh assets westmint target/$target $steps $repeat

${__dir}/benchmarks-ci.sh bridge-hubs bridge-hub-polkadot target/production
${__dir}/benchmarks-ci.sh bridge-hubs bridge-hub-kusama target/production
${__dir}/benchmarks-ci.sh bridge-hubs bridge-hub-rococo target/production
${__dir}/benchmarks-ci.sh bridge-hubs bridge-hub-polkadot target/$target $steps $repeat
${__dir}/benchmarks-ci.sh bridge-hubs bridge-hub-kusama target/$target $steps $repeat
${__dir}/benchmarks-ci.sh bridge-hubs bridge-hub-rococo target/$target $steps $repeat