bench all weekly - and fix for pallet_multisig lib

[mordamax]

Closes #6196
Closes #7204

Example of PR: #6816

Every sunday 01:00 AM it's going to start to benchmark (with /cmd bench) all runtimes and all pallets
Then diff total will be pushed to a branch and PR open,. I assume review-bot is going assign required reviewers per changed files

I afraid each weeks will be too much to review & merge, but we can adjust later

Bonus: fix for pallet_multisig lib and substrate/.maintain/frame-weight-template.hbs , which didn't let to compile new weights

[bkchr]

The pr should be merged automatically if in the difference is not that much more. The cherry on top would be to compare multiple weeks and if it goes up, to create some sort of alarm.

[mordamax]

The pr should be merged automatically if in the difference is not that much more. The cherry on top would be to compare multiple weeks and if it goes up, to create some sort of alarm.

There's only auto-merge feature available for automatic merging, but it will still require some (even formal) approvals, to avoid giving too much permissions for bot which would have to overcome at least github minimum requirements if we would like it to merge without approvals. Obviously we don't want it.

Given the fact that this is going to produce a diff in many folders, naturally it will assign too many people/teams for review (via Review-Bot), so I will turn this PR to draft again, and I'm going to think how to make this requirement more loose, as well as how to notify the owners of the weight files, if this went too much up

[paritytech-workflow-stopper]

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12939954000
Failed job name: check-runtime-migration

[paritytech-workflow-stopper]

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12939954000
Failed job name: check-runtime-migration

[mordamax]

/cmd bench --pallet pallet_multisig --clean

[github-actions]

Command "bench --pallet pallet_multisig --clean" has started 🚀 See logs here

[iulianbarbu]

LGTM overall! Have a few questions. Disclaimer: I do not have enough context on benchmarking.

1. Why don't we use bench-omni?

2. When it comes to reviewing weights changes to all pallets/all runtimes, should we do checks against weights we committed a few weeks back too (e.g. 1/2 months ago)? I am not sure which are the thresholds we'll be accepting for regressions, and I am curious how we'd defend against constant increases below the thresholds week after week.

3. I am seeing we run subweight compare commits somewhere in the workflow. Can the report be added to the PR - can be done in a follow up too? I am thinking such report might be useful for the comparisons with older weights as I suggested at point 2.

Curious about your thoughts.

[mordamax]

LGTM overall! Have a few questions. Disclaimer: I do not have enough context on benchmarking.

1. Why don't we use bench-omni?

we will switch to bench-omni as default bench as soon as the #6797 fix is confirmed 👍

2. When it comes to reviewing weights changes to all pallets/all runtimes, should we do checks against weights we committed a few weeks back too (e.g. 1/2 months ago)? I am not sure which are the thresholds we'll be accepting for regressions, and I am curious how we'd defend against constant increases below the thresholds week after week.

Yep, we discussed this too, and the way to go could be comparing with the previous stable release tag. But need to explore this more

3. I am seeing we run subweight compare commits somewhere in the workflow. Can the report be added to the PR - can be done in a follow up too? I am thinking such report might be useful for the comparisons with older weights as I suggested at point 2.

Yes, my plan was to replace this with link to https://weights.tasty.limo/compare - this shows a diff dynamically to a given branch. so we could do 2 links - 1) compare with master, 2) compare with prev release and add this to a body of a PR