Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Include StorageInfo in Benchmarking Pipeline #9090

Merged
60 commits merged into from
Jul 7, 2021

Conversation

shawntabrizi
Copy link
Member

@shawntabrizi shawntabrizi commented Jun 12, 2021

This PR is the first step to fully integrating StorageInfo into the benchmarking pipeline.

As it sits, this PR now gives comment metadata to all the benchmarks, taking note of which storage items have been touched while running a benchmark.

Here is an example:

	// Storage: Assets Asset (r:1 w:1)
	// Storage: Assets Account (r:2 w:2)
	// Storage: System Account (r:1 w:1)
	fn transfer_keep_alive() -> Weight {
		(121_000_000 as Weight)
			.saturating_add(T::DbWeight::get().reads(4 as Weight))
			.saturating_add(T::DbWeight::get().writes(4 as Weight))
	}

These comments are now added automatically to the benchmarking through the handlebars template.

This information is derived from the StorageInfo and the tracked prefixes from the benchmarking db.

Further work will be included in the future which will pull more storage information from the StorageInfo such as the max size and length for PoV tracking.

polkadot companion: paritytech/polkadot#3424

@github-actions github-actions bot added the A3-in_progress Pull request is in progress. No review needed at this stage. label Jun 12, 2021
@shawntabrizi shawntabrizi marked this pull request as ready for review June 29, 2021 22:36
@shawntabrizi shawntabrizi requested a review from a team as a code owner June 29, 2021 22:36
@github-actions github-actions bot added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Jun 29, 2021
@shawntabrizi shawntabrizi added B7-runtimenoteworthy C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. labels Jun 29, 2021
@shawntabrizi
Copy link
Member Author

/benchmark runtime pallet pallet_balances

@parity-benchapp
Copy link

parity-benchapp bot commented Jun 29, 2021

Finished benchmark for branch: shawntabrizi-benchmarking-storage-info

Benchmark: Benchmark Runtime Pallet

cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results

Pallet: "pallet_balances", Extrinsic: "transfer", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 92.14
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 92.14
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "transfer_keep_alive", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 67.49
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 67.49
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "set_balance_creating", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 36.34
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 36.34
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "set_balance_killing", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 44.07
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 44.07
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "force_transfer", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 91.03
µs

Reads = 2
Writes = 2
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 91.03
µs

Reads = 2
Writes = 2
Pallet: "pallet_balances", Extrinsic: "transfer_all", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 84.07
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 84.07
µs

Reads = 1
Writes = 1

Copy link
Contributor

@TriplEight TriplEight left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I wonder tho how much does it take.

Copy link
Contributor

@gui1117 gui1117 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

@shawntabrizi
Copy link
Member Author

/benchmark runtime pallet frame_system

1 similar comment
@shawntabrizi
Copy link
Member Author

/benchmark runtime pallet frame_system

@parity-benchapp
Copy link

Starting benchmark for branch: shawntabrizi-benchmarking-storage-info (vs master)

Comment will be updated.

@shawntabrizi
Copy link
Member Author

/benchmark runtime pallet frame_system

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 7, 2021

Benchmark Runtime Pallet for branch "shawntabrizi-benchmarking-storage-info" with command cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
Pallet: "frame_system", Extrinsic: "remark", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    1.138
    + b        0
              µs

Reads = 0 + (0 * b)
Writes = 0 + (0 * b)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    b   mean µs  sigma µs       %
    0     1.033     0.003    0.2%
78643     1.128     0.005    0.4%
157286     1.129     0.003    0.2%
235929     1.154     0.003    0.2%
314572     1.142     0.003    0.2%
393215      1.15     0.003    0.2%
471858     1.155     0.002    0.1%
550501     1.151     0.002    0.1%
629144     1.157     0.003    0.2%
707787     1.143     0.005    0.4%
786430     1.149     0.004    0.3%
865073      1.15     0.009    0.7%
943716     1.166     0.011    0.9%
1022359     1.171     0.007    0.5%
1101002      1.19     0.007    0.5%
1179645     1.152     0.006    0.5%
1258288     1.158     0.004    0.3%
1336931     1.141     0.011    0.9%
1415574     1.142     0.011    0.9%
1494217     1.153     0.009    0.7%
1572860     1.136     0.015    1.3%
1651503     1.144     0.008    0.6%
1730146     1.162     0.013    1.1%
1808789     1.168     0.008    0.6%
1887432     1.147     0.013    1.1%
1966075     1.151     0.012    1.0%
2044718     1.149      0.01    0.8%
2123361     1.144     0.007    0.6%
2202004     1.156      0.01    0.8%
2280647     1.175     0.009    0.7%
2359290     1.167     0.012    1.0%
2437933     1.164     0.017    1.4%
2516576     1.152     0.017    1.4%
2595219     1.173     0.032    2.7%
2673862     1.168      0.01    0.8%
2752505     1.183     0.016    1.3%
2831148     1.167     0.021    1.7%
2909791      1.18     0.011    0.9%
2988434     1.177     0.029    2.4%
3067077     1.205     0.019    1.5%
3145720     1.175     0.012    1.0%
3224363     1.182     0.014    1.1%
3303006     1.178     0.019    1.6%
3381649     1.185     0.012    1.0%
3460292     1.189     0.017    1.4%
3538935       1.2     0.021    1.7%
3617578     1.198     0.022    1.8%
3696221     1.193      0.02    1.6%
3774864     1.177     0.021    1.7%
3853507     1.182     0.014    1.1%
3932150     1.188     0.029    2.4%

Quality and confidence:
param     error
b             0

Model:
Time ~=    1.131
    + b        0
              µs

Reads = 0 + (0 * b)
Writes = 0 + (0 * b)
Pallet: "frame_system", Extrinsic: "remark_with_event", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    13.03
    + b    0.001
              µs

Reads = 0 + (0 * b)
Writes = 0 + (0 * b)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    b   mean µs  sigma µs       %
    0     19.19     0.027    0.1%
78643     97.81     0.054    0.0%
157286     175.9     0.413    0.2%
235929     252.7     0.971    0.3%
314572     332.1     0.081    0.0%
393215     409.6     0.098    0.0%
471858     488.4     0.141    0.0%
550501     566.1     0.084    0.0%
629144     643.9     0.151    0.0%
707787     737.8     12.92    1.7%
786430     799.8     0.698    0.0%
865073     878.6     0.511    0.0%
943716     957.7     1.521    0.1%
1022359      1034     1.038    0.1%
1101002      1111     1.171    0.1%
1179645      1193     7.382    0.6%
1258288      1271     6.497    0.5%
1336931      1346     0.904    0.0%
1415574      1425     2.015    0.1%
1494217      1507     15.54    1.0%
1572860      1576     1.344    0.0%
1651503      1662     11.04    0.6%
1730146      1734      4.29    0.2%
1808789      1824     14.48    0.7%
1887432      1895     11.56    0.6%
1966075      1976     14.61    0.7%
2044718      2049     9.601    0.4%
2123361      2140     13.77    0.6%
2202004      2215     14.51    0.6%
2280647      2283     8.051    0.3%
2359290      2360     2.263    0.0%
2437933      2454     16.13    0.6%
2516576      2531     16.22    0.6%
2595219      2606     16.84    0.6%
2673862      2696      15.6    0.5%
2752505      2766     6.218    0.2%
2831148      2853     15.25    0.5%
2909791      2942     15.06    0.5%
2988434      3046     38.06    1.2%
3067077      3170     3.051    0.0%
3145720      3258     10.09    0.3%
3224363      3275     11.58    0.3%
3303006      3345     14.08    0.4%
3381649      3429      12.9    0.3%
3460292      3494     14.65    0.4%
3538935      3577     15.09    0.4%
3617578      3670     8.124    0.2%
3696221      3748     9.203    0.2%
3774864      3827     14.98    0.3%
3853507      3909     14.56    0.3%
3932150      4004     3.318    0.0%

Quality and confidence:
param     error
b             0

Model:
Time ~=    1.875
    + b    0.001
              µs

Reads = 0 + (0 * b)
Writes = 0 + (0 * b)
Pallet: "frame_system", Extrinsic: "set_heap_pages", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    1.959
              µs

Reads = 0
Writes = 1
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    1.959
              µs

Reads = 0
Writes = 1
Pallet: "frame_system", Extrinsic: "set_changes_trie_config", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    8.721
              µs

Reads = 1
Writes = 2
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    8.721
              µs

Reads = 1
Writes = 2
Pallet: "frame_system", Extrinsic: "set_storage", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=        0
    + i     0.78
              µs

Reads = 0 + (0 * i)
Writes = 0 + (1 * i)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    i   mean µs  sigma µs       %
    1     2.034     0.019    0.9%
   20     14.75     0.051    0.3%
   39     28.33     0.124    0.4%
   58     42.89     0.243    0.5%
   77     56.88     0.117    0.2%
   96     71.28     0.132    0.1%
  115     86.34      0.15    0.1%
  134     101.3     0.739    0.7%
  153     115.4     0.298    0.2%
  172     130.3     1.052    0.8%
  191     148.9     0.901    0.6%
  210     160.4     0.278    0.1%
  229     174.8     0.116    0.0%
  248     189.7     0.223    0.1%
  267     203.8     0.421    0.2%
  286     218.6      0.27    0.1%
  305     232.1     0.231    0.0%
  324     248.7     0.172    0.0%
  343     263.5     0.646    0.2%
  362     276.6      0.21    0.0%
  381     291.9     0.286    0.0%
  400     305.8     0.796    0.2%
  419     320.4     0.551    0.1%
  438     337.6     0.424    0.1%
  457     350.5     0.451    0.1%
  476     365.9     0.188    0.0%
  495     381.5     0.883    0.2%
  514     395.2     0.275    0.0%
  533       412       1.3    0.3%
  552     425.7     0.494    0.1%
  571     438.6     0.795    0.1%
  590     454.4      1.15    0.2%
  609     470.3     0.362    0.0%
  628     484.4     0.473    0.0%
  647     499.3     0.705    0.1%
  666     513.5     1.239    0.2%
  685     529.8     3.512    0.6%
  704     545.5     0.491    0.0%
  723     561.6     3.933    0.7%
  742     579.5     6.747    1.1%
  761     591.9     5.289    0.8%
  780     606.5      0.46    0.0%
  799     616.4     0.954    0.1%
  818     634.1     0.425    0.0%
  837     648.8     0.722    0.1%
  856     668.7     1.288    0.1%
  875     683.2     0.986    0.1%
  894     695.2     0.566    0.0%
  913     712.1     0.404    0.0%
  932     727.5     3.751    0.5%
  951     741.5     0.826    0.1%
  970       754     0.634    0.0%
  989       773     4.421    0.5%

Quality and confidence:
param     error
i             0

Model:
Time ~=        0
    + i    0.782
              µs

Reads = 0 + (0 * i)
Writes = 0 + (1 * i)
Pallet: "frame_system", Extrinsic: "kill_storage", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    0.149
    + i    0.596
              µs

Reads = 0 + (0 * i)
Writes = 0 + (1 * i)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    i   mean µs  sigma µs       %
    1     1.917      0.02    1.0%
   20     12.18     0.025    0.2%
   39     22.82     0.093    0.4%
   58        34      0.07    0.2%
   77     45.36     0.093    0.2%
   96     56.56     0.152    0.2%
  115      69.1     0.078    0.1%
  134     80.61     0.207    0.2%
  153     91.53     0.165    0.1%
  172       103     0.178    0.1%
  191     116.1     0.179    0.1%
  210     127.4     0.152    0.1%
  229     139.5     0.348    0.2%
  248     151.5     0.211    0.1%
  267     163.6     0.147    0.0%
  286     175.3     0.232    0.1%
  305     186.1     0.469    0.2%
  324     199.7     0.341    0.1%
  343       213     0.399    0.1%
  362       223     0.186    0.0%
  381     235.8      0.25    0.1%
  400     246.7     0.267    0.1%
  419       259     0.369    0.1%
  438     273.7     0.376    0.1%
  457       265     0.417    0.1%
  476     276.8     0.214    0.0%
  495     288.5      0.25    0.0%
  514     298.6     0.373    0.1%
  533     311.1     0.338    0.1%
  552     322.8     0.468    0.1%
  571     332.6     0.427    0.1%
  590     344.6     0.113    0.0%
  609     357.9     0.306    0.0%
  628     368.8      0.49    0.1%
  647     380.3     0.578    0.1%
  666     389.7     0.248    0.0%
  685     403.8     0.696    0.1%
  704     416.2      0.47    0.1%
  723     427.6     0.197    0.0%
  742     438.8     0.318    0.0%
  761     450.4     0.268    0.0%
  780     463.9     0.565    0.1%
  799     471.6     0.345    0.0%
  818     485.9      0.39    0.0%
  837     496.7     0.401    0.0%
  856     511.4     0.705    0.1%
  875     524.3     0.725    0.1%
  894     532.5     0.601    0.1%
  913     545.2     0.955    0.1%
  932     556.5      0.92    0.1%
  951     570.4     1.132    0.1%
  970     579.5     0.949    0.1%
  989     598.7     10.24    1.7%

Quality and confidence:
param     error
i             0

Model:
Time ~=    1.807
    + i    0.593
              µs

Reads = 0 + (0 * i)
Writes = 0 + (1 * i)
Pallet: "frame_system", Extrinsic: "kill_prefix", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    5.041
    + p    1.345
              µs

Reads = 0 + (0 * p)
Writes = 0 + (1 * p)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    p   mean µs  sigma µs       %
    1      6.17     0.052    0.8%
   20     30.74     0.107    0.3%
   39     55.55      0.18    0.3%
   58      82.9     0.204    0.2%
   77     102.6     0.217    0.2%
   96     127.9     0.308    0.2%
  115     151.6     0.397    0.2%
  134     178.3     0.824    0.4%
  153       205     0.338    0.1%
  172     262.8     15.29    5.8%
  191     259.8     8.515    3.2%
  210     296.3     5.414    1.8%
  229     321.3     7.239    2.2%
  248     338.5     3.553    1.0%
  267     367.7     3.425    0.9%
  286     398.2      3.97    0.9%
  305     420.6     8.043    1.9%
  324     457.9     2.324    0.5%
  343     470.2     4.255    0.9%
  362     503.8     1.408    0.2%
  381     530.5     1.113    0.2%
  400     548.6     1.091    0.1%
  419     583.2     0.954    0.1%
  438     616.7     1.384    0.2%
  457     610.1     1.286    0.2%
  476     652.7     1.674    0.2%
  495     670.9     1.519    0.2%
  514     689.7      2.31    0.3%
  533     723.5     1.103    0.1%
  552     740.8     2.007    0.2%
  571     766.2     3.878    0.5%
  590     804.8     9.825    1.2%
  609     814.1     5.975    0.7%
  628     849.5     1.486    0.1%
  647     886.9     4.459    0.5%
  666     897.7     3.001    0.3%
  685     925.7     0.775    0.0%
  704     965.2     7.897    0.8%
  723     959.8     2.263    0.2%
  742      1006      6.55    0.6%
  761      1029     2.934    0.2%
  780      1049     1.964    0.1%
  799      1088     11.37    1.0%
  818      1099     2.068    0.1%
  837      1128     2.341    0.2%
  856      1165     9.623    0.8%
  875      1171     2.188    0.1%
  894      1202     2.639    0.2%
  913      1231     5.237    0.4%
  932      1259     4.864    0.3%
  951      1287     4.469    0.3%
  970      1324     2.313    0.1%
  989      1332     10.31    0.7%

Quality and confidence:
param     error
p         0.001

Model:
Time ~=     7.87
    + p    1.343
              µs

Reads = 0 + (0 * p)
Writes = 0 + (1 * p)

Parity Bot added 2 commits July 7, 2021 13:13
…/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs
@shawntabrizi
Copy link
Member Author

bot merge

@ghost
Copy link

ghost commented Jul 7, 2021

Waiting for commit status.

@ghost
Copy link

ghost commented Jul 7, 2021

Merge aborted: Checks failed for 4e5c8a6

@shawntabrizi
Copy link
Member Author

bot merge

@ghost
Copy link

ghost commented Jul 7, 2021

Trying merge.

@ghost ghost merged commit 65ac8a5 into master Jul 7, 2021
@ghost ghost deleted the shawntabrizi-benchmarking-storage-info branch July 7, 2021 22:06
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. C1-low PR touches the given topic and has a low impact on builders. D2-breaksapi D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants