Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add execution fuzzing #832

Merged
merged 25 commits into from
Dec 4, 2023
Merged

Add execution fuzzing #832

merged 25 commits into from
Dec 4, 2023

Conversation

Robbepop
Copy link
Member

@Robbepop Robbepop commented Dec 4, 2023

This also adds a 2 minutes wasmi execution fuzzing run to the CI as a job that is going to run on every commit.

@paritytech-cicd-pr
Copy link

paritytech-cicd-pr commented Dec 4, 2023

BENCHMARKS

NATIVEWASMTIME
BENCHMARKMASTERPRDIFFMASTERPRDIFFWASMTIME OVERHEAD
execute/
br_table
1.48ms 1.47ms ⚪ -0.61% 1.26ms 1.48ms 🔴 17.27% 🟢 0%
execute/
call/host/1
46.31µs 45.41µs ⚪ -1.95% 72.73µs 64.74µs 🟢 -11.09% 🟢 43%
execute/
call/rec
167.91µs 167.52µs ⚪ -0.24% 347.22µs 360.34µs 🔴 3.78% 🔴 115%
execute/
count_until
6.67ms 6.73ms ⚪ 0.21% 7.51ms 7.52ms ⚪ 0.10% 🟢 12%
execute/
divrem
6.23ms 6.25ms ⚪ 0.19% 7.78ms 6.96ms 🟢 -10.55% 🟢 11%
execute/
factorial/iter
240.11µs 243.09µs ⚪ 0.79% 315.23µs 314.23µs ⚪ -0.36% 🟢 29%
execute/
factorial/rec
681.56µs 677.68µs ⚪ -0.53% 1.26ms 1.26ms ⚪ 0.02% 🟡 86%
execute/
fibonacci/iter
1.29ms 1.29ms ⚪ -0.04% 1.26ms 1.27ms 🔴 1.15% 🟢 -1%
execute/
fibonacci/rec
6.09ms 6.11ms ⚪ 0.19% 13.35ms 13.49ms 🔴 1.43% 🔴 121%
execute/
fibonacci/tail
1.45ms 1.38ms 🟢 -4.63% 3.55ms 3.54ms ⚪ -0.40% 🔴 156%
execute/
fuse
7.33ms 7.32ms ⚪ -0.12% 11.42ms 11.41ms ⚪ -0.03% 🟡 56%
execute/
global/bump
1.32ms 1.32ms ⚪ -0.17% 1.54ms 1.55ms ⚪ 0.85% 🟢 18%
execute/
global/get_const
701.10µs 699.85µs ⚪ -0.36% 750.55µs 751.77µs ⚪ 0.13% 🟢 7%
execute/
is_even/rec
1.09ms 1.09ms ⚪ 0.20% 2.19ms 2.27ms 🔴 3.92% 🔴 108%
execute/
memory/fill_bytes
1.08ms 1.08ms ⚪ -0.02% 1.32ms 1.32ms ⚪ -0.07% 🟢 22%
execute/
memory/sum_bytes
1.09ms 1.11ms ⚪ 0.32% 1.25ms 1.27ms 🔴 1.29% 🟢 14%
execute/
memory/vec_add
2.95ms 2.95ms ⚪ -0.02% 3.61ms 3.61ms ⚪ -0.07% 🟢 23%
execute/
recursive_scan
191.86µs 193.83µs ⚪ -0.55% 380.36µs 393.34µs 🔴 3.40% 🔴 103%
execute/
recursive_trap
15.69µs 15.63µs ⚪ -0.46% 34.46µs 35.51µs 🔴 3.03% 🔴 127%
execute/
regex_redux
592.25µs 621.57µs 🔴 5.12% 991.13µs 1.01ms 🔴 2.02% 🟡 63%
execute/
rev_complement
442.33µs 442.80µs ⚪ 0.20% 627.12µs 628.84µs ⚪ 0.28% 🟢 42%
execute/
tiny_keccak
349.89µs 350.11µs ⚪ 0.09% 380.51µs 378.29µs ⚪ -0.59% 🟢 8%
execute/
trunc_f2i
616.41µs 615.98µs ⚪ 0.03% 961.40µs 960.66µs ⚪ -0.06% 🟡 56%
instantiate/
wasm_kernel
55.66µs 53.07µs 🟢 -3.88% 55.28µs 54.79µs ⚪ -1.00% 🟢 3%
overhead/
call/typed/0
1.20ms 1.22ms ⚪ 1.24% 777.99µs 862.66µs 🔴 10.75% 🟢 -29%
overhead/
call/typed/16
1.61ms 1.74ms 🔴 7.92% 1.95ms 1.95ms ⚪ -0.16% 🟢 12%
overhead/
call/untyped/0
1.64ms 1.61ms ⚪ -1.52% 1.19ms 1.21ms 🔴 1.93% 🟢 -25%
overhead/
call/untyped/16
2.51ms 2.58ms 🔴 2.73% 4.07ms 4.17ms 🔴 2.44% 🟡 62%
translate/
bz2/checked/default
1.38ms 1.38ms ⚪ -0.23% 2.52ms 2.50ms ⚪ -0.43% 🟡 81%
translate/
bz2/checked/fuel
1.42ms 1.42ms ⚪ -0.10% 2.65ms 2.64ms ⚪ -1.18% 🟡 86%
translate/
bz2/unchecked/default
1.13ms 1.12ms ⚪ 0.07% 1.92ms 1.91ms ⚪ -0.10% 🟡 70%
translate/
bz2/unchecked/fuel
1.16ms 1.16ms ⚪ -0.54% 2.03ms 2.04ms ⚪ 0.02% 🟡 76%
translate/
erc1155/checked/default
280.61µs 284.06µs ⚪ 1.04% 486.48µs 480.17µs ⚪ -1.03% 🟡 69%
translate/
erc1155/checked/fuel
300.07µs 302.57µs ⚪ 0.90% 517.94µs 511.08µs 🟢 -1.60% 🟡 69%
translate/
erc1155/unchecked/default
233.70µs 231.61µs ⚪ -0.70% 376.41µs 375.34µs ⚪ -0.31% 🟡 62%
translate/
erc1155/unchecked/fuel
250.74µs 249.88µs ⚪ -0.36% 402.82µs 400.04µs ⚪ -0.15% 🟡 60%
translate/
erc20/checked/default
136.21µs 137.64µs ⚪ 1.17% 233.98µs 230.85µs ⚪ -1.24% 🟡 68%
translate/
erc20/checked/fuel
143.97µs 144.98µs ⚪ 0.76% 245.20µs 242.34µs ⚪ -1.12% 🟡 67%
translate/
erc20/unchecked/default
112.85µs 112.77µs ⚪ 0.00% 181.02µs 180.85µs ⚪ -0.40% 🟡 60%
translate/
erc20/unchecked/fuel
120.01µs 120.22µs ⚪ 0.18% 191.97µs 190.96µs ⚪ -0.86% 🟡 59%
translate/
erc721/checked/default
194.31µs 194.93µs ⚪ 0.37% 339.28µs 336.09µs ⚪ -0.90% 🟡 72%
translate/
erc721/checked/fuel
204.85µs 205.39µs ⚪ 0.15% 355.13µs 350.21µs 🟢 -1.32% 🟡 71%
translate/
erc721/unchecked/default
159.17µs 158.75µs ⚪ -0.23% 258.96µs 258.91µs ⚪ 0.11% 🟡 63%
translate/
erc721/unchecked/fuel
167.68µs 167.95µs ⚪ 0.10% 272.68µs 270.17µs ⚪ -1.24% 🟡 61%
translate/
pulldown_cmark/checked/default
3.75ms 3.78ms ⚪ 0.51% 6.63ms 6.58ms ⚪ -0.76% 🟡 74%
translate/
pulldown_cmark/checked/fuel
3.85ms 3.84ms ⚪ -0.20% 6.86ms 6.83ms ⚪ -0.32% 🟡 78%
translate/
pulldown_cmark/unchecked/default
3.11ms 3.10ms ⚪ -0.27% 5.06ms 5.06ms ⚪ 0.19% 🟡 63%
translate/
pulldown_cmark/unchecked/fuel
3.19ms 3.20ms ⚪ 0.28% 5.30ms 5.31ms ⚪ -0.19% 🟡 66%
translate/
spidermonkey/checked/default
0.00ns 0.00ns ⚪ 0.47% 0.00ns 0.00ns ⚪ -0.84% 🟢 0%
translate/
spidermonkey/checked/fuel
0.00ns 0.00ns ⚪ 0.59% 0.00ns 0.00ns 🟢 -1.42% 🟢 0%
translate/
spidermonkey/unchecked/default
71.70ms 71.57ms ⚪ -0.29% 0.00ns 0.00ns ⚪ -0.03% 🟢 -100%
translate/
spidermonkey/unchecked/fuel
0.00ns 0.00ns ⚪ -0.06% 0.00ns 0.00ns ⚪ -0.07% 🟢 0%
translate/
wasm_kernel/checked/default
5.03ms 5.05ms ⚪ 0.58% 8.94ms 8.82ms ⚪ -1.25% 🟡 75%
translate/
wasm_kernel/checked/fuel
5.15ms 5.17ms ⚪ 0.43% 9.30ms 9.21ms ⚪ -0.63% 🟡 78%
translate/
wasm_kernel/unchecked/default
4.12ms 4.10ms ⚪ -0.37% 6.73ms 6.73ms ⚪ 0.09% 🟡 64%
translate/
wasm_kernel/unchecked/fuel
4.24ms 4.22ms ⚪ -0.62% 7.14ms 7.31ms 🔴 2.37% 🟡 73%

Link to pipeline

@codecov-commenter
Copy link

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (050ba4f) 81.07% compared to head (48d8fd8) 80.93%.

Files Patch % Lines
fuzz/fuzz_targets/execute.rs 0.00% 33 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #832      +/-   ##
==========================================
- Coverage   81.07%   80.93%   -0.14%     
==========================================
  Files         256      257       +1     
  Lines       22428    22461      +33     
==========================================
- Hits        18183    18179       -4     
- Misses       4245     4282      +37     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Robbepop Robbepop merged commit 41ca061 into master Dec 4, 2023
19 checks passed
@Robbepop Robbepop deleted the rf-execution-fuzzing branch December 4, 2023 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants