Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Even more testing of benchmarks... #629

Draft
wants to merge 1 commit into
base: ban/tmp-test-benchmarks-bs
Choose a base branch
from
Draft

[WIP] Even more testing of benchmarks... #629

wants to merge 1 commit into from

Conversation

bantonsson
Copy link
Contributor

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

@codecov-commenter
Copy link

codecov-commenter commented Sep 13, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.08%. Comparing base (66cc801) to head (0b7281d).

Additional details and impacted files 
@@                      Coverage Diff                       @@
##           ban/tmp-test-benchmarks-bs     #629      +/-   ##
==============================================================
+ Coverage                       73.04%   73.08%   +0.03%     
==============================================================
  Files                             252      252              
  Lines                           36076    36076              
==============================================================
+ Hits                            26352    26365      +13     
+ Misses                           9724     9711      -13
Components Coverage Δ
crashtracker 20.48% <ø> (-0.06%) ⬇️
datadog-alloc 98.73% <ø> (ø)
data-pipeline 90.12% <ø> (ø)
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.08% <ø> (ø)
ddcommon-ffi 69.52% <ø> (ø)
ddtelemetry 59.10% <ø> (ø)
ipc 83.63% <ø> (ø)
profiling 84.26% <ø> (ø)
profiling-ffi 77.42% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 40.12% <ø> (ø)
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 50.36% <ø> (ø)
tinybytes 91.66% <ø> (ø)
trace-mini-agent 72.49% <ø> (ø)
trace-normalization 98.25% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.67% <ø> (ø)
trace-utils 93.30% <ø> (ø)

@pr-commenter
Copy link

pr-commenter bot commented Sep 13, 2024 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2024-09-18 15:21:18

Comparing candidate commit 0b7281d in PR branch ban/tmp-srsly-benchmarks with baseline commit 66cc801 in branch ban/tmp-test-benchmarks-bs.

Found 14 performance improvements and 16 performance regressions! Performance is the same for 21 metrics, 2 unstable metrics.

scenario:benching string interning on wordpress profile

  • 🟩 execution_time [-5.174µs; -4.983µs] or [-3.566%; -3.434%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟥 execution_time [+57.405µs; +59.085µs] or [+10.810%; +11.126%]
  • 🟥 throughput [-946235.451op/s; -917325.410op/s] or [-10.048%; -9.741%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟥 execution_time [+59.313µs; +61.192µs] or [+12.017%; +12.397%]
  • 🟥 throughput [-1121850.875op/s; -1085104.494op/s] or [-11.073%; -10.710%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 execution_time [+49.697µs; +51.655µs] or [+10.281%; +10.686%]
  • 🟥 throughput [-1003577.635op/s; -963032.006op/s] or [-9.700%; -9.308%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟥 execution_time [+18.718µs; +21.392µs] or [+3.828%; +4.375%]
  • 🟥 throughput [-435719.827op/s; -379756.599op/s] or [-4.259%; -3.712%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟩 execution_time [-3.981µs; -3.143µs] or [-3.594%; -2.837%]
  • 🟩 throughput [+1318786.697op/s; +1665018.333op/s] or [+2.920%; +3.687%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟥 execution_time [+11.240µs; +13.025µs] or [+2.468%; +2.860%]
  • 🟥 throughput [-308501.286op/s; -265868.541op/s] or [-2.809%; -2.421%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟥 execution_time [+53.523µs; +54.952µs] or [+12.751%; +13.092%]
  • 🟥 throughput [-1385277.820op/s; -1345118.535op/s] or [-11.627%; -11.290%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟥 execution_time [+16.480µs; +18.965µs] or [+3.354%; +3.859%]
  • 🟥 throughput [-383991.078op/s; -332617.162op/s] or [-3.772%; -3.268%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟩 execution_time [-5.336µs; -4.533µs] or [-4.765%; -4.047%]
  • 🟩 throughput [+1880514.556op/s; +2208425.201op/s] or [+4.210%; +4.944%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟥 execution_time [+2.108µs; +2.200µs] or [+2.595%; +2.708%]
  • 🟥 throughput [-1623643.841op/s; -1555895.773op/s] or [-2.638%; -2.528%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟩 execution_time [-153.598µs; -152.711µs] or [-5.700%; -5.668%]
  • 🟩 throughput [+111512.854op/s; +112147.168op/s] or [+6.009%; +6.044%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

  • 🟩 execution_time [-47.249µs; -46.791µs] or [-5.105%; -5.056%]
  • 🟩 throughput [+287765.901op/s; +290607.447op/s] or [+5.326%; +5.379%]

scenario:normalization/normalize_service/normalize_service/test_ASCII

  • 🟩 execution_time [-41.202µs; -41.098µs] or [-16.581%; -16.539%]
  • 🟩 throughput [+3988034.013op/s; +3998629.972op/s] or [+19.820%; +19.873%]

scenario:redis/obfuscate_redis_string

  • 🟩 execution_time [-1.315µs; -1.269µs] or [-3.241%; -3.128%]

scenario:sql/obfuscate_sql_string

  • 🟩 execution_time [-6.738µs; -6.674µs] or [-8.861%; -8.776%]

scenario:tags/replace_trace_tags

  • 🟩 execution_time [-69.888ns; -67.881ns] or [-2.589%; -2.515%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 18.084µs 36.742µs ± 19.616µs 35.304µs ± 1.354µs 36.699µs 42.540µs 117.849µs 251.896µs 613.50% 8.268 78.388 53.28% 1.241µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [34.310µs; 39.173µs] or [-6.618%; +6.618%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 6.057µs 6.059µs ± 0.001µs 6.059µs ± 0.001µs 6.060µs 6.061µs 6.064µs 6.064µs 0.09% 1.078 1.657 0.02% 0.000µs 1 250
credit_card/is_card_number/ throughput 824514285.416op/s 825230521.865op/s ± 185077.291op/s 825273092.805op/s ± 104846.056op/s 825357587.511op/s 825474611.753op/s 825529461.269op/s 825540837.707op/s 0.03% -1.076 1.651 0.02% 11705.316op/s 1 250
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 581.550µs 589.290µs ± 2.441µs 589.489µs ± 1.589µs 591.090µs 592.908µs 594.302µs 594.982µs 0.93% -0.446 0.205 0.41% 0.154µs 1 250
credit_card/is_card_number/ 3782-8224-6310-005 throughput 8403610.023op/s 8484932.079op/s ± 35216.908op/s 8481920.167op/s ± 22920.622op/s 8504748.178op/s 8555160.136op/s 8572261.538op/s 8597719.872op/s 1.37% 0.471 0.237 0.41% 2227.313op/s 1 250
credit_card/is_card_number/ 378282246310005 execution_time 539.896µs 553.836µs ± 3.638µs 554.415µs ± 1.955µs 555.887µs 559.715µs 560.623µs 563.306µs 1.60% -0.593 0.974 0.66% 0.230µs 1 250
credit_card/is_card_number/ 378282246310005 throughput 8876169.236op/s 9028332.520op/s ± 59539.213op/s 9018508.858op/s ± 31682.706op/s 9059840.555op/s 9134952.775op/s 9202259.895op/s 9261044.051op/s 2.69% 0.645 1.076 0.66% 3765.590op/s 1 250
credit_card/is_card_number/37828224631 execution_time 6.057µs 6.059µs ± 0.001µs 6.059µs ± 0.001µs 6.060µs 6.062µs 6.063µs 6.064µs 0.08% 0.882 1.104 0.02% 0.000µs 1 250
credit_card/is_card_number/37828224631 throughput 824487466.126op/s 825155655.899op/s ± 161445.510op/s 825187435.545op/s ± 80566.203op/s 825263016.839op/s 825353932.422op/s 825464065.457op/s 825558901.175op/s 0.05% -0.881 1.101 0.02% 10210.711op/s 1 250
credit_card/is_card_number/378282246310005 execution_time 523.183µs 534.055µs ± 3.546µs 535.050µs ± 1.994µs 536.106µs 539.257µs 542.232µs 543.485µs 1.58% -0.509 0.434 0.66% 0.224µs 1 250
credit_card/is_card_number/378282246310005 throughput 9199879.150op/s 9362750.566op/s ± 62393.395op/s 9344921.911op/s ± 34840.374op/s 9400449.379op/s 9477174.069op/s 9522311.258op/s 9556876.925op/s 2.27% 0.552 0.466 0.67% 3946.105op/s 1 250
credit_card/is_card_number/37828224631000521389798 execution_time 506.709µs 509.032µs ± 1.041µs 508.975µs ± 0.680µs 509.609µs 511.034µs 511.729µs 512.525µs 0.70% 0.526 0.286 0.20% 0.066µs 1 250
credit_card/is_card_number/37828224631000521389798 throughput 9755625.246op/s 9822598.352op/s ± 20067.380op/s 9823673.958op/s ± 13114.632op/s 9837841.173op/s 9852251.723op/s 9861062.974op/s 9867592.329op/s 0.45% -0.514 0.266 0.20% 1269.173op/s 1 250
credit_card/is_card_number/x371413321323331 execution_time 102.731µs 107.213µs ± 2.094µs 107.096µs ± 1.495µs 108.597µs 110.977µs 111.911µs 112.180µs 4.75% 0.233 -0.520 1.95% 0.132µs 1 250
credit_card/is_card_number/x371413321323331 throughput 44571200.818op/s 46653756.202op/s ± 907780.741op/s 46687178.735op/s ± 652720.827op/s 47340119.908op/s 48072051.543op/s 48456819.082op/s 48670911.728op/s 4.25% -0.150 -0.557 1.94% 57413.095op/s 1 250
credit_card/is_card_number_no_luhn/ execution_time 6.056µs 6.060µs ± 0.003µs 6.059µs ± 0.001µs 6.060µs 6.063µs 6.065µs 6.091µs 0.52% 6.802 75.618 0.04% 0.000µs 1 250
credit_card/is_card_number_no_luhn/ throughput 820932490.526op/s 825135817.463op/s ± 357322.790op/s 825197833.801op/s ± 148819.651op/s 825324124.760op/s 825448586.213op/s 825578160.698op/s 825643207.814op/s 0.05% -6.762 74.964 0.04% 22599.078op/s 1 250
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 457.949µs 467.569µs ± 1.979µs 468.048µs ± 1.010µs 468.843µs 469.882µs 470.800µs 471.650µs 0.77% -1.315 2.671 0.42% 0.125µs 1 250
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 10601076.173op/s 10693791.462op/s ± 45507.275op/s 10682656.223op/s ± 23022.402op/s 10709967.747op/s 10785546.158op/s 10836485.687op/s 10918234.775op/s 2.21% 1.352 2.833 0.42% 2878.133op/s 1 250
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 472.243µs 473.984µs ± 0.946µs 473.770µs ± 0.462µs 474.429µs 475.622µs 476.666µs 479.631µs 1.24% 1.558 5.391 0.20% 0.060µs 1 250
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 10424683.197op/s 10548910.172op/s ± 20987.767op/s 10553652.036op/s ± 10292.089op/s 10561989.153op/s 10576424.504op/s 10584953.539op/s 10587772.070op/s 0.32% -1.528 5.205 0.20% 1327.383op/s 1 250
credit_card/is_card_number_no_luhn/37828224631 execution_time 6.054µs 6.058µs ± 0.001µs 6.058µs ± 0.001µs 6.059µs 6.061µs 6.062µs 6.063µs 0.09% 0.650 1.209 0.02% 0.000µs 1 250
credit_card/is_card_number_no_luhn/37828224631 throughput 824636651.149op/s 825340588.224op/s ± 173529.034op/s 825354539.434op/s ± 102790.960op/s 825456513.052op/s 825577515.687op/s 825634750.427op/s 825876802.203op/s 0.06% -0.648 1.206 0.02% 10974.940op/s 1 250
credit_card/is_card_number_no_luhn/378282246310005 execution_time 407.900µs 412.761µs ± 2.579µs 411.912µs ± 1.441µs 414.437µs 417.622µs 419.701µs 421.733µs 2.38% 0.972 0.377 0.62% 0.163µs 1 250
credit_card/is_card_number_no_luhn/378282246310005 throughput 11855838.836op/s 12114005.333op/s ± 75243.973op/s 12138508.245op/s ± 42453.616op/s 12171350.900op/s 12199093.933op/s 12209374.768op/s 12257917.908op/s 0.98% -0.945 0.301 0.62% 4758.847op/s 1 250
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 506.452µs 509.137µs ± 1.033µs 509.002µs ± 0.625µs 509.657µs 511.001µs 511.765µs 514.718µs 1.12% 1.023 2.912 0.20% 0.065µs 1 250
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 9714057.607op/s 9820585.351op/s ± 19878.552op/s 9823145.320op/s ± 12057.967op/s 9834748.413op/s 9845782.524op/s 9855514.646op/s 9872611.838op/s 0.50% -1.000 2.801 0.20% 1257.230op/s 1 250
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 103.324µs 107.066µs ± 1.776µs 107.039µs ± 1.402µs 108.501µs 110.174µs 110.906µs 111.628µs 4.29% 0.198 -0.616 1.66% 0.112µs 1 250
credit_card/is_card_number_no_luhn/x371413321323331 throughput 44791530.364op/s 46712760.353op/s ± 772606.592op/s 46712085.765op/s ± 610260.516op/s 47298870.307op/s 47977598.074op/s 48229913.500op/s 48391620.610op/s 3.60% -0.131 -0.645 1.65% 48863.931op/s 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [6.059µs; 6.059µs] or [-0.003%; +0.003%] None None None
credit_card/is_card_number/ throughput [825207579.867op/s; 825253463.862op/s] or [-0.003%; +0.003%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [588.987µs; 589.593µs] or [-0.051%; +0.051%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [8480566.626op/s; 8489297.532op/s] or [-0.051%; +0.051%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [553.385µs; 554.287µs] or [-0.081%; +0.081%] None None None
credit_card/is_card_number/ 378282246310005 throughput [9020952.098op/s; 9035712.942op/s] or [-0.082%; +0.082%] None None None
credit_card/is_card_number/37828224631 execution_time [6.059µs; 6.060µs] or [-0.002%; +0.002%] None None None
credit_card/is_card_number/37828224631 throughput [825135643.274op/s; 825175668.524op/s] or [-0.002%; +0.002%] None None None
credit_card/is_card_number/378282246310005 execution_time [533.615µs; 534.494µs] or [-0.082%; +0.082%] None None None
credit_card/is_card_number/378282246310005 throughput [9355016.343op/s; 9370484.789op/s] or [-0.083%; +0.083%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [508.903µs; 509.161µs] or [-0.025%; +0.025%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [9820110.820op/s; 9825085.885op/s] or [-0.025%; +0.025%] None None None
credit_card/is_card_number/x371413321323331 execution_time [106.954µs; 107.473µs] or [-0.242%; +0.242%] None None None
credit_card/is_card_number/x371413321323331 throughput [46541228.604op/s; 46766283.801op/s] or [-0.241%; +0.241%] None None None
credit_card/is_card_number_no_luhn/ execution_time [6.059µs; 6.060µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ throughput [825091524.085op/s; 825180110.841op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [467.324µs; 467.815µs] or [-0.052%; +0.052%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [10688150.426op/s; 10699432.499op/s] or [-0.053%; +0.053%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [473.867µs; 474.102µs] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [10546308.549op/s; 10551511.795op/s] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [6.058µs; 6.058µs] or [-0.003%; +0.003%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [825319077.737op/s; 825362098.711op/s] or [-0.003%; +0.003%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [412.442µs; 413.081µs] or [-0.077%; +0.077%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [12104678.165op/s; 12123332.501op/s] or [-0.077%; +0.077%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [509.009µs; 509.265µs] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [9818121.226op/s; 9823049.477op/s] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [106.846µs; 107.286µs] or [-0.206%; +0.206%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [46616988.808op/s; 46808531.899op/s] or [-0.205%; +0.205%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 308.774ns 309.928ns ± 0.803ns 309.808ns ± 0.261ns 310.060ns 310.802ns 311.938ns 319.927ns 3.27% 8.073 94.714 0.26% 0.051ns 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [309.828ns; 310.027ns] or [-0.032%; +0.032%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.332µs 3.259µs ± 2.328µs 3.062µs ± 0.041µs 3.102µs 3.522µs 6.991µs 31.822µs 939.37% 10.683 118.229 71.29% 0.147µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.971µs; 3.548µs] or [-8.855%; +8.855%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 1.404ms 1.408ms ± 0.002ms 1.407ms ± 0.001ms 1.408ms 1.410ms 1.414ms 1.420ms 0.90% 2.938 16.846 0.12% 0.000ms 1 250
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3521443.084op/s 3552384.557op/s ± 4082.608op/s 3553066.409op/s ± 2048.910op/s 3554782.807op/s 3556851.495op/s 3559113.364op/s 3561730.770op/s 0.24% -2.903 16.536 0.11% 258.207op/s 1 250
normalization/normalize_name/normalize_name/bad-name execution_time 137.342µs 138.504µs ± 0.472µs 138.511µs ± 0.345µs 138.842µs 139.300µs 139.603µs 139.773µs 0.91% 0.200 -0.437 0.34% 0.030µs 1 250
normalization/normalize_name/normalize_name/bad-name throughput 35772380.990op/s 36100541.230op/s ± 122902.979op/s 36098314.746op/s ± 90137.652op/s 36193491.191op/s 36281102.154op/s 36352028.413op/s 36405461.525op/s 0.85% -0.184 -0.445 0.34% 7773.067op/s 1 250
normalization/normalize_name/normalize_name/good execution_time 82.682µs 83.394µs ± 0.245µs 83.328µs ± 0.097µs 83.477µs 83.908µs 84.102µs 84.149µs 0.99% 0.937 0.935 0.29% 0.015µs 1 250
normalization/normalize_name/normalize_name/good throughput 59418336.242op/s 59956769.276op/s ± 175616.966op/s 60003869.440op/s ± 70008.441op/s 60061819.847op/s 60161271.089op/s 60310700.065op/s 60472733.335op/s 0.78% -0.918 0.919 0.29% 11106.992op/s 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [1.407ms; 1.408ms] or [-0.014%; +0.014%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3551878.481op/s; 3552890.633op/s] or [-0.014%; +0.014%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [138.445µs; 138.562µs] or [-0.042%; +0.042%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [36085306.299op/s; 36115776.162op/s] or [-0.042%; +0.042%] None None None
normalization/normalize_name/normalize_name/good execution_time [83.364µs; 83.424µs] or [-0.036%; +0.036%] None None None
normalization/normalize_name/normalize_name/good throughput [59934999.972op/s; 59978538.581op/s] or [-0.036%; +0.036%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.620µs 2.630µs ± 0.005µs 2.630µs ± 0.004µs 2.634µs 2.639µs 2.645µs 2.652µs 0.84% 0.626 0.669 0.20% 0.000µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.630µs; 2.631µs] or [-0.025%; +0.025%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 1.197µs 1.200µs ± 0.001µs 1.200µs ± 0.000µs 1.200µs 1.201µs 1.204µs 1.209µs 0.80% 3.873 30.513 0.09% 0.000µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [1.200µs; 1.200µs] or [-0.011%; +0.011%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 69.096µs 69.335µs ± 0.229µs 69.309µs ± 0.034µs 69.348µs 69.410µs 69.854µs 72.149µs 4.10% 9.576 105.032 0.33% 0.014µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [69.307µs; 69.364µs] or [-0.041%; +0.041%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 2.536ms 2.541ms ± 0.002ms 2.541ms ± 0.001ms 2.542ms 2.544ms 2.547ms 2.566ms 0.98% 6.232 69.319 0.09% 0.000ms 1 250
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1948348.628op/s 1967470.512op/s ± 1667.282op/s 1967512.693op/s ± 660.310op/s 1968255.096op/s 1969397.876op/s 1970065.154op/s 1971923.879op/s 0.22% -6.149 68.091 0.08% 105.448op/s 1 250
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 2.312ms 2.317ms ± 0.002ms 2.317ms ± 0.001ms 2.318ms 2.319ms 2.321ms 2.324ms 0.32% 0.233 1.232 0.08% 0.000ms 1 250
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2151008.698op/s 2157966.201op/s ± 1707.780op/s 2157937.272op/s ± 1145.158op/s 2159065.129op/s 2160783.179op/s 2161880.647op/s 2163069.170op/s 0.24% -0.225 1.217 0.08% 108.010op/s 1 250
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 874.928µs 878.449µs ± 1.379µs 878.718µs ± 0.618µs 879.274µs 880.211µs 881.172µs 882.445µs 0.42% -0.534 -0.083 0.16% 0.087µs 1 250
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5666077.976op/s 5691863.898op/s ± 8939.442op/s 5690105.891op/s ± 4001.560op/s 5695994.702op/s 5708705.769op/s 5710977.553op/s 5714756.459op/s 0.43% 0.541 -0.085 0.16% 565.380op/s 1 250
normalization/normalize_service/normalize_service/[empty string] execution_time 215.030µs 216.384µs ± 0.430µs 216.398µs ± 0.298µs 216.700µs 217.088µs 217.208µs 217.370µs 0.45% -0.213 -0.296 0.20% 0.027µs 1 250
normalization/normalize_service/normalize_service/[empty string] throughput 23002249.518op/s 23107204.080op/s ± 45955.088op/s 23105627.376op/s ± 31846.137op/s 23136912.346op/s 23183087.982op/s 23220562.006op/s 23252591.036op/s 0.64% 0.223 -0.286 0.20% 2906.455op/s 1 250
normalization/normalize_service/normalize_service/test_ASCII execution_time 206.833µs 207.345µs ± 0.313µs 207.293µs ± 0.194µs 207.500µs 207.977µs 208.278µs 208.387µs 0.53% 0.912 0.672 0.15% 0.020µs 1 250
normalization/normalize_service/normalize_service/test_ASCII throughput 23993857.572op/s 24114467.677op/s ± 36295.695op/s 24120451.747op/s ± 22546.518op/s 24141466.094op/s 24161282.562op/s 24169522.359op/s 24174088.246op/s 0.22% -0.903 0.650 0.15% 2295.541op/s 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [2.541ms; 2.542ms] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1967263.837op/s; 1967677.187op/s] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [2.317ms; 2.317ms] or [-0.010%; +0.010%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2157754.507op/s; 2158177.896op/s] or [-0.010%; +0.010%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [878.278µs; 878.620µs] or [-0.019%; +0.019%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5690755.774op/s; 5692972.022op/s] or [-0.019%; +0.019%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [216.330µs; 216.437µs] or [-0.025%; +0.025%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [23101507.533op/s; 23112900.628op/s] or [-0.025%; +0.025%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [207.306µs; 207.384µs] or [-0.019%; +0.019%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [24109968.498op/s; 24118966.855op/s] or [-0.019%; +0.019%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.061ms 9.099ms ± 0.033ms 9.096ms ± 0.010ms 9.106ms 9.120ms 9.130ms 9.491ms 4.34% 8.890 95.776 0.36% 0.002ms 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.095ms; 9.103ms] or [-0.045%; +0.045%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 138.836µs 140.027µs ± 0.529µs 139.958µs ± 0.222µs 140.217µs 140.707µs 141.713µs 144.288µs 3.09% 3.599 23.828 0.38% 0.033µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [139.961µs; 140.092µs] or [-0.047%; +0.047%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0b7281d 1726663683 ban/tmp-srsly-benchmarks
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 38.855µs 39.275µs ± 0.067µs 39.268µs ± 0.026µs 39.296µs 39.348µs 39.503µs 39.795µs 1.34% 2.451 27.016 0.17% 0.004µs 1 250
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [39.267µs; 39.284µs] or [-0.021%; +0.021%] None None None

Baseline

Omitted due to size.

@bantonsson bantonsson force-pushed the ban/tmp-srsly-benchmarks branch 2 times, most recently from 35bb8f4 to 177d3f3 Compare September 13, 2024 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bantonsson @codecov-commenter