Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ekump/eval bench perf for io read impl #636

Closed
wants to merge 14 commits into from

Conversation

ekump
Copy link
Contributor

@ekump ekump commented Sep 17, 2024

What does this PR do?

NOT A REAL PR. Just trying to benchmark changes.

ekump and others added 14 commits September 17, 2024 15:14
Bytes itself is already a structure with a slice into some underlying bytes.
Thus, we are adding a cheap way to directly modify the slice of the Bytes struct.

To facilitate this, we now store the tuple (ptr, len) as a &[] slice within the Bytes struct. The representation of these is exactly identical to before, but it allows trivial manipulation of the slice via direct assignment or &mut &[u8] reference.

The way the underlying field was exposed on the BytesWrapper struct was also unsound. Access to it was not wrapped within an unsafe method.

The decoder code can now trivially clone() the given Bytes struct and carry it around, directly shrinking the slice size of the underlying Bytes instance as the data is being processed.

Additionally, there now exist two helper methods for numbers and strings (read_string_bytes and read_number_bytes) which handle a bit of the boilerplate.
Sadly it is necessary to always write "unsafe { buf.as_mut_slice() }", but this is necessary for soundness, we only can actually guarantee safety when calling the actual decode functions, as long as we want to carry Bytes around.

Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
@ekump ekump force-pushed the ekump/eval-bench-perf-for-io-read-impl branch from 5572486 to 6f6ad62 Compare September 17, 2024 19:14
@codecov-commenter
Copy link

codecov-commenter commented Sep 17, 2024

Codecov Report

Attention: Patch coverage is 96.99399% with 15 lines in your changes missing coverage. Please review.

Project coverage is 73.20%. Comparing base (89f48d4) to head (16f094c).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #636      +/-   ##
==========================================
+ Coverage   73.05%   73.20%   +0.15%     
==========================================
  Files         252      254       +2     
  Lines       36093    36316     +223     
==========================================
+ Hits        26366    26585     +219     
- Misses       9727     9731       +4     
Components Coverage Δ
crashtracker 20.53% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 90.17% <0.00%> (+0.05%) ⬆️
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.08% <ø> (ø)
ddcommon-ffi 69.52% <ø> (ø)
ddtelemetry 59.10% <ø> (ø)
ipc 83.58% <0.00%> (-0.06%) ⬇️
profiling 84.26% <ø> (ø)
profiling-ffi 77.42% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 40.06% <0.00%> (-0.07%) ⬇️
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 50.36% <ø> (ø)
tinybytes 94.77% <100.00%> (+3.10%) ⬆️
trace-mini-agent 72.47% <100.00%> (-0.03%) ⬇️
trace-normalization 98.25% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.67% <ø> (ø)
trace-utils 93.54% <100.00%> (+0.65%) ⬆️

@pr-commenter
Copy link

pr-commenter bot commented Sep 17, 2024

Benchmarks

Comparison

Benchmark execution time: 2024-09-17 19:52:17

Comparing candidate commit 16f094c in PR branch ekump/eval-bench-perf-for-io-read-impl with baseline commit 89f48d4 in branch main.

Found 7 performance improvements and 33 performance regressions! Performance is the same for 11 metrics, 2 unstable metrics.

scenario:benching deserializing traces from msgpack to their internal representation

  • 🟩 execution_time [-178.228ns; -162.009ns] or [-12.994%; -11.811%]

scenario:credit_card/is_card_number/

  • 🟥 execution_time [+1.206µs; +1.206µs] or [+99.640%; +99.685%]
  • 🟥 throughput [-412647223.552op/s; -412307233.869op/s] or [-49.936%; -49.895%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟥 execution_time [+4.264µs; +4.617µs] or [+3.831%; +4.148%]
  • 🟥 throughput [-357942.450op/s; -330735.935op/s] or [-3.983%; -3.681%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟥 execution_time [+2.866µs; +3.439µs] or [+2.743%; +3.292%]
  • 🟥 throughput [-305029.037op/s; -254601.152op/s] or [-3.186%; -2.659%]

scenario:credit_card/is_card_number/37828224631

  • 🟥 execution_time [+1.208µs; +1.208µs] or [+99.941%; +99.985%]
  • 🟥 throughput [-413764053.597op/s; -413623022.201op/s] or [-49.999%; -49.982%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 execution_time [+4.502µs; +4.979µs] or [+4.455%; +4.927%]
  • 🟥 throughput [-464926.078op/s; -420810.458op/s] or [-4.698%; -4.252%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟥 execution_time [+11.404µs; +11.715µs] or [+12.024%; +12.353%]
  • 🟥 throughput [-1162772.591op/s; -1130248.174op/s] or [-11.026%; -10.718%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟥 execution_time [+1.332µs; +1.473µs] or [+6.303%; +6.973%]
  • 🟥 throughput [-3086916.402op/s; -2795602.739op/s] or [-6.521%; -5.906%]

scenario:credit_card/is_card_number_no_luhn/

  • 🟥 execution_time [+1.205µs; +1.206µs] or [+99.531%; +99.575%]
  • 🟥 throughput [-412154604.181op/s; -411835270.361op/s] or [-49.907%; -49.869%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟩 execution_time [-8.898µs; -8.734µs] or [-9.259%; -9.088%]
  • 🟩 throughput [+1041511.962op/s; +1059915.109op/s] or [+10.009%; +10.186%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟩 execution_time [-7.325µs; -7.075µs] or [-8.379%; -8.093%]
  • 🟩 throughput [+1009060.823op/s; +1044716.080op/s] or [+8.821%; +9.133%]

scenario:credit_card/is_card_number_no_luhn/37828224631

  • 🟥 execution_time [+1.208µs; +1.208µs] or [+99.949%; +99.978%]
  • 🟥 throughput [-413784996.374op/s; -413613737.220op/s] or [-50.001%; -49.981%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

  • 🟩 execution_time [-6.108µs; -5.816µs] or [-7.336%; -6.985%]
  • 🟩 throughput [+904262.816op/s; +950517.970op/s] or [+7.529%; +7.914%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟥 execution_time [+11.595µs; +11.819µs] or [+12.242%; +12.480%]
  • 🟥 throughput [-1173504.197op/s; -1150341.279op/s] or [-11.114%; -10.894%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟥 execution_time [+1.308µs; +1.439µs] or [+6.200%; +6.820%]
  • 🟥 throughput [-3024824.653op/s; -2755722.963op/s] or [-6.382%; -5.814%]

scenario:normalization/normalize_name/normalize_name/bad-name

  • 🟥 execution_time [+2.268µs; +2.281µs] or [+8.781%; +8.832%]
  • 🟥 throughput [-3141998.763op/s; -3124006.140op/s] or [-8.117%; -8.070%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟥 execution_time [+1.131µs; +1.152µs] or [+7.356%; +7.497%]
  • 🟥 throughput [-4536842.533op/s; -4456733.813op/s] or [-6.973%; -6.850%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟥 execution_time [+30.353µs; +30.483µs] or [+5.980%; +6.006%]
  • 🟥 throughput [-111636.214op/s; -111148.322op/s] or [-5.666%; -5.642%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

  • 🟥 execution_time [+4.307µs; +4.429µs] or [+2.385%; +2.452%]
  • 🟥 throughput [-132615.765op/s; -128920.717op/s] or [-2.395%; -2.328%]

scenario:normalization/normalize_service/normalize_service/[empty string]

  • 🟥 execution_time [+1.344µs; +1.371µs] or [+3.080%; +3.141%]
  • 🟥 throughput [-698429.966op/s; -684573.962op/s] or [-3.048%; -2.987%]

scenario:redis/obfuscate_redis_string

  • 🟥 execution_time [+1.046µs; +1.402µs] or [+2.709%; +3.631%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 39.396µs 39.838µs ± 0.695µs 39.522µs ± 0.050µs 39.625µs 41.263µs 41.341µs 43.232µs 9.39% 1.915 2.806 1.74% 0.049µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [39.741µs; 39.934µs] or [-0.242%; +0.242%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 276.049µs 276.299µs ± 0.155µs 276.277µs ± 0.107µs 276.404µs 276.563µs 276.760µs 276.914µs 0.23% 0.855 1.256 0.06% 0.011µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3611235.775op/s 3619264.537op/s ± 2035.796op/s 3619555.510op/s ± 1406.102op/s 3620537.688op/s 3622075.242op/s 3622397.642op/s 3622548.440op/s 0.08% -0.850 1.243 0.06% 143.952op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 28.058µs 28.108µs ± 0.037µs 28.112µs ± 0.038µs 28.125µs 28.171µs 28.193µs 28.219µs 0.38% 0.647 -0.531 0.13% 0.003µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 35437080.612op/s 35577388.382op/s ± 46294.079op/s 35572174.147op/s ± 47893.309op/s 35620470.057op/s 35626983.699op/s 35629088.635op/s 35640739.970op/s 0.19% -0.643 -0.541 0.13% 3273.486op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 16.324µs 16.512µs ± 0.074µs 16.520µs ± 0.051µs 16.556µs 16.618µs 16.638µs 16.966µs 2.70% 0.939 5.967 0.45% 0.005µs 1 200
normalization/normalize_name/normalize_name/good throughput 58941412.236op/s 60564849.420op/s ± 269547.081op/s 60533992.110op/s ± 186415.640op/s 60745483.532op/s 60967798.806op/s 61116584.836op/s 61258572.956op/s 1.20% -0.847 5.390 0.44% 19059.857op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [276.278µs; 276.321µs] or [-0.008%; +0.008%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3618982.396op/s; 3619546.679op/s] or [-0.008%; +0.008%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [28.103µs; 28.113µs] or [-0.018%; +0.018%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [35570972.468op/s; 35583804.296op/s] or [-0.018%; +0.018%] None None None
normalization/normalize_name/normalize_name/good execution_time [16.501µs; 16.522µs] or [-0.062%; +0.062%] None None None
normalization/normalize_name/normalize_name/good throughput [60527492.787op/s; 60602206.053op/s] or [-0.062%; +0.062%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 73.671µs 73.802µs ± 0.202µs 73.778µs ± 0.034µs 73.811µs 73.924µs 74.046µs 76.469µs 3.65% 11.601 149.751 0.27% 0.014µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [73.774µs; 73.830µs] or [-0.038%; +0.038%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 143.572µs 144.340µs ± 0.623µs 144.232µs ± 0.185µs 144.456µs 145.026µs 146.247µs 150.501µs 4.35% 5.875 49.952 0.43% 0.044µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [144.254µs; 144.427µs] or [-0.060%; +0.060%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 16.781µs 22.274µs ± 10.337µs 17.680µs ± 0.760µs 19.863µs 38.262µs 46.099µs 93.887µs 431.03% 3.287 15.861 46.29% 0.731µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [20.841µs; 23.706µs] or [-6.432%; +6.432%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 298.795ns 310.111ns ± 13.235ns 303.230ns ± 3.961ns 316.633ns 340.748ns 353.177ns 354.726ns 16.98% 1.696 2.318 4.26% 0.936ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [308.277ns; 311.945ns] or [-0.591%; +0.591%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 537.534µs 537.995µs ± 0.231µs 537.957µs ± 0.159µs 538.140µs 538.383µs 538.555µs 539.042µs 0.20% 0.578 1.278 0.04% 0.016µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1855142.203op/s 1858753.584op/s ± 797.976op/s 1858886.055op/s ± 550.434op/s 1859284.433op/s 1860058.010op/s 1860222.456op/s 1860347.438op/s 0.08% -0.574 1.265 0.04% 56.425op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 466.653µs 467.177µs ± 0.256µs 467.174µs ± 0.179µs 467.355µs 467.538µs 467.742µs 468.452µs 0.27% 0.611 1.926 0.05% 0.018µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2134690.667op/s 2140517.179op/s ± 1173.445op/s 2140530.202op/s ± 820.029op/s 2141338.302op/s 2142308.092op/s 2142684.150op/s 2142921.108op/s 0.11% -0.605 1.900 0.05% 82.975op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 184.546µs 184.967µs ± 0.223µs 184.952µs ± 0.155µs 185.098µs 185.372µs 185.483µs 185.823µs 0.47% 0.620 0.392 0.12% 0.016µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5381452.850op/s 5406372.865op/s ± 6519.978op/s 5406809.716op/s ± 4547.451op/s 5411398.895op/s 5415142.784op/s 5418309.762op/s 5418712.005op/s 0.22% -0.613 0.374 0.12% 461.032op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 44.916µs 44.992µs ± 0.040µs 44.985µs ± 0.024µs 45.017µs 45.062µs 45.087µs 45.162µs 0.39% 0.838 1.427 0.09% 0.003µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 22142315.837op/s 22226092.452op/s ± 19824.036op/s 22229445.922op/s ± 11862.409op/s 22240229.325op/s 22255089.752op/s 22261437.017op/s 22263756.846op/s 0.15% -0.830 1.403 0.09% 1401.771op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 49.720µs 49.810µs ± 0.038µs 49.805µs ± 0.024µs 49.835µs 49.872µs 49.914µs 49.949µs 0.29% 0.454 0.591 0.08% 0.003µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 20020567.797op/s 20076178.022op/s ± 15465.685op/s 20078155.336op/s ± 9516.169op/s 20086857.805op/s 20095601.477op/s 20111163.248op/s 20112608.259op/s 0.17% -0.448 0.583 0.08% 1093.589op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [537.963µs; 538.027µs] or [-0.006%; +0.006%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1858642.992op/s; 1858864.175op/s] or [-0.006%; +0.006%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [467.141µs; 467.212µs] or [-0.008%; +0.008%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2140354.551op/s; 2140679.808op/s] or [-0.008%; +0.008%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [184.936µs; 184.998µs] or [-0.017%; +0.017%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5405469.258op/s; 5407276.471op/s] or [-0.017%; +0.017%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [44.987µs; 44.998µs] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [22223345.032op/s; 22228839.873op/s] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [49.805µs; 49.816µs] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [20074034.627op/s; 20078321.418op/s] or [-0.011%; +0.011%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 1.130µs 1.202µs ± 0.024µs 1.208µs ± 0.012µs 1.219µs 1.228µs 1.233µs 1.236µs 2.25% -1.370 1.320 2.03% 0.002µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [1.198µs; 1.205µs] or [-0.282%; +0.282%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.414µs 3.132µs ± 1.508µs 2.962µs ± 0.022µs 2.981µs 3.349µs 13.542µs 17.270µs 483.02% 7.946 63.777 48.02% 0.107µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.923µs; 3.341µs] or [-6.672%; +6.672%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.012ms 9.053ms ± 0.025ms 9.052ms ± 0.008ms 9.058ms 9.075ms 9.097ms 9.343ms 3.22% 7.628 84.615 0.28% 0.002ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.049ms; 9.056ms] or [-0.039%; +0.039%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.417µs 2.418µs 2.419µs 0.13% 0.625 1.026 0.03% 0.000µs 1 200
credit_card/is_card_number/ throughput 413324817.184op/s 413871558.193op/s ± 128647.560op/s 413867201.024op/s ± 89346.879op/s 413975105.434op/s 414049950.262op/s 414095475.438op/s 414127645.967op/s 0.06% -0.623 1.018 0.03% 9096.756op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 111.785µs 115.729µs ± 1.073µs 115.991µs ± 0.508µs 116.430µs 116.920µs 117.332µs 118.344µs 2.03% -1.299 2.097 0.93% 0.076µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 8449972.907op/s 8641650.627op/s ± 81148.818op/s 8621346.695op/s ± 37598.964op/s 8671765.382op/s 8798516.205op/s 8920449.797op/s 8945772.577op/s 3.76% 1.367 2.302 0.94% 5738.088op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 103.849µs 107.616µs ± 1.601µs 107.602µs ± 1.213µs 108.858µs 110.217µs 110.703µs 111.601µs 3.72% 0.073 -0.773 1.48% 0.113µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 8960504.409op/s 9294328.512op/s ± 138149.606op/s 9293481.140op/s ± 104789.069op/s 9394664.110op/s 9497225.779op/s 9573291.386op/s 9629341.785op/s 3.61% -0.019 -0.774 1.48% 9768.652op/s 1 200
credit_card/is_card_number/37828224631 execution_time 2.415µs 2.416µs ± 0.002µs 2.416µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.439µs 0.93% 9.519 114.268 0.07% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 410080185.092op/s 413847814.069op/s ± 305941.749op/s 413908257.476op/s ± 100047.096op/s 413988055.259op/s 414039912.369op/s 414063354.618op/s 414077225.052op/s 0.04% -9.461 113.310 0.07% 21633.349op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 101.722µs 105.789µs ± 1.370µs 105.963µs ± 1.024µs 106.794µs 107.693µs 107.926µs 108.328µs 2.23% -0.583 -0.064 1.29% 0.097µs 1 200
credit_card/is_card_number/378282246310005 throughput 9231255.086op/s 9454399.584op/s ± 123424.916op/s 9437278.527op/s ± 91202.444op/s 9531330.200op/s 9684279.454op/s 9786788.052op/s 9830757.813op/s 4.17% 0.646 0.059 1.30% 8727.460op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 104.581µs 106.397µs ± 0.499µs 106.464µs ± 0.261µs 106.724µs 107.012µs 107.431µs 107.528µs 1.00% -1.011 1.528 0.47% 0.035µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 9299909.722op/s 9398939.951op/s ± 44261.698op/s 9392869.987op/s ± 23025.623op/s 9415969.956op/s 9485972.719op/s 9530472.512op/s 9561945.897op/s 1.80% 1.047 1.609 0.47% 3129.775op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 21.753µs 22.531µs ± 0.410µs 22.481µs ± 0.296µs 22.856µs 23.257µs 23.500µs 23.741µs 5.60% 0.480 -0.394 1.81% 0.029µs 1 200
credit_card/is_card_number/x371413321323331 throughput 42120961.593op/s 44397611.120op/s ± 801136.416op/s 44481162.737op/s ± 593224.001op/s 45022669.448op/s 45597772.219op/s 45793452.572op/s 45971138.731op/s 3.35% -0.406 -0.486 1.80% 56648.899op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.420µs 0.17% 1.205 2.123 0.04% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 413166093.069op/s 413842801.338op/s ± 154076.405op/s 413857745.151op/s ± 103143.591op/s 413972383.071op/s 414023285.786op/s 414043625.533op/s 414069892.877op/s 0.05% -1.202 2.110 0.04% 10894.847op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 86.955µs 87.286µs ± 0.204µs 87.266µs ± 0.069µs 87.318µs 87.626µs 88.355µs 88.726µs 1.67% 3.865 20.376 0.23% 0.014µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 11270621.440op/s 11456686.745op/s ± 26566.159op/s 11459236.631op/s ± 9048.401op/s 11469962.462op/s 11482708.609op/s 11489309.645op/s 11500166.752op/s 0.36% -3.813 19.928 0.23% 1878.511op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 79.609µs 80.222µs ± 0.650µs 79.994µs ± 0.273µs 80.437µs 81.917µs 82.503µs 82.653µs 3.32% 1.954 3.763 0.81% 0.046µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 12098738.092op/s 12466281.302op/s ± 99457.723op/s 12500937.571op/s ± 42642.890op/s 12534815.137op/s 12553772.074op/s 12555122.941op/s 12561379.927op/s 0.48% -1.908 3.562 0.80% 7032.723op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.420µs 0.17% 1.086 1.339 0.04% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 413190708.981op/s 413849661.199op/s ± 152228.313op/s 413874104.167op/s ± 101317.919op/s 413969489.462op/s 414025527.986op/s 414045171.466op/s 414050334.832op/s 0.04% -1.083 1.329 0.04% 10764.167op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 76.351µs 77.300µs ± 0.879µs 76.910µs ± 0.121µs 77.225µs 79.201µs 80.324µs 80.639µs 4.85% 1.937 2.824 1.13% 0.062µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 12400914.310op/s 12938248.873op/s ± 143923.298op/s 13002241.594op/s ± 20483.061op/s 13017349.524op/s 13037278.209op/s 13053057.237op/s 13097468.743op/s 0.73% -1.902 2.642 1.11% 10176.914op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 104.740µs 106.415µs ± 0.462µs 106.466µs ± 0.268µs 106.700µs 106.983µs 107.570µs 107.970µs 1.41% -0.398 1.708 0.43% 0.033µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 9261851.154op/s 9397305.322op/s ± 40836.622op/s 9392706.094op/s ± 23583.443op/s 9421232.190op/s 9465473.535op/s 9517259.228op/s 9547416.890op/s 1.65% 0.444 1.739 0.43% 2887.585op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 21.758µs 22.475µs ± 0.393µs 22.435µs ± 0.300µs 22.745µs 23.192µs 23.342µs 23.398µs 4.30% 0.340 -0.776 1.74% 0.028µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 42737817.541op/s 44506748.924op/s ± 773082.727op/s 44574008.971op/s ± 598762.562op/s 45146703.723op/s 45607363.737op/s 45783832.109op/s 45959383.369op/s 3.11% -0.282 -0.829 1.73% 54665.204op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [2.416µs; 2.416µs] or [-0.004%; +0.004%] None None None
credit_card/is_card_number/ throughput [413853728.879op/s; 413889387.508op/s] or [-0.004%; +0.004%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [115.580µs; 115.877µs] or [-0.129%; +0.129%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [8630404.181op/s; 8652897.073op/s] or [-0.130%; +0.130%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [107.394µs; 107.838µs] or [-0.206%; +0.206%] None None None
credit_card/is_card_number/ 378282246310005 throughput [9275182.305op/s; 9313474.719op/s] or [-0.206%; +0.206%] None None None
credit_card/is_card_number/37828224631 execution_time [2.416µs; 2.417µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/37828224631 throughput [413805413.485op/s; 413890214.653op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/378282246310005 execution_time [105.599µs; 105.979µs] or [-0.179%; +0.179%] None None None
credit_card/is_card_number/378282246310005 throughput [9437294.078op/s; 9471505.090op/s] or [-0.181%; +0.181%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [106.328µs; 106.466µs] or [-0.065%; +0.065%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [9392805.705op/s; 9405074.197op/s] or [-0.065%; +0.065%] None None None
credit_card/is_card_number/x371413321323331 execution_time [22.474µs; 22.588µs] or [-0.252%; +0.252%] None None None
credit_card/is_card_number/x371413321323331 throughput [44286581.318op/s; 44508640.922op/s] or [-0.250%; +0.250%] None None None
credit_card/is_card_number_no_luhn/ execution_time [2.416µs; 2.417µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ throughput [413821447.830op/s; 413864154.846op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [87.257µs; 87.314µs] or [-0.032%; +0.032%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [11453004.931op/s; 11460368.560op/s] or [-0.032%; +0.032%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [80.131µs; 80.312µs] or [-0.112%; +0.112%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [12452497.419op/s; 12480065.186op/s] or [-0.111%; +0.111%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [2.416µs; 2.416µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [413828563.819op/s; 413870758.579op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [77.178µs; 77.422µs] or [-0.158%; +0.158%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [12918302.488op/s; 12958195.258op/s] or [-0.154%; +0.154%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [106.352µs; 106.479µs] or [-0.060%; +0.060%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [9391645.759op/s; 9402964.885op/s] or [-0.060%; +0.060%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [22.421µs; 22.530µs] or [-0.242%; +0.242%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [44399607.093op/s; 44613890.754op/s] or [-0.241%; +0.241%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 16f094c 1726600471 ekump/eval-bench-perf-for-io-read-impl
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.665µs 2.720µs ± 0.020µs 2.723µs ± 0.009µs 2.728µs 2.746µs 2.755µs 2.866µs 5.25% 1.554 15.311 0.72% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.717µs; 2.723µs] or [-0.099%; +0.099%] None None None

Baseline

Omitted due to size.

@ekump ekump force-pushed the ekump/eval-bench-perf-for-io-read-impl branch from 6f6ad62 to 16f094c Compare September 17, 2024 19:42
@ekump ekump closed this Sep 19, 2024
@ekump ekump deleted the ekump/eval-bench-perf-for-io-read-impl branch September 19, 2024 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants