Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix token cancellation flakiness #646

Merged
merged 3 commits into from
Sep 23, 2024
Merged

Fix token cancellation flakiness #646

merged 3 commits into from
Sep 23, 2024

Conversation

VianneyRuhlmann
Copy link
Contributor

@VianneyRuhlmann VianneyRuhlmann commented Sep 23, 2024 •
edited
Loading

What does this PR do?

The token cancellation test is flaky in the coverage pipeline. I fixed this by increasing the delay for the force flush to complete.

@pr-commenter
Copy link

pr-commenter bot commented Sep 23, 2024 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2024-09-23 12:20:02

Comparing candidate commit eb2e387 in PR branch vianney/data-pipeline/fix-flaky-test with baseline commit 48d6360 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.010ms 9.046ms ± 0.014ms 9.046ms ± 0.009ms 9.054ms 9.069ms 9.085ms 9.095ms 0.55% 0.245 0.432 0.16% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.044ms; 9.048ms] or [-0.022%; +0.022%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 39.082µs 39.481µs ± 0.638µs 39.194µs ± 0.056µs 39.276µs 40.806µs 40.861µs 42.394µs 8.16% 1.849 2.242 1.61% 0.045µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [39.392µs; 39.569µs] or [-0.224%; +0.224%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 143.093µs 143.778µs ± 0.392µs 143.710µs ± 0.176µs 143.896µs 144.307µs 145.463µs 145.710µs 1.39% 2.215 7.650 0.27% 0.028µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [143.723µs; 143.832µs] or [-0.038%; +0.038%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 297.836ns 308.228ns ± 12.181ns 301.729ns ± 3.037ns 313.972ns 337.507ns 340.728ns 340.884ns 12.98% 1.291 0.416 3.94% 0.861ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [306.540ns; 309.916ns] or [-0.548%; +0.548%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.392µs 3.179µs ± 1.529µs 3.026µs ± 0.027µs 3.051µs 3.125µs 13.709µs 17.539µs 479.57% 7.958 63.984 47.96% 0.108µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.967µs; 3.391µs] or [-6.664%; +6.664%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 276.115µs 276.469µs ± 0.194µs 276.448µs ± 0.120µs 276.581µs 276.859µs 276.965µs 277.012µs 0.20% 0.584 0.029 0.07% 0.014µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3609948.770op/s 3617043.991op/s ± 2538.678op/s 3617322.008op/s ± 1567.042op/s 3618782.254op/s 3620648.478op/s 3621468.227op/s 3621678.920op/s 0.12% -0.580 0.024 0.07% 179.512op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 27.996µs 28.122µs ± 0.056µs 28.110µs ± 0.035µs 28.153µs 28.229µs 28.264µs 28.280µs 0.61% 0.625 -0.023 0.20% 0.004µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 35360606.788op/s 35559577.507op/s ± 71062.215op/s 35574601.650op/s ± 44392.247op/s 35602538.745op/s 35658870.580op/s 35682724.016op/s 35719952.822op/s 0.41% -0.615 -0.033 0.20% 5024.857op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 15.953µs 16.252µs ± 0.121µs 16.226µs ± 0.100µs 16.353µs 16.436µs 16.477µs 16.564µs 2.08% 0.182 -0.938 0.74% 0.009µs 1 200
normalization/normalize_name/normalize_name/good throughput 60371489.308op/s 61535859.872op/s ± 457489.557op/s 61627874.313op/s ± 378013.791op/s 61894033.507op/s 62161057.692op/s 62374398.691op/s 62683302.178op/s 1.71% -0.159 -0.941 0.74% 32349.397op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [276.442µs; 276.496µs] or [-0.010%; +0.010%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3616692.155op/s; 3617395.828op/s] or [-0.010%; +0.010%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [28.114µs; 28.130µs] or [-0.028%; +0.028%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [35549728.968op/s; 35569426.047op/s] or [-0.028%; +0.028%] None None None
normalization/normalize_name/normalize_name/good execution_time [16.235µs; 16.268µs] or [-0.103%; +0.103%] None None None
normalization/normalize_name/normalize_name/good throughput [61472456.220op/s; 61599263.525op/s] or [-0.103%; +0.103%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 73.368µs 73.535µs ± 0.172µs 73.513µs ± 0.043µs 73.558µs 73.678µs 73.746µs 75.621µs 2.87% 9.581 109.248 0.23% 0.012µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [73.511µs; 73.559µs] or [-0.032%; +0.032%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 1.134µs 1.198µs ± 0.022µs 1.206µs ± 0.012µs 1.215µs 1.218µs 1.219µs 1.220µs 1.19% -1.398 1.192 1.85% 0.002µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [1.195µs; 1.201µs] or [-0.257%; +0.257%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.666µs 2.719µs ± 0.017µs 2.718µs ± 0.006µs 2.727µs 2.743µs 2.746µs 2.753µs 1.31% -1.022 1.874 0.61% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.716µs; 2.721µs] or [-0.085%; +0.085%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 18.092µs 23.651µs ± 10.866µs 19.100µs ± 0.388µs 19.953µs 37.513µs 50.072µs 104.539µs 447.31% 4.279 26.330 45.83% 0.768µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [22.145µs; 25.157µs] or [-6.367%; +6.367%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 537.573µs 538.169µs ± 0.410µs 538.130µs ± 0.200µs 538.320µs 538.885µs 539.361µs 540.572µs 0.45% 1.999 7.226 0.08% 0.029µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1849893.343op/s 1858152.651op/s ± 1412.158op/s 1858288.758op/s ± 689.583op/s 1859007.664op/s 1859940.242op/s 1860152.454op/s 1860212.881op/s 0.10% -1.987 7.148 0.08% 99.855op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 466.525µs 467.103µs ± 0.244µs 467.103µs ± 0.175µs 467.274µs 467.478µs 467.746µs 467.914µs 0.17% 0.184 0.054 0.05% 0.017µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2137143.658op/s 2140858.092op/s ± 1119.661op/s 2140857.373op/s ± 803.994op/s 2141701.027op/s 2142739.228op/s 2143000.417op/s 2143510.027op/s 0.12% -0.181 0.050 0.05% 79.172op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 184.567µs 185.132µs ± 0.235µs 185.134µs ± 0.119µs 185.231µs 185.510µs 185.760µs 186.156µs 0.55% 0.672 2.046 0.13% 0.017µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5371849.494op/s 5401552.798op/s ± 6856.968op/s 5401506.102op/s ± 3477.030op/s 5405900.520op/s 5411490.883op/s 5415755.560op/s 5418099.867op/s 0.31% -0.659 2.008 0.13% 484.861op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 44.690µs 44.786µs ± 0.058µs 44.780µs ± 0.026µs 44.807µs 44.854µs 44.900µs 45.380µs 1.34% 5.499 52.956 0.13% 0.004µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 22035911.756op/s 22328339.993op/s ± 28791.887op/s 22331507.798op/s ± 13026.503op/s 22343077.093op/s 22360146.801op/s 22372881.694op/s 22376458.088op/s 0.20% -5.403 51.667 0.13% 2035.894op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 49.740µs 49.825µs ± 0.070µs 49.817µs ± 0.059µs 49.875µs 49.952µs 49.997µs 50.020µs 0.41% 0.597 -0.516 0.14% 0.005µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 19992040.378op/s 20070108.076op/s ± 28306.499op/s 20073356.447op/s ± 23747.653op/s 20097739.279op/s 20102923.227op/s 20104398.213op/s 20104520.127op/s 0.16% -0.592 -0.526 0.14% 2001.572op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [538.112µs; 538.226µs] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1857956.940op/s; 1858348.363op/s] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [467.069µs; 467.136µs] or [-0.007%; +0.007%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2140702.918op/s; 2141013.266op/s] or [-0.007%; +0.007%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [185.100µs; 185.165µs] or [-0.018%; +0.018%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5400602.488op/s; 5402503.108op/s] or [-0.018%; +0.018%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [44.778µs; 44.794µs] or [-0.018%; +0.018%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [22324349.714op/s; 22332330.271op/s] or [-0.018%; +0.018%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [49.816µs; 49.835µs] or [-0.020%; +0.020%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [20066185.067op/s; 20074031.084op/s] or [-0.020%; +0.020%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz eb2e387 1727093288 vianney/data-pipeline/fix-flaky-test
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 2.415µs 2.417µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.419µs 0.13% 0.759 -0.111 0.04% 0.000µs 1 200
credit_card/is_card_number/ throughput 413310907.963op/s 413809715.149op/s ± 158401.317op/s 413848020.707op/s ± 107566.322op/s 413945389.114op/s 413985427.965op/s 414018244.879op/s 414086835.980op/s 0.06% -0.758 -0.115 0.04% 11200.665op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 111.332µs 115.859µs ± 1.295µs 115.905µs ± 0.875µs 116.806µs 117.750µs 118.659µs 119.210µs 2.85% -0.366 0.491 1.11% 0.092µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 8388550.727op/s 8632226.890op/s ± 96890.798op/s 8627768.961op/s ± 64866.122op/s 8689011.235op/s 8794931.967op/s 8888024.574op/s 8982149.990op/s 4.11% 0.446 0.606 1.12% 6851.214op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 104.085µs 107.283µs ± 1.458µs 107.156µs ± 0.952µs 108.173µs 110.073µs 110.691µs 111.255µs 3.82% 0.429 -0.276 1.36% 0.103µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 8988369.630op/s 9322815.496op/s ± 126032.933op/s 9332145.866op/s ± 83644.925op/s 9414201.812op/s 9506546.067op/s 9568293.075op/s 9607518.410op/s 2.95% -0.367 -0.328 1.35% 8911.874op/s 1 200
credit_card/is_card_number/37828224631 execution_time 2.415µs 2.417µs ± 0.001µs 2.417µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.430µs 0.55% 6.138 62.222 0.05% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 411522020.007op/s 413791651.809op/s ± 213833.004op/s 413789497.540op/s ± 120324.937op/s 413931179.366op/s 413986649.479op/s 414028606.819op/s 414055312.148op/s 0.06% -6.097 61.637 0.05% 15120.277op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 101.709µs 105.696µs ± 1.441µs 105.744µs ± 1.115µs 106.870µs 107.869µs 108.198µs 108.278µs 2.40% -0.306 -0.502 1.36% 0.102µs 1 200
credit_card/is_card_number/378282246310005 throughput 9235466.681op/s 9462882.878op/s ± 129559.187op/s 9456818.277op/s ± 99694.538op/s 9555383.769op/s 9673483.612op/s 9790035.885op/s 9831939.044op/s 3.97% 0.364 -0.427 1.37% 9161.218op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 104.945µs 106.575µs ± 0.420µs 106.607µs ± 0.227µs 106.819µs 107.190µs 107.881µs 107.987µs 1.29% -0.191 1.744 0.39% 0.030µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 9260393.092op/s 9383233.590op/s ± 37038.406op/s 9380203.558op/s ± 19934.621op/s 9403400.072op/s 9450010.877op/s 9466949.311op/s 9528820.119op/s 1.58% 0.235 1.756 0.39% 2619.011op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 21.676µs 22.463µs ± 0.360µs 22.447µs ± 0.220µs 22.665µs 23.119µs 23.297µs 23.349µs 4.02% 0.219 -0.222 1.60% 0.025µs 1 200
credit_card/is_card_number/x371413321323331 throughput 42827822.271op/s 44529843.172op/s ± 711880.430op/s 44549872.939op/s ± 432682.942op/s 45000191.227op/s 45657577.571op/s 46049952.805op/s 46133217.075op/s 3.55% -0.136 -0.245 1.59% 50337.548op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 2.415µs 2.417µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.420µs 0.17% 0.950 1.300 0.04% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 413137822.698op/s 413819091.364op/s ± 156891.303op/s 413848399.662op/s ± 110811.977op/s 413946733.768op/s 414012632.749op/s 414039499.081op/s 414052240.625op/s 0.05% -0.947 1.288 0.04% 11093.890op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 86.959µs 87.321µs ± 0.362µs 87.250µs ± 0.077µs 87.322µs 87.880µs 89.092µs 89.576µs 2.67% 3.699 15.610 0.41% 0.026µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 11163678.554op/s 11452164.436op/s ± 46798.144op/s 11461319.194op/s ± 10152.124op/s 11472101.001op/s 11489265.126op/s 11499210.867op/s 11499737.806op/s 0.34% -3.651 15.217 0.41% 3309.128op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 79.485µs 80.313µs ± 0.736µs 80.072µs ± 0.276µs 80.443µs 82.342µs 82.619µs 82.815µs 3.43% 1.894 2.863 0.91% 0.052µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 12075090.836op/s 12452362.702op/s ± 112125.578op/s 12488760.749op/s ± 43126.639op/s 12526453.091op/s 12547307.116op/s 12552713.241op/s 12580915.416op/s 0.74% -1.860 2.737 0.90% 7928.476op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.418µs 2.419µs 2.421µs 0.18% 0.936 1.713 0.04% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 413104402.232op/s 413827388.520op/s ± 153400.883op/s 413835731.311op/s ± 117585.876op/s 413954290.105op/s 414027077.331op/s 414040811.143op/s 414066289.022op/s 0.06% -0.933 1.698 0.04% 10847.080op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 76.555µs 77.168µs ± 0.602µs 76.937µs ± 0.128µs 77.222µs 78.826µs 79.391µs 79.786µs 3.70% 2.265 4.766 0.78% 0.043µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 12533577.937op/s 12959504.527op/s ± 99307.518op/s 12997694.859op/s ± 21620.467op/s 13013022.784op/s 13036586.208op/s 13056607.667op/s 13062526.395op/s 0.50% -2.228 4.574 0.76% 7022.102op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 105.501µs 106.630µs ± 0.386µs 106.607µs ± 0.221µs 106.870µs 107.297µs 107.521µs 107.903µs 1.22% 0.103 0.778 0.36% 0.027µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 9267602.110op/s 9378321.529op/s ± 33926.924op/s 9380246.631op/s ± 19409.100op/s 9398139.876op/s 9437861.445op/s 9463503.030op/s 9478621.373op/s 1.05% -0.073 0.763 0.36% 2398.996op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 21.722µs 22.599µs ± 0.395µs 22.599µs ± 0.292µs 22.869µs 23.308µs 23.466µs 23.613µs 4.49% 0.184 -0.535 1.74% 0.028µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 42349586.120op/s 44263399.801op/s ± 771302.914op/s 44250080.155op/s ± 573815.613op/s 44838719.251op/s 45469732.840op/s 45752904.615op/s 46036582.823op/s 4.04% -0.109 -0.570 1.74% 54539.352op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [2.416µs; 2.417µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number/ throughput [413787762.250op/s; 413831668.048op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [115.680µs; 116.039µs] or [-0.155%; +0.155%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [8618798.757op/s; 8645655.023op/s] or [-0.156%; +0.156%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [107.081µs; 107.485µs] or [-0.188%; +0.188%] None None None
credit_card/is_card_number/ 378282246310005 throughput [9305348.544op/s; 9340282.448op/s] or [-0.187%; +0.187%] None None None
credit_card/is_card_number/37828224631 execution_time [2.417µs; 2.417µs] or [-0.007%; +0.007%] None None None
credit_card/is_card_number/37828224631 throughput [413762016.611op/s; 413821287.007op/s] or [-0.007%; +0.007%] None None None
credit_card/is_card_number/378282246310005 execution_time [105.496µs; 105.895µs] or [-0.189%; +0.189%] None None None
credit_card/is_card_number/378282246310005 throughput [9444927.221op/s; 9480838.535op/s] or [-0.190%; +0.190%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [106.516µs; 106.633µs] or [-0.055%; +0.055%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [9378100.423op/s; 9388366.757op/s] or [-0.055%; +0.055%] None None None
credit_card/is_card_number/x371413321323331 execution_time [22.413µs; 22.512µs] or [-0.222%; +0.222%] None None None
credit_card/is_card_number/x371413321323331 throughput [44431183.391op/s; 44628502.953op/s] or [-0.222%; +0.222%] None None None
credit_card/is_card_number_no_luhn/ execution_time [2.416µs; 2.417µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ throughput [413797347.738op/s; 413840834.990op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [87.271µs; 87.371µs] or [-0.057%; +0.057%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [11445678.663op/s; 11458650.209op/s] or [-0.057%; +0.057%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [80.211µs; 80.415µs] or [-0.127%; +0.127%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [12436823.175op/s; 12467902.228op/s] or [-0.125%; +0.125%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [2.416µs; 2.417µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [413806128.633op/s; 413848648.407op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [77.085µs; 77.251µs] or [-0.108%; +0.108%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [12945741.460op/s; 12973267.594op/s] or [-0.106%; +0.106%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [106.577µs; 106.684µs] or [-0.050%; +0.050%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [9373619.583op/s; 9383023.474op/s] or [-0.050%; +0.050%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [22.544µs; 22.654µs] or [-0.242%; +0.242%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [44156504.636op/s; 44370294.967op/s] or [-0.241%; +0.241%] None None None

Baseline

Omitted due to size.

@codecov-commenter
Copy link

codecov-commenter commented Sep 23, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.62%. Comparing base (48d6360) to head (eb2e387).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #646      +/-   ##
==========================================
+ Coverage   73.56%   73.62%   +0.05%     
==========================================
  Files         255      255              
  Lines       36941    36938       -3     
==========================================
+ Hits        27176    27194      +18     
+ Misses       9765     9744      -21
Components Coverage Δ
crashtracker 20.53% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 91.64% <100.00%> (-0.02%) ⬇️
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.08% <ø> (ø)
ddcommon-ffi 69.52% <ø> (ø)
ddtelemetry 59.10% <ø> (ø)
ipc 83.58% <ø> (ø)
profiling 84.26% <ø> (ø)
profiling-ffi 77.42% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 40.10% <ø> (ø)
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 50.36% <ø> (ø)
tinybytes 94.77% <ø> (ø)
trace-mini-agent 72.47% <ø> (ø)
trace-normalization 98.25% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.67% <ø> (ø)
trace-utils 93.41% <ø> (ø)

@VianneyRuhlmann VianneyRuhlmann marked this pull request as ready for review September 23, 2024 12:08
@VianneyRuhlmann VianneyRuhlmann requested a review from a team as a code owner September 23, 2024 12:08
@VianneyRuhlmann VianneyRuhlmann merged commit 5635814 into main Sep 23, 2024
30 checks passed
@VianneyRuhlmann VianneyRuhlmann deleted the vianney/data-pipeline/fix-flaky-test branch September 23, 2024 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bantonsson bantonsson bantonsson approved these changes

Assignees
No one assigned
Labels
data-pipeline
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@VianneyRuhlmann @codecov-commenter @bantonsson