improve assert failure message for send_data retry tests and increase tolerance for time measurements

[ekump]

What does this PR do?

It has been observed that some of the send_data retry strategy logic can be flaky on the macOS github actions runners. This PR adds more time to the retry observation tolerances for these tests. Also, a bit more info was added to the relevant test assertions when a failure does occur.

Motivation

@bwoebi

[pr-commenter]

Benchmarks

Comparison

Benchmark execution time: 2024-09-06 20:34:39

Comparing candidate commit 94befa0 in PR branch ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac with baseline commit c48d94b in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	38.046µs	38.608µs ± 1.006µs	38.143µs ± 0.034µs	38.209µs	40.770µs	40.818µs	41.452µs	8.67%	1.702	0.953	2.60%	0.071µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[38.468µs; 38.747µs] or [-0.361%; +0.361%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	295.118ns	307.260ns ± 13.914ns	301.801ns ± 4.222ns	309.474ns	340.483ns	344.861ns	359.556ns	19.14%	1.664	1.784	4.52%	0.984ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[305.332ns; 309.189ns] or [-0.628%; +0.628%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	618.585µs	619.237µs ± 0.271µs	619.211µs ± 0.187µs	619.425µs	619.653µs	620.038µs	620.394µs	0.19%	0.716	1.594	0.04%	0.019µs	1	200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	1611878.383op/s	1614891.621op/s ± 705.846op/s	1614957.608op/s ± 486.903op/s	1615389.841op/s	1615816.247op/s	1616361.398op/s	1616592.185op/s	0.10%	-0.712	1.582	0.04%	49.911op/s	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	466.013µs	467.582µs ± 1.115µs	467.492µs ± 1.001µs	468.491µs	469.398µs	469.544µs	470.149µs	0.57%	0.324	-1.209	0.24%	0.079µs	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	2126986.227op/s	2138674.486op/s ± 5096.332op/s	2139071.870op/s ± 4589.549op/s	2143660.693op/s	2145168.207op/s	2145614.214op/s	2145863.853op/s	0.32%	-0.319	-1.214	0.24%	360.365op/s	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	190.546µs	190.827µs ± 0.144µs	190.815µs ± 0.091µs	190.903µs	191.074µs	191.272µs	191.325µs	0.27%	0.766	0.891	0.08%	0.010µs	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	5226709.881op/s	5240346.384op/s ± 3961.979op/s	5240685.891op/s ± 2508.816op/s	5243262.509op/s	5245811.603op/s	5247638.786op/s	5248080.817op/s	0.14%	-0.761	0.878	0.08%	280.154op/s	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	46.880µs	47.321µs ± 0.163µs	47.331µs ± 0.097µs	47.423µs	47.577µs	47.662µs	47.707µs	0.79%	-0.279	-0.011	0.34%	0.011µs	1	200
normalization/normalize_service/normalize_service/[empty string]	throughput	20961293.990op/s	21132379.684op/s ± 72649.466op/s	21127622.242op/s ± 43221.157op/s	21171769.885op/s	21258652.809op/s	21323282.213op/s	21331260.978op/s	0.96%	0.299	0.001	0.34%	5137.093op/s	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	51.630µs	51.849µs ± 0.140µs	51.834µs ± 0.075µs	51.913µs	52.033µs	52.176µs	53.103µs	2.45%	3.919	31.347	0.27%	0.010µs	1	200
normalization/normalize_service/normalize_service/test_ASCII	throughput	18831205.370op/s	19286944.545op/s ± 51382.443op/s	19292431.701op/s ± 27797.199op/s	19318458.580op/s	19345869.182op/s	19357371.952op/s	19368744.933op/s	0.40%	-3.778	29.670	0.27%	3633.287op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[619.199µs; 619.274µs] or [-0.006%; +0.006%]	None	None	None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	[1614793.798op/s; 1614989.445op/s] or [-0.006%; +0.006%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[467.427µs; 467.737µs] or [-0.033%; +0.033%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	[2137968.184op/s; 2139380.789op/s] or [-0.033%; +0.033%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[190.807µs; 190.847µs] or [-0.010%; +0.010%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	[5239797.292op/s; 5240895.476op/s] or [-0.010%; +0.010%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[47.299µs; 47.344µs] or [-0.048%; +0.048%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	throughput	[21122311.167op/s; 21142448.201op/s] or [-0.048%; +0.048%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[51.830µs; 51.868µs] or [-0.037%; +0.037%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	throughput	[19279823.432op/s; 19294065.657op/s] or [-0.037%; +0.037%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	9.027ms	9.062ms ± 0.016ms	9.062ms ± 0.012ms	9.073ms	9.088ms	9.102ms	9.116ms	0.59%	0.277	0.066	0.18%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[9.059ms; 9.064ms] or [-0.024%; +0.024%]	None	None	None

Group 5

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.677µs	2.736µs ± 0.018µs	2.740µs ± 0.009µs	2.748µs	2.755µs	2.760µs	2.762µs	0.80%	-1.422	1.808	0.65%	0.001µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.733µs; 2.738µs] or [-0.091%; +0.091%]	None	None	None

Group 6

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	17.076µs	22.958µs ± 14.864µs	17.235µs ± 0.066µs	17.466µs	51.572µs	54.306µs	135.534µs	686.39%	4.057	22.567	64.58%	1.051µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[20.898µs; 25.018µs] or [-8.973%; +8.973%]	None	None	None

Group 7

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	138.013µs	138.371µs ± 0.250µs	138.350µs ± 0.097µs	138.449µs	138.635µs	138.875µs	141.115µs	2.00%	6.736	70.687	0.18%	0.018µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[138.336µs; 138.405µs] or [-0.025%; +0.025%]	None	None	None

Group 8

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	75.449µs	75.705µs ± 0.134µs	75.708µs ± 0.037µs	75.740µs	75.787µs	76.174µs	77.093µs	1.83%	6.118	58.207	0.18%	0.009µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[75.686µs; 75.723µs] or [-0.025%; +0.025%]	None	None	None

Group 9

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching deserializing traces from msgpack to their internal representation	execution_time	1.227µs	1.391µs ± 0.066µs	1.393µs ± 0.055µs	1.447µs	1.479µs	1.502µs	1.503µs	7.84%	-0.394	-0.779	4.72%	0.005µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching deserializing traces from msgpack to their internal representation	execution_time	[1.382µs; 1.400µs] or [-0.656%; +0.656%]	None	None	None

Group 10

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	2.413µs	2.414µs ± 0.001µs	2.414µs ± 0.000µs	2.415µs	2.415µs	2.416µs	2.416µs	0.07%	0.413	-0.652	0.03%	0.000µs	1	200
credit_card/is_card_number/	throughput	413948112.573op/s	414229011.407op/s ± 107182.896op/s	414249337.096op/s ± 78980.384op/s	414316277.863op/s	414377923.525op/s	414402579.044op/s	414419559.483op/s	0.04%	-0.412	-0.654	0.03%	7578.975op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	111.193µs	113.295µs ± 0.928µs	113.188µs ± 0.525µs	113.839µs	114.865µs	115.541µs	117.710µs	3.99%	0.860	1.946	0.82%	0.066µs	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	8495490.613op/s	8827116.830op/s ± 71819.764op/s	8834861.451op/s ± 40863.541op/s	8870925.990op/s	8929028.773op/s	8960228.831op/s	8993358.253op/s	1.79%	-0.783	1.666	0.81%	5078.424op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	107.086µs	107.858µs ± 0.410µs	107.760µs ± 0.244µs	108.093µs	108.599µs	109.046µs	109.155µs	1.29%	0.887	0.385	0.38%	0.029µs	1	200
credit_card/is_card_number/ 378282246310005	throughput	9161304.266op/s	9271578.777op/s ± 35143.009op/s	9279842.790op/s ± 20990.101op/s	9299175.628op/s	9313045.578op/s	9323546.795op/s	9338249.470op/s	0.63%	-0.869	0.342	0.38%	2484.986op/s	1	200
credit_card/is_card_number/37828224631	execution_time	2.413µs	2.415µs ± 0.005µs	2.414µs ± 0.001µs	2.415µs	2.416µs	2.416µs	2.483µs	2.84%	13.566	186.260	0.20%	0.000µs	1	200
credit_card/is_card_number/37828224631	throughput	402762113.997op/s	414147542.489op/s ± 818829.986op/s	414210033.582op/s ± 93612.290op/s	414305085.596op/s	414355407.605op/s	414385305.861op/s	414455793.933op/s	0.06%	-13.540	185.778	0.20%	57900.024op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	104.364µs	105.042µs ± 0.424µs	104.968µs ± 0.287µs	105.303µs	105.845µs	106.325µs	106.543µs	1.50%	0.930	0.626	0.40%	0.030µs	1	200
credit_card/is_card_number/378282246310005	throughput	9385861.568op/s	9520121.587op/s ± 38242.154op/s	9526691.163op/s ± 26103.226op/s	9551497.259op/s	9566976.520op/s	9580727.008op/s	9581830.932op/s	0.58%	-0.909	0.566	0.40%	2704.129op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	101.876µs	102.235µs ± 0.172µs	102.210µs ± 0.092µs	102.319µs	102.585µs	102.718µs	102.900µs	0.67%	0.901	1.225	0.17%	0.012µs	1	200
credit_card/is_card_number/37828224631000521389798	throughput	9718188.403op/s	9781435.400op/s ± 16467.287op/s	9783738.453op/s ± 8789.848op/s	9792159.663op/s	9802365.368op/s	9814296.860op/s	9815829.989op/s	0.33%	-0.889	1.194	0.17%	1164.413op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	22.113µs	23.255µs ± 0.533µs	23.241µs ± 0.417µs	23.694µs	24.083µs	24.284µs	24.435µs	5.14%	0.038	-0.797	2.28%	0.038µs	1	200
credit_card/is_card_number/x371413321323331	throughput	40924336.720op/s	43023725.790op/s ± 985353.404op/s	43027766.699op/s ± 785306.250op/s	43778227.525op/s	44632818.295op/s	45050548.109op/s	45222358.018op/s	5.10%	0.045	-0.777	2.28%	69675.007op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	2.413µs	2.414µs ± 0.001µs	2.414µs ± 0.001µs	2.415µs	2.416µs	2.416µs	2.417µs	0.12%	0.653	0.483	0.03%	0.000µs	1	200
credit_card/is_card_number_no_luhn/	throughput	413702943.911op/s	414194794.791op/s ± 121138.962op/s	414200509.039op/s ± 94125.191op/s	414297628.146op/s	414352923.229op/s	414388212.482op/s	414405217.485op/s	0.05%	-0.651	0.478	0.03%	8565.818op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	88.858µs	89.914µs ± 0.586µs	89.725µs ± 0.324µs	90.210µs	91.029µs	91.605µs	92.018µs	2.56%	1.052	0.786	0.65%	0.041µs	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	10867480.104op/s	11122218.547op/s ± 71966.537op/s	11145154.698op/s ± 40301.111op/s	11179352.016op/s	11204265.859op/s	11233256.886op/s	11253884.725op/s	0.98%	-1.020	0.690	0.65%	5088.803op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	85.482µs	86.297µs ± 0.418µs	86.269µs ± 0.266µs	86.544µs	86.906µs	87.281µs	89.107µs	3.29%	1.670	8.872	0.48%	0.030µs	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	11222523.682op/s	11588154.337op/s ± 55722.462op/s	11591622.735op/s ± 35882.128op/s	11627252.536op/s	11666830.111op/s	11686663.311op/s	11698330.315op/s	0.92%	-1.557	7.979	0.48%	3940.173op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	2.413µs	2.414µs ± 0.001µs	2.414µs ± 0.000µs	2.415µs	2.415µs	2.417µs	2.421µs	0.29%	3.090	19.619	0.04%	0.000µs	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	413043599.689op/s	414201127.359op/s ± 146084.236op/s	414224598.767op/s ± 84796.532op/s	414304262.359op/s	414347174.667op/s	414380609.861op/s	414448011.996op/s	0.05%	-3.077	19.487	0.04%	10329.715op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	82.347µs	83.106µs ± 0.285µs	83.062µs ± 0.157µs	83.237µs	83.652µs	83.850µs	84.137µs	1.29%	0.659	0.644	0.34%	0.020µs	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	11885386.760op/s	12032959.336op/s ± 41147.151op/s	12039155.912op/s ± 22794.799op/s	12059798.664op/s	12089816.083op/s	12119084.131op/s	12143664.976op/s	0.87%	-0.637	0.611	0.34%	2909.543op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	101.824µs	102.222µs ± 0.189µs	102.209µs ± 0.106µs	102.305µs	102.574µs	102.789µs	103.006µs	0.78%	1.069	1.994	0.18%	0.013µs	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	9708207.923op/s	9782643.528op/s ± 18078.336op/s	9783880.080op/s ± 10183.540op/s	9794448.752op/s	9805478.469op/s	9814942.711op/s	9820839.822op/s	0.38%	-1.053	1.943	0.18%	1278.331op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	21.891µs	23.326µs ± 0.543µs	23.321µs ± 0.355µs	23.693µs	24.202µs	24.436µs	24.589µs	5.44%	-0.178	-0.260	2.32%	0.038µs	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	40668199.322op/s	42894404.844op/s ± 1004632.780op/s	42879627.901op/s ± 654829.418op/s	43486859.205op/s	44586903.714op/s	45370707.903op/s	45681043.270op/s	6.53%	0.298	-0.184	2.34%	71038.265op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[2.414µs; 2.414µs] or [-0.004%; +0.004%]	None	None	None
credit_card/is_card_number/	throughput	[414214156.889op/s; 414243865.926op/s] or [-0.004%; +0.004%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[113.166µs; 113.423µs] or [-0.114%; +0.114%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[8817163.301op/s; 8837070.358op/s] or [-0.113%; +0.113%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[107.801µs; 107.915µs] or [-0.053%; +0.053%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[9266708.293op/s; 9276449.260op/s] or [-0.053%; +0.053%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[2.414µs; 2.415µs] or [-0.028%; +0.028%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[414034060.528op/s; 414261024.449op/s] or [-0.027%; +0.027%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[104.984µs; 105.101µs] or [-0.056%; +0.056%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[9514821.592op/s; 9525421.582op/s] or [-0.056%; +0.056%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[102.211µs; 102.259µs] or [-0.023%; +0.023%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[9779153.192op/s; 9783717.607op/s] or [-0.023%; +0.023%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[23.181µs; 23.329µs] or [-0.317%; +0.317%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[42887165.285op/s; 43160286.295op/s] or [-0.317%; +0.317%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[2.414µs; 2.414µs] or [-0.004%; +0.004%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[414178006.095op/s; 414211583.486op/s] or [-0.004%; +0.004%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[89.833µs; 89.995µs] or [-0.090%; +0.090%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[11112244.677op/s; 11132192.417op/s] or [-0.090%; +0.090%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[86.239µs; 86.355µs] or [-0.067%; +0.067%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[11580431.740op/s; 11595876.934op/s] or [-0.067%; +0.067%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[2.414µs; 2.414µs] or [-0.005%; +0.005%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[414180881.489op/s; 414221373.229op/s] or [-0.005%; +0.005%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[83.067µs; 83.146µs] or [-0.047%; +0.047%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[12027256.736op/s; 12038661.935op/s] or [-0.047%; +0.047%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[102.196µs; 102.248µs] or [-0.026%; +0.026%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[9780138.045op/s; 9785149.012op/s] or [-0.026%; +0.026%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[23.250µs; 23.401µs] or [-0.323%; +0.323%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[42755172.403op/s; 43033637.285op/s] or [-0.325%; +0.325%]	None	None	None

Group 11

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.428µs	2.897µs ± 1.388µs	2.742µs ± 0.022µs	2.756µs	2.944µs	12.556µs	15.708µs	472.94%	7.944	63.235	47.78%	0.098µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[2.705µs; 3.090µs] or [-6.638%; +6.638%]	None	None	None

Group 12

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	94befa0	1725654259	ekump/APMSP-1347-fix-flaky-send-data-retry-test-on-mac

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	307.706µs	310.075µs ± 0.833µs	310.173µs ± 0.484µs	310.634µs	311.351µs	311.679µs	312.035µs	0.60%	-0.415	0.031	0.27%	0.059µs	1	200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	3204764.131op/s	3225044.432op/s ± 8678.414op/s	3224009.558op/s ± 5023.357op/s	3230764.537op/s	3241545.414op/s	3246969.094op/s	3249860.467op/s	0.80%	0.430	0.044	0.27%	613.657op/s	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	28.182µs	28.416µs ± 0.095µs	28.410µs ± 0.055µs	28.467µs	28.544µs	28.673µs	28.864µs	1.60%	0.845	3.216	0.33%	0.007µs	1	200
normalization/normalize_name/normalize_name/bad-name	throughput	34644836.608op/s	35192242.720op/s ± 117433.429op/s	35198272.215op/s ± 68604.746op/s	35264951.056op/s	35379357.056op/s	35439617.201op/s	35484118.790op/s	0.81%	-0.800	3.047	0.33%	8303.797op/s	1	200
normalization/normalize_name/normalize_name/good	execution_time	16.501µs	16.529µs ± 0.027µs	16.531µs ± 0.021µs	16.543µs	16.564µs	16.596µs	16.750µs	1.32%	3.017	20.024	0.17%	0.002µs	1	200
normalization/normalize_name/normalize_name/good	throughput	59701830.530op/s	60501541.701op/s ± 100083.098op/s	60491729.994op/s ± 75297.483op/s	60584892.297op/s	60596048.328op/s	60601280.577op/s	60601627.885op/s	0.18%	-2.954	19.351	0.17%	7076.944op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[309.960µs; 310.191µs] or [-0.037%; +0.037%]	None	None	None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	[3223841.688op/s; 3226247.177op/s] or [-0.037%; +0.037%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[28.402µs; 28.429µs] or [-0.046%; +0.046%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	throughput	[35175967.576op/s; 35208517.864op/s] or [-0.046%; +0.046%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[16.525µs; 16.532µs] or [-0.023%; +0.023%]	None	None	None
normalization/normalize_name/normalize_name/good	throughput	[60487671.146op/s; 60515412.255op/s] or [-0.023%; +0.023%]	None	None	None

Baseline

Omitted due to size.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.00000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.62%. Comparing base (c48d94b) to head (94befa0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #610      +/-   ##
==========================================
- Coverage   72.63%   72.62%   -0.02%     
==========================================
  Files         246      246              
  Lines       34964    34971       +7     
==========================================
+ Hits        25396    25397       +1     
- Misses       9568     9574       +6     

Components	Coverage Δ
crashtracker	20.44% <ø> (+0.05%)	⬆️
datadog-alloc	98.73% <ø> (ø)
data-pipeline	90.12% <ø> (ø)
data-pipeline-ffi	0.00% <ø> (ø)
ddcommon	82.82% <ø> (ø)
ddcommon-ffi	68.11% <ø> (ø)
ddtelemetry	59.02% <ø> (ø)
ipc	83.61% <ø> (ø)
profiling	84.26% <ø> (ø)
profiling-ffi	77.42% <ø> (ø)
serverless	0.00% <ø> (ø)
sidecar	40.13% <ø> (ø)
sidecar-ffi	0.00% <ø> (ø)
spawn-worker	50.36% <ø> (ø)
tinybytes	91.66% <ø> (ø)
trace-mini-agent	70.88% <ø> (ø)
trace-normalization	98.25% <ø> (ø)
trace-obfuscation	95.73% <ø> (ø)
trace-protobuf	77.67% <ø> (ø)
trace-utils	92.83% <50.00%> (-0.18%)	⬇️

[bwoebi]

Thanks, looks good.