Enable all benchmarks #522

bantonsson · 2024-07-09T08:26:38Z

What does this PR do?

This PR makes all Criterion benchmarks run on the benchmarking platform.

Motivation

It should be easy to add new benchmarks and we should run them to catch regressions early.

How to test the change?

It's run on this PR

APMSP-1228

codecov-commenter · 2024-07-09T08:33:24Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.14%. Comparing base (d57d6d4) to head (64a0e70).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #522      +/-   ##
==========================================
- Coverage   70.30%   70.14%   -0.17%     
==========================================
  Files         206      206              
  Lines       27806    27806              
==========================================
- Hits        19549    19504      -45     
- Misses       8257     8302      +45

Components	Coverage Δ
crashtracker	`16.86% <ø> (ø)`
datadog-alloc	`98.73% <ø> (ø)`
data-pipeline	`51.15% <ø> (ø)`
data-pipeline-ffi	`0.00% <ø> (ø)`
ddcommon	`86.68% <ø> (ø)`
ddcommon-ffi	`75.31% <ø> (ø)`
ddtelemetry	`59.02% <ø> (ø)`
ipc	`84.13% <ø> (ø)`
profiling	`77.94% <ø> (-0.75%)`	⬇️
profiling-ffi	`58.26% <ø> (ø)`
serverless	`0.00% <ø> (ø)`
sidecar	`35.69% <ø> (ø)`
sidecar-ffi	`0.00% <ø> (ø)`
spawn-worker	`54.98% <ø> (ø)`
trace-mini-agent	`70.93% <ø> (ø)`
trace-normalization	`98.24% <ø> (ø)`
trace-obfuscation	`95.75% <ø> (ø)`
trace-protobuf	`77.16% <ø> (ø)`
trace-utils	`90.68% <ø> (ø)`

pr-commenter · 2024-07-09T13:29:18Z

Benchmarks

Comparison

Benchmark execution time: 2024-07-09 15:22:14

Comparing candidate commit 64a0e70 in PR branch ban/enable-all-benchmarks with baseline commit d57d6d4 in branch main.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 0 metrics, 0 unstable metrics.

scenario:sql/obfuscate_sql_string

🟥 execution_time [+5.950µs; +5.983µs] or [+9.387%; +9.440%]

Candidate

Candidate benchmark details

Group 1

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	310.060ns	316.039ns ± 3.688ns	316.509ns ± 1.194ns	317.336ns	318.820ns	324.905ns	351.108ns	10.93%	4.345	39.780	1.16%	0.261ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[315.528ns; 316.550ns] or [-0.162%; +0.162%]	None	None	None

Group 2

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	575.383ns	584.197ns ± 6.158ns	583.124ns ± 3.231ns	586.700ns	594.658ns	608.134ns	613.144ns	5.15%	1.623	4.098	1.05%	0.435ns	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	473.763ns	482.646ns ± 1.888ns	482.837ns ± 0.779ns	483.595ns	484.570ns	486.442ns	491.993ns	1.90%	-0.900	8.502	0.39%	0.133ns	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	191.661ns	200.796ns ± 3.836ns	200.999ns ± 3.240ns	203.123ns	207.096ns	210.681ns	213.076ns	6.01%	0.264	0.101	1.91%	0.271ns	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	36.788ns	43.254ns ± 7.181ns	39.913ns ± 2.433ns	46.100ns	58.672ns	58.848ns	59.187ns	48.29%	1.328	0.395	16.56%	0.508ns	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	63.060ns	70.059ns ± 5.695ns	68.139ns ± 2.994ns	69.313ns	80.558ns	81.735ns	81.973ns	20.30%	0.975	-0.602	8.11%	0.403ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[583.343ns; 585.050ns] or [-0.146%; +0.146%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[482.385ns; 482.908ns] or [-0.054%; +0.054%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[200.265ns; 201.328ns] or [-0.265%; +0.265%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[42.259ns; 44.250ns] or [-2.301%; +2.301%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[69.270ns; 70.849ns] or [-1.127%; +1.127%]	None	None	None

Group 3

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	277.914ns	278.823ns ± 0.453ns	278.900ns ± 0.290ns	279.105ns	279.455ns	279.684ns	280.980ns	0.75%	0.280	1.540	0.16%	0.032ns	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	33.652ns	35.209ns ± 0.884ns	35.220ns ± 0.211ns	35.368ns	35.550ns	35.736ns	46.910ns	33.19%	11.606	151.848	2.51%	0.063ns	1	200
normalization/normalize_name/normalize_name/good	execution_time	22.191ns	22.588ns ± 0.196ns	22.548ns ± 0.105ns	22.649ns	22.923ns	23.216ns	23.549ns	4.44%	1.625	3.845	0.87%	0.014ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[278.761ns; 278.886ns] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[35.086ns; 35.331ns] or [-0.348%; +0.348%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[22.561ns; 22.616ns] or [-0.120%; +0.120%]	None	None	None

Group 4

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.301µs	2.312µs ± 0.004µs	2.312µs ± 0.003µs	2.315µs	2.320µs	2.323µs	2.325µs	0.56%	0.354	0.235	0.19%	0.000µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.312µs; 2.313µs] or [-0.026%; +0.026%]	None	None	None

Group 5

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	0.017ms	5.085ms ± 36.350ms	0.020ms ± 0.003ms	0.022ms	0.690ms	155.983ms	416.858ms	2048594.04%	9.116	90.555	713.02%	2.570ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[0.048ms; 10.123ms] or [-99.066%; +99.066%]	None	None	None

Group 6

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	69.137µs	69.347µs ± 0.069µs	69.342µs ± 0.019µs	69.359µs	69.399µs	69.513µs	70.014µs	0.97%	5.862	51.314	0.10%	0.005µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[69.338µs; 69.357µs] or [-0.014%; +0.014%]	None	None	None

Group 7

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.411µs	2.766µs ± 3.001µs	1.452µs ± 0.007µs	4.084µs	7.416µs	15.901µs	23.929µs	1547.89%	3.778	18.813	108.22%	0.212µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[2.350µs; 3.182µs] or [-15.036%; +15.036%]	None	None	None

Group 8

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	134.755µs	135.361µs ± 1.062µs	135.222µs ± 0.109µs	135.335µs	135.858µs	137.050µs	149.377µs	10.47%	11.755	150.537	0.78%	0.075µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[135.214µs; 135.509µs] or [-0.109%; +0.109%]	None	None	None

Group 9

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	3.452ns	3.595ns ± 0.018ns	3.600ns ± 0.003ns	3.603ns	3.604ns	3.606ns	3.662ns	1.71%	-4.144	26.170	0.50%	0.001ns	1	200
credit_card/is_card_number/	throughput	273109409.692op/s	278174778.842op/s ± 1432788.422op/s	277777664.590op/s ± 233089.027op/s	278210926.704op/s	279967381.794op/s	285114597.277op/s	289660142.714op/s	4.28%	4.311	27.580	0.51%	101313.441op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	84.652ns	87.063ns ± 0.905ns	87.069ns ± 0.528ns	87.562ns	88.608ns	89.416ns	89.702ns	3.02%	0.129	0.250	1.04%	0.064ns	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	11148077.956op/s	11487172.089op/s ± 119279.133op/s	11485125.261op/s ± 69611.416op/s	11561434.057op/s	11671542.394op/s	11756888.035op/s	11813084.754op/s	2.86%	-0.060	0.231	1.04%	8434.308op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	80.300ns	82.305ns ± 0.925ns	82.282ns ± 0.569ns	82.846ns	83.839ns	84.454ns	86.344ns	4.94%	0.618	1.616	1.12%	0.065ns	1	200
credit_card/is_card_number/ 378282246310005	throughput	11581527.464op/s	12151435.122op/s ± 135682.498op/s	12153356.702op/s ± 83535.152op/s	12236443.986op/s	12362690.270op/s	12424666.879op/s	12453256.666op/s	2.47%	-0.512	1.310	1.11%	9594.201op/s	1	200
credit_card/is_card_number/37828224631	execution_time	3.278ns	3.594ns ± 0.027ns	3.600ns ± 0.003ns	3.602ns	3.604ns	3.607ns	3.649ns	1.38%	-8.841	98.195	0.74%	0.002ns	1	200
credit_card/is_card_number/37828224631	throughput	274024788.864op/s	278249634.687op/s ± 2210868.131op/s	277804295.019op/s ± 239639.858op/s	278190622.909op/s	280135023.416op/s	283603180.442op/s	305026585.129op/s	9.80%	9.343	106.945	0.79%	156331.985op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	78.076ns	80.353ns ± 0.972ns	80.308ns ± 0.603ns	80.925ns	81.765ns	82.565ns	86.039ns	7.14%	1.585	7.952	1.21%	0.069ns	1	200
credit_card/is_card_number/378282246310005	throughput	11622600.918op/s	12446922.333op/s ± 147963.673op/s	12452104.938op/s ± 93991.877op/s	12545383.878op/s	12656829.930op/s	12767025.626op/s	12808034.459op/s	2.86%	-1.331	6.497	1.19%	10462.612op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	65.613ns	67.819ns ± 0.502ns	67.853ns ± 0.308ns	68.131ns	68.474ns	68.929ns	70.375ns	3.72%	0.093	4.349	0.74%	0.035ns	1	200
credit_card/is_card_number/37828224631000521389798	throughput	14209636.865op/s	14745951.779op/s ± 109032.949op/s	14737712.820op/s ± 66915.373op/s	14810009.085op/s	14911879.683op/s	15017235.432op/s	15240818.700op/s	3.41%	0.046	4.190	0.74%	7709.794op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	21.201ns	22.500ns ± 0.410ns	22.527ns ± 0.276ns	22.801ns	23.103ns	23.419ns	23.448ns	4.09%	-0.509	0.757	1.82%	0.029ns	1	200
credit_card/is_card_number/x371413321323331	throughput	42647711.606op/s	44459743.271op/s ± 818358.589op/s	44391675.744op/s ± 544967.256op/s	44943962.310op/s	45684023.615op/s	47166116.712op/s	47167436.077op/s	6.25%	0.650	1.063	1.84%	57866.691op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	3.427ns	3.582ns ± 0.030ns	3.591ns ± 0.013ns	3.603ns	3.605ns	3.614ns	3.684ns	2.59%	-2.122	7.638	0.85%	0.002ns	1	200
credit_card/is_card_number_no_luhn/	throughput	271464157.476op/s	279156147.350op/s ± 2412258.858op/s	278500129.825op/s ± 974534.070op/s	279881939.900op/s	283356413.595op/s	289678416.897op/s	291766803.547op/s	4.76%	2.254	8.249	0.86%	170572.460op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	67.777ns	69.841ns ± 0.850ns	69.873ns ± 0.431ns	70.255ns	71.051ns	72.758ns	73.235ns	4.81%	0.488	2.078	1.21%	0.060ns	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	13654608.553op/s	14320367.197op/s ± 173364.731op/s	14311640.644op/s ± 88233.426op/s	14411844.685op/s	14618793.934op/s	14741562.703op/s	14754178.259op/s	3.09%	-0.352	1.801	1.21%	12258.738op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	61.705ns	64.946ns ± 0.729ns	64.983ns ± 0.485ns	65.465ns	66.007ns	66.274ns	67.097ns	3.25%	-0.672	1.983	1.12%	0.052ns	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	14903700.050op/s	15399310.066op/s ± 174218.921op/s	15388715.581op/s ± 114318.318op/s	15504267.756op/s	15687865.266op/s	15952806.533op/s	16206040.982op/s	5.31%	0.794	2.354	1.13%	12319.138op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	3.502ns	3.586ns ± 0.021ns	3.593ns ± 0.010ns	3.602ns	3.604ns	3.608ns	3.619ns	0.72%	-1.743	3.338	0.58%	0.001ns	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	276302745.829op/s	278840773.294op/s ± 1642878.314op/s	278299503.937op/s ± 766037.443op/s	279446311.530op/s	281990132.683op/s	285051317.191op/s	285551680.617op/s	2.61%	1.784	3.529	0.59%	116169.040op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	61.034ns	62.648ns ± 0.632ns	62.696ns ± 0.429ns	63.102ns	63.686ns	64.052ns	64.114ns	2.26%	-0.215	-0.150	1.01%	0.045ns	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	15597123.139op/s	15963725.460op/s ± 161439.583op/s	15950034.515op/s ± 108461.921op/s	16071977.231op/s	16273587.363op/s	16339547.551op/s	16384423.128op/s	2.72%	0.269	-0.139	1.01%	11415.502op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	65.783ns	69.025ns ± 0.588ns	69.139ns ± 0.276ns	69.379ns	69.626ns	70.106ns	71.966ns	4.09%	-0.491	7.373	0.85%	0.042ns	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	13895470.325op/s	14488559.772op/s ± 123993.812op/s	14463572.752op/s ± 57990.549op/s	14549376.461op/s	14679048.204op/s	14792137.862op/s	15201554.003op/s	5.10%	0.723	7.621	0.85%	8767.687op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	21.606ns	23.204ns ± 0.547ns	23.198ns ± 0.382ns	23.596ns	24.120ns	24.272ns	24.930ns	7.47%	0.008	-0.003	2.35%	0.039ns	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	40112763.186op/s	43120000.693op/s ± 1018218.524op/s	43108082.408op/s ± 704411.632op/s	43771079.538op/s	44805971.545op/s	45497186.540op/s	46283048.750op/s	7.37%	0.133	0.006	2.36%	71998.922op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[3.592ns; 3.597ns] or [-0.070%; +0.070%]	None	None	None
credit_card/is_card_number/	throughput	[277976208.146op/s; 278373349.537op/s] or [-0.071%; +0.071%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[86.938ns; 87.188ns] or [-0.144%; +0.144%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[11470641.148op/s; 11503703.030op/s] or [-0.144%; +0.144%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[82.177ns; 82.433ns] or [-0.156%; +0.156%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[12132630.833op/s; 12170239.412op/s] or [-0.155%; +0.155%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[3.590ns; 3.598ns] or [-0.103%; +0.103%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[277943229.627op/s; 278556039.747op/s] or [-0.110%; +0.110%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[80.218ns; 80.487ns] or [-0.168%; +0.168%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[12426415.991op/s; 12467428.675op/s] or [-0.165%; +0.165%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[67.749ns; 67.888ns] or [-0.102%; +0.102%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[14730840.861op/s; 14761062.697op/s] or [-0.102%; +0.102%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[22.443ns; 22.557ns] or [-0.252%; +0.252%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[44346326.641op/s; 44573159.900op/s] or [-0.255%; +0.255%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[3.578ns; 3.587ns] or [-0.118%; +0.118%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[278821831.473op/s; 279490463.228op/s] or [-0.120%; +0.120%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[69.723ns; 69.959ns] or [-0.169%; +0.169%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[14296340.513op/s; 14344393.881op/s] or [-0.168%; +0.168%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[64.845ns; 65.047ns] or [-0.156%; +0.156%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[15375164.999op/s; 15423455.133op/s] or [-0.157%; +0.157%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[3.584ns; 3.589ns] or [-0.081%; +0.081%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[278613086.160op/s; 279068460.427op/s] or [-0.082%; +0.082%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[62.561ns; 62.736ns] or [-0.140%; +0.140%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[15941351.486op/s; 15986099.434op/s] or [-0.140%; +0.140%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[68.944ns; 69.106ns] or [-0.118%; +0.118%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[14471375.422op/s; 14505744.122op/s] or [-0.119%; +0.119%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[23.128ns; 23.280ns] or [-0.327%; +0.327%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[42978885.399op/s; 43261115.988op/s] or [-0.327%; +0.327%]	None	None	None

Group 10

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
candidate	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`64a0e70`	1720537825	ban/enable-all-benchmarks

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	37.923µs	38.057µs ± 0.026µs	38.057µs ± 0.014µs	38.071µs	38.094µs	38.111µs	38.193µs	0.36%	-0.186	5.926	0.07%	0.002µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[38.053µs; 38.061µs] or [-0.010%; +0.010%]	None	None	None

Warnings

Click to expand

Scenario: normalization/normalize_trace/test_trace, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..5.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: normalization/normalize_service/normalize_service/[empty string], Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: normalization/normalize_service/normalize_service/test_ASCII, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo..., Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: normalization/normalize_name/normalize_name/good, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 2, 4.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: tags/replace_trace_tags, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 3..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: two way interface, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: write only interface, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: benching string interning on wordpress profile, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 9..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 9..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/  378282246310005, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/  378282246310005, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/37828224631000521389798, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/37828224631000521389798, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/x371413321323331, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number/x371413321323331, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/  3782-8224-6310-005 , Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/  3782-8224-6310-005 , Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/  378282246310005, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/  378282246310005, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/378282246310005, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/378282246310005, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 1.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/37828224631000521389798, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 2..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/37828224631000521389798, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 2..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/x371413321323331, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 4..5.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: credit_card/is_card_number_no_luhn/x371413321323331, Metric: throughput

Measurements are autocorrelated.

Autocorrelation is present for lags 4..7.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Scenario: redis/obfuscate_redis_string, Metric: execution_time

Measurements are autocorrelated.

Autocorrelation is present for lags 1..10.

The measurements are not independent, thus confidence intervals
may be less precise.
---------------------------------------------------------------------------

Baseline

Baseline benchmark details

Group 1

config	cpu_model	ci_job_date	ci_job_id	ci_pipeline_id	git_commit_sha	git_commit_date	git_branch
baseline	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	1720538514	566839048	38725769	`d57d6d4`	1720532757	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	63.238µs	63.381µs ± 0.071µs	63.367µs ± 0.043µs	63.428µs	63.491µs	63.611µs	63.619µs	0.40%	0.878	1.097	0.11%	0.007µs	1	100

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[63.367µs; 63.394µs] or [-0.022%; +0.022%]	None	None	None

Warnings

Click to expand

Scenario: sql/obfuscate_sql_string, Metric: execution_time

Sample size is 100, which is lower than 105.

The minimal sample size in case of normal distribution to achieve significance
level of 0.05 for difference of means with effect size Cohen's d = 0.5 must be at
least 105.

The conclusions from confidence intervals may be invalid.
---------------------------------------------------------------------------

github-actions bot added profiling Relates to the profiling* modules. telemetry mini-agent sidecar common data-pipeline labels Jul 9, 2024

bantonsson force-pushed the ban/enable-all-benchmarks branch 3 times, most recently from 436ee9a to c831e36 Compare July 9, 2024 13:19

bantonsson force-pushed the ban/enable-all-benchmarks branch 2 times, most recently from ff88699 to b630e91 Compare July 9, 2024 14:36

Enable all benchmarks

64a0e70

bantonsson force-pushed the ban/enable-all-benchmarks branch from d18d238 to 64a0e70 Compare July 9, 2024 15:10

bantonsson marked this pull request as ready for review July 9, 2024 15:14

bantonsson requested review from a team as code owners July 9, 2024 15:14

bantonsson requested a review from yahya-mouman July 9, 2024 15:14

bantonsson changed the title ~~[WIP] Enable all benchmarks~~ Enable all benchmarks Jul 9, 2024

paullegranddc approved these changes Jul 9, 2024

View reviewed changes

ekump approved these changes Jul 9, 2024

View reviewed changes

pawelchcki approved these changes Jul 9, 2024

View reviewed changes

bantonsson merged commit ea37c78 into main Jul 9, 2024
32 checks passed

bantonsson deleted the ban/enable-all-benchmarks branch July 9, 2024 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable all benchmarks #522

Enable all benchmarks #522

bantonsson commented Jul 9, 2024 •

edited

Loading

codecov-commenter commented Jul 9, 2024 •

edited

Loading

pr-commenter bot commented Jul 9, 2024 •

edited

Loading

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Warnings

Group 1

Warnings

Enable all benchmarks #522

Enable all benchmarks #522

Conversation

bantonsson commented Jul 9, 2024 • edited Loading

What does this PR do?

Motivation

How to test the change?

codecov-commenter commented Jul 9, 2024 • edited Loading

Codecov Report

pr-commenter bot commented Jul 9, 2024 • edited Loading

Benchmarks

Comparison

scenario:sql/obfuscate_sql_string

Candidate

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Warnings

Baseline

Group 1

Warnings

bantonsson commented Jul 9, 2024 •

edited

Loading

codecov-commenter commented Jul 9, 2024 •

edited

Loading

pr-commenter bot commented Jul 9, 2024 •

edited

Loading