Make `compare_throughput` stage more robust #4739

andrewlock · 2023-10-20T09:51:18Z

Summary of changes

Don't try to upload results if a throughput test run fails
Always use the same throughput results
Don't fail the compare_throughput stage if some results are missing

Reason for change

We had several issues:

We have separate throughput and throughput_asm stages currently (side point, I think we can merge them). We don't always want to run them both.
The compare_throughput stage would previously fail if we didn't have one of the expected artifacts (e.g. because the ASM results were missing because the stage was skipped)
If the throughput tests failed, and were retried, we produced different artifact names, which weren't supported properly.

Implementation details

Create consistent artifact names - if the throughput test fails, it's unlikely the artifacts will be useful on their own (unlike for other stages)
Use continueOnError when downloading the throughput results

Test coverage

This is the test

Other details

Pre-requisite for @NachoEchevarria's work on enabling appsec throughput tests again

datadog-ddstaging · 2023-10-20T10:13:25Z

Datadog Report

Branch report: andrew/ci/fix_throughput_comparison
Commit report: e8f8deb

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 305533 Passed, 1172 Skipped, 36m 15.46s Wall Time

andrewlock · 2023-10-20T10:14:20Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4739) - mean (71ms)  : 63, 78
     .   : milestone, 71,
    master - mean (69ms)  : 63, 76
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (4739) - mean (1,023ms)  : 1002, 1044
     .   : milestone, 1023,
    master - mean (1,013ms)  : 994, 1032
     .   : milestone, 1013,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4739) - mean (107ms)  : 102, 111
     .   : milestone, 107,
    master - mean (107ms)  : 96, 118
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (4739) - mean (718ms)  : 698, 738
     .   : milestone, 718,
    master - mean (716ms)  : 695, 737
     .   : milestone, 716,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4739) - mean (91ms)  : 87, 94
     .   : milestone, 91,
    master - mean (90ms)  : 84, 96
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4739) - mean (684ms)  : 668, 701
     .   : milestone, 684,
    master - mean (682ms)  : 653, 711
     .   : milestone, 682,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4739) - mean (188ms)  : 184, 193
     .   : milestone, 188,
    master - mean (188ms)  : 185, 192
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4739) - mean (1,126ms)  : 1097, 1155
     .   : milestone, 1126,
    master - mean (1,127ms)  : 1109, 1146
     .   : milestone, 1127,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4739) - mean (271ms)  : 267, 275
     .   : milestone, 271,
    master - mean (272ms)  : 269, 274
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (4739) - mean (1,087ms)  : 1065, 1108
     .   : milestone, 1087,
    master - mean (1,090ms)  : 1069, 1110
     .   : milestone, 1090,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4739) - mean (260ms)  : 257, 262
     .   : milestone, 260,
    master - mean (261ms)  : 255, 267
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4739) - mean (1,055ms)  : 1038, 1072
     .   : milestone, 1055,
    master - mean (1,055ms)  : 1022, 1087
     .   : milestone, 1055,

andrewlock · 2023-10-20T11:09:02Z

Benchmarks Report 🐌

Benchmarks for #4739 compared to master:

1 benchmarks are faster, with geometric mean 1.153
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8μs	42.9ns	235ns	0.0205	0.00818	0	7.29 KB
master	`StartStopWithChild`	netcoreapp3.1	9.89μs	53.2ns	310ns	0.0342	0.0147	0	7.38 KB
master	`StartStopWithChild`	net472	15.5μs	46.5ns	180ns	1.26	0.277	0.0846	7.66 KB
#4739	`StartStopWithChild`	net6.0	8.19μs	42.5ns	199ns	0.0159	0.00398	0	7.29 KB
#4739	`StartStopWithChild`	netcoreapp3.1	9.85μs	53.6ns	321ns	0.0244	0.00978	0	7.38 KB
#4739	`StartStopWithChild`	net472	15.8μs	64.8ns	251ns	1.28	0.311	0.0933	7.66 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	468μs	248ns	960ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	638μs	506ns	1.96μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	801μs	270ns	1.01μs	0.401	3.3 KB
#4739	`WriteAndFlushEnrichedTraces`	net6.0	464μs	309ns	1.2μs	0	2.7 KB
#4739	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	640μs	173ns	623ns	0	2.7 KB
#4739	`WriteAndFlushEnrichedTraces`	net472	793μs	188ns	652ns	0.396	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	44.1μs	64.9ns	251ns	0.0218	0	2.03 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	45.3μs	53.7ns	208ns	0.0226	0	2.01 KB
master	`AllCycleSimpleBody`	net472	46.5μs	114ns	443ns	0.322	0	2.08 KB
master	`AllCycleMoreComplexBody`	net6.0	229μs	65.5ns	227ns	0.115	0	8.63 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	234μs	102ns	383ns	0	0	8.52 KB
master	`AllCycleMoreComplexBody`	net472	239μs	137ns	531ns	1.31	0	8.7 KB
master	`ObjectExtractorSimpleBody`	net6.0	126ns	0.107ns	0.399ns	0.00394	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	178ns	0.187ns	0.699ns	0.00368	0	272 B
master	`ObjectExtractorSimpleBody`	net472	144ns	0.151ns	0.585ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.02μs	0.662ns	2.56ns	0.0544	0	3.88 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.05μs	2.04ns	7.91ns	0.0526	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.14μs	3.19ns	12.4ns	0.618	0.00622	3.89 KB
#4739	`AllCycleSimpleBody`	net6.0	43.6μs	65.7ns	246ns	0.0219	0	2.03 KB
#4739	`AllCycleSimpleBody`	netcoreapp3.1	44.8μs	91.7ns	355ns	0.0225	0	2.01 KB
#4739	`AllCycleSimpleBody`	net472	46.6μs	22.4ns	86.7ns	0.32	0	2.08 KB
#4739	`AllCycleMoreComplexBody`	net6.0	229μs	175ns	655ns	0.114	0	8.63 KB
#4739	`AllCycleMoreComplexBody`	netcoreapp3.1	234μs	96.5ns	361ns	0	0	8.52 KB
#4739	`AllCycleMoreComplexBody`	net472	238μs	96.5ns	361ns	1.3	0	8.7 KB
#4739	`ObjectExtractorSimpleBody`	net6.0	126ns	0.0552ns	0.214ns	0.00395	0	280 B
#4739	`ObjectExtractorSimpleBody`	netcoreapp3.1	181ns	0.268ns	1.04ns	0.00376	0	272 B
#4739	`ObjectExtractorSimpleBody`	net472	144ns	0.122ns	0.455ns	0.0446	0	281 B
#4739	`ObjectExtractorMoreComplexBody`	net6.0	3.06μs	0.988ns	3.69ns	0.0551	0	3.88 KB
#4739	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.04μs	2.3ns	8.92ns	0.0527	0	3.78 KB
#4739	`ObjectExtractorMoreComplexBody`	net472	4.16μs	2.54ns	9.83ns	0.617	0.00621	3.89 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	12.7μs	4.28ns	16ns	0.128	0	9.42 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	19.7μs	8.5ns	31.8ns	0.118	0	9.42 KB
master	`RunWaf(args=NestedMap (10))`	net472	28.3μs	8.66ns	32.4ns	1.5	0	9.48 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	77.2μs	16.8ns	62.8ns	0.192	0	15.77 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	87.1μs	111ns	432ns	0.173	0	15.72 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	98.3μs	42.8ns	166ns	2.5	0.0482	16.04 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	24.1μs	10.8ns	40.2ns	0.267	0	19.66 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	38μs	13.2ns	51.3ns	0.265	0	20.42 KB
master	`RunWaf(args=NestedMap (100))`	net472	55.1μs	13.7ns	51.4ns	3.25	0.0276	20.63 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	90.9μs	51.9ns	201ns	0.364	0	26.01 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	109μs	220ns	853ns	0.324	0	26.72 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	133μs	48.9ns	183ns	4.31	0.0654	27.19 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	23.6μs	9.01ns	34.9ns	0.273	0	19.4 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	37.7μs	21.9ns	84.7ns	0.264	0	19.84 KB
master	`RunWaf(args=NestedMap (20))`	net472	56.2μs	30.1ns	113ns	3.16	0.056	20.04 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	90.5μs	35.5ns	133ns	0.363	0	25.74 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	108μs	250ns	936ns	0.323	0	26.14 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	134μs	113ns	424ns	4.18	0.0664	26.6 KB
#4739	`RunWaf(args=NestedMap (10))`	net6.0	12.9μs	3.47ns	13.4ns	0.131	0	9.42 KB
#4739	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	19.6μs	6.3ns	23.6ns	0.128	0	9.42 KB
#4739	`RunWaf(args=NestedMap (10))`	net472	28.3μs	10.9ns	39.1ns	1.5	0	9.48 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	76μs	46.2ns	179ns	0.191	0	15.77 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	85.6μs	18ns	65ns	0.216	0	15.72 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	96.5μs	40.4ns	156ns	2.51	0	16.04 KB
#4739	`RunWaf(args=NestedMap (100))`	net6.0	24μs	9.89ns	37ns	0.275	0	19.66 KB
#4739	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	38.3μs	15.1ns	58.7ns	0.267	0	20.42 KB
#4739	`RunWaf(args=NestedMap (100))`	net472	57.7μs	13.3ns	49.8ns	3.26	0.0288	20.63 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	91.7μs	57.8ns	224ns	0.324	0	26.01 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	111μs	319ns	1.15μs	0.328	0	26.72 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	133μs	72.5ns	281ns	4.29	0.0659	27.19 KB
#4739	`RunWaf(args=NestedMap (20))`	net6.0	23.8μs	12.3ns	47.8ns	0.263	0	19.4 KB
#4739	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	37.9μs	11.7ns	43.9ns	0.264	0	19.84 KB
#4739	`RunWaf(args=NestedMap (20))`	net472	53.9μs	20.7ns	77.5ns	3.17	0.0537	20.04 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	93.2μs	15.3ns	52.8ns	0.329	0	25.74 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	108μs	157ns	609ns	0.321	0	26.14 KB
#4739	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	130μs	71.7ns	278ns	4.19	0.0644	26.6 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	168μs	183ns	711ns	0.168	18.04 KB
master	`SendRequest`	netcoreapp3.1	188μs	282ns	1.09μs	0.186	20.2 KB
master	`SendRequest`	net472	0.00126ns	0.000539ns	0.00209ns	0	0 b
#4739	`SendRequest`	net6.0	167μs	142ns	550ns	0.167	18.04 KB
#4739	`SendRequest`	netcoreapp3.1	189μs	236ns	883ns	0.188	20.2 KB
#4739	`SendRequest`	net472	0.000277ns	0.000142ns	0.000552ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #4739

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.56 KB	42.09 KB	529 B	1.27%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	553μs	1.15μs	4.45μs	0.553	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	651μs	813ns	3.04μs	0.331	0	0	41.56 KB
master	`WriteAndFlushEnrichedTraces`	net472	856μs	3.56μs	13.8μs	8.13	2.57	0.428	53.36 KB
#4739	`WriteAndFlushEnrichedTraces`	net6.0	554μs	1.37μs	5.29μs	0.553	0	0	41.85 KB
#4739	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	2.63μs	10.2μs	0.34	0	0	42.09 KB
#4739	`WriteAndFlushEnrichedTraces`	net472	821μs	3.7μs	14.3μs	8.08	2.55	0.425	53.24 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.03μs	0.979ns	3.79ns	0.0108	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.28μs	0.562ns	2.18ns	0.0101	768 B
master	`ExecuteNonQuery`	net472	1.63μs	0.373ns	1.4ns	0.116	730 B
#4739	`ExecuteNonQuery`	net6.0	1.03μs	0.584ns	2.26ns	0.0106	768 B
#4739	`ExecuteNonQuery`	netcoreapp3.1	1.28μs	0.578ns	2.24ns	0.0103	768 B
#4739	`ExecuteNonQuery`	net472	1.63μs	0.767ns	2.87ns	0.115	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.17μs	0.644ns	2.41ns	0.0128	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.41μs	0.567ns	2.2ns	0.0121	936 B
master	`CallElasticsearch`	net472	2.35μs	0.777ns	2.8ns	0.151	955 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.785ns	2.94ns	0.0128	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.48μs	2.62ns	10.1ns	0.0133	984 B
master	`CallElasticsearchAsync`	net472	2.55μs	1.09ns	4.08ns	0.16	1.01 KB
#4739	`CallElasticsearch`	net6.0	1.1μs	0.356ns	1.28ns	0.0132	936 B
#4739	`CallElasticsearch`	netcoreapp3.1	1.48μs	5.41ns	21ns	0.0128	936 B
#4739	`CallElasticsearch`	net472	2.33μs	0.587ns	2.12ns	0.151	955 B
#4739	`CallElasticsearchAsync`	net6.0	1.26μs	0.386ns	1.44ns	0.0125	912 B
#4739	`CallElasticsearchAsync`	netcoreapp3.1	1.52μs	0.912ns	3.29ns	0.0129	984 B
#4739	`CallElasticsearchAsync`	net472	2.49μs	1.72ns	6.65ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.21μs	1.11ns	4.3ns	0.0127	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.49μs	0.664ns	2.49ns	0.012	912 B
master	`ExecuteAsync`	net472	1.62μs	0.526ns	2.04ns	0.138	875 B
#4739	`ExecuteAsync`	net6.0	1.2μs	1.3ns	4.88ns	0.0126	912 B
#4739	`ExecuteAsync`	netcoreapp3.1	1.6μs	0.412ns	1.59ns	0.012	912 B
#4739	`ExecuteAsync`	net472	1.74μs	0.41ns	1.48ns	0.139	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	3.73μs	1.25ns	4.84ns	0.0261	1.9 KB
master	`SendAsync`	netcoreapp3.1	4.42μs	1.31ns	4.91ns	0.0332	2.43 KB
master	`SendAsync`	net472	7.17μs	2.21ns	8.57ns	0.474	2.99 KB
#4739	`SendAsync`	net6.0	3.71μs	0.9ns	3.37ns	0.026	1.9 KB
#4739	`SendAsync`	netcoreapp3.1	4.41μs	2.33ns	9.02ns	0.0331	2.43 KB
#4739	`SendAsync`	net472	7.1μs	1.78ns	6.89ns	0.474	2.99 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.35μs	0.449ns	1.68ns	0.0217	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	1.95μs	1.88ns	7.03ns	0.0214	1.57 KB
master	`EnrichedLog`	net472	2.29μs	1.39ns	5.21ns	0.236	1.49 KB
#4739	`EnrichedLog`	net6.0	1.32μs	1.24ns	4.64ns	0.0218	1.57 KB
#4739	`EnrichedLog`	netcoreapp3.1	2.07μs	0.849ns	3.29ns	0.0216	1.57 KB
#4739	`EnrichedLog`	net472	2.34μs	2.02ns	7.3ns	0.236	1.49 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	66.5ns	257ns	0.0566	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	199ns	769ns	0	0	4.21 KB
master	`EnrichedLog`	net472	147μs	86.7ns	336ns	0.658	0.219	4.38 KB
#4739	`EnrichedLog`	net6.0	113μs	98.7ns	369ns	0	0	4.21 KB
#4739	`EnrichedLog`	netcoreapp3.1	120μs	110ns	413ns	0.0595	0	4.21 KB
#4739	`EnrichedLog`	net472	146μs	79.7ns	309ns	0.66	0.22	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.8μs	0.833ns	3.12ns	0.0293	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	3.87μs	0.989ns	3.7ns	0.029	2.13 KB
master	`EnrichedLog`	net472	4.43μs	1.13ns	4.37ns	0.307	1.93 KB
#4739	`EnrichedLog`	net6.0	2.99μs	0.936ns	3.63ns	0.0297	2.13 KB
#4739	`EnrichedLog`	netcoreapp3.1	3.92μs	1.02ns	3.96ns	0.0294	2.13 KB
#4739	`EnrichedLog`	net472	4.61μs	4.52ns	17.5ns	0.306	1.93 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.28μs	0.428ns	1.6ns	0.0153	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.65μs	1.38ns	5.33ns	0.0148	1.1 KB
master	`SendReceive`	net472	1.89μs	2.69ns	10.4ns	0.177	1.12 KB
#4739	`SendReceive`	net6.0	1.37μs	0.669ns	2.59ns	0.015	1.1 KB
#4739	`SendReceive`	netcoreapp3.1	1.63μs	0.611ns	2.36ns	0.0146	1.1 KB
#4739	`SendReceive`	net472	1.96μs	0.611ns	2.29ns	0.176	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.66μs	0.615ns	2.38ns	0.0213	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.65μs	0.768ns	2.87ns	0.02	1.58 KB
master	`EnrichedLog`	net472	3.99μs	2.56ns	9.93ns	0.309	1.96 KB
#4739	`EnrichedLog`	net6.0	2.45μs	0.86ns	3.22ns	0.0208	1.53 KB
#4739	`EnrichedLog`	netcoreapp3.1	3.54μs	1.29ns	4.98ns	0.0214	1.58 KB
#4739	`EnrichedLog`	net472	3.99μs	1.12ns	4.19ns	0.31	1.96 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4739

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.153	684.06	593.49

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	471ns	0.189ns	0.732ns	0.00755	536 B
master	`StartFinishSpan`	netcoreapp3.1	580ns	0.271ns	1.05ns	0.00728	536 B
master	`StartFinishSpan`	net472	684ns	0.117ns	0.451ns	0.0854	538 B
master	`StartFinishScope`	net6.0	535ns	0.154ns	0.577ns	0.00903	656 B
master	`StartFinishScope`	netcoreapp3.1	754ns	0.158ns	0.59ns	0.00866	656 B
master	`StartFinishScope`	net472	881ns	0.359ns	1.34ns	0.098	618 B
#4739	`StartFinishSpan`	net6.0	444ns	0.251ns	0.972ns	0.00742	536 B
#4739	`StartFinishSpan`	netcoreapp3.1	566ns	3.1ns	17ns	0.00731	536 B
#4739	`StartFinishSpan`	net472	593ns	0.185ns	0.715ns	0.0852	538 B
#4739	`StartFinishScope`	net6.0	526ns	0.135ns	0.504ns	0.0092	656 B
#4739	`StartFinishScope`	netcoreapp3.1	743ns	0.451ns	1.75ns	0.00895	656 B
#4739	`StartFinishScope`	net472	854ns	0.611ns	2.36ns	0.0979	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	586ns	0.177ns	0.686ns	0.00908	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	812ns	2.49ns	9.66ns	0.00872	656 B
master	`RunOnMethodBegin`	net472	917ns	0.284ns	1.1ns	0.098	618 B
#4739	`RunOnMethodBegin`	net6.0	634ns	0.209ns	0.81ns	0.0092	656 B
#4739	`RunOnMethodBegin`	netcoreapp3.1	902ns	1.86ns	7.19ns	0.00851	656 B
#4739	`RunOnMethodBegin`	net472	978ns	0.231ns	0.896ns	0.0981	618 B

This solves problems around retrying throughput tests

…able

andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc labels Oct 20, 2023

andrewlock requested a review from a team as a code owner October 20, 2023 09:51

andrewlock mentioned this pull request Oct 20, 2023

Merge throughput_appsec into throughput stage #4740

Merged

2 tasks

andrewlock added 2 commits October 23, 2023 10:21

Always upload throughput results with the same name

133db1c

This solves problems around retrying throughput tests

Don't crash the compare_throughput stage if some results aren't avail…

e8f8deb

…able

andrewlock force-pushed the andrew/ci/fix_throughput_comparison branch from 16352f8 to e8f8deb Compare October 23, 2023 09:22

kevingosse approved these changes Oct 23, 2023

View reviewed changes

andrewlock merged commit fe10727 into master Oct 23, 2023
57 checks passed

andrewlock deleted the andrew/ci/fix_throughput_comparison branch October 23, 2023 14:30

github-actions bot added this to the vNext milestone Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `compare_throughput` stage more robust #4739

Make `compare_throughput` stage more robust #4739

andrewlock commented Oct 20, 2023

datadog-ddstaging bot commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #4739

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4739

Raw results

Raw results

Make compare_throughput stage more robust #4739

Make compare_throughput stage more robust #4739

Conversation

andrewlock commented Oct 20, 2023

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Oct 20, 2023 • edited Loading

Datadog Report

andrewlock commented Oct 20, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Oct 20, 2023 • edited Loading

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #4739

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4739

Raw results

Raw results

Make `compare_throughput` stage more robust #4739

Make `compare_throughput` stage more robust #4739

datadog-ddstaging bot commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading