Fix `Non-serializable data ('System.Object[]') found` in tests #5089

andrewlock · 2024-01-19T11:53:09Z

Summary of changes

Fix cases where we had non-serializable data

Reason for change

Spotted in the logs:

Datadog.Trace.Tests: Non-serializable data ('System.Object[]') found for 'Datadog.Trace.Tests.Logging.ExceptionRedactorTests.RedactStackTrace_DoesNotRedactBclAndDatadog'; falling back to single test case

It means we're not actually running all the test cases

Implementation details

Use SerializableDictionary where possible, or where not possible, convert the Theory into a Fact

Test coverage

Should be more now!

Other details

datadog-ddstaging · 2024-01-19T12:08:12Z

Datadog Report

Branch report: andrew/ci/fix-non-serializable-tests
Commit report: 36aa7d2
Test service: dd-trace-dotnet

✅ 0 Failed, 307897 Passed, 1506 Skipped, 1h 1m 42.51s Wall Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

AllCycleSimpleBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 63.08µs (+18.51µs, +42%) - Details

andrewlock · 2024-01-19T12:25:17Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5089) - mean (72ms)  : 63, 81
     .   : milestone, 72,
    master - mean (71ms)  : 62, 80
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (5089) - mean (956ms)  : 932, 980
     .   : milestone, 956,
    master - mean (951ms)  : 928, 975
     .   : milestone, 951,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5089) - mean (106ms)  : 101, 111
     .   : milestone, 106,
    master - mean (106ms)  : 101, 111
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (5089) - mean (691ms)  : 673, 710
     .   : milestone, 691,
    master - mean (691ms)  : 673, 710
     .   : milestone, 691,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5089) - mean (89ms)  : 86, 92
     .   : milestone, 89,
    master - mean (89ms)  : 85, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (5089) - mean (651ms)  : 629, 673
     .   : milestone, 651,
    master - mean (649ms)  : 630, 669
     .   : milestone, 649,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5089) - mean (189ms)  : 186, 192
     .   : milestone, 189,
    master - mean (188ms)  : 185, 192
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5089) - mean (1,069ms)  : 1048, 1090
     .   : milestone, 1069,
    master - mean (1,066ms)  : 1041, 1090
     .   : milestone, 1066,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5089) - mean (271ms)  : 267, 276
     .   : milestone, 271,
    master - mean (271ms)  : 267, 276
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (5089) - mean (1,053ms)  : 1027, 1079
     .   : milestone, 1053,
    master - mean (1,049ms)  : 1026, 1071
     .   : milestone, 1049,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5089) - mean (262ms)  : 259, 265
     .   : milestone, 262,
    master - mean (262ms)  : 258, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (5089) - mean (1,013ms)  : 995, 1032
     .   : milestone, 1013,
    master - mean (1,014ms)  : 986, 1041
     .   : milestone, 1014,

andrewlock · 2024-01-21T03:40:11Z

Benchmarks Report 🐌

Benchmarks for #5089 compared to master:

5 benchmarks are faster, with geometric mean 1.157
2 benchmarks are slower, with geometric mean 1.290
1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.88μs	47ns	230ns	0.0226	0.00451	0	7.49 KB
master	`StartStopWithChild`	netcoreapp3.1	10.9μs	60.1ns	351ns	0.027	0.0108	0	7.58 KB
master	`StartStopWithChild`	net472	17.2μs	46.9ns	182ns	1.34	0.357	0.119	7.96 KB
#5089	`StartStopWithChild`	net6.0	8.74μs	49ns	306ns	0.0169	0.00846	0	7.48 KB
#5089	`StartStopWithChild`	netcoreapp3.1	11μs	61.8ns	410ns	0.0227	0.0114	0	7.58 KB
#5089	`StartStopWithChild`	net472	17.2μs	56.2ns	218ns	1.34	0.353	0.12	7.95 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	464μs	68.3ns	256ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	637μs	127ns	475ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	793μs	282ns	1.09μs	0.396	3.3 KB
#5089	`WriteAndFlushEnrichedTraces`	net6.0	458μs	117ns	439ns	0	2.7 KB
#5089	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	646μs	136ns	491ns	0	2.7 KB
#5089	`WriteAndFlushEnrichedTraces`	net472	795μs	395ns	1.53μs	0.398	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5089

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	1.435	43,957.63	63,083.83

Faster 🎉 in #5089

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1	1.226	250.27	204.11

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	38.6μs	20.3ns	78.5ns	0.019	0	1.77 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	41.2μs	21.1ns	78.8ns	0.0205	0	1.74 KB
master	`AllCycleSimpleBody`	net472	43.9μs	19.6ns	73.3ns	0.286	0	1.81 KB
master	`AllCycleMoreComplexBody`	net6.0	199μs	44.7ns	167ns	0.101	0	9.25 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	211μs	222ns	832ns	0.105	0	9.14 KB
master	`AllCycleMoreComplexBody`	net472	225μs	122ns	474ns	1.47	0	9.32 KB
master	`ObjectExtractorSimpleBody`	net6.0	142ns	0.238ns	0.923ns	0.00395	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	250ns	0.139ns	0.521ns	0.00379	0	272 B
master	`ObjectExtractorSimpleBody`	net472	173ns	0.271ns	1.01ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.99μs	2.27ns	8.8ns	0.0524	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.12μs	2.27ns	8.49ns	0.0491	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.83μs	4.53ns	16.9ns	0.603	0.00572	3.8 KB
#5089	`AllCycleSimpleBody`	net6.0	38.1μs	19.9ns	74.5ns	0.019	0	1.77 KB
#5089	`AllCycleSimpleBody`	netcoreapp3.1	41.7μs	93.9ns	364ns	0.0208	0	1.74 KB
#5089	`AllCycleSimpleBody`	net472	63.1μs	19.2ns	71.7ns	0.283	0	1.81 KB
#5089	`AllCycleMoreComplexBody`	net6.0	200μs	93.9ns	364ns	0.1	0	9.25 KB
#5089	`AllCycleMoreComplexBody`	netcoreapp3.1	212μs	155ns	601ns	0.107	0	9.14 KB
#5089	`AllCycleMoreComplexBody`	net472	228μs	84.6ns	316ns	1.48	0	9.32 KB
#5089	`ObjectExtractorSimpleBody`	net6.0	143ns	0.155ns	0.601ns	0.0039	0	280 B
#5089	`ObjectExtractorSimpleBody`	netcoreapp3.1	204ns	0.17ns	0.635ns	0.0037	0	272 B
#5089	`ObjectExtractorSimpleBody`	net472	161ns	0.0948ns	0.367ns	0.0446	0	281 B
#5089	`ObjectExtractorMoreComplexBody`	net6.0	3.02μs	2.07ns	7.74ns	0.0529	0	3.78 KB
#5089	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4μs	3.62ns	14ns	0.0499	0	3.69 KB
#5089	`ObjectExtractorMoreComplexBody`	net472	3.76μs	2.02ns	7.27ns	0.603	0.00562	3.8 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5089

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net6.0	1.129	111,125.43	98,435.97

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	56.4μs	30.5ns	114ns	0.225	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	71.8μs	49.5ns	185ns	0.208	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	net472	99.1μs	507ns	2.38μs	2.55	0.0964	16.14 KB
master	`RunWafTwice(args=NestedMap (10))`	net6.0	53.6μs	19.3ns	72.1ns	0.218	0	16.6 KB
master	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	70.3μs	143ns	608ns	0.224	0	16.58 KB
master	`RunWafTwice(args=NestedMap (10))`	net472	104μs	48.4ns	188ns	2.65	0.104	16.69 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	116μs	185ns	718ns	0.289	0	22.41 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	132μs	629ns	2.44μs	0.253	0	22.36 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	165μs	394ns	1.53μs	3.53	0.161	22.7 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	111μs	79.8ns	309ns	0.461	0	32.76 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	134μs	702ns	3.37μs	0.404	0	33.33 KB
master	`RunWaf(args=NestedMap (100))`	net472	188μs	96ns	359ns	5.27	0.377	33.67 KB
master	`RunWafTwice(args=NestedMap (100))`	net6.0	103μs	33.5ns	130ns	0.435	0	33.3 KB
master	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	136μs	672ns	2.77μs	0.466	0	33.86 KB
master	`RunWafTwice(args=NestedMap (100))`	net472	193μs	97.5ns	365ns	5.42	0.387	34.23 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	163μs	852ns	4.43μs	0.505	0	39.1 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	199μs	972ns	3.89μs	0.506	0	39.63 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	261μs	998ns	3.86μs	6.39	0.511	40.23 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	100μs	28.6ns	111ns	0.425	0	32.18 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	140μs	86.1ns	334ns	0.386	0	32.3 KB
master	`RunWaf(args=NestedMap (20))`	net472	185μs	109ns	376ns	5.1	0.371	32.63 KB
master	`RunWafTwice(args=NestedMap (20))`	net6.0	100μs	46.7ns	175ns	0.423	0	32.72 KB
master	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	135μs	700ns	3.21μs	0.414	0	32.82 KB
master	`RunWafTwice(args=NestedMap (20))`	net472	194μs	122ns	474ns	5.25	0.389	33.19 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	167μs	325ns	1.26μs	0.505	0	38.53 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	205μs	928ns	3.59μs	0.505	0	38.6 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	260μs	1.07μs	4.14μs	6.22	0.508	39.2 KB
#5089	`RunWaf(args=NestedMap (10))`	net6.0	51.9μs	40.3ns	145ns	0.204	0	16.06 KB
#5089	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	70.4μs	379ns	1.97μs	0.198	0	16.06 KB
#5089	`RunWaf(args=NestedMap (10))`	net472	96.2μs	75.7ns	293ns	2.55	0.0962	16.14 KB
#5089	`RunWafTwice(args=NestedMap (10))`	net6.0	56.8μs	326ns	2.46μs	0.227	0	16.6 KB
#5089	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	75.3μs	381ns	1.7μs	0.192	0	16.58 KB
#5089	`RunWafTwice(args=NestedMap (10))`	net472	107μs	519ns	2.08μs	2.61	0.104	16.69 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	115μs	149ns	579ns	0.287	0	22.41 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	135μs	426ns	1.54μs	0.255	0	22.36 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	160μs	127ns	492ns	3.58	0.159	22.7 KB
#5089	`RunWaf(args=NestedMap (100))`	net6.0	98.5μs	23.3ns	90.3ns	0.442	0	32.76 KB
#5089	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	135μs	730ns	4.13μs	0.448	0	33.33 KB
#5089	`RunWaf(args=NestedMap (100))`	net472	185μs	124ns	428ns	5.33	0.368	33.67 KB
#5089	`RunWafTwice(args=NestedMap (100))`	net6.0	102μs	30.2ns	121ns	0.435	0	33.3 KB
#5089	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	140μs	679ns	2.8μs	0.424	0	33.86 KB
#5089	`RunWafTwice(args=NestedMap (100))`	net472	194μs	128ns	479ns	5.4	0.386	34.23 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	158μs	79.3ns	297ns	0.553	0	39.1 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	200μs	788ns	3.05μs	0.496	0	39.63 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	254μs	236ns	915ns	6.31	0.505	40.23 KB
#5089	`RunWaf(args=NestedMap (20))`	net6.0	109μs	511ns	2.05μs	0.429	0	32.18 KB
#5089	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	134μs	762ns	5.11μs	0.441	0	32.3 KB
#5089	`RunWaf(args=NestedMap (20))`	net472	191μs	763ns	2.96μs	5.12	0.372	32.63 KB
#5089	`RunWafTwice(args=NestedMap (20))`	net6.0	107μs	28.7ns	107ns	0.422	0	32.72 KB
#5089	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	139μs	725ns	3.48μs	0.399	0	32.82 KB
#5089	`RunWafTwice(args=NestedMap (20))`	net472	193μs	128ns	494ns	5.22	0.386	33.19 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	168μs	63.9ns	248ns	0.55	0	38.53 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	195μs	977ns	4.48μs	0.491	0	38.6 KB
#5089	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	255μs	642ns	2.4μs	6.19	0.505	39.2 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	176μs	201ns	778ns	0.174	18.25 KB
master	`SendRequest`	netcoreapp3.1	194μs	277ns	1.07μs	0.193	20.41 KB
master	`SendRequest`	net472	0.00079ns	0.000282ns	0.00105ns	0	0 b
#5089	`SendRequest`	net6.0	174μs	137ns	514ns	0.173	18.25 KB
#5089	`SendRequest`	netcoreapp3.1	193μs	249ns	963ns	0.192	20.41 KB
#5089	`SendRequest`	net472	0.000852ns	0.000338ns	0.00131ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	551μs	1.17μs	4.53μs	0.551	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	661μs	253ns	980ns	0.334	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	net472	831μs	2.49μs	9.63μs	8.28	2.48	0.414	53.24 KB
#5089	`WriteAndFlushEnrichedTraces`	net6.0	549μs	462ns	1.79μs	0.534	0	0	41.54 KB
#5089	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	1.1μs	4.25μs	0.329	0	0	41.59 KB
#5089	`WriteAndFlushEnrichedTraces`	net472	812μs	3.29μs	12.3μs	8.08	2.55	0.425	53.22 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.13μs	0.593ns	2.14ns	0.0107	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.49μs	0.746ns	2.89ns	0.0104	768 B
master	`ExecuteNonQuery`	net472	1.84μs	0.385ns	1.49ns	0.115	730 B
#5089	`ExecuteNonQuery`	net6.0	1.02μs	0.409ns	1.53ns	0.0108	768 B
#5089	`ExecuteNonQuery`	netcoreapp3.1	1.45μs	1.17ns	4.39ns	0.0103	768 B
#5089	`ExecuteNonQuery`	net472	1.81μs	0.685ns	2.47ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5089

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0	1.148	1,459.17	1,270.82

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.18μs	0.461ns	1.79ns	0.013	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.61μs	0.974ns	3.77ns	0.012	936 B
master	`CallElasticsearch`	net472	2.46μs	0.841ns	3.03ns	0.151	955 B
master	`CallElasticsearchAsync`	net6.0	1.46μs	0.469ns	1.82ns	0.0124	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	0.491ns	1.77ns	0.013	984 B
master	`CallElasticsearchAsync`	net472	2.67μs	1.85ns	7.15ns	0.16	1.01 KB
#5089	`CallElasticsearch`	net6.0	1.1μs	0.49ns	1.83ns	0.0131	936 B
#5089	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.34ns	5.19ns	0.0124	936 B
#5089	`CallElasticsearch`	net472	2.52μs	1.14ns	4.43ns	0.151	955 B
#5089	`CallElasticsearchAsync`	net6.0	1.27μs	0.445ns	1.72ns	0.0127	912 B
#5089	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.84ns	3.03ns	0.0126	984 B
#5089	`CallElasticsearchAsync`	net472	2.59μs	0.781ns	3.02ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5089

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0	1.160	1,228.00	1,424.99

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.23μs	0.553ns	2.07ns	0.013	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.61μs	1.95ns	7.29ns	0.0118	912 B
master	`ExecuteAsync`	net472	1.88μs	1.02ns	3.93ns	0.138	875 B
#5089	`ExecuteAsync`	net6.0	1.42μs	1.49ns	5.57ns	0.0124	912 B
#5089	`ExecuteAsync`	netcoreapp3.1	1.55μs	3.37ns	13ns	0.0123	912 B
#5089	`ExecuteAsync`	net472	1.77μs	0.829ns	3.21ns	0.139	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendAsync`	net6.0	4.37μs	1.71ns	6.18ns	0.0285	0	2.1 KB
master	`SendAsync`	netcoreapp3.1	4.89μs	1.58ns	6.13ns	0.0366	0	2.63 KB
master	`SendAsync`	net472	7.6μs	2.69ns	10.1ns	0.523	0.00379	3.31 KB
#5089	`SendAsync`	net6.0	4.13μs	1.74ns	6.52ns	0.0288	0	2.1 KB
#5089	`SendAsync`	netcoreapp3.1	5.13μs	1.4ns	5.24ns	0.0334	0	2.63 KB
#5089	`SendAsync`	net472	7.58μs	3.55ns	13.8ns	0.523	0	3.31 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #5089

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	62.18 KB	59.39 KB	-2.79 KB	-4.49%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	61.4μs	825ns	8.13μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	54μs	168ns	650ns	42.64 KB
master	`StringConcatBenchmark`	net472	37.9μs	86.9ns	325ns	62.18 KB
master	`StringConcatAspectBenchmark`	net6.0	285μs	3.06μs	28μs	202.62 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	279μs	7.05μs	64.3μs	201.97 KB
master	`StringConcatAspectBenchmark`	net472	238μs	3.67μs	35.2μs	221.18 KB
#5089	`StringConcatBenchmark`	net6.0	61.6μs	825ns	8.12μs	43.44 KB
#5089	`StringConcatBenchmark`	netcoreapp3.1	53.4μs	215ns	745ns	42.64 KB
#5089	`StringConcatBenchmark`	net472	39.6μs	189ns	781ns	59.39 KB
#5089	`StringConcatAspectBenchmark`	net6.0	267μs	1.37μs	6.15μs	202.58 KB
#5089	`StringConcatAspectBenchmark`	netcoreapp3.1	283μs	3.32μs	30.2μs	201.09 KB
#5089	`StringConcatAspectBenchmark`	net472	257μs	5.24μs	50.3μs	221.18 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.5μs	0.638ns	2.39ns	0.0219	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	2.39μs	0.958ns	3.71ns	0.0215	1.57 KB
master	`EnrichedLog`	net472	2.68μs	2.57ns	9.95ns	0.238	1.5 KB
#5089	`EnrichedLog`	net6.0	1.43μs	0.71ns	2.66ns	0.0222	1.57 KB
#5089	`EnrichedLog`	netcoreapp3.1	2.18μs	1.09ns	3.93ns	0.0209	1.57 KB
#5089	`EnrichedLog`	net472	2.66μs	2.2ns	7.92ns	0.237	1.5 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	59.3ns	222ns	0.0558	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	98ns	379ns	0	0	4.21 KB
master	`EnrichedLog`	net472	150μs	71.6ns	258ns	0.678	0.226	4.39 KB
#5089	`EnrichedLog`	net6.0	113μs	91.3ns	353ns	0.0567	0	4.21 KB
#5089	`EnrichedLog`	netcoreapp3.1	118μs	88.4ns	342ns	0.0596	0	4.21 KB
#5089	`EnrichedLog`	net472	147μs	59.7ns	231ns	0.663	0.221	4.39 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3μs	0.761ns	2.95ns	0.0299	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	4.15μs	2.31ns	8.95ns	0.0293	2.13 KB
master	`EnrichedLog`	net472	4.91μs	2.29ns	8.89ns	0.309	1.95 KB
#5089	`EnrichedLog`	net6.0	3.01μs	2.17ns	8.39ns	0.0301	2.13 KB
#5089	`EnrichedLog`	netcoreapp3.1	4.41μs	0.914ns	3.3ns	0.0286	2.13 KB
#5089	`EnrichedLog`	net472	4.77μs	1.5ns	5.62ns	0.309	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.44μs	0.964ns	3.61ns	0.015	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.71μs	2.58ns	9.98ns	0.0145	1.1 KB
master	`SendReceive`	net472	2.09μs	1.68ns	6.3ns	0.176	1.12 KB
#5089	`SendReceive`	net6.0	1.32μs	0.398ns	1.54ns	0.0151	1.1 KB
#5089	`SendReceive`	netcoreapp3.1	1.69μs	0.526ns	1.97ns	0.0152	1.1 KB
#5089	`SendReceive`	net472	2.09μs	2.46ns	9.2ns	0.177	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.75μs	1.74ns	6.73ns	0.0206	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.83μs	1.77ns	6.84ns	0.0213	1.58 KB
master	`EnrichedLog`	net472	4.35μs	1.66ns	6.44ns	0.312	1.97 KB
#5089	`EnrichedLog`	net6.0	2.92μs	1.04ns	3.89ns	0.0205	1.53 KB
#5089	`EnrichedLog`	netcoreapp3.1	3.82μs	1.2ns	4.66ns	0.0211	1.58 KB
#5089	`EnrichedLog`	net472	4.34μs	1.55ns	6.02ns	0.311	1.97 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5089

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.151	522.43	453.99

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	522ns	0.194ns	0.752ns	0.00761	536 B
master	`StartFinishSpan`	netcoreapp3.1	682ns	0.784ns	3.04ns	0.00718	536 B
master	`StartFinishSpan`	net472	763ns	0.367ns	1.42ns	0.0854	538 B
master	`StartFinishScope`	net6.0	577ns	0.135ns	0.506ns	0.0093	656 B
master	`StartFinishScope`	netcoreapp3.1	815ns	0.468ns	1.75ns	0.00884	656 B
master	`StartFinishScope`	net472	959ns	0.985ns	3.69ns	0.098	618 B
#5089	`StartFinishSpan`	net6.0	455ns	0.609ns	2.36ns	0.00757	536 B
#5089	`StartFinishSpan`	netcoreapp3.1	696ns	0.807ns	3.12ns	0.00718	536 B
#5089	`StartFinishSpan`	net472	755ns	0.445ns	1.72ns	0.0852	538 B
#5089	`StartFinishScope`	net6.0	603ns	0.173ns	0.671ns	0.00905	656 B
#5089	`StartFinishScope`	netcoreapp3.1	867ns	0.667ns	2.31ns	0.00867	656 B
#5089	`StartFinishScope`	net472	955ns	0.748ns	2.9ns	0.0978	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5089

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.132	693.20	612.43

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	694ns	0.523ns	1.96ns	0.00913	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	905ns	0.766ns	2.97ns	0.00868	656 B
master	`RunOnMethodBegin`	net472	1.08μs	0.211ns	0.762ns	0.098	618 B
#5089	`RunOnMethodBegin`	net6.0	612ns	0.127ns	0.477ns	0.00917	656 B
#5089	`RunOnMethodBegin`	netcoreapp3.1	970ns	0.551ns	2.14ns	0.00891	656 B
#5089	`RunOnMethodBegin`	net472	1.07μs	0.314ns	1.22ns	0.0977	618 B

andrewlock · 2024-01-22T11:18:27Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5089) (11.205M)   : 0, 11204918
    master (11.262M)   : 0, 11261656
    benchmarks/2.9.0 (11.015M)   : 0, 11014948

    section Automatic
    This PR (5089) (7.668M)   : 0, 7668197
    master (7.747M)   : 0, 7747211
    benchmarks/2.9.0 (7.926M)   : 0, 7925773

    section Trace stats
    This PR (5089) (7.996M)   : 0, 7995785
    master (8.059M)   : 0, 8058958

    section Manual
    This PR (5089) (9.637M)   : 0, 9637379
    master (9.825M)   : 0, 9825345

    section Manual + Automatic
    This PR (5089) (7.093M)   : 0, 7093095
    master (7.259M)   : 0, 7259332

    section Version Conflict
    This PR (5089) (6.508M)   : 0, 6508126
    master (6.615M)   : 0, 6615074

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5089) (9.551M)   : 0, 9550737
    master (9.478M)   : 0, 9478111
    benchmarks/2.9.0 (9.652M)   : 0, 9651615

    section Automatic
    This PR (5089) (6.516M)   : 0, 6515889
    master (6.620M)   : 0, 6620487

    section Trace stats
    This PR (5089) (6.986M)   : 0, 6985797
    master (6.849M)   : 0, 6848670

    section Manual
    This PR (5089) (8.414M)   : 0, 8414077
    master (8.249M)   : 0, 8249061

    section Manual + Automatic
    This PR (5089) (6.200M)   : 0, 6199557
    master (6.297M)   : 0, 6297028

    section Version Conflict
    This PR (5089) (5.701M)   : 0, 5701208
    master (5.820M)   : 0, 5820337

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5089) (10.065M)   : 0, 10064715
    master (9.914M)   : 0, 9914032
    benchmarks/2.9.0 (10.371M)   : 0, 10371072

    section Automatic
    This PR (5089) (7.206M)   : 0, 7206189
    master (7.079M)   : 0, 7078578
    benchmarks/2.9.0 (7.518M)   : 0, 7518013

    section Trace stats
    This PR (5089) (7.397M)   : 0, 7397068
    master (7.478M)   : 0, 7477500

    section Manual
    This PR (5089) (8.776M)   : 0, 8775723
    master (8.632M)   : 0, 8632473

    section Manual + Automatic
    This PR (5089) (6.884M)   : 0, 6883717
    master (6.726M)   : 0, 6725927

    section Version Conflict
    This PR (5089) (6.114M)   : 0, 6114118
    master (6.119M)   : 0, 6118860

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.543M)   : 0, 7542709
    benchmarks/2.9.0 (7.958M)   : 0, 7957509

    section No attack
    master (1.784M)   : 0, 1783760
    benchmarks/2.9.0 (3.247M)   : 0, 3247318

    section Attack
    master (1.428M)   : 0, 1428109
    benchmarks/2.9.0 (2.539M)   : 0, 2539306

    section Blocking
    master (3.111M)   : 0, 3110933

    section IAST default
    master (6.515M)   : 0, 6514760

    section IAST full
    master (5.814M)   : 0, 5814104

    section Base vuln
    master (0.972M)   : 0, 971808

    section IAST vuln
    master (0.863M)   : 0, 862917

andrewlock added the area:tests unit tests, integration tests label Jan 19, 2024

andrewlock requested a review from a team as a code owner January 19, 2024 11:53

andrewlock force-pushed the andrew/ci/fix-non-serializable-tests branch from 91c052d to a4ec4d2 Compare January 19, 2024 13:40

zacharycmontoya approved these changes Jan 19, 2024

View reviewed changes

andrewlock added 2 commits January 22, 2024 09:49

Fix Non-serializable data ('System.Object[]') found in tests

ae0ea31

Fix incorrect casts

36aa7d2

andrewlock force-pushed the andrew/ci/fix-non-serializable-tests branch from a4ec4d2 to 36aa7d2 Compare January 22, 2024 09:49

lucaspimentel approved these changes Jan 23, 2024

View reviewed changes

andrewlock merged commit 9ab015f into master Feb 12, 2024
55 checks passed

andrewlock deleted the andrew/ci/fix-non-serializable-tests branch February 12, 2024 13:47

github-actions bot added this to the vNext milestone Feb 12, 2024

lucaspimentel mentioned this pull request Sep 30, 2024

Remove warning about non-serializable test data #6102

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `Non-serializable data ('System.Object[]') found` in tests #5089

Fix `Non-serializable data ('System.Object[]') found` in tests #5089

andrewlock commented Jan 19, 2024

datadog-ddstaging bot commented Jan 19, 2024 •

edited

Loading

andrewlock commented Jan 19, 2024 •

edited

Loading

andrewlock commented Jan 21, 2024 •

edited

Loading

Raw results

Raw results

Slower ⚠️ in #5089

Faster 🎉 in #5089

Raw results

Faster 🎉 in #5089

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5089

Raw results

Slower ⚠️ in #5089

Raw results

Raw results

Fewer allocations 🎉 in #5089

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5089

Raw results

Faster 🎉 in #5089

Raw results

andrewlock commented Jan 22, 2024

Fix Non-serializable data ('System.Object[]') found in tests #5089

Fix Non-serializable data ('System.Object[]') found in tests #5089

Conversation

andrewlock commented Jan 19, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Jan 19, 2024 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented Jan 19, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 21, 2024 • edited Loading

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Slower ⚠️ in #5089

Faster 🎉 in #5089

Raw results

Faster 🎉 in #5089

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5089

Raw results

Slower ⚠️ in #5089

Raw results

Raw results

Fewer allocations 🎉 in #5089

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5089

Raw results

Faster 🎉 in #5089

Raw results

andrewlock commented Jan 22, 2024

Throughput/Crank Report:zap:

Fix `Non-serializable data ('System.Object[]') found` in tests #5089

Fix `Non-serializable data ('System.Object[]') found` in tests #5089

datadog-ddstaging bot commented Jan 19, 2024 •

edited

Loading

andrewlock commented Jan 19, 2024 •

edited

Loading

andrewlock commented Jan 21, 2024 •

edited

Loading