Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase Redis test matrix #2472

Merged
merged 2 commits into from
Feb 23, 2022
Merged

Increase Redis test matrix #2472

merged 2 commits into from
Feb 23, 2022

Conversation

andrewlock
Copy link
Member

Summary of changes

  • Try some things to reduce redis flakiness in CI
  • Add testing against 2.x versions of StackExchange.Redis

Reason for change

We haven't ever been testing against 2.x, as far as I can tell 😬

Also we relatively frequently see errors like RedisConnectionException: No connection is available to service this operation, so trying some things from this issue (Update NuGet, increase timeout, don't abort on initial failure)

Implementation details

We have another case of NuGet packages lying about supporting .NET Standard 2.0, so excluding 2.2.0+ from .NET Core 2.1. Apparently they work on 3.0 though 🤷‍♂️

Test coverage

Ran an additional "comprehensive" suite to avoid issues after merging to master.

@andrewlock andrewlock added area:tests unit tests, integration tests area:test-apps apps used to test integrations labels Feb 18, 2022
@andrewlock andrewlock requested a review from a team as a code owner February 18, 2022 12:45
@andrewlock
Copy link
Member Author

Code Coverage Report 📊

✔️ Merging #2472 into master will not change line coverage
✔️ Merging #2472 into master will not change branch coverage
✔️ Merging #2472 into master will not change complexity

master #2472 Change
Lines 11627 / 15971 11591 / 15971
Lines % 73% 73% 0% ✔️
Branches 6255 / 9336 6238 / 9336
Branches % 67% 67% 0% ✔️
Complexity 10546 10546 0 ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #2472 Change
Lines % 73% 73% 0% ✔️
Branches % 67% 67% 0% ✔️
Complexity 10546 10546 0 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.ClrProfiler.AutoInstrumentation.MongoDb.MongoDbIntegration -7% ⛔ 0% ✔️ 0 ✔️

View the full reports for further details:

@andrewlock
Copy link
Member Author

Benchmarks Report 🐌

Benchmarks for #2472 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 912μs 4.56μs 20.4μs 0 0 0 3.16 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 695μs 1.67μs 6.01μs 0 0 0 2.57 KB
#2472 WriteAndFlushEnrichedTraces net472 995μs 3.9μs 19.1μs 0 0 0 3.16 KB
#2472 WriteAndFlushEnrichedTraces netcoreapp3.1 719μs 3.24μs 17.8μs 0 0 0 2.57 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0.00013ns 0.00013ns 0.000759ns 0 0 0 0 b
master SendRequest netcoreapp3.1 347μs 1.05μs 4.06μs 0.173 0 0 19.71 KB
#2472 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#2472 SendRequest netcoreapp3.1 349μs 1.41μs 5.47μs 0.177 0 0 19.71 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.75μs 7.56ns 28.3ns 0.0925 0.000849 0 586 B
master ExecuteNonQuery netcoreapp3.1 1.56μs 7.4ns 30.5ns 0.00851 0 0 624 B
#2472 ExecuteNonQuery net472 1.77μs 8.9ns 39.8ns 0.0929 0.00086 0 586 B
#2472 ExecuteNonQuery netcoreapp3.1 1.59μs 5.67ns 26ns 0.00822 0 0 624 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.75μs 15.5ns 99.2ns 0.124 0 0 794 B
master CallElasticsearch netcoreapp3.1 1.93μs 10.6ns 85.5ns 0.0107 0 0 784 B
master CallElasticsearchAsync net472 2.92μs 12.4ns 46.5ns 0.147 0.00145 0 931 B
master CallElasticsearchAsync netcoreapp3.1 1.94μs 8.41ns 32.6ns 0.0126 0 0 904 B
#2472 CallElasticsearch net472 2.72μs 11.6ns 54.3ns 0.125 0 0 794 B
#2472 CallElasticsearch netcoreapp3.1 1.85μs 8.22ns 30.7ns 0.011 0 0 784 B
#2472 CallElasticsearchAsync net472 2.9μs 14.6ns 67ns 0.148 0.00141 0 931 B
#2472 CallElasticsearchAsync netcoreapp3.1 1.92μs 7.34ns 26.5ns 0.0125 0 0 904 B
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 3.15μs 13.1ns 48.9ns 0.167 0.00157 0 1.05 KB
master ExecuteAsync netcoreapp3.1 2.04μs 7.43ns 25.7ns 0.0144 0 0 1.02 KB
#2472 ExecuteAsync net472 3.15μs 16ns 71.6ns 0.167 0.00157 0 1.05 KB
#2472 ExecuteAsync netcoreapp3.1 2.28μs 12.8ns 81.7ns 0.0143 0 0 1.02 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 7.85μs 46.1ns 454ns 0.358 0 0 2.3 KB
master SendAsync netcoreapp3.1 4.98μs 11.5ns 41.3ns 0.0302 0 0 2.2 KB
#2472 SendAsync net472 7.57μs 37.5ns 172ns 0.361 0 0 2.3 KB
#2472 SendAsync netcoreapp3.1 5.04μs 13.5ns 52.3ns 0.0306 0 0 2.2 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.39μs 16.1ns 68.3ns 0.224 0 0 1.44 KB
master EnrichedLog netcoreapp3.1 3.24μs 7.22ns 26ns 0.0208 0 0 1.52 KB
#2472 EnrichedLog net472 3.52μs 14.1ns 54.7ns 0.224 0 0 1.44 KB
#2472 EnrichedLog netcoreapp3.1 3.27μs 5.85ns 22.7ns 0.0209 0 0 1.52 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 303μs 1.1μs 4.12μs 0.456 0.152 0 4.31 KB
master EnrichedLog netcoreapp3.1 242μs 551ns 1.99μs 0 0 0 4.19 KB
#2472 EnrichedLog net472 304μs 703ns 2.63μs 0.312 0.156 0 4.31 KB
#2472 EnrichedLog netcoreapp3.1 247μs 1.27μs 6.1μs 0 0 0 4.19 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 7.95μs 43.4ns 316ns 0.504 0 0 3.21 KB
master EnrichedLog netcoreapp3.1 6.55μs 22.2ns 80.1ns 0.0509 0 0 3.58 KB
#2472 EnrichedLog net472 7.92μs 40.8ns 267ns 0.501 0 0 3.21 KB
#2472 EnrichedLog netcoreapp3.1 6.81μs 19.6ns 73.4ns 0.0483 0 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.36μs 8.7ns 33.7ns 0.159 0.00116 0 1 KB
master SendReceive netcoreapp3.1 2.22μs 6.65ns 24.9ns 0.0141 0 0 1 KB
#2472 SendReceive net472 2.42μs 8.31ns 31.1ns 0.159 0.00122 0 1 KB
#2472 SendReceive netcoreapp3.1 2.34μs 10.8ns 41.7ns 0.0141 0 0 1 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 6.09μs 29.1ns 116ns 0.29 0 0 1.85 KB
master EnrichedLog netcoreapp3.1 5.95μs 32.1ns 173ns 0.0204 0 0 1.47 KB
#2472 EnrichedLog net472 6.17μs 29.7ns 123ns 0.289 0 0 1.85 KB
#2472 EnrichedLog netcoreapp3.1 5.8μs 32.6ns 216ns 0.0193 0 0 1.47 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 955ns 5.45ns 38.1ns 0.0705 0 0 449 B
master StartFinishSpan netcoreapp3.1 974ns 5.03ns 23.1ns 0.0063 0 0 448 B
master StartFinishScope net472 1.14μs 5.29ns 25.9ns 0.0833 0 0 530 B
master StartFinishScope netcoreapp3.1 1.1μs 4.41ns 16.5ns 0.00782 0 0 568 B
#2472 StartFinishSpan net472 946ns 4.62ns 20.7ns 0.0706 0 0 449 B
#2472 StartFinishSpan netcoreapp3.1 912ns 3ns 11.2ns 0.00638 0 0 448 B
#2472 StartFinishScope net472 1.13μs 4.18ns 15.1ns 0.083 0 0 530 B
#2472 StartFinishScope netcoreapp3.1 1.09μs 2.83ns 10.6ns 0.00798 0 0 568 B

@andrewlock andrewlock merged commit dfb0492 into master Feb 23, 2022
@andrewlock andrewlock deleted the andrew/increase-redis-testing branch February 23, 2022 09:50
@github-actions github-actions bot added this to the vNext milestone Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kevingosse kevingosse kevingosse approved these changes

Assignees
No one assigned
Labels
area:test-apps apps used to test integrations area:tests unit tests, integration tests
Projects
None yet
Milestone
2.5.0
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewlock @kevingosse