Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks don't run for .NET 6+ #2095

Open
idg10 opened this issue Mar 20, 2024 · 0 comments
Open

Benchmarks don't run for .NET 6+ #2095

idg10 opened this issue Mar 20, 2024 · 0 comments
Assignees

Comments

@idg10
Copy link
Collaborator

idg10 commented Mar 20, 2024

Rx.NET has a benchmark suite, but it currently works only for .NET Framework, and not for .NET 6.0 and later.

If we change the <TargetFramework> to net6.0, the benchmarks fail to run. This seems to be because BenchmarkDotNet expects benchmark projects to use the Release build configuration, but the Rx.NET benchmark project has multiple configurations with the goal of being able to run the benchmarks against older Rx.NET libraries.

This seems like it should offer the following advantages:

  • we can introduce new benchmarks to address performance scenarios not previously covered, and run them against older libraries to see whether the scenario regressed
  • we can rerun benchmarks against older libraries on different hardware

That second point is useful if we want to start tracking performance on processor architectures for which we did not have historical data. It's also useful if we want to change the machine type on which we regularly run benchmarks—in addition to being able to re-establish baselines for the current version against new hardware, we can see if the change in performance over time looked different on the new hardware.

Currently it can run benchmarks back as far as v3.1.1.

The problem is that that libraries are selected by choosing different build configurations, and this conflicts with how BenchmarkDotNet wants to do things. (The symptom is a build failure, which I think is caused by inconsistent setting of the CURRENT compilation symbol, meaning that BenchmarkDotNet generates some source code trying to use a benchmark that isn't actually present.)

BenchmarkDotNet now has ways of configuring which external library versions to use. E.g., I found this in a Microsoft blog:

var config = DefaultConfig.Instance
    .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core70)
        .WithNuGet("Microsoft.Extensions.Logging", "7.0.0")
        .AsBaseline())
    .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core80)
        .WithNuGet("Microsoft.Extensions.Logging", "8.0.0-rc.1.23419.4"));

So I think it should be possible for us to use this mechanism to change the library references. (We'd also need to find some way to control the preprocessors defines so that we can omit benchmarks that just can't run on older versions.)

@idg10 idg10 self-assigned this Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant