Can't filter by traits on ITheoryDataRow

[justindbaur]

I had reported xunit/xunit#3030 before, on later versions I no longer get a Catastrophic failure but a trait filter with traits that come from ITheoryDataRow still don't appear to work.

Minimal repo: https://github.com/justindbaur/xunit-v3-problem

I expect dotnet test --filter "MyTrait=One" to run just the theory data row with 1, 2. But instead it finds no tests with the specified filter.

[
    new TheoryDataRow(1, 2)
        .WithTrait("MyTrait", "One"),
    new TheoryDataRow(2, 1)
        .WithTrait("MyTrait", "Two"),
]

I'm using:

xunit.v3: 0.6.0-pre.7
xunit.runner.visualstudio: 3.0.0-pre.49

[bradwilson]

I am working to reproduce the problem, but I'm running into what appears to be an environmental issue on my machine: microsoft/vstest#10605

[bradwilson]

Okay, I am able to reproduce this and the good news is, there is a workaround. If you enable theory pre-enumeration, then you can filter by the trait:

This verifies my hypothesis. For quite a while, we have tried to optimize performance by disabling theory pre-enumeration if you were running tests from the command line, on the idea that it was unnecessary:

visualstudio.xunit/src/xunit.runner.visualstudio/VsTestRunner.cs

Lines 210 to 212 in 9154f3f

	// IDE users get pre-enumeated theories by default
	if (testPlatformContext.DesignMode)
	assembly.Configuration.PreEnumerateTheories ??= true;

What's changed between v2 and v3 is that traits are now available via ITheoryDataRow but we never updated the code here and so when we don't pre-enumerate your theory data, we never see the traits and you can't filter by them.

The good news is, the fix is quite simple, so that's my next stop. 😄

[bradwilson]

Fix available in 3.0.0-pre.53 https://xunit.net/docs/using-ci-builds

[bradwilson]

Not quite fixed yet. 😬

[bradwilson]

Okay, but for real this time. 😂

Fix available in 3.0.0-pre.54 https://xunit.net/docs/using-ci-builds

[bradwilson]

And of course I see now my commit message says "off" instead of "on", but whatever. 💀