NUnit3TestAdapter 3.15 not running tests with custom TestCaseSource

[MattKeenum]

NUnit 3.12.0
NUnit3TestAdapter 3.15.0
Visual Studio 2019 Version 16.1.3
.NET Framework 4.6.1

namespace NUnitTest
{
   public class UnitTest1
    {
        public class DataProvider<T> where T : TestData, new()
        {
            public static IEnumerable<TestCaseData> GetData()
            {
                string[] names = new string[] { "Matthew", "Kelly", "Shawn" };
                char[] genders = new char[] { 'M', 'F', 'M' };
                int[] ages = new int[] { 35, 34, 30 };

                for (int i = 0; i < names.Length; i++)
                {
                    TestData testData = new TestData()
                    {
                        Name = names[i],
                        Gender = genders[i],
                        Age = ages[i]
                    };

                    TestCaseData tcd = new TestCaseData(testData);
                    tcd.SetName("{m} [" + testData.Name + " - " + testData.Gender + " - " + testData.Age + "]");

                    yield return tcd;
                }
            }
        }


        [Test]
        [TestCaseSource(typeof(DataProvider<TestData>), "GetData")]
        public void Persons(TestData test)
        {
            Assert.IsTrue(test.Name == "Matthew");
        }
    }
}

The above code correctly generates 3 test cases that have the following names in the Test Explorer:

Persons [Kelly - F - 34]
Persons [Matthew - M - 35]
Persons [Shawn - M - 30]

However, when I attempt to run/debug any of those test cases, they disappear from the Test Explorer, and nothing happens. A simple rebuild (or another clean and build) brings them back, but they still cannot be executed as they just disappear again.

Using the debugger, the custom data

If I downgrade to NUnit3TestAdapater 3.14, everything works as expected, I get 2 failed and 1 passed.

[OsirisTerje]

Thanks @MattKeenum ! Based on this case, in addition to the other, I am pretty sure we need to rollback 3.15. That performance work is unfortunately not finished.

[turcunicusor]

@MattKeenum How can I downgrade to NUnit3TestAdapater 3.14, without using NuGet?

[MattKeenum]

@turcunicusor You would need to manually remove the reference to the TestAdapter, and then add a reference to the previous version. As a double-check, you can check your App.config, csproj file and packages.config files for any references pointing to the 3.15 version and change them to the 3.14 version.

[turcunicusor]

I found https://github.com/nunit/nunit3-vs-adapter/releases all the Visual Studio test Adapters. Just uninstall current version of NUnit, download the proper version(.vsix file format), and then install it.

[MattKeenum]

Be aware that the vsix extension is being deprecated. It is advised that you use NuGet to install NUnit.
http://hermit.no/vsix-based-test-adapters-to-be-deprecated-in-visual-studio-for-c-and-visual-basic-to-speed-up-testing/

[jnm2]

@MattKeenum I'm not sure exactly what's happening with SetName. I think it might be losing the namespace information from the test names. We will keep looking into it, but in the meantime, NUnit Framework has an API that doesn't break in this situation which is IMO a superior way of setting test names. (It doesn't have funky problems like unescapable tokens getting replaced the way SetName does.)

Instead of:

tcd.SetName("{m} [" + testData.Name + " - " + testData.Gender + " - " + testData.Age + "]");

See what you think of:

tcd.SetArgDisplayNames(testData.Name, testData.Gender.ToString(), testData.Age.ToString());

[OsirisTerje]

@jnm2 Should SetName be deprecated? And have a comment added it to that warns about this kind of stuff, and pointing to the SetArgDisplayNames ? Ref documentation at https://github.com/nunit/docs/wiki/TestCaseData which doesnt mention SetArgDisplayNames, nor warns about this issue with SetName. I see there are multiple NUnit framework issues around the same thing.

LOL: You beat me to it :-)

[jnm2]

@OsirisTerje I wouldn't mind if you commented on that thread though :D

[MattKeenum]

@jnm2 @OsirisTerje Thanks for the workaround. Unfortunately we have some complex ways we build tests and the Set Names worked great since it gave full control over how the names were set. Any chance we can get that feature with SetArgsDisplayNames?

[jnm2]

@MattKeenum Maybe, can you explain more? SetArgDisplayNames only allows you to customize the display text of each parameter, but it uses the standard namespace, type name, method name, parentheses, and commas. Are some of those things you need to override?

[MattKeenum]

Edit: Just to be clear, I am only talking about how the Test Explorer displays the names of the tests. All of the other functionality works great.

@jnm2 Sure. Our company has over 500 applications that all need automated testing. Since we are mostly a C# shop, we chose NUnit as the testing framework of choice. As each team developed their own framework to write automated tests, we realized that every team was solving the same problems in different ways.

We wrote a generic wrapper around NUnit and distribute it as a NuGet package. This lets us solve the common problems once, and easily allows team members to move from team to team without having to learn a different framework.

Apart of this, we heavily customize the TestCaseSource to be able to create tests from different data sources, like Excel, CSV, Databases, etc. We then duplicate these tests changing slight configurations, like which environment to test, where the tests are executed (Local, Selenium Grid, Sauce Labs, etc.). If the team is using mobile devices, it allows the tests to be duplicated for different devices. This allows us to build once and run the tests we need to when approaching the CI/CD pipeline.

Unfortunately, I am prohibited by my company to upload pictures, but here are some example test names:

[Local : Dev : Win : Chrome] - [Card] - [MyMethod1] - [MyDataSourceTestName]
[Grid : Dev : Win : Chrome] - [Card] - [MyMethod1] - [MyDataSourceTestName]
[Sauce Labs : QA : Mac : Safari] - [Card] - [MyMethod2] - [MyDataSourceTestName]

Having full control over how the names are presented, makes it much easier to look through the Test Explorer for the tests the end user wants to run.

I'm open to ideas. We aren't 100% sold on SetNames. Just realize that any changes we make to the way the Test Names are displayed in the Test Explorer, affects 100s of people using the framework. So we want to be consistent in how we display these names to reduce confusion among the teams.

[jnm2]

@MattKeenum That makes sense. Thanks for the detail. SetArgDisplayNames is by design not the way to go for your use case. We'll continue to look at what it's going to take to get SetNames interacting well with the prefilter.

[OsirisTerje]

@MattKeenum @turcunicusor There is now a beta version of the fix in https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.15.1-dev-01134 . Would appreciate if you checked it.

[OsirisTerje]

@jnm2 Will you add a new issue for the framework ? I think this one can be closed, right ?

[jnm2]

Should we transfer this issue there or start a new one?

[turcunicusor]

@OsirisTerje I checked the beta version fix. Now it behaves as expected. I want to mention that I am using SetUpFixture with OneTimeSetUp, and custom TestCaseSource. Thank's a lot!

[OsirisTerje]

@turcunicusor Thanks for the check-up :-) !

@jnm2 I think it would be good to start a new one. Then this one here will serve as documentation for the issue in the adapter. I've linked to this one in the release notes, the "top" item #651 and in the Tips and Tricks page https://github.com/nunit/docs/wiki/Tips-And-Tricks#prefilter

[OsirisTerje]

@MattKeenum 3.15.1 hotfix released now.

[MattKeenum]

@OsirisTerje I can confirm that the fix is working for 3.15.1. Thanks everyone!