Inheritance support for base classes that resides in different assemblies.

[smadala]

Description

Try to have derive test class in different assembly than base test class.

Steps to reproduce

Add a base test class TestClassBase in assembly1.dll
Add a derived test class TestClassDerived in assembly2.dll

Expected behavior

Test methods in TestClassDerived class should be discovered/run

Actual behavior

Test methods on TestClassDerived.cs class not discovered/run

Environment

Package version of MSTest - preview-1.1.4

[AbhitejJohn]

This could be another setting that we can expose in the "MSTestV2" section of the runsettings..

[pvlakshm]

See here for the ask on uservoice: https://visualstudio.uservoice.com/forums/330519-team-services/suggestions/6030736-support-test-inheritance-for-base-classes-in-differ
We should take this up for a future release.

[ajryan]

This looks like a relatively simple issue for a newcomer to pick up. I'd like to help out.

It looks like this TODO (lline 100 of testfx/src/Adapter/MSTest.CoreAdapter/Discovery/TypeEnumerator.cs) is the place to implement it.

Is MsTestSettings the right place to add the new setting? Would a good name be BaseClassTestMethodDiscovery with possible values DeclaringAssembly and AnyAssembly?

[AbhitejJohn]

@ajryan: That would be wonderful.

It looks like this TODO (lline 100 of testfx/src/Adapter/MSTest.CoreAdapter/Discovery/TypeEnumerator.cs) is the place to implement it.

Yes, that's exactly the place.

Is MsTestSettings the right place to add the new setting? Would a good name be BaseClassTestMethodDiscovery with possible values DeclaringAssembly and AnyAssembly?

Right again. We can name the setting EnableDerivedTypesFromOtherAssemblies and it would be a bool. We are only interested in the declaring assembly.

[ajryan]

Thanks for the quick response. I don't think your suggested name is correct, a "derived type" would be an inheriting class (which already works) but we're concerned with enabling test methods found in base/parent classes. Plus it's not a type that is/isn't being enabled, it's a test method.

Maybe one of these?

• EnableTestMethodsInBaseClassesFromOtherAssemblies
• EnableBaseClassTestMethodsFromOtherAssemblies

[AbhitejJohn]

ah that could be confusing. EnableBaseClassTestMethodsFromOtherAssemblies looks good to me.

[AbhitejJohn]

Thanks to @ajryan this is now fixed and would be available in the Sprint 116 nuget release.

[AbhitejJohn]

After speaking with @pvlakshm, reopening this to track making this the default behavior. This would involve the following as well:

1. perf validations.
2. doc update to call out this being a difference in behavior from MSTest V1.

[AbhitejJohn]

Removing up-for-grabs for now since this involves perf infrastructure to be setup.

[ajryan]

FWIW, I think it's highly unlikely that making this option default true will affect performance of the framework itself - the expensive reflection was already being performed. There's no new loops or branching.

Impact on users' performance experience will be dependent on whether they have "other assembly" base tests that become newly valid.

[AbhitejJohn]

@ajryan : That's actually a yes and a no. We actually get all runtime methods for a type and then determine if its a test method here. After this change however, we would be performing this reflection check for APIs in System.Object (at the minimum) as well. We don't really know the numbers for IsValidTestMethod as such. However, since we check for the presence of TestMethod attribute at the beginning of that method, this should ideally not cause much of an issue in performance.

[ajryan]

@AbhitejJohn Ahh - I see your point. We would short-circuit any other assembly method and skip checking for attributes. Perhaps when the "other assembly" option is enabled it would be more efficient to perform the attribute check first.

[AbhitejJohn]

While trying to get the perf numbers out to make this the default, I hit into the following:

1. The setting does not seem to get honored in the desktop workflows - The settings object does not get re-created in the discovery appdomain and hence is always the default(not what the user specified).
2. The tests defined in the base class show up under the External node in Test Explorer instead of the derived project. This is happening because the DIA source information logic is searching for these methods in the derived assembly and cannot find it. Need to figure out what should be done here since setting the source to the base class would start showing this test twice under the base project which is not the intention.

[AbhitejJohn]

Here are the top level discovery numbers I got from VS for a solution with 9 projects each having 10 test classes each with 100 tests:

With the config enabled: 4.67 s. (average of 5 runs)
Without the config enabled: 4.76 s. (average of 5 runs)

This is on a 16 Gb, 2 core machine.
Although the numbers should ideally be reversed based on our theory above, the point is that this does not affect performance and can be turned on by default.

I'm wary of doing so however due to the 2nd issue above. @ajryan , do you want to give a shot at fixing that? It is to do with the assembly full path we pass in here. If we pass in the (base)assembly where the test method is actually defined this might be fixed.

[ajryan]

sure, I'll take a look and come back with questions

[AbhitejJohn]

Sure. Here is little more context on how the 2nd part works: This code gets the line number and file name where a test is defined based on the source. We use DIA internally to populate this data(DIA in-turn works on pdb files). In our case, we need to be searching for the base test methods in the base assembly and not in the derived test assembly. After we make a change to set the appropriate source on a test element as I detailed above, we would also need to change this code to pass in the source from the test element so it gets the right data. Hope that makes sense.

[ajryan]

Thanks for the pointers, this makes sense. I think TestMethod should carry a DeclaringAssemblyName parallel to the way it exposes DeclaringClassFullName. I'll thread it through from the discovery end and then make sure the navigator can locate the source correctly.

[MikeChristensen]

Just tried this today and it works great! Thanks for the hard work, guys!

Few issues:

When you upgrade the NuGet package to the .17 version, you have to restart Visual Studio before you can run tests again. Otherwise you get some error about a constructor not being found. No biggie.

The inherited test methods in the base class must be public to get run; protected won't work. This probably isn't a huge deal, but perhaps should be noted somewhere.

[AbhitejJohn]

@ajryan: Yes, that should be great. There could be issues in figuring out the location of the assembly though, MSTest.CoreAdapter being a PCL. We might need to pass it through another API in IFileOperations.

@MikeChristensen:

When you upgrade the NuGet package to the .17 version, you have to restart Visual Studio before you can run tests again. Otherwise you get some error about a constructor not being found

This is an issue with a cache the Test Explorer maintains. It does not get refreshed - a bug we are tracking internally.

The inherited test methods in the base class must be public to get run; protected won't work.

The test method signature does not like protected methods today. It looks like we need to reconsider that.

[AbhitejJohn]

This is now part of 1.1.17 on nuget.org. This isn't the default workflow in this release though (due to the issue being discussed above) and can be enabled using the following settings:

<RunSettings>    
  <MSTest> 
    <EnableBaseClassTestMethodsFromOtherAssemblies>true</EnableBaseClassTestMethodsFromOtherAssemblies> 
  </MSTest> 
</RunSettings>

Happy unit-testing. :)

[ajryan]

About determining assembly location from TypeEnumerator in the PCL... I now understand your comment about about farming it out to PlatformServices/IFileOperations, I came to the same conclusion while I was working it through.

For the concrete implementations:

• The Desktop flavor has Assembly.Location available.
• For NetCore, it's currently targeting NetStandard1.3 -- if we can move up to NetStandard1.5, it will be available there. Not sure what the restrictions are.
• For UWP, it's simply not an option. Seems like this is OK since navigation session is not a thing here.

I am doing exploratory testing locally in VS2017 with nuget packages built with my changes. I think things are looking good with the base methods appearing under the derived class assembly, meanwhile navigating to source of the base methods locates the correct file+line.

One thing I noticed, somewhat related to the ClassInitialize discussion -- when executing an inherited remote assembly base method: ClassInitialize is called correctly on the base assembly class and a TestContext is passed, but WriteLine calls to that context are not captured as output of the base method.

You can see my progress so far here: ajryan@5c0d2b6

[AbhitejJohn]

That sounds great @ajryan . Looked through your changes as well - looks good.

if we can move up to NetStandard1.5, it will be available there.

We can surely move over to ns1.5. We just wanted to leave it at a minimal standard as a principle.
UWP unfortunately does not seem to have an API to get file path(By design). We can leave it as is.

ClassInitialize is called correctly on the base assembly class and a TestContext is passed, but WriteLine calls to that context are not captured as output of the base method.

Oh I see, this would be coming out in one of the test methods of the derived class. Will file an issue for this one.

I'm gonna re-open a new issue for this with all the data above - The issue in the title is actually already fixed..

[AbhitejJohn]

Pulled out the remaining items in this issue as separate issues - #163 and #164. Closing this one since it is fixed.