No tests found when running dotnet test with TC TestAdapter

[frederik-h]

I have some trouble with test projects on TeamCity for which the tests stopped executing at some point (TC update, dotnet SDK update?!). Unfortunately, this did not cause the dotnet build step to fail and hence it went unnoticed for quite some time. I have traced back the problems to the TeamCity.VSTest.TestAdapter.
As a minimal example with similar behavior, consider a project with the following .csproject file

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>net471;netcoreapp2.0</TargetFrameworks>
    <RestorePackagesPath>packages</RestorePackagesPath>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.8.0" />
    <PackageReference Include="NUnit" Version="3.10.01" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.10.00" />
    <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.10" />
  </ItemGroup>

</Project>

which contains a single source file (whose content is not really important, but included for completeness):

using NUnit.Framework;

namespace Example
{
    public class Tests
    {
        [Test]
        public void Test()
        {
            Assert.AreEqual(true, true);
        }
    }
}

If I run the tests with my local dotnet core SDK version 2.1.401, I get the expected results with both the netcoreapp2.0 and the net471 tests.

dotnet test --test-adapter-path:packages\teamcity.vstest.testadapter\1.0.10\build\_common\vstest15\ --logger:teamcity /p:TargetFramework=netcoreapp2.0
Build started, please wait...
Build completed.

Test run for C:\Users\frederik-h\tcbuildproblem\minimal-example\TestProject-With-TCAdapter\bin\Debug\netcoreapp2.0\Example.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
##teamcity[testSuiteStarted name='Example.dll' timestamp='2018-09-12T12:56:44.976+0000' flowId='f1c6ccda30164f138917f275df305d28']
##teamcity[testStarted name='Example.Tests.Test' captureStandardOutput='false' timestamp='2018-09-12T12:56:44.983+0000' flowId='f1c6ccda30164f138917f275df305d28']
##teamcity[testFinished name='Example.Tests.Test' duration='23' timestamp='2018-09-12T12:56:44.986+0000' flowId='f1c6ccda30164f138917f275df305d28']
##teamcity[testSuiteFinished name='Example.dll' timestamp='2018-09-12T12:56:44.987+0000' flowId='f1c6ccda30164f138917f275df305d28']

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.9936 Seconds

I get the same result if I run with /p:TargetFramework=net471.

Now, if I set the SDK version to 2.1.301 (the version on our build agents) with dotnet new globaljson --sdk-version 2.1.301, the netcoreapp2.0 tests still run with the same result.
But for net471 , I get:

dotnet test --test-adapter-path:packages\teamcity.vstest.testadapter\1.0.10\build\_common\vstest15\ --logger:teamcity /p:TargetFramework=net471
Build started, please wait...
Build completed.

Test run for C:\Users\frederik-h\tcbuildproblem\minimal-example\TestProject-With-TCAdapter\bin\Debug\net471\Example.dll(.NETFramework,Version=v4.7.1)
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test is available in C:\Users\frederik-h\tcbuildproblem\minimal-example\TestProject-With-TCAdapter\bin\Debug\net471\Example.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

If I set the test adapter path by --test-adapter-path:., the net471 build works as well.
But for my actual project, using this variant does not work at all, whereas the variant with the explicit paths shows the same behavior as described here.

[NikolayPianikov]

full .net vstest and .net core vstest have a difference how these tools find adapters in different versions of .net SDK.

But your project file contains package reference to <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.10" />. This package configures the test adapters path and the logger automatically. So try avoiding these arguments --test-adapter-path and --logger in your command line.

For the case under TeamCity the command line could look like dotnet test /p:TargetFramework=net471 or just dotnet test to run all tests.

For the case not under TeamCity you cold add an additional argument like/p:TEAMCITY_VERSION=2018 to notify TeamCity logger to be active this time:

dotnet test /p:TargetFramework=net471 /p:TEAMCITY_VERSION=2018

[frederik-h]

Thanks for your quick reply. The perhaps most important thing first:
Invoking the tests with /p:TEAMCITY_VERSION=2018 yields the desired result for both SDK versions. Thanks!

Some things that don't work:

1. Running dotnet test with the "command line" runner (with the TC adapter as a package reference in the project file) within TeamCity executes the tests but does not produce the output format expected by TC. Same result on my local machine.

2. Running dotnet test using the "command line" runner within TeamCity with the explicit --test-adapter-path (pointing to a path that contains the adapter DLL) and --logger arguments and without the Package Reference does not find the tests. (We have SDK 2.1.301 on our build agents.)

3. Running the tests with the ".NET CLI (dotnet)" runner and the "Command" "test" does not execute the tests (same behavior for "vstest"). But, if you look at the command line in the build logs, this does the same as in the previous step.

Perhaps the TEAMCITY_VERSION parameter could be documented more prominently? Of course, it would be even better, to find out why it is necessary and make the adapter behave as expected without it.

I would also expect the ".NET CLI (dotnet)" runner to fail if no tests were discovered (you don't add a test build step if you do not have any tests to run), but that should probably be discussed elsewhere.

[NikolayPianikov]

@frederik-h I would like to mention that the additional argument /p:TEAMCITY_VERSION=2018 is not required under TeamCity, TeamCity makes it automatically. For local test runs on your computer TeamCity output is hidden by design because of it does not suppose TeamCity integration. For the docker scenario there is a small trick.

[NikolayPianikov]

@frederik-h

Running dotnet test with the "command line" runner (with the TC adapter as a package reference in the project file) within TeamCity executes the tests but does not produce the output format expected by TC. Same result on my local machine.

Is it possible to share some screenshots and build log files?

Running dotnet test using the "command line" runner within TeamCity with the explicit --test-adapter-path (pointing to a path that contains the adapter DLL) and --logger arguments and without the Package Reference does not find the tests. (We have SDK 2.1.301 on our build agents.)

Unfortunately --test-adapter-path does not allow to use several paths, but it is required because vstest should find the specific test adapter and TeamCity adapter. But there is an environment variable VSTestTestAdapterPath. It allows to combine several paths using ; (begining from the some version of .net SDK).

Running the tests with the ".NET CLI (dotnet)" runner and the "Command" "test" does not execute the tests (same behavior for "vstest"). But, if you look at the command line in the build logs, this does the same as in the previous step.

Could you add details?

Could you describe your scenario in details? I mean:

• TeamCity version
• dotnet CLI plugin version if it was installed manually
• OS and version on the TeamCity agent
• .net SDK version (dotnet --info)
• samples in zip files or repo to reproduce
• scnreenshots of your build steps and build log files

⚠️ If there is some confidential information just create an issue here and add comments/artifacts for JetBrains team only.

[frederik-h]

@NikolayPianikov It is not clear to me for which runner this should work? If I use the ".NET CLI (dotnet)" runner, it does not work for me. It works if I replace this build step by a command line build step which invokes dotnet test manually with the version parameter. By the way, the version line of the UI says that we are running "TeamCity Enterprise 2018.1.1 (build 58406)".

What you said regarding local test runs is also not consistent with the animated gif at the project's github page which shows dotnet test being invoked without the version parameter producing the service messages.

[NikolayPianikov]

@frederik-h Unfortunately vstest has this issue, but the vstest team does not want to fix it. May be it is this issue. Thus you could update your TeamCity to the latest version because we have tried to apply some workarounds here. Or just manually set the verbosity level to normal for test step in the dotnet CLI plugin or in the command line runner.

[NikolayPianikov]

Also we have the related issue here

[frederik-h]

@NikolayPianikov It is probably not related to those issues since the output of the dotnet test command in the build log of the dotnet cli build step explicitly says that "No test is available in ..." (for target framework net462, with SDK 2.1.301). I cannot update anything on the build agents, but I have several workarounds that I can use until we get updated TC and dotnet SDK versions. And the same behavior with my minimal test project that I described above goes aways with SDK 2.1.401. Hence, I can use the workarounds for now and wait for the updates to be installed on our agents.

[frederik-h]

To me, issues #10 and #11 seem to be related to this issue.

[ghost]

In scenarios where you have mixed test projects, eg XUnit and Nunit, we had added the Logger console and verbosity commands to make the Xunit output visible and were confused why the NUnit output would not appear when we added the Adapter nuget package.... The gold part from the above threads is you can't just add it to one project if you are providing the logger command... The correct way is to stop overriding the logger for dotnet test and just add the VSTest.TestAdapter to all projects .. works like a charm