Support multiple test projects in a solution/workspace.

[erikaellr]

I really appreciate this extension but it would be even more useful if it supported multiple test projects per solution/workspace in settings.json like:

{
"dotnet-test-explorer.testProjectPaths": ["path/to/some/test/proj", "path/to/other/test/proj"]
}

[paulinka129]

It would be great!

[grankko]

I came here for this..

[AlexTeixeira]

Very good idea.
Better if the plugin be able to discover project inside a folder

[krawuzikapuzi]

I'd love to have this feature too, however since my solution has several test projects, I'm not fond of providing a list of all test projects. Instead, I have a naming convention for all test projects "Some.Thing.Tests". It would be great if it could support a regular expression or similar to show projects matching the expression inside the currently opened folder and allow execution of those tests.

[christophla]

Instead of rewriting the test explorer tree, it may be easier to display tests based on the project that is focused (via menu in footer) in OmniSharp. You could then set the project root as the working directory for the dotnet test -t -v=q command.

Currently, it appears to use the working directory for the first project in the .workspace file.

[davidparker]

thumbs up for this one, I just installed this plugin, and realised it wont actually work as I work with a workspace and have multiple test projects for my dependant projects. This is not based on solution file, they are individual projects on VSTS that I nuget and then have separate, standalone test projects for. for now its a deal breaker :(

[stefanforsberg]

@christophla That sounds somewhat tempting but do you think it will be good enough? I've moved away from having multiple test projects personally but from what I remember I usually executed more than one at a time which would not be possible with the workspace root solution no?

[christophla]

@stefanforsberg When using workspaces, you may, for example, have multiple microservice solutions open, each with its own unit and integration test projects. It's not realistic to combine them into a single test project - especially if they are each coming from different repos.

I currently use an orchestration repo for running tests across all workspace projects that is called by CI/CD. The normal workflow, though, is to focus on a single project with OmniSharp and develop against the other staged microservices (docker/minikube). Having the test panel also focus on this project would be very beneficial.

[stefanforsberg]

@christophla I see, that is basically like my current workflow but I've used one vscode window per service. But I can definitely see the merit in running one instance of the editor and adding multiple workspaces.

But you mention that each solution tend to have a unit and a integration test project, so that would mean that you'd need to two manage at least two test projects in one workspace?

[stefanforsberg]

@christophla Maybe these are two different feature requests for two different workflows.

1. We have people who are developing smaller services that are connected using multiple workspaces where each workspace has it's own set of tests. For this workflow it would be beneficial that the extension used the currently active workspace as it's base for test discovery and test running.

2. We have people developing more of a classic bigger .net "monolith" with a solution file containing a number of test projects. For this workflow it would be helpful to be able to discover/run tests from all test projects at the same time.

Or am I completely of base here?

[christophla]

@stefanforsberg You are correct.

It's probably best to approach the multiple test project scenario first and then approach the workspace (multiple solution project) issue as a separate feature. I would guess that the monolith issue would both block and feed into the bigger workspace issue.

We often have several solution projects per-workspace, each with several test-projects. We dynamically generate the workspace via CLI for faceted views of a much larger set of microservices. :-)

[stefanforsberg]

Just a small update. I've been working on a bit of restructuring to be able to handle multiple test project per solution / workspace.

Instead of taking a path dotnet-test-explorer.testProjectPath takes a glob that points to the directories that contains tests. So if we were to use our own test directory as example the glob pattern "+(mstest|nunit|xunittests)" will find all those test directories and then will perform the usual test discovery of those directories in parallel. The log looks something like this:

Starting extension
Finding projects for pattern d:/dev\vscode-dotnet-test-explorer\test/+(mstest|nunit|xunittests)
Found 3 matches for pattern
Adding directory d:/dev/vscode-dotnet-test-explorer/test/mstest
Adding directory d:/dev/vscode-dotnet-test-explorer/test/nunit
Adding directory d:/dev/vscode-dotnet-test-explorer/test/xunittests
Executing dotnet test -t -v=q in d:/dev/vscode-dotnet-test-explorer/test/mstest
Executing dotnet test -t -v=q in d:/dev/vscode-dotnet-test-explorer/test/nunit
Executing dotnet test -t -v=q in d:/dev/vscode-dotnet-test-explorer/test/xunittests

Actual execution of the test are handled in a similar fashion, they are executed in parallel and when they are all finished the tree and code lens and problem views are updated as they are today. The big difference against today is that the test are not executed in the terminal but rather behind the scenes (like test discovery is handled).

Since I'm not actively using any project with multiple test projects I'm keen for someone else to try this version and see if it's something that will be usable at all. Any takers?

[AlexTeixeira]

Hi,

I have multiple project with complex test tree. So in can test a preview version in a macOs system.

And I found a thread about dotnet test command that talk about discovering multi project inside a directory : microsoft/vstest#705

[stefanforsberg]

@AlexTeixeira Cheers!

Unpack the file below and install the .vsix in vscode.

dotnet-test-explorer-0.3.1.zip

[TimBarley]

@stefanforsberg Just installed the preview extension and it worked for me without any initial issues.

[teamdynamiq]

@TimBarley: Ok great. Out of interest, how many test projects were there in your solution/workspace and about how many tests?

[qubicaamf]

@teamdynamiq, in our case we have at least 2 test projects per solution: Unit Tests, IntegrationTests

[TimBarley]

@TimBarley: Ok great. Out of interest, how many test projects were there in your solution/workspace and about how many tests?

@teamdynamiq There are 6 total projects. 2 of them are test projects.

_{Sent with GitHawk}

[AlexTeixeira]

Hello,

Sorry for the delay, I have been working only on front-end tasks.

For my side, my architecture seems to be quite more complexe and do not macth any of my tests.

This is my project architecture :

All my tests are under the test folder.
For this project, I have for class library projects and any one is detected by the plugin.

Should I configure something in my settings ?

[stefanforsberg]

@AlexTeixeira You would need to enter a glob that matches the test projects you want added in the testProjectPath setting.

[AlexTeixeira]

Back from some test :

I set as project path : _"dotnet-test-explorer.testProjectPath": "./test//*"_**

But when I reload the tree view I have :

1. Any tests in the tree view
2. CPU go to 100% for few seconds to one minutes, and some dotnet process runs at same time ( for my case 3 at same time)

[stefanforsberg]

@AlexTeixeira

If you look at the Output/Test Explorer terminal you can see which folders it tries to discover tests in. Currently it tries to do this in parallel so if you've specified an unlucky glob and/or have many test project you're bound to end up with much cpu-usage. Vsocde it's not really built for that kind of processing.

Just out of interest, how many folder can you see it's trying to discover tests in? You should see something like this in the log

Starting extension
Finding projects for pattern d:/dev\vscode-dotnet-test-explorer\test/+(mstest|nunit|xunittests)
Found 3 matches for pattern

I'm afraid that until MS provides us with multiple test projects supports built in (as a first class citizen, not something that uses undocumented things and requires a serious of burning hoops you'd have to jump through) we're just not getting there. #83 might be such a thing.

[AlexTeixeira]

@stefanforsberg .

I found the good glob pattern.
If anyone have something similar to me, this is the pattern that match my tests :

test/**/*.*[Test]/

The build and test execution seems to not take to much CPU and time.
Few seconds and 44% CPU Usage.

[AlexTeixeira]

Hi,

Just a question. This feature is included into the master branch ? It seems that is not working anymore with the update 4.0 :(

[stefanforsberg]

Hey @AlexTeixeira, this feature is not in the master branch.

[AlexTeixeira]

Hey @stefanforsberg. Ok, I will back to 0.3.1 with the multi workspace feature :)

[stefanforsberg]

@AlexTeixeira Fair enough =)

Just to be clear, that feature is not in the 0.3.1 that was released publically, it's only in the zip file linked above.

[garfbradaz]

@stefanforsberg: Any news on when this will be released into master?

FYI, I love this extension btw.

[stefanforsberg]

@garfbradaz I still have not found a way to implement this feature that feels right. Right now I’m on vacation but I will continue to work on a solution later in august. I am well aware that this is a feature many people are waiting for. But right now it’s time for another glass of sangria 💖

[garfbradaz]

Fair enough @stefanforsberg enjoy your hols :)

FYI, I tried using your VSIX mentioned above, and I get the following error:

Starting extension
Finding projects for pattern c:/code\a2/tests***[test].csproj
Found 2 matches for pattern
Evaluating match c:/code/a2/tests/xunit-integration/xunit-integration-tests.csproj
Adding directory c:/code/a2/tests/xunit-integration/xunit-integration-tests.csproj
Evaluating match c:/code/a2/tests/XUnit/apitests.csproj
Adding directory c:/code/a2/tests/XUnit/apitests.csproj
Executing dotnet test -t -v=q in c:/code/a2/tests/xunit-integration/xunit-integration-tests.csproj
Executing dotnet test -t -v=q in c:/code/a2/tests/XUnit/apitests.csproj
[ERROR] Error while executing dotnet test -t -v=q - Error: spawn C:\WINDOWS\system32\cmd.exe ENOENT
at exports._errnoException (util.js:1050:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
at onErrorNT (internal/child_process.js:367:16)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)
[ERROR] Error while executing dotnet test -t -v=q - Error: spawn C:\WINDOWS\system32\cmd.exe ENOENT
at exports._errnoException (util.js:1050:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
at onErrorNT (internal/child_process.js:367:16)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)

Checked and C:\WINDOWS\system32 is included in my PATH environment variable (WIndows 10 machine).

[Neutrino-Sunset]

I've read through this issue and every issue it links to, but I don't understand how to get Test Explorer to discover my test directories. I have a workspace with a project WebServiceTests. If I set testProjectPath to ./WebServiceTests it works. But obviously I have multiple workspaces with tests in different directories and testProjectPath is a global user setting which won't work for multiple projects and will not work for anyone else who checks out my project, since they won't have that user setting.

How am I supposed to configure this? There is plenty of talk her, but nothing in the project doumentation page.

[stefanforsberg]

@Neutrino-Sunset Currently we do not support multiple workspaces or test projects. We could be clearar on this in the known issues though, thanks for feedback.

[Neutrino-Sunset]

Are you saying it is necessary to manually set testProjectPath to the name of whichever test project you are currently working on? It doesn't accept wildcards or anything?

[stefanforsberg]

It has to point to a specific test project

[TimBarley]

I've been using the VSIX, from this thread, for about two months now and I've not had any issues.
My current directory structure is:
Project.One
Project.One.Tests
Project.Two
Project.Two.Tests
...
There are a total of 6 projects with 4 of them having corresponding test projects. Within my workspace settings (I have nothing in user settings), I'm using: "dotnet-test-explorer.testProjectPath": "./*.Tests" and it has no problem finding (and running) all of my tests.
The only 'issue' I ran into was when VSCode updated the extension to v4... Took me 30 minutes to realize what happen. :-)

[Edit]
Right after I posted this, I saw the v5 update. Updated and everything works without any modification to my setup.

Thanks again for this extension and this update!

[stefanforsberg]

This should be fixed with the v05 update. Please open new issues if something is not working as expected. Thanks for your patience 🌈

[garfbradaz]

@stefanforsberg Just back from DIY hols 🛠

Just seen the update, will have a goosey now and a good test. Top marks for smashing this one out.