ReportPortal.addins integration with TestCentric.GuiRunner.1.3.2

[eddie101AMR]

Hi,
I'm trying to run tests via TestCentric.GuiRunner.1.3.2, but I can't see the results uploaded to ReportPortal, I added ReportPortal.addins to folder in packages folder,
the same thing I did with NUnit.ConsoleRunner.3.11.1 and it's working fine when I run my tests via Nunit console runner,
what I'm missing?
I need to run tests of my dll tests file from some nunit gui runner?
any ideas ?

Thanks

[nvborisenko]

@eddie101AMR please help me to see internal logs of TestCentric, seems it's not trivial.

[eddie101AMR]

@eddie101AMR please help me to see internal logs of TestCentric, seems it's not trivial.

Which logs should I upload?

[nvborisenko]

Any available logs. You are the first who is using this tool to execute nunit tests.

I just tried to get TestCentric logs quickly, but unsuccessful. I suggest to ask help from TestCentric team.

[eddie101AMR]

I also tried to get TestCentric logs but didn't find any, I'll try to ask TestCentric team if they support ReportPortal addins.

[CharliePoole]

@nvborisenko See Eddie's issue report for TestCentric GUI: TestCentric/testcentric-gui#558.

Do you have a special reason why you want the extension to be copied to the output directory? All the NUnit extensions are in the tools directory of the package and so are not copied. I ask because the solution is probably different if you actually require the extensions to be copied.

I think this problem arises because neither NUnit nor TestCentric publishes requirements for 3rd party extension packages to be found automatically. By following some (undocumented) conventions, both the NUnit engine and the TestCentric engine are able to find any extensions without your user needing to create an addins file. Although the TestCentric engine has deviated from the NUnit engine in some ways I take it as a requirement that NUnit extensions should still be loadable. If @eddie101AMR runs the experiments I listed in estcentric/testcentric-gui#558 we'll know more. If what I believe is the problem turns out to be correct, I can give you modified instructions so that TestCentric will work with your current package.

[nvborisenko]

Shortly this extension should be used per test project. This extension publishes report to some external server. Imagine I have:

• NUnit runner (extension doesn't know where runner is located)
• Test project A
• Test project B

As a user, I want to publish results only for A, and not for B. This is why I chose nuget as a platform for shipping the extension. User installs nuget package to test project which he is aware of reporting.

One more requirement for the extension: extension should be configurable, per test project. This is why package copies dlls (with dependencies) to output directory, where it expects configuration. Output directory guarantees that extension is per test project. I don't want to use physical local location of nuget package, because of this location is shared by test projects. And, what is more important, nuget location is available at test project build phase. I want to support scenario when user builds test project on one machine, put artifacts (actual tests) to some other, and execute tests on that other machine (where .NET SDK is not installed).

Related issue about locating addins: nunit/nunit-console#488

[CharliePoole]

@nvborisenko I think you may misunderstand my question. Let me try to be clearer.

Take a typical NUnit extension nuget package, such as NUnit.Extension.NUnitV2Driver. It is installed into the user packages directory for a particular project. It is not shared by other projects as you suggest. The NUnit.ConsoleRunner package already comes with a .addins file, which will locate that extension and any other nuget package extensions that follow the naming convention of NUnit.Extension.* and are installed in the tools directory. Nothing else is needed.

Of course, ReportPortal doesn't follow the convention so you instruct users to create a .addins file on their own. This works at the moment but will need to be changed once the NUnit Project moves it's agents to different directories - I'm only assuming they will, of course, but it seems likely when there are more and more agents.

So, in questioning, I'm trying to find out whether you require the extension to be in the output directory because it's actually more convenient in some ways for it not to be copied there. In your answer, the thing I see that seems to point this way is related to configuration. When I designed the extension facility, my idea was that the extensions would be configured externally, by the runner in most cases. It does make a lot of sense to have an extension self-configure based on a file. It's too bad none of this discussion was ever held before! We might have provided that. 😢

So, it sounds like this needs to be solved in a way that doesn't ask you to change anything. I'll wait to hear back from @eddie101AMR and then produce some instructions that ought to work.

[CharliePoole]

One question, just to be clear... When you copy and run the tests on another machine, does that machine have a central installation of the test runner?

It's possible to have a central installation and turn your extension on or off from the command-line but not so easy if you want a central installation with multiple configurations.

[nvborisenko]

It is installed into the user packages directory for a particular project.

It's true for old projects, new PackageReference csproj style uses global nuget cache which is shared for all projects.

When you copy and run the tests on another machine, does that machine have a central installation of the test runner?

Cannot answer, I even don't know which runner users use to execute tests and how they install it. ReportPortal tries to cover as much as possible, nunit-console, dotnet test, dotnet vstest, vstest-console, VS Test Explorer with minimal pain. Now only one additional action is required in case if user uses nunit-console: create file in the location where console runner is located. If we remember that .net core is supported only by vstest, and RP wants to support .net core, it's clear why RP lives in tests output folder. I don't speak about a case when global/local nuget cache has several versions of addin.

[CharliePoole]

I'll have to experiment with the new PackageReference. So far, it has not caused me a problem. All packages appear to be located at the relative path specified by the .addins file. Where a problem would arise is if one of the two packages (runner and extension) is in the global cache and the other is not.

Anyway, I see now why it has to be done the way you did it and once @eddie101AMR gets back to me I think I can give you some modified instructions for use with the GUI. If you want to try it without waiting, here is my theory... I think your approach works when running in process. When running out of process, however, you would need an addins file in each target agent directory, with an appropriate relative path.

[nvborisenko]

Closing it. @eddie101AMR fell free to reopen if the issue is still not resolved.