PTM not generating index file for HtmlTestResults : Fileserver Test suite 3.19.9.0

[damansaini]

I am not sure if this issue comes under your scope or not but sending you this email in hope that you might be aware of this issue or have some workaround for it.
I was successfully able to run latest Fileserver Test suite 3.19.9.0 but for some reasons, the PTM stopped generating index file under htmlresults folder.
Please share if you know any workaround or steps to fix this.
All other files are there.

[JiajunW]

Hi Daman Saini,

I have made a private build to help better diagnosing this issue. Please download HtmlTestLogger.zip. Extract the zip file, and you will get two files, Microsoft.Protocols.TestTools.Html.TestLogger.dll and Microsoft.Protocols.TestTools.Html.TestLogger.pdb. Go to C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\Extensions, replace the .dll and .pdb file with the new one.

Note: The above location is for Visual Studio 2019 community, if you are using a different version of Visual Studio, please modify the corresponding part.

Then, re-run the test cases in PTM. A debug file would be generated at C:\MicrosoftProtocolTests\LoggerDebug.txt, please upload that file so that I can check what is wrong with the index.html file.

Best Regards,
Windows Protocol Test Suites Development Team

[damansaini]

Log file attached. Below is the error in log file. If fixing this issue will take time then please see if you can provide any workaround for the time being,

C:\MicrosoftProtocolTests\FileServer\Server-Endpoint\3.19.9.0\HtmlTestResults\FileServer-2019-10-18-09-54-57\Html\BVT_ResilientHandle_LockSequence.html generated All test finished! Error when generating index.html at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at System.Collections.Generic.List1.AddRange(IEnumerable1 collection) at Microsoft.Protocols.TestTools.TxtToJSON.GetTestCaseCategortyList(List1 testCaseList) in C:\Users\jiajwa\Projects\ProtocolTestFramework\src\htmltestlogger\TxtToJSON.cs:line 251
at Microsoft.Protocols.TestTools.TxtToJSON.TestCasesString(String resultFolder, String captureFolder) in C:\Users\jiajwa\Projects\ProtocolTestFramework\src\htmltestlogger\TxtToJSON.cs:line 48
at Microsoft.Protocols.TestTools.HtmlTestLogger.ConstructListAndSummaryObj(TestRunCompleteEventArgs e) in C:\Users\jiajwa\Projects\ProtocolTestFramework\src\htmltestlogger\HtmlTestLogger.cs:line 343
at Microsoft.Protocols.TestTools.HtmlTestLogger.ConstructIndexHtml(TestRunCompleteEventArgs e) in C:\Users\jiajwa\Projects\ProtocolTestFramework\src\htmltestlogger\HtmlTestLogger.cs:line 369
at Microsoft.Protocols.TestTools.HtmlTestLogger.TestRunCompleteHandler(Object sender, TestRunCompleteEventArgs e) in C:\Users\jiajwa\Projects\ProtocolTestFramework\src\htmltestlogger\HtmlTestLogger.cs:line 167
`

LoggerDebug.txt

[JiajunW]

Hi Daman Saini,

I have come up with a possible fix. Please download HtmlTestLogger.zip, and follow the same steps above to re-run the test suite to see if it works.

Besides, I also want to understand your scenario, so that I can provide some other workaround. If you want to programmatically analyze the test results, you can export a test report as shown below.

Best Regards,
Windows Protocol Test Suites Development Team

[damansaini]

Hi JiaJun,

I tested the attached html logger and now its creating the index file.
As far as my requirement is concerned , we have a non windows SMB implementation and I am trying to run the latest protocol suite to find pass/fail/inconclusive tests so that our developers can fix the failures. Index file is important for me to share the results with them and for them to go through failure logs.

Thanks for the timely help. Will run it multiple times and incase see any issue, will report here.

Regards,
Daman

[JiajunW]

Hi Daman,

I believe the issue is caused by test categories not being generated correctly. The previous fix is kind of a workaround. If you check LoggerDebug.txt, there is a section where test categories is printed, like below:

==========================
Generated test categories:
BVT_AppInstanceId: BVT,Smb30,AppInstanceId
AppInstanceId_Negative_DifferentAppInstanceIdInReopen: Smb30,AppInstanceId,InvalidIdentifier
AppInstanceId_Smb311: Smb311,AppInstanceId,Positive
AppInstanceId_Encryption: Smb30,CombinedFeatureNonClusterRequired,Positive
……

If there is no category shown there, the problem still remains. Can you try HtmlTestLogger-CheckCategory.zip and send the log file to help us diagnosing this issue?

Thanks,
Windows Protocol Test Suites Development Team

[MattMcmullanQumulo]

We're also seeing this same issue. Additionally, the packet captures stopped being generated at the same time this issue showed up. Do you think that they'll have the same root cause, or shall we open another bug?

[JiajunW]

Hi @MattMcmullanQumulo

Can you try downloading HtmlTestLogger-CheckCategory.zip and following the steps I have mentioned above to see whether index.html is generated. And sharing C:\MicrosoftProtocolTests\LoggerDebug.txt would be helpful for us to diagnose the issue.

As for the packet capture issue, do you mean that you have enabled network capture in Protocol Test Manager but no capture files generated in C:\FileServerCaptureFileDirectory?

BTW, are those two issues happened after you installed a new version of test suite and PTM, or it just doesn't work at some time?

[MattMcmullanQumulo]

I had at some point seen the index.html and the packet capture working on this version of the software. I then noticed when trying to diagnose a failure that neither the index.html nor the packet capture were generated.

I have attached the debug log. Did your modified DLL contain a fix? When I ran a test this time, it generated both the index.html and the packet capture.

LoggerDebug.txt

[JiajunW]

Hi Matt,

The above dll contains a work around to fix the issue that index.html is not generated. However, from the log you attached, it seems this issue didn’t happen on your side. Categories are correctly detected.

[18:44:16.378] Method CloseFileTestCaseS0 is in class CloseFileTestCase, belongs to category Model,FSA,CloseFile,NonSmb,Positive
[18:44:16.378] Method CreateFileTestCaseS0 is in class CreateFileTestCase, belongs to category Model,FSA,CreateFile,NonSmb,Positive
……

Can you try to re-run, with both the original Microsoft.Protocols.TestTools.Html.TestLogger.dll and the new one, to see whether the issue can still repro. And send us the debug log if so.

Thanks,
Windows Protocol Test Suites Development Team

[Damudan]

I'm seeing the issue as well with vstest to test my dll generated with netcoreapp2.2.
At first the index.html is there, but no log files generated, figured out it's due to the test names contains special character colon( : ) , so I changed the test names which contains colon, then I see logs generated, but the index.html is gone.
Then from debugging, I see exception is from below:
Assembly assembly = Assembly.LoadFrom(dllFile);
Type[] types = assembly.GetTypes();
the assembly returns null, therefore types are null.

The exception is saying "Could not load file or assembly 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies".

Please advise how to fix/workaround it.

[Damudan]

@JiajunW any ideas how to fix it?

[JiajunW]

Hi @Damudan,

We run Windows protocol test suite on .NET Framework. While according to your description, you are using the html test logger with unit test on .NET Core. I am not sure whether the exception is caused by the different runtimes.

Can you share the steps for me to reproduce this issue? And what command did you use to run the test cases?

Thanks,
Windows Protocol Test Suites Development Team

[Damudan]

Hi @JiajunW

Sure thing.
I know my test names contains colon( : ) which causes issues on generating files, so then I did following:
1, Replaced the part in Htmltestlogger.cs: result.DisplayName = result.DisplayName.Replace(":", "-");
and then compiled and replaced the file Microsoft.Protocols.TestTools.Html.TestLogger.dll
2, Run my test from vs2019 command line: vstest.console.exe MyIntegration.dll /Logger:"Html;MyTestResults.html"
3, Check results, the logs and files generated under folders, but no index.html

4, By debugging, it looks the exceptions happens here in GetTestCaseCategories():
Assembly assembly = Assembly.LoadFrom(dllFile);
Type[] types = assembly.GetTypes();

The runtime exceptions throws...

[JiajunW]

Hi @Damudan,

Here is a quick workaround, you can have a try to check if it works.
Modify line 232 of TxtToJSON.cs from
Category = testCaseToCategory.ContainsKey(caseName) ? testCaseToCategory[caseName] : null,
to
Category = testCaseToCategory.ContainsKey(caseName) ? testCaseToCategory[caseName] : new List<string>(),

As for the runtime exception, HtmlTestLogger is currently used along with Windows Protocol Test Suite, which runs on .NET Framework, and we haven't tested it against .NET Core applications. We will investigate and try to solve this issue.

[Damudan]

@JiajunW, thanks a lot Jiajun.

Yes, the workaround works to some extent, and I can view the results with under some groups.

[JiajunW]

This issue has been fixed in commit microsoft/ProtocolTestFramework@b9c6a6b. Category info is now directly retrieved in TestResultHandler function instead of dll files.
You can get the latest code of Protocol Test Framework and build it yourself to get the html test logger.

@Damudan, for the special character in test name issue, as you already find a way to fix it, a pull request is welcomed to recognize your contribution. BTW, I am also curious how to put a special character in the test name. Currently we use the test name as part of the result filename. If the test name can be customized, we need to escape more illegal characters to make sure the result file can be created.

Regards,
Windows Protocol Test Suites Development Team

[Damudan]

Thanks @JiajunW

I tried to push but fails to login:
fatal: unable to access 'https://github.com/microsoft/ProtocolTestFramework.git/': The requested URL returned error: 403

It's an easy workaround though, in the ConvertToTestCase function, just replace the colon the result test case name may contain:
if (!string.IsNullOrEmpty(result.TestCase.DisplayName))
{
result.TestCase.DisplayName = result.TestCase.DisplayName.Replace(":", "-");
}
I see other issues with .NetCore, which is some test results are missing from the report, but it shows correctly in vsconsole, I don't have a chance to look into the details yet.

[JiajunW]

Hi @Damudan,

Can you confirm that all test categories are generated correctly with my latest fix?

Currently, the purpose of the Protocol Test Framework is to use with Windows Protocol Test Suite, and .NET Core is not our target. For Windows Protocol Test Suite, colon is not a valid character in our test suite.
However, contributions are always welcomed. To do so, you need to fork the Protocol Test Framework repo and open a pull request, instead of pushing to our repo (you will get a 403 error since you do not have the permission).

Regards,
Windows Protocol Test Suites Development Team

[Damudan]

@JiajunW , thanks I will try that.
Test categories still doesn't show data for me.

[JiajunW]

Hi @Damudan,

Do you mean that you have tried the latest code on staging branch of Protocol Test Framework but it doesn’t work?

I have added some debugging output to help diagnosing the issue. Please download HtmlTestLogger.zip. Extract the zip file, and you will get two files, Microsoft.Protocols.TestTools.Html.TestLogger.dll and Microsoft.Protocols.TestTools.Html.TestLogger.pdb. Go to C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\Extensions, replace the .dll and .pdb file with the new one.
Note: The above location is for Visual Studio 2019 community, if you are using a different version of Visual Studio, please modify the corresponding part.

Re-run your cases and check if the categories are generated in index.html. If categories are not generated, please upload the debug file C:\MicrosoftProtocolTests\LoggerDebug.txt.

Thanks,
Windows Protocol Test Suites Development Team

[Damudan]

@JiajunW It looks you didn't apply the fix for replace colon in the dll, anyway the log should be helpful as well, it shows path too long error and path format is not supported.
I have attached the log.
LoggerDebug.txt

Thanks
Kevin

[JiajunW]

Hi Kevin,

I didn’t apply your workaround because it won’t happen in our test suite cases. I want to focus on the category issue right now.

According to the log, it seems that you are using xUnit for unit test. However, Protocol Test Framework is designed for Windows Protocol Test Suites, which uses .NET Framework and MSTest. Currently we haven't supported test framework other than MSTest. When it comes to other test frameworks, categories are handled differently, which is out of our scope.

Thanks,
Windows Protocol Test Suites Development Team

[JiajunW]

Since this thread has been inactive for a long time, we close it for now. Feel free to reopen it if the problem remains.

Windows Protocol Test Suites Development Team