Fix ILVerification.Tests project #5471

jcouv · 2018-03-01T05:31:20Z

The two source files were missing from the project
The tests are signed with Test.snk (this is set by F:\repos\corert\Tools\sign.targets), so the InternalsVisibleTo from ILVerification to ILVerification.Tests should use that public key
Restore failed without TargetFramework set to netstandard1.5 because Microsoft.TestPlatform.TestHost 15.3.0-preview-20170427-09 requires it
Missing Newtonsoft.Json reference added
There is still an issue with some test code not compiling because APIs are missing (Type.Assembly). Any ideas how to solve that?

jkotas · 2018-03-01T05:40:53Z

src/ILVerification/tests/ILVerification.Tests.csproj

@@ -3,16 +3,21 @@

  <PropertyGroup>
    <OutputType>Library</OutputType>
+    <TargetFramework>netstandard1.5</TargetFramework>


Does this need anything from 1.5 ? I would keep it at netstandard1.3.

I was getting errors when doing msbuild /t:Restore that Microsoft.TestPlatform.TestHost 15.3.0-preview-20170427-09 requires netstandard 1.5. I'll try again, to confirm.

Tried again and confirmed. That said, I'm not exactly sure how that package is getting pulled into the graph...

F:\repos\corert>msbuild /v:m /m "F:\repos\corert\src\ILVerification\tests\ILVerification.Tests.csproj" /t:Restore Microsoft (R) Build Engine version 15.6.82.30579 for .NET Framework Copyright (C) Microsoft Corporation. All rights reserved. Restoring packages for F:\repos\corert\src\ILVerification\tests\ILVerification.Tests.csproj... Restore completed in 56.61 ms for F:\repos\corert\src\ILVerification\src\ILVerification.csproj. F:\repos\corert\src\ILVerification\tests\ILVerification.Tests.csproj : error NU1202: Package Microsoft.TestPlatform.TestHost 15.3.0-preview-20170427-09 is not compatible with netstandard1.3 (.NETStandard,Version=v1.3). Package Microsoft.TestPlatform.TestHost 15.3.0-preview-20170427-09 supports: F:\repos\corert\src\ILVerification\tests\ILVerification.Tests.csproj : error NU1202: - net45 (.NETFramework,Version=v4.5) F:\repos\corert\src\ILVerification\tests\ILVerification.Tests.csproj : error NU1202: - netstandard1.5 (.NETStandard,Version=v1.5) Generating MSBuild file F:\repos\corert\bin/obj/Windows_NT.x64.Debug/ILVerification.Tests\ILVerification.Tests.csproj.nuget.g.props. Restore failed in 529.32 ms for F:\repos\corert\src\ILVerification\tests\ILVerification.Tests.csproj.

jkotas · 2018-03-01T05:41:47Z

Type.Assembly

Change it to Type.GetTypeInfo().Assembly ?

jcouv · 2018-03-01T05:51:19Z

Type.GetTypeInfo().Assembly worked. Thanks

jkotas · 2018-03-01T06:10:14Z

Do the tests run automatically during the build with this change?

jcouv · 2018-03-01T06:17:14Z

I think I can go back to netstandard1.3, by removing the Microsoft.NET.Test.Sdk reference. But even with that, I'm still not able to run the tests in TestExplorer :-(
It may be a problem specific to my box, because I'm also seeing this in Roslyn repo.

Would you know what is a netstandard1.3 equivalent for Assembly.Location by any chance?

No, this change would not yet automate the tests in the build. I would love that though.
That would need two things:

we need a way to compile IL to dll as part of a test. In Roslyn, we find and invoke ilasm. Can we assume that ilasm or something equivalent is present on all platforms in CI?
then we need to hook the test to CI. Who would know enough about the infrastructure (build and CI) to advise on that?

jkotas · 2018-03-01T06:23:06Z

we need a way to compile IL to dll as part of a test

This is the way to do it: https://github.com/dotnet/corert/tree/master/src/ILCompiler.TypeSystem/tests/ILTestAssembly

then we need to hook the test to CI

I think it can be hooked up same way as out TypeSystem tests, e.g.: https://github.com/dotnet/corert/blob/master/src/ILCompiler.TypeSystem/tests/TypeSystem.Tests.csproj . They just run during the build.

jcouv · 2018-03-01T06:29:26Z

Thanks.
Any idea for Assembly.Location in netstandard1.3? If not, I'll leave as netstandard1.5 for now.

jkotas · 2018-03-01T06:33:30Z

I am not sure. I think it is ok to leave it as 1.5.

jcouv · 2018-03-01T06:43:34Z

Ok. Good to merge them.
It would be great if someone could try to run the tests in TestExplorer after this PR.

jcouv · 2018-03-01T07:46:31Z

I took a look at ILTestAssembly. It's using an ilproj to compile some IL into one assembly.
The trouble is that ILVerification has multiple tests, some which involve one assembly and some which involve more than one (for example AccessTests).
So I was thinking of compiling IL as part of the test running.
It looks like Roslyn uses ilasm from package Microsoft.NETCore.ILAsm for this.

jkotas · 2018-03-01T07:58:19Z

The trouble is that ILVerification has multiple tests, some which involve one assembly

The low-tech way to deal with this is to create multiple .ilproj files.

It looks like Roslyn uses ilasm from package Microsoft.NETCore.ILAsm for this.

It is what the current CoreRT build does as well.

jcouv · 2018-03-01T08:50:35Z

I set up corert on a different machine, and it seems TestExplorer is still not working. That'll be for another day...

For the record, here's the TestExplorer log I captured (after removing the Microsoft.NET.Test.Sdk package reference, which was conflicting with the xunit runner package).

[3/1/2018 12:48:20 AM Diagnostic] About to Enqueue operation 'RunAllOperation', hashcode:27486230 
[3/1/2018 12:48:20 AM Diagnostic] Enqueue operation 'RunAllOperation', hashcode:27486230 
[3/1/2018 12:48:20 AM Diagnostic] Operation left in the the queue: 1
[3/1/2018 12:48:20 AM Diagnostic] 	'RunAllOperation', hashcode:27486230
[3/1/2018 12:48:20 AM Diagnostic] 

[3/1/2018 12:48:20 AM Diagnostic] Processing Queue .....
[3/1/2018 12:48:20 AM Diagnostic] Operation Dequeue : 'RunAllOperation'
[3/1/2018 12:48:20 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetStarted, operationInProgress=False
[3/1/2018 12:48:20 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=OperationSetStarted, InProgress=False
[3/1/2018 12:48:20 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=ChangeDetectionStarting, operationInProgress=True
[3/1/2018 12:48:20 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=ChangeDetectionStarting, InProgress=False
[3/1/2018 12:48:20 AM Diagnostic] We programmatically start the build in TestWindowHost.UpdateContainer... 
[3/1/2018 12:48:20 AM Diagnostic] After we await the build that programmatically started in TestWindowHost.UpdateContainer... 
[3/1/2018 12:48:20 AM Diagnostic] TestContainer update (build) complete : 192 ms
[3/1/2018 12:48:20 AM Diagnostic] Adding the Project in Project Map for Guid 56aa4730-39a4-4b48-95e9-89e8a29f0a06
[3/1/2018 12:48:20 AM Diagnostic] Adding the Project in Project Map for Guid 6166b258-3d41-4431-88d9-510faf5e6927
[3/1/2018 12:48:20 AM Diagnostic] Adding the Project in Project Map for Guid 894f0bc9-31d1-42d0-9c3f-4fc9a41cd07e
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://vsprojectoutputcontainerdiscoverer/v1, discovered 2 containers
[3/1/2018 12:48:21 AM Diagnostic] Containers from 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] 	C:\repos\corert\bin\Windows_NT.x64.Debug\ILVerification.Tests\ILVerification.Tests.dll:executor://vsprojectoutputcontainerdiscoverer/v1
[3/1/2018 12:48:21 AM Diagnostic] 	C:\repos\corert\bin\Windows_NT.x64.Debug\ILVerification\ILVerification.dll:executor://vsprojectoutputcontainerdiscoverer/v1
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://testexplorerservicecontainerdiscoverer/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.TestWindow.Extensibility.TestExplorerServiceContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://21a96f1d-b3c6-4cae-9e2f-f37f7c1c65d4/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.CMake.Project.TestExplorer.BoostTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://bebdf2c4-78db-4d12-9430-8eb2b1e116bd/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.CMake.Project.TestExplorer.BoostVsTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://cmakectest/1.0, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.CMake.Project.TestExplorer.CTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://e599a6e4-53cb-48eb-ba1e-5ef2ff29c179/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.CMake.Project.TestExplorer.GoogleVsContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://72b6a973-5936-4c9a-b000-a0df8d911b41/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.CMake.Project.TestExplorer.GoogleVsInBoxContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://a0019ece-853d-488f-9989-e4d98fa5f7ec/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.CMake.Project.TestExplorer.GoogleGitHubContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://cpsprojectoutputtestcontainerdiscoverer/v2, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.TestWindow.Extensions.CPSTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://orderedtestadapter/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.OrderedTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://generictestadapter/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.GenericTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] test container discoverer executor://webtestadapter/v1, discovered 0 containers
[3/1/2018 12:48:21 AM Diagnostic] No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.WebTestContainerDiscoverer' :
[3/1/2018 12:48:21 AM Diagnostic] DiscoveryOperation<RunAllOperation> FinishedChangedCotainers, changed container count is 0
[3/1/2018 12:48:21 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=ChangeDetectionFinished, operationInProgress=False
[3/1/2018 12:48:21 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=ChangeDetectionFinished, InProgress=False
[3/1/2018 12:48:21 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionStarting, operationInProgress=False
[3/1/2018 12:48:21 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionStarting, InProgress=False
[3/1/2018 12:48:21 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionStarted, operationInProgress=False
[3/1/2018 12:48:21 AM Informational] ------ Run test started ------
[3/1/2018 12:48:21 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionStarted, InProgress=False
[3/1/2018 12:48:21 AM Diagnostic] RunSettings Content:
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>C:\repos\corert\src\ILVerify\TestResults</ResultsDirectory>
    <SolutionDirectory>C:\repos\corert\src\ILVerify\</SolutionDirectory>
    <TargetPlatform>X86</TargetPlatform>
    <TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
  </RunConfiguration>
</RunSettings>
[3/1/2018 12:48:21 AM Warning] [xUnit.net 00:00:00.1197901] Skipping: ILVerification.Tests (Could not find/load any of the following assemblies: xunit.execution.desktop.dll)
[3/1/2018 12:48:21 AM Warning] No test is available in C:\repos\corert\bin\Windows_NT.x64.Debug\ILVerification.Tests\ILVerification.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[3/1/2018 12:48:21 AM Diagnostic] ***    Run finished using 'InMemoryUnitTestWriter' ***
[3/1/2018 12:48:21 AM Informational] ========== Run test finished: 0 run (0:00:00.5725716) ==========
[3/1/2018 12:48:21 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False
[3/1/2018 12:48:21 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False
[3/1/2018 12:48:21 AM Diagnostic] VirtualReadOnlyTestDataStore.OperationStateChanged State=TestExecutionFinished, operationInProgress=False
[3/1/2018 12:48:21 AM Diagnostic] TestDiscoveryStats.OperationStateChanged State=TestExecutionFinished, InProgress=False
[3/1/2018 12:48:21 AM Diagnostic] HIERARCHY: Index rebuild not needed (2 tests indexed)

MichalStrehovsky · 2018-03-01T09:42:30Z

I set up corert on a different machine, and it seems TestExplorer is still not working. That'll be for another day...

I'm not sure Test Explorer ever worked. We share a lot of the infrastructure with CoreFX repo through buildtools.

The way I used to run tests in the past is by setting the test project as the startup project and running it with F5 (as described here).

ArztSamuel · 2018-03-01T10:09:33Z

I'm not sure Test Explorer ever worked.

The test explorer did work for ILVerify tests, however I also wasn't able to get it to work for other projects like the TypeSystemTests.

I think it was mentioned that there were problems with getting the tests to work with the old csproj format, when the ILVerify tests were originally introduced in #3725. So it could be that there is a problem related to the changes in project format.

MichalStrehovsky · 2018-03-01T10:12:15Z

The test explorer did work for ILVerify tests, however I also wasn't able to get it to work for other projects like the TypeSystemTests.

Right, it only worked as long as ILVerify wasn't using any of the official build infrastructure. What I meant to say is "I'm not sure Test Explorer ever worked with the official unit testing and build tooling in the repo".

ArztSamuel · 2018-03-01T10:13:34Z

I'm not sure Test Explorer ever worked with the official unit testing and build tooling in the repo

I see. Yes, that is correct!

jcouv · 2018-03-01T18:16:25Z

Thanks for the clarification. I'll try via F5, and I'll use Jan's ilproj approach for now.

jcouv · 2018-03-01T18:42:13Z

Hum, F5 didn't work. I also tried it on ILCompiler.TypeSystem.
In both projects, I get a popup "A project with an Output Type of Class Library cannot be started directly". It looks like it's not hooked up to invoke xunit.

Do you have an example of a project that works with F5?

jcouv · 2018-03-01T19:17:22Z

@jkotas I tried adding an ilproj, but I'm getting "Project file is incomplete. Expected imports are missing." when trying to add the ilproj to the solution.
This seems the same issue Greg initially encountered.

jkotas · 2018-03-01T19:51:25Z

VS does not work well for editing the buildtools-based solution. You have to often do it manually in raw text editor.

jcouv · 2018-03-01T20:26:39Z

That's what I did. But in TypeSystem tests, the ilproj loads fine, while in ILVerify it fails.
The discussion I linked to said that this should work after #3682 was fixed (which it is).

* Use correct name to fetch embedded resource * Fix the test project * Use GetTypeInfo().Assembly

jcouv added 2 commits February 27, 2018 21:04

Use correct name to fetch embedded resource

c33fabc

Fix the test project

034d2c2

jcouv self-assigned this Mar 1, 2018

jkotas reviewed Mar 1, 2018

View reviewed changes

Use GetTypeInfo().Assembly

434514c

jkotas approved these changes Mar 1, 2018

View reviewed changes

jcouv mentioned this pull request Mar 1, 2018

[Umbrella] Refresh IL verification dotnet/roslyn#22872

Open

26 tasks

jkotas merged commit 3a7b082 into dotnet:master Mar 1, 2018

jcouv deleted the test-issue branch March 1, 2018 08:08

kbaladurin pushed a commit to kbaladurin/corert that referenced this pull request Mar 15, 2018

Fix ILVerification.Tests project (dotnet#5471)

865542c

* Use correct name to fetch embedded resource * Fix the test project * Use GetTypeInfo().Assembly

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ILVerification.Tests project #5471

Fix ILVerification.Tests project #5471

jcouv commented Mar 1, 2018 •

edited

Loading

jkotas Mar 1, 2018

jcouv Mar 1, 2018

jcouv Mar 1, 2018

jkotas Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

MichalStrehovsky commented Mar 1, 2018 •

edited

Loading

ArztSamuel commented Mar 1, 2018

MichalStrehovsky commented Mar 1, 2018

ArztSamuel commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

Fix ILVerification.Tests project #5471

Fix ILVerification.Tests project #5471

Conversation

jcouv commented Mar 1, 2018 • edited Loading

jkotas Mar 1, 2018

Choose a reason for hiding this comment

jcouv Mar 1, 2018

Choose a reason for hiding this comment

jcouv Mar 1, 2018

Choose a reason for hiding this comment

jkotas Mar 1, 2018

Choose a reason for hiding this comment

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

MichalStrehovsky commented Mar 1, 2018 • edited Loading

ArztSamuel commented Mar 1, 2018

MichalStrehovsky commented Mar 1, 2018

ArztSamuel commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018

jkotas commented Mar 1, 2018

jcouv commented Mar 1, 2018

jcouv commented Mar 1, 2018 •

edited

Loading

MichalStrehovsky commented Mar 1, 2018 •

edited

Loading