[WIP] Enable CoreFX tests on CoreRT #5287

A-And · 2018-01-25T00:31:40Z

Work on #5059. Enables running CoreFX tests on CoreRT. A couple of points

A number of tests have a dependency on the CoreFX.Private.TestUtilities assembly. We need a way to distribute this - as either a NuGet package or fetching built assemblies as correlation payloads from build machines. Any suggestions on this would be great.
No shell script changes - most of this change is cross-plat, but only called on Windows
Test URL - will change according to whichever version we pick, run and "freeze"
Some minor points marked with TODOs

jkotas · 2018-01-25T00:40:12Z

tests/TopN.CoreFX.issues.json

+[
+    "System.Collections.Tests",
+    "System.Collections.Immutable.Tests",
+    "System.Net.Security.Tests"


Why System.Net.Security.Tests ?

All three are placeholders. System.Net.Security.Tests has an explicit dependenency on CoreFX.Private.TestUitilities, so that's why I included it.
The actual tests need to be decided.

jkotas · 2018-01-25T00:42:41Z

tests/CoreFXTestListURL.txt

@@ -0,0 +1 @@
+https://dotnetbuilddrops.blob.core.windows.net/build-e0a70e19967c4fbfa9e6a9e7fed567cc/TestList.json?sv=2015-04-05&sr=c&sig=rwqOIDM585nrCg4ydSleaYi%2FN1%2BcFqEivySw06MiZRs%3D&st=2018-01-12T01%3A34%3A06.4065828Z&se=2018-02-11T01%3A34%3A06.4065828Z&sp=r


Just curious - is 2015-04-05 in the URL a date? It seems pretty old.

The build itself should be fairly new. "sv" should stand for signed version, but I'm not certain how that relates.

jkotas · 2018-01-25T00:44:12Z

We need a way to distribute this - as either a NuGet package or fetching built assemblies as correlation payloads from build machines.

How does the .NET Core Helix test runs get this? We should be using same mechanism, ideally.

jkotas · 2018-01-25T00:53:59Z

tests/CoreFX/runtest/CoreFXTestHarness/Test.csproj

+    <ItemGroup>
+        <!-- Some tests consist of multiple assemblies - make sure ILC sees them -->
+        <IlcCompileInput Include="$(MSBuildProjectDirectory)\*.dll" />
+        <!-- TODO Get both assemblies from somewhere else?  -->


It should be included here https://github.com/dotnet/corert/blob/master/src/Framework/Framework-uapaot.depproj#L26; or stop being excluded here https://github.com/dotnet/corert/blob/master/src/Framework/Framework.depproj#L42 .

A-And · 2018-01-25T00:59:32Z

@jkotas Helix builds CoreFX. The closest alternative we should have to this is either "freezing" a specific CoreFX build and fetching the dll from there or adding the assembly as a correlation payload to the built tests (as opposed to the built repo)

sergiy-k · 2018-01-25T04:33:23Z

tests/CoreFX/runtest/src/TestUtils/TestFileSetup/TestFileSetup.cs

@@ -0,0 +1,205 @@
+using System;


All source files must have a license header (like this one:

corert/samples/WebApi/Program.cs

Line 1 in 3781c96

// Licensed to the .NET Foundation under one or more agreements.

)

In addition, please add comments describing what this program does and what kind of arguments it takes.

sergiy-k · 2018-01-25T04:33:40Z

tests/CoreFX/runtest/src/TestUtils/XUnit/ResultFormatter.cs

@@ -0,0 +1,108 @@
+using System;


All source files must have a license header (like this one:

corert/samples/WebApi/Program.cs

Line 1 in 3781c96

// Licensed to the .NET Foundation under one or more agreements.

)

sergiy-k · 2018-01-25T04:34:44Z

tests/CoreFX/build-and-run-test.cmd

+echo Build %TestFileName%
+
+call "%CoreRT_CliDir%\dotnet.exe" publish %TestFolder%\Test.csproj /ConsoleLoggerParameters:ForceNoAlign "/p:IlcPath=%CoreRT_ToolchainDir%" "/p:DebugSymbols=false" "/p:Configuration=%CoreRT_BuildType%" "/p:FrameworkLibPath=%~dp0..\..\bin\%CoreRT_BuildOS%.%CoreRT_BuildArch%.%CoreRT_BuildType%\lib" "/p:FrameworkObjPath=%~dp0..\..\bin\obj\%CoreRT_BuildOS%.%CoreRT_BuildArch%.%CoreRT_BuildType%\Framework" /p:DisableFrameworkLibGeneration=true /p:TestRootDir=%~dp0  /p:ExecutableName=%TestExecutable%
+


Do we need to check that the build has succeeded before starting running the test?

MichalStrehovsky · 2018-01-25T11:45:27Z

tests/CoreFX/build-and-run-test.cmd

+if not exist %XunitLogDir% md %XunitLogDir%
+
+if not exist %TestFolder%\%TestExecutable%.exe ( 
+    :: Not a test we support yet, exit silently


What are these tests?

Tests not run by Helix (e.g. AssemblyResolve). Technically, this shouldn't ever happen, but in case the test format changes, this degrades more gracefully.
Ideally, the need for this will be completely eliminated once we add a CoreRT parallel to the uap-aot target for CoreFX.

MichalStrehovsky · 2018-01-25T13:33:05Z

tests/CoreFX/runtest/runtest.cmd

+if not exist %__LogsDir% md %__LogsDir%
+
+:: Check presence of VS
+if not defined VS%__VSProductVersion%COMNTOOLS goto NoVS


Is there a NoVS label anywhere?

MichalStrehovsky · 2018-01-25T13:34:12Z

tests/CoreFX/runtest/runtest.cmd

+::       The issue is that we extend the build with our own targets which
+::       means that that rebuilding cannot successfully delete the task
+::       assembly. 
+set __msbuildCommonArgs=/nologo /nodeReuse:false %__msbuildExtraArgs% /p:Platform=%__MSBuildBuildArch%


Is this used anywhere?

MichalStrehovsky · 2018-01-25T13:34:26Z

tests/CoreFX/runtest/runtest.cmd

+set __VSToolsRoot=!VS%__VSProductVersion%COMNTOOLS!
+if %__VSToolsRoot:~-1%==\ set "__VSToolsRoot=%__VSToolsRoot:~0,-1%"
+
+set _msbuildexe="%VSINSTALLDIR%\MSBuild\15.0\Bin\MSBuild.exe"


Is this used anywhere?

MichalStrehovsky · 2018-01-25T13:37:15Z

tests/runtest.cmd

+:RestoreCoreFXTests
+
+    :: Explicitly restore the test helper project
+    "%CoreRT_CliDir%\dotnet.exe" msbuild /t:Restore "%CoreRT_TestFileHelperProjectPath%"


Should we check error level after this too?

MichalStrehovsky · 2018-01-25T13:38:19Z

tests/runtest.cmd

+    )
+
+    :: Build the test helper project
+    "%CoreRT_CliDir%\dotnet.exe" msbuild /m /ConsoleLoggerParameters:ForceNoAlign "/p:IlcPath=%CoreRT_ToolchainDir%" "/p:Configuration=%CoreRT_BuildType%" "/p:OSGroup=%CoreRT_BuildOS%" "/p:Platform=%CoreRT_BuildArch%" "/p:RepoLocalBuild=true" "/p:FrameworkLibPath=%CoreRT_TestRoot%..\bin\%CoreRT_BuildOS%.%CoreRT_BuildArch%.%CoreRT_BuildType%\lib" "/p:FrameworkObjPath=%~dp0..\bin\obj\%CoreRT_BuildOS%.%CoreRT_BuildArch%.%CoreRT_BuildType%\Framework" "/p:OutputPath=%CoreRT_TestingUtilitiesOutputDir%" /p:RepoLocalBuild=true "%CoreRT_TestFileHelperProjectPath%" 


Check errorlevel here too?

MichalStrehovsky · 2018-01-30T13:55:00Z

tests/CoreFX/runtest/CoreFXTestHarness/Test.csproj

+
+    <PropertyGroup>
+        <!-- TODO - need a general way to specify version without relying on the test directory structure being intact, i.e. import dependencies from \tests\ -->
+        <ToolsDir>$(FrameworkLibPath)\..\tools\</ToolsDir>


The only place where ToolsDir is used seems unused?

jkotas reviewed Jan 25, 2018

View reviewed changes

sergiy-k reviewed Jan 25, 2018

View reviewed changes

MichalStrehovsky reviewed Jan 25, 2018

View reviewed changes

A-And force-pushed the FXTestingFramework branch from a187b80 to 00c532c Compare January 27, 2018 02:22

MichalStrehovsky reviewed Jan 30, 2018

View reviewed changes

A-And and others added 15 commits January 30, 2018 09:55

Generate Test projects - Intermittent checkin

1c4a957

Intermittent check-in running exe, tests need restoring

c5c3901

Add Test utitilites project

6f9bfab

End-to-End test build Windows

d81f654

Change structure

88a8d07

Fix RDXML and add reflectable types

4c5c1e2

Add json parsing and xml logs

ca4b774

End-to-end test run and display

16ec04b

Helper project style changes

1d7e420

Remove direct dll references add package dependencies

b4c1b7c

Stylistic changes + deps

d0a6b8a

Style fixup

ca7fb82

XML reader escape

067d611

Dependency and comment cleanup

f39867a

Respond to PR feedback

48d4acc

A-And force-pushed the FXTestingFramework branch 4 times, most recently from ba7c398 to 208e7da Compare January 30, 2018 19:42

Intermittent checkin - Build helper projects

d1cb32e

A-And force-pushed the FXTestingFramework branch from 208e7da to d1cb32e Compare January 30, 2018 19:45

A-And closed this Jan 30, 2018

A-And mentioned this pull request Feb 2, 2018

Enable CoreFX tests on CoreRT #5329

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Enable CoreFX tests on CoreRT #5287

[WIP] Enable CoreFX tests on CoreRT #5287

A-And commented Jan 25, 2018 •

edited

Loading

jkotas Jan 25, 2018

A-And Jan 25, 2018

jkotas Jan 25, 2018

A-And Jan 25, 2018

jkotas commented Jan 25, 2018 •

edited

Loading

jkotas Jan 25, 2018

A-And commented Jan 25, 2018

sergiy-k Jan 25, 2018

sergiy-k Jan 25, 2018

sergiy-k Jan 25, 2018

MichalStrehovsky Jan 25, 2018

A-And Jan 27, 2018

MichalStrehovsky Jan 25, 2018

A-And Jan 27, 2018

MichalStrehovsky Jan 25, 2018

MichalStrehovsky Jan 25, 2018

MichalStrehovsky Jan 25, 2018

A-And Jan 27, 2018

MichalStrehovsky Jan 25, 2018

MichalStrehovsky Jan 30, 2018

		@@ -0,0 +1 @@
		https://dotnetbuilddrops.blob.core.windows.net/build-e0a70e19967c4fbfa9e6a9e7fed567cc/TestList.json?sv=2015-04-05&sr=c&sig=rwqOIDM585nrCg4ydSleaYi%2FN1%2BcFqEivySw06MiZRs%3D&st=2018-01-12T01%3A34%3A06.4065828Z&se=2018-02-11T01%3A34%3A06.4065828Z&sp=r

		echo Build %TestFileName%

		call "%CoreRT_CliDir%\dotnet.exe" publish %TestFolder%\Test.csproj /ConsoleLoggerParameters:ForceNoAlign "/p:IlcPath=%CoreRT_ToolchainDir%" "/p:DebugSymbols=false" "/p:Configuration=%CoreRT_BuildType%" "/p:FrameworkLibPath=%~dp0..\..\bin\%CoreRT_BuildOS%.%CoreRT_BuildArch%.%CoreRT_BuildType%\lib" "/p:FrameworkObjPath=%~dp0..\..\bin\obj\%CoreRT_BuildOS%.%CoreRT_BuildArch%.%CoreRT_BuildType%\Framework" /p:DisableFrameworkLibGeneration=true /p:TestRootDir=%~dp0 /p:ExecutableName=%TestExecutable%

[WIP] Enable CoreFX tests on CoreRT #5287

[WIP] Enable CoreFX tests on CoreRT #5287

Conversation

A-And commented Jan 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkotas commented Jan 25, 2018 • edited Loading

Choose a reason for hiding this comment

A-And commented Jan 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

A-And commented Jan 25, 2018 •

edited

Loading

jkotas commented Jan 25, 2018 •

edited

Loading