[XHarness] Add support for xunit and nunit tests on Mac OS X. #5182

mandel-macaque · 2018-11-27T01:32:20Z

This branch adds support to execute the tests provided by mono as assemblies. The PR includes:

App generation to run the bcl tests on modern and full.
Needed workaround to delay the compilation of the assemblies until we have them from the SDK.
Some test fail. THIS IS ON PURPOSE. I want to show that we do catch errors.

…ded. This is a workaround that does not use reflection to build the bcl test projects. We need this until we have complete support of using mono as an SDK.

…mono test assemblies.

…n-macios into dont-compile-xunit

monojenkins · 2018-11-27T01:35:06Z

Build failure
❌ Build was aborted

monojenkins · 2018-11-27T02:55:36Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

tests/bcl-test/BCLTests/BCLTests-mac.csproj.in

tests/bcl-test/BCLTests/Info-mac.plist.in

tests/bcl-test/BCLTests/templates/macOS/MacTestMain.cs

tools/bcl-test-importer/BCLTestImporter/BCLTestProjectGenerator.cs

1. Addresses the reviews from rolf. 2. Ensure that the dependencies are ran on mac test projects. 3. Group the tests for ios and mac using their name.

mandel-macaque · 2018-11-27T12:12:53Z

@spouliot @rolfbjarne the tests should build and some of them fail (I want to see that we do report errors correctly). Once that is done, I'll exclude the failing ones as we did with ios/tvos/watchos.

spouliot

Why the renames from MONOTOUCH* to monotouch* ? it's rather noisy and not related to XM support. Also, and more important, how the name format for the mono SDK .zip file ?

mandel-macaque · 2018-11-27T14:41:03Z

@spouliot the rename was done precisely due to that. I was told by @luhenry that should not be upper case, as far as I can remember.

monojenkins · 2018-11-27T23:44:43Z

Build failure
❌ Build was aborted

✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

# Test run in progress: Built: 177, Running: 1, RunQueued: 40, Succeeded: 80, Failed: 15, Ignored: 1570, BuildFailure: 12

Failed tests

[NUnit] Mono MonoCSharpTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono MonoDataSqilteTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono MonoPoxisTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemConfigurationTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemCoreTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemDataTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemIOCompressionFileSystemTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemIOCompressionTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemIdentityModelTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemSecurityTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemServiceModelTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemTransactionsTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemXmlTests/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemTests/Mac/Full: Failed (Test run crashed (exit code: 255).)
[xUnit] Mono SystemCoreXunit/Mac/Full: Failed (Test run failed.)
[NUnit] Mono SystemJsonMicrosoftTests/iOS Unified 32-bits - simulator/Debug: BuildFailure
[NUnit] Mono MonoParallelTests/iOS Unified 32-bits - simulator/Debug: BuildFailure
[NUnit] Mono MonoTaskletsTests/iOS Unified 32-bits - simulator/Debug: BuildFailure
[NUnit] Mono SystemThreadingTasksDataflowTests/iOS Unified 32-bits - simulator/Debug: BuildFailure
[NUnit] Mono SystemJsonMicrosoftTests/iOS Unified 64-bits - simulator/Debug: BuildFailure
[NUnit] Mono MonoParallelTests/iOS Unified 64-bits - simulator/Debug: BuildFailure
[NUnit] Mono MonoTaskletsTests/iOS Unified 64-bits - simulator/Debug: BuildFailure
[NUnit] Mono SystemThreadingTasksDataflowTests/iOS Unified 64-bits - simulator/Debug: BuildFailure
[NUnit] Mono SystemJsonMicrosoftTests/watchOS - simulator/Debug: BuildFailure
[NUnit] Mono MonoParallelTests/watchOS - simulator/Debug: BuildFailure
[NUnit] Mono MonoTaskletsTests/watchOS - simulator/Debug: BuildFailure
[NUnit] Mono SystemThreadingTasksDataflowTests/watchOS - simulator/Debug: BuildFailure

monojenkins · 2018-11-28T07:29:51Z

Build failure
❌ Build was aborted

✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)

tests/bcl-test/BCLTests/templates/macOS/MacTestMain.cs

mandel-macaque · 2018-11-30T17:50:15Z

Do not merge until #4596 is done.

monojenkins · 2018-12-01T03:42:26Z

Build failure
❌ Build was aborted

monojenkins · 2018-12-01T03:44:54Z

Build failure
❌ Build was aborted

rolfbjarne · 2018-12-03T06:27:29Z

build

monojenkins · 2018-12-03T11:09:17Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

# Test run in progress: Built: 137, Succeeded: 142, Ignored: 1570, TimedOut: 10

Failed tests

mscorlib/iOS Unified 32-bits - simulator/Debug: TimedOut
System/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Core/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Data/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Net.Http/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Numerics/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Runtime.Serialization/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Transactions/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Web.Services/iOS Unified 32-bits - simulator/Debug: TimedOut
System.Xml/iOS Unified 32-bits - simulator/Debug: TimedOut

monojenkins · 2018-12-05T16:50:04Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

8 tests failed, 0 tests skipped, 281 tests passed.

Failed tests

[xUnit] Mono SystemJsonXunit/iOS Unified 32-bits - simulator/Debug: Crashed
[xUnit] Mono SystemNumericsXunit/iOS Unified 32-bits - simulator/Debug: Crashed
[xUnit] Mono SystemJsonXunit/iOS Unified 64-bits - simulator/Debug: Crashed
[xUnit] Mono SystemNumericsXunit/iOS Unified 64-bits - simulator/Debug: Crashed
[NUnit] Mono SystemNumericsTests/watchOS - simulator/Debug: Crashed
[NUnit] Mono SystemXmlLinqTests/watchOS - simulator/Debug: Crashed
[NUnit] Mono SystemJsonTests/watchOS - simulator/Debug: Crashed
[xUnit] Mono SystemJsonXunit/watchOS - simulator/Debug: Crashed

mandel-macaque · 2018-12-07T10:50:28Z

Mono 2018-08 got merged, if tests are ok, we should be able to merge this. Any comments against it @rolfbjarne @spouliot ?

monojenkins · 2018-12-07T11:32:34Z

Build failure
❌ Build was aborted

🔥 Build failed 🔥

monojenkins · 2018-12-07T12:40:22Z

Build failure
❌ Build was aborted

🔥 Build failed 🔥

monojenkins · 2018-12-08T23:25:53Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

10 tests failed, 0 tests skipped, 279 tests passed.

Failed tests

[xUnit] Mono SystemSecurityXunit/Mac/Full: Failed (Test run failed.)
[xUnit] Mono SystemSecurityXunit/Mac/Modern: Failed (Test run failed.)
[xUnit] Mono SystemJsonXunit/iOS Unified 32-bits - simulator/Debug: Crashed
[xUnit] Mono SystemNumericsXunit/iOS Unified 32-bits - simulator/Debug: Crashed
[xUnit] Mono SystemJsonXunit/iOS Unified 64-bits - simulator/Debug: Crashed
[xUnit] Mono SystemNumericsXunit/iOS Unified 64-bits - simulator/Debug: Crashed
[NUnit] Mono SystemNumericsTests/watchOS - simulator/Debug: Crashed
[NUnit] Mono SystemXmlLinqTests/watchOS - simulator/Debug: Crashed
[NUnit] Mono SystemJsonTests/watchOS - simulator/Debug: Crashed
[xUnit] Mono SystemJsonXunit/watchOS - simulator/Debug: Crashed

monojenkins · 2018-12-11T01:57:53Z

Build failure
❌ Build was aborted

✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)

monojenkins · 2018-12-11T13:04:29Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
🔥 Test run failed 🔥

Test results

8 tests failed, 0 tests skipped, 279 tests passed.

Failed tests

[xUnit] Mono SystemJsonXunit/iOS Unified 32-bits - simulator/Debug: Crashed
[xUnit] Mono SystemNumericsXunit/iOS Unified 32-bits - simulator/Debug: Crashed
[xUnit] Mono SystemJsonXunit/iOS Unified 64-bits - simulator/Debug: Crashed
[xUnit] Mono SystemNumericsXunit/iOS Unified 64-bits - simulator/Debug: Crashed
[NUnit] Mono SystemNumericsTests/watchOS - simulator/Debug: Crashed
[NUnit] Mono SystemXmlLinqTests/watchOS - simulator/Debug: Crashed
[NUnit] Mono SystemJsonTests/watchOS - simulator/Debug: Crashed
[xUnit] Mono SystemJsonXunit/watchOS - simulator/Debug: Crashed

… the ios apps.

monojenkins · 2018-12-11T16:10:14Z

Build failure
❌ Build was aborted

monojenkins · 2018-12-11T23:41:46Z

Build failure
❌ Build was aborted

monojenkins · 2018-12-12T09:05:05Z

Build success
✅ Build succeeded
✅ API Diff (from stable)
ℹ️ API Diff (from PR only) (please review changes)
✅ Generator Diff (no change)
✅ Test run succeeded

mandel-macaque · 2018-12-12T09:37:54Z

Previous abort was due to the bots getting stuck while building mono, weird. Merging as everything is green,

mandel-macaque added 14 commits November 20, 2018 16:18

[Tests] Add support for xUnit and NUnit on MacOS X.

48aa46b

[Xharness] Add a workaround to not build the xunit tests when not nee…

fb94c85

…ded. This is a workaround that does not use reflection to build the bcl test projects. We need this until we have complete support of using mono as an SDK.

Add the missing parts for the deferred execution of the built of the …

284f9b2

…mono test assemblies.

Merge branch 'master' into dont-compile-xunit

ccf21ee

Address review.

115fa00

Remove noise.

cc6388c

Merge branch 'master' into xunit-mac-support

4e24a35

Merge branch 'dont-compile-xunit' into xunit-mac-support

34cffe8

Add mac tests to the factory.

471f478

Remove lines that do not make longer any sense.

f492809

Merge branch 'dont-compile-xunit' of github.com:mandel-macaque/xamari…

89a4c40

…n-macios into dont-compile-xunit

Merge branch 'dont-compile-xunit' into xunit-mac-support

2fbabc9

Complete xunit and nunit support for mac.

8b53091

Add workaround for mac tests.

906a71b

mandel-macaque requested a review from rolfbjarne as a code owner November 27, 2018 01:32

Small change in the test generation.

0f0332e

mandel-macaque added the run-bcl-tests Run the BCL tests label Nov 27, 2018

rolfbjarne requested changes Nov 27, 2018

View reviewed changes

mandel-macaque added 2 commits November 27, 2018 13:07

Several changes:

43baffe

1. Addresses the reviews from rolf. 2. Ensure that the dependencies are ran on mac test projects. 3. Group the tests for ios and mac using their name.

no need to check if dir exists.

b1f94c8

spouliot reviewed Nov 27, 2018

View reviewed changes

rolfbjarne approved these changes Nov 28, 2018

View reviewed changes

tests/bcl-test/BCLTests/templates/macOS/MacTestMain.cs Outdated Show resolved Hide resolved

mandel-macaque added 2 commits November 28, 2018 12:07

Un-do noisy change for MONOTOUCH to monotouch. Can be done later.

f4af05f

Merge branch 'master' into xunit-mac-support

e64a153

Merge branch 'master' into xunit-mac-support

697c056

Merge branch 'master' into xunit-mac-support

3c4672e

mandel-macaque removed the do-not-merge Do not merge this pull request label Dec 7, 2018

rolfbjarne approved these changes Dec 7, 2018

View reviewed changes

Mono 2018-08 changed the paths to the xammac dlls. Update those.

e62673b

spouliot approved these changes Dec 7, 2018

View reviewed changes

Merge branch 'master' into xunit-mac-support

8a9536d

mandel-macaque added 2 commits December 10, 2018 12:46

Merge branch 'master' into xunit-mac-support

7503ac8

Fix the securtiy project and realize we have failing tests.

7e4d449

mandel-macaque added 3 commits December 11, 2018 16:43

Add extra logging to understand why some mac assemblies are leaked to…

703a11f

… the ios apps.

Mac must use a diff path for the type registration.

5733542

Merge branch 'master' into xunit-mac-support

7ed7f2a

mandel-macaque merged commit ffae762 into xamarin:master Dec 12, 2018

mandel-macaque deleted the xunit-mac-support branch December 12, 2018 09:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[XHarness] Add support for xunit and nunit tests on Mac OS X. #5182

[XHarness] Add support for xunit and nunit tests on Mac OS X. #5182

mandel-macaque commented Nov 27, 2018 •

edited

Loading

monojenkins commented Nov 27, 2018

monojenkins commented Nov 27, 2018

mandel-macaque commented Nov 27, 2018

spouliot left a comment

mandel-macaque commented Nov 27, 2018

monojenkins commented Nov 27, 2018

Failed tests

monojenkins commented Nov 28, 2018

mandel-macaque commented Nov 30, 2018 •

edited

Loading

monojenkins commented Dec 1, 2018

monojenkins commented Dec 1, 2018

rolfbjarne commented Dec 3, 2018

monojenkins commented Dec 3, 2018

Failed tests

monojenkins commented Dec 5, 2018

Failed tests

mandel-macaque commented Dec 7, 2018

monojenkins commented Dec 7, 2018

monojenkins commented Dec 7, 2018

monojenkins commented Dec 8, 2018

Failed tests

monojenkins commented Dec 11, 2018

monojenkins commented Dec 11, 2018

Failed tests

monojenkins commented Dec 11, 2018

monojenkins commented Dec 11, 2018

monojenkins commented Dec 12, 2018

mandel-macaque commented Dec 12, 2018

[XHarness] Add support for xunit and nunit tests on Mac OS X. #5182

[XHarness] Add support for xunit and nunit tests on Mac OS X. #5182

Conversation

mandel-macaque commented Nov 27, 2018 • edited Loading

monojenkins commented Nov 27, 2018

monojenkins commented Nov 27, 2018

mandel-macaque commented Nov 27, 2018

spouliot left a comment

Choose a reason for hiding this comment

mandel-macaque commented Nov 27, 2018

monojenkins commented Nov 27, 2018

Test results

Failed tests

monojenkins commented Nov 28, 2018

mandel-macaque commented Nov 30, 2018 • edited Loading

monojenkins commented Dec 1, 2018

monojenkins commented Dec 1, 2018

rolfbjarne commented Dec 3, 2018

monojenkins commented Dec 3, 2018

Test results

Failed tests

monojenkins commented Dec 5, 2018

Test results

Failed tests

mandel-macaque commented Dec 7, 2018

monojenkins commented Dec 7, 2018

monojenkins commented Dec 7, 2018

monojenkins commented Dec 8, 2018

Test results

Failed tests

monojenkins commented Dec 11, 2018

monojenkins commented Dec 11, 2018

Test results

Failed tests

monojenkins commented Dec 11, 2018

monojenkins commented Dec 11, 2018

monojenkins commented Dec 12, 2018

mandel-macaque commented Dec 12, 2018

mandel-macaque commented Nov 27, 2018 •

edited

Loading

mandel-macaque commented Nov 30, 2018 •

edited

Loading