Add testing for Xamarin

[divega]

Based on recent reports things are generally working with improvements made to Mono and Xamarin in recent versions. We however don't have any automated testing and adding it would likely require different infrastructure from what we use for testing this for other .NET platforms.

[bricelam]

Should we make this a more general cross-platform testing automation bug or file a new one for UWP so it can be prioritized separately?

[divega]

@bricelam My intention is to track them as separate issues, precisely to prioritize them separately. I haven't created the UWP issue yet, mostly because 2.0 cannot be tested on UWP yet.

[cwrea]

I've been mucking around with EF Core 2.0.1 and 2.0.2 with Xamarin.iOS lately, trying to get a meaningful cross-section of EF Core unit tests copied into and passing in a mobile app deployed to a real iOS device, i.e. AOT compiled. I'm doing this to get more confidence in EF Core for Xamarin mobile app development. (EF Core is working in samples I've tried, but they're relatively shallow.)

This remains a work in progress, but I thought I'd remark here about one specific class of issue I needed to work around, so far: there are handfuls of tests that use the Moq framework. Moq depends on generating code at runtime with Reflection.Emit, which isn't permitted for AOT compiled applications. I imagine this would also be an issue for UWP.

Fortunately, I noticed issue #7627 and it looks like 2.1.0's tests won't depend on Moq. 👍

[ajcvickers]

@cwrea It's great to hear that you are doing this testing--really interested to see what you find. And, yes, Moq is gone in the current codebase.

@divega We should look into whether we can leverage the dotnet ci bot to run our tests on Xamarin. It's already running our PRs, so it seems not unreasonable to go the next step.

[cwrea]

In late March, I did get mobile test-runner apps working with a meaningful subset of the EF Core unit tests. As you can see from the issue references just above, I'd already filed a couple of issues for Xamarin.Android and Xamarin.iOS — issues that prevented certain unit tests from running.

I put the rest on the back burner for a little while to wait for VS 15.7 to be released. I'd tried the previews and I knew 15.7 would include Xamarin updates that would address some of the test failures reported on Xamarin.iOS.

Since VS 15.7 was released earlier this week, I just published my results from using the official release. Please find them at Entity Framework Core 2.0.2 Unit Test Subset Results on Xamarin.Android and Xamarin.iOS.

The repository also contains the entire Visual Studio solution for the mobile test runner apps, including the subset of Entity Framework Core 2.0.2 unit test classes that I judged as meaningful to test on mobile.

The EF Core unit tests subset does not include tests that relate to benchmarks, design-time, and SQL Server. Also, tests that used Moq were commented out, and some tests that required SQL Server as accessory to test non-SQL Server functionality were adapted for Sqlite.

The results from both the Xamarin.Android and Xamarin.iOS test runner mobile apps (based on xUnit.net Runners for Devices) were my primary concern, but the solution also includes console-based projects for running the same tests on full .NET Framework and .NET Core. I added these console projects to sanity-check that the test class subset plus manual fixups did not break tests that already pass on those platforms.

There is also a UWP test runner app project. I did get the UWP runner working initially with default Debug project build settings — and all tests from the subset were passing for UWP x86 and x64 on those settings.

However, UWP builds with the all-important option "Compile with .NET Native tool chain" enabled would not compile, not on 15.6.x before, and not on 15.7 now: error : ILT0005: '[...]\nutc_driver.exe [...]"' returned exit code 1.

This surprised me, as I expected if Xamarin.iOS could get an AOT-based build compiled, then so should the .NET Native tool chain for UWP. Plus, UWP is mentioned here as working with EF Core 2.0.1. I didn't venture any further down the UWP rabbit hole, but my guess is that if others have successfully been able to compile UWP native apps that use Entity Framework Core, then something else about the test runner app (size or complexity, perhaps) is causing the .NET Native tool chain to choke.

[cwrea]

p.s. I'll add that I do intend to repeat this exercise for EF Core 2.1 when released, to check for regressions. At the same time, with the issues already identified on Android, iOS, and UWP being Xamarin/Mono framework or UWP tooling issues, I don't expect EF Core 2.1 would improve on current results on those platforms. Platform fixes are necessary, and I still have problems to isolate for filing a couple of new issues in the respective places.

[cwrea]

Today I attempted to update my Xamarin-based test runners to use the EF Core 2.1 release.

I encountered a major issue. An iOS app that references EF Core 2.1 won't even build. I filed the following issue for Xamarin.iOS: #4168 - iOS app build fails when referencing package (e.g. Entity Framework Core 2.1) that depends on newer System.Memory.

[divega]

@cwrea Thanks for trying this. We appreciate what you are doing a lot. I think you hit the same issue reported at There was another issue reported with System.Memory, but with different symptoms: #12081. It was suggested to copying System.Memory.dll as workaround, but we haven't had a chance to verify.

[cwrea]

@divega You're welcome.

I actually just filed a similar issue for Xamarin.Android just a couple of moments ago: #1769 - Android app build fails when referencing package (e.g. Entity Framework Core 2.1) that depends on newer System.Memory. I only noticed your comment about #12081 after opening the Xamarin.Android issue — but I think my issue is in the right place. Includes a sample that reproduces the issue, and downgrading EF Core 2.1 to 2.0.3 gets the sample building and working.

[ToddThomson]

My 2 cents: For the past couple of months I've been building an app which utilizes a common data services library (NetStandard v2.0) using EFCore.Sqlite. For me, cross platform mobile app development using c# with Xamarin.Android, Xamarin.iOS and NetStandard v2.0 libraries is a HUGE advantage. I feel that having an EFCore testing suite for Xamarin is a necessary step forward for EF Core ( and MS as the lead in cross platform development ).

[cwrea]

More generally...

... I know that getting a fully automated EF Core test suite running on iOS and Android devices remains an open issue and presents a handful of challenges as originally noted above.

But maybe now is a good time for the Xamarin and Entity Framework teams at Microsoft to begin co-ordinating pre-release testing of EF Core on for iOS and Android? Start small, if necessary? Build a sufficiently non-trivial sample app on devices once in a while, to ensure no obvious regression before a release?

Consider:

• Each of EF Core and Xamarin are high-profile strategic technology offerings from Microsoft.

• Each is sufficiently mature and being used by Microsoft customers to solve real-world problems.

• And yet, EF Core + Xamarin are not sufficiently mature __when used together__. (sorry!)

That remains a strategic gap. One of the early goals I remember Rowan Miller mentioning about EF7 way back in 2014 that got me interested, and paying attention ever since, was that not only would EF vNext work for cross-platform desktop and server OSs like macOS and Linux, but that EF vNext could/would, eventually, also work on mobile devices.

I assume Windows Phone was the likely intended benefactor of a mobile-capable version of EF as envisioned in 2014, but once Microsoft pivoted from competing as mobile OS vendor to providing awesome development tools for iOS and Android, I expected EF Core would eventually include Xamarin.iOS and Xamarin.Android as first class platforms.

That these two strategic Microsoft development technologies do now sorta, kinda, sometimes work together is nice, but it would be even better if they were intentionally made to work well together.

So now that desktop and server OSs have been given a highly-performant, mature, and feature-packed release of EF Core in v2.1, could mobile platforms please be shown love in a coming-soon release?

🙏 🙏

[divega]

@cwrea, @ToddThomson thanks for your great feedback!

Build a sufficiently non-trivial sample app on devices once in a while, to ensure no obvious regression before a release?

We are discussing this with the Xamarin team. It seems they already regularly build certain samples and customer apps as a way to guard against regressions. It would be great if you can point us to good candidates. @cwrea I know from before that you have built some samples using EF Core with Xamarin together. I would like to know if you think they are sufficient or at least a good starting point. Also, if you can point us to the latest code, that would be great.

[divega]

@smitpatel see #8792 (comment), for the iOS repro of the issue with System.Memory. We don't have a separate issue on the EF Core side tracking this.

[ToddThomson]

@divega Worth investigating is xunit.runner.devices referenced on the xamarin app (iOS,Android). I have zero experience with this (xunit.runner.devices). However, I modified my EFCore Sqlite 2.1 test applet and was able to build and run the common netstandard 2 lib with EFCore Sqlite 2.1 and XUnit along with the mono android app with the xunit runner (2.4 beta nuget package).

I'll poke about to see what can be accomplished...

Additionally...

xunit.runner.devices is a really cool package! I only tried it with mono android, but the xunit tests were truly simple to setup and run. The main activity in the xamarin android app is derived from RunnerActivity and provides all the UI. I added 1) a test class to the common netstandard 2 lib with EFCoreSqlite 2.1 and Xunit; 2) a test class in the app itself. Both were discovered with proper setup in the override of OnCreate().

using Android.App;
using Android.OS;
using System.Reflection;
using Xunit.Runners.UI;
using Xunit.Sdk;

namespace SystemMemoryTest
{
    [Activity( Label = "xUnit Android Runner", MainLauncher = true, Theme = "@android:style/Theme.Material.Light" )]
    public class MainActivity : RunnerActivity
    {
        protected override void OnCreate( Bundle savedInstanceState )
        {
            // tests can be inside the main assembly
            AddTestAssembly( Assembly.GetExecutingAssembly() );

            AddExecutionAssembly( typeof( ExtensibilityPointFactory ).Assembly );
            // or in any reference assemblies			

            AddTestAssembly(typeof( EFCoreSqliteTestLibrary.Class1).Assembly);
            // or in any assembly that you load (since JIT is available)

#if false
            // you can use the default or set your own custom writer (e.g. save to web site and tweet it ;-)
            Writer = new TcpTextWriter ("10.0.1.2", 16384);
            // start running the test suites as soon as the application is loaded
            AutoStart = true;
            // crash the application (to ensure it's ended) and return to springboard
            TerminateAfterExecution = true;
#endif
            // you cannot add more assemblies once calling base
            base.OnCreate( savedInstanceState );
        }
    }
}

[ToddThomson]

@divega I just ran your public class DbSetAsTableNameSqliteTest : DbSetAsTableNameTest test on my android phone. I just copied a couple of your EF Core test files into my net standard 2 lib with EFCore Sqllite.

I'd say that this approach may be a starting point to fit your needs (I will definitely be using it).

[cwrea]

@divega Thanks! Here's a link to the Xamarin Todo sample that I ported from another ORM popular with Xamarin apps (SQLite.Net-PCL) to run on EF Core instead, some time ago: https://github.com/cwrea/XamarinTodo/tree/NETStd20_EFCore203. That's currently the latest, using EF Core 2.0.3. I haven't updated for EF Core 2.1 yet, mainly because the iOS build will fail due to open issue xamarin/xamarin-macios#4168.

Also of interest— but not in the category of a simple sample—in my May 10 comment on this issue I mentioned and linked to another solution I hacked together more recently. The following repository contains both Xamarin.Android and Xamarin.iOS test runner apps. They exercise a meaningful subset of EF Core unit tests: https://github.com/cwrea/EFCoreCPTestSoln/tree/EFCore2.0.2_VS15.7. The test runner apps use the same xunit.runner.devices package that @ToddThomson just mentioned.

That test runners solution also isn't updated for EF Core 2.1 yet—or, rather, I haven't published my work-in-progress— also due to the same iOS build failure issue. But locally I do at least have a working version for Xamarin.Android and have been discovering a handful of new issues for Xamarin.Android that new tests in EF Core 2.1 have uncovered. Tomorrow, I could probably share a half-baked update to the solution that at least compiles and runs the EF Core 2.1 subset on Xamarin.Android. This solution I hacked together is far from perfect but might be a good starting point.

[cwrea]

@divega Re:

There was another issue reported with System.Memory, but with different symptoms: #12081. It was suggested to copying System.Memory.dll as workaround, but we haven't had a chance to verify.

FWIW, for Xamarin.iOS a number of potential workarounds have been attempted e.g. referencing the netstandard2.0 version of System.Memory (xamarin/xamarin-macios#4168 (comment)) but none have succeeded so far.

I did find a related set of commits at xamarin/xamarin-macios#4147 which pre-dated the issue by a couple of days, however it isn't clear to me when those changes will make it into a preview release and whether they would definitively address the System.Memory issue.

Even if the removal of the broken Xamarin.iOS facade assembly permits the netstandard2.0 version of System.Memory to be used by the build, I suspect that the AOT assertion failure seen as result of some of the promising workarounds may end up as the next hurdle.

[divega]

Action item for @divega and @ajcvickers: review sample applications from the perspective of EF Core usage. Once we have something we will contact the Xamarin team for inclusion in regression protection tests.

[ToddThomson]

@divega @ajcvickers You may want to look at NUnitLite in the xamarin.android repo.

[ajcvickers]

Note from triage: focus has shifted to MAUI.