Early <AndroidAsset> caching results in differences w.r.t. other platforms, use case limits.

[plynkus]

Steps to Reproduce

1. Extract the attached workspace, load the top-level solution.
2. For each of the four client projects (under Clients solution folder), build and run.
3. Observe the on-screen text (Android, iOS, UWP) or command line output (CLI).

BuildArtifacts.zip

Expected Behavior

In all cases, "This is actual content for asset A." is indicated.

Actual Behavior

For all of { CLI, iOS, UWP }, "This is actual content for asset A." is indicated.
For { Android }, "This is a placeholder for asset A." is indicated.

(Further commentary to follow.)

Version Information

Microsoft Visual Studio Community 2017
Version 15.5.2
VisualStudio.15.Release/15.5.2+27130.2010
Microsoft .NET Framework
Version 4.7.02556

Installed Version: Community

Visual Basic 2017 00369-60000-00001-AA028
Microsoft Visual Basic 2017

Visual C# 2017 00369-60000-00001-AA028
Microsoft Visual C# 2017

Visual C++ 2017 00369-60000-00001-AA028
Microsoft Visual C++ 2017

Visual F# 4.1 00369-60000-00001-AA028
Microsoft Visual F# 4.1

Application Insights Tools for Visual Studio Package 8.10.01106.1
Application Insights Tools for Visual Studio

ASP.NET and Web Tools 2017 15.0.31125.0
ASP.NET and Web Tools 2017

ASP.NET Core Razor Language Services 1.0
Provides languages services for ASP.NET Core Razor.

Azure App Service Tools v3.0.0 15.0.31106.0
Azure App Service Tools v3.0.0

Common Azure Tools 1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

JavaScript Language Service 2.0
JavaScript Language Service

JavaScript Project System 2.0
JavaScript Project System

JavaScript UWP Project System 2.0
JavaScript UWP Project System

Merq 1.1.17-rc (cba4571)
Command Bus, Event Stream and Async Manager for Visual Studio extensions.

Microsoft Continuous Delivery Tools for Visual Studio 0.3
Simplifying the configuration of continuous build integration and continuous build delivery from within the Visual Studio IDE.

Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft MI-Based Debugger 1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards 1.0
Microsoft Visual C++ Wizards

Microsoft Visual Studio VC Package 1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio 4.8.4-pre (3fe64e3)
Support for debugging Mono processes with Visual Studio.

NuGet Package Manager 4.5.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

SQL Server Data Tools 15.1.61710.120
Microsoft SQL Server Data Tools

TypeScript Tools 15.5.11025.1
TypeScript Tools for Microsoft Visual Studio

Visual Studio Code Debug Adapter Host Package 1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Tools for CMake 1.0
Visual Studio Tools for CMake

Visual Studio Tools for Universal Windows Apps 15.0.27128.01
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.

VisualStudio.Mac 1.0
Mac Extension for Visual Studio

Xamarin 4.8.0.753 (6575bd113)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 4.8.188 (c5813fa34)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin.Android SDK 8.1.0.25 (HEAD/d8c6e504f)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK 11.6.1.2 (6857dfc)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Log File

[plynkus]

The use case here, generalized, is the ability to author external packages that provide specialized asset generation/conditioning pipelines for purposes related to package functionality.

The cause of the problem appears to be the timing/method of Xamarin.Android's production of an intermediate asset ZIP archive (e.g. obj/Debug/android/bin/packaged_resources). I am assuming this is done to provide a ready-made set of assets to include when APK production ultimately occurs later. The issue, though, is that its acquisition of <AndroidAsset>s (by contents and not by reference) occurs early enough that it captures placeholder or incomplete assets not yet fully conditioned by the build itself (which may include processing steps by the aforementioned external packages).

This results in a difference in behavior under Xamarin.Android vs. other platforms---one that assumes all asset content is static at build start, breaking the use of any MSBuild task-implemented asset production and/or conditioning.

[plynkus]

FWIW 1: The reason true (i.e. not placeholder) asset content in the attached example is done post-compile isn't just for the benefit of demonstrating the problem here. Some pipeline use cases require successful compilation to occur before processing is meaningful---they can't be done earlier, say, in advance of BeforeBuild where Xamarin.Android's implementation would currently pick up the contents.

[plynkus]

FWIW 2: The need to identify and emit placeholders in a before-build preprocess step (as you'll see in the .targets files within the solution/nuget package) is an unrelated appeasement to a subset of the platforms---in addition to Android with <AndroidAsset>, I believe UWP also won't include any <Content> not identified up front before the build. Placeholders are not ideal but get the job done.

[plynkus]

FWIW 3: If the timing of the generation of packaged_resources cannot be changed, an alternative might be to simply store within it (or alongside it) a manifest of <AndroidAsset>s by original reference instead of the (potentially stale) files themselves. APK production could then enumerate and gather the artifacts from where you were told they were going to be anyway---and you'd end up with current contents.

(Storing references vs. copies also has scaling benefits---consider a very asset-heavy application having to endure what look to be unnecessary extra copies.)

[dellis1972]

Thanks for the detailed description of the issue. This will require some creative thinking. One of the things that the android process does is create a Zip file which contains all the library resources and assets (layout files etc). This is embedded into the library as part of the compilation process via an EmbeddedResource. Note this is not the packaged_resources file, but the __AndroidLibraryProjects__.zip which you can find in the library intermediate directory.

There is a reason for this. Nuget and all the google libraries. A library needs to ship as a single assembly with all the required dependencies included. The Embedded Resource is then extracted as part of the app build process into an intermediate directory which is then used for building the app.

One option for now might be to do the post processing in the app itself. While we don't have any extension points at the moment you can do

<Target Name="Foo" BeforeTargets="_CreateBaseApk"> </Target>

We can add a public OnBeforeCreateBaseApk property which would allow developers to hook into this.

I'll have a think about this and see if I can figure out a good way to support this use case.
It would be helpful to have a real life example of what kind of assets you are creating post build. i.e what information from the assembly do you use ?

[plynkus]

Thanks for the quick response, Dean.

Thanks for the detailed description of the issue. This will require some creative thinking. One of the things that the android process does is create a Zip file which contains all the library resources and assets (layout files etc). This is embedded into the library as part of the compilation process via an EmbeddedResource. Note this is not the packaged_resources file, but the AndroidLibraryProjects.zip which you can find in the library intermediate directory.

I was initially a little confused here, but I follow now after some local playing. What you speak of are Xamarin.Android class library build artifacts (__AndroidLibraryProjects__.zip), and I understand your rationale there for the need to embed the assets within. Our primary blocking case here deals with the topology in the example project, though---a Xamarin.Android application project that employs a NuGet package providing the asset production/conditioning feature. In the application case, we see the assets placed within packaged_resources. So there is an apparent difference between the build process for each kind of project. The longer asset propagation chain (class library project employing the pipeline-containing nuget package, then an application employing the class library) is an interesting case but not one we're immediately faced with.

One option for now might be to do the post processing in the app itself. While we don't have any extension points at the moment you can do

<Target Name="Foo" BeforeTargets="_CreateBaseApk"> </Target>

Tried this just now, and while I will recheck my work it did not appear to change the outcome. The exercise resulted in additional insight, however. Consider the following:

plynkus@XYZ MINGW64 /C/WS/Bugs/BuildArtifacts/Clients/BuildArtifacts.Android/obj/Debug (master)
$ stat artifacts/ASSETA
  File: artifacts/ASSETA
  Size: 35              Blocks: 1          IO Block: 65536  regular file
Device: 1eca4c7bh/516574331d    Inode: 4785074604623554  Links: 1
Access: (0644/-rw-r--r--)  Uid: (197608/  plynkus)   Gid: (197121/ UNKNOWN)
Access: 2018-01-03 08:38:44.790064600 -0800
Modify: 2018-01-03 08:38:46.915175600 -0800
Change: 2018-01-03 08:38:46.915175600 -0800
 Birth: 2018-01-03 08:38:44.790064600 -0800

plynkus@XYZ MINGW64 /C/WS/Bugs/BuildArtifacts/Clients/BuildArtifacts.Android/obj/Debug (master)
$ stat assets/artifacts/ASSETA
  File: assets/artifacts/ASSETA
  Size: 34              Blocks: 1          IO Block: 65536  regular file
Device: 1eca4c7bh/516574331d    Inode: 4503599627912902  Links: 1
Access: (0644/-rw-r--r--)  Uid: (197608/  plynkus)   Gid: (197121/ UNKNOWN)
Access: 2018-01-03 08:38:44.836931700 -0800
Modify: 2018-01-03 08:38:44.836931700 -0800
Change: 2018-01-03 08:38:44.836931700 -0800
 Birth: 2018-01-03 08:38:44.821312800 -0800

plynkus@XYZ MINGW64 /C/WS/Bugs/BuildArtifacts/Clients/BuildArtifacts.Android/obj/Debug (master)
$ stat android/bin/packaged_resources
  File: android/bin/packaged_resources
  Size: 2709            Blocks: 4          IO Block: 65536  regular file
Device: 1eca4c7bh/516574331d    Inode: 3096224744361619  Links: 1
Access: (0644/-rw-r--r--)  Uid: (197608/  plynkus)   Gid: (197121/ UNKNOWN)
Access: 2018-01-03 08:38:47.149445800 -0800
Modify: 2018-01-03 08:38:47.149445800 -0800
Change: 2018-01-03 08:38:47.149445800 -0800
 Birth: 2018-01-03 08:38:47.149445800 -0800

The chain of events (in time order) from the above is consistent with the observed outcome. Specifically (and everything relative to obj/Debug here):

1. 08:38:44.790064600: We first write/report the placeholder (artifacts/ASSETA, size 34).
2. 08:38:44.821312800: Xamarin makes a copy (assets/artifacts/ASSETA, size 34).
3. 08:38:46.915175600: We then generate the actual content (artifacts/ASSETA, now size 35).
4. 08:38:47.149445800: Xamarin creates android/bin/packaged_resources with its copy from (2).

And for completeness, the contents of (4) above...

Archive:  android/bin/packaged_resources
  Length      Date    Time    Name
---------  ---------- -----   ----
     3384  1980-12-31 16:00   AndroidManifest.xml
       34  1980-12-31 16:00   assets/artifacts/ASSETA
       34  1980-12-31 16:00   assets/artifacts/ASSETB
      372  1980-12-31 16:00   res/layout/main.xml
      756  1980-12-31 16:00   resources.arsc
---------                     -------
     4580                     5 files

... shows the embedded size (34) is in agreement with the placeholder, consistent with the timeline.

I seems to me another way to resolve this, with additional performance/storage benefits through elimination of redundancies, might be to simply eliminate your copy of <AndroidAsset> artifacts (e.g. artifacts/ASSETA) into an apparent staging area (e.g. assets/artifacts/ASSETA). You know where to find each original (<AndroidAsset>-provided, which you used to source your copy), and you know where they are going (inside the packaged_resources ZIP, but simply beneath a new root-level assets folder with relative paths preserved underneath otherwise). Pure speculation here on the implementation, but perhaps the staging folder is used to facilitate an easy bulk folder ZIP addition (assets/ -> packaged_resources) vs. having to add each asset to the ZIP individually?

I'll next try, knowing the above, emitting assets directly into the staging folder (assets/) and see if Xamarin.Android accepts it or croaks (e.g. file copy onto itself issues). Will report back.

Thanks again!

[plynkus]

Forgot to respond to the tail of your last, Dean:

It would be helpful to have a real life example of what kind of assets you are creating post build. i.e what information from the assembly do you use ?

Can't go into much detail, but we are certainly looking at attributes, etc. and in some cases even performing code analysis to emit build-coordinated data for Useful Purposes™. ;).

[plynkus]

Quick followup on my test, as promised. In the example workspace, if I change a single line in Postprocess.cs from:

var root = Path.Combine(OutputPath, "artifacts");

to:

var root = Path.Combine(new[] { OutputPath, "assets", "artifacts" });

...I get lucky and see proper from-clean operation. This of course means the placeholders remain, orphaned, in their correct locations and I am injecting the true content instead into Xamarin's asset staging folder, which I really shouldn't be doing. Inelegant, but it might at least unblock us until a better fix is identified. I'll test it more generally (real projects, incremental build changes, etc.).

[dellis1972]

another option might be to have your update process emit the updated files directly into $(MonoAndroidAssetsDirIntermediate). That is the property name for the directory we output assets to.

I am going to look at changing the build order of UpdateAndroidAssets so that it happens just before we create a base apk. In theory it should mean that it happens after the compile has run, but it should not effect anything else. I'll keep you posted.

[plynkus]

Thanks for that. I'll give it a shot.

[dellis1972]

PR #1161 should fix this issue.

[plynkus]

Much appreciated, Dean! Thanks for the quick look/resolution.