Skip to content

Commit

Permalink
New code to perform managed <-> java lookups (typemap)
Browse files Browse the repository at this point in the history
Xamarin.Assembly needs to "translate" managed types to Java types and
vice versa in order to provide a bridge between the two world. So far it
has been done using a straightforward (and fast) method of performing
the lookups - all the type pairs were stored in two tables of the same
size, with all type names padded to the width of the longest name so
that the `bsearch` C function can be used to quickly perform a binary
search over the data set. This approach works very well at the expense
of data size (shorter strings are 0-padded to the maximum width) and a
slightly degraded performace because of the requirement to perform
string comparisons. Furthermore, the lookup required that reflection is
used to obtain full managed type name (when translating from managed to
Java) or to get a `Type` instance from type name (when translating from
Java to managed).

For Release builds all the above data is placed in the
`libxamarin-app.so` library, for Debug builds it is also placed in two
files - one for each direction of lookup, described above.

This commit is a slight improvement over the above scheme. It eliminates
reflection from the process by using managed type tokens (which are
integers) and using UUID/Guid of the module in which the type is found.
This allows us to perform the binary search over the set of 20 bytes (16
bytes for the UUID and 4 bytes for the token ID) for managed to Java
lookups and a single string comparison + binary search over a set of
integers for the Java to managed lookup.

Java type names must still be used because Java doesn't provide any
equivalent to the .NET's type token and module UUID. Those names are
still 0-padded to the width of the longest name but there are no longer
duplicated. Managed type names are eliminated completely.

If Xamarin.Android Instant Run is not used (which is the case for OSS
code) for Debug builds, the operation is performed in the same way for
both Release and Debug builds. If, however, Instant Run is in effect,
the type maps are stored in several files with the .typemap extension -
one per **module**. The files contain both the Java to managed maps as
well as managed to Java maps (which use indexes into the Java to managed
maps). All of those files are loaded during Debug app startup and used
to construct a dataset which is the searched during all the lookups.

Typemap index file format, all data is little-endian:
----

 **Header format**

 `[Magic string]`             # XATI
 `[Format version]`           # 32-bit unsigned integer, 4 bytes
 `[Entry count]`              # 32-bit unsigned integer, 4 bytes
 `[Module file name width]`   # 32-bit unsigned integer, 4 bytes
 `[Index entries]`            # Format described below, `Entry count` entries

 **Index entry format:**

 `[Module UUID][File name]<NUL>`

  *Where:*

 `[Module UUID]` is 16 bytes long
 `[File name]` is right-padded with `<NUL>` characters to the `[Module file name width]` boundary.

Typemap file format, all data is little-endian:
----

 **Header format**

 `[Magic string]`                    # XATM
 `[Format version]`                  # 32-bit integer, 4 bytes
 `[Module UUID]`                     # 16 bytes
 `[Entry count]`                     # unsigned 32-bit integer, 4 bytes
 `[Duplicate count]`                 # unsigned 32-bit integer, 4 bytes (might be 0)
 `[Java type name width]`            # unsigned 32-bit integer, 4 bytes
 `[Assembly name size]`              # unsigned 32-bit integer, 4 bytes
 `[Assembly name]`                   # Non-null terminated assembly name
 `[Java-to-managed map]`             # Format described below, `[Entry count]` entries
 `[Managed-to-java map]`             # Format described below, `[Entry count]` entries
 `[Managed-to-java duplicates map]`  # Map of unique managed IDs which point to the same Java type name (might be empty)

 **Java-to-managed map format:**

 `[Java type name]<NUL>[Managed type token ID]`

 Each name is padded with `<NUL>` to the width specified in the `[Java type name width]` field above.
 Names are written without the size prefix, instead they are always terminated with a nul character
 to make it easier and faster to handle by the native runtime.

 Each token ID is an unsigned 32-bit integer, 4 bytes

 **Managed-to-java map format:**

 `[Managed type token ID][Java type name table index]`

 Both fields are unsigned 32-bit integers, to a total of 8 bytes per entry. Index points into the
 `[Java-to-managed map]` table above.

 **Managed-to-java duplicates map format:**

 Format is identical to `[Managed-to-java]` above.

Size changes (XF integration test, `libxamarin-app.so`, Release build):
----

  - armeabi-v7a
	  - before: 376616
		-  after: 97860
  - arm64-v8a
	  - before: 377408
		-  after: 104192
  - x86
	  - before: 376424
    -  after: 97604

Performance changes (XF integration test, Release build):
----

        Device name: **Pixel 3 XL**
Device architecture: **arm64-v8a**
Number of test runs: **10**

|                 | **Native to managed**  | **Runtime init** | **Displayed** | **Notes**                      |
|-----------------|------------------------|------------------|---------------|--------------------------------|
| **master**      | 141.102                | 160.606          | 839.80        |  preload enabled; 32-bit build |
| **this commit** | 134.539                | 154.701          | 836.10        |  |
| **master**      | 141.743                | 158.325          | 837.20        | preload disabled; 32-bit build |
| **this commit** | 134.064                | 149.137          | 831.90        |  |
| **master**      | 134.526                | 152.640          | 805.10        |  preload enabled; 64-bit build |
| **this commit** | 126.376                | 143.226          | 788.60        |  |
| **master**      | 134.049                | 149.543          | 779.40        | preload disabled; 64-bit build |
| **this commit** | 124.847                | 139.227          | 776.10        |  |

Build performance (**Release** build):
----

**Before**

    389 ms  GenerateJavaStubs                          1 calls

**After**

    247 ms  GenerateJavaStubs                          1 calls

New code generates only native assembly or only the binary typemap
files, unlike the old code which generated both. Initially the new
generator code was moved to a separate task, but Jonathan Peppers
determined that it was suboptimal and re-integrated the code back with
`GenerateJavaStubs`
  • Loading branch information
grendello committed Feb 10, 2020
1 parent 8acd915 commit 18fd1d5
Show file tree
Hide file tree
Showing 63 changed files with 1,939 additions and 860 deletions.
1 change: 1 addition & 0 deletions Documentation/guides/messages/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ ms.date: 01/24/2020
+ [XA4305](xa4305.md): MultiDex is enabled, but '{nameof (MultiDexMainDexListFile)}' was not specified.
+ [XA4306](xa4306.md): R8 does not support \`@(MultiDexMainDexList)\` files when android:minSdkVersion >= 21
+ [XA4307](xa4307.md): Invalid ProGuard configuration file.
+ [XA4308](xa4308.md): Failed to generate type maps

## XA5xxx: GCC and toolchain

Expand Down
18 changes: 18 additions & 0 deletions Documentation/guides/messages/xa4308.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: Xamarin.Android error XA4308
description: XA4308 error code
ms.date: 02/07/2020
---
# Xamarin.Android error XA4308

## Issue

The `GenerateJavaStubs` task was unable to generate type maps. Detailed diagnostic will be found before this
error in the build log.

## Solution

Consider submitting a [bug][bug] if you are getting this warning under
normal circumstances.

[bug]: https://github.com/xamarin/xamarin-android/wiki/Submitting-Bugs,-Feature-Requests,-and-Pull-Requests
50 changes: 32 additions & 18 deletions build-tools/xaprepare/xaprepare/Steps/Step_DownloadMonoArchive.cs
Original file line number Diff line number Diff line change
Expand Up @@ -60,26 +60,39 @@ async Task<bool> DownloadMonoArchive (Context context)
localPath = Path.Combine (context.Properties.GetRequiredValue (KnownProperties.AndroidToolchainCacheDirectory), archiveFileName);
}

bool result = await DownloadAndUpackIfNeeded (
context,
"Mono",
context.MonoArchiveCustomUrl,
localPath,
archiveFileName,
Configurables.Paths.MonoSDKSOutputDir
);
bool result = false;
for (uint i = 0; i < 3; i++) {
result = await DownloadAndUpackIfNeeded (
context,
"Mono",
context.MonoArchiveCustomUrl,
localPath,
archiveFileName,
Configurables.Paths.MonoSDKSOutputDir
);

if (result)
break;
}

if (!result)
return false;

return await DownloadAndUpackIfNeeded (
context,
"Windows Mono",
customUrl: null,
localPath: Configurables.Paths.MonoArchiveWindowsLocalPath,
archiveFileName: Configurables.Paths.MonoArchiveWindowsFileName,
destinationDirectory: Configurables.Paths.BCLWindowsOutputDir
);
for (uint i = 0; i < 3; i++) {
result = await DownloadAndUpackIfNeeded (
context,
"Windows Mono",
customUrl: null,
localPath: Configurables.Paths.MonoArchiveWindowsLocalPath,
archiveFileName: Configurables.Paths.MonoArchiveWindowsFileName,
destinationDirectory: Configurables.Paths.BCLWindowsOutputDir
);

if (result)
break;
}

return result;
}

async Task<bool> DownloadAndUpackIfNeeded (Context context, string name, string customUrl, string localPath, string archiveFileName, string destinationDirectory)
Expand Down Expand Up @@ -109,7 +122,7 @@ async Task<bool> DownloadAndUpackIfNeeded (Context context, string name, string
await Download (context, url, localPath, $"{name} Archive", archiveFileName, downloadStatus);

if (!File.Exists (localPath)) {
Log.InfoLine ($"Download of {name} archive from {url} failed, Mono will be rebuilt");
Log.InfoLine ($"Download of {name} archive from {url} failed");
return false;
}
}
Expand All @@ -118,7 +131,8 @@ async Task<bool> DownloadAndUpackIfNeeded (Context context, string name, string
if (!await Utilities.Unpack (localPath, tempDir, cleanDestinatioBeforeUnpacking: true)) {
Utilities.DeleteFileSilent (localPath);
Utilities.DeleteDirectorySilent (destinationDirectory);
Log.WarningLine ($"Failed to unpack {name} archive {localPath}, Mono will be rebuilt");
Log.WarningLine ($"Failed to unpack {name} archive {localPath}");
Utilities.DeleteFileSilent (localPath);
return false;
}

Expand Down
12 changes: 6 additions & 6 deletions src/Mono.Android/Android.Runtime/AndroidRuntime.cs
Original file line number Diff line number Diff line change
Expand Up @@ -231,9 +231,9 @@ protected override IEnumerable<Type> GetTypesForSimpleReference (string jniSimpl

protected override string GetSimpleReference (Type type)
{
var j = JNIEnv.monodroid_typemap_managed_to_java (type.FullName + ", " + type.Assembly.GetName ().Name);
if (j != IntPtr.Zero) {
return Marshal.PtrToStringAnsi (j);
string j = JNIEnv.TypemapManagedToJava (type);
if (j != null) {
return j;
}
if (JNIEnv.IsRunningOnDesktop) {
return JavaNativeTypeManager.ToJniName (type);
Expand All @@ -243,9 +243,9 @@ protected override string GetSimpleReference (Type type)

protected override IEnumerable<string> GetSimpleReferences (Type type)
{
var j = JNIEnv.monodroid_typemap_managed_to_java (type.FullName + ", " + type.Assembly.GetName ().Name);
if (j != IntPtr.Zero) {
yield return Marshal.PtrToStringAnsi (j);
string j = JNIEnv.TypemapManagedToJava (type);
if (j != null) {
yield return j;
}
if (JNIEnv.IsRunningOnDesktop) {
yield return JavaNativeTypeManager.ToJniName (type);
Expand Down
53 changes: 49 additions & 4 deletions src/Mono.Android/Android.Runtime/JNIEnv.cs
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,14 @@ public static partial class JNIEnv {
static UncaughtExceptionHandler defaultUncaughtExceptionHandler;

internal static bool IsRunningOnDesktop;
internal static bool LogTypemapMissStackTrace;

static AndroidRuntime androidRuntime;
static BoundExceptionType BoundExceptionType;

[ThreadStatic]
static byte[] mvid_bytes;

internal static AndroidValueManager AndroidValueManager;

[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
Expand Down Expand Up @@ -146,6 +150,8 @@ internal static unsafe void Initialize (JnienvInitializeArgs* args)
partial_timing_sequence = monodroid_timing_start (null);
}

LogTypemapMissStackTrace = (args->logCategories & (uint)LogCategories.Assembly) != 0;

gref_gc_threshold = args->grefGcThreshold;

java_vm = args->javaVm;
Expand Down Expand Up @@ -632,16 +638,55 @@ public static string GetClassNameFromInstance (IntPtr jobject)
}

[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr monodroid_typemap_managed_to_java (string managed);
static extern IntPtr monodroid_typemap_managed_to_java (byte[] mvid, int token);

internal static void LogTypemapTrace (StackTrace st)
{
string trace = st.ToString ()?.Trim ();
if (String.IsNullOrEmpty (trace))
return;

monodroid_log (LogLevel.Warn, LogCategories.Assembly, "typemap: called from");
foreach (string line in trace.Split ('\n')) {
monodroid_log (LogLevel.Warn, LogCategories.Assembly, line);
}
}

internal static string TypemapManagedToJava (Type type)
{
if (mvid_bytes == null)
mvid_bytes = new byte[16];

Span<byte> mvid = new Span<byte>(mvid_bytes);
byte[] mvid_slow = null;
if (!type.Module.ModuleVersionId.TryWriteBytes (mvid)) {
monodroid_log (LogLevel.Warn, LogCategories.Default, $"Failed to obtain module MVID using the fast method, falling back to the slow one");
mvid_slow = type.Module.ModuleVersionId.ToByteArray ();
}

IntPtr ret = monodroid_typemap_managed_to_java (mvid_slow == null ? mvid_bytes : mvid_slow, type.MetadataToken);

if (ret == IntPtr.Zero) {
if (LogTypemapMissStackTrace) {
monodroid_log (LogLevel.Warn, LogCategories.Default, $"typemap: failed to map managed type to Java type: {type.AssemblyQualifiedName} (Module ID: {type.Module.ModuleVersionId}; Type token: {type.MetadataToken})");
LogTypemapTrace (new StackTrace (true));
}

return null;
}

return Marshal.PtrToStringAnsi (ret);
}

public static string GetJniName (Type type)
{
if (type == null)
throw new ArgumentNullException ("type");
var java = monodroid_typemap_managed_to_java (type.FullName + ", " + type.Assembly.GetName ().Name);
return java == IntPtr.Zero

string java = TypemapManagedToJava (type);
return java == null
? JavaNativeTypeManager.ToJniName (type)
: Marshal.PtrToStringAnsi (java);
: java;
}

public static IntPtr ToJniHandle (IJavaObject value)
Expand Down
16 changes: 10 additions & 6 deletions src/Mono.Android/Java.Interop/TypeManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
using System.Collections.Generic;
using System.Reflection;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using Java.Interop.Tools.TypeNameMappings;

Expand Down Expand Up @@ -203,22 +204,25 @@ static Exception CreateJavaLocationException ()
return new JavaLocationException (loc.ToString ());
}

[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
internal static extern IntPtr monodroid_typemap_java_to_managed (string java);
[MethodImplAttribute(MethodImplOptions.InternalCall)]
static extern Type monodroid_typemap_java_to_managed (string java_type_name);

internal static Type GetJavaToManagedType (string class_name)
{
var t = monodroid_typemap_java_to_managed (class_name);
if (t != IntPtr.Zero)
return Type.GetType (Marshal.PtrToStringAnsi (t));
Type type = monodroid_typemap_java_to_managed (class_name);
if (type != null)
return type;

if (!JNIEnv.IsRunningOnDesktop) {
// Miss message is logged in the native runtime
if (JNIEnv.LogTypemapMissStackTrace)
JNIEnv.LogTypemapTrace (new System.Diagnostics.StackTrace (true));
return null;
}

__TypeRegistrations.RegisterPackages ();

var type = (Type) null;
type = null;
int ls = class_name.LastIndexOf ('/');
var package = ls >= 0 ? class_name.Substring (0, ls) : "";
List<Converter<string, Type>> mappers;
Expand Down
4 changes: 4 additions & 0 deletions src/Mono.Android/Properties/AssemblyInfo.cs.in
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,7 @@ using System.Runtime.CompilerServices;
[assembly: System.Runtime.CompilerServices.TypeForwardedToAttribute(typeof(System.Drawing.Size))]
[assembly: System.Runtime.CompilerServices.TypeForwardedToAttribute(typeof(System.Drawing.SizeF))]
[assembly: System.Runtime.CompilerServices.TypeForwardedToAttribute(typeof(System.Drawing.SystemColors))]
[assembly: InternalsVisibleTo("Mono.Android-Tests, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
[assembly: InternalsVisibleTo("Java.Interop-Tests, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
[assembly: InternalsVisibleTo("Mono.Android-TestsMultiDex, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
[assembly: InternalsVisibleTo("Mono.Android-TestsAppBundle, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
<GenerateSerializationAssemblies>Off</GenerateSerializationAssemblies>
<TargetFrameworkVersion>v10.0</TargetFrameworkVersion>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<SignAssembly>true</SignAssembly>
<AssemblyOriginatorKeyFile>..\..\..\..\product.snk</AssemblyOriginatorKeyFile>
</PropertyGroup>
<Import Project="..\..\..\..\Configuration.props" />
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
Expand Down Expand Up @@ -48,12 +50,11 @@
<Reference Include="Mono.Android" />
<Reference Include="Xamarin.Android.NUnitLite" />
<Reference Include="Mono.Linq.Expressions">
<HintPath>..\..\..\..\packages\mono.linq.expressions.2.0.0\lib\netstandard2.0\Mono.Linq.Expressions.dll</HintPath>
<HintPath>..\..\..\..\packages\Mono.Linq.Expressions.2.0.0\lib\netstandard2.0\Mono.Linq.Expressions.dll</HintPath>
</Reference>
</ItemGroup>
<ItemGroup>
<Compile Include="Resources\Resource.designer.cs" />
<Compile Include="Properties\AssemblyInfo.cs" />
<Compile Include="Java.InteropTests\JavaInterop_Tests_Reference.cs" />
</ItemGroup>
<ItemGroup>
Expand Down

This file was deleted.

26 changes: 11 additions & 15 deletions src/Mono.Android/Test/Java.Interop/JnienvTest.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using System;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading;

Expand Down Expand Up @@ -373,32 +374,27 @@ public void MoarThreadingTests ()
Assert.IsNull (ignore_t2, string.Format ("No exception should be thrown [t2]! Got: {0}", ignore_t2));
}

[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
static extern IntPtr monodroid_typemap_java_to_managed (string java);

[Test]
public void JavaToManagedTypeMapping ()
{
var m = monodroid_typemap_java_to_managed ("android/content/res/Resources");
Assert.AreNotEqual (IntPtr.Zero, m);
m = monodroid_typemap_java_to_managed ("this/type/does/not/exist");
Assert.AreEqual (IntPtr.Zero, m);
Type m = Java.Interop.TypeManager.GetJavaToManagedType ("android/content/res/Resources");
Assert.AreNotEqual (null, m);
m = Java.Interop.TypeManager.GetJavaToManagedType ("this/type/does/not/exist");
Assert.AreEqual (null, m);
}

[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
static extern IntPtr monodroid_typemap_managed_to_java (string java);

string GetTypeName (Type type)
{
return type.FullName + ", " + type.Assembly.GetName ().Name;
}
static extern IntPtr monodroid_typemap_managed_to_java (byte[] mvid, int token);

[Test]
public void ManagedToJavaTypeMapping ()
{
var m = monodroid_typemap_managed_to_java (GetTypeName (typeof (Activity)));
Type type = typeof(Activity);
var m = monodroid_typemap_managed_to_java (type.Module.ModuleVersionId.ToByteArray (), type.MetadataToken);
Assert.AreNotEqual (IntPtr.Zero, m, "`Activity` subclasses Java.Lang.Object, it should be in the typemap!");
m = monodroid_typemap_managed_to_java (GetTypeName (typeof (JnienvTest)));

type = typeof (JnienvTest);
m = monodroid_typemap_managed_to_java (type.Module.ModuleVersionId.ToByteArray (), type.MetadataToken);
Assert.AreEqual (IntPtr.Zero, m, "`JnienvTest` does *not* subclass Java.Lang.Object, it should *not* be in the typemap!");
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@
</ItemGroup>
<ItemGroup>
<Compile Include="Resources\Resource.designer.cs" />
<Compile Include="Properties\AssemblyInfo.cs" />
<Compile Include="CustomTextView.cs" />
</ItemGroup>
<ItemGroup>
Expand Down
Loading

0 comments on commit 18fd1d5

Please sign in to comment.