D8 and R8 integration #2019

jonathanpeppers · 2018-07-31T03:29:31Z

Fixes: #1423
Fixes: #2040

Overview

This enables d8 and r8 integration for Xamarin.Android. d8 is a
"next-generation" dex compiler (over dx), and r8 is a replacement for
proguard. For full details on the feature, see
Documentation/guides/D8andR8.md.

MSBuild targets changes

New MSBuild properties include:

$(AndroidDexTool) - an enum-style property with options: dx
and d8. Defaults to dx.
$(AndroidLinkTool) - an enum-style property, that can be left
blank to disable code shrinking. Valid values are proguard and
r8.

Existing MSBuild properties still retain the old behavior:

$(AndroidEnableProguard) will set $(AndroidLinkTool) to
proguard, although it is no longer used internally by
Xamarin.Android targets.
$(AndroidEnableDesugar) will default to true if d8 or r8 are
used.

New MSBuild tasks include:

<D8 /> - runs d8.jar with the required options for
Xamarin.Android.
<R8 /> - subclasses <D8 />, and adds functionality for multi-dex
and code-shrinking.

Additionally, any new MSBuild targets are placed in a new
Xamarin.Android.D8.targets file. This is good first step to make
Xamarin.Android.Common.targets smaller.

Additionally:

build.props is now invalidated via the $(AndroidDexTool)
and $(AndroidLinkTool) properties, instead of
$(AndroidEnableProguard)
build.props is invalidated via $(AndroidEnableDesugar)

Test changes

Tests that need to validate various combinations of properties are now
using parameters such as:

[Values ("dx", "d8")] string dexTool, [Values ("", "proguard", "r8")] string linkTool

Set on the XamarinAndroidApplicationProject such as:

var proj = new XamarinAndroidApplicationProject {
    DexTool = dexTool,
    LinkTool = linkTool,
};

In other cases, a simple useD8 flag was added to set DexTool="d8".

Since adding test parameters, exponentially causes our test cases to
expand, I removed some non-essential parameters:

BuildProguardEnabledProject dropped useLatestSdk, as it does not
seem applicable here (and is deprecated). Otherwise would have 24
test cases...
BuildApplicationWithSpacesInPath dropped isRelease and defaulted
it to true. We aren't going to likely encounter issues with spaces
in a path that happen only in a Debug build. Otherwise we would
have 24 test cases here...
Desugar dropped enableDesugar because it is certain this
application project will not build without desugar. We don't need
to test this, and would have 24 test cases otherwise...

Also dropped some [TestCaseSource] attributes where the [Values]
parameter was much cleaner.

Changes to test/sample projects

HelloWorld - $(AndroidDexTool) set to d8 if unspecified. So
we can track the performance benefit.

Xamarin.Forms Integration - uses d8 and $(AndroidLinkTool) set
to r8, using a proguard.cfg file for Xamarin.Forms. Will help us
track startup performance benefits of Java code shrinking and build
performance.

Mono.Android-Tests - uses d8 and $(AndroidLinkTool) set to r8,
to verify these on-device tests pass.

Runtime-MultiDex - $(AndroidDexTool) set to d8 if
unspecified, to validate multi-dex is working properly with d8.

Xamarin.Android build changes

This adds https://r8.googlesource.com/r8 as a submodule in
external/r8, and builds d8.jar and r8.jar via src/r8/r8.csproj.

Currently using version 1.2.50 of r8.

Deployment changes

Three new files will need to be included in Xamarin.Android
installers:

d8.jar
r8.jar
Xamarin.Android.D8.targets

General changes

Fixed doc for xa4304 error code
Added xa4305 error code for CompileToDalvik, R8, and
CreateMultiDexMainDexClassList
Removed log messages in CreateMultiDexMainDexClassList

Performance Comparison

MSBuild Target	Options Enabled	Time	APK size (bytes)	dex size (bytes)
_CompileToDalvikWithDx	n/a	11074ms	13378157	3894720
_CompileToDalvikWithD8	d8, (desugar enabled)	8543ms	13124205	3314064
_CompileToDalvikWithD8	d8, (desugar disabled)	9550ms	13124205	3314064
_CompileToDalvikWithDx	multi-dex	15632ms	13390498	3916496
_CompileToDalvikWithD8	d8, multi-dex	25979ms	13054626	3264096
_CompileToDalvikWithDx	proguard	11903ms	12804717	2446964
_CompileToDalvikWithD8	d8, r8	13799ms	12513901	1835588
_CompileToDalvikWithDx	multi-dex, proguard	17279ms	12804770	2449512
_CompileToDalvikWithD8	d8, multi-dex, r8	13792ms	12513954	1837588

NOTE: desugar is enabled by default with d8/r8

I timed this builds with this script, with a "Hello World" Xamarin.Forms app. Build logs here: d8andr8.zip

One can draw their own conclusions on which options are faster, better, smaller.

Some of my thoughts:

Default options for d8 and r8 seem to be faster?
Disabling desugar is slower?
Enabling multi-dex makes the dex file larger, because new classes are required. The app wasn't large enough to warrant a classes2.dex.
d8 does not support multi-dex, and so choosing d8 + multi-dex actually runs r8 with --no-tree-shaking --no-minification. These options are slower?

dellis1972 · 2018-07-31T12:11:07Z

@jonathanpeppers looks ok so far. I do think we need to make sure we test the new R8 stuff on windows where the path has spaces. Historically, the .bat files shipped by google are really bad a handling those cases. We just need to make sure we can rely on the .bat and not have to call the various tooling directly.

jonathanpeppers · 2018-07-31T18:33:02Z

Currently hitting this:

adb: failed to install Mono.Android_Tests-Signed.apk: Failure [INSTALL_FAILED_INVALID_APK: Package couldn't be installed in /data/app/Mono.Android_Tests-MLAbuMOK1C8r3HkS2FKmHg==: Pac
  kage /data/app/Mono.Android_Tests-MLAbuMOK1C8r3HkS2FKmHg==/base.apk code is missing]

classes.dex is missing from the APK!

jonathanpeppers · 2018-07-31T21:41:57Z

I think the PR build is actually really green now!

To get the right warning codes, I think I need #1986 merged so I can use XA4304.

Then I think all that is left here is some cleanup, and I can remove the do-not-merge label.

dellis1972 · 2018-08-01T08:39:47Z

Merged #1986

…

On 31 July 2018 at 22:42, Jonathan Peppers ***@***.***> wrote: I think the PR build is actually *really green* now! To get the right warning codes, I think I need #1986 <#1986> merged so I can use XA4304. Then I think all that is left here is some cleanup, and I can remove the do-not-merge label. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#2019 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAxeeYHShbrThh5PVBEzvuBvv2qlQ5A0ks5uMM8sgaJpZM4Vnijr> .

jonathanpeppers · 2018-08-01T14:29:36Z

Ok been doing more testing today... And I started seeing some odd behavior when switching the multidex options around.

I expect things to look like this:

But in some situations, classes2.dex is not there with multidex enabled, and I get warnings from r8:

Warnings
    R8 Application does not contain `android.support.multidex.MultiDex$V14` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.ZipUtil$CentralDirectory` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.MultiDex$V19` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.MultiDex` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.MultiDexExtractor` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.MultiDexApplication` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.ZipUtil` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.MultiDexExtractor$1` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]
    R8 Application does not contain `android.support.multidex.MultiDex$V4` as referenced in main-dex-list. [xamarin-android\bin\TestDebug\temp\BuildReleaseAppWithA InIt(FalseTrueTrueTrue)\UnnamedProject.csproj]

Looks like this PR is not running d8.jar at all, which doesn't seem quite right... It should be a two-step process where d8.jar is run, followed by r8.jar if proguard enabled, right?

I also noticed usage of the Debug $(Configuration), so I believe this should instead use $(AndroidIncludeDebugSymbols): https://github.com/xamarin/xamarin-android/pull/2019/files#diff-79b3c1185394482eea2f7d5553e3e5d4R67

You can name your configurations whatever you like. We should also specify --debug and --release explicitly.

jonathanpeppers · 2018-08-01T14:36:08Z

Ok actually, I think we should be running d8 or r8: https://r8.googlesource.com/r8/#running-d8

d8 - if proguard is off
r8 - if proguard is on

So I need to make a new <D8 /> MSBuild task.

For reference, d8 --help:

Usage: d8 [options] <input-files>
 where <input-files> are any combination of dex, class, zip, jar, or apk files
 and options are:
  --debug                 # Compile with debugging information (default).
  --release               # Compile without debugging information.
  --output <file>         # Output result in <outfile>.
                          # <file> must be an existing directory or a zip file.
  --lib <file>            # Add <file> as a library resource.
  --classpath <file>      # Add <file> as a classpath resource.
  --min-api               # Minimum Android API level compatibility
  --intermediate          # Compile an intermediate result intended for later
                          # merging.
  --file-per-class        # Produce a separate dex file per input class
  --no-desugaring         # Force disable desugaring.
  --main-dex-list <file>  # List of classes to place in the primary dex file.
  --version               # Print the version of d8.
  --help                  # Print this message.

For reference, r8 --help:

Usage: r8 [options] <input-files>
 where <input-files> are any combination of dex, class, zip, jar, or apk files
 and options are:
  --release                # Compile without debugging information (default).
  --debug                  # Compile with debugging information.
  --output <file>          # Output result in <file>.
                           # <file> must be an existing directory or a zip file.
  --lib <file>             # Add <file> as a library resource.
  --min-api                # Minimum Android API level compatibility.
  --pg-conf <file>         # Proguard configuration <file>.
  --pg-map-output <file>   # Output the resulting name and line mapping to <file>.
  --no-tree-shaking        # Force disable tree shaking of unreachable classes.
  --no-minification        # Force disable minification of names.
  --no-desugaring          # Force disable desugaring.
  --main-dex-rules <file>  # Proguard keep rules for classes to place in the
                           # primary dex file.
  --main-dex-list <file>   # List of classes to place in the primary dex file.
  --main-dex-list-output <file>  # Output the full main-dex list in <file>.
  --version                # Print the version of r8.
  --help                   # Print this message.

jonathanpeppers · 2018-09-26T19:01:25Z

I'm going to start changing this PR drastically, rebasing, squashing, etc.

If we ever need the history of what was originally here: master...jonathanpeppers:d8-and-r8-history

jonathanpeppers · 2018-10-24T00:47:55Z

Changes to android-toolchain seem to have worked, from the macOS build log:

Target Name=_UnzipAndroidSdkItems Project=android-toolchain.csproj
  Skipping target "_UnzipAndroidSdkItems" because all output files are up-to-date with respect to the input files.
Target Name=_UnzipAndroidNdkItems Project=android-toolchain.csproj
  Skipping target "_UnzipAndroidNdkItems" because all output files are up-to-date with respect to the input files.
Target Name=_UnzipAntItems Project=android-toolchain.csproj
  Skipping target "_UnzipAntItems" because all output files are up-to-date with respect to the input files.
Target Name=_UnzipChromeItems Project=android-toolchain.csproj
  Building target "_UnzipChromeItems" completely.
  Output file "/Users/builder/android-toolchain/depot_tools/.stamp-depot_tools-crc32c=8HATIw==" does not exist.

jonathanpeppers · 2018-10-24T17:08:58Z

Split out part of the changes in the last commit, for a future PR, diff is smaller now:

Before:
+1,189 −278
After:
+1,136 −233

jonathanpeppers · 2018-10-24T19:09:37Z

I think this build was green, the test failures are network-related, or a file sharing violation (assume it's a new bot?). I added PR 2343 to hopefully help the "flakiness" of some of these tests.

Fixes: dotnet#1423 Fixes: dotnet#2040 ~~ Overview ~~ This enables d8 and r8 integration for Xamarin.Android. d8 is a "next-generation" dex compiler (over dx), and r8 is a replacement for proguard. For full details on the feature, see `Documentation/guides/D8andR8.md`. ~~ MSBuild targets changes ~~ New MSBuild properties include: - `$(AndroidDexTool)` - an enum-style property with options: `dx` and `d8`. Defaults to `dx`. - `$(AndroidLinkTool)` - an enum-style property, that can be left blank to disable code shrinking. Valid values are `proguard` and `r8`. Existing MSBuild properties still retain the old behavior: - `$(AndroidEnableProguard)` will set `$(AndroidLinkTool)` to `proguard`, although it is no longer used internally by Xamarin.Android targets. - `$(AndroidEnableDesugar)` will default to `true` if `d8` or `r8` are used. New MSBuild tasks include: - `<D8 />` - runs `d8.jar` with the required options for Xamarin.Android. - `<R8 />` - subclasses `<D8 />`, and adds functionality for multi-dex and code-shrinking. Additionally, any new MSBuild targets are placed in a new `Xamarin.Android.D8.targets` file. This is good first step to make `Xamarin.Android.Common.targets` smaller. Additionally: * `build.props` is now invalidated via the `$(AndroidDexTool)` and `$(AndroidLinkTool)` properties, instead of `$(AndroidEnableProguard)` * `build.props` is invalidated via `$(AndroidEnableDesugar)` ~~ Test changes ~~ Tests that need to validate various combinations of properties are now using parameters such as: [Values ("dx", "d8")] string dexTool, [Values ("", "proguard", "r8")] string linkTool Set on the `XamarinAndroidApplicationProject` such as: var proj = new XamarinAndroidApplicationProject { DexTool = dexTool, LinkTool = linkTool, }; In other cases, a simple `useD8` flag was added to set `DexTool="d8"`. Since adding test parameters, exponentially causes our test cases to expand, I removed some non-essential parameters: - `BuildProguardEnabledProject` dropped `useLatestSdk`, as it does not seem applicable here (and is deprecated). Otherwise would have 24 test cases... - `BuildApplicationWithSpacesInPath` dropped `isRelease` and defaulted it to `true`. We aren't going to likely encounter issues with spaces in a path that happen *only* in a `Debug` build. Otherwise we would have 24 test cases here... - `Desugar` dropped `enableDesugar` because it is certain this application project will not build *without* desugar. We don't need to test this, and would have 24 test cases otherwise... Also dropped some `[TestCaseSource]` attributes where the `[Values]` parameter was much cleaner. ~~ Changes to test/sample projects ~~ `HelloWorld` - `$(AndroidDexTool)` set to `d8` if unspecified. So we can track the performance benefit. `Xamarin.Forms Integration` - uses `d8` and `$(AndroidLinkTool)` set to `r8`, using a `proguard.cfg` file for Xamarin.Forms. Will help us track startup performance benefits of Java code shrinking and build performance. `Mono.Android-Tests` - uses `d8` and `$(AndroidLinkTool)` set to `r8`, to verify these on-device tests pass. `Runtime-MultiDex` - `$(AndroidDexTool)` set to `d8` if unspecified, to validate multi-dex is working properly with `d8`. ~~ Xamarin.Android build changes ~~ This adds https://r8.googlesource.com/r8 as a submodule in `external/r8`, and builds `d8.jar` and `r8.jar` via `src/r8/r8.csproj`. Currently using version 1.2.50 of r8. ~~ Deployment changes ~~ Three new files will need to be included in Xamarin.Android installers: - `d8.jar` - `r8.jar` - `Xamarin.Android.D8.targets` ~~ General changes ~~ * Fixed doc for `xa4304` error code * Added `xa4305` error code for `CompileToDalvik`, `R8`, and `CreateMultiDexMainDexClassList` * Removed log messages in `CreateMultiDexMainDexClassList` ~~ Performance Comparison ~~ | MSBuild Target | Options Enabled | Time | APK size (bytes) | dex size (bytes) | | --- | --- | ---: | ---: | ---: | | _CompileToDalvikWithDx | n/a | 11074ms | 13378157 | 3894720 | | _CompileToDalvikWithD8 | d8, (desugar enabled) | 8543ms | 13124205 | 3314064 | | _CompileToDalvikWithD8 | d8, (desugar disabled) | 9550ms | 13124205 | 3314064 | | _CompileToDalvikWithDx | multi-dex | 15632ms | 13390498 | 3916496 | | _CompileToDalvikWithD8 | d8, multi-dex | 25979ms | 13054626 | 3264096 | | _CompileToDalvikWithDx | proguard | 11903ms | 12804717 | 2446964 | | _CompileToDalvikWithD8 | d8, r8 | 13799ms | 12513901 | 1835588 | | _CompileToDalvikWithDx | multi-dex, proguard | 17279ms | 12804770 | 2449512 | | _CompileToDalvikWithD8 | d8, multi-dex, r8 | 13792ms | 12513954 | 1837588 | _NOTE: desugar is enabled by default with d8/r8_ I timed this builds with [this script][powershell_script], with a "Hello World" Xamarin.Forms app. Build logs here: [d8andr8.zip][d8andr8_zip] One can draw their own conclusions on which options are faster, better, smaller. See further detail in `D8andR8.md`. [powershell_script]: https://github.com/jonathanpeppers/HelloWorld/blob/39e2854f6ca39c0941fb8bd6f2a16d8b7663003e/build.ps1 [d8andr8_zip]: https://github.com/xamarin/xamarin-android/files/2470385/d8andr8.zip Co-authored-by: Atsushi Eno <atsushieno@gmail.com>

~~ xa-prep-tasks ~~ The `<DownloadUri/>` MSBuild task now has an optional `HashHeader` property: - If set, a http HEAD request is made to the URL, and the destination file path gets a suffix added for the value. - To make this work properly, `DestinationFiles` is also an `[Output]` property, since these paths could change as a result. ~~ android-toolchain ~~ Chromium's `depot_tools` are downloaded the same as all of our other dependencies. I had to update the `<DownloadUri/>` MSBuild task to have an optional `HashHeader` property and made `DestinationFiles` an `<Output/>`. We can use this to populate `@(_DownloadedItem)`. `_UnzipFiles` also "bootstraps" `depot_tools`, and has to set/unset `$PATH` appropriately. I did this in the same place as the license acceptance for the Android SDK. ~~ r8.targets ~~ - Removed all of the `depot_tools` downloading code, and setup `android-toolchain` as a `@(ProjectReference)` - All targets that set `$PATH`, unset it when finished - Use `$(_GradleArgs)` to supply `--no-daemon` - `_CleanR8` should delete `*.jar` files in our build tree ~~ Xamarin.Android.Build.Tasks ~~ - Fixed any formatting that we noticed - Renamed `$(AndroidD8JarPath)` and `$(AndroidR8JarPath)` to be prefixed with `Android` - Refactored `$(IntermediateOutputPath)_dex_stamp` stamp file in `_CompileToDalvikWithDx` and `_CompileToDalvikWithD8` to match the new convention of `$(_AndroidStampDirectory)_CompileToDalvik.stamp` - `*.dex` files weren't in `FileWrites`??? - `CompileToDalvik` had a `DexOutputs` output property that was completely unused, so I removed it. Also removed extra log messages.

Context: dotnet#2019 Since 4bb4b2e, our builds have been failing on VSTS on macOS and Windows. Jenkins is green, however. ~~ macOS ~~ Currently getting a failure building `r8` such as: 2018-10-30T20:19:05.2225960Z * What went wrong: 2018-10-30T20:19:05.2227400Z Could not determine java version from '11.0.1'. So I believe the problem here is that Java 11 is in `$PATH`, and this is only a scenario on the VSTS macOS build agents. It turns out I was able to clean up quite a bit in `r8.targets`: - We don't need to call `SetEnvironmentVariable`, I was able to set `EnvironmentVariables="...;PATH=$(ChromeToolsDirectory)"` and things still worked fine. - We should pass `--stacktrace` to gradle so that we get detailed error messages on a failure. ~~ Windows ~~ Currently getting a failure in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj` such as: 2018-10-31T14:02:28.8336608Z .\gradlew assembleDebug --stacktrace --no-daemon 2018-10-31T14:02:40.7122197Z NDK is missing a "platforms" directory. 2018-10-31T14:02:40.7122937Z If you are using NDK, verify the ndk.dir is set to a valid NDK directory. It is currently set to C:\Users\dlab14\android-toolchain\sdk\ndk-bundle. 2018-10-31T14:02:40.7124870Z If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning. It doesn't really make sense to me why this started happening with d8/r8 support... The `~\android-toolchain\sdk\ndk-bundle` path seems completely wrong. However, it looks like we should be setting `ANDROID_NDK_HOME=$(AndroidNdkDirectory)`. This is the only other place we are calling `gradle`, so it is somehow *related* to the d8/r8 changes.

Context: dotnet#2019 Context: http://build.devdiv.io/2168276 Context: http://build.devdiv.io/2169681 Since 4bb4b2e, our builds have been failing on VSTS on Windows. Jenkins is green, however. Currently getting a failure in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj` such as: 2018-10-31T14:02:28.8336608Z .\gradlew assembleDebug --stacktrace --no-daemon 2018-10-31T14:02:40.7122197Z NDK is missing a "platforms" directory. 2018-10-31T14:02:40.7122937Z If you are using NDK, verify the ndk.dir is set to a valid NDK directory. It is currently set to C:\Users\dlab14\android-toolchain\sdk\ndk-bundle. 2018-10-31T14:02:40.7124870Z If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning. It doesn't really make sense to me why this started happening with d8/r8 support... The `~\android-toolchain\sdk\ndk-bundle` path seems completely wrong. However, it looks like we should be setting `ANDROID_NDK_HOME=$(AndroidNdkDirectory)`. After that change, we got a new error: 2018-10-31T17:19:30.1558223Z Checking the license for package Android SDK Build-Tools 25.0.2 in C:\Users\dlab14\android-toolchain\sdk\licenses 2018-10-31T17:19:30.1562674Z EXEC : warning : License for package Android SDK Build-Tools 25.0.2 not accepted. [E:\A\_work\14\s\tests\CodeGen-Binding\Xamarin.Android.LibraryProjectZip-LibBinding\Xamarin.Android.LibraryProjectZip-LibBinding.csproj] So this is the actual problem! In fa57aa8, I changed `android-toolchain` to not run `<AcceptAndroidSdkLicense/>` all the time. That was actually making this `.\gradlew` command successfully install Android SDK Build-Tools 25.0.2 and use it without a license prompt. We started seeing the problem since 4bb4b2e, because it caused a new `~\android-toolchain\sdk` directory to get setup. The fix here is to just use Build-Tools 28.0.0, since that is the version we boostrap and install during our build. Other changes: - Added `--stacktrace` to the gradle calls in `r8.targets`, so we get better error messages. It also matches the gradle command in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj`.

Context: dotnet#2019 Context: http://build.devdiv.io/2168276 Context: http://build.devdiv.io/2169681 Since 4bb4b2e, our builds have been failing on VSTS on Windows. Jenkins is green, however. Currently getting a failure in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj` such as: 2018-10-31T14:02:28.8336608Z .\gradlew assembleDebug --stacktrace --no-daemon 2018-10-31T14:02:40.7122197Z NDK is missing a "platforms" directory. 2018-10-31T14:02:40.7122937Z If you are using NDK, verify the ndk.dir is set to a valid NDK directory. It is currently set to C:\Users\dlab14\android-toolchain\sdk\ndk-bundle. 2018-10-31T14:02:40.7124870Z If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning. It doesn't really make sense to me why this started happening with d8/r8 support... The `~\android-toolchain\sdk\ndk-bundle` path seems completely wrong. However, it looks like we should be setting `ANDROID_NDK_HOME=$(AndroidNdkDirectory)`. After that change, we got a new error: 2018-10-31T17:19:30.1558223Z Checking the license for package Android SDK Build-Tools 25.0.2 in C:\Users\dlab14\android-toolchain\sdk\licenses 2018-10-31T17:19:30.1562674Z EXEC : warning : License for package Android SDK Build-Tools 25.0.2 not accepted. [E:\A\_work\14\s\tests\CodeGen-Binding\Xamarin.Android.LibraryProjectZip-LibBinding\Xamarin.Android.LibraryProjectZip-LibBinding.csproj] So this is the actual problem! In fa57aa8, I changed `android-toolchain` to not run `<AcceptAndroidSdkLicense/>` all the time. That was actually making this `.\gradlew` command successfully install Android SDK Build-Tools 25.0.2 and use it without a license prompt. We started seeing the problem since 4bb4b2e, because it caused a new `~\android-toolchain\sdk` directory to get setup. The fix here is to just use Build-Tools 28.0.0, since that is the version we boostrap and install during our build. But we would never actually remember to update this value... So we should make a `build.gradle.in` file, and replace with our existing property from `Configuration.props`: <XABuildToolsFolder Condition="'$(XABuildToolsFolder)' == ''">28.0.0</XABuildToolsFolder> Other changes: - Added `--stacktrace` to the gradle calls in `r8.targets`, so we get better error messages. It also matches the gradle command in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj`.

Context: http://build.devdiv.io/2168276 Context: http://build.devdiv.io/2169681 Context: #2019 Since 4bb4b2e, our builds have been failing on VSTS on Windows. Jenkins+macOS & Linux is green, however. Windows is currently getting a failure in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj` such as: .\gradlew assembleDebug --stacktrace --no-daemon NDK is missing a "platforms" directory. If you are using NDK, verify the ndk.dir is set to a valid NDK directory. It is currently set to C:\Users\dlab14\android-toolchain\sdk\ndk-bundle. If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning. It doesn't really make sense to me why this started happening with d8/r8 support. The `~\android-toolchain\sdk\ndk-bundle` path seems completely wrong. Regardless, it looks like we should be setting `ANDROID_NDK_HOME=$(AndroidNdkDirectory)`. After that change, we got a new error: Checking the license for package Android SDK Build-Tools 25.0.2 in C:\Users\dlab14\android-toolchain\sdk\licenses EXEC : warning : License for package Android SDK Build-Tools 25.0.2 not accepted. [E:\A\_work\14\s\tests\CodeGen-Binding\Xamarin.Android.LibraryProjectZip-LibBinding\Xamarin.Android.LibraryProjectZip-LibBinding.csproj] So this is the actual problem! In fa57aa8, I changed `android-toolchain` to not run `<AcceptAndroidSdkLicense/>` all the time. That was actually making this `.\gradlew` command successfully install Android SDK Build-Tools 25.0.2 and use it without a license prompt. We started seeing the problem since 4bb4b2e, because it caused a new `~\android-toolchain\sdk` directory to get created. The fix here is to just use Build-Tools 28.0.0, since that is the version we bootstrap and install during our build. This requires updating the `buildToolsVersion` value within `build.gradle`. However, we would never actually remember to update this value. To simplify maintenance, make a `build.gradle.in` which sets `buildToolsVersion` to `@BUILD_TOOLS_VERSION@`, and replace `@BUILD_TOOLS_VERSION@` with `$(XABuildToolsFolder)`, specified within `Configuration.props`. Additionally: add `--stacktrace` to the gradle calls within `r8.targets`, so we get better error messages. It also matches the `gradle` command in `Xamarin.Android.LibraryProjectZip-LibBinding.csproj`.

tests/Xamarin.Forms-Performance-Integration/Droid/proguard.cfg

gmoney494 · 2018-11-21T17:44:53Z

@jonathanpeppers @jonpryor how long until this is released?

jonathanpeppers · 2018-11-21T18:27:26Z

@gmoney494 it will be in preview 2 of the next Visual Studio. Unfortunately I have no idea if there is a public date for that.

If you really need this feature, try one of the download links on the homepage for commercial Xamarin.Android. You need 9.1.199.x or greater.

gmoney494 · 2018-11-29T18:31:33Z

@jonathanpeppers is there a good documentation to install it externally instead of waiting for the preview? I downloaded the latest .pkg (i have a Mac) and installed it, but i run into 'ResolveLibraryProjectImports' error, it seems its missing the new system dependencies. Any help here?

jonathanpeppers · 2018-11-30T20:31:59Z

@gmoney494 I heard from someone else they had trouble installing the latest master build on top of Visual Studio Windows (stable).

What is the error message you got on Mac?

gmoney494 · 2018-11-30T21:21:04Z

Well i get the installer to show and go through installing no problem. But when i try to do a build, i get this error:

/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(2,2): Error MSB4018: The "ResolveLibraryProjectImports" task failed unexpectedly. System.DllNotFoundException: libzip.5.0.dylib at (wrapper managed-to-native) Xamarin.Tools.Zip.Native.zip_open(intptr,Xamarin.Tools.Zip.OpenFlags,Xamarin.Tools.Zip.ErrorCode&) at Xamarin.Tools.Zip.Native.zip_open (System.String path, Xamarin.Tools.Zip.OpenFlags flags, Xamarin.Tools.Zip.ErrorCode& errorp) [0x00007] in <f520cbcb66df4e7b86c625f491fb4cc0>:0 at Xamarin.Tools.Zip.ZipArchive.Open (System.String path, Xamarin.Tools.Zip.OpenFlags flags) [0x00009] in <f520cbcb66df4e7b86c625f491fb4cc0>:0 at Xamarin.Tools.Zip.ZipArchive.Open (System.String path, System.IO.FileMode mode, System.String defaultExtractionDir, System.Boolean strictConsistencyChecks, Xamarin.Tools.Zip.IPlatformOptions options) [0x0005a] in <f520cbcb66df4e7b86c625f491fb4cc0>:0 at Xamarin.Android.Tools.Files.ReadZipFile (System.String filename, System.Boolean strictConsistencyChecks) [0x00000] in <8fec845047d140a1b41722f962b81712>:0 at Xamarin.Android.Tasks.MonoAndroidHelper.ReadZipFile (System.String filename) [0x00000] in <8fec845047d140a1b41722f962b81712>:0 at Xamarin.Android.Tasks.ResolveLibraryProjectImports.Extract (Java.Interop.Tools.Cecil.DirectoryAssemblyResolver res, System.Collections.Generic.ICollection1[T] jars, System.Collections.Generic.ICollection1[T] resolvedResourceDirectories, System.Collections.Generic.ICollection1[T] resolvedAssetDirectories, System.Collections.Generic.ICollection1[T] resolvedEnvironments) [0x006cd] in <8fec845047d140a1b41722f962b81712>:0 at Xamarin.Android.Tasks.ResolveLibraryProjectImports.Execute () [0x00058] in <8fec845047d140a1b41722f962b81712>:0 at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute () [0x00023] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-06/external/bockbuild/builds/msbuild-15/src/Build/BackEnd/TaskExecutionHost/TaskExecutionHost.cs:573 at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask (Microsoft.Build.BackEnd.ITaskExecutionHost taskExecutionHost, Microsoft.Build.BackEnd.Logging.TaskLoggingContext taskLoggingContext, Microsoft.Build.BackEnd.TaskHost taskHost, Microsoft.Build.BackEnd.ItemBucket bucket, Microsoft.Build.BackEnd.TaskExecutionMode howToExecuteTask) [0x001f6] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-06/external/bockbuild/builds/msbuild-15/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs:784 (MSB4018) (***.Forms.Android)

Which is funny, because i can manually build it from termninal successfully with the /xamarin.android-oss_v9.1.199.17_Darwin-x86_64_master_79b220a0-Release/bin/Release/bin/xabuild /t:SignAndroidPackage {path to Forms.Android.csproj} @jonathanpeppers

dellis1972 · 2018-11-30T22:51:45Z

this might be that the current stable VSForMac is trying to run the build in 32bit more but the libzip .dylib is only 64bit now. which would explain why it would work from the terminal.

gmoney494 · 2018-12-03T16:31:16Z

@dellis1972 is there a way to specify Vs to use the 64 bit?
or get a libzip thats 32 bit?
or to get the build to run on my device from the terminal via commands?

jonathanpeppers · 2018-12-03T17:44:59Z

@gmoney494 /t:Install will deploy command line.

flchaux · 2018-12-11T18:51:28Z

Hello @jonathanpeppers @dellis1972, any workaround to build in 64bits in VSForMac ? I encounter the same issue and need the debugger.

gmoney494 · 2018-12-14T15:56:03Z

i got the install to work on my device, i have limited debugger support (have to use the old Android studio log cat to see anything in the console). However, i would need the breakpoint functionality to fully debug my code. Would the 2019 VS pre-release for Mac include these fixes/changes? There has not been a new release on the Alpha or Beta channels on my mac for months now. Just this latest security patch. Luckily my deadlines have been pushed back (for other reasons), but this has been a literal stopper for me.

flchaux · 2018-12-14T16:09:26Z

Yes ! VS 2019 for Mac include theses changes. You have to download VS pre-release. It works on my side.

wiwski · 2018-12-25T20:48:33Z

Hello @joelmartinez @dellis1972
I've been trying to use this feature with VS2017 by downloading Xamarin.Android 9.1.99 following Building Xamarin.Android on Windows.
I then set :
<AndroidDexTool>d8</AndroidDexTool>
<AndroidLinkTool>r8</AndroidLinkTool>

But when I checkthe output log, I see that CompileToDalvik is being called and not _CompileToDalvikWithD8.

I know that I'm doing something wrong but I don't know what it is and where to get help.
Thank you in advance

EDIT: I used the installer from the Download section and it worked well

jonathanpeppers · 2019-01-02T14:44:17Z

Hi @wiwski I think you need 9.1.199.x: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-d16-0/

I would only use it on top of the Visual Studio 2019 preview. When 2019 Preview 2 ships, you can just use that, but it is not out yet.

jonathanpeppers added do-not-merge PR should not be merged. use-merge-commit Normally we squash-and-merge PRs. Use this label so we instead use a merge commit. labels Jul 31, 2018

jonathanpeppers requested review from jonpryor and dellis1972 July 31, 2018 03:29

jonathanpeppers mentioned this pull request Jul 31, 2018

R8 integration #1489

Closed

jonathanpeppers force-pushed the d8-and-r8 branch 3 times, most recently from 3766aa7 to 7681387 Compare July 31, 2018 04:49

jonathanpeppers force-pushed the d8-and-r8 branch from 7681387 to 59c68e7 Compare July 31, 2018 15:03

jonathanpeppers force-pushed the d8-and-r8 branch from 59c68e7 to 9b5436d Compare July 31, 2018 19:22

jonathanpeppers mentioned this pull request Aug 6, 2018

D8 and R8 Integration #2040

Closed

jonathanpeppers force-pushed the d8-and-r8 branch from 9b5436d to 281012b Compare September 25, 2018 21:11

jonathanpeppers force-pushed the d8-and-r8 branch 4 times, most recently from 7b727ef to 51f12d8 Compare September 27, 2018 19:43

jonathanpeppers added run-tests-release and removed use-merge-commit Normally we squash-and-merge PRs. Use this label so we instead use a merge commit. labels Sep 27, 2018

jonathanpeppers force-pushed the d8-and-r8 branch 2 times, most recently from 1b6fdcb to 4d3fd2a Compare September 28, 2018 20:14

jonathanpeppers force-pushed the d8-and-r8 branch from 5b21404 to 1f0a82a Compare October 9, 2018 20:57

jonathanpeppers mentioned this pull request Oct 9, 2018

Added --classpath_entry for Desugar #2158

Merged

jonathanpeppers force-pushed the d8-and-r8 branch from 1f0a82a to d5363e8 Compare October 11, 2018 19:26

jonathanpeppers force-pushed the d8-and-r8 branch from acbeaac to 55b88a8 Compare October 24, 2018 17:07

jonathanpeppers and others added 2 commits October 29, 2018 13:34

jonathanpeppers force-pushed the d8-and-r8 branch from 55b88a8 to 2e8acea Compare October 29, 2018 18:37

jonpryor merged commit 4bb4b2e into dotnet:master Oct 30, 2018

jonathanpeppers mentioned this pull request Oct 31, 2018

[r8] fixes to building with gradle #2364

Closed

jonathanpeppers mentioned this pull request Nov 1, 2018

[build] fixes for failing Windows build #2368

Merged

radekdoulik reviewed Nov 7, 2018

View reviewed changes

tests/Xamarin.Forms-Performance-Integration/Droid/proguard.cfg Show resolved Hide resolved

jonathanpeppers deleted the d8-and-r8 branch November 30, 2018 20:32

github-actions bot locked and limited conversation to collaborators Feb 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

D8 and R8 integration #2019

D8 and R8 integration #2019

jonathanpeppers commented Jul 31, 2018 •

edited

Loading

dellis1972 commented Jul 31, 2018

jonathanpeppers commented Jul 31, 2018 •

edited

Loading

jonathanpeppers commented Jul 31, 2018

dellis1972 commented Aug 1, 2018 via email

jonathanpeppers commented Aug 1, 2018

jonathanpeppers commented Aug 1, 2018

jonathanpeppers commented Sep 26, 2018

jonathanpeppers commented Oct 24, 2018

jonathanpeppers commented Oct 24, 2018 •

edited

Loading

jonathanpeppers commented Oct 24, 2018 •

edited

Loading

gmoney494 commented Nov 21, 2018

jonathanpeppers commented Nov 21, 2018

gmoney494 commented Nov 29, 2018

jonathanpeppers commented Nov 30, 2018

gmoney494 commented Nov 30, 2018 •

edited

Loading

dellis1972 commented Nov 30, 2018

gmoney494 commented Dec 3, 2018

jonathanpeppers commented Dec 3, 2018

flchaux commented Dec 11, 2018

gmoney494 commented Dec 14, 2018

flchaux commented Dec 14, 2018

wiwski commented Dec 25, 2018 •

edited

Loading

jonathanpeppers commented Jan 2, 2019

D8 and R8 integration #2019

D8 and R8 integration #2019

Conversation

jonathanpeppers commented Jul 31, 2018 • edited Loading

Overview

MSBuild targets changes

Test changes

Changes to test/sample projects

Xamarin.Android build changes

Deployment changes

General changes

Performance Comparison

dellis1972 commented Jul 31, 2018

jonathanpeppers commented Jul 31, 2018 • edited Loading

jonathanpeppers commented Jul 31, 2018

dellis1972 commented Aug 1, 2018 via email

jonathanpeppers commented Aug 1, 2018

jonathanpeppers commented Aug 1, 2018

jonathanpeppers commented Sep 26, 2018

jonathanpeppers commented Oct 24, 2018

jonathanpeppers commented Oct 24, 2018 • edited Loading

jonathanpeppers commented Oct 24, 2018 • edited Loading

gmoney494 commented Nov 21, 2018

jonathanpeppers commented Nov 21, 2018

gmoney494 commented Nov 29, 2018

jonathanpeppers commented Nov 30, 2018

gmoney494 commented Nov 30, 2018 • edited Loading

dellis1972 commented Nov 30, 2018

gmoney494 commented Dec 3, 2018

jonathanpeppers commented Dec 3, 2018

flchaux commented Dec 11, 2018

gmoney494 commented Dec 14, 2018

flchaux commented Dec 14, 2018

wiwski commented Dec 25, 2018 • edited Loading

jonathanpeppers commented Jan 2, 2019

jonathanpeppers commented Jul 31, 2018 •

edited

Loading

jonathanpeppers commented Jul 31, 2018 •

edited

Loading

jonathanpeppers commented Oct 24, 2018 •

edited

Loading

jonathanpeppers commented Oct 24, 2018 •

edited

Loading

gmoney494 commented Nov 30, 2018 •

edited

Loading

wiwski commented Dec 25, 2018 •

edited

Loading