From c8719f038944d2008e8907074151f8d57e33de44 Mon Sep 17 00:00:00 2001 From: Jonathan Peppers Date: Thu, 29 Mar 2018 10:31:59 +0900 Subject: [PATCH 1/2] [Xamarin.Android.Build.Tasks] d8 and r8 support Fixes: https://github.com/xamarin/xamarin-android/issues/1423 Fixes: https://github.com/xamarin/xamarin-android/issues/2040 ~~ Overview ~~ This enables d8 and r8 integration for Xamarin.Android. d8 is a "next-generation" dex compiler (over dx), and r8 is a replacement for proguard. For full details on the feature, see `Documentation/guides/D8andR8.md`. ~~ MSBuild targets changes ~~ New MSBuild properties include: - `$(AndroidDexTool)` - an enum-style property with options: `dx` and `d8`. Defaults to `dx`. - `$(AndroidLinkTool)` - an enum-style property, that can be left blank to disable code shrinking. Valid values are `proguard` and `r8`. Existing MSBuild properties still retain the old behavior: - `$(AndroidEnableProguard)` will set `$(AndroidLinkTool)` to `proguard`, although it is no longer used internally by Xamarin.Android targets. - `$(AndroidEnableDesugar)` will default to `true` if `d8` or `r8` are used. New MSBuild tasks include: - `` - runs `d8.jar` with the required options for Xamarin.Android. - `` - subclasses ``, and adds functionality for multi-dex and code-shrinking. Additionally, any new MSBuild targets are placed in a new `Xamarin.Android.D8.targets` file. This is good first step to make `Xamarin.Android.Common.targets` smaller. Additionally: * `build.props` is now invalidated via the `$(AndroidDexTool)` and `$(AndroidLinkTool)` properties, instead of `$(AndroidEnableProguard)` * `build.props` is invalidated via `$(AndroidEnableDesugar)` ~~ Test changes ~~ Tests that need to validate various combinations of properties are now using parameters such as: [Values ("dx", "d8")] string dexTool, [Values ("", "proguard", "r8")] string linkTool Set on the `XamarinAndroidApplicationProject` such as: var proj = new XamarinAndroidApplicationProject { DexTool = dexTool, LinkTool = linkTool, }; In other cases, a simple `useD8` flag was added to set `DexTool="d8"`. Since adding test parameters, exponentially causes our test cases to expand, I removed some non-essential parameters: - `BuildProguardEnabledProject` dropped `useLatestSdk`, as it does not seem applicable here (and is deprecated). Otherwise would have 24 test cases... - `BuildApplicationWithSpacesInPath` dropped `isRelease` and defaulted it to `true`. We aren't going to likely encounter issues with spaces in a path that happen *only* in a `Debug` build. Otherwise we would have 24 test cases here... - `Desugar` dropped `enableDesugar` because it is certain this application project will not build *without* desugar. We don't need to test this, and would have 24 test cases otherwise... Also dropped some `[TestCaseSource]` attributes where the `[Values]` parameter was much cleaner. ~~ Changes to test/sample projects ~~ `HelloWorld` - `$(AndroidDexTool)` set to `d8` if unspecified. So we can track the performance benefit. `Xamarin.Forms Integration` - uses `d8` and `$(AndroidLinkTool)` set to `r8`, using a `proguard.cfg` file for Xamarin.Forms. Will help us track startup performance benefits of Java code shrinking and build performance. `Mono.Android-Tests` - uses `d8` and `$(AndroidLinkTool)` set to `r8`, to verify these on-device tests pass. `Runtime-MultiDex` - `$(AndroidDexTool)` set to `d8` if unspecified, to validate multi-dex is working properly with `d8`. ~~ Xamarin.Android build changes ~~ This adds https://r8.googlesource.com/r8 as a submodule in `external/r8`, and builds `d8.jar` and `r8.jar` via `src/r8/r8.csproj`. Currently using version 1.2.50 of r8. ~~ Deployment changes ~~ Three new files will need to be included in Xamarin.Android installers: - `d8.jar` - `r8.jar` - `Xamarin.Android.D8.targets` ~~ General changes ~~ * Fixed doc for `xa4304` error code * Added `xa4305` error code for `CompileToDalvik`, `R8`, and `CreateMultiDexMainDexClassList` * Removed log messages in `CreateMultiDexMainDexClassList` ~~ Performance Comparison ~~ | MSBuild Target | Options Enabled | Time | APK size (bytes) | dex size (bytes) | | --- | --- | ---: | ---: | ---: | | _CompileToDalvikWithDx | n/a | 11074ms | 13378157 | 3894720 | | _CompileToDalvikWithD8 | d8, (desugar enabled) | 8543ms | 13124205 | 3314064 | | _CompileToDalvikWithD8 | d8, (desugar disabled) | 9550ms | 13124205 | 3314064 | | _CompileToDalvikWithDx | multi-dex | 15632ms | 13390498 | 3916496 | | _CompileToDalvikWithD8 | d8, multi-dex | 25979ms | 13054626 | 3264096 | | _CompileToDalvikWithDx | proguard | 11903ms | 12804717 | 2446964 | | _CompileToDalvikWithD8 | d8, r8 | 13799ms | 12513901 | 1835588 | | _CompileToDalvikWithDx | multi-dex, proguard | 17279ms | 12804770 | 2449512 | | _CompileToDalvikWithD8 | d8, multi-dex, r8 | 13792ms | 12513954 | 1837588 | _NOTE: desugar is enabled by default with d8/r8_ I timed this builds with [this script][powershell_script], with a "Hello World" Xamarin.Forms app. Build logs here: [d8andr8.zip][d8andr8_zip] One can draw their own conclusions on which options are faster, better, smaller. See further detail in `D8andR8.md`. [powershell_script]: https://github.com/jonathanpeppers/HelloWorld/blob/39e2854f6ca39c0941fb8bd6f2a16d8b7663003e/build.ps1 [d8andr8_zip]: https://github.com/xamarin/xamarin-android/files/2470385/d8andr8.zip Co-authored-by: Atsushi Eno --- .gitmodules | 4 + Documentation/README.md | 1 + Documentation/guides/BuildProcess.md | 25 ++ Documentation/guides/D8andR8.md | 270 ++++++++++++++++++ Documentation/guides/MSBuildBestPractices.md | 14 + Documentation/guides/messages/xa4304.md | 2 +- Documentation/guides/messages/xa4305.md | 21 ++ Xamarin.Android.sln | 9 +- external/r8 | 1 + external/r8.tpnitems | 9 + samples/HelloWorld/HelloWorld.csproj | 1 + .../Test/Mono.Android-Tests.csproj | 2 + .../Tasks/CompileToDalvik.cs | 10 +- .../Tasks/CreateMultiDexMainDexClassList.cs | 24 +- src/Xamarin.Android.Build.Tasks/Tasks/D8.cs | 106 +++++++ .../Tasks/Proguard.cs | 32 +-- src/Xamarin.Android.Build.Tasks/Tasks/R8.cs | 118 ++++++++ .../BuildTest.OSS.cs | 65 ----- .../Xamarin.Android.Build.Tests/BuildTest.cs | 159 +++++++---- .../Android/KnownProperties.cs | 2 + .../XamarinAndroidApplicationProject.cs | 10 + .../Xamarin.Android.Build.Tasks.csproj | 7 + .../Xamarin.Android.Build.Tasks.targets | 3 + .../Xamarin.Android.Common.targets | 57 +++- .../Xamarin.Android.D8.targets | 137 +++++++++ src/r8/r8.csproj | 16 ++ src/r8/r8.props | 3 + src/r8/r8.targets | 119 ++++++++ .../Mono.Android-TestsMultiDex.csproj | 1 + ...Forms.Performance.Integration.Droid.csproj | 3 + .../Droid/proguard.cfg | 2 + 31 files changed, 1057 insertions(+), 176 deletions(-) create mode 100644 Documentation/guides/D8andR8.md create mode 100644 Documentation/guides/messages/xa4305.md create mode 160000 external/r8 create mode 100644 external/r8.tpnitems create mode 100644 src/Xamarin.Android.Build.Tasks/Tasks/D8.cs create mode 100644 src/Xamarin.Android.Build.Tasks/Tasks/R8.cs create mode 100644 src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets create mode 100644 src/r8/r8.csproj create mode 100644 src/r8/r8.props create mode 100644 src/r8/r8.targets create mode 100644 tests/Xamarin.Forms-Performance-Integration/Droid/proguard.cfg diff --git a/.gitmodules b/.gitmodules index 970428fd05d..4808f742de6 100644 --- a/.gitmodules +++ b/.gitmodules @@ -38,6 +38,10 @@ path = external/proguard url = https://github.com/xamarin/proguard.git branch = master +[submodule "external/r8"] + path = external/r8 + url = https://r8.googlesource.com/r8 + branch = d8-1.2 [submodule "external/sqlite"] path = external/sqlite url = https://github.com/xamarin/sqlite.git diff --git a/Documentation/README.md b/Documentation/README.md index dade88725c6..d318de72909 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -20,6 +20,7 @@ * [Build Process](guides/BuildProcess.md) * [`.axml` CodeBehind Support](guides/LayoutCodeBehind.md) * [MSBuild Best Practices](guides/MSBuildBestPractices.md) + * [D8 and R8 Integration](guides/D8andR8.md) # Building from Source diff --git a/Documentation/guides/BuildProcess.md b/Documentation/guides/BuildProcess.md index deca390dcd6..b9e67b02486 100644 --- a/Documentation/guides/BuildProcess.md +++ b/Documentation/guides/BuildProcess.md @@ -210,6 +210,23 @@ when packaing Release applications. This property is `False` by default. +- **AndroidDexTool** – An enum-style property with valid + values of `dx` or `d8`. Indicates which Android [dex][dex] + compiler is used during the Xamarin.Android build process. + Currently defaults to `dx`. For further information see our + documentation on [D8 and R8][d8-r8]. + + [dex]: https://source.android.com/devices/tech/dalvik/dalvik-bytecode + [d8-r8]: D8andR8.md + +- **AndroidEnableDesugar** – A boolean property that + determines if `desugar` is enabled. Android does not currently + support all Java 8 features, and the default toolchain implements + the new language features by performing bytecode transformations, + called `desugar`, on the output of the `javac` compiler. Defaults + to `False` if using `AndroidDexTool=dx` and defaults to `True` if + using `AndroidDexTool=d8`. + - **AndroidEnableMultiDex** – A boolean property that determines whether or not multi-dex support will be used in the final `.apk`. @@ -360,6 +377,14 @@ when packaing Release applications. Assembly1;Assembly2 ``` +- **AndroidLinkTool** – An enum-style property with valid + values of `proguard` or `r8`. Indicates which code shrinker is + used for Java code. Currently defaults to an empty string, or + `proguard` if `$(AndroidEnableProguard)` is `True`. For further + information see our documentation on [D8 and R8][d8-r8]. + + [d8-r8]: D8andR8.md + - **LinkerDumpDependencies** – A bool property which enables generating of linker dependencies file. This file can be used as input for diff --git a/Documentation/guides/D8andR8.md b/Documentation/guides/D8andR8.md new file mode 100644 index 00000000000..e12b3599891 --- /dev/null +++ b/Documentation/guides/D8andR8.md @@ -0,0 +1,270 @@ +This is the D8 and R8 integration specification for Xamarin.Android. + +# What is D8? What is R8? + +At a high level, here are the steps that occur during an Android +application's Java compilation: +- `javac` compiles Java code +- `desugar` remove's the "sugar" (from Java 8 features) that are not + fully supported on Android +- ProGuard shrinks compiled Java code +- `dx` "dexes" compiled Java code into Android [dex][dex] format. This + is an alternate Java bytecode format supported by the Android + platform. + +This process has a few issues, such as: +- [proguard](https://www.guardsquare.com/en/products/proguard/manual) + is made by a third party, and aimed for Java in general (not Android + specific) +- `dx` is slower than it _could_ be + +So in 2017, Google announced a "next-generation" dex compiler named +[D8](https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html). + +- D8 is a direct replacement for `dx` +- R8 is a replacement for ProGuard, that also "dexes" at the same + time. If using R8, a D8 call is not needed. + +Both tools have support for various other Android-specifics: +- Both `desugar` by default unless the `--no-desugaring` switch is + specified +- Both support [multidex][multidex], although `d8` does not have + support for using the ProGuard rules format (the + `--main-dex-rules` switch). +- R8 has full support for [multidex][multidex]. + +Additionally, R8 is geared to be backwards compatible to ProGuard. +It uses the same file format for configuration and command-line +parameters as ProGuard. However, at the time of writing this, there +are still several flags/features not implemented in R8 yet. + +For more information on how R8 compares to ProGuard, please see +[this comparison from the ProGuard team](https://www.guardsquare.com/en/blog/proguard-and-r8). + +You can find the source for D8 and R8 at: + + +For reference, `d8 --help`: +``` +Usage: d8 [options] + where are any combination of dex, class, zip, jar, or apk files + and options are: + --debug # Compile with debugging information (default). + --release # Compile without debugging information. + --output # Output result in . + # must be an existing directory or a zip file. + --lib # Add as a library resource. + --classpath # Add as a classpath resource. + --min-api # Minimum Android API level compatibility + --intermediate # Compile an intermediate result intended for later + # merging. + --file-per-class # Produce a separate dex file per input class + --no-desugaring # Force disable desugaring. + --main-dex-list # List of classes to place in the primary dex file. + --version # Print the version of d8. + --help # Print this message. +``` + +For reference, `r8 --help`: +``` +Usage: r8 [options] + where are any combination of dex, class, zip, jar, or apk files + and options are: + --release # Compile without debugging information (default). + --debug # Compile with debugging information. + --output # Output result in . + # must be an existing directory or a zip file. + --lib # Add as a library resource. + --min-api # Minimum Android API level compatibility. + --pg-conf # Proguard configuration . + --pg-map-output # Output the resulting name and line mapping to . + --no-tree-shaking # Force disable tree shaking of unreachable classes. + --no-minification # Force disable minification of names. + --no-desugaring # Force disable desugaring. + --main-dex-rules # Proguard keep rules for classes to place in the + # primary dex file. + --main-dex-list # List of classes to place in the primary dex file. + --main-dex-list-output # Output the full main-dex list in . + --version # Print the version of r8. + --help # Print this message. +``` + +# What did Xamarin.Android do *before* D8/R8? + +In other words, what is currently happening *before* we introduce D8/R8 support? + +1. The [Javac](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Javac.cs) + MSBuild task compiles `*.java` files to a `classes.zip` file. +2. The [Desugar](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Desugar.cs) + MSBuild task "desugars" using `desugar_deploy.jar` if + `$(AndroidEnableDesugar)` is `True`. +3. The [Proguard](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Proguard.cs) + MSBuild task shrinks the compiled Java code if + `$(AndroidEnableProguard)` is `True`. Developers may also supply + custom proguard configuration files via `ProguardConfiguration` + build items. +4. The [CreateMultiDexMainDexClassList](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/CreateMultiDexMainDexClassList.cs) + MSBuild task runs `proguard` to generate a final, combined + `multidex.keep` file if `$(AndroidEnableMultiDex)` is `True`. + Developers can also supply custom `multidex.keep` files via + `MultiDexMainDexList` build items. +5. The [CompileToDalvik](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs) + MSBuild task runs `dx.jar` to generate a final `classes.dex` file + in `$(IntermediateOutputPath)android\bin`. If `multidex` is + enabled, a `classes2.dex` (and potentially more) are also generated + in this location. + +# What does this process look like with D8 / R8 enabled? + +Xamarin.Android now has two new MSBuild tasks: `` and ``. + +1. The [Javac](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Javac.cs) + MSBuild task will remain unchanged. +2. `D8` will run if `$(AndroidEnableMultiDex)` is `False`, + `$(AndroidLinkTool)` is not `r8`, and "desugar" by default. +3. Otherwise, `R8` will run if `$(AndroidEnableMultiDex)` is `True` or + `$(AndroidLinkTool)` is `r8` and will also "desugar" by default. + +So in addition to be being faster in general (if Google's claims are +true), we will be calling a *single* command line tool to produce dex +files! + +# So how do developers use it? What are sensible MSBuild property defaults? + +Currently, a `csproj` file might have the following properties: +```xml + + + True + True + True + + +``` + +To enable the new behavior, we have introduced two new enum-style +properties: +- `$(AndroidDexTool)` - supports `dx` or `d8` +- `$(AndroidLinkTool)` - supports `proguard` or `r8` + +But for an existing project, a developer could opt-in to the new +behavior with two properties: +```xml + + + True + True + True + + d8 + r8 + + +``` + +There should be two new MSBuild properties to configure here, because: +- You could use `D8` in combination with `proguard`, as `R8` is not + "feature complete" in comparison to `proguard`. +- You may not want to use code shrinking at all, but still use `D8` + instead of `dx`. +- You shouldn't be able to use `dx` in combination with `R8`, it + doesn't make sense. +- Developers should be able to use the existing properties for + enabling code shrinking, `multidex`, and `desugar`. + +Our reasonable defaults would be: +- If `AndroidDexTool` is omitted, `dx` and `CompileToDalvik` + should be used. Until D8/R8 integration is deemed stable and enabled + by default. +- If `AndroidDexTool` is `d8` and `AndroidEnableDesugar` is + omitted, `AndroidEnableDesugar` should be enabled. +- If `AndroidLinkTool` is omitted and `AndroidEnableProguard` is + `true`, we should default `AndroidLinkTool` to `proguard`. + +MSBuild properties default to: +```xml +dx + +proguard +True +``` + +If a user specifies combinations of properties: +- `AndroidDexTool` = `d8` and `AndroidEnableProguard` = `True` + - `AndroidLinkTool` will get set to `proguard` +- `AndroidDexTool` = `dx` and `AndroidLinkTool` = `r8` + - This combination doesn't really *make sense*, but we don't need to + do anything: only `R8` will be called because it dexes and shrinks + at the same time. +- `AndroidEnableDesugar` is enabled when omitted, if either `d8` or + `r8` are used + +For new projects that want to use D8/R8, code shrinking, and +`multidex`, it would make sense to specify: +```xml + + + True + d8 + r8 + + +``` + +# Additional D8 / R8 settings? + +`--debug` or `--release` needs to be explicitly specified for both D8 +and R8. We use the [AndroidIncludeDebugSymbols][debug_symbols] +property for this. + +`$(AndroidD8ExtraArguments)` and `$(AndroidR8ExtraArguments)` can be +used to explicitly pass additional flags to D8 and R8. + +# How are we compiling / shipping D8 and R8? + +We have added a submodule to `xamarin-android` for +[r8](https://r8.googlesource.com/r8/). It will be pinned to a commit +with a reasonable release tag, such as `1.2.50` for now. + +To build r8, we have to: +- Download and unzip a tool named [depot_tools][depot_tools] from the + Chromium project +- Put the path to `depot_tools` in `$PATH` +- Run `gclient` so it will download/bootstrap gradle, python, and + other tools +- Run `python tools\gradle.py d8 r8` to compile `d8.jar` and `r8.jar` +- We will need to ship `d8.jar` and `r8.jar` in our installers, + similar to how we are shipping `desugar_deploy.jar` + +# Performance Comparison + +| MSBuild Target | Options Enabled | Time | APK size (bytes) | dex size (bytes) | +| --- | --- | ---: | ---: | ---: | +| _CompileToDalvikWithDx | n/a | 11074ms | 13378157 | 3894720 | +| _CompileToDalvikWithD8 | d8, (desugar enabled) | 8543ms | 13124205 | 3314064 | +| _CompileToDalvikWithD8 | d8, (desugar disabled) | 9550ms | 13124205 | 3314064 | +| _CompileToDalvikWithDx | multi-dex | 15632ms | 13390498 | 3916496 | +| _CompileToDalvikWithD8 | d8, multi-dex | 25979ms | 13054626 | 3264096 | +| _CompileToDalvikWithDx | proguard | 11903ms | 12804717 | 2446964 | +| _CompileToDalvikWithD8 | d8, r8 | 13799ms | 12513901 | 1835588 | +| _CompileToDalvikWithDx | multi-dex, proguard | 17279ms | 12804770 | 2449512 | +| _CompileToDalvikWithD8 | d8, multi-dex, r8 | 13792ms | 12513954 | 1837588 | + +_NOTE: desugar is enabled by default with d8/r8_ + +I timed this builds with [this script][powershell_script], with a "Hello World" Xamarin.Forms app. Build logs here: [d8andr8.zip][d8andr8_zip] + +One can draw their own conclusions on which options are faster, better, smaller. + +Some of my thoughts: +- Default options for d8 and r8 seem to be faster? +- Disabling `desugar` is slower? +- Enabling `multi-dex` makes the dex file larger, because new classes are required. The app wasn't large enough to warrant a `classes2.dex`. +- `d8` does not support multi-dex, and so choosing `d8` + `multi-dex` actually runs `r8` with `--no-tree-shaking --no-minification`. These options are _slower_? + +[dex]: https://source.android.com/devices/tech/dalvik/dalvik-bytecode +[multidex]: https://developer.android.com/studio/build/multidex +[debug_symbols]: https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets#L315-L336 +[depot_tools]: http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html +[powershell_script]: https://github.com/jonathanpeppers/HelloWorld/blob/39e2854f6ca39c0941fb8bd6f2a16d8b7663003e/build.ps1 +[d8andr8_zip]: https://github.com/xamarin/xamarin-android/files/2470385/d8andr8.zip diff --git a/Documentation/guides/MSBuildBestPractices.md b/Documentation/guides/MSBuildBestPractices.md index 7827540e5e9..490c63717a0 100644 --- a/Documentation/guides/MSBuildBestPractices.md +++ b/Documentation/guides/MSBuildBestPractices.md @@ -201,6 +201,20 @@ use `Inputs` or `Outputs`. # Best Practices for Xamarin.Android MSBuild targets +## Naming in Xamarin.Android targets + +As mentioned [above](/MSBuildBestPractices.md#naming), a good amount +of consideration should be done before adding new public-facing +MSBuild properties. This is pretty clear when adding a new feature, +since an obvious feature flag will be needed to enable it. + +The main thing to keep in mind here is that almost all of our +public-facing MSBuild properties should be prefixed with `Android`. +This is a good convention so it is easy to know which properties are +specific to Xamarin.Android, and this will prevent them from +conflicting with MSBuild properties from other products. All MSBuild +properties are effectively "global variables"... + ## Stamp Files From now on, we should try to put new stamp files in diff --git a/Documentation/guides/messages/xa4304.md b/Documentation/guides/messages/xa4304.md index 384c2b38478..f554ea1c2b4 100644 --- a/Documentation/guides/messages/xa4304.md +++ b/Documentation/guides/messages/xa4304.md @@ -1,4 +1,4 @@ -# Compiler Error XA4304 +# Compiler Warning XA4304 The `Proguard` MSBuild task encountered a proguard configuration file that was not found on disk. These files are generally declared in your diff --git a/Documentation/guides/messages/xa4305.md b/Documentation/guides/messages/xa4305.md new file mode 100644 index 00000000000..c33a510c2bc --- /dev/null +++ b/Documentation/guides/messages/xa4305.md @@ -0,0 +1,21 @@ +# Compiler Warning XA4305 + +The `CreateMultiDexMainDexClassList`, `CompileToDalvik` or `R8` +MSBuild task encountered a `multidex.keep` file that was not found on +disk. You can customize `multidex` settings for your Xamarin.Android +application by adding files with the `MultiDexMainDexList` build item, +which are combined into a final `multidex.keep` file. + +To learn more about `multidex` and how it relates to Android +development, see the [Android documentation][android]. + +## Resolution + +Verify you are not declaring a `MultiDexMainDexList` build item that +does not exist. + +Consider submitting a [bug][bug] if you are getting this warning under +normal circumstances. + +[android]: https://developer.android.com/studio/build/multidex +[bug]: https://github.com/xamarin/xamarin-android/wiki/Submitting-Bugs,-Feature-Requests,-and-Pull-Requests diff --git a/Xamarin.Android.sln b/Xamarin.Android.sln index b53520fd5ea..89e51cdc8ce 100644 --- a/Xamarin.Android.sln +++ b/Xamarin.Android.sln @@ -123,6 +123,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Xamarin.Android.Tools.Andro EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Xamarin.Android.Tools.AndroidSdk-Tests", "external\xamarin-android-tools\src\Xamarin.Android.Tools.AndroidSdk\Tests\Xamarin.Android.Tools.AndroidSdk-Tests.csproj", "{1E5501E8-49C1-4659-838D-CC9720C5208F}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "r8", "src\r8\r8.csproj", "{1BAFA0CC-0377-46CE-AB7B-7BB2E7B62F63}" +EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "proprietary", "build-tools\proprietary\proprietary.csproj", "{D93CAC27-3893-42A3-99F1-2BCA72E186F4}" EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "download-bundle", "build-tools\download-bundle\download-bundle.csproj", "{1DA0CB12-5508-4E83-A242-0C8D6D99A49B}" @@ -358,6 +360,10 @@ Global {1DA0CB12-5508-4E83-A242-0C8D6D99A49B}.Debug|AnyCPU.Build.0 = Debug|Any CPU {1DA0CB12-5508-4E83-A242-0C8D6D99A49B}.Release|AnyCPU.ActiveCfg = Release|Any CPU {1DA0CB12-5508-4E83-A242-0C8D6D99A49B}.Release|AnyCPU.Build.0 = Release|Any CPU + {1BAFA0CC-0377-46CE-AB7B-7BB2E7B62F63}.Debug|AnyCPU.ActiveCfg = Debug|Any CPU + {1BAFA0CC-0377-46CE-AB7B-7BB2E7B62F63}.Debug|AnyCPU.Build.0 = Debug|Any CPU + {1BAFA0CC-0377-46CE-AB7B-7BB2E7B62F63}.Release|AnyCPU.ActiveCfg = Release|Any CPU + {1BAFA0CC-0377-46CE-AB7B-7BB2E7B62F63}.Release|AnyCPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -418,8 +424,7 @@ Global {B8105878-D423-4159-A3E7-028298281EC6} = {04E3E11E-B47D-4599-8AFC-50515A95E715} {E34BCFA0-CAA4-412C-AA1C-75DB8D67D157} = {04E3E11E-B47D-4599-8AFC-50515A95E715} {1E5501E8-49C1-4659-838D-CC9720C5208F} = {CAB438D8-B0F5-4AF0-BEBD-9E2ADBD7B483} - {D93CAC27-3893-42A3-99F1-2BCA72E186F4} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62} - {1DA0CB12-5508-4E83-A242-0C8D6D99A49B} = {E351F97D-EA4F-4E7F-AAA0-8EBB1F2A4A62} + {1BAFA0CC-0377-46CE-AB7B-7BB2E7B62F63} = {04E3E11E-B47D-4599-8AFC-50515A95E715} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {53A1F287-EFB2-4D97-A4BB-4A5E145613F6} diff --git a/external/r8 b/external/r8 new file mode 160000 index 00000000000..125b72d352a --- /dev/null +++ b/external/r8 @@ -0,0 +1 @@ +Subproject commit 125b72d352a07470b96c05689e490f5159b17e27 diff --git a/external/r8.tpnitems b/external/r8.tpnitems new file mode 100644 index 00000000000..8d945808d81 --- /dev/null +++ b/external/r8.tpnitems @@ -0,0 +1,9 @@ + + + + + $(MSBuildThisFileDirectory)\r8\LICENSE + https://r8.googlesource.com/r8/ + + + diff --git a/samples/HelloWorld/HelloWorld.csproj b/samples/HelloWorld/HelloWorld.csproj index 804fa092f18..e48903c2270 100644 --- a/samples/HelloWorld/HelloWorld.csproj +++ b/samples/HelloWorld/HelloWorld.csproj @@ -16,6 +16,7 @@ Assets true v7.1 + d8 False v8.1 true + d8 @@ -41,6 +42,7 @@ 4 false true + r8 diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs b/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs index 9bb448ea876..92c273bf747 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs @@ -116,8 +116,14 @@ protected override string GenerateCommandLineCommands () cmd.AppendSwitchIfNotNull ("--input-list=", inputListFile); if (MultiDexEnabled) { - cmd.AppendSwitch ("--multi-dex"); - cmd.AppendSwitchIfNotNull ("--main-dex-list=", MultiDexMainDexListFile); + if (string.IsNullOrEmpty (MultiDexMainDexListFile)) { + Log.LogCodedWarning ("XA4305", $"MultiDex is enabled, but '{nameof (MultiDexMainDexListFile)}' was not specified."); + } else if (!File.Exists (MultiDexMainDexListFile)) { + Log.LogCodedWarning ("XA4305", MultiDexMainDexListFile, 0, $"MultiDex is enabled, but main dex list file '{MultiDexMainDexListFile}' does not exist."); + } else { + cmd.AppendSwitch ("--multi-dex"); + cmd.AppendSwitchIfNotNull ("--main-dex-list=", MultiDexMainDexListFile); + } } cmd.AppendSwitchIfNotNull ("--output ", Path.GetDirectoryName (ClassesOutputDirectory)); diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/CreateMultiDexMainDexClassList.cs b/src/Xamarin.Android.Build.Tasks/Tasks/CreateMultiDexMainDexClassList.cs index a138f28a91c..2d75b3db999 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/CreateMultiDexMainDexClassList.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/CreateMultiDexMainDexClassList.cs @@ -37,16 +37,6 @@ public class CreateMultiDexMainDexClassList : JavaToolTask public override bool Execute () { - Log.LogDebugMessage ("CreateMultiDexMainDexClassList"); - Log.LogDebugMessage (" ClassesOutputDirectory: {0}", ClassesOutputDirectory); - Log.LogDebugTaskItems (" JavaLibraries:", JavaLibraries); - Log.LogDebugMessage (" MultiDexMainDexListFile: {0}", MultiDexMainDexListFile); - Log.LogDebugTaskItems (" CustomMainDexListFiles:", CustomMainDexListFiles); - Log.LogDebugMessage (" ToolExe: {0}", ToolExe); - Log.LogDebugMessage (" ToolPath: {0}", ToolPath); - Log.LogDebugMessage (" ProguardJarPath: {0}", ProguardJarPath); - Log.LogDebugMessage (" ProguardInputJarFilter: {0}", ProguardInputJarFilter); - tempJar = Path.Combine (Path.GetTempPath (), Path.GetRandomFileName () + ".jar"); commandlineAction = GenerateProguardCommands; // run proguard first @@ -62,13 +52,19 @@ public override bool Execute () var result = base.Execute () && !Log.HasLoggedErrors; - if (result && CustomMainDexListFiles != null && CustomMainDexListFiles.Any (x => File.Exists (x.ItemSpec))) { - foreach (var content in CustomMainDexListFiles.Select (i => File.ReadAllLines (i.ItemSpec))) - File.AppendAllLines (MultiDexMainDexListFile, content); + if (result && CustomMainDexListFiles?.Length > 0) { + var content = new List (); + foreach (var file in CustomMainDexListFiles) { + if (File.Exists (file.ItemSpec)) { + content.Add (File.ReadAllText (file.ItemSpec)); + } else { + Log.LogCodedWarning ("XA4305", file.ItemSpec, 0, $"'MultiDexMainDexList' file '{file.ItemSpec}' does not exist."); + } + } + File.AppendAllText (MultiDexMainDexListFile, string.Concat (content)); } return result; - } protected override string GenerateCommandLineCommands () diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/D8.cs b/src/Xamarin.Android.Build.Tasks/Tasks/D8.cs new file mode 100644 index 00000000000..9bf4ac1ca43 --- /dev/null +++ b/src/Xamarin.Android.Build.Tasks/Tasks/D8.cs @@ -0,0 +1,106 @@ +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; +using System.Collections.Generic; +using System.IO; +using Xamarin.Android.Tools; + +namespace Xamarin.Android.Tasks +{ + /// + /// This task invokes d8, and is also subclassed by the R8 task + /// + public class D8 : JavaToolTask + { + [Required] + public string JarPath { get; set; } + + /// + /// Output for *.dex files. R8 can be invoked for just --main-dex-list-output, so this is not [Required] + /// + public string OutputDirectory { get; set; } + + /// + /// It is loaded to calculate --min-api, which is used by desugaring part to determine which levels of desugaring it performs. + /// + public string AndroidManifestFile { get; set; } + + // general d8 feature options. + public bool Debug { get; set; } + public bool EnableDesugar { get; set; } = true; + + // Java libraries to embed or reference + [Required] + public string ClassesZip { get; set; } + [Required] + public string JavaPlatformJarPath { get; set; } + public ITaskItem [] JavaLibrariesToEmbed { get; set; } + public ITaskItem [] AlternativeJarLibrariesToEmbed { get; set; } + public ITaskItem [] JavaLibrariesToReference { get; set; } + + public string ExtraArguments { get; set; } + + protected override string GenerateCommandLineCommands () + { + return GetCommandLineBuilder ().ToString (); + } + + protected virtual CommandLineBuilder GetCommandLineBuilder () + { + var cmd = new CommandLineBuilder (); + + if (!string.IsNullOrEmpty (JavaOptions)) { + cmd.AppendSwitch (JavaOptions); + } + cmd.AppendSwitchIfNotNull ("-Xmx", JavaMaximumHeapSize); + cmd.AppendSwitchIfNotNull ("-jar ", JarPath); + + if (!string.IsNullOrEmpty (ExtraArguments)) + cmd.AppendSwitch (ExtraArguments); // it should contain "--dex". + if (Debug) + cmd.AppendSwitch ("--debug"); + else + cmd.AppendSwitch ("--release"); + + //NOTE: if this is blank, we can omit --min-api in this call + if (!string.IsNullOrEmpty (AndroidManifestFile)) { + var doc = AndroidAppManifest.Load (AndroidManifestFile, MonoAndroidHelper.SupportedVersions); + int minApiVersion = doc.MinSdkVersion == null ? 4 : (int)doc.MinSdkVersion; + cmd.AppendSwitchIfNotNull ("--min-api ", minApiVersion.ToString ()); + } + + if (!EnableDesugar) + cmd.AppendSwitch ("--no-desugaring"); + + var injars = new List (); + var libjars = new List (); + if (AlternativeJarLibrariesToEmbed?.Length > 0) { + Log.LogDebugMessage (" processing AlternativeJarLibrariesToEmbed..."); + foreach (var jar in AlternativeJarLibrariesToEmbed) { + injars.Add (jar.ItemSpec); + } + } else if (JavaLibrariesToEmbed != null) { + Log.LogDebugMessage (" processing ClassesZip, JavaLibrariesToEmbed..."); + if (File.Exists (ClassesZip)) { + injars.Add (ClassesZip); + } + foreach (var jar in JavaLibrariesToEmbed) { + injars.Add (jar.ItemSpec); + } + } + libjars.Add (JavaPlatformJarPath); + if (JavaLibrariesToReference != null) { + foreach (var jar in JavaLibrariesToReference) { + libjars.Add (jar.ItemSpec); + } + } + + cmd.AppendSwitchIfNotNull ("--output ", OutputDirectory); + foreach (var jar in libjars) + cmd.AppendSwitchIfNotNull ("--lib ", jar); + foreach (var jar in injars) + cmd.AppendFileNameIfNotNull (jar); + + return cmd; + } + } +} \ No newline at end of file diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/Proguard.cs b/src/Xamarin.Android.Build.Tasks/Tasks/Proguard.cs index 3ed19e8afc4..744b7f683e6 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/Proguard.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/Proguard.cs @@ -41,23 +41,17 @@ public class Proguard : ToolTask [Required] public string ProguardJarOutput { get; set; } - [Required] public string ProguardGeneratedReferenceConfiguration { get; set; } - - [Required] public string ProguardGeneratedApplicationConfiguration { get; set; } - - [Required] public string ProguardCommonXamarinConfiguration { get; set; } + [Required] public string ProguardConfigurationFiles { get; set; } public ITaskItem[] JavaLibrariesToEmbed { get; set; } - public ITaskItem[] ExternalJavaLibraries { get; set; } + public ITaskItem[] JavaLibrariesToReference { get; set; } - public ITaskItem[] DoNotPackageJavaLibraries { get; set; } - public bool UseProguard { get; set; } public string JavaOptions { get; set; } @@ -119,15 +113,9 @@ protected override string GenerateCommandLineCommands () // skip invalid lines } - var injars = new List (); - var libjars = new List (); - injars.Add (classesZip); - if (JavaLibrariesToEmbed != null) - foreach (var jarfile in JavaLibrariesToEmbed) - injars.Add (jarfile.ItemSpec); - - using (var xamcfg = File.Create (ProguardCommonXamarinConfiguration)) - GetType ().Assembly.GetManifestResourceStream ("proguard_xamarin.cfg").CopyTo (xamcfg); + if (!string.IsNullOrWhiteSpace (ProguardCommonXamarinConfiguration)) + using (var xamcfg = File.Create (ProguardCommonXamarinConfiguration)) + GetType ().Assembly.GetManifestResourceStream ("proguard_xamarin.cfg").CopyTo (xamcfg); var configs = ProguardConfigurationFiles .Replace ("{sdk.dir}", AndroidSdkDirectory + Path.DirectorySeparatorChar) @@ -148,9 +136,15 @@ protected override string GenerateCommandLineCommands () Log.LogCodedWarning ("XA4304", file, 0, "Proguard configuration file '{0}' was not found.", file); } + var injars = new List (); + var libjars = new List (); + injars.Add (classesZip); + if (JavaLibrariesToEmbed != null) + foreach (var jarfile in JavaLibrariesToEmbed) + injars.Add (jarfile.ItemSpec); libjars.Add (JavaPlatformJarPath); - if (ExternalJavaLibraries != null) - foreach (var jarfile in ExternalJavaLibraries.Select (p => p.ItemSpec)) + if (JavaLibrariesToReference != null) + foreach (var jarfile in JavaLibrariesToReference.Select (p => p.ItemSpec)) libjars.Add (jarfile); cmd.AppendSwitchUnquotedIfNotNull ("-injars ", "\"'" + string.Join ($"'{ProguardInputJarFilter}{Path.PathSeparator}'", injars.Distinct ()) + $"'{ProguardInputJarFilter}\""); diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/R8.cs b/src/Xamarin.Android.Build.Tasks/Tasks/R8.cs new file mode 100644 index 00000000000..8c9762b09b2 --- /dev/null +++ b/src/Xamarin.Android.Build.Tasks/Tasks/R8.cs @@ -0,0 +1,118 @@ +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; +using System.Collections.Generic; +using System.IO; +using System.Linq; + +namespace Xamarin.Android.Tasks +{ + /// + /// This task invokes r8 in order to: + /// - Compile to dex format + code shrinking (replacement for proguard) + /// - Enable multi-dex, even if code shrinking is not used + /// + public class R8 : D8 + { + [Required] + public string AndroidSdkBuildToolsPath { get; set; } + [Required] + public string AndroidSdkDirectory { get; set; } + + // multidex + public bool EnableMultiDex { get; set; } + public ITaskItem [] CustomMainDexListFiles { get; set; } + public string MultiDexMainDexListFile { get; set; } + + // proguard-like configuration settings + public bool EnableShrinking { get; set; } = true; + public string AcwMapFile { get; set; } + public string ProguardGeneratedReferenceConfiguration { get; set; } + public string ProguardGeneratedApplicationConfiguration { get; set; } + public string ProguardCommonXamarinConfiguration { get; set; } + public string ProguardConfigurationFiles { get; set; } + + string temp; + + public override bool Execute () + { + try { + temp = Path.GetTempFileName (); + return base.Execute (); + } finally { + if (!string.IsNullOrEmpty (temp)) + File.Delete (temp); + } + } + + protected override CommandLineBuilder GetCommandLineBuilder () + { + var cmd = base.GetCommandLineBuilder (); + + if (EnableMultiDex) { + if (string.IsNullOrEmpty (MultiDexMainDexListFile)) { + Log.LogCodedWarning ("XA4305", $"MultiDex is enabled, but '{nameof (MultiDexMainDexListFile)}' was not specified."); + } else { + var content = new List (); + if (CustomMainDexListFiles != null) { + foreach (var file in CustomMainDexListFiles) { + if (File.Exists (file.ItemSpec)) { + content.Add (File.ReadAllText (file.ItemSpec)); + } else { + Log.LogCodedWarning ("XA4305", file.ItemSpec, 0, $"'MultiDexMainDexList' file '{file.ItemSpec}' does not exist."); + } + } + } + File.WriteAllText (temp, string.Concat (content)); + + cmd.AppendSwitchIfNotNull ("--main-dex-list ", temp); + cmd.AppendSwitchIfNotNull ("--main-dex-rules ", Path.Combine (AndroidSdkBuildToolsPath, "mainDexClasses.rules")); + cmd.AppendSwitchIfNotNull ("--main-dex-list-output ", MultiDexMainDexListFile); + } + } + + if (EnableShrinking) { + if (!string.IsNullOrEmpty (AcwMapFile)) { + var acwLines = File.ReadAllLines (AcwMapFile); + using (var appcfg = File.CreateText (ProguardGeneratedApplicationConfiguration)) { + for (int i = 0; i + 2 < acwLines.Length; i += 3) { + try { + var line = acwLines [i + 2]; + var java = line.Substring (line.IndexOf (';') + 1); + appcfg.WriteLine ("-keep class " + java + " { *; }"); + } catch { + // skip invalid lines + } + } + } + } + if (!string.IsNullOrWhiteSpace (ProguardCommonXamarinConfiguration)) + using (var xamcfg = File.Create (ProguardCommonXamarinConfiguration)) + GetType ().Assembly.GetManifestResourceStream ("proguard_xamarin.cfg").CopyTo (xamcfg); + if (!string.IsNullOrEmpty (ProguardConfigurationFiles)) { + var configs = ProguardConfigurationFiles + .Replace ("{sdk.dir}", AndroidSdkDirectory + Path.DirectorySeparatorChar) + .Replace ("{intermediate.common.xamarin}", ProguardCommonXamarinConfiguration) + .Replace ("{intermediate.references}", ProguardGeneratedReferenceConfiguration) + .Replace ("{intermediate.application}", ProguardGeneratedApplicationConfiguration) + .Replace ("{project}", string.Empty) // current directory anyways. + .Split (';') + .Select (s => s.Trim ()) + .Where (s => !string.IsNullOrWhiteSpace (s)); + foreach (var file in configs) { + if (File.Exists (file)) + cmd.AppendSwitchIfNotNull ("--pg-conf ", file); + else + Log.LogCodedWarning ("XA4304", file, 0, "Proguard configuration file '{0}' was not found.", file); + } + } + } else { + //NOTE: we may be calling r8 *only* for multi-dex, and all shrinking is disabled + cmd.AppendSwitch ("--no-tree-shaking"); + cmd.AppendSwitch ("--no-minification"); + } + + return cmd; + } + } + +} diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.OSS.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.OSS.cs index 6bce79f2d34..d761074918a 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.OSS.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.OSS.cs @@ -55,28 +55,6 @@ public partial class BuildTest : BaseTest }, }; - static object [] ProguardChecks = new object [] { - new Object [] { - /* isRelease */ true, - /* enableProguard */ true, - /* useLatestSdk */ true, - }, - new Object [] { - /* isRelease */ true, - /* enableProguard */ false, - /* useLatestSdk */ true, - }, - new Object [] { - /* isRelease */ false, - /* enableProguard */ true, - /* useLatestSdk */ true, - }, - new Object [] { - /* isRelease */ false, - /* enableProguard */ false, - /* useLatestSdk */ true, - }, - }; static object [] TakeSimpleFlag = new object [] { new Object [] { false }, // Disabled because Jack DOESN'T work @@ -233,49 +211,6 @@ public partial class BuildTest : BaseTest /* expectedRuntime */ "release", }, }; - - static object [] DesugarChecks = new object [] { - new Object [] { - /* isRelease */ true, - /* enableDesugar */ false, - /* enableProguard */ true, - }, - new Object [] { - /* isRelease */ true, - /* enableDesugar */ false, - /* enableProguard */ false, - }, - new Object [] { - /* isRelease */ true, - /* enableDesugar */ true, - /* enableProguard */ true, - }, - new Object [] { - /* isRelease */ true, - /* enableDesugar */ true, - /* enableProguard */ false, - }, - new Object [] { - /* isRelease */ false, - /* enableDesugar */ false, - /* enableProguard */ true, - }, - new Object [] { - /* isRelease */ false, - /* enableDesugar */ false, - /* enableProguard */ false, - }, - new Object [] { - /* isRelease */ false, - /* enableDesugar */ true, - /* enableProguard */ true, - }, - new Object [] { - /* isRelease */ false, - /* enableDesugar */ true, - /* enableProguard */ false, - }, - }; #pragma warning restore 414 } } diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs index 0778c1d382a..f3893570e92 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs @@ -652,18 +652,26 @@ public void BuildAotApplicationAndBundle (string supportedAbis, bool enableLLVM, } [Test] - [TestCaseSource ("ProguardChecks")] - public void BuildProguardEnabledProject (bool isRelease, bool enableProguard, bool useLatestSdk) + public void BuildProguardEnabledProject ([Values (true, false)] bool isRelease, [Values ("dx", "d8")] string dexTool, [Values ("", "proguard", "r8")] string linkTool) { - var proj = new XamarinAndroidApplicationProject () { IsRelease = isRelease, EnableProguard = enableProguard, UseLatestPlatformSdk = useLatestSdk, TargetFrameworkVersion = useLatestSdk ? "v7.1" : "v5.0" }; - using (var b = CreateApkBuilder (Path.Combine ("temp", $"BuildProguard Enabled Project(1){isRelease}{enableProguard}{useLatestSdk}"))) { + var proj = new XamarinAndroidApplicationProject { + IsRelease = isRelease, + DexTool = dexTool, + LinkTool = linkTool, + }; + using (var b = CreateApkBuilder (Path.Combine ("temp", $"BuildProguard Enabled Project(1){isRelease}{dexTool}{linkTool}"))) { Assert.IsTrue (b.Build (proj), "Build should have succeeded."); - if (isRelease && enableProguard) { + if (isRelease && !string.IsNullOrEmpty (linkTool)) { var proguardProjectPrimary = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "proguard", "proguard_project_primary.cfg"); FileAssert.Exists (proguardProjectPrimary); - StringAssertEx.ContainsText (File.ReadAllLines (proguardProjectPrimary), "-keep class md52d9cf6333b8e95e8683a477bc589eda5.MainActivity"); + Assert.IsTrue (StringAssertEx.ContainsText (File.ReadAllLines (proguardProjectPrimary), "-keep class md52d9cf6333b8e95e8683a477bc589eda5.MainActivity"), "`md52d9cf6333b8e95e8683a477bc589eda5.MainActivity` should exist in `proguard_project_primary.cfg`!"); } + + var className = "Lmono/MonoRuntimeProvider;"; + var dexFile = b.Output.GetIntermediaryPath (Path.Combine ("android", "bin", "classes.dex")); + FileAssert.Exists (dexFile); + Assert.IsTrue (DexUtils.ContainsClass (className, dexFile, b.AndroidSdkDirectory), $"`{dexFile}` should include `{className}`!"); } } @@ -687,23 +695,28 @@ XamarinAndroidApplicationProject CreateMultiDexRequiredApplication (string debug [Test] [Category ("Minor")] - public void BuildApplicationOver65536Methods () + public void BuildApplicationOver65536Methods ([Values (true, false)] bool useD8) { var proj = CreateMultiDexRequiredApplication (); - using (var b = CreateApkBuilder ("temp/BuildApplicationOver65536Methods")) { + if (useD8) { + proj.DexTool = "d8"; + } + using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { b.ThrowOnBuildFailure = false; Assert.IsFalse (b.Build (proj), "Without MultiDex option, build should fail"); - b.Clean (proj); } } [Test] - public void CreateMultiDexWithSpacesInConfig () + public void CreateMultiDexWithSpacesInConfig ([Values (true, false)] bool useD8) { var proj = CreateMultiDexRequiredApplication (releaseConfigurationName: "Test Config"); + if (useD8) { + proj.DexTool = "d8"; + } proj.IsRelease = true; proj.SetProperty ("AndroidEnableMultiDex", "True"); - using (var b = CreateApkBuilder ("temp/CreateMultiDexWithSpacesInConfig")) { + using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { Assert.IsTrue (b.Build (proj), "Build should have succeeded."); } } @@ -737,10 +750,13 @@ public void BuildMultiDexApplication (bool useJackAndJill, string fxVersion) } [Test] - public void BuildAfterMultiDexIsNotRequired () + public void BuildAfterMultiDexIsNotRequired ([Values (true, false)] bool useD8) { var proj = CreateMultiDexRequiredApplication (); proj.SetProperty ("AndroidEnableMultiDex", "True"); + if (useD8) { + proj.DexTool = "d8"; + } using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { string intermediateDir = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath); @@ -765,45 +781,60 @@ public void BuildAfterMultiDexIsNotRequired () Assert.IsTrue (b.Build (proj), "Build should have succeeded."); FileAssert.Exists (Path.Combine (androidBinDir, "classes.dex")); - FileAssert.DoesNotExist (Path.Combine (androidBinDir, "classes2.dex")); + //NOTE: d8 always creates classes2.dex, even if not needed + if (useD8) { + FileAssert.Exists (Path.Combine (androidBinDir, "classes2.dex")); + } else { + FileAssert.DoesNotExist (Path.Combine (androidBinDir, "classes2.dex")); + } FileAssert.DoesNotExist (Path.Combine (androidBinDir, "classes3.dex")); using (var zip = ZipHelper.OpenZip (apkPath)) { var entries = zip.Select (e => e.FullName).ToList (); Assert.IsTrue (entries.Contains ("classes.dex"), "APK must contain `classes.dex`."); - Assert.IsFalse (entries.Contains ("classes2.dex"), "APK must *not* contain `classes2.dex`."); + //NOTE: d8 always creates classes2.dex, even if not needed + if (useD8) { + Assert.IsTrue (entries.Contains ("classes2.dex"), "APK must contain `classes2.dex`."); + } else { + Assert.IsFalse (entries.Contains ("classes2.dex"), "APK must *not* contain `classes2.dex`."); + } Assert.IsFalse (entries.Contains ("classes3.dex"), "APK must *not* contain `classes3.dex`."); } } } [Test] - public void MultiDexCustomMainDexFileList () + public void MultiDexCustomMainDexFileList ([Values (true, false)] bool useD8) { - var expected = @"android/support/multidex/ZipUtil$CentralDirectory.class -android/support/multidex/MultiDexApplication.class -android/support/multidex/MultiDex$V19.class -android/support/multidex/MultiDex$V4.class -android/support/multidex/ZipUtil.class -android/support/multidex/MultiDexExtractor$1.class -android/support/multidex/MultiDexExtractor.class -android/support/multidex/MultiDex$V14.class -android/support/multidex/MultiDex.class -MyTest -"; + var expected = new [] { + "android/support/multidex/ZipUtil$CentralDirectory.class", + "android/support/multidex/MultiDexApplication.class", + "android/support/multidex/MultiDex$V19.class", + "android/support/multidex/MultiDex$V4.class", + "android/support/multidex/ZipUtil.class", + "android/support/multidex/MultiDexExtractor$1.class", + "android/support/multidex/MultiDexExtractor.class", + "android/support/multidex/MultiDex$V14.class", + "android/support/multidex/MultiDex.class", + "MyTest.class", + }; var proj = CreateMultiDexRequiredApplication (); + if (useD8) { + proj.DexTool = "d8"; + } proj.SetProperty ("AndroidEnableMultiDex", "True"); - proj.OtherBuildItems.Add (new BuildItem ("MultiDexMainDexList", "mymultidex.keep") { TextContent = () => "MyTest", Encoding = Encoding.ASCII }); + proj.OtherBuildItems.Add (new BuildItem ("MultiDexMainDexList", "mymultidex.keep") { TextContent = () => "MyTest.class", Encoding = Encoding.ASCII }); proj.OtherBuildItems.Add (new BuildItem ("AndroidJavaSource", "MyTest.java") { TextContent = () => "public class MyTest {}", Encoding = Encoding.ASCII }); - var b = CreateApkBuilder ("temp/MultiDexCustomMainDexFileList"); - b.ThrowOnBuildFailure = false; - Assert.IsTrue (b.Build (proj), "build should succeed. Run will fail."); - var data = File.ReadAllText (Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "multidex.keep")); - data = Regex.Replace (data, @"\r\n|\n\r|\n|\r", "\r\n"); - expected = Regex.Replace (expected, @"\r\n|\n\r|\n|\r", "\r\n"); - Assert.AreEqual (expected, data, "unexpected multidex.keep content"); - b.Clean (proj); - b.Dispose (); + using (var b = CreateApkBuilder (Path.Combine ("temp", $"{nameof (MultiDexCustomMainDexFileList)}{useD8}"))) { + Assert.IsTrue (b.Build (proj), "build should succeed. Run will fail."); + + //NOTE: d8 has the list in a different order, so we should do an unordered comparison + var actual = File.ReadAllLines (Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "multidex.keep")); + foreach (var item in expected) { + Assert.IsTrue (actual.Contains (item), $"multidex.keep did not contain `{item}`"); + } + Assert.AreEqual (expected.Length, actual.Length, "multidex.keep file contained more items than expected!"); + } } [Test] @@ -2130,12 +2161,13 @@ public void BuildReleaseApplication () } [Test] - public void BuildApplicationWithSpacesInPath ([Values (true, false)] bool isRelease, [Values (true, false)] bool enableProguard, [Values (true, false)] bool enableMultiDex) + public void BuildApplicationWithSpacesInPath ([Values (true, false)] bool enableMultiDex, [Values ("dx", "d8")] string dexTool, [Values ("", "proguard", "r8")] string linkTool) { var proj = new XamarinAndroidApplicationProject () { - IsRelease = isRelease, - AotAssemblies = isRelease, - EnableProguard = enableProguard, + IsRelease = true, + AotAssemblies = true, + DexTool = dexTool, + LinkTool = linkTool, }; proj.OtherBuildItems.Add (new BuildItem ("AndroidJavaLibrary", "Hello (World).jar") { BinaryContent = () => Convert.FromBase64String (@" UEsDBBQACAgIAMl8lUsAAAAAAAAAAAAAAAAJAAQATUVUQS1JTkYv/soAAAMAUEsHCAAAAAACAAAAA @@ -2153,9 +2185,8 @@ public void BuildApplicationWithSpacesInPath ([Values (true, false)] bool isRele if (enableMultiDex) proj.SetProperty ("AndroidEnableMultiDex", "True"); - if (isRelease) { - proj.Imports.Add (new Import ("foo.targets") { - TextContent = () => @" + proj.Imports.Add (new Import ("foo.targets") { + TextContent = () => @" @@ -2165,11 +2196,15 @@ public void BuildApplicationWithSpacesInPath ([Values (true, false)] bool isRele ", - }); - } - using (var b = CreateApkBuilder (Path.Combine ("temp", $"BuildReleaseAppWithA InIt({isRelease}{enableProguard}{enableMultiDex})"))) { + }); + using (var b = CreateApkBuilder (Path.Combine ("temp", $"BuildReleaseAppWithA InIt({enableMultiDex}{dexTool}{linkTool})"))) { Assert.IsTrue (b.Build (proj), "Build should have succeeded."); Assert.IsFalse (b.LastBuildOutput.ContainsText ("Duplicate zip entry"), "Should not get warning about [META-INF/MANIFEST.MF]"); + + var className = "Lmono/MonoRuntimeProvider;"; + var dexFile = b.Output.GetIntermediaryPath (Path.Combine ("android", "bin", "classes.dex")); + FileAssert.Exists (dexFile); + Assert.IsTrue (DexUtils.ContainsClass (className, dexFile, b.AndroidSdkDirectory), $"`{dexFile}` should include `{className}`!"); } } @@ -3027,24 +3062,28 @@ public void RunXABuildInParallel () } [Test] - [TestCaseSource ("DesugarChecks")] - public void Desugar (bool isRelease, bool enableDesugar, bool enableProguard) + public void Desugar ([Values (true, false)] bool isRelease, [Values ("dx", "d8")] string dexTool, [Values ("", "proguard", "r8")] string linkTool) { var proj = new XamarinAndroidApplicationProject () { IsRelease = isRelease, - EnableDesugar = enableDesugar, - EnableProguard = enableProguard, + EnableDesugar = true, //It is certain this test would fail without desugar + DexTool = dexTool, + LinkTool = linkTool, }; //Okhttp and Okio //https://github.com/square/okhttp //https://github.com/square/okio - if (enableProguard) { + if (!string.IsNullOrEmpty (linkTool)) { //NOTE: these are just enough rules to get it to build, not optimal - var rules = new [] { + var rules = new List { "-dontwarn com.google.devtools.build.android.desugar.**", "-dontwarn javax.annotation.**", "-dontwarn org.codehaus.mojo.animal_sniffer.*", }; + //NOTE: If using d8 + proguard, then proguard needs an additional rule because d8 is desugaring, which occurs *after* proguard + if (dexTool == "d8" && linkTool == "proguard") { + rules.Add ("-dontwarn java.lang.invoke.LambdaMetafactory"); + } //FIXME: We aren't de-BOM'ing proguard files? var encoding = new UTF8Encoding (encoderShouldEmitUTF8Identifier: false); var bytes = encoding.GetBytes (string.Join (Environment.NewLine, rules)); @@ -3107,16 +3146,14 @@ public void foo() AAAAAAAAAAAAPQAAAE1FVEEtSU5GL01BTklGRVNULk1GUEsBAhQAFAAICAgAJZFnS7uHtAn+AQAA 0QMAAAwAAAAAAAAAAAAAAAAAwwAAAExhbWJkYS5jbGFzc1BLBQYAAAAAAwADALcAAAD7AgAAAAA= ") }); - using (var builder = CreateApkBuilder (Path.Combine ("temp", TestContext.CurrentContext.Test.Name))) { - builder.ThrowOnBuildFailure = enableDesugar; - Assert.AreEqual (enableDesugar, builder.Build (proj), "Unexpected build result"); + using (var builder = CreateApkBuilder (Path.Combine ("temp", TestName))) { + Assert.IsTrue (builder.Build (proj), "Build should have succeeded"); Assert.IsFalse (builder.LastBuildOutput.ContainsText ("Duplicate zip entry"), "Should not get warning about [META-INF/MANIFEST.MF]"); - - if (enableDesugar) { - var className = "Lmono/MonoRuntimeProvider;"; - var dexFile = builder.Output.GetIntermediaryPath (Path.Combine ("android", "bin", "classes.dex")); - Assert.IsTrue (DexUtils.ContainsClass (className, dexFile, builder.AndroidSdkDirectory), $"`{dexFile}` should include `{className}`!"); - } + + var className = "Lmono/MonoRuntimeProvider;"; + var dexFile = builder.Output.GetIntermediaryPath (Path.Combine ("android", "bin", "classes.dex")); + FileAssert.Exists (dexFile); + Assert.IsTrue (DexUtils.ContainsClass (className, dexFile, builder.AndroidSdkDirectory), $"`{dexFile}` should include `{className}`!"); } } diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownProperties.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownProperties.cs index c0d0ca37714..dec43790f1a 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownProperties.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownProperties.cs @@ -17,6 +17,8 @@ public static class KnownProperties public const string BundleAssemblies = "BundleAssemblies"; public const string EnableProguard = "EnableProguard"; public const string AndroidEnableDesugar = "AndroidEnableDesugar"; + public const string AndroidDexTool = "AndroidDexTool"; + public const string AndroidLinkTool = "AndroidLinkTool"; public const string UseJackAndJill = "UseJackAndJill"; public const string AotAssemblies = "AotAssemblies"; diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidApplicationProject.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidApplicationProject.cs index a3804b50f52..3c84dd93d5e 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidApplicationProject.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/XamarinAndroidApplicationProject.cs @@ -85,6 +85,16 @@ public bool EnableDesugar { set { SetProperty (KnownProperties.AndroidEnableDesugar, value.ToString ()); } } + public string DexTool { + get { return GetProperty (KnownProperties.AndroidDexTool); } + set { SetProperty (KnownProperties.AndroidDexTool, value); } + } + + public string LinkTool { + get { return GetProperty (KnownProperties.AndroidLinkTool); } + set { SetProperty (KnownProperties.AndroidLinkTool, value); } + } + public string AndroidFastDeploymentType { get { return GetProperty (KnownProperties.AndroidFastDeploymentType); } set { SetProperty (KnownProperties.AndroidFastDeploymentType, value); } diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj index 989eed5a633..fede85c5b62 100644 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj @@ -569,6 +569,8 @@ + + @@ -732,6 +734,11 @@ proguard False + + {1bafa0cc-0377-46ce-ab7b-7bb2e7b62f63} + r8 + False + {E248B2CA-303B-4645-ADDC-9D4459D550FD} libZipSharp diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets index 7d513e8c940..fb6499f4fb7 100644 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets @@ -94,6 +94,9 @@ PreserveNewest + + PreserveNewest + PreserveNewest diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets index 50fd3e8dd35..16e1fe66470 100755 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets @@ -73,6 +73,8 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. + + @@ -278,9 +280,16 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. <_AndroidStampDirectory>$(IntermediateOutputPath)stamp\ - $(EnableProguard) - False - False + $(EnableProguard) + True + False + dx + d8 + + proguard + True + True + False 1G @@ -353,6 +362,14 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. + + + + @@ -947,6 +964,18 @@ because xbuild doesn't support framework reference assemblies. /> + + + + + + + + <_PropertyCacheItems Include="AndroidAotMode=$(AndroidAotMode)" /> <_PropertyCacheItems Include="ExplicitCrunch=$(AndroidExplicitCrunch)" /> - <_PropertyCacheItems Include="EnableProguard=$(AndroidEnableProguard)" /> + <_PropertyCacheItems Include="AndroidDexTool=$(AndroidDexTool)" /> + <_PropertyCacheItems Include="AndroidLinkTool=$(AndroidLinkTool)" /> <_PropertyCacheItems Include="UseSharedRuntime=$(AndroidUseSharedRuntime)" /> <_PropertyCacheItems Include="EmbedAssembliesIntoApk=$(EmbedAssembliesIntoApk)" /> <_PropertyCacheItems Include="AndroidLinkMode=$(AndroidLinkMode)" /> @@ -2115,12 +2145,12 @@ because xbuild doesn't support framework reference assemblies. Outputs="$(_AndroidLinkFlag)"> - + @@ -2550,7 +2581,7 @@ because xbuild doesn't support framework reference assemblies. + Condition=" '$(AndroidLinkTool)' == 'proguard' And '$(_ProguardProjectConfiguration)' != '' "> + Condition=" ('$(AndroidLinkTool)' != 'proguard' Or '$(_ProguardProjectConfiguration)' == '') And '$(AndroidEnableDesugar)' == 'True' "> @@ -2625,11 +2655,14 @@ because xbuild doesn't support framework reference assemblies. AlternativeJarFiles="@(_AlternativeJarForAppDx)" /> + + + + DependsOnTargets="_CompileToDalvikWithDx;_CompileToDalvikWithD8"> <_DexFile Include="$(IntermediateOutputPath)android\bin\dex\*.dex" /> diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets new file mode 100644 index 00000000000..f603cc270cd --- /dev/null +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets @@ -0,0 +1,137 @@ + + + + + + + + + + <_UseR8>False + <_UseR8 Condition=" ('$(AndroidLinkTool)' == 'r8' And '$(_ProguardProjectConfiguration)' != '') Or '$(AndroidEnableMultiDex)' == 'True' ">True + + <_R8EnableShrinking>False + <_R8EnableShrinking Condition=" '$(AndroidLinkTool)' == 'r8' ">True + + + + + + + + <_DexesToDelete Include="$(IntermediateOutputPath)android\bin\*.dex" /> + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/r8/r8.csproj b/src/r8/r8.csproj new file mode 100644 index 00000000000..5a4390d6714 --- /dev/null +++ b/src/r8/r8.csproj @@ -0,0 +1,16 @@ + + + Debug + 1bafa0cc-0377-46ce-ab7b-7bb2e7b62f63 + Exe + bin\$(Configuration) + v2.0 + + + + + + + + + diff --git a/src/r8/r8.props b/src/r8/r8.props new file mode 100644 index 00000000000..6d7600ad463 --- /dev/null +++ b/src/r8/r8.props @@ -0,0 +1,3 @@ + + + diff --git a/src/r8/r8.targets b/src/r8/r8.targets new file mode 100644 index 00000000000..826dedcade9 --- /dev/null +++ b/src/r8/r8.targets @@ -0,0 +1,119 @@ + + + + + + + + + + + _DownloadDepotTools; + _UnzipDepotTools; + _SetDepotToolsEnvironment; + _BootstrapDepotTools; + _BuildR8; + _CopyR8; + + + _SetDepotToolsEnvironment; + _CleanR8; + _CleanDepotTools; + + + + + + + + + <_DepotToolsZip>..\..\bin\Build$(Configuration)\depot_tools.zip + <_PathToDepotTools>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\..\bin\Build$(Configuration)\depot_tools')) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/Runtime-MultiDex/Mono.Android-TestsMultiDex.csproj b/tests/Runtime-MultiDex/Mono.Android-TestsMultiDex.csproj index 6814e784536..b716ea7acd1 100644 --- a/tests/Runtime-MultiDex/Mono.Android-TestsMultiDex.csproj +++ b/tests/Runtime-MultiDex/Mono.Android-TestsMultiDex.csproj @@ -20,6 +20,7 @@ False false true + d8 diff --git a/tests/Xamarin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.csproj b/tests/Xamarin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.csproj index a2bbf29f3d7..8ac906b0dca 100644 --- a/tests/Xamarin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.csproj +++ b/tests/Xamarin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.csproj @@ -17,6 +17,7 @@ Assets false True + d8 @@ -40,6 +41,7 @@ true false armeabi-v7a;x86 + r8 @@ -134,6 +136,7 @@ + diff --git a/tests/Xamarin.Forms-Performance-Integration/Droid/proguard.cfg b/tests/Xamarin.Forms-Performance-Integration/Droid/proguard.cfg new file mode 100644 index 00000000000..26d27461626 --- /dev/null +++ b/tests/Xamarin.Forms-Performance-Integration/Droid/proguard.cfg @@ -0,0 +1,2 @@ +-keep class android.support.v7.widget.** { *; } +-dontwarn android.support.v7.widget.* From 2e8aceac2f3ad711ecc396166d54f60debee2178 Mon Sep 17 00:00:00 2001 From: Jonathan Peppers Date: Tue, 23 Oct 2018 15:36:41 -0500 Subject: [PATCH 2/2] Requested changes from review + build refactoring ~~ xa-prep-tasks ~~ The `` MSBuild task now has an optional `HashHeader` property: - If set, a http HEAD request is made to the URL, and the destination file path gets a suffix added for the value. - To make this work properly, `DestinationFiles` is also an `[Output]` property, since these paths could change as a result. ~~ android-toolchain ~~ Chromium's `depot_tools` are downloaded the same as all of our other dependencies. I had to update the `` MSBuild task to have an optional `HashHeader` property and made `DestinationFiles` an ``. We can use this to populate `@(_DownloadedItem)`. `_UnzipFiles` also "bootstraps" `depot_tools`, and has to set/unset `$PATH` appropriately. I did this in the same place as the license acceptance for the Android SDK. ~~ r8.targets ~~ - Removed all of the `depot_tools` downloading code, and setup `android-toolchain` as a `@(ProjectReference)` - All targets that set `$PATH`, unset it when finished - Use `$(_GradleArgs)` to supply `--no-daemon` - `_CleanR8` should delete `*.jar` files in our build tree ~~ Xamarin.Android.Build.Tasks ~~ - Fixed any formatting that we noticed - Renamed `$(AndroidD8JarPath)` and `$(AndroidR8JarPath)` to be prefixed with `Android` - Refactored `$(IntermediateOutputPath)_dex_stamp` stamp file in `_CompileToDalvikWithDx` and `_CompileToDalvikWithD8` to match the new convention of `$(_AndroidStampDirectory)_CompileToDalvik.stamp` - `*.dex` files weren't in `FileWrites`??? - `CompileToDalvik` had a `DexOutputs` output property that was completely unused, so I removed it. Also removed extra log messages. --- Configuration.props | 1 + Documentation/guides/BuildProcess.md | 10 ++ .../android-toolchain.projitems | 12 ++ .../android-toolchain.targets | 57 +++++++--- .../DownloadUri.cs | 64 +++++++---- .../Tasks/CompileToDalvik.cs | 22 ---- .../Xamarin.Android.Common.targets | 15 ++- .../Xamarin.Android.D8.targets | 28 ++--- src/r8/r8.csproj | 7 ++ src/r8/r8.targets | 104 +++++------------- 10 files changed, 170 insertions(+), 150 deletions(-) diff --git a/Configuration.props b/Configuration.props index ec78ddb493c..cae8d38367a 100644 --- a/Configuration.props +++ b/Configuration.props @@ -64,6 +64,7 @@ $(AndroidSdkCmakeDirectory)\bin\cmake $(AndroidSdkCmakeDirectory)\bin\ninja $(AndroidToolchainDirectory)\ant + $(AndroidToolchainDirectory)\depot_tools $(AntDirectory)\bin $(HostOS) armeabi-v7a:x86 diff --git a/Documentation/guides/BuildProcess.md b/Documentation/guides/BuildProcess.md index b9e67b02486..f910446adbb 100644 --- a/Documentation/guides/BuildProcess.md +++ b/Documentation/guides/BuildProcess.md @@ -210,6 +210,11 @@ when packaing Release applications. This property is `False` by default. +- **AndroidD8JarPath** – The path to `d8.jar` for use with the + d8 dex-compiler. Defaults to a path in the Xamarin.Android + installation. For further information see our documentation on [D8 + and R8][d8-r8]. + - **AndroidDexTool** – An enum-style property with valid values of `dx` or `d8`. Indicates which Android [dex][dex] compiler is used during the Xamarin.Android build process. @@ -406,6 +411,11 @@ when packaing Release applications. produce the actual `AndroidManifest.xml`. The `$(AndroidManifest)` must contain the package name in the `/manifest/@package` attribute. +- **AndroidR8JarPath** – The path to `r8.jar` for use with the + r8 dex-compiler and shrinker. Defaults to a path in the + Xamarin.Android installation. For further information see our + documentation on [D8 and R8][d8-r8]. + - **AndroidSdkBuildToolsVersion** – The Android SDK build-tools package provides the **aapt** and **zipalign** tools, among others. Multiple different versions of the build-tools package diff --git a/build-tools/android-toolchain/android-toolchain.projitems b/build-tools/android-toolchain/android-toolchain.projitems index dd2a478e801..cd9480f3f03 100644 --- a/build-tools/android-toolchain/android-toolchain.projitems +++ b/build-tools/android-toolchain/android-toolchain.projitems @@ -3,6 +3,7 @@ https://dl.google.com/android/repository https://archive.apache.org/dist/ant/binaries + https://storage.googleapis.com/chrome-infra @@ -162,4 +163,15 @@ + + + + + True + x-goog-hash + + diff --git a/build-tools/android-toolchain/android-toolchain.targets b/build-tools/android-toolchain/android-toolchain.targets index 9b156b9a062..e778180a384 100644 --- a/build-tools/android-toolchain/android-toolchain.targets +++ b/build-tools/android-toolchain/android-toolchain.targets @@ -1,7 +1,8 @@ - - + + + ResolveReferences; @@ -27,17 +28,17 @@ Condition=" '%(HostOS)' == '$(HostOS)' Or '%(HostOS)' == '' "> + + + <_SdkStampFiles Include="@(_PlatformAndroidSdkItem->'$(AndroidToolchainDirectory)\sdk\.stamp-%(Identity)')" /> <_SdkStampFiles Include="@(_PlatformAntItem->'$(AntDirectory)\.stamp-%(Identity)')" /> - - <_DownloadedItem Include="@(_PlatformAndroidSdkItem->'$(AndroidToolchainCacheDirectory)\%(Identity).zip')" /> - <_DownloadedItem Include="@(_PlatformAndroidNdkItem->'$(AndroidToolchainCacheDirectory)\%(Identity).zip')" /> - <_DownloadedItem Include="@(_PlatformAntItem->'$(AndroidToolchainCacheDirectory)\%(Identity).zip')" /> - + DestinationFiles="@(_PlatformAndroidSdkItem->'$(AndroidToolchainCacheDirectory)\%(Identity).zip');@(_PlatformAndroidNdkItem->'$(AndroidToolchainCacheDirectory)\%(Identity).zip')"> + + + DestinationFiles="@(_PlatformAntItem->'$(AndroidToolchainCacheDirectory)\%(Identity).zip')"> + + + + + + + + <_SdkStampFiles Include="@(_DownloadedChromeItem->'$(ChromeToolsDirectory)\.stamp-%(FileName)')" /> + + + <_OriginalPath>$(PATH) + @@ -66,8 +82,8 @@ - - + + + + + + [SourceUris.Length]; using (var client = new HttpClient ()) { client.Timeout = TimeSpan.FromHours (3); for (int i = 0; i < SourceUris.Length; ++i) { - tasks [i] = DownloadFile (client, source, SourceUris [i], DestinationFiles [i].ItemSpec); + tasks [i] = DownloadFile (client, source, SourceUris [i], DestinationFiles [i]); } TTask.WaitAll (tasks, source.Token); } + DestinationFiles = tasks.Select (t => t.Result).ToArray (); + return !Log.HasLoggedErrors; } - async TTask DownloadFile (HttpClient client, CancellationTokenSource source, string uri, string destinationFile) + async Task DownloadFile (HttpClient client, CancellationTokenSource source, string uri, ITaskItem destinationFile) { - if (File.Exists (destinationFile)) { + if (!string.IsNullOrEmpty (HashHeader)) { + var hashSuffix = await CheckHashHeader (client, source, uri); + if (!string.IsNullOrEmpty (hashSuffix)) { + var directory = Path.GetDirectoryName (destinationFile.ItemSpec); + var fileName = Path.GetFileNameWithoutExtension (destinationFile.ItemSpec); + var extension = Path.GetExtension (destinationFile.ItemSpec); + destinationFile.ItemSpec = Path.Combine (directory, fileName + "-" + hashSuffix + extension); + Log.LogMessage (MessageImportance.Normal, $"Hash found using '{HashHeader}', destination file changing to '{destinationFile}'."); + } + } + if (File.Exists (destinationFile.ItemSpec)) { Log.LogMessage (MessageImportance.Normal, $"Skipping uri '{uri}' as destination file already exists '{destinationFile}'."); - return; + return destinationFile; } - var dp = Path.GetDirectoryName (destinationFile); - var dn = Path.GetFileName (destinationFile); + var dp = Path.GetDirectoryName (destinationFile.ItemSpec); + var dn = Path.GetFileName (destinationFile.ItemSpec); var tempPath = Path.Combine (dp, "." + dn + ".download"); Directory.CreateDirectory(dp); @@ -82,12 +87,33 @@ async TTask DownloadFile (HttpClient client, CancellationTokenSource source, str } } Log.LogMessage (MessageImportance.Low, $"mv '{tempPath}' '{destinationFile}'."); - File.Move (tempPath, destinationFile); + File.Move (tempPath, destinationFile.ItemSpec); } catch (Exception e) { Log.LogError ("Unable to download URL `{0}` to `{1}`: {2}", uri, destinationFile, e.Message); Log.LogErrorFromException (e); } + return destinationFile; + } + + async Task CheckHashHeader (HttpClient client, CancellationTokenSource source, string uri) + { + var request = new HttpRequestMessage (HttpMethod.Head, uri); + using (var response = await client.SendAsync (request, source.Token)) { + response.EnsureSuccessStatusCode (); + if (response.Headers.TryGetValues (HashHeader, out var values)) { + foreach (var value in values) { + Log.LogMessage (MessageImportance.Low, $"{HashHeader}: {value}"); + + //Current format: `x-goog-hash: crc32c=8HATIw==` + if (!string.IsNullOrWhiteSpace (value)) { + return value.Trim (); + } + } + } + } + + return null; } } } diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs b/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs index 92c273bf747..7de8e800c69 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs @@ -36,9 +36,6 @@ public class CompileToDalvik : JavaToolTask public string MultiDexMainDexListFile { get; set; } - [Output] - public string [] DexOutputs { get; set; } - string inputListFile; protected override string ToolName { @@ -51,21 +48,6 @@ protected override string ToolName { public override bool Execute () { - Log.LogDebugMessage ("CompileToDalvik"); - Log.LogDebugMessage (" JavaOptions: {0}", JavaOptions); - Log.LogDebugMessage (" JavaMaximumHeapSize: {0}", JavaMaximumHeapSize); - Log.LogDebugMessage (" ClassesOutputDirectory: {0}", ClassesOutputDirectory); - Log.LogDebugMessage (" JavaToolPath: {0}", JavaToolPath); - Log.LogDebugMessage (" DxJarPath: {0}", DxJarPath); - Log.LogDebugMessage (" ToolExe: {0}", ToolExe); - Log.LogDebugMessage (" ToolPath: {0}", ToolPath); - Log.LogDebugMessage (" UseDx: {0}", UseDx); - Log.LogDebugMessage (" DxExtraArguments: {0}", DxExtraArguments); - Log.LogDebugMessage (" MultiDexEnabled: {0}", MultiDexEnabled); - Log.LogDebugMessage (" MultiDexMainDexListFile: {0}", MultiDexMainDexListFile); - Log.LogDebugTaskItems (" JavaLibrariesToCompile:", JavaLibrariesToCompile); - Log.LogDebugTaskItems (" AlternativeJarFiles:", AlternativeJarFiles); - if (!Directory.Exists (ClassesOutputDirectory)) Directory.CreateDirectory (ClassesOutputDirectory); @@ -73,10 +55,6 @@ public override bool Execute () inputListFile = Path.GetTempFileName (); try { ret = base.Execute (); - - DexOutputs = Directory.GetFiles (Path.GetDirectoryName (ClassesOutputDirectory), "*.dex", SearchOption.TopDirectoryOnly); - - Log.LogDebugTaskItems (" DexOutputs: ", DexOutputs); } catch (FileNotFoundException ex) { Log.LogCodedError ("XA1003", ex.ToString ()); } finally { diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets index 16e1fe66470..fb3950df16b 100755 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets @@ -965,14 +965,14 @@ because xbuild doesn't support framework reference assemblies. - - @@ -2549,7 +2549,7 @@ because xbuild doesn't support framework reference assemblies. DependsOnTargets="$(_BeforeCompileToDalvikWithDx);$(_CompileToDalvikDependsOnTargets)" Condition=" '$(AndroidDexTool)' == 'dx' " Inputs="$(_CompileToDalvikInputs)" - Outputs="$(IntermediateOutputPath)_dex_stamp"> + Outputs="$(_AndroidStampDirectory)_CompileToDalvik.stamp"> - + - + @@ -3123,7 +3123,6 @@ because xbuild doesn't support framework reference assemblies. - diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets index f603cc270cd..a7080236479 100644 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets @@ -15,10 +15,10 @@ Copyright (C) 2018 Xamarin. All rights reserved. + Condition=" '$(AndroidDexTool)' == 'd8' " + DependsOnTargets="$(_CompileToDalvikDependsOnTargets)" + Inputs="$(_CompileToDalvikInputs)" + Outputs="$(_AndroidStampDirectory)_CompileToDalvik.stamp"> @@ -31,12 +31,12 @@ Copyright (C) 2018 Xamarin. All rights reserved. @@ -77,8 +77,8 @@ Copyright (C) 2018 Xamarin. All rights reserved. /> + Include="$(IntermediateOutputPath)proguard\__proguard_output__.jar" + Condition=" '$(AndroidLinkTool)' == 'proguard' And '$(_ProguardProjectConfiguration)' != '' "> @@ -87,7 +87,7 @@ Copyright (C) 2018 Xamarin. All rights reserved. ToolPath="$(JavaToolPath)" JavaMaximumHeapSize="$(JavaMaximumHeapSize)" JavaOptions="$(JavaOptions)" - JarPath="$(R8JarPath)" + JarPath="$(AndroidR8JarPath)" AndroidManifestFile="$(IntermediateOutputPath)android\AndroidManifest.xml" OutputDirectory="$(IntermediateOutputPath)android\bin" Debug="$(AndroidIncludeDebugSymbols)" @@ -114,7 +114,7 @@ Copyright (C) 2018 Xamarin. All rights reserved. ToolPath="$(JavaToolPath)" JavaMaximumHeapSize="$(JavaMaximumHeapSize)" JavaOptions="$(JavaOptions)" - JarPath="$(D8JarPath)" + JarPath="$(AndroidD8JarPath)" AndroidManifestFile="$(IntermediateOutputPath)android\AndroidManifest.xml" OutputDirectory="$(IntermediateOutputPath)android\bin" Debug="$(AndroidIncludeDebugSymbols)" @@ -127,9 +127,9 @@ Copyright (C) 2018 Xamarin. All rights reserved. ExtraArguments="$(AndroidD8ExtraArguments)" /> - + - + diff --git a/src/r8/r8.csproj b/src/r8/r8.csproj index 5a4390d6714..94e1d2b6ac0 100644 --- a/src/r8/r8.csproj +++ b/src/r8/r8.csproj @@ -12,5 +12,12 @@ + + + {8FF78EB6-6FC8-46A7-8A15-EBBA9045C5FA} + android-toolchain + False + + diff --git a/src/r8/r8.targets b/src/r8/r8.targets index 826dedcade9..fff9a42d62e 100644 --- a/src/r8/r8.targets +++ b/src/r8/r8.targets @@ -1,119 +1,75 @@ - - - + + <_GradleArgs>--no-daemon - _DownloadDepotTools; - _UnzipDepotTools; - _SetDepotToolsEnvironment; - _BootstrapDepotTools; + ResolveReferences; _BuildR8; _CopyR8; - _SetDepotToolsEnvironment; _CleanR8; - _CleanDepotTools; - - - - <_DepotToolsZip>..\..\bin\Build$(Configuration)\depot_tools.zip - <_PathToDepotTools>$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\..\bin\Build$(Configuration)\depot_tools')) - - - - - - - - - - - - + + + <_OriginalPath>$(PATH) + - - - - - - - - - - - - - - - - - + - + + <_OriginalPath>$(PATH) + + + +