-
Notifications
You must be signed in to change notification settings - Fork 534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[build] Use Microsoft OpenJDK 21.0.5 #9651
Conversation
Context: 298da11 Does It Build™? We have customers asking if they can use JDK 21. @jonpryor has no idea! Let's find out! Update `Step_InstallAdoptOpenJDK.MacOS.cs` to run `xattr -d -r com.apple.quarantine $HOME/android-toolchain/jdk-21`. This is because `tar xzf` will apply the "quarantined" attribute to extracted files, which prevents execution of e.g. `javac`: % $HOME/android-toolchain/jdk-21/bin/javac zsh: operation not permitted: ./jdk-21/bin/javac Removing the `com.apple.quarantine` attribute fixes this.
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Context: #9651 Recent versions of macOS, Safari, and **tar**(1) [^0] interact such that if you manually download a tarball and extract it, *all the extracted files* contain the `com.apple.quarantine` extended attribute. This is a security feature, but it also means that the provisioned JDK *cannot be used*: % $HOME/android-toolchain/jdk-21/bin/javac zsh: operation not permitted: ./jdk-21/bin/javac Which in turn means if you do something "reasonable" like download Microsoft OpenJDK and place it into `$HOME/android-archives` -- so that `xaprepare` doesn't need to download it *again* -- then the provisioned JDK will be *unusable*. Which makes @jonpryor sad. Update `Step_InstallAdoptOpenJDK.MacOS.cs` to run `xattr -d -r com.apple.quarantine $HOME/android-toolchain/jdk-21`. This will *delete* the offending extended attribute, allowing e.g. `javac` to run without error. [^0]: Which versions? ¯\_(ツ)_/¯
Context: #9651 Recent versions of macOS, Safari, and **tar**(1) [^0] interact such that if you manually download a tarball and extract it, *all the extracted files* contain the `com.apple.quarantine` extended attribute. This is a security feature, but it also means that the provisioned JDK *cannot be used*: % $HOME/android-toolchain/jdk-21/bin/javac zsh: operation not permitted: ./jdk-21/bin/javac Which in turn means if you do something "reasonable" like download Microsoft OpenJDK and place it into `$HOME/android-archives` -- so that `xaprepare` doesn't need to download it *again* -- then the provisioned JDK will be *unusable*. Which makes @jonpryor sad. Update `Step_InstallAdoptOpenJDK.MacOS.cs` to run `xattr -d -r com.apple.quarantine $HOME/android-toolchain/jdk-21`. This will *delete* the offending extended attribute, allowing e.g. `javac` to run without error. [^0]: Which versions? ¯\\_(ツ)_/¯
Context: 5bb0d24 Context: 4273e5c Context: 0355acf Context: dotnet/android#9651 Does dotnet/android build with JDK-21? We don't know! But in order to answer that question, dotnet/java-interop needs to be able to build under JDK-21; a'la 5bb0d24: 1. [Install JDK-21][0] 2. Run `dotnet build -t:Prepare`, overriding `$(JdksRoot)`: dotnet build -t:Prepare Java.Interop.sln -p:JdksRoot=/Library/Java/JavaVirtualMachines/microsoft-21.jdk/Contents/Home 3. Build: `dotnet build` Unfortunately, this *fails* for three reasons: 1. `class-parse` crashes when processing `java.base.jmod`. 2. `src/Java.Base` needs updates to bind JDK-21's `java.base.jmod`. 3. On Linux and Windows, Gradle 8.1 and JDK-21 don't mix. ~~ class-parse ~~ This only impacts Debug builds of dotnet/java-interop, but: % dotnet "bin/Debug-net8.0/class-parse.dll" \ "$HOME/android-toolchain/jdk-21/jmods/java.base.jmod" \ "-o=obj/Debug-net8.0//mcw/api.xml" … Process terminated. Assertion failed. Unexpected number of method parameters in `Ljdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter$Instantiation;.get(Ljava/lang/Object;)Ljava/lang/Object;`: expected 1, got 0 at Xamarin.Android.Tools.Bytecode.MethodInfo.UpdateParametersFromMethodParametersAttribute(ParameterInfo[] parameters) in …/src/Xamarin.Android.Tools.Bytecode/Methods.cs:line 308 … The assertion? var pinfo = methodParams.ParameterInfo; int startIndex = 0; while (startIndex < pinfo.Count && pinfo [startIndex].AccessFlags.HasFlag (.Synthetic)) startIndex++; Debug.Assert (parameters.Length == pinfo.Count - startIndex, …); This is part of 4273e5c and 0355acf, attempting to "skip over" the constructor parameters which non-static inner classes have which contain the outer class instance: // Java class Outer { /* non-static */ class Inner { public Inner () { … } } } At the ABI boundary, the `Outer.Inner` constructor is actually: /* partial */ class Outer { /* partial */ class Inner { public Inner (Outer outer) { … } } } and we need to skip over the first parameter. Which brings us to `jdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter.Instantiation.get()`: % mkdir _x % unzip $HOME/android-toolchain/jdk-21/jmods/java.base.jmod -d _x % dotnet bin/Debug-net8.0/class-parse.dll --dump \ _x/classes/jdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter\$Instantiation.class … 8: get (Ljava/lang/Object;)Ljava/lang/Object; Public, Bridge, Synthetic Code(6, Unknown[LineNumberTable](6), LocalVariableTableAttribute(LocalVariableTableEntry(Name='this', Descriptor='Ljdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter$Instantiation;', StartPC=0, Index=0))) MethodParametersAttribute(MethodParameterInfo(Name='', AccessFlags=Final, Synthetic)) The parameter for `JSRInlinerAdapter.Instantiation.get(Object)` is `Synthetic`, causing us to skip over it, which is why we have a parameter mismatch. The thing is, this parameter *shouldn't* be skipped; the skipping is intended for *constructor* parameters! Update the code so that the loop only occurs for constructors. This allows `class-parse` to *not assert*, allowing `src/Java.Base` to build. ~~ Gradle ~~ Via dotnet/android#9651, `gradle` fails when building `tools/java-source-utils`, but only on Linux and Windows: "/mnt/vss/_work/1/s/xamarin-android/external/Java.Interop/build-tools/gradle/gradlew" -d --stacktrace --no-daemon -PjavaSourceVer=11 -PjavaTargetVer=11 jar … [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: [org.gradle.internal.buildevents.BuildExceptionReporter] Could not open settings generic class cache for settings file '/mnt/vss/_work/1/s/xamarin-android/external/Java.Interop/tools/java-source-utils/settings.gradle' (/home/cloudtest/.gradle/caches/8.1.1/scripts/aiw0k2bokig45bv5yvkog3o3j). [org.gradle.internal.buildevents.BuildExceptionReporter] > BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 65 [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try: [org.gradle.internal.buildevents.BuildExceptionReporter] > Run with --scan to get full insights. [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is: [org.gradle.internal.buildevents.BuildExceptionReporter] org.gradle.cache.CacheOpenException: Could not open settings generic class cache for settings file '/mnt/vss/_work/1/s/xamarin-android/external/Java.Interop/tools/java-source-utils/settings.gradle' (/home/cloudtest/.gradle/caches/8.1.1/scripts/aiw0k2bokig45bv5yvkog3o3j). [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.cache.internal.DefaultPersistentDirectoryStore.open(DefaultPersistentDirectoryStore.java:91) [org.gradle.internal.buildevents.BuildExceptionReporter] … [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 65 [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler.compileScript(DefaultScriptCompilationHandler.java:147) [org.gradle.internal.buildevents.BuildExceptionReporter] … [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 65 [org.gradle.internal.buildevents.BuildExceptionReporter] at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:199) 🤔 As per the [Gradle Compatibility Matrix][1], Java 21 requires Gradle 8.5 or later. (No idea why this works on macOS…) Bump to Gradle 8.12, whic is the current latest stable version. ~~ TODO ~~ While dotnet/java-interop now *builds* with JDK-21, unit tests don't fully pass. In particular, `tests/Xamarin.Android.Tools.Bytecode-Tests` will need to be updated because `javac` output has changed. (Again.) [0]: https://learn.microsoft.com/en-us/java/openjdk/download#openjdk-21 [1]: https://docs.gradle.org/8.12/userguide/compatibility.html
Try dotnet/java-interop#1287, which contains a Gradle bump, which is probably needed because previously Linux & Windows failed to build because Gradle 8.1 and JDK-21 don't mix. Update `$(LatestSupportedJavaVersion)` to 21.0.99 so that our unit tests will actually *try* to do something under JDK-21, instead of insta-failing with an XA0030 error: > error XA0030: Building with JDK version `21.0.5` is not supported. > Please install JDK version `17.0`. See https://aka.ms/xamarin/jdk9-errors Doh. Update to Gradle 8.12, for harmony with dotnet/java-interop#1287.
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Context: #9651 Recent versions of macOS, Safari, and **tar**(1) [^0] interact such that if you manually download a tarball and extract it, *all the extracted files* contain the `com.apple.quarantine` extended attribute. This is a security feature, but it also means that the provisioned JDK *cannot be used*: % $HOME/android-toolchain/jdk-21/bin/javac zsh: operation not permitted: ./jdk-21/bin/javac Which in turn means if you do something "reasonable" like download Microsoft OpenJDK and place it into `$HOME/android-archives` -- so that `xaprepare` doesn't need to download it *again* -- then the provisioned JDK will be *unusable*. Which makes @jonpryor sad. Update `Step_InstallAdoptOpenJDK.MacOS.cs` to run `xattr -d -r com.apple.quarantine $HOME/android-toolchain/jdk-21`. This will *delete* the offending extended attribute, allowing e.g. `javac` to run without error. [^0]: Which versions? ¯\\_(ツ)_/¯
Context: 5bb0d24 Context: 4273e5c Context: 0355acf Context: dotnet/android#9651 Does dotnet/android build with JDK-21? We don't know! But in order to answer that question, dotnet/java-interop needs to be able to build under JDK-21; a'la 5bb0d24: 1. [Install JDK-21][0] 2. Run `dotnet build -t:Prepare`, overriding `$(JdksRoot)` to refer to the JDK-21 installation directory: dotnet build -t:Prepare Java.Interop.sln -p:JdksRoot=/Library/Java/JavaVirtualMachines/microsoft-21.jdk/Contents/Home 3. Build: `dotnet build` Unfortunately, this *fails* for three reasons: 1. `class-parse` crashes when processing JDK-21's `java.base.jmod`. 2. `src/Java.Base` needs updates to bind JDK-21's `java.base.jmod`. 3. On Linux and Windows, Gradle 8.1 and JDK-21 don't mix. ~~ class-parse ~~ This only impacts Debug builds of dotnet/java-interop, but: % dotnet "bin/Debug-net8.0/class-parse.dll" \ "$HOME/android-toolchain/jdk-21/jmods/java.base.jmod" \ "-o=obj/Debug-net8.0//mcw/api.xml" … Process terminated. Assertion failed. Unexpected number of method parameters in `Ljdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter$Instantiation;.get(Ljava/lang/Object;)Ljava/lang/Object;`: expected 1, got 0 at Xamarin.Android.Tools.Bytecode.MethodInfo.UpdateParametersFromMethodParametersAttribute(ParameterInfo[] parameters) in …/src/Xamarin.Android.Tools.Bytecode/Methods.cs:line 308 … The assertion? var pinfo = methodParams.ParameterInfo; int startIndex = 0; while (startIndex < pinfo.Count && pinfo [startIndex].AccessFlags.HasFlag (.Synthetic)) startIndex++; Debug.Assert (parameters.Length == pinfo.Count - startIndex, …); This is part of 4273e5c and 0355acf, attempting to "skip over" the constructor parameters which non-static inner classes have which contain the outer class instance: // Java class Outer { /* non-static */ class Inner { public Inner () { … } } } At the ABI boundary, the `Outer.Inner` constructor is actually: // "Equivalent" Java for JNI purposes /* partial */ class Outer { /* partial */ class Inner { public Inner (Outer outer) { … } // note added constructor parameter } } and we need to skip over the first parameter. Which brings us to `jdk.internal.org.objectweb.asm.commons.JSRInlinerAdapter.Instantiation.get()`: % mkdir _x % unzip $HOME/android-toolchain/jdk-21/jmods/java.base.jmod -d _x % dotnet bin/Debug-net8.0/class-parse.dll --dump \ _x/classes/jdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter\$Instantiation.class … 8: get (Ljava/lang/Object;)Ljava/lang/Object; Public, Bridge, Synthetic Code(6, Unknown[LineNumberTable](6), LocalVariableTableAttribute(LocalVariableTableEntry(Name='this', Descriptor='Ljdk/internal/org/objectweb/asm/commons/JSRInlinerAdapter$Instantiation;', StartPC=0, Index=0))) MethodParametersAttribute(MethodParameterInfo(Name='', AccessFlags=Final, Synthetic)) The parameter for `JSRInlinerAdapter.Instantiation.get(Object)` is `Synthetic`, causing us to skip over it, which is why we have a parameter mismatch. The thing is, this parameter *shouldn't* be skipped; the skipping is intended for *constructor* parameters! Update the code so that the loop only occurs for constructors. This allows `class-parse` to *not assert*, allowing `src/Java.Base` to build. ~~ Gradle ~~ Via dotnet/android#9651, `gradle` fails when building `tools/java-source-utils`, but only on Linux and Windows: "/mnt/vss/_work/1/s/xamarin-android/external/Java.Interop/build-tools/gradle/gradlew" -d --stacktrace --no-daemon -PjavaSourceVer=11 -PjavaTargetVer=11 jar … [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: [org.gradle.internal.buildevents.BuildExceptionReporter] Could not open settings generic class cache for settings file '/mnt/vss/_work/1/s/xamarin-android/external/Java.Interop/tools/java-source-utils/settings.gradle' (/home/cloudtest/.gradle/caches/8.1.1/scripts/aiw0k2bokig45bv5yvkog3o3j). [org.gradle.internal.buildevents.BuildExceptionReporter] > BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 65 [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try: [org.gradle.internal.buildevents.BuildExceptionReporter] > Run with --scan to get full insights. [org.gradle.internal.buildevents.BuildExceptionReporter] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is: [org.gradle.internal.buildevents.BuildExceptionReporter] org.gradle.cache.CacheOpenException: Could not open settings generic class cache for settings file '/mnt/vss/_work/1/s/xamarin-android/external/Java.Interop/tools/java-source-utils/settings.gradle' (/home/cloudtest/.gradle/caches/8.1.1/scripts/aiw0k2bokig45bv5yvkog3o3j). [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.cache.internal.DefaultPersistentDirectoryStore.open(DefaultPersistentDirectoryStore.java:91) [org.gradle.internal.buildevents.BuildExceptionReporter] … [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 65 [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler.compileScript(DefaultScriptCompilationHandler.java:147) [org.gradle.internal.buildevents.BuildExceptionReporter] … [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 65 [org.gradle.internal.buildevents.BuildExceptionReporter] at groovyjarjarasm.asm.ClassReader.<init>(ClassReader.java:199) 🤔 As per the [Gradle Compatibility Matrix][1], Java 21 requires Gradle 8.5 or later. (No idea why this works on macOS…) Bump to Gradle 8.12, which is the current latest stable version. ~~ TODO ~~ While dotnet/java-interop now *builds* with JDK-21, unit tests don't fully pass. In particular, `tests/Xamarin.Android.Tools.Bytecode-Tests` will need to be updated because `javac` output has changed. (Again.) [0]: https://learn.microsoft.com/en-us/java/openjdk/download#openjdk-21 [1]: https://docs.gradle.org/8.12/userguide/compatibility.html
Otherwise we're trying to replace the JDK-17 install with our own JDK-21, which fails horrifically.
/azp run |
Pull request contains merge conflicts. |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Tests are now actually run. So how are we faring? CheckSignApk(False,True) is failing because it emitted a warning:
Which is also why |
So what is our "offending"
|
Fortunately this is actually easy to repro: I thus turn to the man page: % man $HOME/android-toolchain/jdk-21/man/man1/javac.1 Which has some interesting things in there:
And indeed, Unfortunately, the docs for
No mention of "modules" or setting a "system modules path". …after remembering that
|
Fortunately JDK-17 supports |
Hopefully fixes Hopefully fies the warning: > JAVAC : warning : [options] system modules path not set in conjunction with -source 17
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
Using But not all. 17 are failing. MauiTargetFramework("net8.0","android",34) fails because it's trying to build net8.0-android34, which fails with XA0030 because .NET 8's DotNetPack("net8.0","android",34) is the same. As is ApplicationRunsWithDebuggerAndBreaks(True,False,null,"apk",False), and all the others. |
Context: #9651 PR #9651 demonstrated that it was fairly straightforward to use JDK-21 on CI. We don't want to fully bump to JDK-21, because we still support building with JDK-17, so we *at minimum* need to run tests on both JDK-17 and JDK-21. As a "minimal introductory step", add support for JDK-21: * Update to Gradle 8.12, for harmony with dotnet/java-interop. * Update the `<Javac/>` task to use `javac --release N` when using JDK-17 and later. JDK-11 doesn't support `javac --release`. (Why care about JDK-11? Because .NET 8 still supports it, and this will make future cherry-picking easier.) * Set `$(LatestSupportedJavaVersion)`=21.0.99, which removes the XA0030 error which would result from using JDK-21. * Update `tools/workload-dependencies` to use `$(LatestSupportedJavaVersion)` property, instead of always using `$(JavaSdkVersion)+1`.
* [build] Support JDK-21 Context: #9651 PR #9651 demonstrated that it was fairly straightforward to use JDK-21 on CI. We don't want to fully bump to JDK-21, because we still support building with JDK-17, so we *at minimum* need to run tests on both JDK-17 and JDK-21. As a "minimal introductory step", add support for JDK-21: * Update to Gradle 8.12, for harmony with dotnet/java-interop. * Update the `<Javac/>` task to use `javac --release N` when using JDK-17 and later. JDK-11 doesn't support `javac --release`. (Why care about JDK-11? Because .NET 8 still supports it, and this will make future cherry-picking easier.) * Set `$(LatestSupportedJavaVersion)`=21.0.99, which removes the XA0030 error which would result from using JDK-21. * Update `tools/workload-dependencies` to use `$(LatestSupportedJavaVersion)` property, instead of always using `$(JavaSdkVersion)+1`. * Address comments.
Context: #9651 Context: 14a6bfb Context: df68c20 As of 14a6bfb, main/.NET 10 supports both JDK 17 and 21. The default version that VS and the `InstallAndroidDependencies` target will install is currently JDK 17, though we may want to upgrade this to JDK 21 as we move further into the .NET 10 cycle. In order to increase our JDK test coverage across versions, the macOS, Windows, and Linux build stages have been updated to install and build with Microsoft OpenJDK 21.0.5. The nightly test jobs have also been updated to use JDK 21, while all other test jobs will continue to use JDK 17. JDK 11 support was removed in df68c20, though customers should be able to add: `<MinimumSupportedJavaVersion>11.0</MinimumSupportedJavaVersion>` to their project files to use it. Some test coverage for this exists in [ValidateJavaVersionTests][0], but this is not fully tested end to end. [0]: https://github.com/dotnet/android/blob/766ac338446f99129168dc02a16669882f6bd34e/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/ValidateJavaVersionTests.cs
Context: 298da11
Does It Build™?
We have customers asking if they can use JDK 21.
@jonpryor has no idea!
Let's find out!
Update
Step_InstallAdoptOpenJDK.MacOS.cs
to runxattr -d -r com.apple.quarantine $HOME/android-toolchain/jdk-21
. This is becausetar xzf
will apply the "quarantined" attribute to extracted files, which prevents execution of e.g.javac
:Removing the
com.apple.quarantine
attribute fixes this.