Upgrade Enso to GraalVM for JDK 21

[JaroslavTulach]

There is new version of GraalVM for JDK21 since yesterday. Here is some information about Truffle Unchained. The upgrade is going to be tougher this time:

Tasks

Preview Give feedback

1. Change the GraalVM version check to non-fatal error - done in Adjusting to GraalVM for JDK21+ #7855
2. Avoid boostrap & copy of truffle-api: ./bin/java -d org.graalvm.truffle seems to exports com.oracle.truffle.api & co. - done in Adjusting to GraalVM for JDK21+ #7855
3. Update documentation that mentiones sbt bootstrap
4. gu is gone! Stop calling it... [GR-46219] Remove the GraalVM Updater oracle/graal#6855 - - done in Adjusting to GraalVM for JDK21+ #7855
5. option -Dtruffle.class.path.append option is deprecated - we need to build module-info.java for the runtime
6. Enable GraalPy on Windows: - Basic Windows support for GraalPy oracle/graalpython#321
7. Fix a lot of Inexhaustive_Pattern_Match.Error TRegexReadOnlyIntArray in test/Tests

Follow-ups

• Remove dumb terminal - implement jline #8313
• Install GraalPython  #8294
• Fix stepping over for Enso debugging #8290
• Remove GraalVM distribution downloading facility #8280
• Add chromeinspector tool #8314
• Re-enable CurriedFunctionBenchmarks #8321

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-06):

Progress: - Introducing modules into our sbt builds

• Trying to compile all the java sources
• Debugging a problem with sbt / zinc problem with mixed compilation of java and scala - cannot compile module-info.java.
• sbt update does not work
• maybe update scala? It should be finished by 2023-10-09.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-09):

Progress: - sbt has a bug - cannot use module-info.java in mixed projects. Luckily, we need just one module-info in pure Java project runtime-with-instruments.

• Still trying to compile just runtime project. Some progress.
• sbt mixes together --module-path and --class-path in compilation and that causes problems.
• Out BuiltinMethodProcessor fails on NoClassDefFoundError because of it. It should be finished by 2023-10-09.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for yesterday (2023-10-10):

Summary: There is 11 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task.
It will cause 11 days delay for the delivery of this weekly plan.

Delay Cause: The initial estimate was too optimistic. Moreover, I worked on different issues in the meantime.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-10-10):

Progress: - Struggling with compilation of runtime.

• Still failing on IllegalAccessError in annotation processor, even though javac receives correct --add-exports argument It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-11):

Progress: - Removed jdk.compiler dependency #8027

• Finally compiled runtime.
• Trying to assembly fat jar with module-info. Thinking where to put module-info.java - in runtime or in runtime-with-instruments? It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-13):

Progress: - Still fixing the compilation of runtime and runtime-with-instruments.

• The progress so far:
  • Implement a workaround for module-info.java compilation in sbt's mixed projects.
  • Add module-info.java to the runtime project.
  • Implement a custom merge strategy for module-info.class files, so that in the Uber jar, we have only one module-info.class file from org.enso.runtime module
  • Assemble engine-runner.
• Everything be able to compile, but the running of runner.jar still fails.
  • I will need to look into the correct javaOptions for that. It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-10-16):

Progress: - Book club, mails, chat, ...

• Still looking into JPMS It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-17):

Progress: - Seems like we will have to ditch fat jars

• Looking into how we can ditch fat jars.
• Getting inspiration from how graalpython and FastR are built and run (module-path, class-path, ...) It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-10-19):

Progress: - Tweaking sbt to provide two fat Jars - runtime.jar and runner.jar. Those fat jars should be almost exactly the same as are now.

• Experimenting with running it.
• The fat jars need to be explicit modules
• Explicit modules cannot have conflicting packages, otherwise JVM initialization fails. It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-20):

Progress: - Investigating how to assembly runner.jar fat jar that has exclusive packages to runtime.jar fat jar. It should be finished by 2023-10-20.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2023-10-23):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task.
It will cause 7 days delay for the delivery of this weekly plan.

Delay Cause: A lot of struggling with assembling fat jars and running them.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-23):

Progress: - Consultation with Jaroslav: Unblocked the problem with runner.jar as fat jar.

• runner.jar shall not be a module, moreover, it shall not be provided on class-path, neither on module-path
• Let's provide only runtime.jar as fat jar that shall be an explicit module provided on the module-path and that shall provide all the Enso-related lagnagues (enso, epb) and instruments
• There will be runner.jar that will be loaded via our custom class loader.
• Trying to launch all this stuff - simple -h option works, but still cannot run a simple Enso script
  • Currently is failing on "ServiceConfigurationError: org.sl4j.spi.SLF4JServiceProvider: module org.enso runtime does not declare uses`" It should be finished by 2023-10-27.

[JaroslavTulach]

A lot of struggling with assembling fat jars and running them.

Yes, the problems with fat JARs were hard to solve. Their occurrence was a kind of unknown unknown to me. I apologize for not being around sooner to help overcome the problems with more advanced ClassLoader usage. I hope you are unblocked now.

will cause 7 days delay

Go for it! Still there is still going to be quite a few know unknowns, so the deadline is tough. Good luck.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-24):

Progress: - Book club.

• Remove slf4j and logback from runtime.jar.
  • We should put them into module-path and treat them as explicit modules.
• Fixing buildEngineDistribution sbt command.
  • We need to copy all the necessary Jar modules inside component directory.
• buildEngineDistribution seems to be working - a simple enso -h works.
• Trying to add graaljs modules, and copy them to the distribution It should be finished by 2023-10-27.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-25):

Progress: - Filter away all the graal-related JAR modules from withing our generated polyglot directories in std libs

• Some small refactorings to build.sbt.
• Fixing runtime / test configuration - module path, --add-exports, etc., to Test/javaOptions.
  • testOnly *VectorTest works now.
• Simple script with simple IO.println works now. It should be finished by 2023-10-27.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-26):

Progress: - Trying to fix class loading issues in runtime / test.

• Trying to run Enso with nothing on module-path and everything on class-path.
  • Inspiration in graal-js and graalpy.
• Disabling class-path isolation with -Dpolyglotimpl.DisableClassPathIsolation=true works, but is not desirable, even for tests.
  • Because then, there is no Truffle runtime compilation at all.
• Debugging polyglot engine initialization errors - some ClassCastException that seems to be caused by different class loaders in ReplInstrumentProvider?
  • Maybe our MessageEndpoint initialization is wrong? It should be finished by 2023-10-27.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-27):

Progress: - Reviews

• Fixing unit tests by adding everything on class-path and disabling class-path isolation.
• Fixing ./run build script.
• Other small fixes: Add special handling for Dataflow_Error passed to Runtime.assert #8168 and Fix bench-processor test #8166
• Starting to look into the native image build. It should be finished by 2023-10-27.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2023-10-30):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task.
It will cause 7 days delay for the delivery of this weekly plan.

Unit tests almost work, locally. We need to fix Rust build scripts, and native image builds, and then check that everything works. Let's hope that one more week is sufficient.

Delay Cause: Let's delay the task for one more week, as discussed

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-30):

Progress: - Fixing native-image build - struggling with sbt task definitions and their dependencies

• Striving to provide the easiest and quickest possible solution
• Adding stuff on module-path when building native image does not work again, let's put everything on class-path
  • Similar to how we do that for unit tests. It should be finished by 2023-11-03.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-10-31):

Progress: - Native image build fixes

• Fixing license review - ublock CI - a lot of manual work, writing some script? It should be finished by 2023-11-03.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-01):

Progress: - Another PR for private module checking disabling fix - #8140

• There is no need to download any GraalVM runtime - removing this functionality from project-manager, runtime-version-manager, and co.
• Fixing invocation of language server from project-manager.
  • We will need to start the language server subprocess in a diametrally different way.
  • Previously, it was enough to just run something like java -jar runner.jar <args>. It should be finished by 2023-11-03.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-02):

Progress: - Fixing language server launch from project-manager, dealing with various class loading issues.

• I was able to finally start the language server after these semi-manual modifications:
  • Add a lot of --add-exports options
  • Copy reference.conf from old runner.jar to the new runtime.jar.
  • Move package akka.actor.typed from new runner.jar to the new runtime.jar.
  • Remove contants of JAR archives that intersect runner.jar and runtime.jar from runner.jar - make runner.jar as thin as possible. It should be finished by 2023-11-03.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-03):

Progress: - language server can be launched from project-manager

• It boots and connects to a client
• In 15 seconds generates 5 GB of logs - the configuration is in runner.jar and not in runtime.jar.
• Now fixing the configuration - runtime module sees different configuration, e.g., for logging, than runner.jar.
  • Playing with specifying class loaders for the logging service It should be finished by 2023-11-03.

[JaroslavTulach]

* In 15 seconds generates 5 GB of logs - the configuration is in `runner.jar` and not in `runtime.jar`.
* Now fixing the configuration - runtime module sees different configuration, e.g., for logging, than `runner.jar`.

Pavel, @hubertp shall know what resources need to be read to configure SLF4J in the runtime.jar properly. The IsolatedClassLoader shall:

• share the org.sfl4j classes between module path and runner.jar
• load each SFL4J API class only once into the JVM
• there should be only one implementation of appenders - it should be in runner.jar
• Make sure the configuration resources from runtime.jar are visible to runner.jar SLF4J implementation
• e.g. override getResource and getResources in IsolatedClassLoader
• to delegate to system class loader to read necessary SLF4J configuration resources

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2023-11-06):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task.
It will cause 7 days delay for the delivery of this weekly plan.

Fixing language-server invocation from project-manager was more difficult than expected.

Delay Cause: The agreed upon delay for additional week.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-06):

Progress: - Still fixing log configuration

• When running engine distribution from the cmd line, the log configuration is applied correctly from runner.jar.
• When launching language-server from project-manager, file appender configuration seems ok, but console appender configuration is set to default, which causes generation of GBs of log messages in first few seconds. It should be finished by 2023-11-10.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-07):

Progress: - It seems that the log configuration on my branch was correct for some time, the weird behavior (generation of 5GB of output) can be reproduced on develop.

• Update license review tool reports.
• Fixing native image building once more. It should be finished by 2023-11-10.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2023-11-07):

Progress: - Fixed native image of engine-runner.

• Fixing other tests in project-manager and language-server. It should be finished by 2023-11-10.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-09):

Progress: - Fixed absolute paths to LanguageHome in some language-server tests.

• Took a lot of time to realize what is going wrong in these async tests.
• Fixing tests in project-manager, tests in language-server succeeds locally. It should be finished by 2023-11-10.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-10):

Progress: - Fixed all project-manager tests.

• It took some time, project-manager used some hacks to install engine and create and open projects in it.
• Fixed build of launcher native image
• Fixed tests in bench-processor.
• Fixed runtime-version-manager tests.
• Create follow-up issue Remove GraalVM distribution downloading facility #8280 It should be finished by 2023-11-10.

[enso-bot]

Pavel Marek reports a new 🔴 DELAY for today (2023-11-13):

Summary: There is 7 days delay in implementation of the Upgrade Enso to GraalVM for JDK 21 (#7851) task.
It will cause 7 days delay for the delivery of this weekly plan.

I hope this is the very last delay report for this issue, let's merge it ASAP.

Delay Cause: Tests in project-manager and co. were more difficult to fix than expected, they contained hacks that allowed them to launch the whole language-server. A week delay is a pessimistic estimation, if there are no blockers, it should be merged sooner.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-13):

Progress: - Fixing test in runtime-version-manager-test, project-manager, etc.

• Trying to merge it ASAP. It should be finished by 2023-11-17.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-14):

Progress: - Fixed bug in sbt-assembly - now we can use the newest sbt-assembly plugin version and no IOException will be thrown when assembling.

• All unit tests are passing, now Enso tests are failing
• Discussing Jaroslav's serialization framework
• Fixing java.sql.Driver service loading - java.sql.DriverManager is a bit dump and uses platform class loader to load drivers, so we need to do service loading ourselves. It should be finished by 2023-11-17.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-15):

Progress: - Fixed bug in Regex - graalvm.regex changed its regex polyglot object representation (groups).

• Added some missing modules to the component directory - fixes more Enso tests.
• Added chromeinspector tool, does not work though - freezes after chrome devtools connects.
  • Won't solve in this PR.
• Meetings It should be finished by 2023-11-17.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-16):

Progress: - Fixing std-benchmarks annotation processor.

• Update parent class loader for our org.enso.runtime.HostClassLoader.
• Will have to fix the benchmark runner such that it runs with truffle-compiler on module-path as well.
• Created follow-up issues after the PR is merged (in the issue description). It should be finished by 2023-11-17.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2023-11-17):

Progress: - Fixed rest of the problems, should be able to merge in the evening.

• Working on enabling JVM assertions on the CI in the meantime. It should be finished by 2023-11-17.