Multi-Release JAR that contains Java 8 binaries + Java 9 module-info.class

[danieldietrich]

I need a helping hand regarding our Gradle build for Vavr 1.0.0 (see v1.0.0 branch).

We have

• The build uses JDK9
• We develop in Java 9 (in order to have modules) but we only use Java 8 API

Additionally we need

The build.gradle needs to be altered in the way that

• for each module we get one jar ./vavr-<module>/build/lib/vavr-<module>-<version>.jar
• the contents of the jar are:

META-INF
META-INF/MANIFEST.MF
META-INF/versions/
META-INF/versions/9/
META-INF/versions/9/module-info.class
io/
io/vavr/
io/vavr/<module>/
io/vavr/<module>/<classes>

• all files are compiled with JDK9 -release 8 compiler arg
• only module-info.java is compiled with JDK9 -release 9 compiler arg

Resources

• JEP 238: Multi-Release JAR Files
• Multi-release JARs - Good or bad idea?
• Building a Multi-Release JAR with Gradle
• Building Java 9 Modules
• Multi-Release Example: MinecraftForge ForgeFlower build

[melix]

Hi @danieldietrich,

In this case I assume that what you want to produce is a real module (under Java 9+), and hope that your jar will not break clients using Java 8 (saying this because libraries that scan jars will fail on your module-info file if they think it's a class and that's one of the reasons mrjars aren't cool). If all you want to do is reserve a module name for the future, then the Automatic-Module-Name entry in the manifest file should be enough.

So, if you want to have 2 sets of things that end up in a jar, so really, really want to produce a mrjar, it's not really different from, for example, having generated sources, compiling them and packaging in the jar eventually. Here it will involve:

1. creating a source set for your Java 9 files (just module-info.java, here)
2. configure the compilation for this source set
3. bundling the result in the jar

So, first step, create a java9 source set:

sourceSets {
   java9 {
      java {
       srcDirs = ['src/main/java9']
      }
   }
}

Next, configure the Java compile tasks:

compileJava {
   sourceCompatibility = 8
   targetCompatibility = 8
}

compileJava9Java {
   sourceCompatibility = 9
   targetCompatibility = 9
}

Then, package everything into the jar:

jar {
  into('META-INF/versions/9') {
     from sourceSets.java9.output
  }

  manifest.attributes(
     'Multi-Release': 'true'
  )
}

And that's all! The following build scan shows that when running the jar task, Gradle will automatically trigger the compilation of both source sets, and include the result where you want.

[melix]

BTW if you want to do this on all modules of your project, I'd suggest you make this a buildSrc plugin, and apply it to all your subprojects.

[danieldietrich]

Hi @melix,

thank you for the details, I can reproduce it.

But I still wasn't able to glue your hints with my Java 9 modules build (see below). Basically, the module-info.java can't be compiled because the dependent modules and exported packages cannot be found.

• Is the java9 sourceSet aware of the java sourceSet?
• Also I'm not sure if I should use the '--module-path' trick anymore
• I'm not familiar with buildSrc plugins or how they might help. Will try to find resources on the web.

Root Project 'vavr'

settings.gradle:

rootProject.name = "vavr"

include 'vavr-core'
include 'vavr-control'

build.gradle

if (!JavaVersion.current().java9Compatible) {
    throw new GradleException("Please build Vavr with JDK 9+")
}

subprojects {
    afterEvaluate {
        repositories {
            jcenter()
        }

        group = 'io.vavr'
        version = '1.0.0'
        
        sourceSets {
            java9 {
                java {
                    srcDirs = ['src/main/java9']
                }
            }
        }
        
        compileJava {
            sourceCompatibility = 8
            targetCompatibility = 8
            options.encoding = 'UTF-8'
        }

        compileJava9Java {
            sourceCompatibility = 9
            targetCompatibility = 9
            options.encoding = 'UTF-8'
            inputs.property("moduleName", moduleName)
            doFirst {
                options.compilerArgs = [ '--module-path', classpath.asPath ]
                classpath = files()
            }
        }
        
        jar {
            into('META-INF/versions/9') {
                from sourceSets.java9.output
            }
            manifest.attributes(
                    'Multi-Release': 'true',
            )
        }
    }
}

Subproject 'vavr-core'

build.gradle

plugins {
    id 'java-library'
}
ext.moduleName = 'io.vavr.core'

src/main/java9/module-info.java

module io.vavr.core {
    exports io.vavr.core;
}

(src/main/java omitted here)

Subproject 'vavr-control'

build.gradle

plugins {
    id 'java-library'
}
ext.moduleName = 'io.vavr.control'

src/main/java9/module-info.java

module io.vavr.control {
    exports io.vavr.control;
}

(src/main/java omitted here)

[danieldietrich]

@melix I think I'm nearly there.

The main problem is, that the src/main/java9/module-info.java files are not aware of the src/main/java/**/*.java sources anymore.

> Task :vavr-control:compileJava9Java FAILED
/Users/daniel/git/vavr-io/vavr/vavr-control/src/main/java9/module-info.java:3: error: module not found: io.vavr.core
    requires transitive io.vavr.core;
                               ^
1 error

[danieldietrich]

Maybe the classpath system and the module system are now mixed-up somehow... 🤔

[danieldietrich]

@Opalo I saw on StackOverflow that you are familiar with Gradle sourceSet magic.

Could you please take a look at the v1.0.0 branch?

git clone https://github.com/vavr-io/vavr.git
cd vavr
git checkout v1.0.0
./gradlew assemble

The build needs to be run with JDK9.

[danieldietrich]

@Opalo @melix sorry for flooding you with messages - no further actions are needed.

I moved a step back and implemented a simple Java 8 multi-project build.

That's the best solution for now, it will work with Java 8 and Java 9 and we do not risk any side-effects by piggy-backing the module-info.class in the jar.

subprojects {
    apply plugin: 'java'

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }

    afterEvaluate {
        jar {
            inputs.property('moduleName', moduleName)
            manifest.attributes(
                'Automatic-Module-Name': moduleName
            )
        }
    }
}

[melix]

Out of curiosity, why did you put the jar configuration in an afterEvaluate?

[danieldietrich]

@melix if I omit afterEvaluate I get the following error because the subprojects define an extra property ext.moduleName that is used by the parent project:

$ ./gradlew assemble
Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/daniel/git/vavr-io/vavr/build.gradle' line: 19

* What went wrong:
A problem occurred evaluating root project 'vavr'.
> Could not get unknown property 'moduleName' for task ':vavr-control:jar' of type org.gradle.api.tasks.bundling.Jar.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 10s

---

build.gradle (failing):

subprojects {
    apply plugin: 'java'

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }

    jar {
        inputs.property('moduleName', moduleName)
        manifest.attributes(
            'Automatic-Module-Name': moduleName
        )
    }
}

vavr-control/build.gradle:

dependencies {
    compile project(':vavr-core')
}
ext.moduleName = 'io.vavr.control'

vavr-core/build.gradle:

ext.moduleName = 'io.vavr.core'

[melix]

I see, it's an ordering problem. You have to define the moduleName before you first use it. One way to do this is to create a plugin that will define it, so you'd do:

apply plugin: 'java'
apply plugin: 'modularity' // yours

[danieldietrich]

@melix I haven't written a Gradle plugin, yet. Are there any good resources on that topic? Do I need buildSrc for that purpose?

This does not work:

// parent build.gradle
class Module {
    String name
}

class modularity implements Plugin<Project> {
    void apply(Project project) {
        def extension = project.extensions.create('module', Module)
    }
}

subprojects {
    apply plugin: 'java'
    apply plugin: 'modularity'

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }

    jar {
        println "Creating module $module.name"
        manifest.attributes(
                'Automatic-Module-Name': module.name
        )
    }
}

// subproject build.gradle
module {
    name = 'io.vavr.core'
}

Error: Plugin with id 'modularity' not found.

[danieldietrich]

@melix Update: it does work. I needed to remove the quotes apply plugin: modularity

Thx for the hint!

[melix]

Yes, using buildSrc is a good practice. See https://docs.gradle.org/current/userguide/custom_plugins.html for more ideas.

[danieldietrich]

@melix Actually it did not work because the original Gradle module name was taken (I think module was a reserved word`).

When I renamed it to module2, it was not initialized, e.g. module2.name was null. Maybe also an afterEvaluate is needed, even when using a plugin?

---

I ended up with a more straight-forward solution without extra plugins - I just renamed the Gradle modules of the multi module project.

build.gradle:

subprojects {
    apply plugin: 'java'

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }
    
    jar {
        manifest.attributes(
            'Automatic-Module-Name': module.name
        )
    }
}

The subprojects only define dependencies, no more extra module names are declared.

[danieldietrich]

@melix oh, that's not possible because the maven coordinates changed, too 🙈

[danieldietrich]

@melix Sorry for the flood of notifications. I run into more problems.

• jar has to depend on jarModule
• the new task jarModule needs to contain a doLast (is that correct? without it, it does not work)
• I'm not able to add manifest attributes to the outer jar task from within the new plugin

class JavaModule {
    String name
}

class modularity implements Plugin<Project> {
    void apply(Project project) {
        def extension = project.extensions.create('javaModule', JavaModule)
        project.task('jarModule') {
            doLast {
                println "Automatic-Module-Name: $extension.name"
                // FAILS
                jar {
                    manifest.attributes(
                        'Automatic-Module-Name': extension.name
                    )
                }
            }
        }
    }
}

subprojects {
    apply plugin: 'java'
    apply plugin: modularity

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }

    jar {
        dependsOn 'jarModule'
    }
}

Subproject:

javaModule { name = 'io.vavr.core' }

Error:

$ ./gradlew clean assemble

Execution failed for task ':vavr-core:jarModule'.
> Could not find method jar() for arguments [modularity$_apply_closure1$_closure2$_closure3@410542b4] on task ':vavr-core:jarModule' of type org.gradle.api.DefaultTask.

[Opalo]

@danieldietrich here you have a simple workaround:

class JavaModule {
    String name
}

class modularity implements Plugin<Project> {
    void apply(Project project) {
        def extension = project.extensions.create('javaModule', JavaModule)
    }
}

subprojects { s ->

    apply plugin: 'java'
    apply plugin: modularity

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    javaModule {
      name = s.name
    }

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }
    jar {
        manifest.attributes(
            'Automatic-Module-Name': javaModule.name
        )
    }
}

What you want to do is quite tricky since you have configured a cycle. Just replace root build.gradle with this piece of code and remove build.gradle files from submodules. I have very little time right now - if this suggestion does not satisfy you - may try to help late in the evening (c.a. 2200 CET).

[danieldietrich]

@Opalo thanks, let's come back this evening.

(The subproject name isn't the java module name. Currently we have 'vavr-core' and 'vavr-control' but we need 'io.vavr.core' and 'io.vavr.control'. However, the artifact names need to remain 'vavr-core' and ' vavr-control'.)

[Opalo]

So add this to settings.gradle:

def conf = ['io.vavr.core' : 'vavr-core']
conf.each { k, v ->
  include k
  project(":$k").name = v
}

Of course map (conf) ^ should have entries for both modules. Time for climbing, keep me posted.

[danieldietrich]

Thanks, have fun!

[danieldietrich]

@Opalo I reverted the Plugin-related changes and went back to the simplest solution. We gained nothing by adding additional lines of codes and workarounds - the result is the same with the following:

subprojects {

    apply plugin: 'java'

    repositories {
        jcenter()
    }

    group = 'io.vavr'
    version = '1.0.0'

    compileJava {
        sourceCompatibility = 8
        targetCompatibility = 8
        options.encoding = 'UTF-8'
        options.compilerArgs = [ '-Xlint:all', '-Werror' ]
    }

    afterEvaluate {
        jar {
            inputs.property('moduleName', moduleName)
            manifest.attributes(
                'Automatic-Module-Name': module.name
            )
        }
    }
}

(+ ext.moduleName definitions in the subproject's build.gradle files).

Simplicity wins.

[Opalo]

Yes, definitely. This is a very good piece of gradle configuration :)

[tlinkowski]

What

If you're still interested in providing Java 9 module-info.class files with your Java 8 binaries, Gradle Modules Plugin v1.5.0 supports it now.

Note that it won't produce a Multi-Release JAR, but it doesn't seem necessary to me (rationale here).

Why

I really hope you're still interested in it, because if popular libraries like yours don't adopt JPMS, we all (as a community) won't be able to benefit from it.

There's a recent post by Nicolas Fränkel about how hardly any popular library is modularized yet.

On the other hand, one of these libraries (JUnit 5) is again working on support for JPMS (5 drafts, 2 working): junit-team/junit5#1091 👍

So I hope you'll at least consider it...

How

You just need to:

1. place your module-info.java files in src/main/java of every subproject (AFAIK, you already have them),
2. insert the following snippet in your main build.gradle:

plugins {
  // your current plugins here

  id 'org.javamodularity.moduleplugin' version '1.5.0' apply false
}

subprojects {
  apply plugin: 'org.javamodularity.moduleplugin'
  modularity.mixedJavaRelease 8 // sets "--release 8" for main code, and "--release 9" for "module-info.java"

  // test.moduleOptions.runOnClasspath = true // optional (if you want your tests to still run on classpath)

  // your current subproject configuration here
}

No need for custom source sets, etc.

PS. Note that you can't set sourceCompatiblity nor targetCompatibility if you use modularity.mixedJavaRelease, or Gradle Modules Plugin will throw an error.

[danieldietrich]

@tlinkowski as long as we support Java 8, we will stay with automatic-module-name only. Sorry, but I think it is not worth the effort for the moment.

[jbduncan]

@danieldietrich I could be wrong about this, but from reading @tlinkowski's comment and from what I've observed of the JUnit 5 folks' work towards modularity, I was under the impression that there is a way of including module-info.java files without needing to sacrifice Java 8 support... 🤔

[tlinkowski]

@tlinkowski as long as we support Java 8, we will stay with automatic-module-name only. Sorry, but I think it is not worth the effort for the moment.

I see. And if I found the time to provide a PR for this, would you be willing to accept it (provided its quality were OK)?

Or are you generally opposed to the idea of providing a Java 9 module-info.class together with Java 8 binaries? Or, perhaps, are you afraid that depending on org.javamodularity.moduleplugin could be a maintenance burden for you?

@danieldietrich I could be wrong about this, but from reading @tlinkowski's comment and from what I've observed of the JUnit 5 folks' work towards modularity, I was under the impression that there is a way of including module-info.java files without needing to sacrifice Java 8 support... 🤔

Yes, precisely.

[danieldietrich]

I do not want to mix up JDK8 binaries/classes with a JDK9 module-info.class in one .jar file. Furthermore, we will not create multi-release .jars in order to separate JDK8 and JDK9 builds. Expect the unexpected - it will lead to problems. (I have been there with class names containing spec-conform unicode characters but tools/cloud platforms errored when processing them).

I don‘t see any problem you try to solve here. But I see problems that might be introduced when applying this change.

VAVR is both, classpath and module-path compatible. That should be enough for now.

I will not accept a pull request, the risk is too high to break Java 8 environments. I care about Java 8 because, as you said and as our users reflected, most Java users are still on Java 8.

The existing tooling is a smell. It looks like a workaround. The Java language architects should work on a proper solution. Developers should not have the option to choose. There should be exactly one simple way to create .jars for all.

[tlinkowski]

I do not want to mix up JDK8 binaries/classes with a JDK9 module-info.class in one .jar file. Furthermore, we will not create multi-release .jars in order to separate JDK8 and JDK9 builds. Expect the unexpected - it will lead to problems. (I have been there with class names containing spec-conform unicode characters but tools/cloud platforms errored when processing them).

I see. Well, I trust your judgment — my experience is scarce here.

I don‘t see any problem you try to solve here. But I see problems that might be introduced when applying this change.

I understand — you see negative gain/cost ratio here. For the record, though, the problem I'm trying to solve is: inappropriately weak encapsulation prevailing in the Java ecosystem.

It's not that you can't write software without stronger encapsulation (surely, people do, so you can — we had so many years without JPMS to get used to it). It's just that, with stronger encapsulation, you can write software better and write better software, I believe.

Analogy (not very strong, though):

• a shift from C (no encapsulation) to C++ (class-based encapsulation);
• here, we have a shift from pre-JPMS Java (class-based encapsulation) to JPMS Java (module-based encapsulation).

Stretching this analogy even further:

• It's a little bit like convincing Java developers to ditch the mutable collections API (which they know very well) for persistent collections API 😉 The gains aren't so obvious at first sight.

But like I said, I understand & respect your stance!

I just disagree that JPMS wouldn't help solve any problems for VAVR (both for its maintainers and for its users) in the long term.

I will not accept a pull request, the risk is too high to break Java 8 environments. I care about Java 8 because, as you said and as our users reflected, most Java users are still on Java 8.

Understood.

The existing tooling is a smell. It looks like a workaround. The Java language architects should work on a proper solution. Developers should not have the option to choose. There should be exactly one simple way to create .jars for all.

I agree with every sentence!

Still, I think that what JPMS brings is worth it, even despite all these hurdles. But it's just my opinion now (subject to change with increasing experience).

[danieldietrich]

I really appreciate your engagement for bringing Java forward.

For VAVR itself and for consumers of VAVR, encapsulation/modularization is not a big deal. We decided to ship VAVR as one standalone jar, because it is small enough. Slicing it into modules that contain only two or three public classes would be too much.

• VAVR has no dependencies
• VAVR has no cross-package utility classes
• VAVR‘s public classes are all intended to be used by consumers

You see, currently I have no specific use-case for the module-info DSL.

[bowbahdoe]

@danieldietrich I would really want this to be able to use vavr in a context where I want users of my library to be able to use jlink.

I know this is a niche use case and I'm probably the first to ask for that reason but...yeah.