Add module picocli-jpms (was: Include module-info.class for full JPMS modularization and jlink support)

[remkop]

Currently the picocli jar is an automatic module (manifest has Automatic-Module-Name: info.picocli).

A JLink trimmed binary image needs explicit modules. This means the jar needs to include a module-info.class.

Update: this is a common misunderstanding. See the comment below for a tutorial for creating a JLink binary image with an automatic module.

module-info.class trade-offs

It is possible to compile only the module-info.java file with Java 9 and include it in the picocli jar where all other classes are compiled with Java 5.

However, there are quite a few tools that scan all classes in a Jar and that would choke on the module-info.class (whether it is in the root of the jar or in META-INF/versions/9/). For example: jandex, and Apache Commons BCEL.

Consider producing a picocli-jpms-3.x.x.jar that has the same contents as picocli-3.x.x.jar but in addition a module-info.class. Users looking to create a JLink custom runtime image can use the picocli-jpms jar.

Solution

The comment below is a brief tutorial for creating a JLink binary image with an automatic module. No module-info.class required in picocli or in the application.

Todo: do a more formal write-up and add it to the picocli documentation.

[remkop]

It turns out that an application does not need to be a full-fledged JPMS module with a module-info.java file to generate a custom jlink runtime image for your application.

Example application:

package hello;

import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;

@Command(name = "hello", description = "picocli demo", version = "3.6.1", mixinStandardHelpOptions = true)
public class HelloWorld implements Runnable{

    @Option(names = {"-u", "--user"}, description = "Specify a user. Default is ${DEFAULT-VALUE}.")
    String user = "unknown";

    @Override
    public void run() {
        System.out.printf("Hello, %s", user);
    }

    public static void main(String[] args) {
        CommandLine.run(new HelloWorld(), args);
    }
}

Example build.gradle:

group 'experiments'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
    jcenter()
}

dependencies {
    compile 'info.picocli:picocli:3.6.1'
    testCompile 'junit:junit:4.12'
}

jar {
    manifest {
        attributes 'Main-Class': "hello.HelloWorld",
                'Class-Path': 'lib/picocli-3.6.1.jar'
    }
}

Build the application:

$ gradlew clean build

If you try to run it now as an executable jar it complains because it is still missing picocli on the classpath:

$ java -jar build\libs\experiments-1.0-SNAPSHOT.jar -h
Exception in thread "main" java.lang.NoClassDefFoundError: picocli/CommandLine
        at hello.HelloWorld.main(HelloWorld.java:19)
Caused by: java.lang.ClassNotFoundException: picocli.CommandLine
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 1 more

Copy the picocli jar into build/libs/lib:

$ mkdir build/libs/lib
$ cp path/to/picocli-3.6.1.jar build/libs/lib/.

Now you can run the jar:

$ java -jar build\libs\experiments-1.0-SNAPSHOT.jar
Hello, unknown

Get the list of dependencies from jdeps:

$ jdeps --list-deps build\libs\experiments-1.0-SNAPSHOT.jar
   java.base

Our HelloWorld app only requires java.base. Let's create a runtime image for that module:

$ jlink --no-header-files --no-man-pages --add-modules java.base --output java-runtime

You now have a ./java-runtime subdirectory with a custom JRE runtime image with all that is necessary to run the application:

$ java-runtime\bin\java -jar build\libs\experiments-1.0-SNAPSHOT.jar
Hello, unknown

And we're done!

You can copy the contents of the build/libs directory to java-runtime/lib so that everything is in one place for easier distribution.
That allows you to run the application like this:

$ java-runtime\bin\java -jar java-runtime\lib\experiments-1.0-SNAPSHOT.jar
Hello, unknown

[jennettwheeler]

I'm having this issue too, but I'm using maven...

[remkop]

All that’s really needed is a jar with the HelloWorld class where the manifest has HelloWorld as the main class so that the application can be run with java -jar myjar.jar.

It would be great if you could provide a minimal maven Pom that accomplishments that, so I can include it in the documentation.

[jennettwheeler]

It's just that I can't run jLink while my module-info.java depends on picocli?

Oh wait, you mean that for now:

• Work out "JPMS supported" dependencies
• Work out non-JPMS dependencies
• Create a JRE using just the "JPMS supported" dependencies without the jar file
• Copy the jar & the non-JPMS dependencies (within class path)

[jennettwheeler]

I think can work with that, when I get to it, but I'm definitely looking forward to the "picocli-jpms" module.

I had a go at modularising it, but was having issues with groovy being found (I think it's because I'm using JDK 11). So I'm guessing it will not be an easy task to convert.

[remkop]

I got my inspiration from Simon Ritter: https://medium.com/azulsystems/using-jlink-to-build-java-runtimes-for-non-modular-applications-9568c5e70ef4

I'm considering separating the groovy stuff into a separate module (#479) but that will likely be a picocli 4.0 thing. There are other things I want to work on first. Especially since there is a workaround as described in the above comment and Simon't article.

[jennettwheeler]

Thank you. i look forward to it. For now I am happy with the workaround.

[jennettwheeler]

Besides it's better to use Picocli than not :P haha

[jennettwheeler]

For now, I've added a new module to my project called picocli with ComandLine.java & AutoComplete.java (plus module-info.java) and named the version 3.7.0-JPMS. This is letting my compile and create my JRE!

The downside is, of course, that I have to manually download changes to your source code.

[remkop]

Did following the steps in the comment (or Simon’s article) not work? You shouldn’t need a module-info.class...

[jennettwheeler]

My application is a JPMS app, so it does not work for me. My solution means that I have Linux & windows zips being created by just running mvn package.

I have made use of service loader (I don't know if this is only java 9+?) to essentially enable installing sub commands prior to run-time...

I have a folder called "jobs", in there maven puts my default jobs (sub commands for picocli) - which are excluded from jlink as they aren't depended upon at any point.

I also have a second git repo that proves to me that building the jar and dropping it into the jobs folder immediately enables the sub-command in picocli and lets me run that jar's code.

[remkop]

In picocli 4.0 I’m planning to do the following:

• Add a picocli-core-module artifact that has only the classes in the picocli package. This will be a Java 9 modular jar (not an automatic module). TBD whether to exclude the AutoComplete class and whether to compile all classes with Java 9 or just the JPMS module-related ones (module-info).
• Add a picocli-groovy artifact that has only the Groovy classes. This artifact depends on picocli-core-module.
• Also, continue to provide a picocli artifact with the same content as the current artifact (the picocli and the picocli.groovy package). All classes in this jar are compiled with Java 5, just like now. This is an automatic module just like it is now. The module name for this automatic module will be the same as that of the picocli-core-module artifact.

[jennettwheeler]

Sounds great.

[jennettwheeler]

Is this still planned for 4.0?

[remkop]

Hi, yes, this is part of the plan for 4.0.
(However, there’s a lot on that list: https://github.com/remkop/picocli/milestone/40)

I haven’t really given much thought on how to implement this. It’ll probably require some Gradle magic.

Will you be able to help out with this one?

[jennettwheeler]

I would be more than willing to but I struggled the last time I tried. I have absolutely zero experience with gradle (I use maven) and was struggling to make it work.

Do you have any pointers.

Are you planning on 4.0 being a breaking change (i.e. no longer supporting Java 5?)

[jennettwheeler]

I did have a quick go, but struggled to modify. I'll have a proper sit down soon to work out exactly how it currently works, because only then can I make adequate decisions.

It has reaffirmed my personal preference of maven haha. Each to their own eh?

[Warkdev]

Hi guys,
Please let me know if you need further assistance for this, I'm also looking forward a picocli JPMS-9 compliant library :)

Warkdev

[remkop]

@Warkdev that would be great!
I jotted down some thoughts in this comment. I was thinking to add Gradle tasks to build the extra artifacts, but haven’t really thought through the details.

Any help would be appreciated.

[jennettwheeler]

@Warkdev if you have experience with Gradle, then I would suggest you take a look at it as i have not had a proper look in earnest yet, so don't worry about taking over if that's what you want. I have absolutely no experience with Gradle, so am not the best person tackle this (unless @remkop wants to switch to Maven haha)

[Warkdev]

@jwheeler91 I think we've the same level of knowledge with Gradle. I just recently adapted a Gradle-projet to JPMS recently. I do have a preference for Maven projects :)

@remkop Will have a look, hopefully I won't make any mistake.

[remkop]

Having worked with both Maven and Gradle I much prefer Gradle. As with any technology, there is some learning involved, but that is an opportunity, right? ;-) Don't worry about making mistakes, we'll just fix them. :-)

One idea is to introduce a new Gradle subproject in the project, picocli-core-module (or maybe picocli-jpms-module?). This subproject will have its own build.gradle and its own src/main/java/ directory (or should that be src/main/java/info.picocli/?) with at least one source file: module-info.java:

module info.picocli {
    exports picocli;
}

In that subproject we need to use Java 9 to compile this source file.

We can let this subproject depend on the main project, so that the main parent project's jar is created before the subproject's jar. (See picocli-codegen/build.gradle for how to depend on the root project).

The next step would be a bit unconventional. Usually a subproject's build.gradle only works with resources in its own subproject, but in this case we may want to take the root project's jar file, and work with that. Concretely, we want to unjar it in a temp folder, remove the Groovy classes, add our module-info.class, and then re-jar it into a modular jar.

TBD:

• does the module-info.class live in the root of the jar or under /META-INF/versions/9/? (Looks like the latter is better.)
• do we (should we) create the modular jar with the --module-version option?
• Would the picocli-core-module Gradle build file also be responsible for creating a separate artifact (picocli-groovy? picocli-groovy-jpms-module?) with just the groovy classes, or should we create another subproject with its own Gradle build for that?
• I guess we should remove the 'Automatic-Module-Name' : 'info.picocli' from the jar manifest attributes of the root project, so that the picocli-4.x.x.jar is no longer an automatic module (two modules cannot contain the same package).

[Warkdev]

Great, so did I start the right way.

However, I've a first remark. To manage JPMS, I need to use gradle plug-in which is compiled with Java 11, I hope this is not an issue.

Currently, I'm also fighting with junit and gradle because I do need a second module info for the test module (why do I need to make it a module too? Or there's something really wrong with the way my IDE and gradle handles it).

Still early stages.. I've forked it to my repo and make some changes but no commit yet.

[Warkdev]

I think it will be wiser if we want to have this being released :)

Maybe, with this start, someone else will jump on it and bring all his gradle knowledge !

[remkop]

No problem, let’s do it that way.

[remkop]

Can you submit a PR for the module bits?

[Warkdev]

Done boss!

[remkop]

PR merged. Thanks for the hard work!

Note to self: keeping this ticket open: still need to remove the automatic module name from the manifest of the picocli-4.x.jar artifact.

[remkop]

Following up on the conversation on PR #662:

If anyone wants to help out further, one thing I can think of is that it would be nice to have a README.md for this module.

Also, creating automated tests turned out to be quite challenging, but it would be good to confirm manually that the artifact produced by the build for this subproject can in fact be used in a modular java application (that is, just using jars on the --module-path, nothing on the classpath). Maybe the README could be a mini-tutorial on creating a modular CLI application with this module.

By the way, I am still undecided, but considering to rename this module to picocli-jpms-module. Feedback welcome.

[remkop]

Credit where credit is due: https://twitter.com/picocli/status/1116722078516826112?s=21
:-)

[Warkdev]

Ni need for credit but thanks. You made this awesome lib'!

[remkop]

Note to self: it turns out that the (nice and short) DSL syntax for configuring the plugin has some issue in some environments where I am testing. The legacy plugin application syntax is more verbose but does not have this issue:

buildscript {
    repositories {
        maven {
            url "https://plugins.gradle.org/m2/"
        }
    }
    dependencies {
        classpath "gradle.plugin.org.beryx:badass-jar:1.1.3"
    }
}

plugins {
    id 'java'
    id 'distribution'
    id 'maven-publish'
    id 'com.jfrog.bintray'
    id 'java-library'
    //id 'org.beryx.jar' version '1.1.3'
}

apply plugin: "org.beryx.jar"

[remkop]

Note to self:
Charles Nutter reported he had issues after putting the module-info.class in /META-INF/releases/9/ :
https://twitter.com/headius/status/1037931427000725504

So it may be safer to put it in the root of the jar after all. Since we publish two artifacts, old tools that cannot handle Java 9 bytecodes can always use the non-modular picocli.jar.

I will take a look at this later, before 4.0-GA.

[remkop]

I created #674 to follow up.