Publish Gradle Module Metadata with Variants

[jjohannes]

Hello again from Gradle 👋

The following is a suggestion open for discussion. I provide this as a PR, instead of an issue only, as a proof of concept to aid the discussion.

Background

With the release of Gradle 6, Gradle fully supports a new metadata format called Gradle Module Metadata (GMM) - docs and spec. Such a metadata (.module) file can be published in addition to a .pom file and Gradle 6 will use it to obtain additional information about a library, its artifacts and transitive dependencies which can not be represented in the pom format. The GMM format has been under development at Gradle since about two years. With the release of Gradle 6, we are now at a point where it is ready to be adopted in larger scale. That's why we have been looking at popular JVM libraries, like Guava, to evaluate if Gradle users depending on a library would profit if the library would publish GMM.

Why should Guava care?

There are a couple of things Guava struggles with to express in their poms. Many of which have already been discussed at length here in other issues. I set up a repository with samples demonstrating these use cases. In detail, GMM offers modeling concepts, and with that solutions, for the following:

Multiple variants in one component

Use case with description: guava-jdk-variants

At the core of Gradle 6 dependency management and GMM is the notion that a component can have multiple, and arbitrary many, variants. A concept Gradle has been adopting since 3.x internally and which is heavily used already by all Gradle JVM plugins to distinguish between runtime and compile time variants and to a larger extend, for example, by the Android build system or by Kotlin Native.

Guava publishes two prominent variants - android and jre. It is solved in probably the most elegant way possible with POM by encoding the variant name in the version number. Still this often causes issues as built tools do not know that the versions also present variants.

Each variant in GMM is identified by attributes. These attributes are used by Gradle to select the best fitting variant. Gradle defines a number of default attributes for JVM libraries. One of them is org.gradle.jvm.version. In the presented solution, we set the attribute to 6 for the android variants and to 8 for the jre variants. Gradle, knowing about the attribute, can now use that information. If you do in your build:

java.targetCompatibility = JavaVersion.VERSION_1_6

Gradle will automatically select android, if you set it to version 8 or higher, Gradle will select jre. If you set it to a lower version, Gradle will fail.
It does not matter if you depend on Guava directly or if it is brought in transitively. The central idea behind the variant-aware mechanism is that dependencies are defined between modules, but the variant selection is done when the dependencies are resolved based on the current context. For example, Gradle can say "give me everything I need to compile this library for Java 6" or "give me everything I need to run this application with Java 9".

Note: The presented solution keeps the way how Guava is publishing two "versions" at once. Only for Gradle now both versions are identical (both provide all variants). So in Gradle it would not matter if com.google.guava:guava:29.0-jre or com.google.guava:guava:29.0-android is in your dependency graph, as both provide the same variants. For Maven users, nothing changes as the poms are no different.

Compile-only API dependencies / annotation processor libraries on runtime classpath

Use case with description: guava-compile-only-dependencies

This addresses what is being discussed in #2824. With the solution presented here, you would get the following classpathes:

=== COMPILE ===
guava-28.1-jre.jar
failureaccess-1.0.1.jar
jsr305-3.0.2.jar
checker-qual-2.8.1.jar
error_prone_annotations-2.3.3.jar
j2objc-annotations-1.3.jar
=== RUNTIME ===
guava-28.1-jre.jar
failureaccess-1.0.1.jar
error_prone_annotations-2.3.3.jar

It can be discussed if that is what is desired, but the essential bit is that you can have things on the compile classpath but not on the runtime classpath (not possible with poms currently).

A component/variant can provide multiple capabilities

Use case 1: guava-listenablefuture-conflict
Use case 2: guava-vs-google-collections

Another concept introduced with variants is capabilities. A capability is essentially an identifier for a "feature" that is implemented by a component. This allows you to express that:

• One component provides multiple features that you can individually use/depend on (in this case usually different variants of the component have different capabilities)
• Two or more components provide the same capability and can not be selected together (this is the interesting case for Guava).

To be more concrete, each component has a default capability corresponding to its GAV coordinates:

• com.google.guava:guava has the com.google.guava:guava capability
• com.google.collections:google-collections has the com.google.collections:google-collections capability
• com.google.guava:listenablefuture has the com.google.guava:listenablefuture capability

Now we can say that guava also has the com.google.collections:google-collections and com.google.guava:listenablefuture capabilities. Gradle will then not allow Guava together with these modules on the classpath and offers simple options to resolve the conflict. The 9999.0-empty-to-avoid-conflict-with-guava workaround is then no longer needed to avoid undesired old versions of com.google.guava:listenablefuture on the classpath.
Note: since the pom is not changed, the workaround will stay in place for Maven.

How does publishing GMM work with Maven builds?

Usually, we would recommend using Gradle 6 as your build system if your library wants to publish individual variants or other GMM specifics, as it is not easy to manually set this up in a Maven build. However, the Guava build is already setup to do it - by combining two Maven builds. So everything is already in place, the only thing is that certain metadata information cannot be expressed/published in the poms.
As you can see in this PR, I added the .module file to the repo. (In Gradle, these files are generated during publishing as are the poms.) Since it mostly contains the additional information and Guava does not have many dependencies (and also does not want to add more), the file is of reasonable size to be maintained manually along the poms. Since the naming of the module file is following Maven conventions, it can be treated and published as an additional artifact by Maven - as you can see in this PR.
If you consider adopting this solution, I am happy to provide some kind of integration test (maybe based on the samples linked above) that validates that the module file is valid and that the information does not contradict the information in the pom files.

Further details

Sorry for this rather long text here. My goal is to illustrating the use cases and provide enough detail for a basic understanding of why we think many Gradle users would profit from having Guava publish Gradle Module Metadata. If you are interested in more details, here are a few links to sections in the Gradle user manual:

• Gradle Module Metadata
• Variant Selection
• Variant Attributes
• Capabilities
• Enriching Guava's current metadata with a rule

We also recently did a Webinar on publishing that covers GMM:
https://gradle.com/blog/dependency-management-with-gradle-part-3-publishing-and-release-strategies/ (you can directly jump to 'Section 5 - Metadata' in the video)

And others have started to publish module files to central, so that is working :)
https://repo1.maven.org/maven2/org/junit/jupiter/junit-jupiter-api/5.6.0-M1/junit-jupiter-api-5.6.0-M1.module

I am happy to answer any questions on the topic and help working out a solution if there is interest in this on your side.

[kevinb9n]

Thanks very much for this.

[jjohannes]

We have published a blog post on the topic of how libraries can profit from Gradle Module Metadata and used Guava as an example: https://blog.gradle.org/guava

It basically covers what I have written in the PR description in better words. :)

While working on the post, I also wrote a Gradle plugin that adds the additional metadata to the existing Guava versions. So whoever is interested in trying this out today can use the plugin in a Gradle build: https://plugins.gradle.org/plugin/de.jjohannes.missing-metadata-guava

(⚠️ If you publish a library, you should not use the plugin as all your consumers would have to apply it as well. That's why it would be great if we get this PR in for future Guava releases, because then the plugin is no longer needed)

@cgdecker As I got back to this topic, I thought about how we could add test coverage to this PR. I added an "integration test" that resolves the locally installed Guava SNAPSHOT version with Gradle and tests all different variations of classpathes. It tests against Gradle 5 (using POM metadata) and Gradle 6 (using GMM). This way, we make sure that both metadata files are in sync and that we only have the expected differences in the resolution result.
Since you don't have this kind tests (as far as I know), I added a script to run it: util/gradle_integration_tests.sh. It runs outside the Maven builds, as it requires both variants (JRE and Android) to be built first. Independent of the changes in this PR, such a test makes it easier for you to see the impact changes in dependencies have on Gradle users in general. This script can maybe be called as an additional step in the CI setup. Not sure if this is the best way to go. I am happy to rework this.

[cpovirk]

Thanks again for this, and sorry that we have let it sit so long. I have a couple small questions, but we should probably err on the side of merging this soon, and we can always make changes later.

[cpovirk]

It sounds like we may need to update this manually (or tweak set_version to update it with sed or something) to test against a version that we're about to release? (And similarly at the top of the file.)

That should be fine. I just want to check my understanding (and invite you to let me know if there are other things we should do that I've overlooked).

[jjohannes]

You are right. I missed this.

I fixed it by extracting the version directly from the pom.xml file.

[cpovirk]

Similarly: Do we need to specify the version number here? I see that it's omitted for checker-qual above.

[jjohannes]

The reason why the version is required here is that the version of checker-compat-qual is not managed in the <dependencyManagement> block below. It's only defined directly in android/pom.xml.

I think the better solution is to add it to the block as well. I made that change: 70eaaf7.
Let me know if that is fine.

Note: The version is then defined in two places, here and in android/pom.xml. I am not sure why there are versions in android/pom.xml at all. The android pom.xml also uses this pom as parent. Maybe all the versions can be removed from android/pom.xml?

[jjohannes]

Thanks for taking the time to get back to this @cpovirk.

I rebased this PR to latest master and addressed both of your comments. Please have a look.

From my perspective this can be merged as it is. The result of Gradle resolving a dependency to Guava is now pinned down in the integration test. That also tests the behaviour when resolving without Gradle Metadata. Which is a good addition to Guava's test coverage in general to make sure that changes to dependencies in pom files do not have unexpected side effects.

What remains to be done is to (automatically) execute the integration test by calling util/gradle_integration_tests.sh. I am not sure where to put this best. (Should it become part of the travis configuration?) This can be done as a follow up.

[cpovirk]

Thank you again, this time for picking this up so quickly after our silence.

(Rewinding: I am going to ponder your earlier question about whether to move our annotation dependencies to compile-only scope. You can leave things as you have them, and I will tweak it if we want to be more conservative.)

What remains to be done is to (automatically) execute the integration test by calling util/gradle_integration_tests.sh. I am not sure where to put this best. (Should it become part of the travis configuration?) This can be done as a follow up.

I would assume Travis as well. A follow-up sounds perfectly fine.

[cpovirk]

In your most recent round of comments, you said:

I am not sure why there are versions in android/pom.xml at all. The android pom.xml also uses this pom as parent. Maybe all the versions can be removed from android/pom.xml?

I don't think that's the case: I see no <parent> element in android/pom.xml.

There is certainly a decent argument that we should have a common parent for both flavors; that's just not how we have it set up at present.

Regardless, I like the idea of having the version of checker-compat-qual defined in both flavors' pom.xml files, since that lets the Gradle files refer to it consistently.

But: Given that the Android flavor has its own parent POM, should the line here refer to HEAD-android-SNAPSHOT?

[jjohannes]

I don't think that's the case: I see no element in android/pom.xml.

Uh, my bad. Yes the POM does not have the parent. And while it could be possible it does make sense that there is no common parent as each flavor is a separate maven build.

I don't know why I did this, but this parentVersion property is actually not needed at all and the version of the parent simply has to match the version of the variant. Then the checker-compat-qual version is also there.

So forget most of what I said earlier.

I fixed this. Thanks for clarifying and asking the right questions.

[cpovirk]

Ah, thanks, I was still not seeing the implications of this :) All the more reason for the Gradle metadata to simplify things for users!

[jjohannes]

I went ahead and added the integration test as a step to travis to make sure it is working before we merge. Looks good as far a I can tell. At least it did run on this PR and passed.

Please feel free to merge. And do further changes later if required (I am happy to help, just ping me).

As a note regarding

(Rewinding: I am going to ponder your earlier question about whether to move our annotation dependencies to compile-only scope. You can leave things as you have them, and I will tweak it if we want to be more conservative.)

You are most likely aware: this is this long-standing feature request in Maven that was never picked up.

[cpovirk]

Sorry, I have one more question, which I should have asked long ago. You said in your original message that Gradle will automatically choose guava-android if you set:

java.targetCompatibility = JavaVersion.VERSION_1_6

But... do Android users do this? It sounds like targetCompatibility corresponds to the javac -target argument, and I think that Android compiles normally set that to 8 (or maybe higher now?) so that they can use lambdas, etc. (which get desugared to work even without support in the Android runtime). But we still want those compiles to get the -android flavor of Guava. (That's both because most Android users don't want to assume that they can use Java 8 APIs and because our -android flavor contains some slightly different implementations that are optimized for Android considerations.)

Would we need to do something different in order to detect Android compiles (presumably by detecting something set by the Android Gradle Plugin)? [edit: I'm not clear on whether the following links are asking about something similar: https://discuss.gradle.org/t/how-to-detect-if-a-plugin-was-not-applied/28995, https://stackoverflow.com/q/46204642/28465]

[cpovirk]

(Maybe Android Gradle plugin should declare an attribute? I could try talking to that team if you think it would be helpful. Historically, we haven't been in touch with them a lot, but I'd like to think they'd be interested in this.)