Multi-module liberty dev mode fails when there exists a test-jar dependency

[lmsurpre]

I have an existing multi-module web application and I want to use liberty dev mode with it.

One of the modules has some test utilities which are used in the tests of the other modules.
Maven supports that through the use of a build plugin configuration like this:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>test-jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Then, in the modules that depend on it, we declare the dependency with a type of test-jar (or, equivalently, with a classifier of tests)

This builds fine with a command like mvn clean install -f parent. However, if one of those modules has our liberty config and we try using mvn liberty:dev -f parent then we end up with an error like the following:

Could not resolve dependencies for project io.openliberty.guides:module2:war:1.0-SNAPSHOT: Could not find artifact io.openliberty.guides:module1:jar:tests:1.0-SNAPSHOT

Attached is a minimal reproducer that adapts the OpenLiberty getting started example to demonstrate the issue.
multimodule-testjar-reproducer.zip
After unzipping, run the following to see the error:

mvn liberty:dev -f parent

[lmsurpre]

I'll also note that the documented mvn liberty:dev -f parent -pl module2 -am syntax didn't work for me at all.
That one produces an error like this:

[ERROR] [ERROR] Could not find the selected project in the reactor: module2 @ 
[ERROR] Could not find the selected project in the reactor: module2 -> [Help 1]

However, using the "reactor" syntax (prefixing the module name with a : to indicate that module2 is an artifactId), I got back to the same issue described above).

I opened https://github.com/OpenLiberty/ci.maven/pull/1562/files to update the docs accordingly, but let me know if you'd like me to open a separate issue for that.

$ mvn liberty:dev -f parent -pl :module2 -am
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] module1                                                            [jar]
[INFO] module2                                                            [war]
[INFO] 
[INFO] -------------------< io.openliberty.guides:module1 >--------------------
[INFO] Building module1 1.0-SNAPSHOT                                      [1/2]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- liberty-maven-plugin:3.5.1:dev (default-cli) @ module1 ---
[INFO] CWWKM2102I: Using artifact based assembly archive : io.openliberty:openliberty-kernel:null:22.0.0.7:zip.
[INFO] CWWKM2102I: Using installDirectory : /Users/lsurprenant/git/guide-getting-started/module1/target/liberty/wlp.
[INFO] CWWKM2102I: Using serverName : defaultServer.
[INFO] CWWKM2102I: Using serverDirectory : /Users/lsurprenant/git/guide-getting-started/module1/target/liberty/wlp/usr/servers/defaultServer.
[INFO] Running maven-dependency-plugin:purge-local-repository
[INFO] Running maven-resources-plugin:resources
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/lsurprenant/git/guide-getting-started/module1/src/main/resources
[INFO] Running maven-compiler-plugin:compile on /Users/lsurprenant/git/guide-getting-started/module1/pom.xml
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] -------------------< io.openliberty.guides:module2 >--------------------
[INFO] Building module2 1.0-SNAPSHOT                                      [2/2]
[INFO] --------------------------------[ war ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for module1 1.0-SNAPSHOT:
[INFO] 
[INFO] module1 ............................................ SUCCESS [  0.553 s]
[INFO] module2 ............................................ FAILURE [  0.004 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.637 s
[INFO] Finished at: 2022-07-12T15:37:30-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project module2: Could not resolve dependencies for project io.openliberty.guides:module2:war:1.0-SNAPSHOT: Could not find artifact io.openliberty.guides:module1:jar:tests:1.0-SNAPSHOT -> [Help 1]

[kathrynkodama]

Thanks @lmsurpre, I am able to reproduce this issue.

Looks as though dev mode is unable to resolve the module1 dependency in this case because of the tests classifier:

        <dependency>
            <groupId>io.openliberty.guides</groupId>
            <artifactId>module1</artifactId>
            <version>1.0-SNAPSHOT</version>
            <classifier>tests</classifier>
            <scope>test</scope>
        </dependency>

In dev mode we should support classifiers when resolving dependencies for multi module projects.

I'll also note that the documented mvn liberty:dev -f parent -pl module2 -am syntax didn't work for me at all. That one produces an error like this:

[ERROR] [ERROR] Could not find the selected project in the reactor: module2 @ 
[ERROR] Could not find the selected project in the reactor: module2 -> [Help 1]

However, using the "reactor" syntax (prefixing the module name with a : to indicate that module2 is an artifactId), I got back to the same issue described above).

I opened https://github.com/OpenLiberty/ci.maven/pull/1562/files to update the docs accordingly, but let me know if you'd like me to open a separate issue for that.

The documentation for the -pl module2 -am syntax assumes a relative path approach, for this project layout, -pl ../module2 -am would work. However, I think to be more robust documenting to always prefix the module name with a : to indicate artifactId is a good idea rather than the relative path approach.

More details on --projects (-pl for short) here: https://maven.apache.org/guides/mini/guide-multiple-modules-4.html#inclusion-and-exclusion

[maroschutte]

I just encountered this problem as well today. To me it seems related to 'maven-dependency-plugin:purge-local-repository' being called during execution of 'liberty-maven-plugin:dev' and I am wondering why this is being done. Does anyone have a pointer for me ?

[kathrynkodama]

I just encountered this problem as well today. To me it seems related to 'maven-dependency-plugin:purge-local-repository' being called during execution of 'liberty-maven-plugin:dev' and I am wondering why this is being done. Does anyone have a pointer for me ?

Hi @maroschutte, hot deployment through dev mode is made possible because the application is installed as a loose application. A loose application loads the associated files directly from the workspace rather than being exported.

Dev mode calls maven-dependency-plugin:purse-local-repository in a multi module scenario to ensure that any downstream modules will not rely on the installed artifacts for their compilation, but are instead loaded from the workspace (loose application).

Unfortunately we do not yet have a workaround for this sort of project configuration.

More on loose applications in Liberty here: https://www.ibm.com/docs/en/was-liberty/base?topic=liberty-loose-applications

[maroschutte]

It really worries me that liberty-maven-plugin clears artifacts from the local repository. At the very least it is really confusing to have artifacts that you put there using a 'mvn install' just disappear, but it could actually mean that existing setups that rely on artifacts being present in the localrepo break once you run dev mode. The test jars are a good example of this.

[scottkurz]

It really worries me that liberty-maven-plugin clears artifacts from the local repository. At the very least it is really confusing to have artifacts that you put there using a 'mvn install' just disappear, but it could actually mean that existing setups that rely on artifacts being present in the localrepo break once you run dev mode. The test jars are a good example of this.

That's a good question @maroschutte. I'm looking over the logic behind the change we made in #1201 and trying to understand better. If the purpose is just to surface a warning message to the user then messing with someone's local .m2 repo seems a bit disruptive. Will let you know what we find and create a new issue if necessary (since this one has already identified a primary issue with the use of the classifier). Thank you for bringing this up.

[scottkurz]

I opened a new issue #1569 to consider an alternative to doing the purge since I have a prototype which may solve the problem in a better way.

[lmsurpre]

I saw #1569 is marked as completed now... are we expecting that to also address the issue presented here?

[cherylking]

@lmsurpre Scott is on vacation right now, but I believe the changes for #1569 were meant to solve this. I'll wait for him to return and confirm before closing this issue though.

[scottkurz]

@lmsurpre good question. The #1569/#1570 change will get past the error in the recreate use case you mentioned, where you did the mvn install but then the dependency couldn't be resolved.

So from that perspective, the problem will be fixed in the next release.

You could also achieve something similar by doing: mvn clean test-compile jar:test-jar liberty:dev without needing to do the local install.

However, I wonder what your intention was in declaring the test dependency and I wonder if you are going to be happy with the dev mode behavior here. I can forsee a potential problem: if the idea was to provide failsafe configuration in module2 with a test packaged in module1 then I think this could be a gotcha, since I believe dev mode would likely run the module1 tests using the module1 pom config.

Perhaps I'm getting ahead of myself though. In any case, when the next release is available, please let us know if you have other issues, either by an issue on this repo or on Gitter at: https://gitter.im/OpenLiberty/developer-experience. Thanks.

[lmsurpre]

if the idea was to provide failsafe configuration in module2 with a test packaged in module1 then I think this could be a gotcha, since I believe dev mode would likely run the module1 tests using the module1 pom config.

I'd actually like to learn how to set up a test like that :-)
However, in the meantime, its more like we have tests in module2 that refer to test utilities in module1.
Happy to try this and report back whenever it gets released...is there any rough timeframe on that (e.g. days vs weeks vs months?)

[cherylking]

Happy to try this and report back whenever it gets released...is there any rough timeframe on that (e.g. days vs weeks vs months?)

We are planning for a release the third week of September as of right now.

[scottkurz]

Continuing my comment: #1561 (comment)

I was able to verify the original problem as reported is fixed in 3.7... though no rush to close if we want to wait for @lmsurpre to have a chance to try it out too.

To follow-up some of the issues raised.... first, I think we can ignore my comment about trying to provide

failsafe configuration in module2 with a test packaged in module1

The example in the original post here involved extending a base class in module1 with a subclass in module2 and configuring this in module2. That should work fine, and as there are other issues or concerns, let's worry about that when they arise.

FOLLOW-UP ISSUE

However, there's another problem, which is there should be a way to resolve the artifact used in the 'module1' dependency with just a:
mvn liberty:dev -f parent.

There are two ways to deal with this issue

• Do a mvn install of the dependency first before running dev mode
• Do a mvn test-compile liberty:dev -f parent

The problem with doing the install first is that you end up "stuck" with the previously-built version of the dependency artifact. You miss out on one of the whole values of dev mode.

The second one will work better, the problem is that it's simply not all that obvious that this is what you need to do.
It would be a nice story too from the perspective that it generally shouldn't be necessary to do a mvn install before a multi-module build.

I looked to see if there was a simple fix and there doesn't seem to be. It might even require a bit of a hack to make this work and to get the artifacts to resolve correctly within dev mode. I believe some of the artifact resolution actually is designed to behave differently in the compile phases (which we don't participate in doing just mvn liberty:dev).

I'm not sure it's worth opening a new issue just yet, though we could. It's probably enough of a corner case that we might not prioritize it for awhile.

But hopefully this analysis has been useful to capture here at least.

[lmsurpre]

@scottkurz Hi Scott, we're starting up a new project and considering using OpenLiberty. My colleague is experiencing some pain related to your "FOLLOW-UP ISSUE" writeup above.

Is that expected to still be an issue? Is there any plan to address it?
Maybe we should

1. close this really old issue since the original issue was addressed; and
2. open a new enhancement request for getting "hot load" to work better for multi-module projects

Agree?

[cherylking]

@lmsurpre Scott is on vacation for the next week. I will go ahead and open a separate issue for the "FOLLOW-UP ISSUE". Nothing has been done to resolve it yet, and it wasn't on our current radar of things to address in the near term. I will tag Scott in the new issue and have him follow up when he gets back. Thanks for reaching out.

[lmsurpre]

Thanks @cherylking now that we have that one open, I'll go ahead and close this one out. Feel free to reopen if you disagree.