ClassLoader can't solve directories on Graal binary

[mageddo]

Context

I'm using embedded flyway to automatically migrate my database scripts and it solves directories on classpath to load sql files

What is expected

java.lang.ClassLoader#getResources(String) returns the URL for the specified directory

What is happening

java.lang.ClassLoader#getResources(String) returns an empty Enumeration

Steps to reproduce

Here an example project reproducing the issue

App.java

public static void main(String[] args) throws Exception {
	System.out.println(getResources("folder/subfolder"));
	System.out.println(getResources("folder/subfolder/resource-001.txt"));
}

public static List<URL> getResources(String name) throws IOException {
	final ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
	return Collections.list(classLoader.getResources(name));
}

$ ./gradlew clean build nativeImage && ./build/graal/resources-resolution
[]
[resource:folder/subfolder/resource-001.txt]

[guanchao-yang]

@olpaw Which version do we plan to release ? Thank you very much

[olpaw]

@guanchao-yang, @mageddo as of now (GraalVM 19.1.1) we only support adding resources that are real data files to native-images. A directory on its own is not seen as a resource because we cannot embed a directory into a native-image. If we'd add support for that, which kind of usage patterns would need to be supported on resources that represent directories to make spring work as expected?

[dsyer]

which kind of usage patterns would need to be supported on resources that represent directories to make spring work as expected?

It's nothing really to do with Spring. The question above was about Flyway, where they sort the filenames in a directory and execute them as scripts in order. Webjars also likes to traverse "directories" in classpath resources (so it can resolve the "best" available resource matching the request). It's just a common pattern, to search for stuff in packages and subpackages.

Here's a library dedicated to scanning classpath directories: https://github.com/classgraph/classgraph. It's used by the most recent versions of webjars, and naturally doesn't work in substratevm because of the above.

[rtfpessoa]

Any progress here? Are there any workarounds we can do to list all files in a directory or in the classpath in general?

[rdehuyss]

Also running in troubles because of this...

[jantoniucci]

Me too. Are there any update nor workaround on it ? Thanks!

[mhalbritter]

We're currently working on implementing Flyway support on native-image and it turns out quite harder than we initially thought it would be.

Any chances that this issue here will be implemented in the foreseeable future?

Right now, we have to come up with workarounds for the classpath enumeration. It would be great if an application running in a native image could just use the classpath enumeration libraries like Reflections / Classgraph / Springs PathMatchingResourcePatternResolver and get back the included resources like on the JVM. This would make all the workarounds no longer necessary.

[mhalbritter]

Interestingly, one can list resources at runtime:

        try (FileSystem fileSystem = FileSystems.newFileSystem(URI.create("resource:/"), Map.of(), classLoader)) {
            Path path = fileSystem.getPath("folder");
            try (Stream<Path> files = Files.walk(path)) {
                files.forEach(file -> {
                    System.out.print(file.toString());
                    if (Files.isDirectory(file)) {
                        System.out.println(" (D)");
                    } else if (Files.isRegularFile(file)) {
                        System.out.println(" (F)");
                    } else {
                        System.out.println(" (?)");
                    }
                });
            }
        }

with one caveat: you can't list starting from the root folder, you have to know the name of the first directory where to start.

With resource-config.json like this:

{
  "resources": {
    "includes": [
      {
        "pattern": "folder.*"
      }
    ]
  }
}

and a directory tree under src/main/resources like this:

src/main/resources
├── folder
│   ├── 1.txt
│   ├── 2.txt
│   └── sub-folder
│       ├── 3.txt
│       └── 4.txt
└── META-INF
    └── native-image
        └── resource-config.json

this prints

folder (D)
folder/2.txt (F)
folder/1.txt (F)
folder/sub-folder (D)
folder/sub-folder/3.txt (F)
folder/sub-folder/4.txt (F)

This works because there is a filesystem named com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider available which handles the resource scheme.

But this doesn't matter, because now we have to convince maintainers of Flyway, Reflections, etc. to add specific support for GraalVM. It would be nice if this would work in a JVM compatible way.

[jovanstevanovic]

I see that the discussion is a bit old, but maybe I could help here.

• The provided example is not building at all because of an old version of the native image.
• After some manual changes, I managed to make it work. Here is a command that I have used: native-image -H:IncludeResources="folder/.*" -cp "build/classes/java/main/:build/resources/main/" com.mageddo.resources.resolution.App. So, the initial problem is not relevant anymore, right?
• I'm the creator of NativeImageResourceFileSystemProvider, so that caveat is actually my mistake. I overlooked that somehow.
• @mhalbritter How exactly we can help from the Native Image side? Is this the exampe that is not working?

[dsyer]

I couldn't build the original sample either. If you could post your changes it might be useful. How do you mean "it worked"? What did you run and what was the output?

Instead I tried the Flyway sample at https://github.com/spring-projects/spring-aot-smoke-tests with a manual resources-config.json:

{
  "resources": {
    "includes": [
      {
        "pattern": "\\Qdb\/migration\/\\E.*"
      }
    ]
  }
}

If you run ./gradlew :flyway:bootRun it works and reports the 2 migrations. If you try ./gradlew :flyway:nativeRun it will run but report that 0 migrations were found.

[jovanstevanovic]

Regarding the original sample (using the latest labsjdk11 and native image):

• export JAVA_HOME=/path/to/jdk11
• ./gradlew clean build
• native-image -H:IncludeResources="folder/.*" -cp "build/classes/java/main/:build/resources/main/" com.mageddo.resources.resolution.App

The output was as expected:

[resource:/folder/subfolder]
[resource:/folder/subfolder/resource-001.txt]

I'm not sure what is the problem with the original sample Gradle setup, so I will need to take a better look to see why it's not building using ./gradle nativeImage.

Regarding the spring aot smoke test, I'm running it right now.

[dsyer]

So the problem is that resource: is the URL prefix for the resources found in the native image, but Flyway doesn't think it knows how to resolve them:

2022-09-14T11:44:56.828Z  WARN 17198 --- [           main] o.f.c.i.s.classpath.ClassPathScanner     : Unable to scan location: /db/migration (unsupported protocol: resource)

I think @mhalbritter already alluded to this, but I wasn't following. It looks to me as if Flyway (and Spring etc.) could use the FileSystems abstraction, so they don't need to know about GraalVM, just java.nio.

[sbrannen]

Right. resource: is non-standard.

Existing libraries have been working for over a decade scanning the classpath using ClassLoader#getResources.

Requiring all applications and libraries to add special support for resource: is therefore not a viable option IMO, since that would require many libraries to be rewritten or augmented to support resource:.

The only reasonable solution I see is for GraalVM to provide the support already available in NativeImageResourceFileSystemProvider transparently via ClassLoader#getResources.

[sdeleuze]

I tend to agree with @sbrannen take above.

[dsyer]

But ClassLoader#getResources returns an array of URI so I don't see how GraalVM can change that in a sensible way. It's how the existing libraries resolve the URI that's the problem. They have been hacking on that for decades, making assumptions about "jar:" and "file:" prefixes, and ignoring the abstractions provided by java.nio.FileSystems. Isn't it time we used the abstractions provided by the JDK (which would work in a native image too)?

[sbrannen]

They have been hacking on that for decades, making assumptions about "jar:" and "file:" prefixes, and ignoring the abstractions provided by java.nio.FileSystems. Isn't it time we used the abstractions provided by the JDK (which would work in a native image too)?

Good point.

Are you proposing that libraries like Spring Framework attempt to use java.nio.FileSystems for any unrecognized protocol (like resource:)?

If so, then I question GraalVM's choice of resource as a protocol since it is a rather generic term that may already be in use. I would expect a protocol supported only within a GraalVM native image to have a more specific name such as native-resource:.

[dsyer]

@mhalbritter's code above fails if the starting URI is a file:/.... The FileSystem implementation likes to have absolute paths that start and end with "/", but the resource: implementation in GraalVM doesn't. So you can't use the same code with both unless you do some ugly hacks:

		String rootPath = rootDirResource.getURI().getRawPath();
		if (!("file").equals(rootDirResource.getURI().getScheme()) && rootPath.startsWith("/")) {
			rootPath = rootPath.substring(1);
			if (rootPath.length()==0) {
				return result;
			}
			if (rootPath.endsWith("/")) {
				rootPath = rootPath.substring(0, rootPath.length()-1);
			}
			if (rootPath.length()==0) {
				return result;
			}
		}

It would be good to not have to do that.

[bclozel]

@jovanstevanovic I've raised #5020, could you have a look? I'll try extracting feedback from here into separate issues so that we can close this one.

[jovanstevanovic]

@bclozel thanks for extracting the issue from this discussion, that is actually something that I've planned to do. Sure, we can close this one now. 💯