Try System.loadLibrary before trying to extract native libraries from a JAR

[manosbatsis]

We use Realm in a multi-platform compose app, one that includes a compose desktop build. Some JVM installers move native shared/JNI libraries are out of the JAR, for a number of reasons, including Windows and macOS security systems' requirement to verify all native code is signed. Conveyor specifically recommends:

Therefore your software should always attempt to load shared libraries by using System.loadLibrary first, before trying to extract native libraries from a JAR. Alternatively you can use System.load in combination with the java.home system property but remember to add either lib on UNIX or bin on Windows.

From what i can see here, it seems Realm will only use System.loadLibrary on Android, which results in exception when trying to initialize our app on desktop JVM apps:

Caused by: java.lang.RuntimeException: Couldn't load Realm native libraries
        at io.realm.kotlin.internal.interop.realmc.<clinit>(realmc.java:28)
        ... 85 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.realm.kotlin.internal.interop.realmc.<clinit>(realmc.java:26)
        ... 85 more
Caused by: java.lang.NullPointerException: lib must not be null
        at io.realm.kotlin.jvm.SoLoader.unpackAndInstall(SoLoader.kt:98)
        at io.realm.kotlin.jvm.SoLoader.load(SoLoader.kt:62)
        at io.realm.kotlin.jvm.SoLoader.load(SoLoader.kt:47)
        ... 90 more

[cmelchior]

Hi @manosbatsis Thank you for the report.

Loading native libraries on JVM are controlled by this file:

realm-kotlin/packages/cinterop/src/jvmMain/kotlin/io/realm/kotlin/jvm/SoLoader.kt

Line 69 in 4fb30d7

System.load(libraryInstallationLocation.absolutePath)

You can also see the logic for where we place the files on disk, which we check before trying to unpack them from the JAR.

If you manually install the native code, is it possible for you to place the files at that location?

Alternatively we need to find a way that users, like you, can override the default behaviour and manually point to the .so file 🤔

[nhachicha]

Hi @manosbatsis
Going quickly through https://conveyor.hydraulic.dev/1.0/configs/jvm/#native-libraries we seem to do the same thing with our loader: we extract and check the checksum for integrity and reuse the cached native lib for performance...

I think a more general approach is to implement #461 which will allow you to bypass the internal loader and then specify the location of the signed/extracted native libs Realm need to load. (without going through extraction/checksum verification etc.) as this will be handled by Conveyor as I understand.

[nhachicha]

FYI this is the content of the dependency that Conveyor needs to extract/copy

jar -t --file cinterop-jvm-1.4.0.jar
...
jni/
jni/linux/
jni/linux/dynamic_libraries.properties
jni/linux/librealmc.so
jni/macos/
jni/macos/dynamic_libraries.properties
jni/macos/librealmc.dylib
jni/windows/
jni/windows/dynamic_libraries.properties
jni/windows/realmc.dll

[mikehearn]

Hiya,

The issue here isn't really Conveyor specific. Rather, anyone who wants to package Realm with their (desktop) app and use jlink to create a bundled JVM setup will hit the issue.

In a completely normal, old-school JVM bundling setup all shared libraries are placed in the library search path by default, so when the JVM executes System.loadLibrary("foo") will work without the app needing to care about file locations, caches, extraction from jars etc. That's why the JDK's own libraries don't need to be extracted to your home directory: they're already next to libjvm and so get found correctly. On Windows this means in the same bin directory, on macOS in the Contents/runtime/Contents/Home/lib directory and so on. Doing things the standard way has many advantages for end users, like not cluttering their home directory, better startup time, better compatibility with code signing and so on.

Therefore, Realm should really be compatible with this setup and the good news is how easy it is: just run System.loadLibrary always, and continue along the existing codepaths from a catch block. If the library can't be loaded then you conclude you're in a developer environment and it's OK to go unpack to the user's home dir. If it can be loaded, happy days, you're running in an end-user package where everything is already laid out properly ahead of time.

Does that make sense?

[nhachicha]

I think it's a good idea to default to System.loadLibrary and continue with the existing loader in case of UnsatisfiedLinkError, the only annoying thing for the user is that he needs to extract the shared lib manually from the JAR downloaded from Maven (as it is our only distribution medium) install it and sets the java.library.path when launching the JVM program ... this can be documented though so it shouldn't prevent defaulting to System.loadLibrary

[mikehearn]

I think if you try loadLibrary first users won't even need to know, right? If it doesn't work you'll do what you're doing now with extraction from the JAR and everything will be just as it was. It only makes a difference if the user or a tool has done the extraction and placement.

[manosbatsis]

Fantastic guys, thank you for coming up with the PR so quickly. Looking forward to the snapshot for our dependencies

[cmelchior]

@manosbatsis We have released a 1.5.0-SNAPSHOT with a fix for this. Can you verify if this fixes your issue?

https://github.com/realm/realm-kotlin#using-snapshots

[manosbatsis]

Native library is loaded with the snapshot, thank you @cmelchior