Please upgrade libffi to ≥3.4.2

[DaveCTurner]

The latest libffi (at time of writing) adds support for specifying its temporary directory via the LIBFFI_TMPDIR environment variable. Earlier versions only use more generally-applicable locations such as TMPDIR and HOME. In Elasticsearch we'd like to be able to control where libffi writes its executables by setting LIBFFI_TMPDIR rather than TMPDIR, because administrators of locked-down systems may prefer to keep TMPDIR pointing somewhere that's mounted noexec.

Relates #1378 which fixes a segfault if TMPDIR is mounted noexec, but doesn't solve the underlying problem that libffi needs to write executables somewhere.

Relates elastic/elasticsearch#77014 which tracks the changes we intend to make in Elasticsearch once JNA is using a sufficiently-recent libffi.

[matthiasblaesing]

A first rough update can be found in PR #1383. This is not complete, but a first working version.

[matthiasblaesing]

Ok - ran unittests on:

• Linux x86-64
• Windows x86-64
• Windows x86
• Solaris x86-64
• Solaris Sparc9 (64Bit)
• macOS x86-64

All existing native libraries, with the exception of AIX, were rebuild with the new libffi. As this does not touch the binary interface, AIX should still work.

I would appreciate a test, before I merge this into master.

[DaveCTurner]

I would appreciate a test, before I merge this into master.

I can confirm that Elasticsearch successfully works with the new version built from your branch. However I'm struggling to test that it supports LIBFFI_TMPDIR as expected, because this only has an effect when SELinux is enabled and I don't have access to such a machine at the moment.

If you would like me to check that too, could you point me to instructions to rebuild the native code? That way I can skip the SELinux checks and force it always to use file-backed pages for its executables. I tried cd native && ant but it failed:

~/jna/native$ ant
Buildfile: /home/davidturner/jna/native/build.xml
     [echo] java.home: /usr/lib/jvm/java-8-oracle/jre
     [echo] jdk.home: /usr/lib/jvm/java-8-oracle
     [echo] JAVA_HOME: /usr/lib/jvm/java-8-oracle

workspace-base:

compile-base:

compile-core:
     [echo] Compile /home/davidturner/jna/src - compatibility=1.6 - target=/home/davidturner/jna/native/target/classes

javah:
     [echo] Create native class headers in /home/davidturner/jna/native/target/native-linux-x86-64
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function_PointerArray.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function_NativeMappedArray.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Function_PostCallRead.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native_AWT.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native_Buffers.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_Native_ffi_callback.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/davidturner/jna/native/target/native-linux-x86-64/com_sun_jna_win32_DLLCallback.h]]

create-version-resource:

compile-native:
     [echo] ARCH: x86-64
     [echo] make.USE_MSVC: IGNORE=
     [echo] make.CC: IGNORE=
     [echo] make.ARCH: ARCH=x86-64
     [echo] make.BUILD: BUILD=/home/davidturner/jna/native/target/native-linux-x86-64
     [echo] make.PATH: IGNORE=
     [echo] make.OPTS: IGNORE=
     [echo] make /home/davidturner/jna/native/Makefile - target=/home/davidturner/jna/native/target/native-linux-x86-64
     [exec] Configuring libffi (x86-64)
     [exec] configure: WARNING: unrecognized options: --enable-static, --disable-shared, --with-pic
     [exec] Makefile:509: recipe for target '/home/davidturner/jna/native/target/native-linux-x86-64/libffi/.libs/libffi.a' failed
     [exec] configure: error: cannot find install-sh, install.sh, or shtool in "/home/davidturner/jna/native/libffi" "/home/davidturner/jna/native/libffi/.." "/home/davidturner/jna/native/libffi/../.."
     [exec] make: *** [/home/davidturner/jna/native/target/native-linux-x86-64/libffi/.libs/libffi.a] Error 1

BUILD FAILED
/home/davidturner/jna/native/build.xml:364: exec returned: 2

Total time: 0 seconds

[matthiasblaesing]

Ah - I just noticed, that there are recipes to build on mac and windows, but not Linux - at some point that needs to be corrected. For debian/ubuntu this should work:

# fetch the build dependencies for the distribution libjna-java package
apt-get build-dep -yq --force-yes libjna-java
# install the remaining required dependencies
apt-get install -yq --force-yes zip unzip libtool automake libltdl-dev texinfo

[matthiasblaesing]

I now also ran unittests on solaris and found no regressions. I intent to merge this the next few days.

[matthiasblaesing]

Change was merged to master.

[DaveCTurner]

Sorry it's taken so long to get back to this - testing this turned into quite a yak-shaving exercise for various reasons. Anyway I can now confirm that with this version of JNA Elasticsearch does indeed respect the $LIBFFI_TMPDIR environment variable if it's prevented from using anonymous mappings, which was what we wanted 🎉

Thanks so much for doing this. Can you give us a sense of how long it'll be until the next release?

[matthiasblaesing]

Thank you for the check and verification!

For a release: It depends if anyone asks for it and you just did. I'll look into it.

[dkocher]

Thank you for the check and verification!

For a release: It depends if anyone asks for it and you just did. I'll look into it.

May I second that.

[matthiasblaesing]

Released: https://groups.google.com/g/jna-users/c/g4essN3m1Fo