"Couldn't register scheme handler" errors for tool linking (likely incorrectly?) to htslib

[jwimberl]

I am working to enable S3 connectivity in a fork of a tool which includes htslib as a dependency. This tool currently configuring and compiles htslib without plugins and statically links to libhts.a. I've set the configure options --enable-libcurl --enable-plugins --enable-s3 and recompiled the tool (with hts_verbose turned up). Setting LD_LIBRARY_PATH to the internal htslib build directory that contains hfile_curl.so and hfile_s3.so, I first confirmed that htsfile (built alongside libhts by the first step in the tool's build script) can load the plugins and query from s3. However, when I run the tool itself pointing to an object in S3, I get the following errors:

[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "crypt4gh-needed"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for dict
[W::hfile_add_scheme_handler] Couldn't register scheme handler for file
...omitted
[D::init_add_plugin] Loaded "/GLnexus/external/src/htslib/hfile_libcurl.so"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for gs
...omitted
[D::init_add_plugin] Loaded "/GLnexus/external/src/htslib/hfile_gcs.so"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3
...omitted
[D::init_add_plugin] Loaded "/GLnexus/external/src/htslib/hfile_s3.so"
[W::hfile_add_scheme_handler] Couldn't register scheme handler for s3w
...omitted
[D::init_add_plugin] Loaded "/GLnexus/external/src/htslib/hfile_s3_write.so"

I saw similar error messages reported in #1176 , and though this system is an Ubuntu container inside CentOS 7, the discussion there did leave me to believe that the issue may have something to do with the conflict between the static libhts.a library that the tool's executable links in, and the shared libhts.so loaded by the plugins. I'd given a though to trying instead to dynamically libhts in the tool's executable, as a test, but I do see that htsfile chooses the static link option and works infe.

What are the essential requirements when building and linking to htslib in order to be able to load the s3 plugins at runtime? I realize that things like htsfile and bcftools do just this, and so I should only need to find the difference to how they compile against and link htslib, but I haven't been able to find any major difference. (When I have a minimal or at least concise reproduction of my issue I could share it).

[jwimberl]

Minutes after writing the issue, I found that the issue was resolved when dynamically linking to libhts.so rather than statically linking to libhts.a. Of course that cannot be a hard requirement, since as I noted above htsfile goes the static route, so I'm uncertain why it was necessary in this case -- perhaps some conflicting in my unshared build settings.

[whitwham]

Does setting HTS_PATH help? Mentioned here.

[jwimberl]

I did have the HTS_PATH set; when it was unset I received different error messages stating that the plugins could not be loaded.

Based on my current workaround of using dynamic linking, a better title for this issue would be the requirements for statically linking to libhts and using its plugins.

[jmarshall]

The plugins are separately compiled, so you should be able to point HTS_PATH at the plugins in a standalone HTSlib directory or installation. Then you can just configure and compile your tool's htslib with --enable-plugins and not compile the actual plugins, simplifying your build. Of course, ideally your tool would not bundle and build HTSlib itself but would be able to build against a previously installed HTSlib.

The plugin machinery does not use LD_LIBRARY_PATH, so you shouldn't set that in an attempt to have it find the plugins. Setting LD_LIBRARY_PATH can confuse matters (e.g. which libhts.so.* is actually used at runtime), so ideally you would compile your tool so as to avoid needing to do that.

The “Couldn't register scheme handler” message appears fairly generic, but in fact it appears in one specific place in the code where sadly there is no errno or similar to report. So you'll need to bring out the big guns to debug this.

First off, what do ldd /path/to/yourtool and ldd /GLnexus/external/src/htslib/hfile_libcurl.so report?

Next, what does the following report (with hts_verbose still cranked up)? The output will be voluminous, so it would probably be best to attach or upload it rather than paste it into a comment.

LD_DEBUG=libs,files,symbols /path/to/yourtool s3://whatever/whatever.bam 2> loader.log

[whitwham]

If there is no update on this by the end of the week I am going to close it.

[jwimberl]

Apologies for the long delay --

# ldd glnexus_cli 
        linux-vdso.so.1 (0x00007ffe8c371000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb6d7118000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb6d6ef9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb6d6b5b000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb6d6943000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb6d6552000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb6d7ec4000)

libhts.so.3 not found for the plugin w/o LD_LIBRARY_PATH (since the library is bundled and not getting installed):

# ldd ./external/src/htslib/hfile_libcurl.so
        linux-vdso.so.1 (0x00007ffe27b9c000)
        libhts.so.3 => not found
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f27f2bad000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f27f298e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f27f259d000)
        libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f27f2378000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f27f215b000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f27f1f3f000)
        libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f27f1d31000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f27f1aa4000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f27f15d8000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f27f138d000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f27f113b000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f27f0f2d000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f27f0d10000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f27f3038000)
        libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f27f0992000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f27f062c000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f27f03f6000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f27f01c0000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f27eff3f000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f27efd3b000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f27efa65000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f27ef833000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f27ef62f000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f27ef424000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f27ef20a000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f27eefef000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f27eedae000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f27eea7f000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f27ee86c000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f27ee668000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f27ee45f000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f27ee1d2000)
        libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f27edf30000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f27edcfa000)
        libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f27edae4000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f27ed8dc000)
        libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f27ed6b3000)
        libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f27ed4a4000)
        libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f27ed25a000)
        libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f27ecf51000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f27ecd19000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f27ec97b000)

Setting LD_LIBRARY_PATH, libhts.so is no longer "not found" of course.

I generated a loader.log file as instructed, attached here: loader.log

Thanks for your help, and apologies again for the delay.

[jmarshall]

What we learn from your loader.log is that when each plugin uses a function like hfile_add_scheme_handler(), it needs to look through all the loaded shared libraries to find it (see lines 147041–49) in the libhts.so.3 pulled in by the plugin. The plugin system expects it to be found in the same place as the load_hfile_plugins() and hopen() functions, namely ./glnexus_cli.

As noted in the issue title, the suspicion here is that the tool is likely being incorrectly linked — but you haven't shown us the link command that is producing the glnexus_cli executable. 😄 I suspect the problem is that you are not using -rdynamic.

At present, if enabling plugins and linking your executable against libhts.a statically, you need to use -rdynamic when linking your executable. This probably needs to be documented better.

It would be possible to revise the plugin mechanism so that the plugin initialisation function was explicitly provided with a table of pointers to plugin-setup functions etc as an argument rather than calling them by name and depending on the loader symbol resolution and so requiring only one copy of hfile.o (and the schemes table in particular) to exist or at least to be in use. (Then it wouldn't matter how many copies of hfile.o there were at load time, RTLD_GLOBAL defaults wouldn't matter, and you wouldn't need to use -rdynamic to ensure looking up symbols found the main executable's copy.) Now that the main plugins have bedded in very well, it's probably time to revise it in that way and sidestep this linking and loading complexity.

[jwimberl]

Indeed, adding rdynamic to the compilation of the executable resolves the issue -- I no longer need to set LD_LIBRARY_PATH and the plugins load successfully -- thanks!

And thanks for the insight despite me not providing a the link command. I didn't mean to be cryptic, this is a fork of an open source tool after all (https://github.com/jwimberl/GLnexus, forked from https://github.com/dnanexus-rnd/GLnexus), but the building and linking is handled by a lengthy CMake config that I hadn't fully digested, let alone stripped down a MRE. I thought including it might add more noise than actual helpful info. As it turns out, there was a change in CMake from version 3.3 to 3.4 in which it stopped adding rdynamic linking flags by default (https://cmake.org/cmake/help/latest/policy/CMP0065.html). The canonical way to resolve this in CMake is to add a line like

set_property(TARGET executable_name PROPERTY ENABLE_EXPORTS 1)

which, while it works, has the somewhat unfortunate downside of obfuscating the setting of the "rdynamic" flag by renaming the concept to ENABLE_EXPORTS.

[jmarshall]

Well, that's CMake for ya 😄

No worries, I should have figured it out sooner but it's been a while since anyone was immersed in this code.

Your issue has prompted me to flesh out what a revised plugin mechanism that was immune to these issues would look like, and in fact it can be done while maintaining good compatibility with existing plugins and htslibs. So I'll turn that into a draft PR if the maintainers are interested.

[jkbonfield]

I wouldn't have any objections to a more robust plugin mechanism, but @daviesrob is the arbiter.

[daviesrob]

I'd certainly be interested in an improved plug-in PR.

[daviesrob]

I guess we can close this now the original problem has been dealt with and #1611 merged. We'll remember the bit about the plugin API separately.