GCC prefers /lib64 libs over $LIBRARY_PATH from EasyBuild modules (!)

[michaelkarlcoleman]

While debugging a Boost link issue, the cause turned out to be that g++ (from GCCcore/6.4.0) will use libraries in /lib64 (actually /lib/../lib64) in preference to the libraries from loaded EasyBuild modules, which are specified in LIBRARY_PATH and LD_LIBRARY_PATH.

Clarification: The actual directories mentioned in LIBRARY_PATH appear at the end of the list of directories that gcc searches. Modified versions of the LIBRARY_PATH directories are placed at the front of the gcc search list, but those directories generally don't exist because they're not created by easyconfigs. The net effect of this is that distributions libraries present in standard locations like /lib will "win" over loaded modules when gcc goes looking for libraries mentioned with -l flags.

This is with vanilla easyconfigs from develop or perhaps a release from December or so. If there's a flag to get gcc to stop doing this, I haven't spotted it.

This is super-awful, obviously. Apparently it "works" most of the time because there's either no library in /lib64 to shadow the module library, or else the library that is there is close enough compatibility-wise to not cause trouble. In my case, it failed because there was an older centos version of Boost installed, and that shadowed the EasyBuild Boost module.

Not sure what the right fix is, but it might be that eb needs to start using the .../lib64 form in addition to or instead of the .../lib form.

Here's some data:

$ ml +Boost/1.66.0-foss-2017b

$ type g++
g++ is /projects/eb-racs/p/software/GCCcore/6.4.0/bin/g++

$ g++ --print-search-dirs | tr ':' '\n'
install
 /gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/
programs
 =/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../libexec/gcc/x86_64-pc-linux-gnu/6.4.0/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../libexec/gcc/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../../x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu/6.4.0/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../../x86_64-pc-linux-gnu/bin/
libraries
 =/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/
/projects/eb-racs/p/software/Boost/1.66.0-foss-2017b/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/Boost/1.66.0-foss-2017b/lib/../lib64/
/projects/eb-racs/p/software/zlib/1.2.11-GCCcore-6.4.0/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/zlib/1.2.11-GCCcore-6.4.0/lib/../lib64/
/projects/eb-racs/p/software/bzip2/1.0.6-GCCcore-6.4.0/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/bzip2/1.0.6-GCCcore-6.4.0/lib/../lib64/
/projects/eb-racs/p/software/ScaLAPACK/2.0.2-gompi-2017b-OpenBLAS-0.2.20/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/ScaLAPACK/2.0.2-gompi-2017b-OpenBLAS-0.2.20/lib/../lib64/
/projects/eb-racs/p/software/FFTW/3.3.6-gompi-2017b/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/FFTW/3.3.6-gompi-2017b/lib/../lib64/
/projects/eb-racs/p/software/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib/../lib64/
/projects/eb-racs/p/software/OpenMPI/2.1.1-GCC-6.4.0-2.28/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/OpenMPI/2.1.1-GCC-6.4.0-2.28/lib/../lib64/
/projects/eb-racs/p/software/hwloc/1.11.7-GCCcore-6.4.0/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/hwloc/1.11.7-GCCcore-6.4.0/lib/../lib64/
/projects/eb-racs/p/software/numactl/2.0.11-GCCcore-6.4.0/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/numactl/2.0.11-GCCcore-6.4.0/lib/../lib64/
/projects/eb-racs/p/software/binutils/2.28-GCCcore-6.4.0/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/binutils/2.28-GCCcore-6.4.0/lib/../lib64/
/projects/eb-racs/p/software/GCCcore/6.4.0/lib64/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/GCCcore/6.4.0/lib64/../lib64/
/projects/eb-racs/p/software/GCCcore/6.4.0/lib/x86_64-pc-linux-gnu/6.4.0/
/projects/eb-racs/p/software/GCCcore/6.4.0/lib/../lib64/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../../x86_64-pc-linux-gnu/lib/x86_64-pc-linux-gnu/6.4.0/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../../x86_64-pc-linux-gnu/lib/../lib64/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../x86_64-pc-linux-gnu/6.4.0/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../../lib64/
/lib/x86_64-pc-linux-gnu/6.4.0/
/lib/../lib64/
/usr/lib/x86_64-pc-linux-gnu/6.4.0/
/usr/lib/../lib64/
/projects/eb-racs/p/software/Boost/1.66.0-foss-2017b/lib/
/projects/eb-racs/p/software/zlib/1.2.11-GCCcore-6.4.0/lib/
/projects/eb-racs/p/software/bzip2/1.0.6-GCCcore-6.4.0/lib/
/projects/eb-racs/p/software/ScaLAPACK/2.0.2-gompi-2017b-OpenBLAS-0.2.20/lib/
/projects/eb-racs/p/software/FFTW/3.3.6-gompi-2017b/lib/
/projects/eb-racs/p/software/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib/
/projects/eb-racs/p/software/OpenMPI/2.1.1-GCC-6.4.0-2.28/lib/
/projects/eb-racs/p/software/hwloc/1.11.7-GCCcore-6.4.0/lib/
/projects/eb-racs/p/software/numactl/2.0.11-GCCcore-6.4.0/lib/
/projects/eb-racs/p/software/binutils/2.28-GCCcore-6.4.0/lib/
/projects/eb-racs/p/software/GCCcore/6.4.0/lib64/
/projects/eb-racs/p/software/GCCcore/6.4.0/lib/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../../x86_64-pc-linux-gnu/lib/
/gpfs/projects/eb-racs/p/software/GCCcore/6.4.0/bin/../lib/gcc/x86_64-pc-linux-gnu/6.4.0/../../../
/lib/
/usr/lib/

[michaelkarlcoleman]

This link provides some useful background. Apparently others have crashed into this problem as well.

[boegel]

@michaelkarlcoleman Thanks for reporting this, and sorry for not picking up on this earlier.

I'm a bit puzzled that this only popped up recently... Although your explanation covers why this problem usually doesn't surface makes sense, I would still expect that this pops up regularly.

@bartoldeman was also hitting this problem when testing #7346, so I'm wondering if he has more to add here...

@geimer: Are you familiar with this, and are you aware of a proper fix for this?

[boegel]

A general workaround for this could be to symlink lib64 to lib in every installation performed by EasyBuild (and vice versa to be sure), but that seems a bit extreme to me (although fairly easy to implement).

For specific packages, the problem is actually that there's no proper -L command line option being used, and that the linking purely relies on $LIBRARY_PATH which is only used as a fallback. We could try fixing this on a per-package basis, but that doesn't seem like an attractive approach to me either.

Yet another option is to use a compiler wrapper to inject -L options in the compiler command, much like we already do for RPATH (and like Spack also does).

[michaelkarlcoleman]

A comment on the StackOverflow page suggests that only g++ does this, but not gcc. If true, that might help explain the rarity of observed problems, since probably fewer packages actually use g++.

Another factor is that perhaps a lot of EB recipes are already using the -L $EBWHATEVER form. Possibly in some cases they did that to work around this issue, without pursuing it further. I'm not that crazy about this, as it seems to work against the whole idea of LIBRARYPATH and the module system, which is that you can just load library modules and expect that programs subsequently compiled will pick up the intended libraries.

The symlink idea sounds like it might work, ugliness notwithstanding. Probably a lot easier than actually trying to change lib to lib64 in the various package build procedures, etc.

More broadly, it'd be great to have EB packages built within a sterilized environment, perhaps provided by a Singularity image. Such an environment could have far fewer libraries present, which would make this issue more obvious when it occurs.

[boegel]

@michaelkarlcoleman My preference would go to the compiler wrapper that silently injects -L flags for every entry in $LIBRARY_PATH, but if this is to work also when you manually build software (i.e. without using EasyBuild), then that compiler wrapper would have to be a part of the compiler installation done by EasyBuild, and not be created dynamically when eb is used (as is done now with the RPATH support, cfr. https://easybuild.readthedocs.io/en/latest/RPATH-support.html).

With symlinking lib to lib64 that's less of an issue, but there corner cases like installations which already have both a lib and lib64 with different files in each should be taken into account as well.

I have been wanting to use a more isolated environment for testing installations for a while, and I'm finally getting to it, but note that this is a double-edged sword. It definitely helps with discovering that certain dependencies are being picked up from the OS rather than via EasyBuild-provided dependencies. But it wouldn't actually help to discover whether or not an installation suffers from the problem being discussed here, since that can only occur if things like zlib are installed in the OS. So you would need to test in both a fully packed isolated environment as well as a trimmed down one...

[michaelkarlcoleman]

All good points. You're right--the sterile environment would mostly be useful for detecting libs/etc. that are missing from a recipe.

Maybe there's also a special flag or two that would convince g++ to not even look for such system libs during build. That might help diagnose these issues. On the other hand, you'd want to make sure they didn't get picked up on the end user's system either. Lots of complexity.

[Flamefire]

I just ran into this with ICU and mpicc: LIBRARY_PATH=/scratch/ws/1/s3248973-EasyBuild/easybuild-ml/software/ICU/65.1-GCCcore-8.3.0/lib mpicc --print-search-dirs yields:

/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/:/scratch/ws/1/s3248973-EasyBuild/easybuild-ml/software/ICU/65.1-GCCcore-8.3.0/lib/powerpc64le-unknown-linux-gnu/8.3.0/:/scratch/ws/1/s3248973-EasyBuild/easybuild-ml/software/ICU/65.1-GCCcore-8.3.0/lib/../lib64/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/../../../../powerpc64le-unknown-linux-gnu/lib/powerpc64le-unknown-linux-gnu/8.3.0/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/../../../../powerpc64le-unknown-linux-gnu/lib/../lib64/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/../../../powerpc64le-unknown-linux-gnu/8.3.0/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/../../../../lib64/:/lib/powerpc64le-unknown-linux-gnu/8.3.0/:/lib/../lib64/:/usr/lib/powerpc64le-unknown-linux-gnu/8.3.0/:/usr/lib/../lib64/:/scratch/ws/1/s3248973-EasyBuild/easybuild-ml/software/ICU/65.1-GCCcore-8.3.0/lib/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/../../../../powerpc64le-unknown-linux-gnu/lib/:/software/ml/GCCcore/8.3.0/bin/../lib/gcc/powerpc64le-unknown-linux-gnu/8.3.0/../../../:/lib/:/usr/lib/

So I'd like to bring this up once again and vote for the solution to symlink lib64 to lib if it doesn't exist. This will solve most of the problems.

Note that in my case -L is impossible because of Bazel -.-

[boegel]

@Flamefire Symlinking lib64 makes sense, thanks for looking into this via easybuilders/easybuild-framework#3401

[boegel]

EasyBuild will now automatically symlinks lib64 to the lib subdirectory in the installation (unless lib64 exists already), thanks to the changes in easybuilders/easybuild-framework#3401), to be included with upcoming EasyBuild release (4.2.3 or 4.3.0, to be decided).