Problem with compilation of terra on Red Hat Linux caused by "missing" std::filesystem

[bodkan]

I have an issue with compiling the latest version of terra from CRAN on my Red Hat Linux system while installing it as a dependency of other R packages (in this case rnaturalearth). Although I haven't used terra directly before, I think I must have managed to install it successfully at some point in the past as a dependency via other R packages -- unfortunately, I don't know at which point has the compilation issue described below started and if this is or isn't a regression of some kind.

I think I have a potential fix for this which I attempt to describe below (I don't really write C or C++ code so excuse some weird terminology or naivety in my approach).

Problem description

When I run install.packages("terra"), I get the following (I'm cutting the log output to what I think is the most useful information -- the error itself is at the end):

> install.packages("terra")

> install.packages("terra")
Installing package into ‘/maps/projects/racimolab/people/krd114/.R_LIBS’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/terra_1.7-37.tar.gz'
Content type 'application/x-gzip' length 814649 bytes (795 KB)
==================================================
downloaded 795 KB

* installing *source* package ‘terra’ ...
** package ‘terra’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++ -std=gnu++14
checking for gdal-config... /usr/bin/gdal-config
checking gdal-config usability... yes
configure: GDAL: 3.0.4
checking GDAL version >= 2.0.1... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for gdal.h... yes
checking GDAL: linking with --libs only... yes
checking GDAL: /usr/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... yes
configure: GDAL: 3.0.4
configure: pkg-config proj exists, will use it
configure: using proj.h.
configure: PROJ: 6.3.2
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... yes
checking for geos-config... /usr/bin/geos-config
checking geos-config usability... yes
configure: GEOS: 3.7.2
checking GEOS version >= 3.4.0... yes
checking for geos_c.h... yes
checking geos: linking with -L/usr/lib64 -lgeos_c... yes
configure: Package CPP flags:   -DHAVE_PROJ_H -I/usr/include/gdal -I/usr/include
configure: Package LIBS: -lproj    -L/usr/lib64 -lgdal -L/usr/lib64 -lgeos_c
configure: creating ./config.status
config.status: creating src/Makevars
** libs
g++ -std=gnu++17 -I"/usr/include/R" -DNDEBUG -DHAVE_PROJ_H -I/usr/include/gdal -I/usr/include -I'/maps/projects/racimolab/people/krd114/.R_LIBS/Rcpp/include' -I/usr/local/include   -fpic  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -c RcppExports.cpp -o RcppExports.o

[...]

*** copying figures
** building package indices
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘terra’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/maps/projects/racimolab/people/krd114/.R_LIBS/00LOCK-terra/00new/terra/libs/terra.so':                                                                                   
  /maps/projects/racimolab/people/krd114/.R_LIBS/00LOCK-terra/00new/terra/libs/terra.so:
  undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv                                              
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/maps/projects/racimolab/people/krd114/.R_LIBS/terra’

The downloaded source packages are in
        ‘/tmp/RtmpebF2cs/downloaded_packages’
Warning message:
In install.packages("terra") :
  installation of package ‘terra’ had non-zero exit status

System information

Compiler version:

> g++ --version
g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)

> gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)

OS version:

> cat /etc/redhat-release
Red Hat Enterprise Linux release 8.7 (Ootpa)

(released on November 9, 2022)

R version:

> version
               _                           
platform       x86_64-redhat-linux-gnu     
arch           x86_64                      
os             linux-gnu                   
system         x86_64, linux-gnu           
status                                     
major          4                           
minor          2.2                         
year           2022                        
month          10                          
day            31                          
svn rev        83211                       
language       R                           
version.string R version 4.2.2 (2022-10-31)
nickname       Innocent and Trusting

Hypothesis

After a bit of googling based on this error from the install.packages call:

unable to load shared object '[...] terra.so:
undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv

I wondered if the issue is with std::filesystem not being properly discovered or linked -- and it does indeed seem to be the case.

Minimal (approximate) example

Consider this minimal C++ program source code test.cpp:

#include <iostream>
#include <filesystem>

int main() {
    std::filesystem::path p("/etc");
    std::cout << "The path is " << p << '\n';
}

If I compile it with this (using the -std=c++17 flag as shown in the install.packages("terra") log above):

g++ -std=c++17 test -o test

I get

/tmp/ccTr9kn2.o: In function `std::filesystem::__cxx11::path::path<char [5], std::filesystem::__cxx11│~                                                                                                   
::path>(char const (&) [5], std::filesystem::__cxx11::path::format)':                                │~                                                                                                   
test.cpp:(.text._ZNSt10filesystem7__cxx114pathC2IA5_cS1_EERKT_NS1_6formatE[_ZNSt10filesystem7__cxx114│~                                                                                                   
pathC5IA5_cS1_EERKT_NS1_6formatE]+0x5e): undefined reference to `std::filesystem::__cxx11::path::_M_s│~                                                                                                   
plit_cmpts()'                                                                                        │~                                                                                                   
collect2: error: ld returned 1 exit status

However, if I compile it with this (note the last flag!)

g++ -std=c++17 test.cpp -o test -lstdc++fs

it compiles without error and I can run the produced ./test binary.

Fix?

Based on this simple compilation test, I created a ~/.R/Makevars file with the following contents:

PKG_LIBS += -lstdc++fs

After this, running install.packages("terra") ran without any issues, and the R package was installed successfully (as do the R packages which previously failed to install due to the terra itself failing to compile).

Conclusion

It seem that although std::filesystem is part of the C++17 standard, which GCC v8.5.0 distributed by my RHEL version supports, specifying -std=c++17 is not sufficient. It needs to be linked explicitly via -lstdc++fs.

Is this something that can be added to the build instructions (or however is configuration of these things called with R packages using C/C++ code) of terra itself so that this is done automatically? I have been using R for a long time, and had my share of issues with compilation, but I have never had to use ~/.R/Makevars in the past. In my experience, 99.9% of compilation issues are solved by installing a missing header library. I don't have any experience with C/C++ compilation in R packages, but I would hope that adding -lstdc++fs would be -- in the worst case -- redundant while fixing compilation using compiler toolchains which do not link to std::filesystem automatically (despite supporting C++17).

(Unfortunately, I work on shared machines in a university setting which maintains corporate-level RHEL installations, so I have no way to influence which versions of GCC or other compiler toolchains are installed on our systems.)

Thanks for your help!

[bodkan]

Upon reading a bit more into the topic, I discovered this. It's probably obvious to experts in C++ standards (or, as in this case, their implementations), but I'm reproducing it here full for posterity in case someone else runs into this or similar C++ compilation issues:

It is not specified in the C++ standards how exactly you must invoke your compiler. I guess this is up to implementations to decide, so there might still be a need for -lstdc++fs.

Note that C++17 is not yet officially a standard, and implementations may or may not yet have implemented it. Even when C++17 finally becomes a standard, this won't magically change the implementation. It doesn't even define what linker flags an implementation should or should not require. It is completely unrelated.

How you can include the officially accepted version after C++17 becomes a standard and your implementation begins to support it, is completely implementation-specific. We'll probably just have to wait and see how that works out.

On a similar note, it does seem like starting with GCC version 9.1, the std::filesystem library became a part of the standard library and is included by default, so the -lstdc++fs flag should not be necessary for 9.1 and later versions.

[rhijmans]

Thank you very much for your report and thorough analysis. This problem was introduced to CRAN yesterday; and I expect that others will also run into it.

I have changed the configure script so that the -lstdc++fs flag is used. I now also use <experimental/filesystem> instead of <filesystem> if I can detect that the latter is not available or when the gcc version is < 8 (I tested with 7.5.0 on Ubuntu 18.04). Further tweaks may be necessary for other compilers (CLANG on OSX). This is not an issue on windows.

The "~/.R/Makevars" file with

PKG_LIBS += -lstdc++fs

is an elegant workaround for installing the CRAN version with compilers that have the <filesystem> library.