undefined reference to ‘memcpy@GLIBC_2.1’ compiling error

[nrnhines]

An issue was posted to the NEURON forum
https://www.neuron.yale.edu/phpBB/viewtopic.php?t=4694
The author writes:

I am attempting to compile the mod files with the command nrnivmodl . for a single cell model and keep encountering the error that there is an undefined reference to ‘memcpy@GLIBC_2.14':

and gives details of the link error and environment. The problem occurs with both 8.2.4 and current master. My last message was

So the issue seems generic with respect to our wheel build environment.
I think the fact that you can launch neurondemo with its libnrniv and the libnrnmech built when the installer was created is an important clue. The issue seems focused on the nrnivmodl make process. I imagine some detail is missing in the compile or link arguments. I've created a GitHub issue to discuss with colleagues about this. Perhaps someone has a suggestion for further diagnosis/fixing. If the problem is specific to nrnivmodl and helpers then it should be more convenient for you to modify or replace the nrnmech_makefile without having to build a new wheel for each diagnostic experiment.
Note. When I create a python3.12 virtual environment in env and pip install neuron, nrnmech_makefile is located in
env/lib/python3.12/site-packages/neuron/.data/bin

[mgeplf]

I think it would be worth knowing the conda version; and also what libc gcc is actually using (from stackoverflow) should do the trick:

echo '#include <errno.h>' | gcc -xc - -E -dM | grep -E '^#define __GLIBC(|_MINOR)__ '

Making sure that the gcc that is being called is from within conda, and not the OS's version (so, perhaps /home/kedoxey/.conda/envs/python3_9-NEW/bin/../lib/gcc/x86_64-conda-linux-gnu/10.4.0/../../../../x86_64-conda-linux-gnu/bin/gcc ?)

[kedoxey]

My conda version is conda 4.10.3.

The output from echo ... is:

#define __GLIBC__ 2
#define __GLIBC_MINOR__ 35

[nrnhines]

I am trying to reproduce on a Virtual box guest (Ubuntu 23.10). Installing https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Linux-x86_64.sh
After several attempts I end up with

(base) hines@ubuntu23-10:~$ echo $PATH
/home/hines/anaconda3/bin:/home/hines/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
(base) hines@ubuntu23-10:~$ conda --version
conda 24.5.0
(base) hines@ubuntu23-10:~$ python
Python 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0] on linux

With pip install neuron I see Successfully installed find-libpython-0.4.0 neuron-8.2.4 and neurondemo works.

I had some mod files in a neuron source folder, ~/neuron/nrn/share/demo/release and from there

(base) hines@ubuntu23-10:~/neuron/nrn/share/demo/release$ nrnivmodl
...
Successfully created x86_64/special

I note that there are a number of differences with your environment (python3.9 and conda 4.10.3) and I don't know whether your problem is related to some specific mod file. With respect to the last are you able to do a successful nrnivmodl from a folder analogous to

~/anaconda3/lib/python3.12/site-packages/neuron/.data/share/nrn/demo/release

You might have to temporarily rename the x86_64 folder there first.

[kedoxey]

I am working on conda upgraded on the server to a newer version. I have found that the error appears after I copy a mod file into the folder I am trying to compile.

Fortunately, I tested nrnivmodl . yesterday and the mod files were able to compile without any error. I am hoping this error does not persist and I can avoid it by not coping files into the directory. Thanks for your help!

[nrnhines]

I likely don't understand. I'm interpreting your last comment as that you have a folder with the property that when you copy a mod file into it, the mod file will no longer compile. Is that the correct interpretation? If so, what is the full pathname of that folder? And what the the contents (all the filenames) in that folder. i.e. the output of ls -al ?

[kedoxey]

Correct. I am using the NEURON version of this single cell model from NeuroML-DB . The error appears when I copy an external mod file into the cell model's main folder.

The pathname is /home/kedoxey/CRCNS/PyramidalCellSimulations/models/NEURON/NMLCL000073-NEURON-C and the output of ls -al is

total 928
drwxrwxr-x 4 kedoxey kedoxey   4096 Jan  3  2024 .
drwxrwxr-x 9 kedoxey kedoxey   4096 Jun 27 15:47 ..
-rw-rw-r-- 1 kedoxey kedoxey   2658 Jan  3  2024 CaDynamics_E2_NML2__decay122__gamma5_09Emin4.mod
-rw-rw-r-- 1 kedoxey kedoxey   2658 Jan  3  2024 CaDynamics_E2_NML2__decay460__gamma5_01Emin4.mod
-rw-rw-r-- 1 kedoxey kedoxey   9155 Jan  3  2024 Ca_HVA.mod
-rw-rw-r-- 1 kedoxey kedoxey   9428 Jan  3  2024 Ca_LVAst.mod
-rw-rw-r-- 1 kedoxey kedoxey   6162 Jan  3  2024 Ih.mod
-rw-rw-r-- 1 kedoxey kedoxey   6233 Jan  3  2024 Im.mod
-rw-rw-r-- 1 kedoxey kedoxey   9683 Jan  3  2024 K_Pst.mod
-rw-rw-r-- 1 kedoxey kedoxey   9415 Jan  3  2024 K_Tst.mod
-rw-rw-r-- 1 kedoxey kedoxey 808918 Jan  3  2024 L5PC.hoc
-rw-rw-r-- 1 kedoxey kedoxey  14260 Jan  3  2024 Nap_Et2.mod
-rw-rw-r-- 1 kedoxey kedoxey  11095 Jan  3  2024 NaTa_t.mod
drwxrwxr-x 2 kedoxey kedoxey   4096 Jan  3  2024 output
-rw-rw-r-- 1 kedoxey kedoxey   1900 Jan  3  2024 pas_nml2.mod
-rw-rw-r-- 1 kedoxey kedoxey   6493 Jan  3  2024 SK_E2.mod
-rw-rw-r-- 1 kedoxey kedoxey   5764 Jan  3  2024 SKv3_1.mod
drwxrwxr-x 3 kedoxey kedoxey   4096 Jan 25 13:34 x86_64

[nrnhines]

The path and file list you sent above doesn't raise any red flags with me. Though I'm not sure what would :)
I take it that when nrnivmodl is launched in the above folder, that there is no error (or if it has a copy of your external mod file, there would be no error if the copy of the external mod file was removed (and which file was the copy)).

If you send me a zip file of that folder (without the x86_64 or output subfolders), I'll try it out on my ubuntu virtualbox guest. The zip command would be ( executed in the parent folder of ``NMLCL000073-NEURON-C```)

zip -r NMLCL000073-NEURON-C -x x86_64 output

and please send to michael.hines@yale.edu .
This whole notion of a copy of an external mod file is confusing me. Perhaps you can send also a zip of one of those files that I can copy into NMLCL000073-NEURON-C

[nrnhines]

Perhaps it would be possible to avoid fruitless experiments on my side by having a zoom meeting with you so that I can see what is going on. We can arrange via email.

[nrnhines]

@kedoxey On my machine I see:

$ readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6|grep memcpy
...
  2825: 00000000000ba870    44 FUNC    GLOBAL DEFAULT   17 memcpy@GLIBC_2.2.5
  2827: 00000000000b1710   273 IFUNC   GLOBAL DEFAULT   17 memcpy@@GLIBC_2.14

I'm curious what you observe on your machine. The above matches well with the wheel version of libnrniv.so

~/tmp/env/lib64/python3.11/site-packages/neuron/.data/lib$ readelf  --dyn-syms -W libnrniv.so | grep memcpy
    33: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_memcpyRtWn
   138: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (16)

[kedoxey]

For these two commands I get

$ readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6|grep memcpy
  2262: 00000000000c4870    44 FUNC    GLOBAL DEFAULT   15 memcpy@GLIBC_2.2.5
  2264: 00000000000a9c10   289 IFUNC   GLOBAL DEFAULT   15 memcpy@@GLIBC_2.14

(python3_9-NEW) kedoxey@spike:~/.conda/envs/python3_9-NEW/lib/python3.9/site-packages/neuron/.data/lib$ readelf  --dyn-syms -W libnrniv.so | grep memcpy
    33: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_memcpyRtWn
   138: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (16)
   150: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_memcpyRnWt

[nrnhines]

I did not expect your libc.so.6 to have an entry for memcpy@@GLIBC_2.14! Recall during our zoom meeting that we were stymied by lack of symbols to see anything via nm -a. Now I wonder why the link step of nrnivmodl can't find the obviously existing memcpy for GLIBC_2.14. Please look again with ldd libnrniv.so to make sure it links against the /lib/x86_64-linux-gnu/libc.so.6 or if it refers to a library without the memcpy@@GLIBC_2.14

[kedoxey]

ldd libnrniv.so gives

kedoxey@spike:~/.conda/envs/python3_9-NEW/lib/python3.9/site-packages/neuron/.data/lib$ ldd libnrniv.so
        linux-vdso.so.1 (0x00007ffc1674c000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f86b0a78000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f86b0000000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f86b0319000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f86b0a58000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f86b0a53000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f86afc00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f86b0a96000)

I'm not sure if anything changed, but I was able to successfully compile my mod folder without the memcpy error after our meeting. I am going to contact a systems administrator for the server to look into it further.

[nrnhines]

I'm certainly confused by the variability exhibited by the error. On my machine, I actually have a lot of libc.so.6 instances and some do not have memcpy@@GLIBC_2.14 (the i386 versions) but those are likely unused. I wonder if your machine has some that are questionable. I.e.

for i in `find / -name libc.so.6 2> /dev/null` ; do echo $i ; readelf  --dyn-syms -W $i | grep 'memcpy.*2.14' ; done
/snap/core20/2264/usr/lib/i386-linux-gnu/libc.so.6
/snap/core20/2264/usr/lib/x86_64-linux-gnu/libc.so.6
  1197: 00000000000a0950   183 IFUNC   GLOBAL DEFAULT   15 memcpy@@GLIBC_2.14
/snap/core20/2318/usr/lib/i386-linux-gnu/libc.so.6
/snap/core20/2318/usr/lib/x86_64-linux-gnu/libc.so.6
  1197: 00000000000a0950   183 IFUNC   GLOBAL DEFAULT   15 memcpy@@GLIBC_2.14
/snap/core18/2812/lib/i386-linux-gnu/libc.so.6
/snap/core18/2812/lib/x86_64-linux-gnu/libc.so.6
  1179: 000000000009eec0   202 IFUNC   GLOBAL DEFAULT   13 memcpy@@GLIBC_2.14
/snap/core18/2829/lib/i386-linux-gnu/libc.so.6
/snap/core18/2829/lib/x86_64-linux-gnu/libc.so.6
  1179: 000000000009eec0   202 IFUNC   GLOBAL DEFAULT   13 memcpy@@GLIBC_2.14
/snap/mesa-2404/44/usr/lib/i386-linux-gnu/libc.so.6
/snap/snapd/21465/lib/x86_64-linux-gnu/libc.so.6
  1134: 0000000000094270   106 IFUNC   GLOBAL DEFAULT   13 memcpy@@GLIBC_2.14
/snap/snapd/21759/lib/x86_64-linux-gnu/libc.so.6
  1134: 0000000000094270   106 IFUNC   GLOBAL DEFAULT   13 memcpy@@GLIBC_2.14
/snap/core22/1380/usr/lib/i386-linux-gnu/libc.so.6
/snap/core22/1380/usr/lib/x86_64-linux-gnu/libc.so.6
  2264: 00000000000a9c10   289 IFUNC   GLOBAL DEFAULT   15 memcpy@@GLIBC_2.14
/snap/core24/423/usr/lib/i386-linux-gnu/libc.so.6
/snap/core24/423/usr/lib/x86_64-linux-gnu/libc.so.6
  2827: 00000000000b1710   273 IFUNC   GLOBAL DEFAULT   17 memcpy@@GLIBC_2.14
/usr/lib/x86_64-linux-gnu/libc.so.6
  2827: 00000000000b1710   273 IFUNC   GLOBAL DEFAULT   17 memcpy@@GLIBC_2.14

[nrnhines]

Thought I'd see what chatgpt has to say about the issue. We've already explored a few of its suggestions. But having the linker run in verbose mode seems like it might be usefully informative.

when linking with g++ on ubuntu, how can one determine the libraries used (in particular libc.so.6). I having trouble with nrnivmodl from a pip install neuron which quite variably exhibits the error ...x86_64-conda-linux-gnu/bin/ld: /home/kedoxey/.conda/envs/python3_9-NEW/lib/python3.9/site-packages/neuron/.data/lib/libnrniv.so: undefined reference to memcpy@GLIBC_2.14

When linking with g++ on Ubuntu, you can determine the libraries used by examining the linking command and using tools to inspect the binaries. The issue you're encountering with memcpy@GLIBC_2.14 suggests a mismatch between the version of the GNU C Library (glibc) expected by the libnrniv.so library and the version available on your system.

Here's how you can approach this:

1. Check the Linking Command

When g++ links a program, it usually outputs the linking command. You can capture this command by using verbose flags:

g++ -v ...

This will show you the detailed linking process, including the libraries being used.

2. Inspect Shared Library Dependencies

To check which libraries are linked by an executable or shared library, you can use the ldd command. This command lists the shared libraries required by a binary and their locations:

ldd /home/kedoxey/.conda/envs/python3_9-NEW/lib/python3.9/site-packages/neuron/.data/lib/libnrniv.so

If ldd shows that libc.so.6 (glibc) is linked but an undefined reference to memcpy@GLIBC_2.14 is present, it indicates a version mismatch. Ensure that the glibc version installed on your system meets or exceeds the version required by the library.

3. Check the GLIBC Version

To check the version of glibc installed on your system, use:

ldd --version

The version number is shown at the top of the output. For example, if the version is 2.31, then GLIBC_2.14 is included, but if it’s older, you might have issues with version-specific symbols.

4. Examine Symbol Versions in Libraries

You can use the objdump or readelf command to examine the symbol versions in a shared library. For example:

objdump -T /home/kedoxey/.conda/envs/python3_9-NEW/lib/python3.9/site-packages/neuron/.data/lib/libnrniv.so | grep memcpy

or

readelf -Ws /home/kedoxey/.conda/envs/python3_9-NEW/lib/python3.9/site-packages/neuron/.data/lib/libnrniv.so | grep memcpy

This will show you the versions of memcpy that libnrniv.so is expecting.

5. Verify Library Path Consistency

Ensure that the libraries in your conda environment are compatible with your system's glibc version. Conda environments sometimes use libraries that may not match the versions available on the system. You may need to update or reconfigure your conda environment.

6. Check for Version Compatibility

Ensure that the version of neuron installed via pip is compatible with your system’s libraries. If the package is compiled with a different version of glibc than what is available on your system, consider installing a version of the package that matches your environment's glibc.

7. Consider Conda-Specific Issues

If you’re using conda, it may be helpful to ensure that all dependencies are correctly aligned within your conda environment. Sometimes recreating the environment or using a different channel can resolve such issues.

By using these steps, you should be able to diagnose and potentially resolve the issue with undefined references related to memcpy@GLIBC_2.14. If problems persist, consider reaching out to the maintainers of the neuron package or consulting additional documentation for compatibility issues.

[nrnhines]

To explore the linking information I created a python virtual environment

mkdir tmp
cd tmp
python -m venv env
source env/bin/activate
pip install neuron

then modified the makefile associated with nrnivmodl

cd tmp/env/lib/python3.11/site-packages/neuron/.data/bin

and edited nrnmech_makefile and modified the relevant line by inserting a -v to read

CXX_LINK_SHARED = $(CXX) -v $(CXXFLAGS) -shared -fPIC

then

~/tmp/env/lib/python3.11/site-packages/neuron/.data/share/nrn/demo/release$ nrnivmodl

generates the output

/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/share/nrn/demo/release
Mod files: "./cabpump.mod" "./cachan1.mod" "./camchan.mod" "./capump.mod" "./invlfire.mod" "./khhchan.mod" "./mcna.mod" "./nacaex.mod" "./nachan.mod" "./release.mod"

 -> Compiling mod_func.cpp
 -> Compiling cabpump.c
 -> Compiling camchan.c
 -> Compiling capump.c
 -> Compiling khhchan.c
 -> Compiling nacaex.c
 -> Compiling nachan.c
 -> Compiling release.c
 => LINKING shared library ./libnrnmech.so
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-linux-gnu/13/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 13.2.0-23ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-13/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-13 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/libexec --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-libstdcxx-backtrace --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-13-uJ7kn6/gcc-13-13.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-13-uJ7kn6/gcc-13-13.2.0/debian/tmp-gcn/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 13.2.0 (Ubuntu 13.2.0-23ubuntu4) 
MAKEFLAGS= -j4  -- LinkCoreNEURON=false UserINCFLAGS= UserLDFLAGS= MODOBJFILES=./cabpump.o\ ./cachan1.o\ ./camchan.o\ ./capump.o\ ./invlfire.o\ ./khhchan.o\ ./mcna.o\ ./nacaex.o\ ./nachan.o\ ./release.o ROOT=/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data
COMPILER_PATH=/usr/libexec/gcc/x86_64-linux-gnu/13/:/usr/libexec/gcc/x86_64-linux-gnu/13/:/usr/libexec/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/13/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/13/:/usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/13/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/13/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-O2' '-Wno-write-strings' '-D' 'VERSION_INFO=8.2.6' '-std=c++11' '-shared' '-fPIC' '-I' '/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/include' '-o' './libnrnmech.so' '-L/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/lib' '-pthread' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' './libnrnmech.so.'
 /usr/libexec/gcc/x86_64-linux-gnu/13/collect2 -plugin /usr/libexec/gcc/x86_64-linux-gnu/13/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/x86_64-linux-gnu/13/lto-wrapper -plugin-opt=-fresolution=/tmp/ccl3eNxO.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -shared -z relro -o ./libnrnmech.so /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/13/crtbeginS.o -L/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/lib -L/usr/lib/gcc/x86_64-linux-gnu/13 -L/usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/13/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/13/../../.. -soname libnrnmech.so ./mod_func.o ./cabpump.o ./cachan1.o ./camchan.o ./capump.o ./invlfire.o ./khhchan.o ./mcna.o ./nacaex.o ./nachan.o ./release.o -lnrniv -rpath /home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/lib -lstdc++ -lm -lgcc_s -lpthread -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/13/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/crtn.o
COLLECT_GCC_OPTIONS='-v' '-O2' '-Wno-write-strings' '-D' 'VERSION_INFO=8.2.6' '-std=c++11' '-shared' '-fPIC' '-I' '/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/include' '-o' './libnrnmech.so' '-L/home/hines/tmp/env/lib/python3.11/site-packages/neuron/.data/lib' '-pthread' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-dumpdir' './libnrnmech.so.'
 => LINKING executable ./special LDFLAGS are:    -pthread
Successfully created x86_64/special

In my case, given -L/lib/x86_64-linux-gnu and -lc, that seems consistent with

~/tmp/env/lib/python3.11/site-packages/neuron/.data/share/nrn/demo/release$ ldd x86_64/libnrnmech.so
...
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6