Building VW statically with Clang does not work

[lokitoth]

From #1691, @arielf found a regression in Linux build post CMAKE-changes

Hi,

I'd like to reopen this issue. Basically, we have a regression.

Since the switch to cmake, I have trouble building vw statically with clang, on Ubuntu 18.04 by following the (new) instructions.

Starting with a fresh clone:

% git clone git://github.com/VowpalWabbit/vowpal_wabbit.git vwclean
Cloning into 'vwclean'...
remote: Enumerating objects: 91, done.
remote: Counting objects: 100% (91/91), done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 57865 (delta 35), reused 51 (delta 25), pack-reused 57774
Receiving objects: 100% (57865/57865), 120.07 MiB | 10.73 MiB/s, done.
Resolving deltas: 100% (43117/43117), done.

Proceeding with build according to instructions:

% cd vwclean
% mkdir build
% cd build
% export CC=clang CXX=clang++
% cmake ..  -DCMAKE_BUILD_TYPE=Release  -DSTATIC_LINK_VW=On
-- VowpalWabbit Version: 8.6.1
-- Number of processors: 8
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   program_options
--   system
--   thread
--   unit_test_framework
--   chrono
--   date_time
--   atomic
-- Configuring done
-- Generating done
-- Build files have been written to: ~/src/vw/vwclean/build

% make
[... long output  trimmed, compilation works, but final linking fails ...]

[100%] Linking CXX executable vw-unit-test.out
CMakeFiles/vw-unit-test.out.dir/main.cc.o: In function `main':
/usr/include/boost/test/unit_test.hpp:63: undefined reference to `boost::unit_test::unit_test_main(bool (*)(), int, char**)'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
test/unit_test/CMakeFiles/vw-unit-test.out.dir/build.make:230: recipe for target 'test/unit_test/vw-unit-test.out' failed
make[2]: *** [test/unit_test/vw-unit-test.out] Error 1
CMakeFiles/Makefile2:821: recipe for target 'test/unit_test/CMakeFiles/vw-unit-test.out.dir/all' failed
make[1]: *** [test/unit_test/CMakeFiles/vw-unit-test.out.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

It is possible that LD=clang++ should be added to environment (?)

Interestingly, going one level up (not exactly following the instructions), seems to work slightly better:

% cd ..
% make
mkdir -p build
cd build; cmake ..
-- VowpalWabbit Version: 8.6.1
-- Number of processors: 8
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   program_options
--   system
--   thread
--   unit_test_framework
--   chrono
--   date_time
--   atomic
-- Configuring done
-- Generating done
-- Build files have been written to: ~/src/vw/vwclean/build
cd build; make -j8 vw-bin
make[1]: Entering directory '~/src/vw/vwclean/build'
make[2]: Entering directory '~/src/vw/vwclean/build'
make[3]: Entering directory '~/src/vw/vwclean/build'
make[4]: Entering directory '~/src/vw/vwclean/build'
make[4]: Leaving directory '~/src/vw/vwclean/build'
[  4%] Built target allreduce
make[4]: Entering directory '~/src/vw/vwclean/build'
make[4]: Leaving directory '~/src/vw/vwclean/build'
[ 97%] Built target vw
make[4]: Entering directory '~/src/vw/vwclean/build'
make[4]: Leaving directory '~/src/vw/vwclean/build'
[100%] Built target vw-bin
...

Now we have a vw executable, but it crashes:

% ./build/vowpalwabbit/vw
Command terminated by signal 11

And make test is failing 5 of 5 tests.

Another related/new issue which didn't exist pre the cmake switch, is that cmake overwrites the original Makefiles so now running git bisect run refuses to check-out the bisected commit because that would overwrite modified files.

To summarize, since the cmake switch we have:

• Possible missing LD setting
• Inability to build statically
• When linked manually, executable crashes
• Instructions not working as they used to
• git bisect run now fails to checkout Makefiles

I'd love to get to the prior state before all the massive changes when I build statically with clang and everything just works. Can someone help? Thanks!

This seems like a big enough problem that it should be tracked under its own issue.

[jackgerrits]

I've got a fix for the linker error you mentioned. For the seg fault it seems to be caused by the thread library being statically linked, so I am not yet sure how to fix this.

Thread 1 "vw" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000000000046f75d in __gthread_equal (__t1=0, __t2=0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/x86_64-linux-gnu/c++/5.4.0/bits/gthr-default.h:680
#2  0x000000000047b139 in std::operator== (__x=..., __y=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/thread:84
#3  0x000000000047b110 in std::thread::joinable (this=0xc46b00)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/thread:170
#4  0x0000000000595250 in std::thread::operator=(std::thread&&) (this=0xc46b00,
    __t=<unknown type in /mnt/c/w/repos/vowpal_wabbit_jagerrit/build/vowpalwabbit/vw, CU 0x299397, DIE 0x2bb855>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/thread:158
#5  0x0000000000594204 in VW::start_parser (all=...)
    at /mnt/c/w/repos/vowpal_wabbit_jagerrit/vowpalwabbit/parser.cc:1049
#6  0x0000000000404810 in main (argc=1, argv=0x7ffffffedd18)
    at /mnt/c/w/repos/vowpal_wabbit_jagerrit/vowpalwabbit/main.cc:124

[jackgerrits]

The second issue was to do with linking pthreads as static, it requires special linker flags in order to retain the symbols. I've submitted a fix for this. This fix may not work on other platforms but I have confirmed it works on Ubuntu with GCC and Clang.