Pip install compile error on OS X Sierra 10.12.5 with Anaconda 3.6

[jaanli]

I'm trying to run sudo pip install git+https://github.com/clab/dynet#egg=dynet but am hitting an error:

    creating build/temp.macosx-10.7-x86_64-3.6
    /usr/bin/gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/anaconda3/include -arch x86_64 -I/usr/local/anaconda3/include -arch x86_64 -I/tmp/pip-build-h95rs6ru/dynet -I/private/tmp/pip-build-h95rs6ru/dynet/build/py3.6-64bit/eigen -I/usr/local/anaconda3/include/python3.6m -c _dynet.cpp -o build/temp.macosx-10.7-x86_64-3.6/_dynet.o -std=c++11 -stdlib=libc++ -mmacosx-version-min=10.7
    _dynet.cpp:16528:27: warning: comparison of integers of different signs: 'Py_ssize_t' (aka 'long') and 'size_type' (aka 'unsigned long') [-Wsign-compare]
      __pyx_t_7 = ((__pyx_t_6 > __pyx_v_self->thisptr.get_storage().values.size()) != 0);
                    ~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    _dynet.cpp:71505:34: error: no member named 'restart' in 'dynet::Trainer'
              __pyx_v_self->thisptr->restart();
              ~~~~~~~~~~~~~~~~~~~~~  ^
    _dynet.cpp:71531:34: error: no member named 'restart' in 'dynet::Trainer'
              __pyx_v_self->thisptr->restart(__pyx_t_8);
              ~~~~~~~~~~~~~~~~~~~~~  ^
    1 warning and 2 errors generated.
    error: command '/usr/bin/gcc' failed with exit status 1
    make[2]: *** [python/timestamp] Error 1
    make[1]: *** [python/CMakeFiles/target.dir/all] Error 2
    make: *** [all] Error 2
    error: /usr/bin/make -j 8

    ----------------------------------------
Command "/usr/local/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-h95rs6ru/dynet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-a59pu6f8-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-h95rs6ru/dynet/

Full paste: https://pastebin.com/raw/P0TWDemW

[jaanli]

Updated to Sierra 10.12.6, same error.

[neubig]

Thanks, we'll take a look into this. In the mean time, hopefully manual install will work?

[neubig]

@altosaar I think this is likely fixed by #783 . Could you take a look?

[jaanli]

Thanks @neubig ! Did you test this with anaconda3?

I'm still getting the same error so I'm not sure if the pip package has been updated: https://pastebin.com/raw/uWg755Dy

[danielhers]

Does it help to remove /usr/local/anaconda3/include/dynet before installing?
The pip package is updated (it's taken from the current master branch) but it might be seeing the old installation on your computer, since build_ext happens before make install.

[jaanli]

That worked, thanks!

It looks like the version number is missing though, and importing doesn't work:

$ pip install git+https://github.com/clab/dynet#egg=dynet

Collecting dynet from git+https://github.com/clab/dynet#egg=dynet
  Cloning https://github.com/clab/dynet to /private/var/folders/qr/s_h2rhqj6kdg8qrj0zw2scnw0000gn/T/pip-build-cg12p66r/dynet
Requirement already satisfied: cython in /usr/local/anaconda3/lib/python3.6/site-packages (from dynet)
Requirement already satisfied: numpy in /usr/local/anaconda3/lib/python3.6/site-packages (from dynet)
Installing collected packages: dynet
  Running setup.py install for dynet ... done
Successfully installed dynet-0.0.0

# jaanaltosaar @ on git:master o [10:55:02]
$ ipython
Python 3.6.2 |Anaconda custom (x86_64)| (default, Jul 20 2017, 13:14:59)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import dynet as dy
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-01ab94540019> in <module>()
----> 1 import dynet as dy

/usr/local/anaconda3/lib/python3.6/site-packages/dynet.py in <module>()
     15     def print_graphviz(**kwarge):
     16         print("Run with --dynet-viz to get the visualization behavior.")
---> 17     from _dynet import *
     18
     19 __version__ = 2.0

ImportError: dlopen(/usr/local/anaconda3/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so, 2): Symbol not found: __ZN5dynet14vanilla_lstm_cERKNS_10ExpressionES2_
  Referenced from: /usr/local/anaconda3/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so
  Expected in: /usr/local/lib/libdynet.dylib
 in /usr/local/anaconda3/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so

[danielhers]

About the version number, see #752.
Please remove /usr/local/lib/libdynet.dylib and try importing again. It also remained from a previous install.

[jaanli]

Thanks -- just tried it, no dice

# jaanaltosaar @ dyn-209-2-211-250 in ~/tmp [14:57:23] C:1
$ rm -rf /usr/local/anaconda3/include/dynet

# jaanaltosaar @ dyn-209-2-211-250 in ~/tmp [14:57:26]
$ rm /usr/local/lib/libdynet.dylib

# jaanaltosaar @ dyn-209-2-211-250 in ~/tmp [14:57:28] C:1
$ pip install git+https://github.com/clab/dynet#egg=dynet

Collecting dynet from git+https://github.com/clab/dynet#egg=dynet
  Cloning https://github.com/clab/dynet to /private/var/folders/qr/s_h2rhqj6kdg8qrj0zw2scnw0000gn/T/pip-build-iy3tt040/dynet
Requirement already satisfied: cython in /usr/local/anaconda3/lib/python3.6/site-packages (from dynet)
Requirement already satisfied: numpy in /usr/local/anaconda3/lib/python3.6/site-packages (from dynet)
Installing collected packages: dynet
  Running setup.py install for dynet ... done
Successfully installed dynet-0.0.0

# jaanaltosaar @ dyn-209-2-211-250 in ~/tmp [14:59:32]
$ ipython
Python 3.6.2 |Anaconda custom (x86_64)| (default, Jul 20 2017, 13:14:59)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import dynet as dy
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-8cf3036352c6> in <module>()
----> 1 import dynet as dy

/usr/local/anaconda3/lib/python3.6/site-packages/dynet.py in <module>()
     15     def print_graphviz(**kwarge):
     16         print("Run with --dynet-viz to get the visualization behavior.")
---> 17     from _dynet import *
     18
     19 __version__ = 2.0

ImportError: dlopen(/usr/local/anaconda3/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so, 2): Library not loaded: libdynet.dylib
  Referenced from: /usr/local/anaconda3/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so
  Reason: image not found

[danielhers]

Well, now it's a different problem. It's not finding the wrong libdynet.dylib, but it's not finding the right one either. Setting DYLD_LIBRARY_PATH to the path where it was installed will fix this - I think it should be (in your ~/.bashrc):

export DYLD_LIBRARY_PATH=/usr/local/anaconda3/lib:$DYLD_LIBRARY_PATH

The documentation should be clearer about this (currently it only explains what to do in the manual installation case).

[jaanli]

Awesome, that worked -- thanks!

Steps I followed:

# add to .bashrc or .zshrc and source it
export DYLD_LIBRARY_PATH=/usr/local/anaconda3/lib:$DYLD_LIBRARY_PATH

pip install git+https://github.com/clab/dynet#egg=dynet