Precompiled wheel

[danielhers]

Installation of the master branch with pip is currently supported by pip install git+https://github.com/clab/dynet#egg=dynet.

To support installation by a simple pip install dynet (and versioning!), we need to upload a binary distribution package to PyPI. I think it would be a good idea to upload a precompiled .whl file, to reduce installation time significantly. It must be a separate file for Linux, OSX and Windows, compiled on the respective systems. I think this should be done once there is a 2.0 release.

However, I'm pretty sure it won't work with the current configuration where there is a main setup.py calling the python/setup.py which is generated from setup.py.in, because this way the main one doesn't know about the Cython extensions, and they are missing from the created wheel. Therefore, I believe the two setup.py files should be combined into a single static one, maybe replacing the CMake ${...} directives by environment variables or parameters.

[neubig]

Your assessment sounds correct, so if you could do something like that it would be greatly appreciated!

[neubig]

I've made the actual v2.0 release here, so I think it'd be great to revisit this:
https://github.com/clab/dynet/releases/tag/v2.0

[danielhers]

Great news! For now I uploaded the source distribution to PyPI, so v2.0 can already be installed by pip install dynet, but it will have to compile everything.

[danielhers]

Once #783 is merged, this should be possible by

pip install auditwheel
python setup.py bdist_wheel
setenv LD_LIBRARY_PATH $PWD/.virtualenv/lib
auditwheel repair dist/dyNET-0.0.0-cp35-cp35m-linux_x86_64.whl

auditwheel puts the libdynet.so into the wheel and fixes RPATH.

[danielhers]

When I'm trying this, I'm getting:

auditwheel: error: cannot repair "dist/dyNET-0.0.0-cp35-cp35m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

Waiting for pypa/auditwheel#36 to be fixed so I know what to change in my environment.
Also, waiting for the next DyNet release before uploading to PyPI again.

[danielhers]

Using the Docker image recommended by auditwheel, I was able to build the wheel:

docker run -i -t -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /bin/bash
wget https://bitbucket.org/eigen/eigen/get/51c95eb5952b.zip
unzip 51c95eb5952b.zip
mv eigen* eigen
export EIGEN3_INCLUDE_DIR=$PWD/eigen
export MINICONDA_OS_NAME=Linux
export PYNUM=3
wget https://repo.continuum.io/miniconda/Miniconda$PYNUM-latest-$MINICONDA_OS_NAME-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
export PATH="$HOME/miniconda/bin:$PATH"
pip install cython auditwheel
git clone https://github.com/clab/dynet
cd dynet
export CMAKE=cmake28
python setup.py bdist_wheel
export LD_LIBRARY_PATH=$HOME/miniconda/lib
auditwheel repair dist/dyNET-0.0.0-cp36-cp36m-linux_x86_64.whl

I uploaded this to PyPI as version 2.0.1.dev1 (so it won't be automatically considered the latest stable version when installing with pip). To try it, run using Python 2.7 or 3.6 on a Linux machine:

pip install --no-cache-dir --pre dynet

[neubig]

Awesome, this is excellent! A couple things:

1. Dp you know if there is a way to get it working on Mac OS?
2. DyNet compiles with -march=native which compiles things for a specific processor architecture. There's a chance that things will break on other architectures, so we might need to test this.
3. We could probably also compile with GPU support, and create a dynet-gpu package right?

(No need to handle all of these at once, just asking for clarification about possibilities.)

[danielhers]

1. I suppose it should work for macOS too, but haven't tried it or checked about auditwheel support for it.
2. The wheels I uploaded should be tested on different architectures to see if it works. If not, it will be necessary to compile arch-specific wheels (don't know how to do that yet).
3. Yes, we can add a separate package for GPU. Should check what Tensorflow does.

If this works well, I'll upload an official one for the next release.

[pmichel31415]

I think tensorflow has a tensorflow and tensorflow-gpu package iirc

[danielhers]

By the way, is there a date planned for the next release? :)

[neubig]

I was resolving bugs that popped up due to the introduction of multi-device support, but I think we're almost finished, so maybe this weekend?

[neubig]

OK, release is done!

[danielhers]

Great! Working on the wheels.

[danielhers]

Uploaded 2.0.1 to PyPI, pre-compiled for Linux (32/64 bit) and macOS (64 bit), for Python 2.7, 3.4, 3.5 and 3.6.