Build and deploy Python wheels to PyPI

[danielhers]

• Test on various architectures and Python versions
  (Build for macOS, test)
  (Build for Windows, test)
  (GPU version)

[danielhers]

Trying this in a separate repository: https://github.com/danielhers/dynet-manylinux1

[neubig]

I'm thinking about fixing the Python 2.0.1 version today. It's OK to fix it now then build the wheel files afterwards, right?

[danielhers]

Yes, I don't expect any actual code changes required in order to be able to build it.

…

On Sat, Sep 2, 2017, 17:46 Graham Neubig ***@***.***> wrote: I'm thinking about fixing the Python 2.0.1 version today. It's OK to fix it now then build the wheel files afterwards, right? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#841 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQEw6cs9gn7ipAlAx2BnXH048tn5rhoks5seWpKgaJpZM4PIq1t> .

[danielhers]

Eventually I'd like to add a section to DyNet's .travis.yml to make it deploy to PyPI automatically whenever a tag is created.

[danielhers]

@neubig The automatic PyPI deployment (upon tag creation in clab/dynet) will require adding the PyPI username and (encrypted) password to .travis.yml. Do you want to use your or mine? To use yours, you can add the TWINE_USERNAME and TWINE_PASSWORD environment variables to the global environment in .travis.yml. The TWINE_PASSWORD variable should be encrypted in a clone of clab/dynet (not a fork).

[danielhers]

Regarding Windows and GPU, I don't know how to do that on Travis CI, so maybe we'll open another issue for that (in addition to #657, which I think can be closed once this is finished). I don't have access to a machine with a GPU and the ability to install Docker on (which I need for building the Linux wheels), but maybe they can be built on a machine with CUDA even if it doesn't have a GPU.

[danielhers]

Turns out march=native does pose a problem (the wheels are created successfully but may cause Illegal Instructions when run on a different machine), so I'm removing it from CMakeLists.txt using sed right before creating the wheel.

[danielhers]

The Linux wheels now work, at least on the machines I've tested them on. I'm not sure about the macOS wheels, though, as I don't have a Mac to try on. It looks like delocate doesn't actually put the shared library into the wheel, although auditwheel works. This might be the same problem as matthew-brett/delocate#15 - with DyNet it's a bit complicated but I think it's also just a Python file at the top rather than a package (there's no __init__.py). It might be necessary to change the structure a bit to make the wheels work on macOS too. Or maybe they work already, I don't know - it would be nice if someone tried them (pip install dynet==2.0.1.post5).

[neubig]

First, this is absolutely awesome that we will be able to install DyNet without compiling. I really appreciate this :)

That being said, it doesn't look like it works on Mac:

(dynet-throwaway) itachi:~ neubig$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dynet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/neubig/anaconda/envs/dynet-throwaway/lib/python3.6/site-packages/dynet.py", line 15, in <module>
    from _dynet import *
ImportError: dlopen(/Users/neubig/anaconda/envs/dynet-throwaway/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so, 2): Library not loaded: libdynet.dylib
  Referenced from: /Users/neubig/anaconda/envs/dynet-throwaway/lib/python3.6/site-packages/_dynet.cpython-36m-darwin.so
  Reason: image not found
>>> 

The work-around here doesn't seem to be working either, as it doesn't look like libdynet.dylib is installed at all #146

[danielhers]

Yes, #146 is just what I was afraid of; that delocate isn't really putting libdynet.dylib into the wheel. As I said in my previous comment, I think this has to do with the directory structure of the DyNet Python files, and the fact that they're not in a proper package.
Maybe we could use wheels for Linux and keep using the .tar.gz source distribution for macOS and Windows, and fix this in a future PR.
If you agree, I'll add a commit avoiding the wheel upload for osx. Please also let me know what you prefer regarding the twine (PyPI) credentials.

[danielhers]

For now I removed the macOS wheels from PyPI, so if your OS doesn't have a wheel, you'll just get the source distribution and compile it automatically.

[neubig]

If this might be working on Mac I can try it one more time.

[danielhers]

It's working, but precompiled wheels are just for Linux for now (for Mac you'll get the source distribution).

[danielhers]

So, I opened issues for the remaining tasks related to this, but I think the current PR is ready and useful:

• Refactored .travis.yml to make it easier to modify and add parts.
• Automatic deployment to PyPI on new tags in clab/dynet.
• Precompiled wheels for Linux, tested in multiple architectures. Other OSs still get the sdist as before.
• Groundwork for macOS precompiled wheels.