Pythonwheels.com publications needed

[paul-hammant]

Watchdog more than many has a finickity build per platform it can run on.

Can we publish binary wheels to https://pythonwheels.com, please?

[nsoranzo]

Having ready-to-install wheels on PyPI would indeed be great.

[BoboTiG]

On Windows and GNU/Linux it will be easy.
We have some work for macOS as XCode is involved and we may have to publish one version of watchdog by macOS version. And this is the hard part.

[di]

Hi, I was recently bit by #689 and would like to work on having this project provide wheels for manylinux, macos and windows. Can a maintainer give me an outline of the current release process?

This would likely require using GitHub Actions as well as https://github.com/pypa/gh-action-pypi-publish.

[BoboTiG]

That would be awesome @di 👍 The current release process is quite simple/naive:

python setup.py sdist
twine upload dist/*

Wheels for GNU/Linux and Windows should be the easy part. The macOS one requires compilation of the C file and it is handled into setup.py. I would be happy to give you all the answers you need to move forward :)

[di]

OK, I took a stab at this: 78a09ed

You can see an example run here: https://github.com/di/watchdog/actions/runs/427357496 (the run fails because PyPI credentials aren't set, so the uploads fail).

This produces the following distributions:

• watchdog-1.0.2-cp36-cp36m-macosx_10_14_x86_64.whl
• watchdog-1.0.2-cp37-cp37m-macosx_10_14_x86_64.whl
• watchdog-1.0.2-cp38-cp38-macosx_10_14_x86_64.whl
• watchdog-1.0.2-cp39-cp39-macosx_10_14_x86_64.whl
• watchdog-1.0.2-py3-none-manylinux2014.whl
• watchdog-1.0.2-py3-none-win_amd64.whl
• watchdog-1.0.2.tar.gz

One thing that was challenging is that the tests seem fairly flakey -- it took quite a number of runs to get them all to pass. Is this expected?

[BoboTiG]

Yes, tests are quite flaky, there is definitely room for improvements.

Wow this is absolutely splendid! :)
Given the wheel names, does that mean the support for macOS 10.9-10.13 would be lost? Same for Windows 32 bit 🤔
But this is a very big step as you also moved tests, thanks a lot! (Travis is messing with OSS and test time is soooo long)

[di]

Given the wheel names, does that mean the support for macOS 10.9-10.13 would be lost?

Great point, I had missed that it's using such a modern version of the SDK. Looks like this is due to actions/setup-python#26. I've updated it to use https://github.com/joerick/cibuildwheel so it now builds the following macOS distributions (which are forwards-compatible):

• watchdog-1.0.2-cp36-cp36m-macosx_10_9_x86_64.whl
• watchdog-1.0.2-cp37-cp37m-macosx_10_9_x86_64.whl
• watchdog-1.0.2-cp38-cp38-macosx_10_9_x86_64.whl
• watchdog-1.0.2-cp39-cp39-macosx_10_9_x86_64.whl
• watchdog-1.0.2-pp36-pypy36_pp73-macosx_10_9_x86_64.whl
• watchdog-1.0.2-pp37-pypy37_pp73-macosx_10_9_x86_64.whl

Same for Windows 32 bit

Sorry, I thought it was clear: the Windows and Linux wheels are actually pure-Python, so we can just rename them to make them available to all Windows/Linux platforms. I've gone ahead and done that, the build now produces these distributions as well:

• watchdog-1.0.2-py3-none-manylinux2014_aarch64.whl
• watchdog-1.0.2-py3-none-manylinux2014_armv7l.whl
• watchdog-1.0.2-py3-none-manylinux2014_i686.whl
• watchdog-1.0.2-py3-none-manylinux2014_ppc64.whl
• watchdog-1.0.2-py3-none-manylinux2014_ppc64le.whl
• watchdog-1.0.2-py3-none-manylinux2014_s390x.whl
• watchdog-1.0.2-py3-none-manylinux2014_x86_64.whl
• watchdog-1.0.2-py3-none-win32.whl
• watchdog-1.0.2-py3-none-win_amd64.whl
• watchdog-1.0.2-py3-none-win_ia64.whl

As well as:

• watchdog-1.0.2.tar.gz

You can see the full run here: https://github.com/di/watchdog/actions/runs/429458829

But this is a very big step as you also moved tests, thanks a lot! (Travis is messing with OSS and test time is soooo long)

I'm not sure I ported them 1:1 (e.g. PyPy tests are missing), I mostly just wanted to have some confidence that broken releases weren't being made. But I think it's fairly close.

I don't think you want to build all distributions and try to upload them as part of your CI though. There's a couple options where this workflow could be run:

• On push to master
• As a manual trigger

Which would you prefer?

[BoboTiG]

A manual trigger is required IMO. No need to automate too much those steps as I do not plan to do continuous deployment ;)

Out of curiosity, instead of renaming wheels for Windows and Linux, does something like generic wheel name could be use? I am not sure at all though.

[BoboTiG]

BTW go ahead and open a PR with your changes. Just add a changelog entry and we will be good :)

[di]

A manual trigger is required IMO. No need to automate too much those steps as I do not plan to do continuous deployment ;)

Sounds good.

Out of curiosity, instead of renaming wheels for Windows and Linux, does something like generic wheel name could be use? I am not sure at all though.

Yes, by default this produces a pure Python wheel, watchdog-1.0.2-py3-none-any.whl on Linux or Windows which is generic and would work on both those platforms.

The issue is that publishing a *-none-any.whl distribution would cause pip on macOS to fall back on that wheel if none of the macOS wheels match (for example, if the user is trying to install from Python 3.10 in the future and this project hasn't started releasing 3.10 wheels for macos).

I'm not entirely sure what the macOS-specific parts of this library are, but I'm assuming it would cause problems if they were unavailable, therefore as long as this project has macOS-specific parts, it should never publish a *-none-any.whl distribution.

BTW go ahead and open a PR with your changes. Just add a changelog entry and we will be good :)

I made a PR at #739. I'm not sure what you want me to put in the changelog, since I would recommend running this workflow as soon as it's merged to add wheels retroactively for the 1.0.2 release that's already on PyPI.

[BoboTiG]

Thanks a lot!