Skip to content

Commit

Permalink
Update notes and scripts for building wheels
Browse files Browse the repository at this point in the history
Wheels for Python 3.7 to 3.9 can be built using manylinux2010, as
there are binary wheels of oldest-supported-numpy available for these
versions. For 3.10 onwards manylinux2014 is required for numpy binary
wheels. Building numpy from source in the manylinux containers is not
viable.

There are now two scripts, one for each manylinux version, which will
build wheels for all upstream-supported Python versions. These can be
run as-is: info is in the first few lines of these files. Also, the
dev notes on building wheels wtih manylinux have been updated to
reference these files.
  • Loading branch information
jacklovell committed Dec 15, 2023
1 parent d14d85c commit 2b377c4
Show file tree
Hide file tree
Showing 4 changed files with 70 additions and 38 deletions.
34 changes: 0 additions & 34 deletions dev/build_wheels.sh

This file was deleted.

25 changes: 25 additions & 0 deletions dev/build_wheels_manylinux2010.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash
# To be run from within manylinux2010 docker container.
# Run the command below from the root of the source folder:
# sudo docker run -ti -v $(pwd):/io quay.io/pypa/manylinux2010_x86_64 ./dev/build_wheels_manylinux2010.sh
# Or, to use singularity instead of docker (e.g. on HPC, or where root not available):
# singularity run -B $(pwd):/io -W /tmp -c docker://quay.io/pypa/manylinux2010_x86_64 /io/dev/build_wheels_manylinux2010.sh

set -e
cd /io || exit
VERSION=$(cat raysect/VERSION)
PLAT=manylinux2010_x86_64

# Numpy provides manylinux2010 wheels only for Python up to 3.9.

# python 3.7
/opt/python/cp37-cp37m/bin/python -m build .
auditwheel repair dist/raysect-$VERSION-cp37-cp37m-linux_x86_64.whl --plat $PLAT

# python 3.8
/opt/python/cp38-cp38/bin/python -m build .
auditwheel repair dist/raysect-$VERSION-cp38-cp38-linux_x86_64.whl --plat $PLAT

# python 3.9
/opt/python/cp39-cp39/bin/python -m build .
auditwheel repair dist/raysect-$VERSION-cp39-cp39-linux_x86_64.whl --plat $PLAT
26 changes: 26 additions & 0 deletions dev/build_wheels_manylinux2014.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash
# To be run from within manylinux2014 docker container.
# Run the command below from the root of the source folder:
# sudo docker run -ti -v $(pwd):/io quay.io/pypa/manylinux2014_x86_64 ./dev/build_wheels_manylinux2014.sh
# Or, to use singularity instead of docker (e.g. on HPC, or where root not available):
# singularity run -B $(pwd):/io -W /tmp -c docker://quay.io/pypa/manylinux2014_x86_64 /io/dev/build_wheels_manylinux2014.sh

set -e
cd /io || exit
VERSION=$(cat raysect/VERSION)
PLAT=manylinux2014_x86_64

# Numpy provides manylinux2010 wheels for Python up to 3.9. So only need manylinux2014
# wheels for 3.10 onwards

# python 3.10
/opt/python/cp310-cp310/bin/python -m build .
auditwheel repair dist/raysect-$VERSION-cp310-cp310-linux_x86_64.whl --plat $PLAT

# python 3.11
/opt/python/cp311-cp311/bin/python -m build .
auditwheel repair dist/raysect-$VERSION-cp311-cp311-linux_x86_64.whl --plat $PLAT

# python 3.12
/opt/python/cp312-cp312/bin/python -m build .
auditwheel repair dist/raysect-$VERSION-cp312-cp312-linux_x86_64.whl --plat $PLAT
23 changes: 19 additions & 4 deletions dev/notes/building_bdist_with_manylinux.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,29 @@ Start the manylinux docker container (this will download the container, mount th
This will drop you into the manylinux container terminal. The /opt/python folder in the container holds the various versions of python. This example targets python 3.7. Adjust the paths as appropriate to build bdists for different python versions.

cd io/source
/opt/python/cp37-cp37m/bin/python -m pip install cython numpy==1.14.6
/opt/python/cp37-cp37m/bin/python setup.py bdist_wheel
/opt/python/cp37-cp37m/bin/python -m pip build .
auditwheel repair dist/raysect-0.8.1-cp37-cp37m-linux_x86_64.whl --plat manylinux2010_x86_64

This will compile the wheel and repair any library references to produce the manylinux wheel files in a folder ./wheelhouse. e.g. raysect-0.8.1-cp37-cp37m-manylinux1_x86_64.whl and raysect-0.8.1-cp37-cp37m-manylinux2010_x86_64.whl.

These can then be uploaded to pypi (just the manylinux2010 packages for now).
These can then be uploaded to pypi.

For more info see:
https://realpython.com/python-wheels/#building-a-platform-wheel-macos-and-windows
https://uwekorn.com/2019/09/15/how-we-build-apache-arrows-manylinux-wheels.html
https://uwekorn.com/2019/09/15/how-we-build-apache-arrows-manylinux-wheels.html

Scripts to semi-automate the wheel building process
===================================================

There are two scripts which automate this for different versions of Python:
- dev/build_wheels_manylinux2010.sh builds manylinux2010 wheels for Python 3.7, 3.8 and 3.9
- dev/build_wheels_manylinux2014.sh builds manylinux2014 wheels for Python 3.10, 3.11 and 3.12

These can be used to produce wheels in a semi-automated fashion:
1. Install docker as above
2. Clone the raysect/source repository
3. Change into the top level of the raysect/source repository.
4. Run `sudo docker run -ti -v $(pwd):/io quay.io/pypa/manylinux2010_x86_64 /io/dev/build_wheels_manylinux2010.sh`
5. Run `sudo docker run -ti -v $(pwd):/io quay.io/pypa/manylinux2014_x86_64 /io/dev/build_wheels_manylinux2014.sh`
6. Upload the wheels in wheelhouse/, and the sdist in dist/, to PyPI using twine.

0 comments on commit 2b377c4

Please sign in to comment.