Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Manylinux2014 -> Manylinux_2_28 and update deprecated actions #1228

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ptheywood
Copy link
Member

@ptheywood ptheywood commented Sep 4, 2024

Node 16 Github actions are deprecated, but node 20 actions are incompatible with Manylinux_2014/CentOS7.

  • Replaces manylinux wheel building to use 2_28 (based on Alma 8)
  • Updates depreacted actions in all workflows.

Produced wheels are still not technically Manylinux compliant due to linkage against non-packaged .so files (see #647)

This will reduce wheel compatibility, by increasing the required glibc to 2.28 or newer.

Closes #1190
Closes #1224

Replaces #1200 and #1194 as they must be coupled.

Todo

  • Test the built wheels on recent linux
  • Test installation on Centos7 to check the error (Liberty Linux 7 is still supported)

The use of download_artifact will not have been tested by CI, as it is only used if a version tag is created. We might want to do a dry run on a fork.

@ptheywood ptheywood changed the title CI: Manylinux2014 -Manylinux_2_28 and update deprecated CI: Manylinux2014 -> Manylinux_2_28 and update deprecated actions Sep 4, 2024
@ptheywood ptheywood force-pushed the manylinux_2_28-actions-node-20 branch 2 times, most recently from ca0c56f to 943e244 Compare September 4, 2024 16:53
@ptheywood
Copy link
Member Author

Wheels generated by the Manylinux_2_28 wheel seem to be missing some wrapped methods. Will need to investiagate when a bit more when on a machine with a GPU.

AttributeError: 'MessageSpatial3D_Description' object has no attribute 'newVariableID'. Did you mean: 'newVariableInt'?

@ptheywood
Copy link
Member Author

Wheels generated by the Manylinux_2_28 wheel seem to be missing some wrapped methods. Will need to investiagate when a bit more when on a machine with a GPU.

AttributeError: 'MessageSpatial3D_Description' object has no attribute 'newVariableID'. Did you mean: 'newVariableInt'?

This will be due to swig 4.2.1 being included in the manylinux_2_28 containers. Could downgrade (and block swig 4.2?), but it would be better to fix the root cause (i.e. make pyflamegpu swig 4.2 compatible).

See #956

…ithub actions

Node 16 Github actions are deprecated, but node 20 actions are incompatible with Manylinux_2014/CentOS7.

- Replaces manylinux wheel building to use 2_28 (based on Alma 8)
- Updates depreacted actions in all workflows.
@ptheywood ptheywood force-pushed the manylinux_2_28-actions-node-20 branch from 42f3959 to 9d234fd Compare February 17, 2025 17:54
@ptheywood
Copy link
Member Author

When installing the wheels produced by the ManyLinux_2_28 image, they are still installable with incompatible glibc, but error at runtime:

  File "path/to/FLAMEGPU2/examples/python_rtc/boids_spatial3D_bounded/boids_spatial3D.py", line 2, in <module>
    from pyflamegpu import *
  File "path/to/venv/lib/python3.10/site-packages/pyflamegpu/__init__.py", line 36, in <module>
    from .pyflamegpu import *
  File "path/to/venv/lib/python3.10/site-packages/pyflamegpu/pyflamegpu.py", line 13, in <module>
    from . import _pyflamegpu
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by path/to/venv/lib/python3.10/site-packages/pyflamegpu/_pyflamegpu.so)

(I believe) this is due to the wheel name being pyflamegpu-2.0.0rc3+cuda120-cp310-cp310-linux_x86_64.whl, i.e. it ends with linux_x86_64 not manylinux_2_28_x86_64, so pip does not understand the incompatibility.
Similar errors would have been encountered prior to this change, if older glibc than provided in manylinux2014 were used.

Our wheels are not strictly manylinux compliant (see #647), so I didn't enable the use of auditwheel repair in CI, so we arent' falsely claiming manylinux compatibility (/ I think it used to error?)

After manually running a repair on an individual wheel (which ran with no error code):

$ auditwheel repair /tmp/pyflamegpu-2.0.0rc3+cuda120-cp310-cp310-linux_x86_64.whl --plat manylinux_2_28_x86_64
INFO:auditwheel.main_repair:Repairing pyflamegpu-2.0.0rc3+cuda120-cp310-cp310-linux_x86_64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: manylinux_2_28_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp310-cp310-linux_x86_64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp310-cp310-manylinux_2_28_x86_64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /tmp/wheelhouse/pyflamegpu-2.0.0rc3+cuda120-cp310-cp310-manylinux_2_28_x86_64.whl

attempting to install this wheel on CentOS7 errors:

ERROR: pyflamegpu-2.0.0rc3+cuda120-cp310-cp310-manylinux_2_28_x86_64.whl is not a supported wheel on this platform.

However locally repairing the wheel with auditwheel repair packages libcuda.so and libnvrtc into the wheel, which we cannot do for licencing reasons.


Given this is an improvement (i.e. fixes CI) we should probably merge this as is, and just ensure that in our next / future release notes and probably in the wheelhouse state we require glibc >= 2_28 + raise #647 as a priority for a better pyflamegpu installation process

(with nvidia providing cuda packages on pip now, we might be able to lean into that a little, though we might have to sitch to cuda 11.6+ iirc?)

@ptheywood ptheywood requested review from Robadob and mondus February 18, 2025 12:20
@ptheywood ptheywood marked this pull request as ready for review February 18, 2025 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replace manylinux2014 with manylinux_2_28 CI Github Actions deprecations
1 participant