Move to manylinux_2_28 for aarch64, ppc64le builds. #1459
Conversation
|
Oh, is 2_28 already out? Sweet. From pypa/manylinux#1332 I understood it's not out yet. Yes, it should address #1396 indeed. |
|
Do I understand correctly that 2_28 is based on Centos rather than on Debian? If so maybe some build script should be adapted. Are you able to create a test build of these packages? Thank you! |
Valid point! I erroneously assumed it was Debian based but I should double check how to get installed. I’ll try to run this locally later today. |
|
😬 pypa/manylinux#1332 (comment) I’ll see how easy this is to get to work but might be a little early with this PR. 🤞 |
|
OK so I pushed an update which install There are some hiccups though when it comes to post-build steps and maturity of [root@746bbcededce psycopg2]# yum info libpq-devel
Last metadata expiration check: 1:11:53 ago on Tue 07 Jun 2022 12:56:30 PM UTC.
Installed Packages
Name : libpq-devel
Version : 13.5
Release : 1.el8
Architecture : x86_64
Size : 324 k
Source : libpq-13.5-1.el8.src.rpm
Repository : @System
From repo : appstream
Summary : Development files for building PostgreSQL client tools
URL : http://www.postgresql.org/
License : PostgreSQL
Description : The libpq package provides the essential shared library for any PostgreSQL
: client program or interface. You will need to install this package to build any
: package or any clients that need to connect to a PostgreSQL server.
Available Packages
Name : libpq-devel
Version : 13.5
Release : 1.el8
Architecture : i686
Size : 98 k
Source : libpq-13.5-1.el8.src.rpm
Repository : appstream
Summary : Development files for building PostgreSQL client tools
URL : http://www.postgresql.org/
License : PostgreSQL
Description : The libpq package provides the essential shared library for any PostgreSQL
: client program or interface. You will need to install this package to build any
: package or any clients that need to connect to a PostgreSQL server.
[root@746bbcededce psycopg2]# ./scripts/build/build_manylinux_2_28.sh
+++ dirname ./scripts/build/build_manylinux_2_28.sh
++ cd ./scripts/build
++ pwd
+ dir=/app/psycopg2/scripts/build
++ cd /app/psycopg2/scripts/build/../..
++ pwd
+ prjdir=/app/psycopg2
+ '[' '!' '' ']'
++ ls /opt/python/
+ PYVERS='cp310-cp310
cp36-cp36m
cp37-cp37m
cp38-cp38
cp39-cp39
pp37-pypy37_pp73
pp38-pypy38_pp73
pp39-pypy39_pp73'
++ grep -e '^PSYCOPG_VERSION' /app/psycopg2/setup.py
++ sed 's/.*'\''\(.*\)'\''/\1/'
+ version=2.9.3
+ distdir=/app/psycopg2/dist/psycopg2-2.9.3
+ [[ -n '' ]]
+ yum install -y libpq-devel
Last metadata expiration check: 0:03:33 ago on Tue 07 Jun 2022 12:56:30 PM UTC.
Package libpq-devel-13.5-1.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
+ for pyver in $PYVERS
+ pybin=/opt/python/cp310-cp310/bin
+ /opt/python/cp310-cp310/bin/pip wheel /app/psycopg2 -w /app/psycopg2/dist/
Processing /app/psycopg2
Preparing metadata (setup.py) ... done
Building wheels for collected packages: psycopg2
Building wheel for psycopg2 (setup.py) ... done
Created wheel for psycopg2: filename=psycopg2-2.9.3-cp310-cp310-linux_x86_64.whl size=490212 sha256=06a8cc19cc9ea2b80de2ccbbeb4796bd063267e1079ff4bc93130419a0c7942d
Stored in directory: /tmp/pip-ephem-wheel-cache-v_6xy8w6/wheels/97/0b/f4/b1e62973ad4dbf6950a3847822f9a8914263514139f5b3b069
Successfully built psycopg2
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/opt/_internal/cpython-3.10.4/bin/python -m pip install --upgrade pip' command.
+ for pyver in $PYVERS
+ pybin=/opt/python/cp36-cp36m/bin
+ /opt/python/cp36-cp36m/bin/pip wheel /app/psycopg2 -w /app/psycopg2/dist/
Processing /app/psycopg2
Preparing metadata (setup.py) ... done
Building wheels for collected packages: psycopg2
Building wheel for psycopg2 (setup.py) ... done
Created wheel for psycopg2: filename=psycopg2-2.9.3-cp36-cp36m-linux_x86_64.whl size=459292 sha256=f21fb7152b4ad7978cc1350b098c381a0cf313d4a4183ea180d2ef6b43ab7379
Stored in directory: /tmp/pip-ephem-wheel-cache-22a1au3p/wheels/87/3a/7c/5fa620050a23f8f66753c5ab5df28a023f7f4375d559cc65f9
Successfully built psycopg2
###############################################
#👉..(continues for all available Python versions. I do need to rerun for -binary)
#👉..(Haven't spun up with Postgres running to run tests but could try to get that to work)
#👉..(Post-build steps fail though, please see below)
###############################################
[root@746bbcededce psycopg2]# ls /opt/python/
cp310-cp310 cp36-cp36m cp37-cp37m cp38-cp38 cp39-cp39 pp37-pypy37_pp73 pp38-pypy38_pp73 pp39-pypy39_pp73
[root@746bbcededce psycopg2]# /opt/python/cp310-cp310/bin/python -c "import psycopg2; print(psycopg2.__libpq_version__)"
130005 # 🎉
[root@746bbcededce psycopg2]# uname -r
5.10.104-linuxkit
[root@746bbcededce psycopg2]# cat /etc/*release
AlmaLinux release 8.5 (Arctic Sphynx)
AlmaLinux release 8.5 (Arctic Sphynx)
NAME="AlmaLinux"
VERSION="8.5 (Arctic Sphynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.5 (Arctic Sphynx)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.5"
AlmaLinux release 8.5 (Arctic Sphynx)
AlmaLinux release 8.5 (Arctic Sphynx)
The build script doesn't get down to testing because it fails to run the [root@746bbcededce psycopg2]# ./scripts/build/build_manylinux_2_28.sh
#...END OF RUNNING BUILD SCRIPT ABOVE
+ wheel=/app/psycopg2/dist/psycopg2-2.9.3-cp310-cp310-linux_x86_64.whl
+ shift
+ which python
+ alias
+ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot python
+ eval declare -f
++ declare -f
/usr/bin/which: no python in (/usr/share/Modules/bin:/opt/rh/gcc-toolset-11/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
+ py=python3
++ mktemp -d
+ tmpdir=/tmp/tmp.BI2r8nv47J
+ trap 'rm -r /tmp/tmp.BI2r8nv47J' EXIT
+ cd /tmp/tmp.BI2r8nv47J
+ python3 -m zipfile -e /app/psycopg2/dist/psycopg2-2.9.3-cp310-cp310-linux_x86_64.whl .
/app/psycopg2/scripts/build/strip_wheel.sh: line 33: python3: command not found
+ rm -r /tmp/tmp.BI2r8nv47J
[root@746bbcededce psycopg2]# which python
/usr/bin/which: no python in (/usr/share/Modules/bin:/opt/rh/gcc-toolset-11/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
[root@746bbcededce psycopg2]# which python3
/usr/bin/which: no python3 in (/usr/share/Modules/bin:/opt/rh/gcc-toolset-11/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
EDIT: Looked at the wrong image name. There is a |
| @@ -1,6 +1,6 @@ | |||
| #!/bin/bash | |||
|
|
|||
| # Create manylinux_2_24 wheels for psycopg2 | |||
| # Create manylinux_2_28 wheels for psycopg2 | |||
| # | |||
| # Look at the .github/workflows/packages.yml file for hints about how to use it. | |||
|
|
|||
There was a problem hiding this comment.
Added the suggestion below which allowed strip_wheels to run and get to tests (and pass!)
| PYTHON_VERSION="cp38" | |
| pythonLocation=$(find /opt/python -maxdepth 1 -name "${PYTHON_VERSION}-*" -print -quit) | |
| export PATH="${pythonLocation}/bin:${PATH}" |
#...
+ for pyver in $PYVERS
+ pybin=/opt/python/cp39-cp39/bin
+ /opt/python/cp39-cp39/bin/pip install psycopg2-binary --no-index -f /app/psycopg2/dist/psycopg2-2.9.3
Looking in links: /app/psycopg2/dist/psycopg2-2.9.3
Processing ./dist/psycopg2-2.9.3/psycopg2_binary-2.9.3-cp39-cp39-manylinux_2_28_x86_64.whl
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.9.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ /opt/python/cp39-cp39/bin/python -c 'import psycopg2; print(psycopg2.__version__)'
2.9.3 (dt dec pq3 ext lo64)
+ /opt/python/cp39-cp39/bin/python -c 'import psycopg2; print(psycopg2.__libpq_version__)'
130005
+ /opt/python/cp39-cp39/bin/python -c 'import psycopg2; print(psycopg2.extensions.libpq_version())'
130005
+ [[ -n '' ]]
+ /opt/python/cp39-cp39/bin/python -c 'import tests; tests.unittest.main(defaultTest='\''tests.test_suite'\'')'
ss..................s.s.s..........ss.........ss.........s..ss.s...s.ssssssssssssssssss.................................ss................s.....s....s........s........s..s.s......s.s....................s.....s.......................................................................s.........................................................................................ss.......ss......................ss.s......................................s....sssss.sssssssssssss..................................................ssssssssss......................................................ss.....................................................s.........s.ssss.sssss...............s..............................................................
----------------------------------------------------------------------
Ran 738 tests in 4.848s
OK (skipped=96)
#...
is libpq 14 not available? It's an important version as it adds the pipeline mode. Also, can you check what libssl is installed? Thank you so far! |
Not out of the box but I can check how hard it would be to get a newer version. I’ll check the version of libssl too. |
[root@682be270549f psycopg2]# yum list installed | grep ssl
openssl.x86_64 1:1.1.1k-6.el8_5 @baseos
openssl-libs.x86_64 1:1.1.1k-6.el8_5 @System
[root@682be270549f psycopg2]# yum info openssl-libs
Last metadata expiration check: 3:11:30 ago on Tue 07 Jun 2022 08:44:34 PM UTC.
Installed Packages
Name : openssl-libs
Epoch : 1
Version : 1.1.1k
Release : 6.el8_5
Architecture : x86_64
Size : 3.6 M
Source : openssl-1.1.1k-6.el8_5.src.rpm
Repository : @System
Summary : A general purpose cryptography library with TLS implementation
URL : http://www.openssl.org/
License : OpenSSL and ASL 2.0
Description : OpenSSL is a toolkit for supporting cryptography. The openssl-libs
: package contains the libraries that are used by various applications which
: support cryptographic algorithms and protocols. |
| > /etc/apt/sources.list.d/pgdg.list | ||
| apt-get -y update | ||
| apt-get install -y libpq-dev | ||
| yum install -y libpq-devel |
There was a problem hiding this comment.
Per Postgres docs, to get to libpq 14.
This does switch to using libpq5-devel which I don't know what that does to the support matrix. Tests seem to pass though!
| yum install -y libpq-devel | |
| dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm | |
| dnf -qy module disable postgresql | |
| yum install -y libpq5-devel |
Built and tested:
+ for pyver in $PYVERS
+ pybin=/opt/python/cp310-cp310/bin
+ /opt/python/cp310-cp310/bin/pip install psycopg2-binary --no-index -f /app/psycopg2/dist/psycopg2-2.9.3
Looking in links: /app/psycopg2/dist/psycopg2-2.9.3
Processing ./dist/psycopg2-2.9.3/psycopg2_binary-2.9.3-cp310-cp310-manylinux_2_28_x86_64.whl
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.9.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ /opt/python/cp310-cp310/bin/python -c 'import psycopg2; print(psycopg2.__version__)'
2.9.3 (dt dec pq3 ext lo64)
+ /opt/python/cp310-cp310/bin/python -c 'import psycopg2; print(psycopg2.__libpq_version__)'
140003
+ /opt/python/cp310-cp310/bin/python -c 'import psycopg2; print(psycopg2.extensions.libpq_version())'
140003
+ [[ -n '' ]]
+ /opt/python/cp310-cp310/bin/python -c 'import tests; tests.unittest.main(defaultTest='\''tests.test_suite'\'')'
ss..................s.s.s..........ss.........ss.........s..ss.s...s.ssssssssssssssssss.................................ss................s.....s....s........s........s..s.s......s.s....................s.....s.......................................................................s.........................................................................................ss.......ss......................ss.s......................................s....sssss.sssssssssssss..................................................ssssssssss......................................................ss.....................................................s.........s.ssss.sssss...............s..............................................................
----------------------------------------------------------------------
Ran 738 tests in 5.230s
OK (skipped=96)
#...
|
@dvarrazzo Can you take a look at any of the feedback above to see if I should update my commits here? Also I assume #1482 is holding up this getting merged? |
How are these related? Manylinux wheels for aarch and osx-arm wheels have no overlap. |
|
Hello, I can't remember what I did for psycopg 3, but probably I will adopt something similar, involving building libpq from source and using manylinux2014 tag for maximum compatibility. |
My mistake @h-vetinari. For some reason I had believed that manylinux aarch was a listed compatibility tag for native macOS execution but I just checked again and it is in fact not listed. |
In that case would you like me to close this PR or attempt to port that work from psycopg 3? |
|
If you would like to work on it, sure |
|
Hi, it's been 3 months now from the last comment and the PR is still open. #1396 has been open for almost a year now, causing major pain for anyone trying to migrate to aarch64. can we please merge this? |
|
@gilbahat no, not in this form, as per last comments. We cannot use libpq from packages on centos, we need to build it from source. |
|
fwiw, pg v15 is available for pypa's For |
|
@ddelange thank you for the heads-up. However, the main problem with yum packages, is that the distro uses an outdated version of libssl. As a consequence we have to build it from source, and build the libpq as a consequence. Psycopg 3 does this (hopefully) right, so, the best thing to do is to use the psycopg 3 script for psycopg 2 too. |
|
thanks for the quick reply! makes sense :) cc @elprans |
|
|
Related to:
I believe this should be the only thing necessary to get to Debian 10 (Buster)
For what it's worth this MAY be able to collapse the CentOS builds from
manylinux2014tomanylinux_2_28as per the compatibility chart above states it supports a minimum of CentOS 8, although I don't want to trigger pitchforks: pypa/manylinux#1332 (comment)I also may be completely missing the issue here. 🤷 Feel free to decline with fervor.