python3: Update to 3.11.3, refresh/redo patches #21030
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Changes:
|
|
Changes:
|
jefferyto
force-pushed
the
python-3.11.3
branch
2 times, most recently
from
bdbb8f9 to
e1ac7eb
Compare
|
Changes:
Just to add: fixes are in separate commits so that they can be cherry-picked into 22.03 when I do the python3 package update after this PR. |
|
Changes:
|
commodo
approved these changes
May 22, 2023
|
Marking this PR as draft as I think I need to fit in one more fix... I believe there is a Python/libressl issue that is causing the hashlib module to not build for host Python (when building from full buildroot, not sdk), which is causing poetry-core to fail to build, causing the issue mentioned in #20994 |
|
Changes:
|
jefferyto
force-pushed
the
python-3.11.3
branch
2 times, most recently
from
30b90b7 to
2ac9736
Compare
--without-pymalloc was added in 7bf1ae6 because leaving it enabled added an "m" flag/suffix to file names. This flag/suffix was removed in Python 3.8[1], so disabling pymalloc is no longer necessary. [1]: https://docs.python.org/3.8/whatsnew/3.8.html#build-and-c-api-changes Signed-off-by: Jeffery To <jeffery.to@gmail.com>
003-do-not-run-distutils-tests.patch was removed in 4e05541. This patch stopped "make install" from, among other things, running compileall. When this patch was removed, "make install" ran compileall as normal and created bytecode files in __pycache__ directories. These files were then packaged in python3-light. This adds a patch to stop compileall from being run during "make install". Fixes: 4e05541 ("python3: bump to version 3.10.0") Signed-off-by: Jeffery To <jeffery.to@gmail.com>
The Makefile lines to add READELF to TARGET_CONFIGURE_OPTS was removed in 4e05541. Without setting READELF, configure finds the symlink to $(TARGET_CROSS)readelf (e.g. arm-openwrt-linux-readelf) instead of $(TARGET_CROSS)readelf (e.g. arm-openwrt-linux-muslgnueabi-readelf). This leads to the symlink name being saved to _sysconfigdata.py, and so the readelf name is not replaced correctly (in Py3Package/python3-base/install). This restores the removed Makefile lines. Fixes: 4e05541 ("python3: bump to version 3.10.0") Signed-off-by: Jeffery To <jeffery.to@gmail.com>
When doing parallel builds, host Python can install the python3 symlink before the Python standard library is installed completely. When this occurs, it is possible for other packages to detect the python3 symlink and try to use host Python before it is fully installed. This adds a patch to make commoninstall (where the standard library is installed) a prerequisite of bininstall (where the python3 symlink is installed), so that commoninstall is fully completed before bininstall begins. Patch has been submitted upstream: python/cpython#104693 Fixes: openwrt#19241 Signed-off-by: Jeffery To <jeffery.to@gmail.com>
By default, the Python build process will add /usr/local/{lib,include},
and multiarch paths (e.g. /usr/{lib,include}/x86_64-linux-gnu) if
building on Debian/Ubuntu, to its library and includes paths.
006-remove-multi-arch-and-local-paths.patch was added in
84202f1 to stop the Python build
process from adding these paths.
006-remove-multi-arch-and-local-paths.patch was removed in
48277ec.
006-do-not-add-multiarch-paths-when-cross-compiling.patch was added in
0c8b0b0 to stop the Python build
process from adding these paths for target Python.
These paths are still added by the Python build process when building
host Python.
This replaces the cross-compiling-only patch with the original patch,
renamed slightly and adapted for Python 3.10.
Fixes: 48277ec ("python3: bump to version 3.8")
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds $(STAGING_DIR_HOST)/include/e2fsprogs to HOST_CFLAGS and HOST_CPPFLAGS so that configure can find uuid/uuid.h. Signed-off-by: Jeffery To <jeffery.to@gmail.com>
026-openssl-feature-flags.patch and 028-host-python-support-ssl-with-libressl.patch were removed in 4ecd9d6 to fix the ssl module after libressl was upgraded to 3.7.0[1]. However, the cause of the ssl module build failure was only 028-host-python-support-ssl-with-libressl.patch. Removing 026-openssl-feature-flags.patch caused a build failure for the hashlib module. This restores 026-openssl-feature-flags.patch with an updated version of the patch from OpenBSD[2]. [1]: openwrt#20107 [2]: https://github.com/openbsd/ports/blob/26a04435bf2a09dcbe22b718bfee08997617a906/lang/python/3.10/patches/patch-Modules__hashopenssl_c Fixes: 4ecd9d6 ("python3: fix ssl support by removing libressl patches") Signed-off-by: Jeffery To <jeffery.to@gmail.com>
* Removed patches:
* 001-enable-zlib.patch: zlib module should be enabled automatically
* 007-distutils-do-not-adjust-path.patch: Not necessary since we
process shebang lines for all scripts (in python3-package.mk)
* 030-bpo-43112-detect-musl-as-a-separate-SOABI-openwrtGH-24502.patch:
Already merged
* Move configure vars from config.site back into Makefile
Centralizing all build information into one file makes it easier to
maintain
* No longer set ac_cv_header_uuid_h=yes as configure should detect
libuuid
* Order configure args by enable-/disable-/with-/without-, then
alphabetically
* Set ac_cv_working_openssl_hashlib=yes for host configure to bypass the
OpenSSL API tests with LibreSSL
* Use the default Host/Compile recipe instead of picking out specific
targets to make
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
|
Changes:
|
This was referenced May 24, 2023
|
hmmm the poetry module still seems broken when compiling from source. build log: base rev: 1c074ac |
Yes the fix is openwrt/openwrt#12710. I have also sent it to the mailing list now: https://patchwork.ozlabs.org/project/openwrt/patch/20230531135834.2150795-1-jeffery.to@gmail.com/ If it's still not merged in a week or so I'll do a special case fix for Python packages, but this is really a general issue. |
|
Thank you for the fix! And sorry I asked a stupid question (didn't check the existing issues first...) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Maintainer: me
Compile tested: armvirt-32/armvirt-64/malta-be/x86-generic/x86-64, 2023-05-15 snapshot sdk
Run tested: armvirt-32/armvirt-64/malta-be/x86-generic/x86-64 (qemu), 2023-05-15 snapshot
Description:
python3: Remove --without-pymalloc
--without-pymallocwas added in 7bf1ae6 because leaving it enabled added an "m" flag/suffix to file names.This flag/suffix was removed in Python 3.8, so disabling pymalloc is no longer necessary.
python3: Fix __pycache__ files included in python3-light
003-do-not-run-distutils-tests.patch was removed in 4e05541. This patch stopped
make installfrom, among other things, running compileall.When this patch was removed,
make installran compileall as normal and created bytecode files in __pycache__ directories. These files were then packaged in python3-light.This adds a patch to stop compileall from being run during
make install.Fixes: 4e05541 ("python3: bump to version 3.10.0")
python3: Fix readelf program name not replaced in _sysconfigdata.py
The Makefile lines to add
READELFtoTARGET_CONFIGURE_OPTSwas removed in 4e05541.Without setting
READELF, configure finds the symlink to$(TARGET_CROSS)readelf(e.g.arm-openwrt-linux-readelf) instead of$(TARGET_CROSS)readelf(e.g.arm-openwrt-linux-muslgnueabi-readelf).This leads to the symlink name being saved to _sysconfigdata.py, and so the readelf name is not replaced correctly (in
Py3Package/python3-base/install).This restores the removed Makefile lines.
Fixes: 4e05541 ("python3: bump to version 3.10.0")
python3: Fix race condition when doing parallel builds
When doing parallel builds, host Python can install the python3 symlink before the Python standard library is installed completely.
When this occurs, it is possible for other packages to detect the python3 symlink and try to use host Python before it is fully installed.
This adds a patch to make
commoninstall(where the standard library is installed) a prerequisite ofbininstall(where the python3 symlink is installed), so thatcommoninstallis fully completed beforebininstallbegins.Patch has been submitted upstream: gh-104692: Include commoninstall as a prerequisite for bininstall python/cpython#104693
Fixes: python3/host: build install race condition with uboot/scripts/dtc/pylibfdt #19241
python3: Fix multiarch/local paths added when building host Python
By default, the Python build process will add /usr/local/{lib,include}, and multiarch paths (e.g. /usr/{lib,include}/x86_64-linux-gnu) if building on Debian/Ubuntu, to its library and includes paths.
006-remove-multi-arch-and-local-paths.patch was added in 84202f1 to stop the Python build process from adding these paths.
006-remove-multi-arch-and-local-paths.patch was removed in 48277ec.
006-do-not-add-multiarch-paths-when-cross-compiling.patch was added in 0c8b0b0 to stop the Python build process from adding these paths for target Python.
These paths are still added by the Python build process when building host Python.
This replaces the cross-compiling-only patch with the original patch, renamed slightly and adapted for Python 3.10.
Fixes: 48277ec ("python3: bump to version 3.8")
python3: Fix uuid module not compiled for host Python
This adds
$(STAGING_DIR_HOST)/include/e2fsprogstoHOST_CFLAGSandHOST_CPPFLAGSso that configure can find uuid/uuid.h.python3: Fix hashlib module not compiled for host Python
026-openssl-feature-flags.patch and 028-host-python-support-ssl-with-libressl.patch were removed in 4ecd9d6 to fix the ssl module after libressl was upgraded to 3.7.0.
However, the cause of the ssl module build failure was only 028-host-python-support-ssl-with-libressl.patch.
Removing 026-openssl-feature-flags.patch caused a build failure for the hashlib module.
This restores 026-openssl-feature-flags.patch with an updated version of the patch from OpenBSD.
Fixes: 4ecd9d6 ("python3: fix ssl support by removing libressl patches")
python3: Update to 3.11.3, refresh/redo patches
Removed patches:
001-enable-zlib.patch: zlib module should be enabled automatically
007-distutils-do-not-adjust-path.patch: Not necessary since we process shebang lines for all scripts (in python3-package.mk)
030-bpo-43112-detect-musl-as-a-separate-SOABI-rust: select correct architecture for armv5 #24502.patch: Already merged
Move configure vars from config.site back into Makefile
Centralizing all build information into one file makes it easier to maintain
No longer set
ac_cv_header_uuid_h=yesas configure should detect libuuidOrder configure args by
enable-/disable-/with-/without-, then alphabeticallySet
ac_cv_working_openssl_hashlib=yesfor host configure to bypass the OpenSSL API tests with LibreSSLUse the default
Host/Compilerecipe instead of picking out specific targets to make