Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libxml2: Fix python build & import #5930

Merged
merged 12 commits into from
Dec 9, 2024
Merged

libxml2: Fix python build & import #5930

merged 12 commits into from
Dec 9, 2024

Conversation

star-hengxing
Copy link
Contributor

Upstream report: https://gitlab.gnome.org/GNOME/libxml2/-/issues/824

@star-hengxing
Copy link
Contributor Author

@waruqi 请帮忙调试 mac 上的问题。

已知信息:

  • 所有 python 产物都在 <installdir>/python 目录,并加上 PYTHONPATH
  • 静态链接 python3.13 和 libxml2 依然报错,应该和 python 版本 & 各种环境变量 path 加载无关。
  • 链接有一些奇怪 warning。
/Applications/Xcode_15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc  -pedantic -Wall -Wextra -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wno-long-long -Wno-format-extra-args  -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk -dynamiclib -Wl,-headerpad_max_install_names -current_version 2.13.4 -o libxml2mod.2.13.4.dylib -install_name @rpath/libxml2mod.2.13.4.dylib "CMakeFiles/LibXml2Mod.dir/libxml2-py.c.o" CMakeFiles/LibXml2Mod.dir/python/libxml.c.o CMakeFiles/LibXml2Mod.dir/python/types.c.o  libxml2.a /Users/runner/.xmake/packages/p/python/3.13.0/3dd64f01e26f48debd2ccd8c08fdb2a3/lib/libpython3.13.a -ldl -lm /Users/runner/.xmake/packages/l/libiconv/1.17/035a4b093d48429a8f361b4ea32b4663/lib/libiconv.a -ldl -framework CoreFoundation
ld: warning: ignoring duplicate libraries: '-ldl'
ld: warning: search path '/opt/homebrew/Cellar/gcc/14.2.0/lib/gcc/14/' not found
ld: warning: object file (/Users/runner/.xmake/packages/p/python/3.13.0/3dd64f01e26f48debd2ccd8c08fdb2a3/lib/libpython3.13.a[2](getbuildinfo.o)) was built for newer 'macOS' version (14.5) than being linked (14.0)
python3 -c "import libxml2"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import libxml2
  File "/Users/runner/.xmake/packages/l/libxml2/v2.13.4/def471559399468b8c364a8c7c220590/python/libxml2.py", line 1, in <module>
    import libxml2mod
ModuleNotFoundError: No module named 'libxml2mod'

@star-hengxing star-hengxing requested a review from waruqi December 8, 2024 09:36
@waruqi
Copy link
Member

waruqi commented Dec 8, 2024

这边检测 python 都没过,还是找的系统的 python2.7

-- The C compiler identification is AppleClang 16.0.0.16000026
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefaul
t.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for F_GETFL
-- Looking for F_GETFL - found
CMake Error at /usr/local/Cellar/cmake/3.25.3/share/cmake/Modules/FindPackageHandleStandardArgs.cm
ake:230 (message):
  Could NOT find Python (missing: Development Development.Module
  Development.Embed) (found version "2.7.17")
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.25.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPH
SA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.25.3/share/cmake/Modules/FindPython.cmake:561 (find_package_handle_sta
ndard_args)
  CMakeLists.txt:69 (find_package)

@star-hengxing
Copy link
Contributor Author

这边检测 python 都没过,还是找的系统的 python2.7

这是啥环境,ci 上这里没问题。

-- Found Python: /Users/runner/.xmake/packages/p/python/3.13.0/3dd64f01e26f48debd2ccd8c08fdb2a3/bin/python3.13 (found version "3.13.0") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Failed
-- Found Iconv: /Users/runner/.xmake/packages/l/libiconv/1.17/035a4b093d48429a8f361b4ea32b4663/lib/libiconv.a (found version "1.17")

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


The python test here has not passed, but I still looked for the system’s python2.7

What kind of environment is this? There is no problem here on ci.

-- Found Python: /Users/runner/.xmake/packages/p/python/3.13.0/3dd64f01e26f48debd2ccd8c08fdb2a3/bin/python3.13 (found version "3.13.0") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Failed
-- Found Iconv: /Users/runner/.xmake/packages/l/libiconv/1.17/035a4b093d48429a8f361b4ea32b4663/lib/libiconv.a (found version "1.17")

@waruqi
Copy link
Member

waruqi commented Dec 9, 2024

macos 下有几个问题

  1. 目前生成的是 libxml2mod.dylib, 得改成 libxml2mod.so 或者 xml2mod.so ,并且 import("xml2mod") 才能找到,否则是找不到的,参考 https://github.com/xmake-io/xmake/blob/46b99517934618116d6b280ffac819ba830af7db/xmake/rules/python/xmake.lua#L42
  2. macos 下编译模块,不能显式加 -lpython link,并且要加 -undefined dynamic_lookup,否则即使 import 找到了,也是会 segment fault,参考:尝试使用pybind11做python的绑定,按照文档操作无果 xmake#2177 (comment) https://github.com/xmake-io/xmake/blob/46b99517934618116d6b280ffac819ba830af7db/xmake/rules/python/xmake.lua#L49C40-L49C65

而cmakelists.txt 里面,它直接加了 Python link,没对 macos 做特殊处理,应该仅仅支持 linux

目前这边本地测试通过

python3 -c "import libxml2"
  => install libxml2 v2.13.4 .. ok

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


There are several problems under macos

  1. Currently, libxml2mod.dylib is generated. It must be changed to libxml2mod.so or xml2mod.so and import("xml2mod") is used to find it. Otherwise, it cannot be found. Please refer to https://github.com/xmake-io/xmake/blob/46b99517934618116d6b280ffac819ba830af7db/xmake/rules/python/xmake.lua#L42
  2. When compiling modules under macos, you cannot explicitly add -lpython link, and you must add -undefined dynamic_lookup, otherwise even if the import is found, segment fault will occur. Reference: https://github.com/xmake-io /xmake/issues/2177#issuecomment-1209423820 https://github.com/xmake-io/xmake/blob/46b99517934618116d6b280ffac819ba830af7db/xmake/rules/python/xmake.lua#L49C40-L49C65

In cmakelists.txt, it directly adds the Python link. No special processing is done for macos. It should only support linux.

At present, the local test here has passed

python3 -c "import libxml2"
  => install libxml2 v2.13.4 .. ok

@waruqi waruqi merged commit 81fb898 into dev Dec 9, 2024
67 checks passed
@waruqi waruqi deleted the libxml2-python branch December 9, 2024 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants