Unable to cross compile on Ubuntu Jammy (amd64 -> aarch64)

[gabrik]

Hi, I'm having some issues in cross-compiling a library that links with the Python interpreter.

I'm using Ubuntu 22.04 as host system with the following packages installed:

build-essential
pkg-config 
crossbuild-essential-arm64
libpython3-dev:arm64 

Rust is version rustc 1.66.0 (69f9c33d7 2022-12-12) and PyO3 is version: 0.16

I ran the build with this command: PYO3_CROSS_PYTHON_VERSION=3.10 PYO3_CROSS_LIB_DIR=/lib/aarch64-linux-gnu/ cargo build --target=aarch64-unknown-linux-gnu

But I get this error:

Compiling pyo3-ffi v0.16.6
error: failed to run custom build command for `pyo3-ffi v0.16.6`

Caused by:
  process didn't exit successfully: `/root/zenoh-flow-python/target/debug/build/pyo3-ffi-49ed1ebc11f3b498/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
  cargo:rerun-if-env-changed=PYO3_NO_PYTHON
  cargo:rerun-if-env-changed=_PYTHON_SYSCONFIGDATA_NAME

  --- stderr
  error: Could not find either libpython.so or _sysconfigdata*.py in /lib/aarch64-linux-gnu/

And ls /lib/aarch64-linux-gnu/ shows that python is actually there:

$ ls /lib/aarch64-linux-gnu/libpython3*
/lib/aarch64-linux-gnu/libpython3.10.a   /lib/aarch64-linux-gnu/libpython3.10.so.1
/lib/aarch64-linux-gnu/libpython3.10.so  /lib/aarch64-linux-gnu/libpython3.10.so.1.0

Any suggestion?

[messense]

I think the error message is kinda outdated, pyo3 actually looks for _sysconfigdata*.py when cross compiling.

[gabrik]

In that case it is right as _sysconfigdata*.py is not there. Any idea which package is providing that file?

[messense]

libpython3.10-minimal: https://packages.ubuntu.com/jammy/arm64/libpython3.10-minimal/filelist

[gabrik]

Apt says it is actually installed, but the files are not there. Magic.

In any case, thank you for your help. I guess I'll try to figure out why they are not there.

And then update the discussion with the solution (hopefully).

[gabrik]

Seems that the PYO3_CROSS_LIB_DIR=/lib/aarch64-linux-gnu/ was not needed.

And that I need to change my build.rs to :

fn main() {
  pyo3_build_config::add_extension_module_link_args();
}

But now I have Symbols not found errors coming from one of the libraries loaded within Python, similar to what we experienced in #2024
And seems that the workaround we where using there is not working when cross-building.

[davidhewitt]

Can you share the symbols which are missing? Possibly you are building / running on different Python versions.

[gabrik]

The missing symbol was PyErr_SetImportError, but the symbol was actually there.
I guess what is happening is the same issue we had in #2024 in which the python library is dynamically linked but symbols are loaded with the LOCAL flags so they are not available when importing python code that is based on native code (e.g., numpy)

The workaround we had was to explicitly load the python library when instantiating the .so with the GLOBAL flag, with the name of the library provided by pyo3_build_config and stored in a static via the build.rs. But this seems to not work when cross-compiling.

So my idea was either to embed the python interpreter with static linking, but it will be a pain, or find another way to have these symbols loaded automatically with the GLOBAL flag.
So far we are focusing on a (alpha) release and we will look more on this issue right after.

[davidhewitt]

Ah I see. You might want to check out the PyOxidizer project. I believe it's normally used to package executables rather than libraries, though you may be able to learn some tricks from it even if it doesn't natively support exactly what you need.

[gabrik]

Indeed it is on my TODO list to check how PyOxidizer works and see what we can learn from it.