Cross-compiling to windows with pyo3

[Waelwindows]

Perhaps related to #216,

I'm trying to cross-compile my project to x86_64-pc-windows-gnu using crane.
However, I get the following error message which I haven't faced before in my devshell environment

error: builder for '/nix/store/xi9ishj1kn1a98jf4gd8l4pzln004py6-objset-deps-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       >   cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
       >   cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
       >
       >   --- stderr
       >   x86_64-w64-mingw32-dlltool: failed to open temporary head file: _build_source_target_x86_64_pc_windows_gnu_release_build_pyo3_ffi_e328b6241fdb89c9_out_lib_python3_dll_a_h.s
       >   error: failed to generate python3.dll import library
       >   caused by:
       >     - 0: "x86_64-w64-mingw32-dlltool" "--input-def" "/build/source/target/x86_64-pc-windows-gnu/release/build/pyo3-ffi-e328b6241fdb89c9/out/lib/python3.def" "--output-lib" "/build/source/target/x86_64-pc-windows-gnu/release/build/pyo3-ffi-e328b6241fdb89c9/out/lib/python3.dll.a" failed with exit status: 1
       >
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/xi9ishj1kn1a98jf4gd8l4pzln004py6-objset-deps-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/8mrp3slqh0yr5mf81vymkh5p69d6ldl1-objset-0.1.0.drv' failed to build

I'm having issues cross compiling it using the given instructions in the windows cross-compilation example, so I tried building in a dev shell.
While it can compile in the devshell, it doesn't compile as is, You have to disable pthread first because that causes linking errors for some reason. You get many linking errors about libpthread symbols culminating in this one

          /nix/store/gqjznrqzxzdxdiqgfvpm93lq1j2x59pp-pthreads-w32-x86_64-w64-mingw32-2.9.1/lib/libpthread.a(pthread.o):(.pdata+0x0): dangerous relocation: /nix/store/wn31i3dzwahz6ccws8bs1nwyqrpgsvj7-gcc-wrapper-11.3.0/bin/ld: line 256: 453975 Segmentation fault      (core dumped) /nix/store/frh9l9nrdysasdi2gs7i241s241ngjw2-binutils-2.39/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}
          collect2: error: ld returned 139 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

After compiling once without pthread it'll proceed till it complains about a missing libpthread. Only then can you add pthread and it finish compiling. I can confirm that the resulting .dll does function normally, at least via python on wine.

So the steps in order to build it in the devshell are:

1. Run cargo build --target="x86_64-pc-windows-gnu" with pkgsCross.mingwW64.windows.pthreads commented out in devShell's `buildInput
2. Do Step 1 without commenting out pkgsCross.mingwW64.windows.pthreads
3. Inspect the built .dll inside of target/x86_64-pc-windows-gnu

Obviously this won't do, and i have to find a fix for this. Ideally i'd like to be able to do all of this in just nix build so that I do this in CI.

Here's my latest attempt so far.

On a side note, i think it would be great if crane had a maturin builder just like nixpkgs's maturinHook.

[ipetkov]

Hi @Waelwindows, thanks for the report!

Unfortunately I'm not very familiar with neither pyo3 nor Windows cross compilation so I'm not sure how to debug this myself. Searching for failed to open temporary head file results in several links suggesting there is something wrong with the cross toolchain itself. Seems like NixOS/nixpkgs#170946 was tracking a similar issue which has supposedly been resolved but that's as much context as I have on the issue...

Though if anyone figures this out it would be great to either fix things or document them appropriately!

[Waelwindows]

I've managed to fix the devshell problem by passing in libpthread in RUSTFLAGS instead.

RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
  pkgs.pkgsCross.mingwW64.windows.pthreads
]);

However this bricks the builds for any other target.

As for the actual nix build itself, no clues yet. I've looked into the issue you mentioned and it should've been fixed by 2.39 which i'm using. Assuming that the passed pkgsCross.mingwW64.stdenv.cc between the devshell and the crane builder are the sames there shouldn't be an error to begin with.

If only there was a way to see exactly what the derivation is actually doing, We'd have a better way to diagnose this

[ipetkov]

@Waelwindows you can set NIX_DEBUG = 10; on the derivation to get a very verbose log of everything the derivation executes. There are other numeric levels you can set to control the verbosity but I forget where that's documented...

I have a sneaking suspicion the issue may be related to strictDeps which influences to how various paths are updated with the derivation's inputs. Either something strictDeps is omitting too much stuff or something else is going wrong when strictDeps is disabled