python3Packages.scipy can't be build with other blasProvider than openBlas

[sheepforce]

Describe the bug

When overriding blasProvider in blas with basically anything else than OpenBLAS, SciPy fails to build. The reason seems to be commit 44ae9b6 where the meson build is introduced. Meson defaults to OpenBLAS and requires flags to build with generic BLAS and Lapack. The current build works by accident, as SciPy uses the blasProvider passthru of NumPy instead of the generic BLAS wrapper or blas (without .blasProvider) in Nixpkgs, which happens to be OpenBLAS.

Steps To Reproduce

Steps to reproduce the behavior:

nix-build -E '
  let blasOvl = final: prev: {
        blas = prev.blas.override { blasProvider = prev.amd-blis; };
        lapack = prev.lapack.override { lapackProvider = prev.amd-libflame; };
      };
      nixpkgs = import ./default.nix {
        overlays = [ blasOvl ];
      };
   in nixpkgs.python3.pkgs.scipy
'

Expected behavior

SciPy should build and work with the wrapper of the generic BLAS interface instead of relying on OpenBLAS.

Notify maintainers

@FRidh

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.81, NixOS, 22.11 (Raccoon), 22.11.20221207.5528350`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.0`
 - channels(phillip): `"nixpkgs"`
 - nixpkgs: `/home/phillip/.nix-defexpr/channels/nixpkgs`

[ConnorBaker]

This is a problem I just ran into. Any ideas?

Here's a flake to reproduce the error with nix build:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs";

  outputs =
    { self
    , nixpkgs
    }:
    let
      system = "x86_64-linux";

      pkgs = import nixpkgs {
        inherit system;
        config.allowUnfree = true;
        overlays = [
          (final: prev: {
            blas = prev.blas.override {
              blasProvider = final.mkl;
            };
            lapack = prev.lapack.override {
              lapackProvider = final.mkl;
            };
          })
        ];
      };
    in
    {
      packages.${system}.default = pkgs.python3.pkgs.scipy;
    };
}

And here's the actual error:

python3.10-scipy> Sourcing python-remove-tests-dir-hook
python3.10-scipy> Sourcing python-catch-conflicts-hook.sh
python3.10-scipy> Sourcing python-remove-bin-bytecode-hook.sh
python3.10-scipy> Sourcing pip-build-hook
python3.10-scipy> Using pipBuildPhase
python3.10-scipy> Using pipShellHook
python3.10-scipy> Sourcing pip-install-hook
python3.10-scipy> Using pipInstallPhase
python3.10-scipy> Sourcing python-imports-check-hook.sh
python3.10-scipy> Using pythonImportsCheckPhase
python3.10-scipy> Sourcing python-namespaces-hook
python3.10-scipy> Sourcing python-catch-conflicts-hook.sh
python3.10-scipy> unpacking sources
python3.10-scipy> unpacking source archive /nix/store/bab0qjg738jl3gy1jb5qgcvbjmjm9v33-scipy-1.9.3.tar.gz
python3.10-scipy> source root is scipy-1.9.3
python3.10-scipy> setting SOURCE_DATE_EPOCH to timestamp 315619200 of file scipy-1.9.3/tox.ini
python3.10-scipy> patching sources
python3.10-scipy> configuring
python3.10-scipy> no configure script, doing nothing
python3.10-scipy> building
python3.10-scipy> Executing pipBuildPhase
python3.10-scipy> Creating a wheel...
python3.10-scipy> WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
python3.10-scipy> Processing /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3
python3.10-scipy>   Running command Preparing metadata (pyproject.toml)
python3.10-scipy>   The Meson build system
python3.10-scipy>   Version: 0.64.1
python3.10-scipy>   Source dir: /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3
python3.10-scipy>   Build dir: /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3/.mesonpy-3wfcasw8/build
python3.10-scipy>   Build type: native build
python3.10-scipy>   Project name: SciPy
python3.10-scipy>   Project version: 1.9.3
python3.10-scipy>   C compiler for the host machine: gcc (gcc 11.3.0 "gcc (GCC) 11.3.0")
python3.10-scipy>   C linker for the host machine: gcc ld.bfd 2.39
python3.10-scipy>   C++ compiler for the host machine: g++ (gcc 11.3.0 "g++ (GCC) 11.3.0")
python3.10-scipy>   C++ linker for the host machine: g++ ld.bfd 2.39
python3.10-scipy>   Host machine cpu family: x86_64
python3.10-scipy>   Host machine cpu: x86_64
python3.10-scipy>   Compiler for C supports arguments -Wno-unused-but-set-variable: YES
python3.10-scipy>   Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
python3.10-scipy>   Compiler for C supports arguments -Wno-unused-function: YES
python3.10-scipy>   Compiler for C supports arguments -Wno-conversion: YES
python3.10-scipy>   Compiler for C supports arguments -Wno-misleading-indentation: YES
python3.10-scipy>   Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
python3.10-scipy>   Library m found: YES
python3.10-scipy>   Fortran compiler for the host machine: gfortran (gcc 11.3.0 "GNU Fortran (GCC) 11.3.0")
python3.10-scipy>   Fortran linker for the host machine: gfortran ld.bfd 2.39
python3.10-scipy>   Compiler for Fortran supports arguments -Wno-conversion: YES
python3.10-scipy>   Program cython found: YES (/nix/store/j3ksmm3idcdnmbb6s73x1k162kbjwgkl-python3.10-cython-0.29.32/bin/cython)
python3.10-scipy>   Program pythran found: YES (/nix/store/fray0v0l6wiigzlvfh1w47df04ma43xp-python3.10-pythran-0.11.0/bin/pythran)
python3.10-scipy>   Program cp found: YES (/nix/store/lljkbrqh1nkayprky3lrigj0fbxj35rv-coreutils-9.1/bin/cp)
python3.10-scipy>   Program python found: YES (/nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9/bin/python3.10)
python3.10-scipy>   Found pkg-config: /nix/store/v73ppkjk196bwsjm6yy9yv80bdq7q68v-pkg-config-wrapper-0.29.2/bin/pkg-config (0.29.2)
python3.10-scipy>   Run-time dependency threads found: YES
python3.10-scipy>   Library npymath found: YES
python3.10-scipy>   Library npyrandom found: YES
python3.10-scipy>   Did not find CMake 'cmake'
python3.10-scipy>   Found CMake: NO
python3.10-scipy>   Run-time dependency openblas found: NO (tried pkgconfig and cmake)
python3.10-scipy>   Run-time dependency openblas found: NO (tried pkgconfig)
python3.10-scipy>   ../../scipy/meson.build:131:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig
python3.10-scipy>   A full log can be found at /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3/.mesonpy-3wfcasw8/build/meson-logs/meson-log.txt
python3.10-scipy>   + meson setup --prefix=/nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9 /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3 /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3/.mesonpy-3wfcasw8/build --native-file=/tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
python3.10-scipy>   error: subprocess-exited-with-error
python3.10-scipy>   × Preparing metadata (pyproject.toml) did not run successfully.
python3.10-scipy>   │ exit code: 1
python3.10-scipy>   ╰─> See above for output.
python3.10-scipy>   note: This error originates from a subprocess, and is likely not a problem with pip.
python3.10-scipy>   full command: /nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9/bin/python3.10 /nix/store/flz0j8byjnpd9fnn9vfkf9s923zfkhb3-python3.10-pip-22.3.1/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/tmp2qes5bo_
python3.10-scipy>   cwd: /tmp/nix-build-python3.10-scipy-1.9.3.drv-0/scipy-1.9.3
python3.10-scipy>   Preparing metadata (pyproject.toml) ... error
python3.10-scipy> error: metadata-generation-failed
python3.10-scipy> × Encountered error while generating package metadata.
python3.10-scipy> ╰─> See above for output.
python3.10-scipy> note: This is an issue with the package mentioned above, not pip.
python3.10-scipy> hint: See above for details.
error: builder for '/nix/store/vijn5kl1jl71gs3k1lw805h7xhkwhdcr-python3.10-scipy-1.9.3.drv' failed with exit code 1

[alexvorobiev]

I am having this issue too and MKL is critical for my use case. Would it be possible to rollback to the old build method?

[FRidh]

Scipy using Meson needs to be configured explicitly with a blas/lapack implementation. They've introduced the -Dblas and -Dlapack flags for that. Because we need a wheel we use the meson-python build backend, which invokes meson. This bypasses our setup hook so it is not possible to just set

mesonFlags = ["-Dblas=${blas.implementation}" "-Dlapack=${lapack.implementation}"];

Instead, we need to pass these options to pip which then passes it to meson-python. As it is now, our pip-build-hook.sh does not support passing --config-settings. This needs to be added. Maybe just a pipBuildFlags analogous to our pipInstallFlags.

A more crude approach would be to patch meson-python to pick up our mesonFlags. Probably easiest though.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/developing-nix-only-python-packages/26342/6

[SomeoneSerge]

-Dblas=${blas.implementation}

Shouldn't it just be -Dblas=cblas to make use of

nixpkgs/pkgs/build-support/alternatives/blas/default.nix

Lines 137 to 141 in 27ccf0e

	'' + lib.optionalString (blasImplementation == "mkl") ''
	mkdir -p $out/nix-support
	echo 'export MKL_INTERFACE_LAYER=${lib.optionalString isILP64 "I"}LP64,GNU' > $out/nix-support/setup-hook
	ln -s $out/lib/libblas${canonicalExtension} $out/lib/libmkl_rt${stdenv.hostPlatform.extensions.sharedLibrary}
	ln -sf ${blasProvider'}/include/* $dev/include

I haven't touched build-support/alternatives before though, so I've no idea how this works

[SomeoneSerge]

@FRidh -Dblas=${blas.implementation} didn't work for me, because implementation evaluates into "mkl" and there's no mkl.pc afaiu. Using the aforementioned cblas and lapacke works, as well as targets like mkl-dynamic-ilp64-gomp

[nviets]

I also ran into this one. Is there any workaround I can use?

[doronbehar]

This is fixed in 5262a7d and the commits before and after it. Closing.