Using PythonCall results in a NumPy compilation error

[zornsllama]

Affects: PythonCall

Describe the bug
On doing using PythonCall, CondaPkg begins to set up a local Python environment. However, on reaching the installation of numpy, I receive the following compilation error:

Collecting numpy<1.19.0,>=1.15.4 (from ipfx)
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [60 lines of output]
      Running from numpy source directory.
      <string>:461: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      /tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
        required_version = LooseVersion('0.29.14')
      /tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py:77: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
        if LooseVersion(cython_version) < required_version:
      warning: /tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/numpy/__init__.pxd:17:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      warning: /tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/numpy/__init__.pxd:17:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          def __init__(self, seed=None):
              BitGenerator.__init__(self, seed)
              self.rng_state.pcg_state = &self.pcg64_random_state
      
              self._bitgen.state = <void *>&self.rng_state
              self._bitgen.next_uint64 = &pcg64_uint64
                                         ^
      ------------------------------------------------------------
      
      _pcg64.pyx:113:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'
      Processing numpy/random/_bounded_integers.pxd.in
      Processing numpy/random/mtrand.pyx
      Processing numpy/random/_pcg64.pyx
      Traceback (most recent call last):
        File "/tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py", line 238, in <module>
          main()
        File "/tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py", line 234, in main
          find_process_files(root_dir)
        File "/tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py", line 225, in find_process_files
          process(root_dir, fromfile, tofile, function, hash_db)
        File "/tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py", line 191, in process
          processor_function(fromfile, tofile)
        File "/tmp/pip-install-cgyfq94t/numpy_63725e2b74454648b28b02acfdf4db11/tools/cythonize.py", line 80, in process_pyx
          subprocess.check_call(
        File "/home/*/.julia/environments/v1.9/.CondaPkg/env/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/home/*/.julia/environments/v1.9/.CondaPkg/env/bin/python3.11', '-m', 'cython', '-3', '--fast-fail', '-o', '_pcg64.c', '_pcg64.pyx']' returned non-zero exit status 1.
      Cythonizing sources
      Traceback (most recent call last):
        File "/home/*/.julia/environments/v1.9/.CondaPkg/env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/*/.julia/environments/v1.9/.CondaPkg/env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/*/.julia/environments/v1.9/.CondaPkg/env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ngmf88ne/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-ngmf88ne/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 488, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ngmf88ne/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 488, in <module>
        File "<string>", line 469, in setup_package
        File "<string>", line 275, in generate_cython
      RuntimeError: Running cythonize failed!
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
ERROR: InitError: failed process: Process(`/home/*/.julia/environments/v1.9/.CondaPkg/env/bin/pip install ipfx pyabf`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base ./process.jl:477
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:370
  [5] (::CondaPkg.var"#29#30"{Base.TTY, Vector{String}, Vector{String}, Vector{String}})()
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:304
  [6] withenv(f::Function)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/env.jl:44
  [7] _resolve_pip_install(io::Any, pip_specs::Any, load_path::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:301
  [8] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:528
  [9] resolve()
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/resolve.jl:384
 [10] envdir()
    @ CondaPkg ~/.julia/packages/CondaPkg/osUdN/src/env.jl:70
 [11] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/1f5yE/src/cpython/context.jl:63
 [12] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/1f5yE/src/cpython/CPython.jl:21
 [13] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1074
 [14] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1020
 [15] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1471
 [16] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1748
 [17] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1625
 [18] macro expansion
    @ ./loading.jl:1613 [inlined]
 [19] macro expansion
    @ ./lock.jl:267 [inlined]
 [20] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1576
during initialization of module C

Your system
Please provide detailed information about your system:

• OS: Arch Linux
• Versions: Julia 1.9.2, PythonCall 0.9.13
• Output of Pkg.status():

Status `~/.julia/environments/v1.9/Project.toml`
  [7d9f7c33] Accessors v0.1.32
  [28f2ccd6] ApproxFun v0.13.20
  [13f3f980] CairoMakie v0.10.6
  [861a8166] Combinatorics v1.0.2
  [992eb4ea] CondaPkg v0.2.18
  [d194a933] Conductor v0.0.6 `~/research/Conductor.jl`
  [39dd38d3] Dierckx v0.5.3
  [aae7a2af] DiffEqFlux v2.1.0
  [77a26b50] DiffEqNoiseProcess v5.17.2
  [0c46a032] DifferentialEquations v7.8.0
  [31c24e10] Distributions v0.25.98
  [61744808] DynamicalSystems v3.2.0
  [eb9bf01b] FastDifferentiation v0.2.8
  [f6369f11] ForwardDiff v0.10.35
  [e9467ef8] GLMakie v0.8.6
  [86223c79] Graphs v1.8.0
  [34c5aeac] HSL v0.4.0
  [09f84164] HypothesisTests v0.11.0
  [7073ff75] IJulia v1.24.2
  [a98d9a8b] Interpolations v0.14.7
  [b6b21f68] Ipopt v1.4.1
  [c8e1da08] IterTools v1.8.0
  [4076af6c] JuMP v1.12.0
  [2621e9c9] MadNLP v0.7.0 `https://github.com/MadNLP/MadNLP.jl.git#master`
  [7fb6135f] MadNLPHSL v0.4.0
  [961ee093] ModelingToolkit v8.62.0
  [a4795742] NLPModels v0.20.0
  [f4238b75] NLPModelsIpopt v0.10.1
  [792afdf1] NLPModelsJuMP v0.12.1
  [1dea7af3] OrdinaryDiffEq v6.53.3
  [18e31ff7] Peaks v0.4.3
  [91a5bcdd] Plots v1.38.16
  [f27b6e38] Polynomials v3.2.13
  [31e2f376] PredefinedDynamicalSystems v1.1.1
  [c46f51b8] ProfileView v1.7.2
  [92933f4c] ProgressMeter v1.7.2
  [6099a3de] PythonCall v0.9.13
  [274fc56d] PythonPlot v1.0.2
  [90137ffa] StaticArrays v1.6.1
  [81625895] StaticCompiler v0.5.3
  [86c06d3c] StaticTools v0.8.8
⌅ [2913bbd2] StatsBase v0.33.21
  [0c5d862f] Symbolics v5.5.0
  [1986cc42] Unitful v1.15.0
  [017b0a0e] HSL_jll v2023.5.26+0 `~/software/coinhsl/HSL_jll.jl-2023.5.26`
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`

• Output of CondaPkg.status():

CondaPkg Status /home/*/.julia/environments/v1.9/CondaPkg.toml
Not Resolved (resolve first for more information)
Packages
  sympy
Pip packages
  ipfx
  pyabf

[cjdoris]

What happens if you try to create an equivalent environment by hand, like this?

conda create -n test sympy
conda activate test
pip install ipfx pyabf

If that fails, then the issue lies with something else, not PythonCall.

[zornsllama]

Do you mean to run this through CondaPkg.jl or an external Conda installation? If the former, "conda create" doesn't seem to be a recognized command. If the latter, I don't actually have an external Conda installation outside Julia.

[cjdoris]

Yeah the latter. Can you install conda (or mamba or micromamba) and try?

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[github-actions]

This issue has been closed because it has been stale for 7 days. You can re-open it if it is still relevant.