Unable to find DLLs on Windows using PythonCall with Python3.8+

[thealanjason]

Affects: PythonCall / JuliaCall(Maybe)

Describe the bug
Using a project folder environment for a Julia project with julia --project=. and when trying to use PythonCall along with CondaPkg with numpy on Windows 10, with Julia 1.10 I get the following error:

julia> using PythonCall
D:\psimpy.jl
ERROR: InitError: Python: ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Python stacktrace:
 [1] <module>
   @ D:\psimpy.jl\.CondaPkg\env\lib\ctypes\__init__.py:8
 [2] init
   @ C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\pysrc\juliacall\__init__.py:54
 [3] <module>
   @ C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\pysrc\juliacall\__init__.py:229
Stacktrace:
  [1] pythrow()
    @ PythonCall C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\err.jl:94
  [2] errcheck
    @ PythonCall C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\err.jl:10 [inlined]
  [3] pyimport(m::String)
    @ PythonCall C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\concrete\import.jl:11
  [4] init_juliacall()
    @ PythonCall C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\juliacall.jl:25
  [5] (::PythonCall.var"#199#203")()
    @ PythonCall C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\PythonCall.jl:91
  [6] with_gil
    @ PythonCall.C C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\cpython\gil.jl:10 [inlined]
  [7] with_gil
    @ PythonCall.C C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\cpython\gil.jl:9 [inlined]
  [8] __init__()
    @ PythonCall C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\src\PythonCall.jl:86
  [9] run_module_init(mod::Module, i::Int64)
    @ Base .\loading.jl:1128
 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base .\loading.jl:1116
 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base .\loading.jl:1061
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base .\loading.jl:1575
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:1932
 [14] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:1806
 [15] #invoke_in_world#3
    @ Base .\essentials.jl:921 [inlined]
 [16] invoke_in_world
    @ Base .\essentials.jl:918 [inlined]
 [17] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:1797
 [18] macro expansion
 [19] macro expansion
    @ Base .\lock.jl:267 [inlined]
 [20] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1747
 [21] #invoke_in_world#3
    @ Base .\essentials.jl:921 [inlined]
 [22] invoke_in_world
    @ Base .\essentials.jl:918 [inlined]
 [23] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1740
during initialization of module PythonCall

This however does not happen when running in the default Julia environment (@v1.10)

Expectation:
using PythonCall should work without issues (as on Linux)

Steps to Reproduce

1. Either use julia --project=. inside a temporary folder OR start Julia and ] activate .
2. Run the following MWE

using Pkg
Pkg.add("CondaPkg")
Pkg.add("PythonCall")
using CondaPkg
CondaPkg.add("numpy")
using PythonCall

Your system
OS: Windows10
Julia Version: 1.10
Python Version: 3.10
PythonCall Version: v0.9.15
Output of Pkg.status():

Status `D:\psimpy.jl\Project.toml`
  [992eb4ea] CondaPkg v0.2.22
  [6099a3de] PythonCall v0.9.15

Output of CondaPkg.status():

CondaPkg Status D:\psimpy.jl\CondaPkg.toml
Environment
  D:\psimpy.jl\.CondaPkg\env
Packages
  numpy v1.26.4
  python v3.10.0 (3.10)
Channels
  conda-forge

Additional context
The issues #146, #245, #339, #373 seem to also be affected by this.

On a bit of further reading on stack overflow, I found out that this issue is related to loading the DLLs on Windows. The Python 3.8 release notes state:

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)

[thealanjason]

In the default (@v1.10) environment I get:

using CondaPkg
] conda run python
Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 07:53:56) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PATH']
'C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Library\\mingw-w64\\bin;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Library\\usr\\bin;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Library\\bin;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\Scripts;C:\\Users\\Alan Jason Correa\\.julia\\environments\\v1.10\\.CondaPkg\\env\\bin;........

This shows that the correct paths have been added in the default Julia enironment.

[thealanjason]

I modified the init() function here locally to print os.environ['PATH'] and get the following output:

 D:\psimpy.jl> julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.    
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0 (2023-12-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release  
|__/                   |

julia> using PythonCall
    CondaPkg Found dependencies: D:\psimpy.jl\CondaPkg.toml
    CondaPkg Found dependencies: C:\Users\Alan Jason Correa\.julia\packages\PythonCall\wXfah\CondaPkg.toml
    CondaPkg Found dependencies: C:\Users\Alan Jason Correa\.julia\environments\v1.10\CondaPkg.toml
    CondaPkg Dependencies already up to date
D:\psimpy.jl\.CondaPkg\env;D:\psimpy.jl\.CondaPkg\env\Library\mingw-w64\bin;D:\psimpy.jl\.CondaPkg\env\Library\usr\bin;D:\psimpy.jl\.CondaPkg\env\Library\bin;D:\psimpy.jl\.CondaPkg\env\Scripts;D:\psimpy.jl\.CondaPkg\env\bin;......

[thealanjason]

Quoting from the OP:

Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library.

The os.environ['PATH'] is ignored on Windows. So I tried to debug by doing something very hacky: Made locally some changes to the init() function in __init__.py file of pysrc/juliacall module as follows:

def init():
    import os
    #D:\psimpy.jl\.CondaPkg\env;
    path = os.path.join(os.getcwd(), ".CondaPkg", "env")
    print(f"Adding {path} to dll search path")
    os.add_dll_directory(path)
    #D:\psimpy.jl\.CondaPkg\env\Library\mingw-w64\bin;
    path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Library", "mingw-w64", "bin")
    print(f"Adding {path} to dll search path")
    os.add_dll_directory(path)
    # # D:\psimpy.jl\.CondaPkg\env\Library\usr\bin;           ##### ERROR: Directory not found.
    # path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Library", "usr", "bin")
    # print(f"Adding {path} to dll search path")
    # os.add_dll_directory(path)
    # D:\psimpy.jl\.CondaPkg\env\Library\bin;
    path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Library", "bin")
    print(f"Adding {path} to dll search path")
    os.add_dll_directory(path)
    # D:\psimpy.jl\.CondaPkg\env\Scripts;
    path = os.path.join(os.getcwd(), ".CondaPkg", "env", "Scripts")
    print(f"Adding {path} to dll search path")
    os.add_dll_directory(path)
    # D:\psimpy.jl\.CondaPkg\env\bin
    path = os.path.join(os.getcwd(), ".CondaPkg", "env", "bin")
    print(f"Adding {path} to dll search path")
    os.add_dll_directory(path)

And this fixes the issue. See below:

julia> 
PS D:\psimpy.jl> julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0 (2023-12-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using PythonCall
Adding D:\psimpy.jl\.CondaPkg\env to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\Library\mingw-w64\bin to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\Library\bin to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\Scripts to dll search path
Adding D:\psimpy.jl\.CondaPkg\env\bin to dll search path

julia> np = pyimport("numpy")
Python: <module 'numpy' from 'D:\\psimpy.jl\\.CondaPkg\\env\\lib\\site-packages\\numpy\\__init__.py'>

Maybe there is a better way or location to make the os.add_dll_directory(<dll_search_path>) calls. However, I don't know how the PythonCall package is structured and its inner workings. It seems very difficult for me to understand. I could create a pull request for now.

[cjdoris]

What you are experiencing is actually a bug in Python 3.10.0 which has been fixed in 3.10.2.

From this line of CondaPkg.status()

  python v3.10.0 (3.10)

I can see that you have the buggy Python 3.10.0 installed. This happened because you added Python like CondaPkg.add("python", version="3.10"). Unlike Julia's package manager, a version of 3.10in Conda means exactly3.10.0`.

You probably actually wanted version="3.10.*" to get the latest Python 3.10.

Note that this prevents any newer Python from being used. Unless you really do need Python 3.10 it's generally recommended to do something like version=">=3.10,<4" to get any Python 3.10+.

[cjdoris]

Closing as I believe this issue is solved.