python - problem with library search order?

[Ede123]

We found an issue downstream where Python searches for libraries of a dynamically linked module (_hashlib.pyd) in an order that makes it likely to cause problems:

_hashlib.pyd is linked against LIBEAY32.dll; when loaded it attempts to load

C:\Program Files\Inkscape\lib\python2.7\LIBEAY32.dll
C:\Windows\System32\LIBEAY32.dll
C:\Windows\System\LIBEAY32.dll
C:\Windows\LIBEAY32.dll
C:\Program Files\Inkscape\libeay32.dll

This is consistent with what is described at
https://github.com/numpy/numpy/wiki/windows-dll-notes

Unfortunately if an incompatible LIBEAY32.dll (or any other DLL that needs to be loaded) is found in one of the system directories the module load fails.

There are easy steps to reproduce this:

• Create an empty file at C:\Windows\System32\LIBEAY32.dll
• Open cmd.exe
• Launch C:\msys64\mingw64\bin\python.exe
• Execute "import hashlib"
• See errors like

ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "C:\msys64\mingw64\lib\python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "C:\msys64\mingw64\lib\python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5

I've found that this does not seem to happen when launched from inside an MSYS2 shell (anybody knows what is different in this scenario?).

A workaround for Inkscape seems to be to call SetDllDirectory but for standalone Python that still won't work.

Any ideas what is going wrong and how/where to fix it best? Interestingly python3.6.exe is not affected either, so maybe there is already a solution available?

[lazka]

Weird, I can reproduce with mingw64 python but not with mingw32 python. Can you confirm that?

[Ede123]

Indeed! Only happens with mingw64 for me, too.

[mingwandroid]

You'd need to put the broken DLL in syswow64 for 32 bit. This is just standard DLL-hell though, no one but Microsoft should put libraries in that folder and the software doing this is broken and inconsiderate. On Python 3 hashlib depends on openssl much less (if at all).

…

On Feb 4, 2018 10:51 AM, "Christoph Reiter" ***@***.***> wrote: Weird, I can reproduce with mingw64 but not with mingw32. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/Alexpux/MINGW-packages/issues/3343#issuecomment-362897957>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_pdHt3jJ1I6C0bepyiNBXBLT4BuVwkks5tRYvDgaJpZM4R4Zih> .

[lazka]

@mingwandroid ah, thanks. Shouldn't python still prefer the DLL in the directory of the python.exe?

[mingwandroid]

@lazka, well, the code to load an extension module (ok for Python 3.6 but I think 2.7 will be much the same) is here:

https://github.com/python/cpython/blob/f7eae0adfcd4c50034281b2c69f461b43b68db84/Python/dynload_win.c#L218-L222

        /* We use LoadLibraryEx so Windows looks for dependent DLLs
            in directory of pathname first. */
        /* XXX This call doesn't exist in Windows CE */
        hDLL = LoadLibraryExW(wpathname, NULL,
                              LOAD_WITH_ALTERED_SEARCH_PATH);

So AFAICT, LOAD_WITH_ALTERED_SEARCH_PATH causes the Windows loader to look not in the folder containing python*.exe but in the dyn-load folder (or somewhere in site-packages) instead. It would be nice if it looked in this folder first then the folder containing the executable after that, but that does not seem to be the case. Search for Alternate Search Order for Desktop Applications on:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx

[lazka]

Thanks @mingwandroid . Makes sense, and the order you mentioned would indeed be nicer.

From what I see you can get that order by combining LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR and LOAD_LIBRARY_SEARCH_DEFAULT_DIRS, but those are only available with Windows 7.

[mingwandroid]

Ah ok, good research. We could think to add the executable's dirname to the dll search path somewhere around Python_Init. Am on my phone so cannot find the correct function names ATM!

[lazka]

Yeah, if the numpy wiki is correct, then

if (GetProcAddress(GetModuleHandleW(L"kernel32.dll"), "SetDefaultDllDirectories"))
    SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);

in init might work.

Experiments suggest that, if you first set DLL search order with LOAD_LIBRARY_SEARCH_* flags to SetDefaultDllDiectories, then call LoadLibraryEx with LOAD_WITH_ALTERED_SEARCH_PATH, then the load-dll path will be prefixed to the search order given by your LOAD_LIBRARY_SEARCH_* flags.

Now I also understand why MSYS2 Python always prepends the executable directory to PATH while the official Python doesn't. Otherwise nothing would be found...

[Ede123]

What about simply using SetDllDirectory and point it to the /bin folder (i.e. where python executable is located)?

It seems to work even when python is spawned as a child process by Inkscape, so should be fine for python itself, too, or am I overlooking any edge cases?

Seems to be the more reliable method that causes less headaches...

[lazka]

As long as SetDllDirectory does not undo the effect of LOAD_WITH_ALTERED_SEARCH_PATH, then that sounds more reliable.

[lazka]

To verify that put the empty LIBEAY32.dll into "/mingw64/lib/python2.7/lib-dynload/" and it should fail despite of SetDllDirectory.

[Ede123]

Ah, that might be a problem...
I think https://msdn.microsoft.com/library/windows/desktop/ms682586.aspx#alternate_search_order_for_desktop_applications describes the modes in question - according to this SetDllDirectory activates an alternate search order, too, but it differs from what LOAD_WITH_ALTERED_SEARCH_PATH achieves (notably by omitting "The directory specified by lpFileName.", i.e. the name of the DLL loaded by LoadLibraryEx).

[Ede123]

Behavior with SetDllDirectory confirmed. Search order is

C:\Program Files\Inkscape\lib\python2.7\LIBEAY32.dll
C:\Program Files\Inkscape\libeay32.dll

(i.e. SetDllDirectory in second place and "The directory from which the application loaded" in first place as documented)

On a side note: In all those cases I wonder how "C:\Program Files\Inkscape\lib\python2.7" is the first directory searched? It does contain neither the python.exe nor the _hashlib.pyd.

Edit: Considering the initial search order and the side note this seems to be actually what we want? As Python does not seem to search in "/mingw64/lib/python2.7/lib-dynload/" at all we do not really loose anything by loosing the effect of LOAD_WITH_ALTERED_SEARCH_PATH that does not really seem to have the effect of searching "The directory specified by lpFileName." in first place...

Edit2: Just to confirm this is not a random glitch with hashlib, default load order when loading C:\Program Files\Inkscape\lib\python2.7\site-packages\lxml\etree.pyd:

C:\Program Files\Inkscape\lib\python2.7\libxml2-2.dll
C:\Windows\System32\libxml2-2.dll
C:\Windows\System\libxml2-2.dll
C:\Windows\libxml2-2.dll
(and then it goes on to search on path...)

Edit3: And to recap my observation from the very first post: If I do the whole thing from within a MinGW shell python immediately loads C:\msys64\mingw64\bin\libeay32.dll - so it seems search order is already modified in this case.

[lazka]

small note: that it searches first in bin seems to be some MSYS patching, starting python with "MSYSTEM= python" makes it search in lib first, but also in lib\python2.7 and not lib-dynload, as you noted.

[Ede123]

Seems python3 is affected, too, see #3381.

Interestingly python3 does try to search in "lib-dynload/" so there seems to be a slight difference here.

Did we reach a consensus on what the best solution might be?

• I'm still unsure wether "SetDllDirectory" might have any undesired side effects by excluding the path the .dll was loaded from. Probably not for most packages but who knows if some extensions ship their dependencies in the module directory?
• A combination of LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR and LOAD_LIBRARY_SEARCH_DEFAULT_DIRS as suggested by @lazka initially therefore does indeed sound more suitable but if I understand correctly this does then disable searching on the PATH (which in turn might cause yet other problems)?.
• Do we have another possibility I overlooked?

Any thoughts?

[lazka]

Some related recent upstream changes for 3.8: python/cpython@2438cdf