Handling PATH breakage on Windows with Python 3.8+

[anderslanglands]

As of Python 3.8, PATH is no longer used for loading DLLs. https://docs.python.org/3.8/whatsnew/3.8.html#bpo-36085-whatsnew

For packages that use PATH on Windows like LD_LIBRARY_PATH on Linux (all of mine are like this) I’m not really sure what to do to fix this.

One option would be to write out a sitecustomize.py that adds all the dll paths and set USER_BASE to point to it. This will probably break in certain circumstances though.

Another option would just be to static link everything on Windows.

I really hope I’m just being stupid and overlooking something obvious here!

[JeanChristopheMorinPerso]

Libraries on PyPI just call os.add_dll_directory in their __init__.py.

Other options are what you said (including all similar/derivative options) and you can also make sure the DLLs are in the same path as your Python code (if I'm not mistaken).

But since this is more a question than an issue, I'll convert it to a discussion.

[anderslanglands]

I love this. "Good news everybody! We fixed DLL hell by forcing every python package to reimplement the existing OS mechanism in their own way! Nothing can possibly go wrong with this plan..."

Looks like OpenImageIO and OpenVDB don't do it at all. OpenColorIO only does it for their test runner (and then only specifically to add the build directory).

[JeanChristopheMorinPerso]

The OCIO Python bindings are built statically, see https://github.com/AcademySoftwareFoundation/OpenColorIO/blob/main/setup.py#L79. OTIO does the same thing. OpenVDB doesn't have a python package (in the sense that they don't distribute it on PyPI) and they only seem to have 4 python files I think?

[anderslanglands]

OK but that only works for the PyPI package. The .pyd produced through the cmake build for ocio will still fail unless static linking is forced. So we're back to static linking everything :\

[JeanChristopheMorinPerso]

If you are unhappy with a package not handling DLLs correctly, open an issue with them explaining your use case. Point to mainstream packages as examples and they might add some os.add_dll_directory calls. That's how I would resolve that situation. They might have explicitly decided to not do that depending on their user base and a bunch of different reasons. For example, in OTIO we know most users simply use the wheels we publish in PyPI and we worked pretty hard to make them work on all platforms and systems from Python 2.7 to Python 3.10.

So that's really a case by case basis, but I still think it's worth discussing it with packages and libraries.

[anderslanglands]

Yep you're right. And worst comes to worst I can just patch the offending libraries at build time to do what USD is doing...

[anderslanglands]

@christian-korneck good to know that's there as a nuclear option :D I agree I wish there was an environment variable to revert to the old behaviour...

[anderslanglands]

Hmm actually @christian-korneck that doesn't seem to work. Just tried patching my python-3.9, then running it and it's still not searching PATH (looking at procmon)

[JeanChristopheMorinPerso]

When he posted the link, I looked at the code and had the impression it would only work when you load a DLL from Python. And yeah, that's the case. The CDLL class isn't involved when loading DLLs from a pyd (since a yd is just a DLL with a custom extension and some other stuff). Or there is something I'm overlooking. Like I don't remember by heart if pyds are loaded using CDLL or not.

[anderslanglands]

Yeah definitely doesn't work unfortunately

[christian-korneck]

ah yes, my usecase is only with ctypes. Haven't tested with pyd. Sorry for the confusion.