Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embedded Python 3.11 on windows warns about <exec_prefix> #98790

Closed
davidhewitt opened this issue Oct 27, 2022 · 9 comments
Closed

Embedded Python 3.11 on windows warns about <exec_prefix> #98790

davidhewitt opened this issue Oct 27, 2022 · 9 comments
Assignees
Labels
3.11 only security fixes 3.12 bugs and security fixes OS-windows type-bug An unexpected behavior, bug, or error

Comments

@davidhewitt
Copy link
Contributor

davidhewitt commented Oct 27, 2022

Bug report

On Windows, with Python 3.11, an embedded Python interpreter prints a new warning at startup Could not find platform dependent libraries <exec_prefix>.

3.10 and older versions do not exhibit this.

Sample program (I'm using Rust / PyO3 because it's what I know, however I'm fairly sure that it's not relevant to the repro. If you've got a recommended way to configure a windows C toolchain for building an embedded Python I'd be happy to demonstrate the same issue in C):

fn main() {
    unsafe { 
        pyo3::ffi::Py_Initialize();
        dbg!(pyo3::ffi::Py_IsInitialized());
    }
}

Running outputs:

Could not find platform dependent libraries <exec_prefix>
[src\main.rs:4] pyo3::ffi::Py_IsInitialized() = 1

Execution then proceeds without issue. (We observe this in PyO3's test suite, which passes all 3.11 tests despite emitting this warning.)

I have a couple of thoughts:

  • Documentation for exec_prefix suggests it is irrelevant on Windows. So is this warning actually useful?
  • I suspect this is because of the reimplementation of getpath into Python? (Rewrite getpath.c in Python #89745)

Your environment

  • CPython versions tested on: 3.11.0
  • Operating system and architecture: Windows x64
@kevinf100
Copy link

kevinf100 commented Oct 29, 2022

I can confirm this isn't a PyO issue. This happening in Visual Studio running C++ code.
Calling Py_Initialize(); gets the exact same error. No matter if x86, x64, debug or release. Rebuild and changing the Include and library Directories back to 3.10 and the error goes away.

@zooba
Copy link
Member

zooba commented Oct 31, 2022

All you probably need to do is create an empty DLLs folder, or use a ._pth file. Either will suppress the warning.

Incidentally, for an embedded Python, you really want to set paths yourself. Your app is very open to attack if it's allowed to pick up libraries from anywhere on disk, and the only ways to prevent this (Py_SetPath or a ._pth file) will prevent this warning. (The warning, of course, does not mention this, because it was unintentional 😉 )

Now, for the case where we are expecting a "standard" layout and don't find it, we actually only run because Windows has always implicitly included the executable's directory in sys.path. We decided a while back that this was incorrect, but unfixable. So perhaps we ought to encode the extra path as a search for an expected .pyd file? (Which will make it even more important to override the paths when embedding, but would allow for manually rearranging into a flat layout.)

@zooba
Copy link
Member

zooba commented Oct 31, 2022

Alternatively, you can set PyConfig.pathconfig_warnings to 0 in the initialisation config.

Changing this default might be the best fix here anyway. Right now, it's based on the Py_FrozenFlag flag, which is documented as internal, but is only ever used by tests. And all it does is disable logging, rather than enabling it, which means if we default to disabled it just becomes a pointless flag. Instead, we could use Py_VerboseFlag (the -v option) to enable logging, and use that as a recommended debugging step when initialisation fails.

Not sure who has the most thoughts about this vs. changing platlib search (which may be unnecessary if we can rely on it always being relative to the stdlib directory)? Maybe @vstinner or @FFY00?

@kevinf100
Copy link

kevinf100 commented Nov 1, 2022

All you probably need to do is create an empty DLLs folder, or use a ._pth file. Either will suppress the warning.

Incidentally, for an embedded Python, you really want to set paths yourself. Your app is very open to attack if it's allowed to pick up libraries from anywhere on disk, and the only ways to prevent this (Py_SetPath or a ._pth file) will prevent this warning. (The warning, of course, does not mention this, because it was unintentional 😉 )

Now, for the case where we are expecting a "standard" layout and don't find it, we actually only run because Windows has always implicitly included the executable's directory in sys.path. We decided a while back that this was incorrect, but unfixable. So perhaps we ought to encode the extra path as a search for an expected .pyd file? (Which will make it even more important to override the paths when embedding, but would allow for manually rearranging into a flat layout.)

Py_SetPath no longer works in Python 3.11.
What is the right way to do this in Python 3.11? The following way didn't work for me
config.module_search_paths_set = 1; PyWideStringList_Append(&config.module_search_paths, L"C:\\Users\\Kevinf100\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\;C:\\Users\\Kevinf100\\source\\repos\\PythonTest\\x64\\Release");
Also Python no longer looks in the Working Directory of the .exe anymore it seems. Having the .py file next to the .exe found the .py file in 3.10, but now it no longer does in 3.11.

@zooba
Copy link
Member

zooba commented Nov 1, 2022

Py_SetPath should still work. If it's not, we need more test cases that cover your scenario (see _testembed.c for our current tests). And it certainly includes the executable path in sys.path by default, though there are some non-obvious ways to affect that once you're embedding.

I've posted two PRs - one which I think is more correct (suppressing debug messages by default), and one that just adds a Windows-specific hack to never warn about not having found DLLs. Neither is good, but I'm trying to stick to things appropriate for 3.11 at this stage.

I still think getpath.py could do with a revamp, but I don't have enough context to do it for all platforms (might be a project for @FFY00 😉 ). The aim is to have more consistency, rather than less, which is why the OS-specific hacks are all marked and are less preferred.

@kevinf100
Copy link

Py_SetPath should still work. If it's not, we need more test cases that cover your scenario (see _testembed.c for our current tests). And it certainly includes the executable path in sys.path by default, though there are some non-obvious ways to affect that once you're embedding.

I've posted two PRs - one which I think is more correct (suppressing debug messages by default), and one that just adds a Windows-specific hack to never warn about not having found DLLs. Neither is good, but I'm trying to stick to things appropriate for 3.11 at this stage.

I still think getpath.py could do with a revamp, but I don't have enough context to do it for all platforms (might be a project for @FFY00 😉 ). The aim is to have more consistency, rather than less, which is why the OS-specific hacks are all marked and are less preferred.

Looks like your right. Py_SetPath still does work. VS 2022 just doesn't compile without #pragma warning(suppress : 4996) in 3.11 it seems. Thanks. Didn't think of that without.

@FFY00
Copy link
Member

FFY00 commented Nov 1, 2022

I still think getpath.py could do with a revamp, but I don't have enough context to do it for all platforms (might be a project for @FFY00 wink ). The aim is to have more consistency, rather than less, which is why the OS-specific hacks are all marked and are less preferred.

Touching that is very tricky, and ought to break things, so I think we probably should only do it with a PEP, but yeah, I think making it more consistent across platforms would be really nice. For now, I think we should try to mimic the old initialization as best-effort, so I am in favor of the PR changing the behavior back to close of what it was.

zooba added a commit to zooba/cpython that referenced this issue Nov 1, 2022
jqnatividad added a commit to dathere/qsv that referenced this issue Nov 2, 2022
* `python`: we now link against python 3.11 - the latest stable python

* change all GH workflows to use python 3.11

* add empty DLLs directory to suppress python 3.11 warning

as per python/cpython#98790
zooba added a commit to zooba/cpython that referenced this issue Nov 2, 2022
zooba added a commit that referenced this issue Nov 2, 2022
@zooba
Copy link
Member

zooba commented Nov 2, 2022

This is resolved in 3.12 and backporting to 3.11, by changing the rule slightly on how DLLs is found on Windows. The visible changes will be:

  • DLLs is only found adjacent to the executable (consistent with 3.10)
  • if it's not found, the executable dir is added instead (in 3.10 the missing DLLs dir would still be added, and exe dir would be later)

@zooba zooba closed this as completed Nov 2, 2022
zooba added a commit to zooba/cpython that referenced this issue Nov 2, 2022
zooba added a commit that referenced this issue Nov 2, 2022
@gpshead
Copy link
Member

gpshead commented Dec 14, 2022

FYI this caused a regression in 3.11.1 that prevents some users from using it at all. #100171

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes OS-windows type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

6 participants