Allow to set non-UTF8 exception messages

[picnixz]

Feature or enhancement

Proposal:

This is a follow-up to the discussion in #126555 (comment).

dlerror() may return non-UTF-8 messages, possibly translated. We should be able to set exception messages according to the current locale. To that end, we'll expose some internal helper:

extern void 
_PyErr_SetLocaleStringTstate(PyThreadState *tstate, PyObject *exception, const char *string);

extern void
_PyErr_SetLocaleString(PyObject *exception, const char *string);

cc @ZeroIntensity @encukou

For now, both functions would be only declared as extern and not part of the public C API.

Linked PRs

• gh-126742: allow to use non-UTF8 exception messages #126746

• [3.13] gh-126742: Add _PyErr_SetLocaleString, use it for gdbm & dlerror messages (GH-126746) #128023

• [3.12] gh-126742: Add _PyErr_SetLocaleString, use it for gdbm & dlerror messages (GH-126746) #128027

• gh-126742: avoid checking for library filename in dlerror() error messages #128034

• [3.13] gh-126742: Avoid checking for library filename in test_ctypes (GH-128034) #128056

• [3.12] gh-126742: Avoid checking for library filename in test_ctypes (GH-128034) #128057

• gh-126742: add missing NEWS entry #128025

• [3.13] gh-126742: add NEWS entry for fix of localized error messages (GH-128025) #128059

• [3.12] gh-126742: add NEWS entry for fix of localized error messages (GH-128025) #128060

[picnixz]

Let's actually mark it as a bugfix since we'll need to backport it in order to use it later.

[ZeroIntensity]

Hmm, so is PyErr_SetLocaleString supposed to be implicitly private? I'm not the biggest fan of that--if something is supposed to be private, then we need to prefix it with _, not only so users don't touch it, but so people reading the source don't mistake it as public. If we want a variation of it that takes a thread state, we could have something like _PyErr_SetLocaleStringTstate.

[picnixz]

Hmm, so is PyErr_SetLocaleString supposed to be implicitly private

For now, but maybe we can directly expose to the C API? I want to be consistent with the naming of each function.

[picnixz]

Nevermind, there is a precedent: _PyErr_FormatFromCauseTstate. Let's have
_PyErr_SetLocaleStringTstate and _PyErr_SetLocaleString and both private.

[encukou]

How many places in the code would use this helper? I don't see the advantage over calling PyUnicode_DecodeLocale and PyErr_SetObject.

[picnixz]

There are a few occurrences of dlerror() in _ctypes.c and callproc.c and dynload_shlib as well.

NVM: the dynload_shlib uses its own logic.

[encukou]

errors.c uses PyUnicode_DecodeLocale (with surrogateescape) on strerror()'s result. That looks like a good precedent.

I don't think it's worth it to add a project-wide helper here. Calling the 2 functions should be fine.

[ZeroIntensity]

My main arguments for adding a new internal function would be that a) it helps remove the risk of reference leaks (which in my experience, are all too common when dealing with string mutations) and b) makes it easier to deal with locale in the future (we have incorrect uses of dlerror() internally that skip the locale check).

[encukou]

As far as I know, this is not a bug on any supported platforms, and so can't be tested in our CI. It's also not been reported/patched on any other platform I know of.

I don't know if it should be called a “bug” (that's more a question about the definition of the term), but I don't think a fix for this issue should be backported. Let's just improve main.

As for the helper: IMO, it's better to first fix the code, and then, if it makes sense, refactor to add a helper. In clear-cut cases you'd combine the steps in one commit, but let's not do that here.

[ZeroIntensity]

As far as I know, this is not a bug on any supported platforms, and so can't be tested in our CI. It's also not been reported/patched on any other platform I know of.

FWIW, I did find BPO-41894 when looking through the places where we use PyUnicode_DecodeLocale.

[serhiy-storchaka]

I do not mind to add a private helper if it can be used in multiple files, but I am not sure about a public API. It is just a combination of two functions, and this pattern may be not so common.

Some examples in the PR looks wrong. Set an error with locale-decoded error message, then clear it and set an error with utf8-decoded error message -- this pattern looks very wrong.

[picnixz]

Set an error with locale-decoded error message, then clear it and set an error with utf8-decoded error message -- this pattern looks very wrong

This pattern is actually a way to fallback to a generic message. Otherwise, there will be no error message at all, which can be more confusing IMO. We only clear the error if the decoding fails (for whatever reason). Such pattern already existed when handling dlerror().

[picnixz]

We just merged the PR for _PyErr_SetLocaleString but it remains to address #126746 (comment). I'll do it in a follow-up PR, but anyone else is free to pick it up. I will however create a separate issue for this and close this one as being completed.

[picnixz]

I changed the labels since we decideed to eventually treat this as a bug fix.