Massively reduce EncodingWarning spam in tests

[Avasam]

Summary of changes

Explicitly add encoding=locale.getpreferredencoding(False) to files opened as string. Starting Python 3.10 this can become encoding="locale"
Done by running ruff check . --select=PLW1514 --fix --unsafe-fixes (no manual code change)
I also added the check to Ruff config
Edit: This was changed by a constant that's "locale" when supported. PLW1514 also wasn't added to the Ruff config because it requires the preview flag which itself causes new lint errors unrelated to this PR

I didn't go for "utf-8" because it could be a breaking change for some projects on Windows.

Works toward #3810 (comment) (I don't think it closes, idk if there's more warnings, probably from vendored packages)

Pull Request Checklist

• Changes have tests (all existing tests should pass, but with a lot less warnings now)
• News fragment added in newsfragments/.
  (See documentation for details)

[abravalheri]

Hi @Avasam, I would like to thank you for all the contributions you are doing to setuptools. You have been very nice to the project and put a lot of effort in tackling very hard and time consuming tasks.

---

My minor comment about this are the following:

• When dealing with .pth files specifically, the advice we got from CPython maintainers is to omit the encoding parameter and ignore the warning specifically, instead of getpreferredencoding: .pth files cannot contain folders with utf-8 names python/cpython#77102 (comment).

  I while ago I left a comment in editable_wheel to remind myself about that in the future.

• .py files should always use UTF-8 right? (Because we only support Python3)

• In the tests folder, for everything else that is not .pth or is explicitly trying to test encoding, we probably can risk utf-8.

[Avasam]

Alright so doing this made me notice another issue that leads me to believe we can't completely hide these warnings in a non-breaking way to stop to spam in tests (until Python 3.9 support is dropped so encoding="locale" can be used everywhere it should):

In diffcov (which runs Python 3.12), the warning is simply changed to EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead. But since I'm only using getpreferredencoding for open, may as well use encoding="locale" at that point. (both available at Python 3.11). Which defeats the purpose of my attempted changes here.

Edit: Unless getting the default encoding is extracted to a helper method, so the getpreferredencoding warning appears only a few times.

---

• In the tests folder, for everything else that is not .pth or is explicitly trying to test encoding, we probably can risk utf-8.

I was thinking that too. Some or most tests could probably still be changed to use "utf-8" directly.

But we could also just blanket ignore this warning (if that's doable) given the special situation setuptools always has. And it would achieve the same result of improving test results readability.

---

• .py files should always use UTF-8 right? (Because we only support Python3)

I have come across python files with non-utf-8 characters in the wild, specifically this one with Windows 1252 chars that I remember tripping up Ruff:
https://github.com/mhammond/pywin32/blob/main/Pythonwin/pywin/test/_dbgscript.py

>>> open("./Pythonwin/pywin/test/_dbgscript.py").readlines()
['# -*- coding: latin-1 -*-\n', '#\n', '# Script for testing pywin.debugger & interactive exec from test_pywin\n', '\n', '# Umlauts for encoding test: áéúäöü\n', '\n', 'aa = 11\n', 'aa = 22\n', '\n', '\n', 'class CC:\n', '    cc = 44\n', '\n', '\n', 'def ff(bb=55):\n', '    global aa\n', '    aa = 77\n', '    return aa + bb\n', '\n', '\n', 'ff()\n', 'aa = 33\n']
>>> open("./Pythonwin/pywin/test/_dbgscript.py", encoding="utf-8").readlines()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 129: invalid continuation byte

And pywin32 also has these files with a different top-of-the-file encoding declaration:
https://github.com/mhammond/pywin32/blob/main/com/win32com/demos/iebutton.py
https://github.com/mhammond/pywin32/blob/main/com/win32com/demos/ietoolbar.py

Granted these are Python 2 & Internet Explorer era files, and the one with non-UTF-8 chars is in a test. But apparently they exist?

[Avasam]

All tests pass (except coverage), and you can see using the diffcov test (because they only appear in failing test) that there's 4 EncodingWarning left down from ~124 ! (2 of which are from pip and I believe are happening outside pytest, ref.: pypa/pip#12070)

That doesn't mean this is done though, I still need to:

1. Validate with you, the maintainers, whether the name and location of the new utilities is good
   2. Manually search the code for .pth to try to set the encoding explicitly to None (unless you tell me that passing tests is enough)
   3. Should I use encoding_for_open_for_mode everywhere where the mode argument to open is dynamic and could potentially be b (bytes read/write)? Rn I only added it where tests fail.

Finally, once I have a full complete solution, this could likely be split up in smaller PRs. For example to start with all changes in tests that certainly won't affect end-users.

[jaraco]

Thanks so much for doing this work. I know it's a lot.

I'd like to explore adopting the future default UTF-8 if possible.

[jaraco]

It's my understanding that encoding='locale' is essentially locking in the legacy behavior, which is UTF-8 on most systems, but cp1252 or similar on Windows. I believe the intention of the EncodingWarnings is to eventually default to UTF-8 regardless of the platform or environment. It would be my preference too if Setuptools could adopt UTF-8 and avoid the platform variance or at the very least minimize the variance to places where it's absolutely necessary.
When I made similar changes to distutils (pypa/distutils#232), I simply hard-coded "utf-8".
I still need to absorb the discussion, which I haven't the time right now, but I wanted to get this review in.

[Avasam]

Your understanding of encoding='locale' seems to align with mine. As a point of precision for this PR's goal: It's not to be a long-term fix that is forward-compatible with Python 3.15 (current ETA of UTF-8 as default), but rather just to stop the warning spam whilst keeping the current behaviour + backwards compatibility (no possible breaking change) and keeping it easy to find where the encoding is set explicitly for an eventual update to UTF-8.

I'd like to explore adopting the future default UTF-8 if possible.
It would be my preference too if Setuptools could adopt UTF-8 and avoid the platform variance or at the very least minimize the variance to places where it's absolutely necessary.

As far as going full utf-8 goes, @abravalheri suggested starting with tests, where we couldn't directly affect any user of setuptools.

[jaraco]

If the goal is to suppress the warnings, I would simply suppress them in the pytest.ini rather than coding in the less desirable behavior. At the very least, if this code is undesirable/transitional, I'd like to see a comment (probably linked to an open issue) indicating that there's anticipated work.

[Avasam]

I discovered pretty late into doing this how the pytest.ini file could be used to suppress these warnings. If you're fine suppressing them there, with an appropriate comment, to make current failed tests more readable, then I'll try to make it work that way instead.

[Avasam]

Something like this #4255 ?
If so, I'll immediately close this PR in favor of that one.

[abravalheri]

In #4261, I am testing out "utf-8" for setuptools/tests discussed in #4234 (review).

[Avasam]

As mentioned in #4255 (comment) , the vast majority of the spam has been greatly reduced. And about half of what's left should be fixed upstream (and can be ignored by that PR). Let's try to use UTF-8 where applicable for non-test changes.