EncodingWarnings in utils.subprocess and cache modules

[jaraco]

Description

Running the tests on pytest-perf with PYTHON_WARN_DEFAULT_ENCODING, I see several warnings:

pytest_perf/runner.py .../Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
/Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
/Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
/Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
/Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
/Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
/Users/jaraco/code/jaraco/pytest-perf/.tox/py/lib/python3.11/site-packages/pip/_internal/cache.py:261: EncodingWarning: 'encoding' argument not specified
  origin = DirectUrl.from_json(origin_path.read_text())

Expected behavior

pip should run clean without EncodingWarnings with PYTHON_WARN_DEFAULT_ENCODING set.

pip version

23.1.2

Python version

3.11.3

OS

macOS

How to Reproduce

1. git clone https://github.com/jaraco/pytest-perf; cd pytest-perf
2. tox -- -k runner

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[jaraco]

It looks like at least one of those uses was fixed in bd1563d... so why am I seeing the warning on pip 23.1.2?

[jaraco]

The broken usage was introduced in e744949. It was fixed again in 1ca4529.

It sure would be nice if there were some tests to enforce the desired behavior.

[pfmoore]

The difficulty with the call_subprocess case(s) is that we don't actually have any way of knowing the correct encoding. It's usually UTF-8, especially on Unix, but it may not be. On Windows, it's often the ANSI or console code page (or in the case of MSVC, I believe it can sometimes be a mixture of both 🙁 although maybe that's been fixed now). We're not yet in the "everything is UTF-8" world we want to get to.

We can of course force UTF-8 and tell users that it's not our problem if build tools don't produce that and as a result the output is mojibake. I genuinely don't know if that's the best thing to do here. Up until now, we've chosen to do whatever Python does by default - not accidentally, because we forgot to specify, but deliberately, to match the best guess of the stdlib. Personally, I dislike the fact that the stdlib is now making it impossible to explicitly say "do what the stdlib does" without getting a warning, but we have to live with that situation.

[jaraco]

impossible to explicitly say "do what the stdlib does"

I think I read somewhere that encoding="locale" will provide the legacy behavior. However, "what the stdlib does" is expected to change in Python 3.14, which is why these warnings exist - to give downstream users the ability to stipulate one path or another independent of what Python does across the transition. If we simply want to "do what stdlib does", we could suppress the warning. In my experience across dozens of libraries, doing what Python 3.14 is planning to do, which is to expect UTF-8 unless indicated otherwise, has had literally no consequence and also means there's no risk when Python 3.14 arrives. Can we try that and see if it causes any problems?