Some setuptools tests run against stale artifacts

[jaraco]

In #2968, the team introduced the setuptools_wheel fixture, but due to a possible deficiency in pytest, that fixture gets used across test runs, even with a clean checkout.

I discovered this issue while working on adding some new vendored dependencies that relied on their package_data being present. During early testing, the package data was missing, but even after correcting the issue in the Setuptools build, for some reason tests relying on setuptools_wheel were still failing. During debugging, I created a breakpoint to test:

diff --git a/setuptools/tests/fixtures.py b/setuptools/tests/fixtures.py
index 9b91d7d71..d6d2be06a 100644
--- a/setuptools/tests/fixtures.py
+++ b/setuptools/tests/fixtures.py
@@ -79,6 +79,7 @@ def setuptools_sdist(tmp_path_factory, request):
 @pytest.fixture(scope="session")
 def setuptools_wheel(tmp_path_factory, request):
     with contexts.session_locked_tmp_dir(tmp_path_factory, "wheel_build") as tmp:
+        breakpoint()
         dist = next(tmp.glob("*.whl"), None)
         if dist:
             return dist

And during debugging, confirmed that the artifact existed already on the first run and had yesterday's date on it:

setuptools/tests/test_distutils_adoption.py 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/jaraco/code/public/pypa/setuptools/setuptools/tests/fixtures.py(83)setuptools_wheel()
-> dist = next(tmp.glob("*.whl"), None)
(Pdb) n
> /Users/jaraco/code/public/pypa/setuptools/setuptools/tests/fixtures.py(84)setuptools_wheel()
-> if dist:
(Pdb) dist
PosixPath('/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/wheel_build/setuptools-60.5.4.post20220128-py3-none-any.whl')

And after exiting the tests, confirmed that the artifact is still present in my temp directory:

setuptools feature/vendor-jaraco-text $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/wheel_build
setuptools-60.5.4.post20220128-py3-none-any.whl

Looking into the implementation, I see a few potential problems:

• I don't see how tmp_path_factory does any cleanup. It seems like a bad design if pytest-of-jaraco is going to get reused across runs and possible across projects.
• session_locked_temp_dir invokes mkdir directly, bypassing mktemp, the primary mechanism for getting a directory managed by the factory.
• session_locked_temp_dir operates on the parent of the temp dir, which is presumably also unmanaged.

As I think about it more, I think I understand the issue. getbasetemp() does by default create a new, numbered temporary directory for the session. Because of (3), that session is bypassed and a directory is created outside of the management of pytest, so doesn't get cleaned up and can get re-used across sessions.