[macOS] test_builtin.PtyTests.test_input_tty hangs if rlcompleter is imported

[Eclips4]

Bug report

Bug description:

./python.exe -m test
...many lines
0:05:14 load avg: 4.04 [ 87/472] test_bool
0:05:14 load avg: 4.04 [ 88/472] test_buffer
0:05:15 load avg: 3.80 [ 89/472] test_bufio
0:05:16 load avg: 3.80 [ 90/472] test_builtin
/Users/admin/Projects/cpython/Lib/pty.py:95: DeprecationWarning: This process (pid=47600) is multi-threaded, use of forkpty() may lead to deadlocks in the child.
  pid, fd = os.forkpty()

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-116402: Avoid readline in test_builtin TTY input tests #122447
• [3.13] gh-116402: Avoid readline in test_builtin TTY input tests (GH-122447) #122472

[Eclips4]

Reduced amount of tests to trigger the hang:

 ./python.exe -m test test___all__ test_builtin 
Using random seed: 2282378935
0:00:00 load avg: 1.58 Run 2 tests sequentially
0:00:00 load avg: 1.58 [1/2] test___all__
0:00:00 load avg: 1.58 [2/2] test_builtin
^C

== Tests result: INTERRUPTED ==

1 test omitted:
    test_builtin

1 test OK.

Test suite interrupted by signal SIGINT.

Total duration: 18.0 sec
Total tests: run=1
Total test files: run=1/2
Result: INTERRUPTED

[sobolevn]

Minimal number of tests:

test.test___all__.AllTest.test_all
test.test_builtin.PtyTests.test_input_tty

It gets stuck here:

cpython/Lib/test/test_builtin.py

Line 2282 in d2f1b0e

line = rpipe.readline().strip()

[sobolevn]

Since test_all actually imports all modules and executes them, the problem can be in any module :(

[sobolevn]

I think that it would be easier to bisect when running only two test cases at least!

[Eclips4]

If we skip import of rlcompleter in the test_all, then everything will be fine:

diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py
index 19dcbb207e..c5bf4bb1ff 100644
--- a/Lib/test/test___all__.py
+++ b/Lib/test/test___all__.py
@@ -111,6 +111,8 @@ def test_all(self):
         lib_dir = os.path.dirname(os.path.dirname(__file__))
         for path, modname in self.walk_modules(lib_dir, ""):
             m = modname
+            if m == "rlcompleter":
+                continue
             denied = False
             while m:
                 if m in denylist:

./python.exe -m test test___all__ test_builtin
Using random seed: 1172128194
0:00:00 load avg: 4.37 Run 2 tests sequentially
0:00:00 load avg: 4.37 [1/2] test___all__
0:00:01 load avg: 4.37 [2/2] test_builtin

== Tests result: SUCCESS ==

All 2 tests OK.

Total duration: 1.7 sec
Total tests: run=133 skipped=1
Total test files: run=2/2
Result: SUCCESS

[Eclips4]

Seems like problem in this import:
(sources of Lib/rlcompleter.py)

try:
    import readline
except ImportError:
    _readline_available = False
else:
    readline.set_completer(Completer().complete)
    # Release references early at shutdown (the readline module's
    # contents are quasi-immortal, and the completer function holds a
    # reference to globals).
    atexit.register(lambda: readline.set_completer(None))
    _readline_available = True

[vstinner]

If a test fails with a completer callback is set, you can try to:

completer = readline.get_completer()
try:
    readline.set_completer(None)
   ... # run the test
finally:
    readline.set_completer(completer)

But only do it if readline is in sys.modules.

[serhiy-storchaka]

test.test_builtin.PtyTests.test_input_tty can also be affected by other tests that import rlcompleter, and importing rlcompleter can also affect other tests. There may be several old issues which will be solved when solve this issue.

I would add a code that restores (or just resets) the completer in Lib/test/libregrtest/save_env.py.

[vstinner]

readline.set_completer(None)

If you fully want to control the TTY and not be affected by previous tests, another approach is to spawn a new process.

[ronaldoussoren]

This might be a duplicate of #89050

[JelleZijlstra]

This test hangs for me on a Mac even without running any other test:

% ./python.exe -m test test_builtin -m test_input_tty -v
== CPython 3.14.0a0 (heads/pep649-inspect:ad8c51c157, May 31 2024, 08:27:05) [Clang 14.0.0 (clang-1400.0.29.202)]
== macOS-13.4.1-arm64-arm-64bit-Mach-O little-endian
== Python build: debug
== cwd: /Users/jelle/py/cpython/build/test_python_worker_85845æ
== CPU count: 8
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1737450533
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 1.34 Run 1 test sequentially
0:00:00 load avg: 1.34 [1/1] test_builtin
test_input_tty (test.test_builtin.PtyTests.test_input_tty) ... ^C

== Tests result: INTERRUPTED ==

[JelleZijlstra]

What I'm seeing is related to #112950, which made it so pdb unconditionally imports rlcompleter; test_builtin imports doctest which imports pdb which now imports rlcompleter. However, obviously there are many code paths that could cause rlcompleter to be imported when this test runs.

I tried locally to temporarily set the readline completer to None as @vstinner suggests above, but the test still hangs. I haven't looked more deeply to figure out what's going wrong.

[AlexWaygood]

If I comment out the import of readline in rlcompleter.py, the test in test_builtin no longer hangs. But getting rid of the readline.set_completer() call in rlcompleter.py is not sufficient to stop the test from hanging: just readline being present in sys.modules is sufficient to cause the hang.

I see that many similar tests in test_builtin.py are simply skipped if "readline" is present in sys.modules:

cpython/Lib/test/test_builtin.py

Line 2400 in 64ff1e2

self.skip_if_readline()

cpython/Lib/test/test_builtin.py

Line 2405 in 64ff1e2

self.skip_if_readline()

cpython/Lib/test/test_builtin.py

Line 2415 in 64ff1e2

self.skip_if_readline()

We could consider doing something similar here, though obviously it would be nice if the test could still be run even if readline is present in sys.modules.