Results from running CPython test suite in a sub interpreter

[tonybaloney]

Test suite in sub interpreters

This issue is about running the CPython test suite in sub interpreters. On the current HEAD there are lots of failures, crashes and related issues.

I'm raising this issue to share the approach I used and discuss how we could automate this into CI somehow, also I want to discuss some of these tests and whether we've tracked issues to the related failures (the XML test failures are a good example)

Approach

Initially I wrote this script to iterate over the list of tests, but soon found that crashes often occur during finalize, so it made more sense to initiate them one at a time so I could capture which test suites cause a finalize crash.

All tests require faulthandler, which is not compatible with sub interpreters so I made a fake module that patches all the methods and load that into sys.modules to force the suite to execute.

Test Script

from test.libregrtest.findtests import findtests

from concurrent.futures import ThreadPoolExecutor

import _xxsubinterpreters as interpreters
import _xxinterpchannels as channels


# Get a list of tests
test_names = findtests()
skip_tests = [
    # "test_builtin" # Crashes
    "test_httpservers", # hangs?
    "test_imaplib", # hangs?
    "test.test_concurrent_futures.test_process_pool", # hangs on test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_traceback
    "test.test_multiprocessing_forkserver.test_manager", # hangs in multiple places
    "test.test_multiprocessing_forkserver.test_processes", # hangs in multiple places
    "test.test_multiprocessing_spawn.test_manager", # hangs in multiple places
]


def run_test():
    # Args- test_name: str
    import sys

    # Use a patched faulthandler because the stdlib can't be imported
    from fake_faulthandler import FaultHandler
    sys.modules['faulthandler'] = FaultHandler()

    # Force ModuleNotFoundError for datetime imports instead of a segfault
    # See GH 102995
    # sys.modules['datetime'] = None

    from test.libregrtest.main import Regrtest, Namespace

    ns = Namespace()
    ns.single = True
    ns.verbose = 1
    # Namespace attributes that are assumed to be set
    ns.fast_ci = True
    ns.args = ()
    ns.xmlpath = None
    ns.use_resources = 'all'
    ns.python = None

    r = Regrtest(ns)
    runtests = r.create_run_tests((test_name, ))
    r.run_test(test_name, runtests, None)
    # TODO : Save test result somewhere.

results = {}

# If argument then use that as the only test
import sys
if len(sys.argv) > 1:
    test_names = [sys.argv[1]]

def run_test_thread(test):
    if test in skip_tests:
        print(f"Skipping test {test}")
        return
    print(f"Running test {test}")
    interp = interpreters.create()
    channel_id = channels.create()
    try:
        result = interpreters.run_func(
                interp,
                run_test,
                shared={
                    "test_name": test,
                })
    except Exception as e:
        print(f"Test {test} failed with exception {e}")
        results[test] = {
            "result": "failed",
            "reason": f"Test {test} failed with exception {e}"
        }
    finally:
        interpreters.destroy(interp)
        channels.destroy(channel_id)

with ThreadPoolExecutor(max_workers=4) as executor:
    for test in test_names:
        executor.submit(run_test_thread, test)

Test Results

The Detail will just be 1 error message captured, for those tests there are often multiple.

Test Results

Test	Result	Detail
test.test_asyncio.test_base_events	🟢 passed [1]
test.test_asyncio.test_buffered_proto	🟢 passed [1]
test.test_asyncio.test_context	🟢 passed [1]
test.test_asyncio.test_eager_task_factory	🟢 passed [1]
test.test_asyncio.test_events	🔴 failed [1]	set_wakeup_fd only works in main thread of the main interpreter
test.test_asyncio.test_futures	🟢 passed [1]
test.test_asyncio.test_futures2	🟢 passed
test.test_asyncio.test_locks	🟢 passed [1]
test.test_asyncio.test_pep492	🟢 passed [1]
test.test_asyncio.test_proactor_events	🔴 failed [1]	set_wakeup_fd only works in main thread of the main interpreter
test.test_asyncio.test_protocols	🟢 passed
test.test_asyncio.test_queues	🟢 passed [1]
test.test_asyncio.test_runners	🔴 failed
test.test_asyncio.test_selector_events	🟢 passed [1]
test.test_asyncio.test_sendfile	🟢 passed [1]
test.test_asyncio.test_server	🔴 failed	'NoneType' object has no attribute 'default_exception_handler'
test.test_asyncio.test_sock_lowlevel	🟢 passed
test.test_asyncio.test_ssl	🔴 failed [1]	daemon threads are disabled in this
test.test_asyncio.test_sslproto	🔴 failed	daemon threads are disabled in this interpreter
test.test_asyncio.test_streams	🔴 failed [1]	daemon threads are disabled in this interpreter
test.test_asyncio.test_subprocess	🔴 failed [1]	Event loop is closed
test.test_asyncio.test_taskgroups	🟢 passed [1]
test.test_asyncio.test_tasks	🟢 passed [1]
test.test_asyncio.test_threads	🟢 passed [1]
test.test_asyncio.test_timeouts	🟢 passed [1]
test.test_asyncio.test_transports	🟢 passed
test.test_asyncio.test_unix_events	🔴 failed [1]	set_wakeup_fd only works in main thread of the main interpreter
test.test_asyncio.test_waitfor	🟢 passed [1]
test.test_asyncio.test_windows_events	🟠 skipped	Windows only
test.test_asyncio.test_windows_utils	🟠 skipped	Windows only
test.test_concurrent_futures.test_as_completed	🔴 failed	fork not supported for isolated subinterpreters
test.test_concurrent_futures.test_deadlock	🔴 failed
test.test_concurrent_futures.test_future	🟢 passed
test.test_concurrent_futures.test_init	🔴 failed
test.test_concurrent_futures.test_process_pool	🟠 skipped
test.test_concurrent_futures.test_shutdown	🔴 failed	fork not supported for isolated subinterpreters
test.test_concurrent_futures.test_thread_pool	🟢 passed
test.test_concurrent_futures.test_wait	🔴 failed	fork not supported for isolated subinterpreters
test.test_future_stmt.test_future	🟢 passed
test.test_future_stmt.test_future_flags	🟢 passed
test.test_future_stmt.test_future_multiple_features	🟢 passed
test.test_future_stmt.test_future_multiple_imports	🟢 passed
test.test_future_stmt.test_future_single_import	🟢 passed
test.test_gdb.test_backtrace	🔴 failed	'Breakpoint 1 at 0x1001f3aca
test.test_gdb.test_cfunction	🟠 skipped
test.test_gdb.test_cfunction_full	🟠 skipped
test.test_gdb.test_misc	🔴 failed	'Breakpoint 1 at 0x1001f3aca
test.test_gdb.test_pretty_print	🔴 failed
test.test_inspect.test_inspect	🔴 failed	module _testcapi does not support loading in subinterpreters
test.test_multiprocessing_fork.test_manager	🟠 skipped	test may crash on macOS (bpo-33725)
test.test_multiprocessing_fork.test_misc	🟠 skipped	test may crash on macOS (bpo-33725)
test.test_multiprocessing_fork.test_processes	🟠 skipped	test may crash on macOS (bpo-33725)
test.test_multiprocessing_fork.test_threads	🟠 skipped	test may crash on macOS (bpo-33725)
test.test_multiprocessing_forkserver.test_manager	🟠 skipped
test.test_multiprocessing_forkserver.test_misc	🔴 failed	'ForkAwareLocal' object has no attribute 'connection'
test.test_multiprocessing_forkserver.test_processes	🟠 skipped
test.test_multiprocessing_forkserver.test_threads	🔴 failed	module pyexpat does not support loading in subinterpreters
test.test_multiprocessing_spawn.test_manager	🟠 skipped
test.test_multiprocessing_spawn.test_misc	🔴 failed	'ForkAwareLocal' object has no attribute 'connection'
test.test_multiprocessing_spawn.test_processes	🔴 failed
test.test_multiprocessing_spawn.test_threads	🔴 failed	module pyexpat does not support loading in subinterpreters
test___all__	🟢 passed
test__locale	🟢 passed
test__opcode	🟢 passed [1]
test__osx_support	🔴 failed	invalid literal for int
test__xxinterpchannels	🔴 failed	channel 35 is closed
test__xxsubinterpreters	🔴 failed	'object' object has no attribute 'spam'
test_abc	🟢 passed
test_abstract_numbers	🟢 passed
test_argparse	🟢 passed
test_array	🟢 passed
test_asdl_parser	🟢 passed
test_ast	🔴 failed	module _testcapi does not support loading in subinterpreters
test_asyncgen	🟢 passed [1]
test_atexit	🟢 passed
test_audit	🟢 passed [1]
test_augassign	🟢 passed
test_base64	🟢 passed [1]
test_baseexception	🟢 passed
test_bdb	🟢 passed
test_bigaddrspace	🟢 passed
test_bigmem	🟢 passed
test_binascii	🟢 passed [1]
test_binop	🟢 passed
test_bisect	🟢 passed [1]
test_bool	🟢 passed [1]
test_buffer	🟢 passed
test_bufio	🟢 passed
test_builtin	🔴 failed	signal only works in main thread of the main interpreter
test_bytes	🟠 skipped	module _testcapi does not support loading in subinterpreters
test_bz2	🟢 passed [1]
test_c_locale_coercion	🟢 passed
test_calendar	🟢 passed
test_call	🔴 failed	'NoneType' object has no attribute 'MethInstance'
test_capi	🔴 failed	module _testcapi does not support loading in subinterpreters
test_charmapcodec	🟢 passed
test_class	🟢 passed
test_clinic	🔴 failed	Function 'group_after_parameter_on_left' has an unsupported group configuration
test_cmath	🟢 passed [1]
test_cmd	🟢 passed
test_cmd_line	🔴 failed	preexec_fn not supported within subinterpreters
test_cmd_line_script	🟢 passed
test_code	🔴 failed	module _testcapi does not support loading in subinterpreters
test_code_module	🟢 passed
test_codeccallbacks	🟢 passed
test_codecencodings_cn	🟢 passed
test_codecencodings_hk	🟢 passed
test_codecencodings_iso2022	🟢 passed
test_codecencodings_jp	🟢 passed
test_codecencodings_kr	🟢 passed
test_codecencodings_tw	🟢 passed
test_codecmaps_cn	🟢 passed [1]
test_codecmaps_hk	🟢 passed [1]
test_codecmaps_jp	🟢 passed [1]
test_codecmaps_kr	🟢 passed [1]
test_codecmaps_tw	🟢 passed [1]
test_codecs	🟢 passed
test_codeop	🟢 passed
test_collections	🟢 passed
test_colorsys	🟢 passed
test_compare	🟢 passed
test_compile	🔴 failed	maximum recursion depth exceeded during compilation
test_compileall	🔴 failed	daemon threads are disabled in this
test_compiler_assemble	🟢 passed
test_compiler_codegen	🟢 passed
test_complex	🟢 passed
test_configparser	🟢 passed
test_contains	🟢 passed
test_context	🔴 failed [1]	'_contextvars
test_contextlib	🟢 passed
test_contextlib_async	🟢 passed [1]
test_copy	🟢 passed
test_copyreg	🟢 passed
test_coroutines	🔴 failed	module _testcapi does not support loading in subinterpreters
test_cppext	🟠 skipped
test_cprofile	🔴 failed	module _lsprof does not support loading in subinterpreters
test_csv	🟢 passed [1]
test_ctypes	🟠 skipped	module _ctypes does not support loading in subinterpreters
test_curses	🟠 skipped	Use of the 'curses' resource not enabled
test_dataclasses	🟢 passed [1]
test_datetime	🔴 failed	'NoneType' object has no attribute 'dict'
test_dbm	🟢 passed
test_dbm_dumb	🟢 passed
test_dbm_gnu	🟢 passed
test_dbm_ndbm	🟢 passed
test_decimal	🟢 passed
test_decorators	🟢 passed
test_defaultdict	🟢 passed
test_deque	🟢 passed
test_descr	🟢 passed
test_descrtut	🟢 passed
test_devpoll	🟠 skipped	test works only on Solaris OS family
test_dict	🟢 passed
test_dict_version	🟠 skipped	module _testcapi does not support loading in subinterpreters
test_dictcomps	🟢 passed
test_dictviews	🟢 passed
test_difflib	🟢 passed
test_dis	🟢 passed
test_doctest	🔴 failed	Failed doctest test for test
test_doctest2	🟢 passed
test_docxmlrpc	🔴 failed	module pyexpat does not support loading in subinterpreters
test_dtrace	🟠 skipped
test_dynamic	🟢 passed
test_dynamicclassattribute	🟢 passed
test_eintr	🟠 skipped
test_email	🔴 failed	module 'datetime' has no attribute 'datetime_CAPI'
test_embed	🔴 failed
test_ensurepip	🟢 passed
test_enum	🔴 failed	'
test_enumerate	🟢 passed
test_eof	🟢 passed
test_epoll	🟠 skipped	test works only on Linux 2.6
test_errno	🟢 passed
test_except_star	🟢 passed
test_exception_group	🟢 passed
test_exception_hierarchy	🟢 passed
test_exception_variations	🟢 passed
test_exceptions	🔴 failed	module _testcapi does not support loading in subinterpreters
test_extcall	🟢 passed
test_faulthandler	🔴 failed	RuntimeError not raised
test_fcntl	🔴 failed	module _testcapi does not support loading in subinterpreters
test_file	🟢 passed
test_file_eintr	🟢 passed
test_filecmp	🟢 passed
test_fileinput	🟢 passed
test_fileio	🔴 failed	module _testcapi does not support loading in subinterpreters
test_fileutils	🟢 passed
test_finalization	🔴 failed	requires _testcapi
test_float	🟢 passed
test_flufl	🟢 passed
test_fnmatch	🟢 passed
test_fork1	🔴 failed	fork not supported for isolated subinterpreters
test_format	🔴 failed	module _testcapi does not support loading in subinterpreters
test_fractions	🟢 passed
test_frame	🟢 passed
test_frozen	🟢 passed
test_fstring	🟢 passed
test_ftplib	🔴 failed	daemon threads are disabled in this interpreter
test_funcattrs	🟢 passed
test_functools	🟢 passed
test_gc	🔴 failed	requires _testcapi
test_generated_cases	🟢 passed
test_generator_stop	🟢 passed
test_generators	🟢 passed
test_genericalias	🟢 passed [1]
test_genericclass	🔴 failed	module _testcapi does not support loading in subinterpreters
test_genericpath	🟢 passed
test_genexps	🟢 passed
test_getopt	🟢 passed
test_getpass	🟢 passed
test_getpath	🟢 passed
test_gettext	🟢 passed
test_glob	🟢 passed
test_global	🟢 passed
test_grammar	🟢 passed
test_graphlib	🟢 passed
test_grp	🟢 passed [1]
test_gzip	🟢 passed
test_hash	🟢 passed
test_hashlib	🟢 passed [1]
test_heapq	🟢 passed
test_hmac	🟢 passed [1]
test_html	🟢 passed
test_htmlparser	🟢 passed
test_http_cookiejar	🟢 passed
test_http_cookies	🟢 passed [1]
test_httplib	🔴 failed	daemon threads are disabled in this interpreter
test_httpservers	🟠 skipped
test_idle	🟠 skipped	No module named '_tkinter'
test_imaplib	🟠 skipped
test_import	🔴 failed	module _testcapi does not support loading in subinterpreters
test_importlib	🔴 failed	module _test_shared_gil_only does not support loading in subinterpreters
test_index	🟢 passed
test_int	🔴 failed	module _testcapi does not support loading in subinterpreters
test_int_literal	🟢 passed
test_interpreters	🔴 failed	too many values to unpack
test_io	🔴 failed	signal only works in main thread of the main interpreter
test_ioctl	🟠 skipped	Neither the process group nor the session are attached to /dev/tty
test_ipaddress	🟢 passed
test_isinstance	🟢 passed
test_iter	🟢 passed
test_iterlen	🟢 passed
test_itertools	🟢 passed
test_json	🟢 passed
test_keyword	🟢 passed
test_keywordonlyarg	🟢 passed
test_kqueue	🔴 failed	fork not supported for isolated subinterpreters
test_largefile	🔴 failed	signal only works in main thread of the main interpreter
test_launcher	🟠 skipped	test only applies to Windows
test_linecache	🟢 passed
test_list	🟢 passed
test_listcomps	🟢 passed
test_lltrace	🟢 passed
test_locale	🟢 passed
test_logging	🔴 failed	daemon threads are disabled in this interpreter
test_long	🟢 passed
test_longexp	🟢 passed
test_lzma	🟢 passed [1]
test_mailbox	🔴 failed	fork not supported for isolated subinterpreters
test_marshal	🟢 passed
test_math	🟢 passed
test_math_property	🟠 skipped
test_memoryio	🟢 passed
test_memoryview	🟢 passed
test_metaclass	🟢 passed
test_mimetypes	🟢 passed
test_minidom	🔴 failed	module pyexpat does not support loading in subinterpreters
test_mmap	🟢 passed
test_module	🟢 passed
test_modulefinder	🟢 passed
test_monitoring	🟢 passed
test_msvcrt	🟠 skipped	windows related tests
test_multibytecodec	🔴 failed	module _testcapi does not support loading in subinterpreters
test_multiprocessing_main_handling	🟢 passed
test_named_expressions	🟢 passed
test_netrc	🟢 passed
test_ntpath	🟢 passed
test_numeric_tower	🟢 passed
test_opcache	🟢 passed
test_opcodes	🟢 passed
test_openpty	🟢 passed
test_operator	🟢 passed
test_optparse	🟢 passed
test_ordered_dict	🟢 passed
test_os	🔴 failed	exec not supported for isolated subinterpreters
test_osx_env	🟠 skipped
test_pathlib	🟢 passed
test_patma	🟢 passed
test_pdb	🔴 failed [1]	Failed doctest test for test
test_peepholer	🟢 passed
test_peg_generator	🟠 skipped	Use of the 'cpu' resource not enabled
test_pep646_syntax	🟢 passed
test_perf_profiler	🟠 skipped	perf trampoline profiling not supported
test_perfmaps	🟠 skipped	Linux only
test_pickle	🔴 failed	object
test_picklebuffer	🟢 passed
test_pickletools	🔴 failed	object
test_pkg	🟢 passed
test_pkgutil	🟢 passed
test_platform	🔴 failed	fork not supported for isolated subinterpreters
test_plistlib	🔴 failed	module pyexpat does not support loading in subinterpreters
test_poll	🔴 failed	module _testcapi does not support loading in subinterpreters
test_popen	🟢 passed
test_poplib	🔴 failed	daemon threads are disabled in this interpreter
test_positional_only_arg	🟢 passed
test_posix	🔴 failed	signal only works in main thread of the main interpreter
test_posixpath	🟢 passed
test_pow	🟢 passed
test_pprint	🟢 passed
test_print	🟢 passed
test_profile	🟢 passed
test_property	🟢 passed
test_pstats	🔴 failed	module _lsprof does not support loading in subinterpreters
test_pty	🔴 failed	signal only works in main thread of the main interpreter
test_pulldom	🔴 failed
test_pwd	🟢 passed
test_py_compile	🟢 passed
test_pyclbr	🟢 passed
test_pydoc	🔴 failed	module _testcapi does not support loading in subinterpreters
test_pyexpat	🔴 failed	module pyexpat does not support loading in subinterpreters
test_queue	🟢 passed [1]
test_quopri	🟢 passed [1]
test_raise	🟢 passed
test_random	🔴 failed	fork not supported for isolated subinterpreters
test_range	🟢 passed
test_re	🟢 passed
test_readline	🟠 skipped	module readline does not support loading in subinterpreters
test_regrtest	🔴 failed	unexpectedly None
test_repl	🟢 passed
test_reprlib	🟢 passed
test_resource	🟢 passed
test_richcmp	🟢 passed
test_rlcompleter	🟢 passed
test_robotparser	🔴 failed	daemon threads are disabled in this interpreter
test_runpy	🟢 passed
test_sax	🟠 skipped	no XML parsers available
test_sched	🟢 passed
test_scope	🟢 passed
test_script_helper	🟢 passed
test_secrets	🟢 passed [1]
test_select	🟢 passed
test_selectors	🔴 failed	signal only works in main thread of the main interpreter
test_set	🟢 passed
test_setcomps	🟢 passed
test_shelve	🟢 passed
test_shlex	🟢 passed
test_shutil	🟢 passed
test_signal	🔴 failed	signal only works in main thread of the main interpreter
test_site	🟢 passed
test_slice	🟢 passed
test_smtplib	🟢 passed [1]
test_smtpnet	🟠 skipped [1]	Use of the 'network' resource not enabled
test_socket	🔴 failed	module _testcapi does not support loading in subinterpreters
test_socketserver	🟠 skipped	Use of the 'network' resource not enabled
test_sort	🟢 passed
test_source_encoding	🟢 passed
test_sqlite3	🔴 failed	module _testcapi does not support loading in subinterpreters
test_ssl	🔴 failed [1]	daemon threads are disabled in this interpreter
test_stable_abi_ctypes	🔴 failed	module _testcapi does not support loading in subinterpreters
test_startfile	🟠 skipped	object <module 'os' from '/Users/anthonyshaw/projects/cpython/Lib/os.py'> has no attribute 'startfile'
test_stat	🟢 passed
test_statistics	🟢 passed [1]
test_str	🔴 failed	'
test_strftime	🟢 passed
test_string	🟢 passed
test_string_literals	🟢 passed
test_stringprep	🟢 passed
test_strptime	🟢 passed
test_strtod	🟢 passed
test_struct	🟢 passed [1]
test_structseq	🟢 passed
test_subclassinit	🟢 passed
test_subprocess	🔴 failed	preexec_fn not supported within subinterpreters
test_sundry	🟢 passed
test_super	🟢 passed
test_support	🔴 failed	fork not supported for isolated subinterpreters
test_symtable	🟢 passed
test_syntax	🟢 passed
test_sys	🔴 failed	module _datetime does not support loading in subinterpreters
test_sys_setprofile	🟢 passed
test_sys_settrace	🟢 passed
test_sysconfig	🟢 passed
test_syslog	🔴 failed	subinterpreter can't use syslog
test_tabnanny	🟢 passed
test_tarfile	🟢 passed
test_tcl	🟠 skipped	No module named '_tkinter'
test_tempfile	🔴 failed	fork not supported for isolated subinterpreters
test_termios	🟢 passed
test_textwrap	🟢 passed
test_thread	🔴 failed	fork not supported for isolated subinterpreters
test_threadedtempfile	🟢 passed
test_threading	🔴 failed	fork not supported for isolated subinterpreters
test_threading_local	🟢 passed
test_threadsignals	🔴 failed	signal only works in main thread of the main interpreter
test_time	🔴 failed	invalid literal for int
test_timeit	🟢 passed
test_timeout	🟠 skipped
test_tkinter	🟠 skipped	No module named '_tkinter'
test_tokenize	🟢 passed
test_tomllib	🟢 passed
test_tools	🟢 passed [1]
test_trace	🟢 passed
test_traceback	🔴 failed	module _testcapi does not support loading in subinterpreters
test_tracemalloc	🔴 failed	fork not supported for isolated subinterpreters
test_ttk	🟠 skipped	No module named '_tkinter'
test_ttk_textonly	🟠 skipped	No module named '_tkinter'
test_tty	🟢 passed
test_tuple	🟢 passed
test_turtle	🟠 skipped	No module named '_tkinter'
test_type_aliases	🟢 passed [1]
test_type_annotations	🟢 passed
test_type_cache	🟠 skipped	module _testcapi does not support loading in subinterpreters
test_type_comments	🟢 passed
test_type_params	🟢 passed
test_typechecks	🟢 passed
test_types	🔴 failed	module _datetime does not support loading in subinterpreters
test_typing	🟢 passed
test_ucn	🟢 passed
test_unary	🟢 passed
test_unicode_file	🟢 passed
test_unicode_file_functions	🟢 passed
test_unicode_identifiers	🟢 passed
test_unicodedata	🟢 passed [1]
test_unittest	🔴 failed [1]	signal only works in main thread of the main interpreter
test_univnewlines	🟢 passed
test_unpack	🟢 passed
test_unpack_ex	🟢 passed
test_unparse	🟢 passed
test_urllib	🟢 passed [1]
test_urllib2	🟢 passed [1]
test_urllib2_localnet	🔴 failed	daemon threads are disabled in this interpreter
test_urllib2net	🟠 skipped	Use of the 'network' resource not enabled
test_urllib_response	🟢 passed
test_urllibnet	🟠 skipped [1]	Use of the 'network' resource not enabled
test_urlparse	🟢 passed
test_userdict	🟢 passed
test_userlist	🟢 passed
test_userstring	🟢 passed
test_utf8_mode	🟢 passed
test_utf8source	🟢 passed
test_uuid	🔴 failed [1]	fork not supported for isolated subinterpreters
test_venv	🟢 passed
test_wait3	🔴 failed	fork not supported for isolated subinterpreters
test_wait4	🔴 failed	fork not supported for isolated subinterpreters
test_warnings	🟢 passed
test_wave	🟢 passed
test_weakref	🔴 failed	module _testcapi does not support loading in subinterpreters
test_weakset	🟢 passed
test_webbrowser	🟢 passed
test_winconsoleio	🟠 skipped	test only relevant on win32
test_winreg	🟠 skipped	No module named 'winreg'
test_winsound	🟠 skipped	Use of the 'audio' resource not enabled
test_with	🟢 passed
test_wmi	🟠 skipped	No module named '_wmi'
test_wsgiref	🔴 failed	signal only works in main thread of the main interpreter
test_xml_dom_minicompat	🟢 passed
test_xml_etree	🔴 failed	module pyexpat does not support loading in subinterpreters
test_xml_etree_c	🔴 failed	module pyexpat does not support loading in subinterpreters
test_xmlrpc	🔴 failed	module pyexpat does not support loading in subinterpreters
test_xxlimited	🟠 skipped	module xxlimited_35 does not support loading in subinterpreters
test_xxtestfuzz	🟠 skipped	module _xxtestfuzz does not support loading in subinterpreters
test_yield_from	🟢 passed
test_zipapp	🟢 passed
test_zipfile	🟢 passed
test_zipfile64	🟠 skipped	test requires loads of disk-space bytes and a long time to run
test_zipimport	🟢 passed
test_zipimport_support	🟢 passed
test_zlib	🟢 passed [1]
test_zoneinfo	🔴 failed	module 'datetime' has no attribute 'datetime_CAPI'

[1] CPython process crashes on finalise operation, most of the cases I've debugged are because of datetime's global state already being destroyed and then CPython trying to iterate the items of a tuple in the module that has already been freed (see #112140).

To loop over a test plan (made by using -m test --list-tests) and write the results to a folder, use this script

# Read test_plan.txt and loop through each line
mkdir testresults
while IFS= read -r line
do
    rm -f testresults/$line.txt
    timeout 2m ./python.exe -X dev test_in_interp.py "$line" &> testresults/$line.txt
done < "test_plan.txt"

And finally to make this Markdown table I wrote this script:

Test Results parser

import os
import pathlib
import re

print("| Test | Result | Detail |")

# Read all txt files in testresults/
test_checks = []
for test_result_fname in pathlib.Path("testresults").glob("test*.txt"):
    test_name = test_result_fname.name.replace('.txt', '')
    test_checks.append((test_name, test_result_fname))

test_checks = sorted(test_checks, key=lambda x: x[0])

for test_name, test_result_fname in test_checks:
    print(f"| {test_name} |", end='')
    result = reason = ""
    with open(test_result_fname, encoding='utf-8', errors='ignore') as test_result:
        contents = test_result.read()

        for l in contents.splitlines():
            check = f"{test_name} skipped"
            if check in l:
                result = "🟠 skipped"
                reason = l.replace(check, '').replace(" -- ", "")

            if test_summary := re.search(r"Ran \d* test(s?) in \d*\.\d*s", l):
                if "\nOK" in contents:
                    result = "🟢 passed"
                elif "FAILED (" in contents:
                    error = re.search(r"\wError: ([\w \']*)", contents)
                    result = "🔴 failed"
                    if error:
                        reason = error.groups()[0]
                elif "NO TESTS RAN" in contents:
                    result = "🟠 skipped"
                else:
                    result = "🟠 ??? "
        if result == "" and f"Skipping test {test_name}" in contents:
            result = "🟠 skipped"
        if result == "":
            error = re.search(r"\wError: ([\w \']*)", contents)
            result = "🔴 failed"
            if error:
                reason = error.groups()[0]

        if "Debug memory block at address" in contents:
            result += " [1]"


    print(f" {result} | {reason} |")

This test was for a9574c6.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

[tonybaloney]

Related: #98627

[hugovk]

[1] CPython process crashes on finalise operation, most of the cases I've debugged are because of datetime's global state already being destroyed and then CPython trying to

This sentence is incomplete.

[tonybaloney]

[1] CPython process crashes on finalise operation, most of the cases I've debugged are because of datetime's global state already being destroyed and then CPython trying to

This sentence is incomplete.

Fixed. it was related to #112140