PyThreadState_Clear, tstate_activate/_PyThreadState_Attach and drop_gil race race on TSAN

[Fidget-Spinner]

Bug report

Bug description:

Reproducer:
Remove all suppressions from https://github.com/python/cpython/blob/main/Tools/tsan/suppressions_free_threading.txt

Build and Run with TSAN #117657

We get races like

SUMMARY: ThreadSanitizer: data race /home/ken/Documents/GitHub/cpython/Python/pystate.c:1665:32 in PyThreadState_Clear
==================
ThreadSanitizer: reported 1 warnings
==================
WARNING: ThreadSanitizer: data race (pid=40494)
  Read of size 4 at 0x72a400014030 by main thread:
    #0 PyThreadState_Clear /home/ken/Documents/GitHub/cpython/Python/pystate.c:1661:5 (python+0x5d9458) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #1 _PyThreadState_DeleteList /home/ken/Documents/GitHub/cpython/Python/pystate.c:1917:9 (python+0x5da7f4) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #2 _Py_Finalize /home/ken/Documents/GitHub/cpython/Python/pylifecycle.c:2038:5 (python+0x59d929) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #3 Py_Exit /home/ken/Documents/GitHub/cpython/Python/pylifecycle.c:3391:9 (python+0x5a0e35) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #4 handle_system_exit /home/ken/Documents/GitHub/cpython/Python/pythonrun.c:604:9 (python+0x5e0bf9) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #5 _PyErr_PrintEx /home/ken/Documents/GitHub/cpython/Python/pythonrun.c:613:5 (python+0x5e0bf9)
    #6 PyErr_PrintEx /home/ken/Documents/GitHub/cpython/Python/pythonrun.c:690:5 (python+0x5e00aa) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #7 PyErr_Print /home/ken/Documents/GitHub/cpython/Python/pythonrun.c:696:5 (python+0x5e00aa)
    #8 _PyRun_SimpleStringFlagsWithName /home/ken/Documents/GitHub/cpython/Python/pythonrun.c:521:9 (python+0x5e00aa)
    #9 pymain_run_command /home/ken/Documents/GitHub/cpython/Modules/main.c:252:11 (python+0x6208ad) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #10 pymain_run_python /home/ken/Documents/GitHub/cpython/Modules/main.c:631:21 (python+0x6208ad)
    #11 Py_RunMain /home/ken/Documents/GitHub/cpython/Modules/main.c:719:5 (python+0x6208ad)
    #12 pymain_main /home/ken/Documents/GitHub/cpython/Modules/main.c:749:12 (python+0x620f43) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #13 Py_BytesMain /home/ken/Documents/GitHub/cpython/Modules/main.c:773:12 (python+0x620fe6) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #14 main /home/ken/Documents/GitHub/cpython/./Programs/python.c:15:12 (python+0x15b850) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)

  Previous write of size 4 at 0x72a400014030 by thread T3:
    #0 tstate_activate /home/ken/Documents/GitHub/cpython/Python/pystate.c:2004:28 (python+0x5dadbd) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #1 _PyThreadState_Attach /home/ken/Documents/GitHub/cpython/Python/pystate.c:2081:9 (python+0x5dadbd)
    #2 _Py_HandlePending /home/ken/Documents/GitHub/cpython/Python/ceval_gil.c:1234:9 (python+0x554778) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #3 _PyEval_EvalFrameDefault /home/ken/Documents/GitHub/cpython/Python/generated_cases.c.h:907:13 (python+0x4bb5f3) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)
    #4 _PyEval_EvalFrame /home/ken/Documents/GitHub/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x297c3b) (BuildId: e8ff96bc7d1e595a25cb4534a571573eaa53d4f4)

Considering it's writing to a bit struct, the race seems plausible.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[colesbury]

This is the same underlying issue as #110052 -- daemon threads (or regular threads if you press ctrl-c) may still be running during _Py_Finalize, but their PyThreadState structs get freed out from underneath them. The underlying bug affects both the default and free-threaded build.

See also my comment on that issue: #110052 (comment)

The fix is a bit tricky, so I think we should probably prioritize the other remaining data races first.