Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-117657: Fix data races reported by TSAN in some set methods #120914

Merged
merged 6 commits into from
Jul 1, 2024

Conversation

aisk
Copy link
Member

@aisk aisk commented Jun 23, 2024

set_add_key and set_discard_key have the same issue and should be fixed, although they have not been detected by TSAN.

After this change, the TSAN doesn't report data races for this method on my machine.

@colesbury
Copy link
Contributor

Thanks! This pattern occurs in a bunch of places outside of setobject.c. Can you fix those as well? I think it's worth fixing the entire pattern at once.

I think it's also probably worth refactoring the fast hash check into its own inline function (maybe in pycore_object.h or pycore_unicodeobject.h) that does the whole check the type, check the unicode hash, return it if it's not -1, otherwise delegate to PyObject_Hash().

@colesbury colesbury self-requested a review June 25, 2024 17:57
@aisk aisk requested a review from markshannon as a code owner July 1, 2024 17:17
@aisk
Copy link
Member Author

aisk commented Jul 1, 2024

Thanks! This pattern occurs in a bunch of places outside of setobject.c. Can you fix those as well? I think it's worth fixing the entire pattern at once.

I think it's also probably worth refactoring the fast hash check into its own inline function (maybe in pycore_object.h or pycore_unicodeobject.h) that does the whole check the type, check the unicode hash, return it if it's not -1, otherwise delegate to PyObject_Hash().

Thanks for the review! I think this new function is not related to unicode object only, so I added it in pycore_object.h, and named it _PyObject_HashFast().

@aisk aisk requested a review from methane as a code owner July 1, 2024 17:59
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - this looks good. Just minor comments about _PyObject_HashFast

Include/internal/pycore_object.h Outdated Show resolved Hide resolved
Include/internal/pycore_object.h Outdated Show resolved Hide resolved
aisk and others added 2 commits July 2, 2024 02:25
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM

@colesbury colesbury merged commit 294e724 into python:main Jul 1, 2024
41 checks passed
@miss-islington-app
Copy link

Thanks @aisk for the PR, and @colesbury for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 1, 2024
…ythonGH-120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
(cherry picked from commit 294e724)

Co-authored-by: AN Long <aisk@users.noreply.github.com>
@bedevere-app
Copy link

bedevere-app bot commented Jul 1, 2024

GH-121240 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jul 1, 2024
@vstinner
Copy link
Member

vstinner commented Jul 1, 2024

Nice change, I like _PyObject_HashFast() which factorizes the code.

colesbury pushed a commit that referenced this pull request Jul 1, 2024
…GH-120914) (#121240)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
(cherry picked from commit 294e724)

Co-authored-by: AN Long <aisk@users.noreply.github.com>
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.13 has failed when building commit 06fd745.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1441/builds/255) and take a look at the build logs.
  4. Check if the failure is related to this commit (06fd745) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1441/builds/255

Failed tests:

  • test_repl

Failed subtests:

  • test_asyncio_repl_is_ok - test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/asyncio/__main__.py", line 109, in run
    raise OSError(errno.ENOTTY, "tty required", "stdin")
xiting asyncio REPL...
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x0000557e5090ff50)


Traceback (most recent call last):
xiting asyncio REPL...
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/asyncio/__main__.py", line 109, in run
    raise OSError(errno.ENOTTY, "tty required", "stdin")
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x0000561498692f50)


Traceback (most recent call last):
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/test_repl.py", line 199, in test_asyncio_repl_is_ok
    assert_python_ok("-m", "asyncio")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 180, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 165, in _assert_python
    res.fail(cmd_line)
    ~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 75, in fail
    raise AssertionError("Process return code is %d\n"
    ...<13 lines>...
                            err))
AssertionError: Process return code is -6
command line: ['/root/buildarea/3.13.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '-m', 'asyncio']

@colesbury
Copy link
Contributor

Buildbot failure is unrelated to this PR.

See #119909

@aisk aisk deleted the tsan-set-keys branch July 2, 2024 08:30
Akasurde pushed a commit to Akasurde/cpython that referenced this pull request Jul 3, 2024
…ython#120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
noahbkim pushed a commit to hudson-trading/cpython that referenced this pull request Jul 11, 2024
…ython#120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
…ython#120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants