Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-66285: fix forking in asyncio #99539

Merged
merged 8 commits into from
Nov 24, 2022
Merged

Conversation

kumaraditya303
Copy link
Contributor

@kumaraditya303 kumaraditya303 commented Nov 16, 2022

@kumaraditya303 kumaraditya303 marked this pull request as ready for review November 17, 2022 11:11
@kumaraditya303 kumaraditya303 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 17, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit d6ef1a6 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 17, 2022
@gvanrossum
Copy link
Member

See my comments on the issue, @kumaraditya303 .

@kumaraditya303
Copy link
Contributor Author

See my comments on the issue,

@gvanrossum: I have added more tests and also added a subprocess test.

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x has failed when building commit 0c1fbc1.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/469/builds/3768) and take a look at the build logs.
  4. Check if the failure is related to this commit (0c1fbc1) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/469/builds/3768

Failed tests:

  • test_asyncio

Failed subtests:

  • test_fork_signal_handling - test.test_asyncio.test_unix_events.TestFork.test_fork_signal_handling
  • test_fork_asyncio_run - test.test_asyncio.test_unix_events.TestFork.test_fork_asyncio_run
  • test_fork_asyncio_subprocess - test.test_asyncio.test_unix_events.TestFork.test_fork_asyncio_subprocess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

416 tests OK.

10 slowest tests:

  • test_gdb: 6 min 6 sec
  • test_concurrent_futures: 2 min 46 sec
  • test_capi: 2 min 7 sec
  • test_asyncio: 1 min 57 sec
  • test_multiprocessing_spawn: 1 min 44 sec
  • test_multiprocessing_forkserver: 1 min 10 sec
  • test_multiprocessing_fork: 1 min 3 sec
  • test_tokenize: 59.6 sec
  • test_signal: 48.9 sec
  • test_socket: 46.8 sec

1 test failed:
test_asyncio

16 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_peg_generator test_startfile
test_tix test_tkinter test_ttk test_winconsoleio test_winreg
test_winsound test_wmi test_zipfile64

1 re-run test:
test_asyncio

Total duration: 10 min 8 sec

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1953, in test_fork_asyncio_subprocess
    result = manager.Value('i', 1)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
    token, exp = self._create(typeid, *args, **kwds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
    digest = hmac.new(authkey, message, 'md5').digest()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in new
    return HMAC(key, msg, digestmod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: no reason supplied


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1902, in test_fork_signal_handling
    child_started = manager.Event()
                    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
    token, exp = self._create(typeid, *args, **kwds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
    digest = hmac.new(authkey, message, 'md5').digest()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in new
    return HMAC(key, msg, digestmod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: no reason supplied


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 90, in _callTestMethod
    if self._callMaybeAsync(method) is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/unittest/async_case.py", line 117, in _callMaybeAsync
    return self._asyncioTestContext.run(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_asyncio/test_unix_events.py", line 1937, in test_fork_asyncio_run
    result = manager.Value('i', 0)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 726, in temp
    token, exp = self._create(typeid, *args, **kwds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/multiprocessing/connection.py", line 822, in answer_challenge
    digest = hmac.new(authkey, message, 'md5').digest()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 184, in new
    return HMAC(key, msg, digestmod)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/hmac.py", line 67, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: no reason supplied

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 0c1fbc1.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/954) and take a look at the build logs.
  4. Check if the failure is related to this commit (0c1fbc1) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/954

Failed tests:

  • test_traceback

Failed subtests:

  • test_import_from_error_bad_suggestions_do_not_trigger_for_small_names - test.test_traceback.CPythonSuggestionFormattingTests.test_import_from_error_bad_suggestions_do_not_trigger_for_small_names

Summary of the results of the build (if available):

== Tests result: FAILURE ==

325 tests OK.

10 slowest tests:

  • test_tokenize: 51.3 sec
  • test_unparse: 42.7 sec
  • test_capi: 33.1 sec
  • test_lib2to3: 29.2 sec
  • test_unicodedata: 25.0 sec
  • test_decimal: 16.0 sec
  • test_statistics: 14.2 sec
  • test_pickle: 13.1 sec
  • test_zipfile: 9.4 sec
  • test_buffer: 7.5 sec

1 test failed:
test_traceback

107 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio test_bz2
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_gzip test_httplib test_httpservers test_idle
test_imaplib test_interpreters test_ioctl test_kqueue
test_launcher test_lzma test_mailbox test_mmap test_msilib
test_multiprocessing_fork test_multiprocessing_forkserver
test_multiprocessing_main_handling test_multiprocessing_spawn
test_nis test_openpty test_ossaudiodev test_pdb test_peg_generator
test_perf_profiler test_pipes test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_spwd test_sqlite3 test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib test_urllib2 test_urllib2_localnet test_urllib2net
test_urllib_response test_urllibnet test_venv test_wait3
test_wait4 test_webbrowser test_winconsoleio test_winreg
test_winsound test_wmi test_wsgiref test_xmlrpc test_xmlrpc_net
test_xxlimited test_zipfile64 test_zipimport_support test_zlib
test_zoneinfo
0:04:47 load avg: 2.78
0:04:47 load avg: 2.78 Re-running failed tests is not supported with --python host runner option.

Total duration: 4 min 47 sec

Click to see traceback logs
TracebackFormatTests.test_format_stack) ... ok


TracebackFormatTests.test_exception_group_deep_recursion_traceback) ... ok


TracebackCases.test_format_exception_only_exc) ... ok


TracebackException.test_compact_no_cause) ... ok


TracebackCases.test_base_exception) ... ok


TracebackErrorCaretTests.test_specialization_variations) ... ok


TracebackException.test_locals) ... ok


TracebackFormatTests.test_print_exception_bad_type_python) ... ok


TracebackErrorCaretTests.test_caret_multiline_expression) ... ok


TracebackCases.test_print_traceback_at_exit) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_traceback_specialization_with_syntax_error) ... ok


TracebackException.test_compact_with_cause) ... ok


TracebackFormatTests.test_recursive_traceback_python) ... ok


TracebackException.test_long_context_chain) ... ok


TracebackException_ExceptionGroups.test_exception_group_format_exception_only) ... ok


TracebackFormatTests.test_print_exception_bad_type_capi) ... ok


TracebackCases.test_extract_stack) ... ok


TracebackCases.test_nocaret) ... ok


TracebackErrorCaretTests.test_multiline_method_call_a) ... ok


TracebackCases.test_no_caret_with_no_debug_ranges_flag) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_traceback_very_long_line) ... ok


TracebackErrorCaretTests.test_caret_exception_group) ... ok


TracebackErrorCaretTests.test_basic_caret) ... ok


Traceback (most recent call last):
  File "/Lib/test/test_traceback.py", line 3173, in test_import_from_error_bad_suggestions_do_not_trigger_for_small_names
    self.assertNotIn("mom", actual)
AssertionError: 'mom' unexpectedly found in "ImportError: cannot import name 'b' from 'nmQWXqKJcBDACXLN' (/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.wasi/build/build_oot/host/build/test_python_496508æ/test_python_worker_830271æ/tmp6fv8mom7/nmQWXqKJcBDACXLN.py)"


TracebackException.test_comparison_equivalent_exceptions_are_equal) ... ok


TracebackErrorCaretTests.test_multiline_method_call_c) ... ok


TracebackErrorCaretTests.test_caret_for_subscript_unicode) ... ok


TracebackCases.test_no_caret_with_no_debug_ranges_flag_python_traceback) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_multiline_method_call_b) ... ok


TracebackCases.test_recursion_error_during_traceback) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_decorator_application_lineno_correct) ... ok


TracebackErrorCaretTests.test_caret_in_type_annotation) ... ok


TracebackException.test_no_refs_to_exception_and_traceback_objects) ... ok


TracebackFormatTests.test_recursive_traceback_cpython_internal) ... ok


TracebackFormatTests.test_stack_format) ... ok


TracebackErrorCaretTests.test_wide_characters_unicode_with_problematic_byte_offset) ... ok


TracebackCases.test_caret) ... ok


TracebackCases.test_format_exception_exc) ... ok


TracebackErrorCaretTests.test_secondary_caret_not_elided) ... ok


TracebackException.test_from_exception) ... ok


TracebackCases.test_print_exception) ... ok


TracebackException.test_print) ... ok


TracebackException.test_no_locals) ... ok


TracebackCases.test_clear) ... ok


TracebackCases.test_format_exception_only_bad__str__) ... ok


TracebackCases.test_exception_is_None) ... ok


TracebackException.test_lookup_lines) ... ok


TracebackException_ExceptionGroups.test_max_group_width) ... ok


TracebackCases.test_bad_indentation) ... ok


TracebackException.test_traceback_header) ... ok


TracebackCases.test_signatures) ... ok


TracebackException_ExceptionGroups.test_exception_group_format) ... ok


TracebackFormatTests.test_print_stack) ... ok


TracebackErrorCaretTests.test_caret_for_binary_operators_with_unicode) ... ok


TracebackErrorCaretTests.test_line_with_unicode) ... ok


TracebackFormatTests.test_exception_group_deep_recursion_capi) ... ok


TracebackException.test_limit) ... ok


TracebackErrorCaretTests.test_caret_for_binary_operators) ... ok


TracebackException.test_comparison_basic) ... ok


TracebackErrorCaretTests.test_byte_offset_with_wide_characters_middle) ... ok


TracebackErrorCaretTests.test_caret_for_subscript) ... ok


TracebackException_ExceptionGroups.test_comparison) ... ok


TracebackErrorCaretTests.test_byte_offset_multiline) ... ok


TracebackException.test_smoke) ... ok


TracebackErrorCaretTests.test_caret_multiline_expression_bin_op) ... ok


TracebackCases.test_encoded_file) ... skipped 'requires subprocess support'


TracebackErrorCaretTests.test_caret_for_binary_operators_two_char) ... ok


TracebackFormatTests.test_traceback_format_with_cleared_frames) ... ok


TracebackFormatTests.test_unhashable) ... ok


TracebackException_ExceptionGroups.test_max_group_depth) ... ok


TracebackException.test_cause) ... ok


TracebackFormatTests.test_traceback_format) ... ok


TracebackException.test_comparison_params_variations) ... ok


TracebackException.test_context) ... ok


TracebackException_ExceptionGroups.test_exception_group_construction) ... ok


TracebackErrorCaretTests.test_caret_multiline_expression_syntax_error) ... ok


TracebackException.test_unhashable) ... ok


TracebackCases.test_print_exception_exc) ... ok

kumaraditya303 added a commit to kumaraditya303/cpython that referenced this pull request Nov 24, 2022
@brettcannon
Copy link
Member

Looks like the failure was random as subsequent builds succeeded.

https://buildbot.python.org/all/#/builders/1046

@vstinner
Copy link
Member

Yeah, I saw the test_traceback failure, but it only occurred once, so I ignored it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

asyncio signal handler receives signals from child processes asyncio: support multiprocessing (support fork)
5 participants