[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) #127257

vstinner · 2024-11-25T14:51:31Z

First failure: https://buildbot.python.org/#/builders/484/builds/6224

Example of failure:

test_stls_context (test.test_poplib.TestPOP3Class.test_stls_context) ...

Warning -- Uncaught thread exception: SSLError
Exception in thread Thread-16:
Traceback (most recent call last):
  (...)
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/test_poplib.py", line 203, in handle_read
    asynchat.async_chat.handle_read(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/support/asynchat.py", line 128, in handle_read
    self.handle_error()
    ~~~~~~~~~~~~~~~~~^^
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/support/asynchat.py", line 124, in handle_read
    data = self.recv(self.ac_in_buffer_size)
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/test/support/asyncore.py", line 377, in recv
    data = self.socket.recv(buffer_size)
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/ssl.py", line 1285, in recv
    return self.read(buflen)
           ~~~~~~~~~^^^^^^^^
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64/build/Lib/ssl.py", line 1140, in read
    return self._sslobj.read(len)
           ~~~~~~~~~~~~~~~~~^^^^^
ssl.SSLError: [SYS] unknown error (_ssl.c:2634)

ok

According to @encukou, the failure started to occur after a system update:

OpenSSL 3.3.0→3.4.0
Linux 5.15.0→6.8.0
glibc 2.39→2.40
GCC 13.2.1→14.2.1
etc.

Linked PRs

The text was updated successfully, but these errors were encountered:

vstinner · 2024-11-25T14:52:11Z

cc @pablogsal

vstinner · 2024-11-25T14:57:12Z

Failing tests:

test_stls (test.test_poplib.TestPOP3Class.test_stls)
test_stls_context (test.test_poplib.TestPOP3Class.test_stls_context)

stls stands for "Start a TLS session":

    def stls(self, context=None):
        """Start a TLS session on the active connection as specified in RFC 2595.

                context - a ssl.SSLContext
        """

vstinner · 2024-11-25T15:00:52Z

First failure: https://buildbot.python.org/#/builders/484/builds/6224

That's AMD64 Arch Linux TraceRefs 3.x. test_poplib is also failing on AMD64 Arch Linux Asan 3.x, first failure: https://buildbot.python.org/#/builders/582/builds/6182

ned-deily · 2024-11-25T18:34:05Z

Are there changes/additions in the OpenSSL 3.4.x branch to its error codes that require updating our header files (see https://github.com/python/cpython/blob/main/Tools/ssl/make_ssl_data.py)?

encukou · 2024-11-27T13:24:42Z

Thanks for the pointer, @ned-deily! Here's where it led me: #127331

encukou · 2024-11-27T14:27:32Z

Oh! The [SYS] library means that the error code is a system errno and the message should be retrieved using perror rather than ERR_reason_error_string.

I wonder if this should raise an OSError using PyErr_SetFromErrno, rather than a SSLError.

The failure is an EPIPE, raised when the server is shutting down:

Fatal Python error: Aborted

Current thread 0x00007ff46ce786c0 (most recent call first):
  File "/workspace/Lib/ssl.py", line 1140 in read
  File "/workspace/Lib/ssl.py", line 1285 in recv
  File "/workspace/Lib/test/support/asyncore.py", line 377 in recv
  File "/workspace/Lib/test/test_ftplib.py", line 416 in recv
  File "/workspace/Lib/test/support/asynchat.py", line 124 in handle_read
  File "/workspace/Lib/test/test_poplib.py", line 203 in handle_read
  File "/workspace/Lib/test/support/asyncore.py", line 426 in handle_read_event
  File "/workspace/Lib/test/test_ftplib.py", line 394 in handle_read_event
  File "/workspace/Lib/test/support/asyncore.py", line 89 in read
  File "/workspace/Lib/test/support/asyncore.py", line 156 in poll
  File "/workspace/Lib/test/support/asyncore.py", line 213 in loop
  File "/workspace/Lib/test/test_poplib.py", line 235 in run
  File "/workspace/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/workspace/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007ff46e1c1740 (most recent call first):
  File "/workspace/Lib/threading.py", line 1092 in join
  File "/workspace/Lib/test/test_poplib.py", line 242 in stop
  File "/workspace/Lib/test/test_poplib.py", line 270 in tearDown
  File "/workspace/Lib/unittest/case.py", line 620 in _callTearDown
  File "/workspace/Lib/unittest/case.py", line 663 in run
  File "/workspace/Lib/unittest/case.py", line 716 in __call__
  File "/workspace/Lib/unittest/suite.py", line 122 in run
  File "/workspace/Lib/unittest/suite.py", line 84 in __call__
  File "/workspace/Lib/unittest/suite.py", line 122 in run
  File "/workspace/Lib/unittest/suite.py", line 84 in __call__
  File "/workspace/Lib/unittest/runner.py", line 240 in run
  File "/workspace/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/workspace/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/workspace/Lib/test/libregrtest/single.py", line 135 in test_func
  File "/workspace/Lib/test/libregrtest/single.py", line 91 in regrtest_runner
  File "/workspace/Lib/test/libregrtest/single.py", line 138 in _load_run_test
  File "/workspace/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc
  File "/workspace/Lib/test/libregrtest/single.py", line 281 in _runtest
  File "/workspace/Lib/test/libregrtest/single.py", line 307 in run_single_test
  File "/workspace/Lib/test/libregrtest/main.py", line 365 in run_test
  File "/workspace/Lib/test/libregrtest/main.py", line 399 in run_tests_sequentially
  File "/workspace/Lib/test/libregrtest/main.py", line 545 in _run_tests
  File "/workspace/Lib/test/libregrtest/main.py", line 580 in run_tests
  File "/workspace/Lib/test/libregrtest/main.py", line 752 in main
  File "/workspace/Lib/test/libregrtest/main.py", line 760 in main
  File "/workspace/Lib/test/__main__.py", line 2 in <module>
  File "/workspace/Lib/runpy.py", line 88 in _run_code
  File "/workspace/Lib/runpy.py", line 198 in _run_module_as_main
Aborted (core dumped)

From the ERR_raise manpage: ERR_LIB_SYS This "library code" indicates that a system error is being reported. In this case, the reason code given to `ERR_raise()` and `ERR_raise_data()` *must* be `errno(3)`. This PR only handles ERR_LIB_SYS for the high-lever error types SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where OpenSSL indicates it has some more information about the issue.

From the ERR_raise manpage: ERR_LIB_SYS This "library code" indicates that a system error is being reported. In this case, the reason code given to `ERR_raise()` and `ERR_raise_data()` *must* be `errno(3)`. This PR only handles ERR_LIB_SYS for the high-lever error types SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where OpenSSL indicates it has some more information about the issue. (cherry picked from commit f4b31ed) Co-authored-by: Petr Viktorin <encukou@gmail.com>

…127812) gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) From the ERR_raise manpage: ERR_LIB_SYS This "library code" indicates that a system error is being reported. In this case, the reason code given to `ERR_raise()` and `ERR_raise_data()` *must* be `errno(3)`. This PR only handles ERR_LIB_SYS for the high-lever error types SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where OpenSSL indicates it has some more information about the issue. (cherry picked from commit f4b31ed) Co-authored-by: Petr Viktorin <encukou@gmail.com>

From the ERR_raise manpage: ERR_LIB_SYS This "library code" indicates that a system error is being reported. In this case, the reason code given to `ERR_raise()` and `ERR_raise_data()` *must* be `errno(3)`. This PR only handles ERR_LIB_SYS for the high-lever error types SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where OpenSSL indicates it has some more information about the issue. (cherry picked from commit f4b31ed) Co-authored-by: Petr Viktorin <encukou@gmail.com>

…127905) gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) From the ERR_raise manpage: ERR_LIB_SYS This "library code" indicates that a system error is being reported. In this case, the reason code given to `ERR_raise()` and `ERR_raise_data()` *must* be `errno(3)`. This PR only handles ERR_LIB_SYS for the high-lever error types SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where OpenSSL indicates it has some more information about the issue. (cherry picked from commit f4b31ed) Co-authored-by: Petr Viktorin <encukou@gmail.com>

From the ERR_raise manpage: ERR_LIB_SYS This "library code" indicates that a system error is being reported. In this case, the reason code given to `ERR_raise()` and `ERR_raise_data()` *must* be `errno(3)`. This PR only handles ERR_LIB_SYS for the high-lever error types SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where OpenSSL indicates it has some more information about the issue.

* ssl: Add hex error code to "unknown error" messages To make it easier to vary the individual parts of the message, replace the if-ladder with constant format strings by building the string piece-wise with PyUnicodeWriter. Use "unknown error (0x%x)" rather than just "unknown error" if we can't get a better error message. (Hex makes sense as the error includes two packed parts.)

vstinner · 2025-01-10T13:12:35Z

Congrats for this fix @encukou !

vstinner added tests Tests in the Lib/test dir topic-SSL labels Nov 25, 2024

vstinner changed the title ~~[tests] test_poplib fails with "env changed" on AMD64 Arch Linux TraceRefs 3.x: [SYS] unknown error (_ssl.c:2634)~~ [tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) Nov 25, 2024

mpage mentioned this issue Nov 26, 2024

gh-115999: Enable specialization of CALL instructions in free-threaded builds #127123

Merged

encukou mentioned this issue Nov 27, 2024

gh-127330: Update for OpenSSL 3.4 #127331

Merged

encukou mentioned this issue Nov 28, 2024

gh-127257: Add hex code to ssl "unknown error" message #127360

Merged

bedevere-app bot mentioned this issue Nov 28, 2024

gh-127257: ssl: Raise OSError for ERR_LIB_SYS #127361

Merged

github-actions bot mentioned this issue Dec 1, 2024

Monthly issue metrics report hugovk/test#88

Closed

vstinner mentioned this issue Dec 3, 2024

gh-59705: Add _thread.set_name() function #127338

Merged

bedevere-app bot mentioned this issue Dec 11, 2024

[3.13] gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) #127812

Merged

bedevere-app bot mentioned this issue Dec 13, 2024

[3.12] gh-127257: ssl: Raise OSError for ERR_LIB_SYS (GH-127361) #127905

Merged

jacobtomlinson mentioned this issue Jan 10, 2025

ssl.SSLError: [SYS] unknown error (_ssl.c:2580) in test_tls_comm_closed_implicit dask/distributed#8936

Closed

encukou closed this as completed in #127360 Jan 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) #127257

[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) #127257

vstinner commented Nov 25, 2024 •

edited by bedevere-app bot

Loading

vstinner commented Nov 25, 2024

vstinner commented Nov 25, 2024

vstinner commented Nov 25, 2024

ned-deily commented Nov 25, 2024

encukou commented Nov 27, 2024

encukou commented Nov 27, 2024 •

edited

Loading

vstinner commented Jan 10, 2025

[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) #127257

[tests] test_poplib fails with "env changed" on Arch Linux with OpenSSL 3.4: [SYS] unknown error (_ssl.c:2634) #127257

Comments

vstinner commented Nov 25, 2024 • edited by bedevere-app bot Loading

Linked PRs

vstinner commented Nov 25, 2024

vstinner commented Nov 25, 2024

vstinner commented Nov 25, 2024

ned-deily commented Nov 25, 2024

encukou commented Nov 27, 2024

encukou commented Nov 27, 2024 • edited Loading

vstinner commented Jan 10, 2025

vstinner commented Nov 25, 2024 •

edited by bedevere-app bot

Loading

encukou commented Nov 27, 2024 •

edited

Loading