Windows error 10022 kills threads

[AdamaTheHutt]

❓ I'm submitting a ...

• 🐞 bug report
• 🐣 feature request
• ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?
Sometimes when closing a connection, we get an error, probably because it's already closed or similar. This is addressed in 384e6d0 , the problem is that in windows the error might instead be WSAEINVAL (10022), and that is not caught.

❓ What is the motivation / use case for changing the behavior?
Because of this, in combination with, I think, the error seen in #358 , we are eventually running out of threads, and the server stops responding completely.

💡 To Reproduce
Don't know how to reproduce, we get this error sporadically in production. All threads are typically gone in a few days or sometimes weeks.

💡 Expected behavior
I expect this error to be ignored, so that the thread keeps on running.

📋 Details
Also tried switching to pyopenssl, but the same error happened there too.

📋 Environment

• Cheroot version: 8.5.2
• CherryPy version: -
• Python version: 3.8.9
• OS: Microsoft Windows Server 2016 Datacenter, 10.0.14393
• Browser: [?]

📋 Additional context
A typical error:

 Exception in thread CP Server Thread-18: 
Traceback (most recent call last): 
  File "c:\py\lib\threading.py", line 917, in _bootstrap_inner 
    self.run() 
  File "c:\py\lib\site-packages\cheroot\workers\threadpool.py", line 130, in run 
    conn.close() 
  File "c:\py\lib\site-packages\cheroot\server.py", line 1360, in close 
    self._close_kernel_socket() 
  File "c:\py\lib\site-packages\cheroot\server.py", line 1483, in _close_kernel_socket 
    shutdown(socket.SHUT_RDWR)  # actually send a TCP FIN 
  File "c:\py\lib\ssl.py", line 1090, in shutdown 
    super().shutdown(how) 
OSError: [WinError 10022] An invalid argument was supplied

Then similar exceptions for thread after thread, until none are left.

In the references added in 9f9de65 there are several examples of dealing with error 10022 also, for example python/cpython@83a2c28 .

[webknjaz]

Looks like this error code is sometimes exposed as EINVAL or WSAEINVAL:

• https://github.com/python/cpython/blob/19ca114/Modules/errnomodule.c#L146-L152
• https://github.com/python/cpython/blob/19ca114/Modules/errnomodule.c#L791-L793

So it seems like maybe it should be covered already. At least in some envs. But somebody needs to check a reproducer under a relevant env. One way to do this is to connect to a GHA worker using https://github.com/marketplace/actions/debugging-with-tmate.

[AzureIP]

Hi,
I experienced this error, too. What is the current state?

[liquidaty]

Same. It is difficult to reproduce but fatal

[webknjaz]

@liquidaty I thought #649 made it better. No?

[liquidaty]

@liquidaty I thought #649 made it better. No?

I did too, but under certain conditions it appears that threads are still either dying or becoming unresponsive. Might be related to line 1291 of cheroot/server.py:

            req.respond() # <== here
            if not req.close_connection:
                return True

don't suppose there is an easy way to spit out, at regular intervals, the number of active and healthy threads?

[liquidaty]

*** UPDATED ***

@liquidaty I thought #649 made it better. No?

@webknjaz here's an example of an actual error log:

[14/Jan/2025:15:08:07] ENGINE A fatal exception happened. Setting the server interrupt flag to OSError(22, 'An invalid argument was supplied', None, 10022, None) and giving up.

Please, report this on the Cheroot tracker at <https://github.com/cherrypy/cheroot/issues/new/choose>, providing a full reproducer with as much context and details as possible.
Traceback (most recent call last):
  File "cheroot\workers\threadpool.py", line 119, in run
    self._process_connections_until_interrupted()
  File "cheroot\workers\threadpool.py", line 216, in _process_connections_until_interrupted
    conn.close()
  File "cheroot\server.py", line 1364, in close
    self._close_kernel_socket()
  File "cheroot\server.py", line 1484, in _close_kernel_socket
    shutdown(socket.SHUT_RDWR)  # actually send a TCP FIN
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ssl.py", line 1353, in shutdown
OSError: [WinError 10022] An invalid argument was supplied

Maybe this would fix, in errors.py:

acceptable_sock_shutdown_error_codes = {
    errno.ENOTCONN,
    10022, # <== add this?
    errno.EPIPE, errno.ESHUTDOWN,  # corresponds to BrokenPipeError in Python 3
    errno.ECONNRESET,  # corresponds to ConnectionResetError in Python 3
}
"""Errors that may happen during the connection close sequence.

and in server.py:

        except OSError as e:
            if e.errno == 10022:
                self.server.error_log('Ignoring OSError 10022')
            else:
                raise

[webknjaz]

Thanks for the traceback! I'm not sure yet. However, I seem to recall having some unfinished improvement attempts in this area somewhere on my laptop. I'll try to remember and see if those might be related.