Fix CI with postgres #954
Labels
1.4.0
bug
Something isn't working
High Priority
Current Priorities & Blocking Issues
testing
testing
Comments
|
|
Cannot reproduce any more. |
|
Perhaps fixed with 1008? To be verified. EDIT: no |
|
It looks like we are really hitting this one issue of asyncpg: |
tcompa
added a commit
that referenced
this issue
Nov 21, 2023
|
Unexpectedly, this bug appeared again (ref https://github.com/fractal-analytics-platform/fractal-server/actions/runs/6966759281/job/18957429979), even after #1012. One more reason to consider #996 |
1 task
|
Possibly relevant entry in sqlalchemy 1.4.41 CHANGELOG (https://docs.sqlalchemy.org/en/20/changelog/changelog_14.html#change-1.4.41-asyncio):
|
|
(with @mfranzon) Ref: where we upgraded asyncpg from 0.27 to 0.29. Possibly related (as of 0.29): MagicStack/asyncpg#1087 and the addition of . Errors still appear when running the CI locally (see below), but they are not re-raised and the tests pass. Note the difference between Another relevant bit of info is
poetry run pytest -s --log-cli-level info tests/test_*_api.py
...
Results (78.27s):
95 passed
1 warning
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa737542080>
transport: <_SelectorSocketTransport closing fd=18>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa7378a51b0>
transport: <_SelectorSocketTransport closing fd=8>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa746081cf0>
transport: <_SelectorSocketTransport closing fd=9>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa737d1fe50>
transport: <_SelectorSocketTransport closing fd=10>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa737f59f60>
transport: <_SelectorSocketTransport closing fd=11>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed |
|
A very promising option to play with is We ran the
This suggests that |
tcompa
added a commit
that referenced
this issue
Nov 29, 2023
1 task
|
For the record, the CI can still fail - even with the |
1 task
|
As part of sqlalchemy 2.0.2, there is this change (https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-72bab6f6b9eb1662c0f5c5030a44f184):
See also the corresponding issue/discussion/commit: |
|
In #1044, by setting event pool The background error logs (as shown in #954 (comment)) are still present, but they are less worrying. For the record, even if they are not always reproducible, some tests suggest that:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See failures in #953.
For the record, as of #949 we are using sqlalchemy 1.4.50 in the CI (while we were using 1.4.41 before that PR).
Ref:
The text was updated successfully, but these errors were encountered: