Integration tests failing with bad file descriptor message

[kimwnasptd]

Encountered during a PR that just updated a random text file in the repo #183

The tests were failing for Seldon with the following message:

OSError: [Errno 9] Bad file descriptor
ERROR    juju.client.connection:connection.py:619 RPC: Automatic reconnect failed
Error: The operation was canceled.

https://github.com/canonical/seldon-core-operator/actions/runs/5963901210/job/16178014925?pr=183

Full logs

WARNING  juju.client.connection:connection.py:611 RPC: Connection closed, reconnecting
WARNING  juju.client.connection:connection.py:611 RPC: Connection closed, reconnecting
ERROR    asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-2000' coro=<Connection.reconnect() done, defined at /home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py:736> exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 745, in reconnect
    res = await connector(
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 823, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 773, in _connect
    result = await task
  File "/usr/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 762, in _try_endpoint
    return await self._open(endpoint, cacert)
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line [402](https://github.com/canonical/seldon-core-operator/actions/runs/5963901210/job/16178014925?pr=183#step:5:403), in _open
    return (await websockets.connect(
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/websockets/py35/client.py", line 12, in __await_impl__
    transport, protocol = await self._creating_connection
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1066, in _create_connection_transport
    sock.setblocking(False)
OSError: [Errno 9] Bad file descriptor
ERROR    juju.client.connection:connection.py:619 RPC: Automatic reconnect failed
ERROR    asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-2001' coro=<Connection.reconnect() done, defined at /home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py:736> exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 745, in reconnect
    res = await connector(
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 823, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 773, in _connect
    result = await task
  File "/usr/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 762, in _try_endpoint
    return await self._open(endpoint, cacert)
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/juju/client/connection.py", line 402, in _open
    return (await websockets.connect(
  File "/home/runner/work/seldon-core-operator/seldon-core-operator/.tox/seldon-servers-integration/lib/python3.8/site-packages/websockets/py35/client.py", line 12, in __await_impl__
    transport, protocol = await self._creating_connection
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1066, in _create_connection_transport
    sock.setblocking(False)
OSError: [Errno 9] Bad file descriptor
ERROR    juju.client.connection:connection.py:619 RPC: Automatic reconnect failed
Error: The operation was canceled.

[i-chvets]

That error - Bad file descriptior - is an effect of tear down of test environment. Test environment is being removed when tests are finished. In this case integration tests are failing, most likely due resources.
This log is the real reason for this issue:

tests/integration/test_seldon_servers.py::test_seldon_predictor_server[MLFLOW_SERVER-mlflowserver.yaml-api/v1.0/predictions-request_data4-response_test_data4] 
-------------------------------- live log call ---------------------------------
INFO     httpx:_client.py:1013 HTTP Request: POST https://10.1.0.25:16443/apis/machinelearning.seldon.io/v1/namespaces/testing/seldondeployments?fieldManager=seldon-tests "HTTP/1.1 201 Created"
INFO     httpx:_client.py:1013 HTTP Request: GET https://10.1.0.25:16443/apis/machinelearning.seldon.io/v1/namespaces/testing/seldondeployments/mlflow "HTTP/1.1 200 OK"
INFO     test_seldon_servers:utils.py:29 seldondeployment/mlflow status == None (waiting for 'Available')
. . .
INFO     httpx:_client.py:1013 HTTP Request: GET https://10.1.0.25:16443/apis/machinelearning.seldon.io/v1/namespaces/testing/seldondeployments/mlflow "HTTP/1.1 200 OK"
INFO     test_seldon_servers:utils.py:25 Deployment of fseldondeployment/mlflow failed, status = Failed
FAILED

Integration test fail in Seldon from time to time. There was a fix for this, but it looks like Seldon still having this problem with integration tests.
#191

[kimwnasptd]

Do we delete each server after it's tested, or do we keep them all running?

seldon-core-operator/tests/integration/test_seldon_servers.py

Line 261 in f15fd7a

async def test_seldon_predictor_server(

@kimwnasptd In main we do delete servers when we done testing them. It is done through fixture.
https://github.com/canonical/seldon-core-operator/blob/main/tests/integration/test_seldon_servers.py#L268

Problem is that integration test pass locally and we need to do debugging in GH runner. The plan was to move to self-hosted runners, but we hit issues with that setup. It is documented in our Jira.

We need to dedicate more time to investigate how we can remedy the problem.

Immediate steps that we can try (no guarantee of success):

• Separate server testing even further (now it is done in its own Juju model, may be we can split them up even more to run in separate Juju models).
• Investigate if size of the ML model that is used for testing is an issue and find smaller ML models for testing.

[i-chvets]

@kimwnasptd @phoevos I re-ran actions and tests passed:
#183

Seldon tests at their current state have 90% probablity passing on weekends when, I guess, GH runners are not used as much.
We will definitely need to investigate what is going on in the runner when this test is execuited.

[orfeas-k]

Testing seldon servers in GH runners isn't be a problem anymore since we split testing each server to a different GH runner in #229. Given also that

error - Bad file descriptior - is an effect of tear down of test environment.

I 'll go ahead and close this. We can have a new issue for more specific issues with testing in GH runners.