bug(api): All connection attempts failed - backend API breakage

[andrewrisse]

Environment

Device and OS: Playwright tests using Chromium hitting API deployed on LF-04
App/package versions: LeapfrogAI API V9.0 and 10.0

Steps to reproduce

1. Run UI playwright tests (increase workers to 4 in playwright.config.ts) to see this bug quicker. If using 1 worker, the API is more stable and it can take many tries to re-create.

Expected result

Tests pass

Actual Result

Tests begin to fail suddenly once backend breaks and gives this error:

│ httpx.ConnectError: All connection attempts failed                                                                                                                                                                                         │
│                                                                                                                                                                                                                                            │
│ During handling of the above exception, another exception occurred:                                                                                                                                                                        │
│                                                                                                                                                                                                                                            │
│ Traceback (most recent call last):                                                                                                                                                                                                         │
│   File "/home/nonroot/.local/lib/python3.11/site-packages/leapfrogai_api/routers/supabase_session.py", line 60, in init_supabase_client                                                                                                    │
│     await client.auth.set_session(                                                                                                                                                                                                         │
│   File "/home/nonroot/.local/lib/python3.11/site-packages/gotrue/_async/gotrue_client.py", line 601, in set_session                                                                                                                        │
│     response = await self.get_user(access_token)                                                                                                                                                                                           │
│                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                           │
│   File "/home/nonroot/.local/lib/python3.11/site-packages/gotrue/_async/gotrue_client.py", line 547, in get_user                                                                                                                           │
│     return await self._request("GET", "user", jwt=jwt, xform=parse_user_response)                                                                                                                                                          │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                          │
│   File "/home/nonroot/.local/lib/python3.11/site-packages/gotrue/_async/gotrue_base_api.py", line 118, in _request                                                                                                                         │
│     raise handle_exception(e)                                                                                                                                                                                                              │
│ gotrue.errors.AuthRetryableError: All connection attempts failed

The tests will not pass until you restart the api-deployment pod as well as both supabase-kong pods.

It also appears to be triggered by Playwright E2E tests that interruupt active message streams (by clicking the pause button or reloading the page). This is not always the case, but the issue did seem to happen more when those tests were the first to start failing.

Visual Proof (screenshots, videos, text, etc)

Severity/Priority

Major - causes catastrophic failure of backend requiring reboot of multiple services

Additional Context