Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Error when aborting mid-generation with SSE enabled #81

Closed
Bassel6254 opened this issue Mar 9, 2024 · 7 comments
Closed

[BUG] Error when aborting mid-generation with SSE enabled #81

Bassel6254 opened this issue Mar 9, 2024 · 7 comments
Labels
bug Something isn't working

Comments

@Bassel6254
Copy link

Describe the bug
Getting errors when aborting mid-generation. I get this error on latest ST and another UI (Mikupad), both with SSE enabled. Works fine on Kobold Lite, but probably because it doesn't have SSE enabled. The error makes the console to hang up a little, so I have to restart the server to fix it.

To Reproduce
Steps to reproduce the behavior:

  1. Enable SSE on the UI.
  2. Generate Text
  3. Abort mid-generation.
  4. See error on console.

Logs

 + Exception Group Traceback (most recent call last):
  |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 408, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 69, in __call__
  |     return await self.app(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 84, in __call__
  |     raise exc from None
  |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 80, in __call__
  |     await self.app(scope, inner_receive, inner_send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\fastapi\applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\applications.py", line 123, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\errors.py", line 186, in __call__
  |     raise exc
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
  |     await self.app(scope, receive, _send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\cors.py", line 83, in __call__
  |     await self.app(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 758, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 778, in app
  |     await route.handle(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 299, in handle
  |     await self.app(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 79, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 77, in app
  |     await response(scope, receive, send)
  |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\responses.py", line 257, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "E:\tabbyAPI\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 678, in __aexit__
  |     raise BaseExceptionGroup(
  | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\responses.py", line 260, in wrap
    |     await func()
    |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\responses.py", line 252, in stream_response
    |     await send({"type": "http.response.body", "body": chunk, "more_body": True})
    |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 50, in sender
    |     await send(message)
    |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 50, in sender
    |     await send(message)
    |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\errors.py", line 161, in _send
    |     await send(message)
    |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 74, in inner_send
    |     await send(message)
    |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 461, in send
    |     raise ClientDisconnected
    | uvicorn.protocols.utils.ClientDisconnected
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\tabbyAPI\venv\lib\site-packages\loguru\_handler.py", line 165, in emit
    formatted = precomputed_format.format_map(formatter_record)
KeyError: '"type"'

System info (Bugs without this information will go lower on our priority list!)

  • OS: Windows 10
  • Python version: 3.10.6
  • CUDA/ROCm version: CUDA12.3

Additional context

  • This happens with SSE enabled, as it works fine with it disabled.
  • This is on a clean installation of TabbyAPI.
  • Worked fine on my PC yesterday. When booting up the server today, I noticed that it updated a requirement.
  • I was on an older version of TabbyAPI and Exl2 when I got the error first, so I made a clean install (thinking the error was because of TabbyAPI and Exl2 being a little old) but I'm still getting the error even after updating.
@Bassel6254 Bassel6254 added the bug Something isn't working label Mar 9, 2024
@Dakraid
Copy link

Dakraid commented Mar 9, 2024

Can confirm, that even when trying to reconnect, any responses from TabbyAPI seem to simply not work anymore and require a restart.

@kingbri1
Copy link
Member

kingbri1 commented Mar 9, 2024

The error should be fixed in c77259b. It was because of a logger issue.

@Bassel6254
Copy link
Author

Bassel6254 commented Mar 10, 2024 •
edited
Loading

Did a clean install with latest commit, but still getting errors when aborting mid-generation. (Tested with SillyTavern and Mikupad)

ERROR:    Exception in ASGI application
ERROR:      + Exception Group Traceback (most recent call last):
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 408, in run_asgi
ERROR:      |     result = await app(  # type: ignore
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 69, in __call__
ERROR:      |     return await self.app(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 84, in __call__
ERROR:      |     raise exc from None
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 80, in __call__
ERROR:      |     await self.app(scope, inner_receive, inner_send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\fastapi\applications.py", line 1054, in __call__
ERROR:      |     await super().__call__(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\applications.py", line 123, in __call__
ERROR:      |     await self.middleware_stack(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\errors.py", line 186, in __call__
ERROR:      |     raise exc
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
ERROR:      |     await self.app(scope, receive, _send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\cors.py", line 91, in __call__
ERROR:      |     await self.simple_response(scope, receive, send, request_headers=headers)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\cors.py", line 146, in simple_response
ERROR:      |     await self.app(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\exceptions.py", line 62, in __call__
ERROR:      |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
ERROR:      |     raise exc
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
ERROR:      |     await app(scope, receive, sender)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 758, in __call__
ERROR:      |     await self.middleware_stack(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 778, in app
ERROR:      |     await route.handle(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 299, in handle
ERROR:      |     await self.app(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 79, in app
ERROR:      |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
ERROR:      |     raise exc
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
ERROR:      |     await app(scope, receive, sender)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\routing.py", line 77, in app
ERROR:      |     await response(scope, receive, send)
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\responses.py", line 257, in __call__
ERROR:      |     async with anyio.create_task_group() as task_group:
ERROR:      |   File "E:\tabbyAPI\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 678, in __aexit__
ERROR:      |     raise BaseExceptionGroup(
ERROR:      | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
ERROR:      +-+---------------- 1 ----------------
ERROR:        | Traceback (most recent call last):
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\responses.py", line 260, in wrap
ERROR:        |     await func()
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\responses.py", line 252, in stream_response
ERROR:        |     await send({"type": "http.response.body", "body": chunk, "more_body": True})
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 50, in sender
ERROR:        |     await send(message)
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\_exception_handler.py", line 50, in sender
ERROR:        |     await send(message)
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\cors.py", line 152, in send
ERROR:        |     await send(message)
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\starlette\middleware\errors.py", line 161, in _send
ERROR:        |     await send(message)
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 74, in inner_send
ERROR:        |     await send(message)
ERROR:        |   File "E:\tabbyAPI\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 461, in send
ERROR:        |     raise ClientDisconnected
ERROR:        | uvicorn.protocols.utils.ClientDisconnected
ERROR:        +------------------------------------

@DocShotgun
Copy link
Member

Can confirm that on a69ee97, after aborting a gen in ST, any further gen attempts are just met with a code 200 OK without anything being generated. Until the server is restarted.

@kingbri1
Copy link
Member

kingbri1 commented Mar 10, 2024 •
edited
Loading

This is fixed in 6b4f100. The issue was due to Uvicorn v0.28 changing a few things under the hood. See encode/uvicorn#2220 for the large error message.

As for the generation being stuck afterwards, this is due to the semaphore getting locked up because of the exception that can't be caught.

v0.27.1 has been pinned in the requirements until further notice. Please install that version.

@Bassel6254
Copy link
Author

Thanks! Now it works fine.

@kingbri1
Copy link
Member

Update: The latest commits fix this issue on Uvicorn v0.28.0. As such, I've unpinned that dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Dakraid @kingbri1 @Bassel6254 @DocShotgun