Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add websocket-based streaming chat to prototype #163

Merged
merged 1 commit into from
Aug 31, 2023

Conversation

mbklein
Copy link
Contributor

@mbklein mbklein commented Aug 31, 2023

This PR abandons the idea of a streaming langchain-js Lambda Function URL (which was proving to be both complicated and unreliable) in favor of using the python-based langchain to set up an API Gateway V2 Websocket API to handle chats.

It adds:

  • A websocket API under the /chat directory, which is deployed as a sub-stack of the main dc-api stack.
  • A /chat-endpoint route to the main API, which returns the URI of the above websocket API and an authentication token to pass to it

After connecting to the websocket URI, the client can issue a message in the format:

{
  "message": "chat",
  "question": "What question should I ask?",
  "auth": "....",
  "ref": "SOME_UNIQUE_REFERENCE_STRING"
}

The socket will respond with a number of messages containing the question, the context documents, a stream of tokens making up the answer, and finally the full answer. Each of these responses will include the ref provided in the request. This allows multiple requests to run simultaneously on the same socket connection without losing track of which response goes with which request.

@mbklein mbklein temporarily deployed to test August 31, 2023 21:04 — with GitHub Actions Inactive
@bmquinn bmquinn merged commit 5c44515 into deploy/prototype Aug 31, 2023
3 checks passed
@bmquinn bmquinn deleted the prototype-streaming branch August 31, 2023 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants