Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add retries to recover from realtime api text mode #1121

Merged
merged 2 commits into from
Nov 26, 2024

Conversation

longcw
Copy link
Collaborator

@longcw longcw commented Nov 21, 2024

Sometimes the real-time API responds in text mode, so a retry method needs to be added to remove the text response, append an empty user audio message, and request a new response.

The example log messages:

WARNING livekit.agents - The OpenAI Realtime API returned a text response instead of audio. Attempting to recover to audio mode... {"item_id": "item_AW3WzU2ZflaraUBqpibe3", "text": "Hello! How can I assist you with finding movies today? Please let me know the city, province, and date you're interested in.", "retries": 1}

Copy link
Member

@davidzhao davidzhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lg, nit about comments

@@ -902,6 +886,27 @@ async def set_chat_ctx(self, new_ctx: llm.ChatContext) -> None:
# wait for all the futures to complete
await asyncio.gather(*_futs)

def _create_empty_user_audio_message(self, duration: float) -> llm.ChatMessage:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add a bit of comments regarding the approach? the PR comments are helpful, but someone reading the code wouldn't be able to find it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment for the _recover_from_text_response

Copy link

changeset-bot bot commented Nov 23, 2024

🦋 Changeset detected

Latest commit: ae05954

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
livekit-plugins-openai Patch
livekit-agents Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants