Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support for custom toolchoice in LLMs #1102

Merged
merged 45 commits into from
Nov 26, 2024
Merged

feat: support for custom toolchoice in LLMs #1102

merged 45 commits into from
Nov 26, 2024

Conversation

jayeshp19
Copy link
Collaborator

@jayeshp19 jayeshp19 commented Nov 17, 2024 •
edited
Loading

What

This PR implements custom tool choice and parallel tool call functionality for LLMs.

Why

Feature Improvement

Example Usage

agent = VoicePipelineAgent(
    vad=ctx.proc.userdata["vad"],
    stt=deepgram.STT(),
    llm=openai.LLM(
        parallel_tool_calls=True, 
        tool_choice="auto",
    ),
    tts=openai.TTS(),
    fnc_ctx=fnc_ctx,
    chat_ctx=initial_chat_ctx,
)

Relevant Links

@changeset-bot changeset-bot
Copy link

changeset-bot bot commented Nov 17, 2024 •
edited
Loading

🦋 Changeset detected

Latest commit: 7490a95

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
livekit-plugins-anthropic Patch
livekit-plugins-openai Patch
livekit-agents Patch
livekit-plugins-llama-index Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@theomonnom
Copy link
Member

theomonnom commented Nov 18, 2024 •
edited
Loading

I think we should introduce tool_choice inside the chat method in the baseclass so we can support it for all LLMs

@jayeshp19
Copy link
Collaborator Author

jayeshp19 commented Nov 20, 2024 •
edited
Loading

@theomonnom Shouldn't these parameters be in the __init__ method of baseclass? End users should be able to configure them, right?

@theomonnom
Copy link
Member

theomonnom commented Nov 20, 2024 •
edited
Loading

@theomonnom Shouldn't these parameters be in the __init__ method of baseclass? End users should be able to configure them, right?

It should be on the chat method

@jayeshp19 jayeshp19 changed the title support for custom tooluse in anthropic support for custom tooluse in llms Nov 22, 2024
@jayeshp19 jayeshp19 changed the title support for custom tooluse in llms feat: support for custom tooluse in llms Nov 23, 2024
@jayeshp19 jayeshp19 changed the title feat: support for custom tooluse in llms feat: support for custom toolchoice in llms Nov 23, 2024
@jayeshp19 jayeshp19 changed the title feat: support for custom toolchoice in llms feat: support for custom toolchoice in LLMs Nov 23, 2024
theomonnom
theomonnom reviewed Nov 25, 2024
View reviewed changes
livekit-plugins/livekit-plugins-anthropic/livekit/plugins/anthropic/llm.py Outdated
@@ -40,11 +50,18 @@
)


class ToolChoice(TypedDict, total=False):
type: Literal["auto", "any", "tool", "none", "required"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the difference between tool and required?

What is any used for?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required (OpenAI specific param) - the model must call one or more tools.
any (Anthropic specific param) - the model must call one or more tools.
tool (Anthropic specific param) - the model must call specific tool

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we merge any and required?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can, but I think we may need to document this otherwise, it might confuse users.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated code accordingly

theomonnom
theomonnom reviewed Nov 26, 2024
View reviewed changes
livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/llm.py Outdated
@@ -578,6 +680,14 @@ def chat(

if fnc_ctx and parallel_tool_calls is not None:
opts["parallel_tool_calls"] = parallel_tool_calls
if tool_choice is not None:
if isinstance(tool_choice, dict):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if isinstance(tool_choice, dict):
if isinstance(tool_choice, FunctionToolChoice):

theomonnom
theomonnom reviewed Nov 26, 2024
View reviewed changes
Copy link
Member

@theomonnom theomonnom left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also update this class: https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-llama-index/livekit/plugins/llama_index/llm.py

@jayeshp19 @theomonnom
Update livekit-plugins/livekit-plugins-anthropic/livekit/plugins/anth…
a89616b 
…ropic/llm.py

Co-authored-by: Théo Monnom <theo.8bits@gmail.com>
theomonnom
theomonnom reviewed Nov 26, 2024
View reviewed changes
livekit-agents/livekit/agents/llm/llm.py Outdated
@@ -51,6 +60,11 @@ class ChatChunk:
usage: CompletionUsage | None = None


class FunctionToolChoice(TypedDict):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class FunctionToolChoice(TypedDict):
class ToolChoice(TypedDict):

theomonnom
theomonnom reviewed Nov 26, 2024
View reviewed changes
.changeset/real-phones-cheat.md
---
"livekit-plugins-anthropic": patch
"livekit-plugins-openai": patch
"livekit-agents": patch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add llama-index here

@jayeshp19 jayeshp19 merged commit 0194e36 into main Nov 26, 2024
16 checks passed
@jayeshp19 jayeshp19 deleted the jp/anthropic-toolchoice branch November 26, 2024 09:58
This was referenced Nov 26, 2024
Merged
Open
Open
Open
@github-actions github-actions bot mentioned this pull request Dec 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@theomonnom theomonnom theomonnom approved these changes

@davidzhao davidzhao Awaiting requested review from davidzhao

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jayeshp19 @theomonnom