Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tool calling & argref changes #71

Merged
merged 10 commits into from
Oct 16, 2024
Merged

Tool calling & argref changes #71

merged 10 commits into from
Oct 16, 2024

Conversation

sidnarayanan
Copy link
Collaborator

These are a set of changes to tool call handling and argref_by_name that I found to help reduce agent failure rates.

Tool calling:

  • Simplify the returned Message to only be the string representation of an exception, if one is raised and handle_exc=True. The old message (a) leaked implementation details and (b) prevented us from using exceptions for control flow in tools.

argref_by_name:

  • If we have def func(a: list[int]), now func(a='foo,bar') is supported.
  • If a string was passed as a kwarg, and the string was not found in the state, then we'd pass the string directly as an argument. This was too ambiguous for agents - it is more stable to force a kwarg to either always or never be argref'd

FWIW - I think we should drop *args in argref_by_name methods, since ToolRequestMessages only support kwargs anyway. Would simplify the code significantly.

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Oct 15, 2024
@whitead
Copy link
Contributor

whitead commented Oct 15, 2024

Hi @sidnarayanan - one issue is in the design PCR, sometimes you need to pass the name of an enzyme. We can change that method to accomodate, but that flexibility for passing in a string or keyname

@@ -217,7 +217,7 @@ async def _exec_tool_call(tool_call: ToolCall) -> ToolResponseMessage:
logger.debug(str(exc), exc_info=True)
tool_exc = exc
if tool_exc:
s_content: str = f"{logger_msg}:\n{tool_exc}"
s_content = str(tool_exc)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep the above logger_msg, but just change the wording from Failed to execute tool call to something like Encountered Exception during tool call.

The reason is:

  • Sometimes people don't put a message inside of a raise (e.g. raise ValueError is valid)
  • Sometimes people don't state in the message if it's a failure or not

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I'm keeping logger_msg separate above, since we report a bit more info (exception type, tool name).

@sidnarayanan
Copy link
Collaborator Author

@whitead I see, let me think this through

@whitead
Copy link
Contributor

whitead commented Oct 15, 2024

@sidnarayanan I can just change that function to have another set of arguments - enzyme_overhang - for example.

logger_msg = (
f"Failed to execute tool call for tool {tool.info.name}: {exc!r}"
)
logger_msg = f"Encountered exception during tool call for tool {tool.info.name}: {exc!r}"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logger_msg = f"Encountered exception during tool call for tool {tool.info.name}: {exc!r}"
logger_msg = f"Encountered exception during tool {tool.info.name}: {exc!r}"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

during tool sounds odd - it's during a tool call

src/aviary/env.py Show resolved Hide resolved
src/aviary/tools/argref.py Show resolved Hide resolved
@sidnarayanan
Copy link
Collaborator Author

Per offline discussion, will modify design_primers to conform to these changes.

Copy link
Collaborator

@jamesbraza jamesbraza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

src/aviary/env.py Show resolved Hide resolved
@sidnarayanan sidnarayanan merged commit 2a3037b into main Oct 16, 2024
5 of 6 checks passed
@sidnarayanan sidnarayanan deleted the argref-tweaks branch October 16, 2024 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants