Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Adds context variable example to tool runtime docs #6975

Merged
merged 2 commits into from
Oct 12, 2024
Merged

docs: Adds context variable example to tool runtime docs #6975

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
251c9b2
Adds context variable example to tool runtime docs
jacoblee93 Oct 11, 2024
85befad
Add langsmith version requirement
jacoblee93 Oct 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
261 changes: 218 additions & 43 deletions docs/core_docs/docs/how_to/tool_runtime.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,17 +28,13 @@
"\n",
"Most of the time, such values should not be controlled by the LLM. In fact, allowing the LLM to control the user ID may lead to a security risk.\n",
"\n",
"Instead, the LLM should only control the parameters of the tool that are meant to be controlled by the LLM, while other parameters (such as user ID) should be fixed by the application logic.\n",
"\n",
"This how-to guide shows a design pattern that creates the tool dynamically at run time and binds to them appropriate values."
"Instead, the LLM should only control the parameters of the tool that are meant to be controlled by the LLM, while other parameters (such as user ID) should be fixed by the application logic."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can bind them to chat models as follows:\n",
"\n",
"```{=mdx}\n",
"import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
"\n",
Expand All @@ -48,26 +44,226 @@
"```"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"// @lc-docs-hide-cell\n",
"\n",
"import { ChatOpenAI } from \"@langchain/openai\";\n",
"\n",
"const llm = new ChatOpenAI({ model: \"gpt-4o-mini\" })"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using context variables\n",
"\n",
"```{=mdx}\n",
":::caution Compatibility\n",
"This functionality was added in `@langchain/core>=0.3.10`. If you are using the LangSmith SDK separately in your project, we also recommend upgrading to `langsmith>=0.1.65`. Please make sure your packages are up to date.\n",
"\n",
"It also requires [`async_hooks`](https://nodejs.org/api/async_hooks.html) support, which is not supported in all environments.\n",
":::\n",
"```\n",
"\n",
"One way to solve this problem is by using **context variables**. Context variables are a powerful feature that allows you to set values at a higher level of your application, then access them within child runnable (such as tools) called from that level.\n",
"\n",
"They work outside of traditional scoping rules, so you don't need to have a direct reference to the declared variable to pass it through\n",
"\n",
"For example, you could set a user ID as a context variable at the beginning of a request, and then access it within your tools without explicitly passing it to each tool.\n",
"\n",
"For example, you could declare a tool that updates a central `userToPets` state based on a `userId` as follows:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import { z } from \"zod\";\n",
"import { tool } from \"@langchain/core/tools\";\n",
"import { getContextVariable } from \"@langchain/core/context\";\n",
"\n",
"let userToPets: Record<string, string[]> = {};\n",
"\n",
"const updateFavoritePets = tool(async (input) => {\n",
" const userId = getContextVariable(\"userId\");\n",
" if (userId === undefined) {\n",
" throw new Error(`No \"userId\" found in current context. Remember to call \"setContextVariable('userId', value)\";`);\n",
" }\n",
" userToPets[userId] = input.pets;\n",
" return \"update_favorite_pets called.\"\n",
"}, {\n",
" name: \"update_favorite_pets\",\n",
" description: \"add to the list of favorite pets.\",\n",
" schema: z.object({\n",
" pets: z.array(z.string())\n",
" }),\n",
"});"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you were to invoke the above tool before setting a context variable at a higher level, `userId` would be `undefined`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Error: No \"userId\" found in current context. Remember to call \"setContextVariable('userId', value)\";\n",
" at updateFavoritePets.name (evalmachine.<anonymous>:14:15)\n",
" at /Users/jacoblee/langchain/langchainjs/langchain-core/dist/tools/index.cjs:329:33\n",
" at AsyncLocalStorage.run (node:async_hooks:346:14)\n",
" at AsyncLocalStorageProvider.runWithConfig (/Users/jacoblee/langchain/langchainjs/langchain-core/dist/singletons/index.cjs:58:24)\n",
" at /Users/jacoblee/langchain/langchainjs/langchain-core/dist/tools/index.cjs:325:68\n",
" at new Promise (<anonymous>)\n",
" at DynamicStructuredTool.func (/Users/jacoblee/langchain/langchainjs/langchain-core/dist/tools/index.cjs:321:20)\n",
" at DynamicStructuredTool._call (/Users/jacoblee/langchain/langchainjs/langchain-core/dist/tools/index.cjs:283:21)\n",
" at DynamicStructuredTool.call (/Users/jacoblee/langchain/langchainjs/langchain-core/dist/tools/index.cjs:111:33)\n",
" at async evalmachine.<anonymous>:3:22\n"
]
}
],
"source": [
"await updateFavoritePets.invoke({ pets: [\"cat\", \"dog\" ]})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instead, set a context variable with a parent of where the tools are invoked:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import { setContextVariable } from \"@langchain/core/context\";\n",
"import { BaseChatModel } from \"@langchain/core/language_models/chat_models\";\n",
"import { RunnableLambda } from \"@langchain/core/runnables\";\n",
"\n",
"const handleRunTimeRequestRunnable = RunnableLambda.from(async (params: {\n",
" userId: string;\n",
" query: string;\n",
" llm: BaseChatModel;\n",
"}) => {\n",
" const { userId, query, llm } = params;\n",
" if (!llm.bindTools) {\n",
" throw new Error(\"Language model does not support tools.\");\n",
" }\n",
" // Set a context variable accessible to any child runnables called within this one.\n",
" // You can also set context variables at top level that act as globals.\n",
" setContextVariable(\"userId\", userId);\n",
" const tools = [updateFavoritePets];\n",
" const llmWithTools = llm.bindTools(tools);\n",
" const modelResponse = await llmWithTools.invoke(query);\n",
" // For simplicity, skip checking the tool call's name field and assume\n",
" // that the model is calling the \"updateFavoritePets\" tool\n",
" if (modelResponse.tool_calls.length > 0) {\n",
" return updateFavoritePets.invoke(modelResponse.tool_calls[0]);\n",
" } else {\n",
" return \"No tool invoked.\";\n",
" }\n",
"});"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Passing request time information\n",
"And when our method invokes the tools, you will see that the tool properly access the previously set `userId` context variable and runs successfully:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ToolMessage {\n",
" \"content\": \"update_favorite_pets called.\",\n",
" \"name\": \"update_favorite_pets\",\n",
" \"additional_kwargs\": {},\n",
" \"response_metadata\": {},\n",
" \"tool_call_id\": \"call_vsD2DbSpDquOtmFlOtbUME6h\"\n",
"}\n"
]
}
],
"source": [
"await handleRunTimeRequestRunnable.invoke({\n",
" userId: \"brace\",\n",
" query: \"my favorite animals are cats and parrots.\",\n",
" llm: llm\n",
"});"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And have additionally updated the `userToPets` object with a key matching the `userId` we passed, `\"brace\"`:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{ brace: [ 'cats', 'parrots' ] }\n"
]
}
],
"source": [
"console.log(userToPets);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Without context variables\n",
"\n",
"If you are on an earlier version of core or an environment that does not support `async_hooks`, you can use the following design pattern that creates the tool dynamically at run time and binds to them appropriate values.\n",
"\n",
"The idea is to create the tool dynamically at request time, and bind to it the appropriate information. For example,\n",
"this information may be the user ID as resolved from the request itself."
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"import { z } from \"zod\";\n",
"import { tool } from \"@langchain/core/tools\";\n",
"\n",
"const userToPets: Record<string, string[]> = {};\n",
"userToPets = {};\n",
"\n",
"function generateToolsForUser(userId: string) {\n",
" const updateFavoritePets = tool(async (input) => {\n",
Expand All @@ -80,63 +276,42 @@
" pets: z.array(z.string())\n",
" }),\n",
" });\n",
"\n",
" const deleteFavoritePets = tool(async () => {\n",
" if (userId in userToPets) {\n",
" delete userToPets[userId];\n",
" }\n",
" return \"delete_favorite_pets called.\";\n",
" }, {\n",
" name: \"delete_favorite_pets\",\n",
" description: \"Delete the list of favorite pets.\",\n",
" schema: z.object({}),\n",
" });\n",
"\n",
" const listFavoritePets = tool(async () => {\n",
" return JSON.stringify(userToPets[userId] ?? []);\n",
" }, {\n",
" name: \"list_favorite_pets\",\n",
" description: \"List favorite pets if any.\",\n",
" schema: z.object({}),\n",
" });\n",
"\n",
" return [updateFavoritePets, deleteFavoritePets, listFavoritePets];\n",
" // You can declare and return additional tools as well:\n",
" return [updateFavoritePets];\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Verify that the tools work correctly"
"Verify that the tool works correctly"
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{ brace: [ 'cat', 'dog' ] }\n",
"[\"cat\",\"dog\"]\n"
"{ cobb: [ 'tiger', 'wolf' ] }\n"
]
}
],
"source": [
"const [updatePets, deletePets, listPets] = generateToolsForUser(\"brace\");\n",
"const [updatePets] = generateToolsForUser(\"cobb\");\n",
"\n",
"await updatePets.invoke({ pets: [\"cat\", \"dog\"] });\n",
"await updatePets.invoke({ pets: [\"tiger\", \"wolf\"] });\n",
"\n",
"console.log(userToPets);\n",
"console.log(await listPets.invoke({}));"
"console.log(userToPets);"
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -156,12 +331,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This code will allow the LLM to invoke the tools, but the LLM is **unaware** of the fact that a **user ID** even exists! You can see that `user_id` is not among the params the LLM generates:"
"This code will allow the LLM to invoke the tools, but the LLM is **unaware** of the fact that a **user ID** even exists. You can see that `user_id` is not among the params the LLM generates:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 11,
"metadata": {},
"outputs": [
{
Expand All @@ -170,16 +345,16 @@
"text": [
"{\n",
" name: 'update_favorite_pets',\n",
" args: { pets: [ 'cats', 'parrots' ] },\n",
" args: { pets: [ 'tigers', 'wolves' ] },\n",
" type: 'tool_call',\n",
" id: 'call_97h0nQ3B3cr0m58HOwq9ZyUz'\n",
" id: 'call_FBF4D51SkVK2clsLOQHX6wTv'\n",
"}\n"
]
}
],
"source": [
"const aiMessage = await handleRunTimeRequest(\n",
" \"brace\", \"my favorite animals are cats and parrots.\", llm,\n",
" \"cobb\", \"my favorite pets are tigers and wolves.\", llm,\n",
");\n",
"console.log(aiMessage.tool_calls[0]);"
]
Expand Down
Loading