github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-04-29 20:35:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
62074bac60
langchain/docs/docs/how_to/tools_error.ipynb

466 lines
24 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"id": "5d60cbb9-2a6a-43ea-a9e9-f67b16ddd2b2",
"metadata": {},
"source": [
"# How to handle tool errors\n",
"\n",
":::info Prerequisites\n",
"\n",
"This guide assumes familiarity with the following concepts:\n",
"- [Chat models](/docs/concepts/chat_models)\n",
"- [LangChain Tools](/docs/concepts/tools)\n",
"- [How to use a model to call tools](/docs/how_to/tool_calling)\n",
"\n",
":::\n",
"\n",
"[Calling tools](/docs/concepts/tool_calling/) with an LLM is generally more reliable than pure prompting, but it isn't perfect. The model may try to call a tool that doesn't exist or fail to return arguments that match the requested schema. Strategies like keeping schemas simple, reducing the number of tools you pass at once, and having good names and descriptions can help mitigate this risk, but aren't foolproof.\n",
"\n",
"This guide covers some ways to build error handling into your chains to mitigate these failure modes."
]
},
{
"cell_type": "markdown",
"id": "712c774f-27c7-4351-a196-39900ca155f5",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"We'll need to install the following packages:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "63056c24-9834-4e3d-8bc5-54b1e6c5df86",
"metadata": {},
"outputs": [],
"source": [
"%pip install --upgrade --quiet langchain-core langchain-openai"
]
},
{
"cell_type": "markdown",
"id": "68107597-0c8c-4bb5-8c12-9992fabdf71a",
"metadata": {},
"source": [
"If you'd like to trace your runs in [LangSmith](https://docs.smith.langchain.com/) uncomment and set the following environment variables:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "08785b6d-722d-4620-b6ec-36deb3842c69",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:25.005243Z",
"iopub.status.busy": "2024-09-11T03:10:25.005074Z",
"iopub.status.idle": "2024-09-11T03:10:25.007679Z",
"shell.execute_reply": "2024-09-11T03:10:25.007361Z"
}
},
"outputs": [],
"source": [
"import getpass\n",
"import os\n",
"\n",
"# os.environ[\"LANGSMITH_TRACING\"] = \"true\"\n",
"# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass()"
]
},
{
"cell_type": "markdown",
"id": "0a50f93a-5d6f-4691-8f98-27239a1c2f95",
"metadata": {},
"source": [
"## Chain\n",
"\n",
"Suppose we have the following (dummy) tool and tool-calling chain. We'll make our tool intentionally convoluted to try and trip up the model.\n",
"\n",
"import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
"\n",
"<ChatModelTabs customVarName=\"llm\"/>\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "86258950-5e61-4340-81b9-84a5d26e8773",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:25.009496Z",
"iopub.status.busy": "2024-09-11T03:10:25.009371Z",
"iopub.status.idle": "2024-09-11T03:10:25.552917Z",
"shell.execute_reply": "2024-09-11T03:10:25.552592Z"
}
},
"outputs": [],
"source": [
"# | echo: false\n",
"# | output: false\n",
"\n",
"from langchain_openai import ChatOpenAI\n",
"\n",
"if \"OPENAI_API_KEY\" not in os.environ:\n",
" os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()\n",
"\n",
"llm = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "1d20604e-c4d1-4d21-841b-23e4f61aec36",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:25.554543Z",
"iopub.status.busy": "2024-09-11T03:10:25.554439Z",
"iopub.status.idle": "2024-09-11T03:10:25.631610Z",
"shell.execute_reply": "2024-09-11T03:10:25.631346Z"
}
},
"outputs": [],
"source": [
"# Define tool\n",
"from langchain_core.tools import tool\n",
"\n",
"\n",
"@tool\n",
"def complex_tool(int_arg: int, float_arg: float, dict_arg: dict) -> int:\n",
" \"\"\"Do something complex with a complex tool.\"\"\"\n",
" return int_arg * float_arg\n",
"\n",
"\n",
"llm_with_tools = llm.bind_tools(\n",
" [complex_tool],\n",
")\n",
"\n",
"# Define chain\n",
"chain = llm_with_tools | (lambda msg: msg.tool_calls[0][\"args\"]) | complex_tool"
]
},
{
"cell_type": "markdown",
"id": "c34f005e-63f0-4841-9461-ca36c36607fc",
"metadata": {},
"source": [
"We can see that when we try to invoke this chain with even a fairly explicit input, the model fails to correctly call the tool (it forgets the `dict_arg` argument)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "d354664c-ac44-4967-a35f-8912b3ad9477",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:25.633050Z",
"iopub.status.busy": "2024-09-11T03:10:25.632978Z",
"iopub.status.idle": "2024-09-11T03:10:26.556508Z",
"shell.execute_reply": "2024-09-11T03:10:26.556233Z"
}
},
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for complex_toolSchema\ndict_arg\n Field required [type=missing, input_value={'int_arg': 5, 'float_arg': 2.1}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.8/v/missing",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[5], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mchain\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43minvoke\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43muse complex tool. the args are 5, 2.1, empty dictionary. don\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mt forget dict_arg\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\n\u001b[1;32m 3\u001b[0m \u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/langchain_core/runnables/base.py:2998\u001b[0m, in \u001b[0;36mRunnableSequence.invoke\u001b[0;34m(self, input, config, **kwargs)\u001b[0m\n\u001b[1;32m 2996\u001b[0m \u001b[38;5;28minput\u001b[39m \u001b[38;5;241m=\u001b[39m context\u001b[38;5;241m.\u001b[39mrun(step\u001b[38;5;241m.\u001b[39minvoke, \u001b[38;5;28minput\u001b[39m, config, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 2997\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 2998\u001b[0m \u001b[38;5;28minput\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[43mcontext\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mstep\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43minvoke\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mconfig\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 2999\u001b[0m \u001b[38;5;66;03m# finish the root run\u001b[39;00m\n\u001b[1;32m 3000\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/langchain_core/tools/base.py:456\u001b[0m, in \u001b[0;36mBaseTool.invoke\u001b[0;34m(self, input, config, **kwargs)\u001b[0m\n\u001b[1;32m 449\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21minvoke\u001b[39m(\n\u001b[1;32m 450\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 451\u001b[0m \u001b[38;5;28minput\u001b[39m: Union[\u001b[38;5;28mstr\u001b[39m, Dict, ToolCall],\n\u001b[1;32m 452\u001b[0m config: Optional[RunnableConfig] \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[1;32m 453\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any,\n\u001b[1;32m 454\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[1;32m 455\u001b[0m tool_input, kwargs \u001b[38;5;241m=\u001b[39m _prep_run_args(\u001b[38;5;28minput\u001b[39m, config, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m--> 456\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtool_input\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/langchain_core/tools/base.py:659\u001b[0m, in \u001b[0;36mBaseTool.run\u001b[0;34m(self, tool_input, verbose, start_color, color, callbacks, tags, metadata, run_name, run_id, config, tool_call_id, **kwargs)\u001b[0m\n\u001b[1;32m 657\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m error_to_raise:\n\u001b[1;32m 658\u001b[0m run_manager\u001b[38;5;241m.\u001b[39mon_tool_error(error_to_raise)\n\u001b[0;32m--> 659\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m error_to_raise\n\u001b[1;32m 660\u001b[0m output \u001b[38;5;241m=\u001b[39m _format_output(content, artifact, tool_call_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname, status)\n\u001b[1;32m 661\u001b[0m run_manager\u001b[38;5;241m.\u001b[39mon_tool_end(output, color\u001b[38;5;241m=\u001b[39mcolor, name\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/langchain_core/tools/base.py:622\u001b[0m, in \u001b[0;36mBaseTool.run\u001b[0;34m(self, tool_input, verbose, start_color, color, callbacks, tags, metadata, run_name, run_id, config, tool_call_id, **kwargs)\u001b[0m\n\u001b[1;32m 620\u001b[0m context \u001b[38;5;241m=\u001b[39m copy_context()\n\u001b[1;32m 621\u001b[0m context\u001b[38;5;241m.\u001b[39mrun(_set_config_context, child_config)\n\u001b[0;32m--> 622\u001b[0m tool_args, tool_kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_to_args_and_kwargs\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtool_input\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 623\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m signature(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_run)\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mrun_manager\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[1;32m 624\u001b[0m tool_kwargs[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mrun_manager\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m run_manager\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/langchain_core/tools/base.py:545\u001b[0m, in \u001b[0;36mBaseTool._to_args_and_kwargs\u001b[0;34m(self, tool_input)\u001b[0m\n\u001b[1;32m 544\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_to_args_and_kwargs\u001b[39m(\u001b[38;5;28mself\u001b[39m, tool_input: Union[\u001b[38;5;28mstr\u001b[39m, Dict]) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Tuple[Tuple, Dict]:\n\u001b[0;32m--> 545\u001b[0m tool_input \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_parse_input\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtool_input\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 546\u001b[0m \u001b[38;5;66;03m# For backwards compatibility, if run_input is a string,\u001b[39;00m\n\u001b[1;32m 547\u001b[0m \u001b[38;5;66;03m# pass as a positional argument.\u001b[39;00m\n\u001b[1;32m 548\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(tool_input, \u001b[38;5;28mstr\u001b[39m):\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/langchain_core/tools/base.py:487\u001b[0m, in \u001b[0;36mBaseTool._parse_input\u001b[0;34m(self, tool_input)\u001b[0m\n\u001b[1;32m 485\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m input_args \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 486\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28missubclass\u001b[39m(input_args, BaseModel):\n\u001b[0;32m--> 487\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[43minput_args\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmodel_validate\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtool_input\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 488\u001b[0m result_dict \u001b[38;5;241m=\u001b[39m result\u001b[38;5;241m.\u001b[39mmodel_dump()\n\u001b[1;32m 489\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28missubclass\u001b[39m(input_args, BaseModelV1):\n",
"File \u001b[0;32m~/langchain/.venv/lib/python3.11/site-packages/pydantic/main.py:568\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m 566\u001b[0m \u001b[38;5;66;03m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 567\u001b[0m __tracebackhide__ \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 568\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mcls\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__pydantic_validator__\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvalidate_python\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 569\u001b[0m \u001b[43m \u001b[49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstrict\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstrict\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfrom_attributes\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfrom_attributes\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcontext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcontext\u001b[49m\n\u001b[1;32m 570\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for complex_toolSchema\ndict_arg\n Field required [type=missing, input_value={'int_arg': 5, 'float_arg': 2.1}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.8/v/missing"
]
}
],
"source": [
"chain.invoke(\n",
" \"use complex tool. the args are 5, 2.1, empty dictionary. don't forget dict_arg\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "890d989d-2d39-4571-9a55-d3496b9b5d27",
"metadata": {},
"source": [
"## Try/except tool call\n",
"\n",
"The simplest way to more gracefully handle errors is to try/except the tool-calling step and return a helpful message on errors:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "8fedb550-683d-45ae-8876-ae7acb332019",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:26.558131Z",
"iopub.status.busy": "2024-09-11T03:10:26.558031Z",
"iopub.status.idle": "2024-09-11T03:10:27.399844Z",
"shell.execute_reply": "2024-09-11T03:10:27.399201Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calling tool with arguments:\n",
"\n",
"{'int_arg': 5, 'float_arg': 2.1}\n",
"\n",
"raised the following error:\n",
"\n",
"<class 'pydantic_core._pydantic_core.ValidationError'>: 1 validation error for complex_toolSchema\n",
"dict_arg\n",
" Field required [type=missing, input_value={'int_arg': 5, 'float_arg': 2.1}, input_type=dict]\n",
" For further information visit https://errors.pydantic.dev/2.8/v/missing\n"
]
}
],
"source": [
"from typing import Any\n",
"\n",
"from langchain_core.runnables import Runnable, RunnableConfig\n",
"\n",
"\n",
"def try_except_tool(tool_args: dict, config: RunnableConfig) -> Runnable:\n",
" try:\n",
" complex_tool.invoke(tool_args, config=config)\n",
" except Exception as e:\n",
" return f\"Calling tool with arguments:\\n\\n{tool_args}\\n\\nraised the following error:\\n\\n{type(e)}: {e}\"\n",
"\n",
"\n",
"chain = llm_with_tools | (lambda msg: msg.tool_calls[0][\"args\"]) | try_except_tool\n",
"\n",
"print(\n",
" chain.invoke(\n",
" \"use complex tool. the args are 5, 2.1, empty dictionary. don't forget dict_arg\"\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"id": "3b2f6393-cb47-49d0-921c-09550a049fe4",
"metadata": {},
"source": [
"## Fallbacks\n",
"\n",
"We can also try to fallback to a better model in the event of a tool invocation error. In this case we'll fall back to an identical chain that uses `gpt-4-1106-preview` instead of `gpt-3.5-turbo`."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "02cc4223-35fa-4240-976a-012299ca703c",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:27.404122Z",
"iopub.status.busy": "2024-09-11T03:10:27.403539Z",
"iopub.status.idle": "2024-09-11T03:10:38.080547Z",
"shell.execute_reply": "2024-09-11T03:10:38.079955Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"10.5"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chain = llm_with_tools | (lambda msg: msg.tool_calls[0][\"args\"]) | complex_tool\n",
"\n",
"better_model = ChatOpenAI(model=\"gpt-4-1106-preview\", temperature=0).bind_tools(\n",
" [complex_tool], tool_choice=\"complex_tool\"\n",
")\n",
"\n",
"better_chain = better_model | (lambda msg: msg.tool_calls[0][\"args\"]) | complex_tool\n",
"\n",
"chain_with_fallback = chain.with_fallbacks([better_chain])\n",
"\n",
"chain_with_fallback.invoke(\n",
" \"use complex tool. the args are 5, 2.1, empty dictionary. don't forget dict_arg\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "412f8c4e-cc83-4d87-84a1-5ba2f8edb1e9",
"metadata": {},
"source": [
"Looking at the [LangSmith trace](https://smith.langchain.com/public/00e91fc2-e1a4-4b0f-a82e-e6b3119d196c/r) for this chain run, we can see that the first chain call fails as expected and it's the fallback that succeeds."
]
},
{
"cell_type": "markdown",
"id": "304b59cd-cd25-4205-9769-36595c8f3b59",
"metadata": {},
"source": [
"## Retry with exception\n",
"\n",
"To take things one step further, we can try to automatically re-run the chain with the exception passed in, so that the model may be able to correct its behavior:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "b5659956-9454-468a-9753-a3ff9052b8f5",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:38.083810Z",
"iopub.status.busy": "2024-09-11T03:10:38.083623Z",
"iopub.status.idle": "2024-09-11T03:10:38.090089Z",
"shell.execute_reply": "2024-09-11T03:10:38.089682Z"
}
},
"outputs": [],
"source": [
"from langchain_core.messages import AIMessage, HumanMessage, ToolCall, ToolMessage\n",
"from langchain_core.prompts import ChatPromptTemplate\n",
"\n",
"\n",
"class CustomToolException(Exception):\n",
" \"\"\"Custom LangChain tool exception.\"\"\"\n",
"\n",
" def __init__(self, tool_call: ToolCall, exception: Exception) -> None:\n",
" super().__init__()\n",
" self.tool_call = tool_call\n",
" self.exception = exception\n",
"\n",
"\n",
"def tool_custom_exception(msg: AIMessage, config: RunnableConfig) -> Runnable:\n",
" try:\n",
" return complex_tool.invoke(msg.tool_calls[0][\"args\"], config=config)\n",
" except Exception as e:\n",
" raise CustomToolException(msg.tool_calls[0], e)\n",
"\n",
"\n",
"def exception_to_messages(inputs: dict) -> dict:\n",
" exception = inputs.pop(\"exception\")\n",
"\n",
" # Add historical messages to the original input, so the model knows that it made a mistake with the last tool call.\n",
" messages = [\n",
" AIMessage(content=\"\", tool_calls=[exception.tool_call]),\n",
" ToolMessage(\n",
" tool_call_id=exception.tool_call[\"id\"], content=str(exception.exception)\n",
" ),\n",
" HumanMessage(\n",
" content=\"The last tool call raised an exception. Try calling the tool again with corrected arguments. Do not repeat mistakes.\"\n",
" ),\n",
" ]\n",
" inputs[\"last_output\"] = messages\n",
" return inputs\n",
"\n",
"\n",
"# We add a last_output MessagesPlaceholder to our prompt which if not passed in doesn't\n",
"# affect the prompt at all, but gives us the option to insert an arbitrary list of Messages\n",
"# into the prompt if needed. We'll use this on retries to insert the error message.\n",
"prompt = ChatPromptTemplate.from_messages(\n",
" [(\"human\", \"{input}\"), (\"placeholder\", \"{last_output}\")]\n",
")\n",
"chain = prompt | llm_with_tools | tool_custom_exception\n",
"\n",
"# If the initial chain call fails, we rerun it withe the exception passed in as a message.\n",
"self_correcting_chain = chain.with_fallbacks(\n",
" [exception_to_messages | chain], exception_key=\"exception\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "4c45f5bd-cbb4-47d5-b4b6-aec50673c750",
"metadata": {
"execution": {
"iopub.execute_input": "2024-09-11T03:10:38.092152Z",
"iopub.status.busy": "2024-09-11T03:10:38.092021Z",
"iopub.status.idle": "2024-09-11T03:10:39.592443Z",
"shell.execute_reply": "2024-09-11T03:10:39.591990Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"10.5"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"self_correcting_chain.invoke(\n",
" {\n",
" \"input\": \"use complex tool. the args are 5, 2.1, empty dictionary. don't forget dict_arg\"\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"id": "50d269a9-3cab-4a37-ba2f-805296453627",
"metadata": {},
"source": [
"And our chain succeeds! Looking at the [LangSmith trace](https://smith.langchain.com/public/c11e804c-e14f-4059-bd09-64766f999c14/r), we can see that indeed our initial chain still fails, and it's only on retrying that the chain succeeds."
]
},
{
"cell_type": "markdown",
"id": "6b97af9f",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"Now you've seen some strategies how to handle tool calling errors. Next, you can learn more about how to use tools:\n",
"\n",
"- Few shot prompting [with tools](/docs/how_to/tools_few_shot/)\n",
"- Stream [tool calls](/docs/how_to/tool_streaming/)\n",
"- Pass [runtime values to tools](/docs/how_to/tool_runtime)\n",
"\n",
"You can also check out some more specific uses of tool calling:\n",
"\n",
"- Getting [structured outputs](/docs/how_to/structured_output/) from models"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink