From 55dd2ea57d50463d784cf01a6c8be262fea6a2bd Mon Sep 17 00:00:00 2001 From: Harrison Chase Date: Mon, 29 Apr 2024 16:29:47 -0700 Subject: [PATCH] cr --- .../version-0.2.x/tutorials/agents.ipynb | 5 +- .../tutorials/qa_chat_history.ipynb | 360 +++------ .../version-0.2.x/tutorials/sql_qa.ipynb | 720 ++++-------------- 3 files changed, 278 insertions(+), 807 deletions(-) diff --git a/docs/versioned_docs/version-0.2.x/tutorials/agents.ipynb b/docs/versioned_docs/version-0.2.x/tutorials/agents.ipynb index c6cc463eb60..c5bed455b6c 100644 --- a/docs/versioned_docs/version-0.2.x/tutorials/agents.ipynb +++ b/docs/versioned_docs/version-0.2.x/tutorials/agents.ipynb @@ -761,7 +761,10 @@ "source": [ "## Conclusion\n", "\n", - "That's a wrap! In this quick start we covered how to create a simple agent. Agents are a complex topic, and there's lot to learn! \n", + "That's a wrap! In this quick start we covered how to create a simple agent. \n", + "We've then shown how to stream back a response - not only the intermediate steps, but also tokens!\n", + "We've also added in memory so you can have a conversation with them.\n", + "Agents are a complex topic, and there's lot to learn! \n", "\n", "For more information on Agents, please check out the [LangGraph](/docs/concepts/#langgraph) documentation. This has it's own set of concepts, tutorials, and how-to guides." ] diff --git a/docs/versioned_docs/version-0.2.x/tutorials/qa_chat_history.ipynb b/docs/versioned_docs/version-0.2.x/tutorials/qa_chat_history.ipynb index b961053342e..0a1babcb46f 100644 --- a/docs/versioned_docs/version-0.2.x/tutorials/qa_chat_history.ipynb +++ b/docs/versioned_docs/version-0.2.x/tutorials/qa_chat_history.ipynb @@ -205,17 +205,17 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 4, "id": "bf55faaf-0d17-4b74-925d-c478b555f7b2", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "'Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model. This process helps in planning and executing tasks efficiently by dividing them into subgoals or smaller components. Task decomposition can be achieved through techniques like Chain of Thought and Tree of Thoughts, which guide the model in thinking step by step or exploring multiple reasoning possibilities at each step.'" + "'Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable. This process can be achieved through techniques like Chain of Thought (CoT) or Tree of Thoughts, which help agents plan and execute tasks effectively by dividing them into sequential subgoals. Task decomposition can be facilitated by using prompting techniques, task-specific instructions, or human inputs to guide the agent through the steps required to accomplish a task.'" ] }, - "execution_count": 6, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -275,7 +275,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 5, "id": "2b685428-8b82-4af1-be4f-7232c5d55b73", "metadata": {}, "outputs": [], @@ -320,7 +320,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 6, "id": "66f275f3-ddef-4678-b90d-ee64576878f9", "metadata": {}, "outputs": [], @@ -352,7 +352,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 7, "id": "0005810b-1b95-4666-a795-08d80e478b83", "metadata": {}, "outputs": [ @@ -360,7 +360,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Task decomposition can be achieved through various methods such as using prompting techniques like \"Steps for XYZ\" with LLMs, providing task-specific instructions like \"Write a story outline,\" or incorporating human inputs. These approaches help in breaking down complex tasks into smaller, more manageable subgoals for better understanding and execution. By decomposing tasks effectively, agents can plan ahead and navigate through the problem-solving process more efficiently.\n" + "Task decomposition can be done in several common ways, such as using Language Model (LLM) with simple prompting like \"Steps for XYZ\" or asking for subgoals to achieve a specific task. Task-specific instructions can also be provided, like requesting a story outline for writing a novel. Additionally, human inputs can be utilized to decompose tasks into smaller components effectively.\n" ] } ], @@ -419,7 +419,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 8, "id": "9c3fb176-8d6a-4dc7-8408-6a22c5f7cc72", "metadata": {}, "outputs": [], @@ -449,17 +449,17 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 9, "id": "1046c92f-21b3-4214-907d-92878d8cba23", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "'Task decomposition involves breaking down a complex task into smaller and simpler steps to make it more manageable for an agent or model to execute. This process helps in organizing the sequence of actions required to achieve the overall goal effectively. Task decomposition can be facilitated through techniques like Chain of Thought and Tree of Thoughts, which guide the model in thinking step by step or exploring multiple reasoning possibilities at each step.'" + "'Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model. This process helps in guiding the agent through the various subgoals required to achieve the overall task efficiently. Different techniques like Chain of Thought and Tree of Thoughts can be used to decompose tasks into manageable components.'" ] }, - "execution_count": 12, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -475,17 +475,17 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 10, "id": "0e89c75f-7ad7-4331-a2fe-57579eb8f840", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "'Task decomposition can be achieved through various methods such as using prompting techniques like \"Steps for XYZ\" to guide the model in breaking down the task into subgoals, providing task-specific instructions like \"Write a story outline\" for specific tasks, or incorporating human inputs to assist in breaking down complex tasks. These approaches help in organizing the steps needed to accomplish a task and enhance the model\\'s performance on challenging tasks by simplifying the problem-solving process.'" + "'Task decomposition can be achieved through various methods such as using prompting techniques like \"Steps for XYZ\" to guide the model through subgoals, providing task-specific instructions like \"Write a story outline\" for specific tasks, or incorporating human inputs to break down complex tasks. These approaches help in dividing a large task into smaller, more manageable components for better understanding and execution.'" ] }, - "execution_count": 13, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -507,7 +507,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 11, "id": "7686b874-3a85-499f-82b5-28a85c4c768c", "metadata": {}, "outputs": [ @@ -517,11 +517,11 @@ "text": [ "User: What is Task Decomposition?\n", "\n", - "AI: Task decomposition involves breaking down a complex task into smaller and simpler steps to make it more manageable for an agent or model to execute. This process helps in organizing the sequence of actions required to achieve the overall goal effectively. Task decomposition can be facilitated through techniques like Chain of Thought and Tree of Thoughts, which guide the model in thinking step by step or exploring multiple reasoning possibilities at each step.\n", + "AI: Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model. This process helps in guiding the agent through the various subgoals required to achieve the overall task efficiently. Different techniques like Chain of Thought and Tree of Thoughts can be used to decompose tasks into manageable components.\n", "\n", "User: What are common ways of doing it?\n", "\n", - "AI: Task decomposition can be achieved through various methods such as using prompting techniques like \"Steps for XYZ\" to guide the model in breaking down the task into subgoals, providing task-specific instructions like \"Write a story outline\" for specific tasks, or incorporating human inputs to assist in breaking down complex tasks. These approaches help in organizing the steps needed to accomplish a task and enhance the model's performance on challenging tasks by simplifying the problem-solving process.\n", + "AI: Task decomposition can be achieved through various methods such as using prompting techniques like \"Steps for XYZ\" to guide the model through subgoals, providing task-specific instructions like \"Write a story outline\" for specific tasks, or incorporating human inputs to break down complex tasks. These approaches help in dividing a large task into smaller, more manageable components for better understanding and execution.\n", "\n" ] } @@ -722,7 +722,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 12, "id": "809cc747-2135-40a2-8e73-e4556343ee64", "metadata": {}, "outputs": [], @@ -747,7 +747,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 13, "id": "931c4fe3-c603-4efb-9b37-5f7cbbb1cbbd", "metadata": {}, "outputs": [ @@ -757,7 +757,7 @@ "'Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\\n\\nFig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\n\\n(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\\n\\nFig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 2023)\\nThe system comprises of 4 stages:\\n(1) Task planning: LLM works as the brain and parses the user requests into multiple tasks. There are four attributes associated with each task: task type, ID, dependencies, and arguments. They use few-shot examples to guide LLM to do task parsing and planning.\\nInstruction:'" ] }, - "execution_count": 5, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -773,37 +773,57 @@ "source": [ "### Agent constructor\n", "\n", - "Here, we will use the high level `create_tool_calling_agent` API to construct the agent.\n", - "\n", - "This builds an instance of `AgentExecutor`, which will manage the execution of the tools via the LLM. It requires an LLM, prompt, and list of tools. The requirements of our prompt are:\n", - "1. It should contain an element representing the chat history;\n", - "2. It should contain the user's input query;\n", - "3. It should contain an element representing a sequence of decisions and observations by the agent (typically identified as the `\"agent_scratchpad\"`).\n", - "\n", - "Let's assemble these components into a prompt template and instantiate the agent:" + "Now that we have defined the tools and the LLM, we can create the agent. We will be using [LangGraph](/docs/concepts/#langgraph) to construct the agent. \n", + "Currently we are using a high level interface to construct the agent, but the nice thing about LangGraph is that this high-level interface is backed by a low-level, highly controllable API in case you want to modify the agent logic." ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 15, "id": "1726d151-4653-4c72-a187-a14840add526", "metadata": {}, "outputs": [], "source": [ - "from langchain.agents import AgentExecutor, create_tool_calling_agent\n", + "from langgraph.prebuilt import chat_agent_executor\n", "\n", + "agent_executor = chat_agent_executor.create_tool_calling_executor(llm, tools)" + ] + }, + { + "cell_type": "markdown", + "id": "6d5152ca-1c3b-4f58-bb28-f31c0be7ba66", + "metadata": {}, + "source": [ + "We can now try it out. Note that so far it is not stateful (we still need to add in memory)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "170403a2-c914-41db-85d8-a2c381da112d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_demTlnha4vYA1IH6CByYupBQ', 'function': {'arguments': '{\"query\":\"Task Decomposition\"}', 'name': 'blog_post_retriever'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 19, 'prompt_tokens': 68, 'total_tokens': 87}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-d1c3f3da-be18-46a5-b3a8-4621ba1f7f2a-0', tool_calls=[{'name': 'blog_post_retriever', 'args': {'query': 'Task Decomposition'}, 'id': 'call_demTlnha4vYA1IH6CByYupBQ'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='Fig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\n\\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\\n\\n(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\\n\\nFig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 2023)\\nThe system comprises of 4 stages:\\n(1) Task planning: LLM works as the brain and parses the user requests into multiple tasks. There are four attributes associated with each task: task type, ID, dependencies, and arguments. They use few-shot examples to guide LLM to do task parsing and planning.\\nInstruction:', name='blog_post_retriever', id='e83e4002-33d2-46ff-82f4-fddb3035fb6a', tool_call_id='call_demTlnha4vYA1IH6CByYupBQ')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='Task decomposition is a technique used in autonomous agent systems to break down complex tasks into smaller and simpler steps. This approach helps agents better understand and plan for the various steps involved in completing a task. One common method for task decomposition is the Chain of Thought (CoT) technique, where models are prompted to \"think step by step\" to decompose hard tasks into manageable steps. Another approach, known as Tree of Thoughts, extends CoT by exploring multiple reasoning possibilities at each step and creating a tree structure of tasks.\\n\\nTask decomposition can be achieved through various methods, such as using simple prompts for language models, task-specific instructions, or human inputs. By breaking down tasks into smaller components, agents can effectively plan and execute tasks with greater efficiency.\\n\\nIn summary, task decomposition is a valuable strategy for autonomous agents to tackle complex tasks by breaking them down into smaller, more manageable steps.', response_metadata={'token_usage': {'completion_tokens': 177, 'prompt_tokens': 588, 'total_tokens': 765}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-808f32b9-ae61-4f31-a55a-f30643594282-0')]}}\n", + "----\n" + ] + } + ], + "source": [ + "query = \"What is Task Decomposition?\"\n", "\n", - "prompt = ChatPromptTemplate.from_messages(\n", - " [\n", - " (\"system\", \"You are a helpful assistant\"),\n", - " MessagesPlaceholder(variable_name=\"chat_history\", optional=True),\n", - " (\"human\", \"{input}\"),\n", - " MessagesPlaceholder(\"agent_scratchpad\"),\n", - " ]\n", - ")\n", - "\n", - "agent = create_tool_calling_agent(llm, tools, prompt)\n", - "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)" + "for s in agent_executor.stream(\n", + " {\"messages\": [HumanMessage(content=query)]},\n", + "):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -811,32 +831,21 @@ "id": "1df703b1-aad6-48fb-b6fa-703e32ea88b9", "metadata": {}, "source": [ - "Because AgentExecutor is a runnable, we can wrap it in a `RunnableWithMessageHistory` as before to manage the chat history. Note that AgentExecutor by convention returns its output in a key `\"output\"`:" + "LangGraph comes with built in persistence, so we don't need to use ChatMessageHistory! Rather, we can pass in a checkpointer to our LangGraph agent directly" ] }, { "cell_type": "code", - "execution_count": 8, - "id": "678552e7-39b7-4ea2-a5f1-241d85dae268", + "execution_count": 21, + "id": "04a3a664-3c3f-4cd1-9995-26662a52da7c", "metadata": {}, "outputs": [], "source": [ - "store = {}\n", + "from langgraph.checkpoint.sqlite import SqliteSaver\n", "\n", + "memory = SqliteSaver.from_conn_string(\":memory:\")\n", "\n", - "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n", - " if session_id not in store:\n", - " store[session_id] = ChatMessageHistory()\n", - " return store[session_id]\n", - "\n", - "\n", - "conversational_rag_agent = RunnableWithMessageHistory(\n", - " agent_executor,\n", - " get_session_history,\n", - " input_messages_key=\"input\",\n", - " history_messages_key=\"chat_history\",\n", - " output_messages_key=\"output\",\n", - ")" + "agent_executor = chat_agent_executor.create_tool_calling_executor(llm, tools, checkpointer=memory)\n" ] }, { @@ -851,7 +860,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 22, "id": "d6d70833-b958-4cd7-9e27-29c1c08bb1b8", "metadata": {}, "outputs": [ @@ -859,32 +868,20 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\n", - "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3mHello Bob! How can I assist you today?\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" + "{'agent': {'messages': [AIMessage(content='Hello Bob! How can I assist you today?', response_metadata={'token_usage': {'completion_tokens': 11, 'prompt_tokens': 67, 'total_tokens': 78}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-1451e59b-b135-4776-985d-4759338ffee5-0')]}}\n", + "----\n" ] - }, - { - "data": { - "text/plain": [ - "'Hello Bob! How can I assist you today?'" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "query = \"Hello, I'm Bob.\"\n", + "config = {\"configurable\": {\"thread_id\": \"abc123\"}}\n", "\n", - "conversational_rag_agent.invoke(\n", - " {\"input\": query},\n", - " config={\"configurable\": {\"session_id\": \"abc123\"}},\n", - ")[\"output\"]" + "for s in agent_executor.stream(\n", + " {\"messages\": [HumanMessage(content=\"Hi! I'm bob\")]},\n", + " config=config\n", + "):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -897,7 +894,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 23, "id": "e2c570ae-dd91-402c-8693-ae746de63b16", "metadata": {}, "outputs": [ @@ -905,55 +902,24 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\n", - "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3m\n", - "Invoking: `blog_post_retriever` with `{'query': 'Task Decomposition'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[36;1m\u001b[1;3mFig. 1. Overview of a LLM-powered autonomous agent system.\n", - "Component One: Planning#\n", - "A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\n", - "Task Decomposition#\n", - "Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n", - "\n", - "Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\n", - "Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\n", - "\n", - "(3) Task execution: Expert models execute on the specific tasks and log results.\n", - "Instruction:\n", - "\n", - "With the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\n", - "\n", - "Fig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 2023)\n", - "The system comprises of 4 stages:\n", - "(1) Task planning: LLM works as the brain and parses the user requests into multiple tasks. There are four attributes associated with each task: task type, ID, dependencies, and arguments. They use few-shot examples to guide LLM to do task parsing and planning.\n", - "Instruction:\u001b[0m\u001b[32;1m\u001b[1;3mTask decomposition is a technique used to break down complex tasks into smaller and simpler steps. This approach helps in managing and solving intricate problems by dividing them into more manageable components. One common method for task decomposition is the Chain of Thought (CoT) technique, which prompts models to think step by step and decompose difficult tasks into smaller steps. Another extension of CoT is the Tree of Thoughts, which explores multiple reasoning possibilities at each step by creating a tree structure of thoughts.\n", - "\n", - "Task decomposition can be achieved through various methods, such as using simple prompts for language models, providing task-specific instructions, or incorporating human inputs. This process is essential for effective planning and execution of tasks by autonomous agents or AI systems.\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_ab2x4iUPSWDAHS5txL7PspSK', 'function': {'arguments': '{\"query\":\"Task Decomposition\"}', 'name': 'blog_post_retriever'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 19, 'prompt_tokens': 91, 'total_tokens': 110}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-f76b5813-b41c-4d0d-9ed2-667b988d885e-0', tool_calls=[{'name': 'blog_post_retriever', 'args': {'query': 'Task Decomposition'}, 'id': 'call_ab2x4iUPSWDAHS5txL7PspSK'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='Fig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\n\\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\\n\\n(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\\n\\nFig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 2023)\\nThe system comprises of 4 stages:\\n(1) Task planning: LLM works as the brain and parses the user requests into multiple tasks. There are four attributes associated with each task: task type, ID, dependencies, and arguments. They use few-shot examples to guide LLM to do task parsing and planning.\\nInstruction:', name='blog_post_retriever', id='e0895fa5-5d41-4be0-98db-10a83d42fc2f', tool_call_id='call_ab2x4iUPSWDAHS5txL7PspSK')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='Task decomposition is a technique used in complex tasks where the task is broken down into smaller and simpler steps. This approach helps in managing and solving difficult tasks by dividing them into more manageable components. One common method for task decomposition is the Chain of Thought (CoT) technique, which prompts the model to think step by step and decompose hard tasks into smaller steps. Another extension of CoT is the Tree of Thoughts, which explores multiple reasoning possibilities at each step by creating a tree structure of thought steps.\\n\\nTask decomposition can be achieved through various methods, such as using language models with simple prompting, task-specific instructions, or human inputs. By breaking down tasks into smaller components, agents can better plan and execute complex tasks effectively.\\n\\nIf you would like more detailed information or examples related to task decomposition, feel free to ask!', response_metadata={'token_usage': {'completion_tokens': 165, 'prompt_tokens': 611, 'total_tokens': 776}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-13296566-8577-4d65-982b-a39718988ca3-0')]}}\n", + "----\n" ] - }, - { - "data": { - "text/plain": [ - "'Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. This approach helps in managing and solving intricate problems by dividing them into more manageable components. One common method for task decomposition is the Chain of Thought (CoT) technique, which prompts models to think step by step and decompose difficult tasks into smaller steps. Another extension of CoT is the Tree of Thoughts, which explores multiple reasoning possibilities at each step by creating a tree structure of thoughts.\\n\\nTask decomposition can be achieved through various methods, such as using simple prompts for language models, providing task-specific instructions, or incorporating human inputs. This process is essential for effective planning and execution of tasks by autonomous agents or AI systems.'" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ "query = \"What is Task Decomposition?\"\n", "\n", - "conversational_rag_agent.invoke(\n", - " {\"input\": query},\n", - " config={\"configurable\": {\"session_id\": \"abc123\"}},\n", - ")[\"output\"]" + "for s in agent_executor.stream(\n", + " {\"messages\": [HumanMessage(content=query)]},\n", + " config=config\n", + "):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -968,7 +934,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 25, "id": "570d8c68-136e-4ba5-969a-03ba195f6118", "metadata": {}, "outputs": [ @@ -976,66 +942,24 @@ "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\n", - "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3m\n", - "Invoking: `blog_post_retriever` with `{'query': 'common ways of task decomposition'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[36;1m\u001b[1;3mTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\n", - "Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\n", - "\n", - "Fig. 1. Overview of a LLM-powered autonomous agent system.\n", - "Component One: Planning#\n", - "A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\n", - "Task Decomposition#\n", - "Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n", - "\n", - "Resources:\n", - "1. Internet access for searches and information gathering.\n", - "2. Long Term memory management.\n", - "3. GPT-3.5 powered Agents for delegation of simple tasks.\n", - "4. File output.\n", - "\n", - "Performance Evaluation:\n", - "1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n", - "2. Constructively self-criticize your big-picture behavior constantly.\n", - "3. Reflect on past decisions and strategies to refine your approach.\n", - "4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\n", - "\n", - "(3) Task execution: Expert models execute on the specific tasks and log results.\n", - "Instruction:\n", - "\n", - "With the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\u001b[0m\u001b[32;1m\u001b[1;3mAccording to the blog post, common ways of task decomposition include:\n", - "\n", - "1. Using Language Models (LLM) with simple prompting like \"Steps for XYZ\" or \"What are the subgoals for achieving XYZ?\"\n", - "2. Utilizing task-specific instructions, such as \"Write a story outline\" for writing a novel.\n", - "3. Incorporating human inputs to assist in task decomposition.\n", - "\n", - "Additionally, the Chain of Thought (CoT) technique is highlighted as a standard prompting technique for enhancing model performance on complex tasks. CoT instructs the model to \"think step by step\" to decompose hard tasks into smaller and simpler steps, transforming big tasks into multiple manageable tasks.\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_KvoiamnLfGEzMeEMlV3u0TJ7', 'function': {'arguments': '{\"query\":\"common ways of task decomposition\"}', 'name': 'blog_post_retriever'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 21, 'prompt_tokens': 930, 'total_tokens': 951}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-dd842071-6dbd-4b68-8657-892eaca58638-0', tool_calls=[{'name': 'blog_post_retriever', 'args': {'query': 'common ways of task decomposition'}, 'id': 'call_KvoiamnLfGEzMeEMlV3u0TJ7'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\\n\\nFig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\n\\nResources:\\n1. Internet access for searches and information gathering.\\n2. Long Term memory management.\\n3. GPT-3.5 powered Agents for delegation of simple tasks.\\n4. File output.\\n\\nPerformance Evaluation:\\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\\n2. Constructively self-criticize your big-picture behavior constantly.\\n3. Reflect on past decisions and strategies to refine your approach.\\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\\n\\n(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.', name='blog_post_retriever', id='c749bb8e-c8e0-4fa3-bc11-3e2e0651880b', tool_call_id='call_KvoiamnLfGEzMeEMlV3u0TJ7')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='According to the blog post, common ways of task decomposition include:\\n\\n1. Using language models with simple prompting like \"Steps for XYZ\" or \"What are the subgoals for achieving XYZ?\"\\n2. Utilizing task-specific instructions, for example, using \"Write a story outline\" for writing a novel.\\n3. Involving human inputs in the task decomposition process.\\n\\nThese methods help in breaking down complex tasks into smaller and more manageable steps, facilitating better planning and execution of the overall task.', response_metadata={'token_usage': {'completion_tokens': 100, 'prompt_tokens': 1475, 'total_tokens': 1575}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-98b765b3-f1a6-4c9a-ad0f-2db7950b900f-0')]}}\n", + "----\n" ] - }, - { - "data": { - "text/plain": [ - "'According to the blog post, common ways of task decomposition include:\\n\\n1. Using Language Models (LLM) with simple prompting like \"Steps for XYZ\" or \"What are the subgoals for achieving XYZ?\"\\n2. Utilizing task-specific instructions, such as \"Write a story outline\" for writing a novel.\\n3. Incorporating human inputs to assist in task decomposition.\\n\\nAdditionally, the Chain of Thought (CoT) technique is highlighted as a standard prompting technique for enhancing model performance on complex tasks. CoT instructs the model to \"think step by step\" to decompose hard tasks into smaller and simpler steps, transforming big tasks into multiple manageable tasks.'" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "query = \"What according to the blog post are common ways of doing it?\"\n", + "query = \"What according to the blog post are common ways of doing it? redo the search\"\n", "\n", - "conversational_rag_agent.invoke(\n", - " {\"input\": query},\n", - " config={\"configurable\": {\"session_id\": \"abc123\"}},\n", - ")[\"output\"]" + "for s in agent_executor.stream(\n", + " {\"messages\": [HumanMessage(content=query)]},\n", + " config=config\n", + "):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -1058,7 +982,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 26, "id": "b1d2b4d4-e604-497d-873d-d345b808578e", "metadata": {}, "outputs": [], @@ -1074,7 +998,9 @@ "from langchain_core.runnables.history import RunnableWithMessageHistory\n", "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n", "from langchain_text_splitters import RecursiveCharacterTextSplitter\n", + "from langgraph.checkpoint.sqlite import SqliteSaver\n", "\n", + "memory = SqliteSaver.from_conn_string(\":memory:\")\n", "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n", "\n", "\n", @@ -1104,63 +1030,7 @@ "tools = [tool]\n", "\n", "\n", - "### Build agent ###\n", - "prompt = ChatPromptTemplate.from_messages(\n", - " [\n", - " (\"system\", \"You are a helpful assistant\"),\n", - " MessagesPlaceholder(variable_name=\"chat_history\", optional=True),\n", - " (\"human\", \"{input}\"),\n", - " MessagesPlaceholder(\"agent_scratchpad\"),\n", - " ]\n", - ")\n", - "\n", - "agent = create_tool_calling_agent(llm, tools, prompt)\n", - "agent_executor = AgentExecutor(agent=agent, tools=tools)\n", - "\n", - "\n", - "### Statefully manage chat history ###\n", - "store = {}\n", - "\n", - "\n", - "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n", - " if session_id not in store:\n", - " store[session_id] = ChatMessageHistory()\n", - " return store[session_id]\n", - "\n", - "\n", - "conversational_rag_agent = RunnableWithMessageHistory(\n", - " agent_executor,\n", - " get_session_history,\n", - " input_messages_key=\"input\",\n", - " history_messages_key=\"chat_history\",\n", - " output_messages_key=\"output\",\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "a0622ac9-523c-4110-b669-ba2a43e9de67", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. This approach helps autonomous agents or models to handle difficult tasks by dividing them into more manageable components. One common method for task decomposition is the Chain of Thought (CoT) technique, where models are instructed to think step by step to decompose hard tasks into smaller steps. Another extension of CoT is the Tree of Thoughts, which explores multiple reasoning possibilities at each step by creating a tree structure of thought steps.\\n\\nTask decomposition can be achieved through various methods, such as using simple prompts for Language Model (LLM) like \"Steps for XYZ\" or task-specific instructions. Human inputs can also be utilized in the task decomposition process. Overall, task decomposition is essential for planning and executing complex tasks effectively.'" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "query = \"What is Task Decomposition?\"\n", - "\n", - "conversational_rag_agent.invoke(\n", - " {\"input\": query},\n", - " config={\"configurable\": {\"session_id\": \"abc123\"}},\n", - ")[\"output\"]" + "agent_executor = chat_agent_executor.create_tool_calling_executor(llm, tools, checkpointer=memory)" ] }, { @@ -1177,12 +1047,18 @@ "\n", "To explore different types of retrievers and retrieval strategies, visit the [retrievers](docs/0.2.x/how_to/#retrievers) section of the how-to guides.\n", "\n", - "For a detailed walkthrough of LangChain's conversation memory abstractions, visit the [How to add message history (memory)](/docs/expression_language/how_to/message_history) LCEL page.\n", + "For a detailed walkthrough of LangChain's conversation memory abstractions, visit the [How to add message history (memory)](/docs/how_to/message_history) LCEL page.\n", "\n", - "To learn more about the built-in generic agent types as well as how to build custom agents, head to the [Agents Modules](/docs/modules/agents/).\n", - "\n", - "The built-in `AgentExecutor` runs a simple Agent action -> Tool call -> Agent action... loop. To build more complex agent runtimes, head to the [LangGraph section](/docs/langgraph)." + "To learn more about agents, head to the [Agents Modules](/docs/modules/agents/)." ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b8d17592-6240-49ac-a904-f0171eddcc14", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { diff --git a/docs/versioned_docs/version-0.2.x/tutorials/sql_qa.ipynb b/docs/versioned_docs/version-0.2.x/tutorials/sql_qa.ipynb index 9664d1c96f4..f4183977941 100644 --- a/docs/versioned_docs/version-0.2.x/tutorials/sql_qa.ipynb +++ b/docs/versioned_docs/version-0.2.x/tutorials/sql_qa.ipynb @@ -142,7 +142,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -156,7 +156,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 3, "metadata": {}, "outputs": [ { @@ -165,7 +165,7 @@ "'SELECT COUNT(\"EmployeeId\") AS \"TotalEmployees\" FROM \"Employee\"\\nLIMIT 1;'" ] }, - "execution_count": 5, + "execution_count": 3, "metadata": {}, "output_type": "execute_result" } @@ -187,7 +187,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 4, "metadata": {}, "outputs": [ { @@ -196,7 +196,7 @@ "'[(8,)]'" ] }, - "execution_count": 6, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -220,7 +220,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 5, "metadata": {}, "outputs": [ { @@ -264,7 +264,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 6, "metadata": {}, "outputs": [ { @@ -273,7 +273,7 @@ "'[(8,)]'" ] }, - "execution_count": 8, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -298,7 +298,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "metadata": {}, "outputs": [ { @@ -307,7 +307,7 @@ "'There are a total of 8 employees.'" ] }, - "execution_count": 6, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -381,35 +381,97 @@ "- It can query the database as many times as needed to answer the user question.\n", "- It will save tokens by only retrieving the schema from relevant tables.\n", "\n", - "To initialize the agent we'll use the [create_sql_agent](https://api.python.langchain.com/en/latest/agent_toolkits/langchain_community.agent_toolkits.sql.base.create_sql_agent.html) constructor. This agent uses the `SQLDatabaseToolkit` which contains tools to: \n", + "To initialize the agent we'll use the `SQLDatabaseToolkit` to create a bunch of tools:\n", "\n", "* Create and execute queries\n", "* Check query syntax\n", "* Retrieve table descriptions\n", - "* ... and more\n", - "\n", - "### Initializing agent\n", - "\n", - "LangChain provides a built-in `create_sql_agent` constructor that accepts an LLM and database as input.\n", - "\n", - "Note that we specify `agent_type`, which determines the principle by which agents execute tools:\n", - "- Tool-calling agents are compatible with LLMs that support function or tool calling features.\n", - "- ReAct agents rely on prompting the LLM to follow a certain natural language pattern (e.g., \"Thought: \", \"Action: \", etc.). They are potentially less reliable, but do not require specialized tool-calling capabilities from an LLM.\n", - "\n", - "See our [agent types](/docs/modules/agents/agent_types/) documentation for more detail.\n", - "\n", - "Below, we create a tool-calling agent." + "* ... and more" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[QuerySQLDataBaseTool(description=\"Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields.\", db=),\n", + " InfoSQLDatabaseTool(description='Input to this tool is a comma-separated list of tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist by calling sql_db_list_tables first! Example Input: table1, table2, table3', db=),\n", + " ListSQLDatabaseTool(db=),\n", + " QuerySQLCheckerTool(description='Use this tool to double check if your query is correct before executing it. Always use this tool before executing a query with sql_db_query!', db=, llm=ChatOpenAI(client=, async_client=, temperature=0.0, openai_api_key=SecretStr('**********'), openai_proxy=''), llm_chain=LLMChain(prompt=PromptTemplate(input_variables=['dialect', 'query'], template='\\n{query}\\nDouble check the {dialect} query above for common mistakes, including:\\n- Using NOT IN with NULL values\\n- Using UNION when UNION ALL should have been used\\n- Using BETWEEN for exclusive ranges\\n- Data type mismatch in predicates\\n- Properly quoting identifiers\\n- Using the correct number of arguments for functions\\n- Casting to the correct data type\\n- Using the proper columns for joins\\n\\nIf there are any of the above mistakes, rewrite the query. If there are no mistakes, just reproduce the original query.\\n\\nOutput the final SQL query only.\\n\\nSQL Query: '), llm=ChatOpenAI(client=, async_client=, temperature=0.0, openai_api_key=SecretStr('**********'), openai_proxy='')))]" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from langchain_community.agent_toolkits import SQLDatabaseToolkit\n", + "\n", + "toolkit = SQLDatabaseToolkit(db=db, llm=llm)\n", + "\n", + "tools = toolkit.get_tools()\n", + "\n", + "tools" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### System Prompt\n", + "\n", + "We will also want to create a system prompt for our agent. This will consist of instructions for how to behave." + ] + }, + { + "cell_type": "code", + "execution_count": 32, "metadata": {}, "outputs": [], "source": [ - "from langchain_community.agent_toolkits import create_sql_agent\n", + "from langchain_core.messages import SystemMessage\n", "\n", - "agent_executor = create_sql_agent(llm, db=db, agent_type=\"tool-calling\", verbose=True)" + "SQL_PREFIX = \"\"\"You are an agent designed to interact with a SQL database.\n", + "Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer.\n", + "Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results.\n", + "You can order the results by a relevant column to return the most interesting examples in the database.\n", + "Never query for all the columns from a specific table, only ask for the relevant columns given the question.\n", + "You have access to tools for interacting with the database.\n", + "Only use the below tools. Only use the information returned by the below tools to construct your final answer.\n", + "You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.\n", + "\n", + "DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.\n", + "\n", + "To start you should ALWAYS look at the tables in the database to see what you can query.\n", + "Do NOT skip this step.\n", + "Then you should query the schema of the most relevant tables.\"\"\"\n", + "\n", + "system_message = SystemMessage(content=SQL_PREFIX)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Initializing agent\n", + "\n", + "We will use a prebuilt [LangGraph](/docs/concepts/#langgraph) agent to build our agent" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": {}, + "outputs": [], + "source": [ + "from langgraph.prebuilt import chat_agent_executor\n", + "from langchain_core.messages import HumanMessage\n", + "\n", + "agent_executor = chat_agent_executor.create_tool_calling_executor(llm, tools, system_message=system_message)" ] }, { @@ -421,115 +483,34 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\n", - "\u001b[1m> Entering new SQL Agent Executor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_list_tables` with `{}`\n", - "\n", - "\n", - "\u001b[0m\u001b[38;5;200m\u001b[1;3mAlbum, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track\u001b[0m\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_schema` with `{'table_names': 'Customer, Invoice, InvoiceLine'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[33;1m\u001b[1;3m\n", - "CREATE TABLE \"Customer\" (\n", - "\t\"CustomerId\" INTEGER NOT NULL, \n", - "\t\"FirstName\" NVARCHAR(40) NOT NULL, \n", - "\t\"LastName\" NVARCHAR(20) NOT NULL, \n", - "\t\"Company\" NVARCHAR(80), \n", - "\t\"Address\" NVARCHAR(70), \n", - "\t\"City\" NVARCHAR(40), \n", - "\t\"State\" NVARCHAR(40), \n", - "\t\"Country\" NVARCHAR(40), \n", - "\t\"PostalCode\" NVARCHAR(10), \n", - "\t\"Phone\" NVARCHAR(24), \n", - "\t\"Fax\" NVARCHAR(24), \n", - "\t\"Email\" NVARCHAR(60) NOT NULL, \n", - "\t\"SupportRepId\" INTEGER, \n", - "\tPRIMARY KEY (\"CustomerId\"), \n", - "\tFOREIGN KEY(\"SupportRepId\") REFERENCES \"Employee\" (\"EmployeeId\")\n", - ")\n", - "\n", - "/*\n", - "3 rows from Customer table:\n", - "CustomerId\tFirstName\tLastName\tCompany\tAddress\tCity\tState\tCountry\tPostalCode\tPhone\tFax\tEmail\tSupportRepId\n", - "1\tLuís\tGonçalves\tEmbraer - Empresa Brasileira de Aeronáutica S.A.\tAv. Brigadeiro Faria Lima, 2170\tSão José dos Campos\tSP\tBrazil\t12227-000\t+55 (12) 3923-5555\t+55 (12) 3923-5566\tluisg@embraer.com.br\t3\n", - "2\tLeonie\tKöhler\tNone\tTheodor-Heuss-Straße 34\tStuttgart\tNone\tGermany\t70174\t+49 0711 2842222\tNone\tleonekohler@surfeu.de\t5\n", - "3\tFrançois\tTremblay\tNone\t1498 rue Bélanger\tMontréal\tQC\tCanada\tH2G 1A7\t+1 (514) 721-4711\tNone\tftremblay@gmail.com\t3\n", - "*/\n", - "\n", - "\n", - "CREATE TABLE \"Invoice\" (\n", - "\t\"InvoiceId\" INTEGER NOT NULL, \n", - "\t\"CustomerId\" INTEGER NOT NULL, \n", - "\t\"InvoiceDate\" DATETIME NOT NULL, \n", - "\t\"BillingAddress\" NVARCHAR(70), \n", - "\t\"BillingCity\" NVARCHAR(40), \n", - "\t\"BillingState\" NVARCHAR(40), \n", - "\t\"BillingCountry\" NVARCHAR(40), \n", - "\t\"BillingPostalCode\" NVARCHAR(10), \n", - "\t\"Total\" NUMERIC(10, 2) NOT NULL, \n", - "\tPRIMARY KEY (\"InvoiceId\"), \n", - "\tFOREIGN KEY(\"CustomerId\") REFERENCES \"Customer\" (\"CustomerId\")\n", - ")\n", - "\n", - "/*\n", - "3 rows from Invoice table:\n", - "InvoiceId\tCustomerId\tInvoiceDate\tBillingAddress\tBillingCity\tBillingState\tBillingCountry\tBillingPostalCode\tTotal\n", - "1\t2\t2021-01-01 00:00:00\tTheodor-Heuss-Straße 34\tStuttgart\tNone\tGermany\t70174\t1.98\n", - "2\t4\t2021-01-02 00:00:00\tUllevålsveien 14\tOslo\tNone\tNorway\t0171\t3.96\n", - "3\t8\t2021-01-03 00:00:00\tGrétrystraat 63\tBrussels\tNone\tBelgium\t1000\t5.94\n", - "*/\n", - "\n", - "\n", - "CREATE TABLE \"InvoiceLine\" (\n", - "\t\"InvoiceLineId\" INTEGER NOT NULL, \n", - "\t\"InvoiceId\" INTEGER NOT NULL, \n", - "\t\"TrackId\" INTEGER NOT NULL, \n", - "\t\"UnitPrice\" NUMERIC(10, 2) NOT NULL, \n", - "\t\"Quantity\" INTEGER NOT NULL, \n", - "\tPRIMARY KEY (\"InvoiceLineId\"), \n", - "\tFOREIGN KEY(\"TrackId\") REFERENCES \"Track\" (\"TrackId\"), \n", - "\tFOREIGN KEY(\"InvoiceId\") REFERENCES \"Invoice\" (\"InvoiceId\")\n", - ")\n", - "\n", - "/*\n", - "3 rows from InvoiceLine table:\n", - "InvoiceLineId\tInvoiceId\tTrackId\tUnitPrice\tQuantity\n", - "1\t1\t2\t0.99\t1\n", - "2\t1\t4\t0.99\t1\n", - "3\t2\t6\t0.99\t1\n", - "*/\u001b[0m\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_query` with `{'query': 'SELECT c.Country, SUM(i.Total) AS TotalSpent FROM Customer c JOIN Invoice i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY TotalSpent DESC LIMIT 1'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[36;1m\u001b[1;3m[('USA', 523.0600000000003)]\u001b[0m\u001b[32;1m\u001b[1;3mCustomers from the USA spent the most, with a total amount spent of $523.06.\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_vnHKe3oul1xbpX0Vrb2vsamZ', 'function': {'arguments': '{\"query\":\"SELECT c.Country, SUM(i.Total) AS Total_Spent FROM customers c JOIN invoices i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY Total_Spent DESC LIMIT 1\"}', 'name': 'sql_db_query'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 53, 'prompt_tokens': 557, 'total_tokens': 610}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-da250593-06b5-414c-a9d9-3fc77036dd9c-0', tool_calls=[{'name': 'sql_db_query', 'args': {'query': 'SELECT c.Country, SUM(i.Total) AS Total_Spent FROM customers c JOIN invoices i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY Total_Spent DESC LIMIT 1'}, 'id': 'call_vnHKe3oul1xbpX0Vrb2vsamZ'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='Error: (sqlite3.OperationalError) no such table: customers\\n[SQL: SELECT c.Country, SUM(i.Total) AS Total_Spent FROM customers c JOIN invoices i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY Total_Spent DESC LIMIT 1]\\n(Background on this error at: https://sqlalche.me/e/20/e3q8)', name='sql_db_query', id='1a5c85d4-1b30-4af3-ab9b-325cbce3b2b4', tool_call_id='call_vnHKe3oul1xbpX0Vrb2vsamZ')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_pp3BBD1hwpdwskUj63G3tgaQ', 'function': {'arguments': '{}', 'name': 'sql_db_list_tables'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 12, 'prompt_tokens': 699, 'total_tokens': 711}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-04cf0e05-61d0-4673-b5dc-1a9b5fd71fff-0', tool_calls=[{'name': 'sql_db_list_tables', 'args': {}, 'id': 'call_pp3BBD1hwpdwskUj63G3tgaQ'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='Album, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track', name='sql_db_list_tables', id='c2668450-4d73-4d32-8d75-8aac8fa153fd', tool_call_id='call_pp3BBD1hwpdwskUj63G3tgaQ')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_22Asbqgdx26YyEvJxBuANVdY', 'function': {'arguments': '{\"query\":\"SELECT c.Country, SUM(i.Total) AS Total_Spent FROM Customer c JOIN Invoice i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY Total_Spent DESC LIMIT 1\"}', 'name': 'sql_db_query'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 53, 'prompt_tokens': 744, 'total_tokens': 797}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-bdd94241-ca49-4f15-b31a-b7c728a34ea8-0', tool_calls=[{'name': 'sql_db_query', 'args': {'query': 'SELECT c.Country, SUM(i.Total) AS Total_Spent FROM Customer c JOIN Invoice i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY Total_Spent DESC LIMIT 1'}, 'id': 'call_22Asbqgdx26YyEvJxBuANVdY'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content=\"[('USA', 523.0600000000003)]\", name='sql_db_query', id='f647e606-8362-40ab-8d34-612ff166dbe1', tool_call_id='call_22Asbqgdx26YyEvJxBuANVdY')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='Customers from the USA spent the most, with a total amount spent of $523.06.', response_metadata={'token_usage': {'completion_tokens': 20, 'prompt_tokens': 819, 'total_tokens': 839}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-92e88de0-ff62-41da-8181-053fb5632af4-0')]}}\n", + "----\n" ] - }, - { - "data": { - "text/plain": [ - "{'input': \"Which country's customers spent the most?\",\n", - " 'output': 'Customers from the USA spent the most, with a total amount spent of $523.06.'}" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "agent_executor.invoke({\"input\": \"Which country's customers spent the most?\"})" + "for s in agent_executor.stream({\"messages\": [HumanMessage(content=\"Which country's customers spent the most?\")]}):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -548,382 +529,34 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\n", - "\u001b[1m> Entering new SQL Agent Executor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_list_tables` with `{}`\n", - "\n", - "\n", - "\u001b[0m\u001b[38;5;200m\u001b[1;3mAlbum, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track\u001b[0m\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_schema` with `{'table_names': 'PlaylistTrack'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[33;1m\u001b[1;3m\n", - "CREATE TABLE \"PlaylistTrack\" (\n", - "\t\"PlaylistId\" INTEGER NOT NULL, \n", - "\t\"TrackId\" INTEGER NOT NULL, \n", - "\tPRIMARY KEY (\"PlaylistId\", \"TrackId\"), \n", - "\tFOREIGN KEY(\"TrackId\") REFERENCES \"Track\" (\"TrackId\"), \n", - "\tFOREIGN KEY(\"PlaylistId\") REFERENCES \"Playlist\" (\"PlaylistId\")\n", - ")\n", - "\n", - "/*\n", - "3 rows from PlaylistTrack table:\n", - "PlaylistId\tTrackId\n", - "1\t3402\n", - "1\t3389\n", - "1\t3390\n", - "*/\u001b[0m\u001b[32;1m\u001b[1;3mThe `PlaylistTrack` table has the following columns:\n", - "- PlaylistId (INTEGER, NOT NULL)\n", - "- TrackId (INTEGER, NOT NULL)\n", - "\n", - "It has a composite primary key on the columns PlaylistId and TrackId. Additionally, there are foreign key constraints on TrackId referencing the Track table's TrackId column, and on PlaylistId referencing the Playlist table's PlaylistId column.\n", - "\n", - "Here are 3 sample rows from the `PlaylistTrack` table:\n", - "1. PlaylistId: 1, TrackId: 3402\n", - "2. PlaylistId: 1, TrackId: 3389\n", - "3. PlaylistId: 1, TrackId: 3390\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" - ] - }, - { - "data": { - "text/plain": [ - "{'input': 'Describe the playlisttrack table',\n", - " 'output': \"The `PlaylistTrack` table has the following columns:\\n- PlaylistId (INTEGER, NOT NULL)\\n- TrackId (INTEGER, NOT NULL)\\n\\nIt has a composite primary key on the columns PlaylistId and TrackId. Additionally, there are foreign key constraints on TrackId referencing the Track table's TrackId column, and on PlaylistId referencing the Playlist table's PlaylistId column.\\n\\nHere are 3 sample rows from the `PlaylistTrack` table:\\n1. PlaylistId: 1, TrackId: 3402\\n2. PlaylistId: 1, TrackId: 3389\\n3. PlaylistId: 1, TrackId: 3390\"}" - ] - }, - "execution_count": 16, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "agent_executor.invoke(\"Describe the playlisttrack table\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Using a dynamic few-shot prompt\n", - "\n", - "To optimize agent performance, we can provide a custom prompt with domain-specific knowledge. In this case we'll create a few shot prompt with an example selector, that will dynamically build the few shot prompt based on the user input. This will help the model make better queries by inserting relevant queries in the prompt that the model can use as reference.\n", - "\n", - "First we need some user input <> SQL query examples:" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "examples = [\n", - " {\"input\": \"List all artists.\", \"query\": \"SELECT * FROM Artist;\"},\n", - " {\n", - " \"input\": \"Find all albums for the artist 'AC/DC'.\",\n", - " \"query\": \"SELECT * FROM Album WHERE ArtistId = (SELECT ArtistId FROM Artist WHERE Name = 'AC/DC');\",\n", - " },\n", - " {\n", - " \"input\": \"List all tracks in the 'Rock' genre.\",\n", - " \"query\": \"SELECT * FROM Track WHERE GenreId = (SELECT GenreId FROM Genre WHERE Name = 'Rock');\",\n", - " },\n", - " {\n", - " \"input\": \"Find the total duration of all tracks.\",\n", - " \"query\": \"SELECT SUM(Milliseconds) FROM Track;\",\n", - " },\n", - " {\n", - " \"input\": \"List all customers from Canada.\",\n", - " \"query\": \"SELECT * FROM Customer WHERE Country = 'Canada';\",\n", - " },\n", - " {\n", - " \"input\": \"How many tracks are there in the album with ID 5?\",\n", - " \"query\": \"SELECT COUNT(*) FROM Track WHERE AlbumId = 5;\",\n", - " },\n", - " {\n", - " \"input\": \"Find the total number of invoices.\",\n", - " \"query\": \"SELECT COUNT(*) FROM Invoice;\",\n", - " },\n", - " {\n", - " \"input\": \"List all tracks that are longer than 5 minutes.\",\n", - " \"query\": \"SELECT * FROM Track WHERE Milliseconds > 300000;\",\n", - " },\n", - " {\n", - " \"input\": \"Who are the top 5 customers by total purchase?\",\n", - " \"query\": \"SELECT CustomerId, SUM(Total) AS TotalPurchase FROM Invoice GROUP BY CustomerId ORDER BY TotalPurchase DESC LIMIT 5;\",\n", - " },\n", - " {\n", - " \"input\": \"Which albums are from the year 2000?\",\n", - " \"query\": \"SELECT * FROM Album WHERE strftime('%Y', ReleaseDate) = '2000';\",\n", - " },\n", - " {\n", - " \"input\": \"How many employees are there\",\n", - " \"query\": 'SELECT COUNT(*) FROM \"Employee\"',\n", - " },\n", - "]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Although stuffing all of these examples in a prompt is an option, we can also intelligently select examples that are most relevant to a given query. For this we can use example selectors. An example selector will take the actual user input and select some number of examples to add to our few-shot prompt. We'll use a SemanticSimilarityExampleSelector, which will perform a semantic search using the embeddings and vector store we configure to find the examples most similar to our input:" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "from langchain_community.vectorstores import FAISS\n", - "from langchain_core.example_selectors import SemanticSimilarityExampleSelector\n", - "from langchain_openai import OpenAIEmbeddings\n", - "\n", - "example_selector = SemanticSimilarityExampleSelector.from_examples(\n", - " examples,\n", - " OpenAIEmbeddings(),\n", - " FAISS,\n", - " k=5,\n", - " input_keys=[\"input\"],\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's try it out:" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[{'input': 'How many employees are there',\n", - " 'query': 'SELECT COUNT(*) FROM \"Employee\"'},\n", - " {'input': 'Find the total number of invoices.',\n", - " 'query': 'SELECT COUNT(*) FROM Invoice;'},\n", - " {'input': 'Who are the top 5 customers by total purchase?',\n", - " 'query': 'SELECT CustomerId, SUM(Total) AS TotalPurchase FROM Invoice GROUP BY CustomerId ORDER BY TotalPurchase DESC LIMIT 5;'},\n", - " {'input': 'List all customers from Canada.',\n", - " 'query': \"SELECT * FROM Customer WHERE Country = 'Canada';\"},\n", - " {'input': 'How many tracks are there in the album with ID 5?',\n", - " 'query': 'SELECT COUNT(*) FROM Track WHERE AlbumId = 5;'}]" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "example_selector.select_examples({\"input\": \"How many employees are there\"})" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that this particular selector is simply identifying examples whose inputs are semantically similar to a given input.\n", - "\n", - "LangChain's `FewShotPromptTemplate` functions as a typical prompt template, but will retrieve relevant few-shot examples using the example selector and format them into the prompt. Let's instantiate one:" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [], - "source": [ - "from langchain_core.prompts import (\n", - " ChatPromptTemplate,\n", - " FewShotPromptTemplate,\n", - " MessagesPlaceholder,\n", - " PromptTemplate,\n", - " SystemMessagePromptTemplate,\n", - ")\n", - "\n", - "system_prefix = \"\"\"You are an agent designed to interact with a SQL database.\n", - "Given an input question, create a syntactically correct {dialect} query to run, then look at the results of the query and return the answer.\n", - "Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most {top_k} results.\n", - "You can order the results by a relevant column to return the most interesting examples in the database.\n", - "Never query for all the columns from a specific table, only ask for the relevant columns given the question.\n", - "You have access to tools for interacting with the database.\n", - "Only use the given tools. Only use the information returned by the tools to construct your final answer.\n", - "You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.\n", - "\n", - "DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.\n", - "\n", - "If the question does not seem related to the database, just return \"I don't know\" as the answer.\n", - "\n", - "Here are some examples of user inputs and their corresponding SQL queries:\"\"\"\n", - "\n", - "few_shot_prompt = FewShotPromptTemplate(\n", - " example_selector=example_selector,\n", - " example_prompt=PromptTemplate.from_template(\n", - " \"User input: {input}\\nSQL query: {query}\"\n", - " ),\n", - " input_variables=[\"input\", \"dialect\", \"top_k\"],\n", - " prefix=system_prefix,\n", - " suffix=\"\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Since our underlying agent is a [tool-calling agent](/docs/modules/agents/agent_types/tool_calling), our full prompt should be a chat prompt with a human message template and an agent_scratchpad `MessagesPlaceholder`. The few-shot prompt will be used for our system message:" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [], - "source": [ - "full_prompt = ChatPromptTemplate.from_messages(\n", - " [\n", - " SystemMessagePromptTemplate(prompt=few_shot_prompt),\n", - " (\"human\", \"{input}\"),\n", - " MessagesPlaceholder(\"agent_scratchpad\"),\n", - " ]\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can invoke the prompt template to inspect how an individual query would be formatted:" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "System: You are an agent designed to interact with a SQL database.\n", - "Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer.\n", - "Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results.\n", - "You can order the results by a relevant column to return the most interesting examples in the database.\n", - "Never query for all the columns from a specific table, only ask for the relevant columns given the question.\n", - "You have access to tools for interacting with the database.\n", - "Only use the given tools. Only use the information returned by the tools to construct your final answer.\n", - "You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.\n", - "\n", - "DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.\n", - "\n", - "If the question does not seem related to the database, just return \"I don't know\" as the answer.\n", - "\n", - "Here are some examples of user inputs and their corresponding SQL queries:\n", - "\n", - "User input: List all artists.\n", - "SQL query: SELECT * FROM Artist;\n", - "\n", - "User input: How many employees are there\n", - "SQL query: SELECT COUNT(*) FROM \"Employee\"\n", - "\n", - "User input: How many tracks are there in the album with ID 5?\n", - "SQL query: SELECT COUNT(*) FROM Track WHERE AlbumId = 5;\n", - "\n", - "User input: List all tracks in the 'Rock' genre.\n", - "SQL query: SELECT * FROM Track WHERE GenreId = (SELECT GenreId FROM Genre WHERE Name = 'Rock');\n", - "\n", - "User input: Which albums are from the year 2000?\n", - "SQL query: SELECT * FROM Album WHERE strftime('%Y', ReleaseDate) = '2000';\n", - "Human: How many arists are there\n" + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_WN0N3mm8WFvPXYlK9P7KvIEr', 'function': {'arguments': '{\"table_names\":\"playlisttrack\"}', 'name': 'sql_db_schema'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 17, 'prompt_tokens': 554, 'total_tokens': 571}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-be278326-4115-4c67-91a0-6dc97e7bffa4-0', tool_calls=[{'name': 'sql_db_schema', 'args': {'table_names': 'playlisttrack'}, 'id': 'call_WN0N3mm8WFvPXYlK9P7KvIEr'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content=\"Error: table_names {'playlisttrack'} not found in database\", name='sql_db_schema', id='fe32b3d3-a40f-4802-a6b8-87a2453af8c2', tool_call_id='call_WN0N3mm8WFvPXYlK9P7KvIEr')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='I apologize for the error. Let me first check the available tables in the database.', additional_kwargs={'tool_calls': [{'id': 'call_CzHt30847ql2MmnGxgYeVSL2', 'function': {'arguments': '{}', 'name': 'sql_db_list_tables'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 30, 'prompt_tokens': 592, 'total_tokens': 622}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-f6c107bb-e945-4848-a83c-f57daec1144e-0', tool_calls=[{'name': 'sql_db_list_tables', 'args': {}, 'id': 'call_CzHt30847ql2MmnGxgYeVSL2'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='Album, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track', name='sql_db_list_tables', id='a4950f74-a0ad-4558-ba54-7bcf99539a02', tool_call_id='call_CzHt30847ql2MmnGxgYeVSL2')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='The database contains a table named \"PlaylistTrack\". Let me retrieve the schema and sample rows from the \"PlaylistTrack\" table.', additional_kwargs={'tool_calls': [{'id': 'call_wX9IjHLgRBUmxlfCthprABRO', 'function': {'arguments': '{\"table_names\":\"PlaylistTrack\"}', 'name': 'sql_db_schema'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 44, 'prompt_tokens': 658, 'total_tokens': 702}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-e8d34372-1159-4654-a185-1e7d0cb70269-0', tool_calls=[{'name': 'sql_db_schema', 'args': {'table_names': 'PlaylistTrack'}, 'id': 'call_wX9IjHLgRBUmxlfCthprABRO'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='\\nCREATE TABLE \"PlaylistTrack\" (\\n\\t\"PlaylistId\" INTEGER NOT NULL, \\n\\t\"TrackId\" INTEGER NOT NULL, \\n\\tPRIMARY KEY (\"PlaylistId\", \"TrackId\"), \\n\\tFOREIGN KEY(\"TrackId\") REFERENCES \"Track\" (\"TrackId\"), \\n\\tFOREIGN KEY(\"PlaylistId\") REFERENCES \"Playlist\" (\"PlaylistId\")\\n)\\n\\n/*\\n3 rows from PlaylistTrack table:\\nPlaylistId\\tTrackId\\n1\\t3402\\n1\\t3389\\n1\\t3390\\n*/', name='sql_db_schema', id='f6ffc37a-188a-4690-b84e-c9f2c78b1e49', tool_call_id='call_wX9IjHLgRBUmxlfCthprABRO')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='The \"PlaylistTrack\" table has the following schema:\\n- PlaylistId: INTEGER (NOT NULL)\\n- TrackId: INTEGER (NOT NULL)\\n- Primary Key: (PlaylistId, TrackId)\\n- Foreign Key: TrackId references Track(TrackId)\\n- Foreign Key: PlaylistId references Playlist(PlaylistId)\\n\\nHere are 3 sample rows from the \"PlaylistTrack\" table:\\n1. PlaylistId: 1, TrackId: 3402\\n2. PlaylistId: 1, TrackId: 3389\\n3. PlaylistId: 1, TrackId: 3390\\n\\nIf you have any specific questions or queries regarding the \"PlaylistTrack\" table, feel free to ask!', response_metadata={'token_usage': {'completion_tokens': 145, 'prompt_tokens': 818, 'total_tokens': 963}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-961a4552-3cbd-4d28-b338-4d2f1ac40ea0-0')]}}\n", + "----\n" ] } ], "source": [ - "prompt_val = full_prompt.invoke(\n", - " {\n", - " \"input\": \"How many arists are there\",\n", - " \"top_k\": 5,\n", - " \"dialect\": \"SQLite\",\n", - " \"agent_scratchpad\": [],\n", - " }\n", - ")\n", - "print(prompt_val.to_string())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can then build the agent in the same way. Calls to the agent will pass the query through the example selector in order to build the few-shot examples into the system prompt." - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [], - "source": [ - "agent = create_sql_agent(\n", - " llm=llm,\n", - " db=db,\n", - " prompt=full_prompt,\n", - " verbose=True,\n", - " agent_type=\"tool-calling\",\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "\n", - "\u001b[1m> Entering new SQL Agent Executor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_query` with `{'query': 'SELECT COUNT(*) FROM Artist;'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[36;1m\u001b[1;3m[(275,)]\u001b[0m\u001b[32;1m\u001b[1;3mThere are 275 artists in the database.\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" - ] - }, - { - "data": { - "text/plain": [ - "{'input': 'How many artists are there?',\n", - " 'output': 'There are 275 artists in the database.'}" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "agent.invoke({\"input\": \"How many artists are there?\"})" + "for s in agent_executor.stream({\"messages\": [HumanMessage(content=\"Describe the playlisttrack table\")]}):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -941,20 +574,20 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "['Great Recordings of the Century - Mahler: Das Lied von der Erde',\n", - " 'Holst: The Planets, Op. & Vaughan Williams: Fantasies',\n", - " 'Chemical Wedding',\n", - " 'By The Way',\n", - " 'The Singles']" + "['Big Ones',\n", + " 'Cidade Negra - Hits',\n", + " 'In Step',\n", + " 'Use Your Illusion I',\n", + " 'Voodoo Lounge']" ] }, - "execution_count": 25, + "execution_count": 36, "metadata": {}, "output_type": "execute_result" } @@ -985,11 +618,13 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "from langchain.agents.agent_toolkits import create_retriever_tool\n", + "from langchain_community.vectorstores import FAISS\n", + "from langchain_openai import OpenAIEmbeddings\n", "\n", "vector_db = FAISS.from_texts(artists + albums, OpenAIEmbeddings())\n", "retriever = vector_db.as_retriever(search_kwargs={\"k\": 5})\n", @@ -1011,7 +646,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 40, "metadata": {}, "outputs": [ { @@ -1045,13 +680,13 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "system = \"\"\"You are an agent designed to interact with a SQL database.\n", - "Given an input question, create a syntactically correct {dialect} query to run, then look at the results of the query and return the answer.\n", - "Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most {top_k} results.\n", + "Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer.\n", + "Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results.\n", "You can order the results by a relevant column to return the most interesting examples in the database.\n", "Never query for all the columns from a specific table, only ask for the relevant columns given the question.\n", "You have access to tools for interacting with the database.\n", @@ -1060,72 +695,40 @@ "\n", "DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.\n", "\n", - "If you need to filter on a proper noun, you must ALWAYS first look up the filter value using the \"search_proper_nouns\" tool! \n", - "\n", "You have access to the following tables: {table_names}\n", "\n", - "If the question does not seem related to the database, just return \"I don't know\" as the answer.\"\"\"\n", - "\n", - "prompt = ChatPromptTemplate.from_messages(\n", - " [(\"system\", system), (\"human\", \"{input}\"), MessagesPlaceholder(\"agent_scratchpad\")]\n", + "If you need to filter on a proper noun, you must ALWAYS first look up the filter value using the \"search_proper_nouns\" tool!\n", + "Do not try to guess at the proper name - use this function to find similar ones.\"\"\".format(\n", + " table_names=db.get_usable_table_names()\n", ")\n", - "agent = create_sql_agent(\n", - " llm=llm,\n", - " db=db,\n", - " extra_tools=[retriever_tool],\n", - " prompt=prompt,\n", - " agent_type=\"tool-calling\",\n", - " verbose=True,\n", - ")" + "\n", + "system_message = SystemMessage(content=system)\n", + "\n", + "agent = chat_agent_executor.create_tool_calling_executor(llm, tools, system_message=system_message)" ] }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 51, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "\n", - "\n", - "\u001b[1m> Entering new SQL Agent Executor chain...\u001b[0m\n", - "\u001b[32;1m\u001b[1;3m\n", - "Invoking: `search_proper_nouns` with `{'query': 'alis in chain'}`\n", - "\n", - "\n", - "\u001b[0m\u001b[36;1m\u001b[1;3mAlice In Chains\n", - "\n", - "Aisha Duo\n", - "\n", - "Xis\n", - "\n", - "Da Lama Ao Caos\n", - "\n", - "A-Sides\u001b[0m\u001b[32;1m\u001b[1;3m\n", - "Invoking: `sql_db_query` with `{'query': \"SELECT COUNT(*) AS album_count FROM Album WHERE ArtistId IN (SELECT ArtistId FROM Artist WHERE Name = 'Alice In Chains')\"}`\n", - "\n", - "\n", - "\u001b[0m\u001b[36;1m\u001b[1;3m[(1,)]\u001b[0m\u001b[32;1m\u001b[1;3mAlice In Chains has 1 album.\u001b[0m\n", - "\n", - "\u001b[1m> Finished chain.\u001b[0m\n" + "{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_r5UlSwHKQcWDHx6LrttnqE56', 'function': {'arguments': '{\"query\":\"SELECT COUNT(*) AS album_count FROM Album WHERE ArtistId IN (SELECT ArtistId FROM Artist WHERE Name = \\'Alice In Chains\\')\"}', 'name': 'sql_db_query'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 40, 'prompt_tokens': 612, 'total_tokens': 652}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-548353fd-b06c-45bf-beab-46f81eb434df-0', tool_calls=[{'name': 'sql_db_query', 'args': {'query': \"SELECT COUNT(*) AS album_count FROM Album WHERE ArtistId IN (SELECT ArtistId FROM Artist WHERE Name = 'Alice In Chains')\"}, 'id': 'call_r5UlSwHKQcWDHx6LrttnqE56'}])]}}\n", + "----\n", + "{'action': {'messages': [ToolMessage(content='[(1,)]', name='sql_db_query', id='093058a9-f013-4be1-8e7a-ed839b0c90cd', tool_call_id='call_r5UlSwHKQcWDHx6LrttnqE56')]}}\n", + "----\n", + "{'agent': {'messages': [AIMessage(content='Alice In Chains has 11 albums.', response_metadata={'token_usage': {'completion_tokens': 9, 'prompt_tokens': 665, 'total_tokens': 674}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-f804eaab-9812-4fb3-ae8b-280af8594ac6-0')]}}\n", + "----\n" ] - }, - { - "data": { - "text/plain": [ - "{'input': 'How many albums does alis in chain have?',\n", - " 'output': 'Alice In Chains has 1 album.'}" - ] - }, - "execution_count": 29, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "agent.invoke({\"input\": \"How many albums does alis in chain have?\"})" + "for s in agent.stream({\"messages\": [HumanMessage(content=\"How many albums does alis in chain have?\")]}):\n", + " print(s)\n", + " print(\"----\")" ] }, { @@ -1134,17 +737,6 @@ "source": [ "As we can see, the agent used the `search_proper_nouns` tool in order to check how to correctly query the database for this specific artist." ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Next steps\n", - "\n", - "Under the hood, `create_sql_agent` is just passing in SQL tools to more generic agent constructors. To learn more about the built-in generic agent types as well as how to build custom agents, head to the [Agents Modules](/docs/modules/agents/).\n", - "\n", - "The built-in `AgentExecutor` runs a simple Agent action -> Tool call -> Agent action... loop. To build more complex agent runtimes, head to the [LangGraph section](/docs/langgraph)." - ] } ], "metadata": {