Suggested diff for guardrails error handling

- ValidationError is now a subclass of Exception
- Each OutputParser can now declare its own subclass of ValidationError that people can reference in a catch statement

README.md

@@ -79,4 +79,4 @@ For more information on these concepts, please see our [full documentation](http
 
 As an open source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infra, or better documentation.
 
-For detailed information on how to contribute, see [here](CONTRIBUTING.md).
+For detailed information on how to contribute, see [here](.github/CONTRIBUTING.md).

docs/conf.py

@@ -30,6 +30,7 @@ version = data["tool"]["poetry"]["version"]
 release = version
 
 html_title = project + " " + version
+html_last_updated_fmt = "%b %d, %Y"
 
 
 # -- General configuration ---------------------------------------------------
@@ -45,6 +46,7 @@ extensions = [
     "sphinx.ext.viewcode",
     "sphinxcontrib.autodoc_pydantic",
     "myst_nb",
+    "sphinx_copybutton",
     "sphinx_panels",
     "IPython.sphinxext.ipython_console_highlighting",
 ]

docs/ecosystem/atlas.md

@@ -1,19 +1,21 @@
 # AtlasDB
 
-This page covers how to Nomic's Atlas ecosystem within LangChain.
+This page covers how to use Nomic's Atlas ecosystem within LangChain.
 It is broken into two parts: installation and setup, and then references to specific Atlas wrappers.
 
 ## Installation and Setup
 - Install the Python package with `pip install nomic`
 - Nomic is also included in langchains poetry extras `poetry install -E all`
-- 
+
 ## Wrappers
 
 ### VectorStore
 
 There exists a wrapper around the Atlas neural database, allowing you to use it as a vectorstore.
 This vectorstore also gives you full access to the underlying AtlasProject object, which will allow you to use the full range of Atlas map interactions, such as bulk tagging and automatic topic modeling.
-Please see [the Nomic docs](https://docs.nomic.ai/atlas_api.html) for more detailed information.
+Please see [the Atlas docs](https://docs.nomic.ai/atlas_api.html) for more detailed information.
+
+
 
 
 
@@ -22,4 +24,4 @@ To import this vectorstore:
 from langchain.vectorstores import AtlasDB
 ```
 
-For a more detailed walkthrough of the Chroma wrapper, see [this notebook](../modules/indexes/examples/vectorstores.ipynb)
+For a more detailed walkthrough of the AtlasDB wrapper, see [this notebook](../modules/indexes/vectorstore_examples/atlas.ipynb)

docs/ecosystem/bananadev.md

@@ -5,7 +5,7 @@ It is broken into two parts: installation and setup, and then references to spec
 
 ## Installation and Setup
 
-- Install with `pip3 install banana-dev`
+- Install with `pip install banana-dev`
 - Get an Banana api key and set it as an environment variable (`BANANA_API_KEY`)
 
 ## Define your Banana Template

docs/ecosystem/graphsignal.md

@@ -1,6 +1,6 @@
 # Graphsignal
 
-This page covers how to use the Graphsignal to trace and monitor LangChain.
+This page covers how to use the Graphsignal ecosystem to trace and monitor LangChain.
 
 ## Installation and Setup
 

docs/ecosystem/helicone.md

@@ -1,6 +1,6 @@
 # Helicone
 
-This page covers how to use the [Helicone](https://helicone.ai) within LangChain.
+This page covers how to use the [Helicone](https://helicone.ai) ecosystem within LangChain.
 
 ## What is Helicone?
 

docs/ecosystem/pgvector.md

@@ -0,0 +1,29 @@
+# PGVector
+
+This page covers how to use the Postgres [PGVector](https://github.com/pgvector/pgvector) ecosystem within LangChain
+It is broken into two parts: installation and setup, and then references to specific PGVector wrappers.
+
+## Installation
+- Install the Python package with `pip install pgvector`
+
+
+## Setup
+1. The first step is to create a database with the `pgvector` extension installed.
+
+    Follow the steps at [PGVector Installation Steps](https://github.com/pgvector/pgvector#installation) to install the database and the extension. The docker image is the easiest way to get started.
+
+## Wrappers
+
+### VectorStore
+
+There exists a wrapper around Postgres vector databases, allowing you to use it as a vectorstore,
+whether for semantic search or example selection.
+
+To import this vectorstore:
+```python
+from langchain.vectorstores.pgvector import PGVector
+```
+
+### Usage
+
+For a more detailed walkthrough of the PGVector Wrapper, see [this notebook](../modules/indexes/vectorstore_examples/pgvector.ipynb)

docs/ecosystem/promptlayer.md

@@ -29,3 +29,5 @@ This LLM is identical to the [OpenAI LLM](./openai), except that
 - all your requests will be logged to your PromptLayer account
 - you can add `pl_tags` when instantializing to tag your requests on PromptLayer
 
+
+PromptLayer also provides native wrappers for [`PromptLayerChatOpenAI`](../modules/chat/examples/promptlayer_chat_openai.ipynb)

docs/ecosystem/unstructured.md

@@ -17,9 +17,12 @@ This page is broken into two parts: installation and setup, and then references
     - `poppler-utils`
     - `tesseract-ocr`
     - `libreoffice`
-- If you are parsing PDFs, run the following to install the `detectron2` model, which
+- If you are parsing PDFs using the `"hi_res"` strategy, run the following to install the `detectron2` model, which
   `unstructured` uses for layout detection:
     - `pip install "detectron2@git+https://github.com/facebookresearch/detectron2.git@v0.6#egg=detectron2"`
+    - If `detectron2` is not installed, `unstructured` will fallback to processing PDFs
+      using the `"fast"` strategy, which uses `pdfminer` directly and doesn't require
+      `detectron2`.
 
 ## Wrappers
 

docs/modules/agents/agent_toolkits/vectorstore.ipynb

@@ -92,7 +92,7 @@
    "id": "f4814175-964d-42f1-aa9d-22801ce1e912",
    "metadata": {},
    "source": [
-    "## Initalize Toolkit and Agent\n",
+    "## Initialize Toolkit and Agent\n",
     "\n",
     "First, we'll create an agent with a single vectorstore."
    ]

docs/modules/agents/examples/chat_conversation_agent.ipynb

@@ -0,0 +1,309 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4658d71a",
+   "metadata": {},
+   "source": [
+    "# Conversation Agent (for Chat Models)\n",
+    "\n",
+    "This notebook walks through using an agent optimized for conversation, using ChatModels. Other agents are often optimized for using tools to figure out the best response, which is not ideal in a conversational setting where you may want the agent to be able to chat with the user as well.\n",
+    "\n",
+    "This is accomplished with a specific type of agent (`chat-conversational-react-description`) which expects to be used with a memory component."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f4f5d1a8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "f65308ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import Tool\n",
+    "from langchain.memory import ConversationBufferMemory\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
+    "from langchain.agents import initialize_agent"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "5fb14d6d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search = SerpAPIWrapper()\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name = \"Current Search\",\n",
+    "        func=search.run,\n",
+    "        description=\"useful for when you need to answer questions about current events or the current state of the world. the input to this should be a single search term.\"\n",
+    "    ),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "dddc34c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "memory = ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "cafe9bc1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm=ChatOpenAI(temperature=0)\n",
+    "agent_chain = initialize_agent(tools, llm, agent=\"chat-conversational-react-description\", verbose=True, memory=memory)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "dc70b454",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m{\n",
+      "    \"action\": \"Final Answer\",\n",
+      "    \"action_input\": \"Hello Bob! How can I assist you today?\"\n",
+      "}\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Hello Bob! How can I assist you today?'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"hi, i am bob\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "3dcf7953",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m{\n",
+      "    \"action\": \"Final Answer\",\n",
+      "    \"action_input\": \"Your name is Bob.\"\n",
+      "}\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Your name is Bob.'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"what's my name?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "aa05f566",
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m{\n",
+      "    \"action\": \"Current Search\",\n",
+      "    \"action_input\": \"Thai food dinner recipes\"\n",
+      "}\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3m59 easy Thai recipes for any night of the week · Marion Grasby's Thai spicy chilli and basil fried rice · Thai curry noodle soup · Marion Grasby's ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m{\n",
+      "    \"action\": \"Final Answer\",\n",
+      "    \"action_input\": \"Here are some Thai food dinner recipes you can make this week: Thai spicy chilli and basil fried rice, Thai curry noodle soup, and many more. You can find 59 easy Thai recipes for any night of the week on Marion Grasby's website.\"\n",
+      "}\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"Here are some Thai food dinner recipes you can make this week: Thai spicy chilli and basil fried rice, Thai curry noodle soup, and many more. You can find 59 easy Thai recipes for any night of the week on Marion Grasby's website.\""
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(\"what are some good dinners to make this week, if i like thai food?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "c5d8b7ea",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m```json\n",
+      "{\n",
+      "    \"action\": \"Current Search\",\n",
+      "    \"action_input\": \"who won the world cup in 1978\"\n",
+      "}\n",
+      "```\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mThe Argentina national football team represents Argentina in men's international football and is administered by the Argentine Football Association, the governing body for football in Argentina. Nicknamed La Albiceleste, they are the reigning world champions, having won the most recent World Cup in 2022.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m```json\n",
+      "{\n",
+      "    \"action\": \"Final Answer\",\n",
+      "    \"action_input\": \"The last letter in your name is 'b'. The Argentina national football team won the World Cup in 1978.\"\n",
+      "}\n",
+      "```\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"The last letter in your name is 'b'. The Argentina national football team won the World Cup in 1978.\""
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"tell me the last letter in my name, and also tell me who won the world cup in 1978?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "f608889b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m{\n",
+      "    \"action\": \"Current Search\",\n",
+      "    \"action_input\": \"weather in pomfret\"\n",
+      "}\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mMostly cloudy with gusty winds developing during the afternoon. A few flurries or snow showers possible. High near 40F. Winds NNW at 20 to 30 mph.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m{\n",
+      "    \"action\": \"Final Answer\",\n",
+      "    \"action_input\": \"The weather in Pomfret is mostly cloudy with gusty winds developing during the afternoon. A few flurries or snow showers are possible. High near 40F. Winds NNW at 20 to 30 mph.\"\n",
+      "}\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'The weather in Pomfret is mostly cloudy with gusty winds developing during the afternoon. A few flurries or snow showers are possible. High near 40F. Winds NNW at 20 to 30 mph.'"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"whats the weather like in pomfret?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0084efd6",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/examples/sharedmemory_for_tools.ipynb

@@ -0,0 +1,552 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "fa6802ac",
+   "metadata": {},
+   "source": [
+    "# Adding SharedMemory to an Agent and its Tools\n",
+    "\n",
+    "This notebook goes over adding memory to **both** of an Agent and its tools. Before going through this notebook, please walk through the following notebooks, as this will build on top of both of them:\n",
+    "\n",
+    "- [Adding memory to an LLM Chain](../../memory/examples/adding_memory.ipynb)\n",
+    "- [Custom Agents](custom_agent.ipynb)\n",
+    "\n",
+    "We are going to create a custom Agent. The agent has access to a conversation memory, search tool, and a summarization tool. And, the summarization tool also needs access to the conversation memory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "8db95912",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import ZeroShotAgent, Tool, AgentExecutor\n",
+    "from langchain.memory import ConversationBufferMemory, ReadOnlySharedMemory\n",
+    "from langchain import OpenAI, LLMChain, PromptTemplate\n",
+    "from langchain.utilities import GoogleSearchAPIWrapper"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "06b7187b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "template = \"\"\"This is a conversation between a human and a bot:\n",
+    "\n",
+    "{chat_history}\n",
+    "\n",
+    "Write a summary of the conversation for {input}:\n",
+    "\"\"\"\n",
+    "\n",
+    "prompt = PromptTemplate(\n",
+    "    input_variables=[\"input\", \"chat_history\"], \n",
+    "    template=template\n",
+    ")\n",
+    "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
+    "readonlymemory = ReadOnlySharedMemory(memory=memory)\n",
+    "summry_chain = LLMChain(\n",
+    "    llm=OpenAI(), \n",
+    "    prompt=prompt, \n",
+    "    verbose=True, \n",
+    "    memory=readonlymemory, # use the read-only memory to prevent the tool from modifying the memory\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "97ad8467",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search = GoogleSearchAPIWrapper()\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name = \"Search\",\n",
+    "        func=search.run,\n",
+    "        description=\"useful for when you need to answer questions about current events\"\n",
+    "    ),\n",
+    "    Tool(\n",
+    "        name = \"Summary\",\n",
+    "        func=summry_chain.run,\n",
+    "        description=\"useful for when you summarize a conversation. The input to this tool should be a string, representing who will read this summary.\"\n",
+    "    )\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "e3439cd6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prefix = \"\"\"Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:\"\"\"\n",
+    "suffix = \"\"\"Begin!\"\n",
+    "\n",
+    "{chat_history}\n",
+    "Question: {input}\n",
+    "{agent_scratchpad}\"\"\"\n",
+    "\n",
+    "prompt = ZeroShotAgent.create_prompt(\n",
+    "    tools, \n",
+    "    prefix=prefix, \n",
+    "    suffix=suffix, \n",
+    "    input_variables=[\"input\", \"chat_history\", \"agent_scratchpad\"]\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0021675b",
+   "metadata": {},
+   "source": [
+    "We can now construct the LLMChain, with the Memory object, and then create the agent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "c56a0e73",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=prompt)\n",
+    "agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)\n",
+    "agent_chain = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "ca4bc1fb",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: I should research ChatGPT to answer this question.\n",
+      "Action: Search\n",
+      "Action Input: \"ChatGPT\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mNov 30, 2022 ... We've trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer ... ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large ... ChatGPT. We've trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer ... Feb 2, 2023 ... ChatGPT, the popular chatbot from OpenAI, is estimated to have reached 100 million monthly active users in January, just two months after ... 2 days ago ... ChatGPT recently launched a new version of its own plagiarism detection tool, with hopes that it will squelch some of the criticism around how ... An API for accessing new AI models developed by OpenAI. Feb 19, 2023 ... ChatGPT is an AI chatbot system that OpenAI released in November to show off and test what a very large, powerful AI system can accomplish. You ... ChatGPT is fine-tuned from GPT-3.5, a language model trained to produce text. ChatGPT was optimized for dialogue by using Reinforcement Learning with Human ... 3 days ago ... Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. Dec 1, 2022 ... ChatGPT is a natural language processing tool driven by AI technology that allows you to have human-like conversations and much more with a ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\""
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"What is ChatGPT?\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "45627664",
+   "metadata": {},
+   "source": [
+    "To test the memory of this agent, we can ask a followup question that relies on information in the previous exchange to be answered correctly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "eecc0462",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: I need to find out who developed ChatGPT\n",
+      "Action: Search\n",
+      "Action Input: Who developed ChatGPT\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large ... Feb 15, 2023 ... Who owns Chat GPT? Chat GPT is owned and developed by AI research and deployment company, OpenAI. The organization is headquartered in San ... Feb 8, 2023 ... ChatGPT is an AI chatbot developed by San Francisco-based startup OpenAI. OpenAI was co-founded in 2015 by Elon Musk and Sam Altman and is ... Dec 7, 2022 ... ChatGPT is an AI chatbot designed and developed by OpenAI. The bot works by generating text responses based on human-user input, like questions ... Jan 12, 2023 ... In 2019, Microsoft invested $1 billion in OpenAI, the tiny San Francisco company that designed ChatGPT. And in the years since, it has quietly ... Jan 25, 2023 ... The inside story of ChatGPT: How OpenAI founder Sam Altman built the world's hottest technology with billions from Microsoft. Dec 3, 2022 ... ChatGPT went viral on social media for its ability to do anything from code to write essays. · The company that created the AI chatbot has a ... Jan 17, 2023 ... While many Americans were nursing hangovers on New Year's Day, 22-year-old Edward Tian was working feverishly on a new app to combat misuse ... ChatGPT is a language model created by OpenAI, an artificial intelligence research laboratory consisting of a team of researchers and engineers focused on ... 1 day ago ... Everyone is talking about ChatGPT, developed by OpenAI. This is such a great tool that has helped to make AI more accessible to a wider ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: ChatGPT was developed by OpenAI.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'ChatGPT was developed by OpenAI.'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"Who developed it?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "c34424cf",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: I need to simplify the conversation for a 5 year old.\n",
+      "Action: Summary\n",
+      "Action Input: My daughter 5 years old\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mThis is a conversation between a human and a bot:\n",
+      "\n",
+      "Human: What is ChatGPT?\n",
+      "AI: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\n",
+      "Human: Who developed it?\n",
+      "AI: ChatGPT was developed by OpenAI.\n",
+      "\n",
+      "Write a summary of the conversation for My daughter 5 years old:\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "Observation: \u001b[33;1m\u001b[1;3m\n",
+      "The conversation was about ChatGPT, an artificial intelligence chatbot. It was created by OpenAI and can send and receive images while chatting.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: ChatGPT is an artificial intelligence chatbot created by OpenAI that can send and receive images while chatting.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'ChatGPT is an artificial intelligence chatbot created by OpenAI that can send and receive images while chatting.'"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"Thanks. Summarize the conversation, for my daughter 5 years old.\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "4ebd8326",
+   "metadata": {},
+   "source": [
+    "Confirm that the memory was correctly updated."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "b91f8c85",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Human: What is ChatGPT?\n",
+      "AI: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\n",
+      "Human: Who developed it?\n",
+      "AI: ChatGPT was developed by OpenAI.\n",
+      "Human: Thanks. Summarize the conversation, for my daughter 5 years old.\n",
+      "AI: ChatGPT is an artificial intelligence chatbot created by OpenAI that can send and receive images while chatting.\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(agent_chain.memory.buffer)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "cc3d0aa4",
+   "metadata": {},
+   "source": [
+    "For comparison, below is a bad example that uses the same memory for both the Agent and the tool."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "3359d043",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## This is a bad practice for using the memory.\n",
+    "## Use the ReadOnlySharedMemory class, as shown above.\n",
+    "\n",
+    "template = \"\"\"This is a conversation between a human and a bot:\n",
+    "\n",
+    "{chat_history}\n",
+    "\n",
+    "Write a summary of the conversation for {input}:\n",
+    "\"\"\"\n",
+    "\n",
+    "prompt = PromptTemplate(\n",
+    "    input_variables=[\"input\", \"chat_history\"], \n",
+    "    template=template\n",
+    ")\n",
+    "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
+    "summry_chain = LLMChain(\n",
+    "    llm=OpenAI(), \n",
+    "    prompt=prompt, \n",
+    "    verbose=True, \n",
+    "    memory=memory,  # <--- this is the only change\n",
+    ")\n",
+    "\n",
+    "search = GoogleSearchAPIWrapper()\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name = \"Search\",\n",
+    "        func=search.run,\n",
+    "        description=\"useful for when you need to answer questions about current events\"\n",
+    "    ),\n",
+    "    Tool(\n",
+    "        name = \"Summary\",\n",
+    "        func=summry_chain.run,\n",
+    "        description=\"useful for when you summarize a conversation. The input to this tool should be a string, representing who will read this summary.\"\n",
+    "    )\n",
+    "]\n",
+    "\n",
+    "prefix = \"\"\"Have a conversation with a human, answering the following questions as best you can. You have access to the following tools:\"\"\"\n",
+    "suffix = \"\"\"Begin!\"\n",
+    "\n",
+    "{chat_history}\n",
+    "Question: {input}\n",
+    "{agent_scratchpad}\"\"\"\n",
+    "\n",
+    "prompt = ZeroShotAgent.create_prompt(\n",
+    "    tools, \n",
+    "    prefix=prefix, \n",
+    "    suffix=suffix, \n",
+    "    input_variables=[\"input\", \"chat_history\", \"agent_scratchpad\"]\n",
+    ")\n",
+    "\n",
+    "llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=prompt)\n",
+    "agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)\n",
+    "agent_chain = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "970d23df",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: I should research ChatGPT to answer this question.\n",
+      "Action: Search\n",
+      "Action Input: \"ChatGPT\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mNov 30, 2022 ... We've trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer ... ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large ... ChatGPT. We've trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer ... Feb 2, 2023 ... ChatGPT, the popular chatbot from OpenAI, is estimated to have reached 100 million monthly active users in January, just two months after ... 2 days ago ... ChatGPT recently launched a new version of its own plagiarism detection tool, with hopes that it will squelch some of the criticism around how ... An API for accessing new AI models developed by OpenAI. Feb 19, 2023 ... ChatGPT is an AI chatbot system that OpenAI released in November to show off and test what a very large, powerful AI system can accomplish. You ... ChatGPT is fine-tuned from GPT-3.5, a language model trained to produce text. ChatGPT was optimized for dialogue by using Reinforcement Learning with Human ... 3 days ago ... Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. Dec 1, 2022 ... ChatGPT is a natural language processing tool driven by AI technology that allows you to have human-like conversations and much more with a ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\""
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"What is ChatGPT?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "d9ea82f0",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: I need to find out who developed ChatGPT\n",
+      "Action: Search\n",
+      "Action Input: Who developed ChatGPT\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large ... Feb 15, 2023 ... Who owns Chat GPT? Chat GPT is owned and developed by AI research and deployment company, OpenAI. The organization is headquartered in San ... Feb 8, 2023 ... ChatGPT is an AI chatbot developed by San Francisco-based startup OpenAI. OpenAI was co-founded in 2015 by Elon Musk and Sam Altman and is ... Dec 7, 2022 ... ChatGPT is an AI chatbot designed and developed by OpenAI. The bot works by generating text responses based on human-user input, like questions ... Jan 12, 2023 ... In 2019, Microsoft invested $1 billion in OpenAI, the tiny San Francisco company that designed ChatGPT. And in the years since, it has quietly ... Jan 25, 2023 ... The inside story of ChatGPT: How OpenAI founder Sam Altman built the world's hottest technology with billions from Microsoft. Dec 3, 2022 ... ChatGPT went viral on social media for its ability to do anything from code to write essays. · The company that created the AI chatbot has a ... Jan 17, 2023 ... While many Americans were nursing hangovers on New Year's Day, 22-year-old Edward Tian was working feverishly on a new app to combat misuse ... ChatGPT is a language model created by OpenAI, an artificial intelligence research laboratory consisting of a team of researchers and engineers focused on ... 1 day ago ... Everyone is talking about ChatGPT, developed by OpenAI. This is such a great tool that has helped to make AI more accessible to a wider ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: ChatGPT was developed by OpenAI.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'ChatGPT was developed by OpenAI.'"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"Who developed it?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "5b1f9223",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: I need to simplify the conversation for a 5 year old.\n",
+      "Action: Summary\n",
+      "Action Input: My daughter 5 years old\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mThis is a conversation between a human and a bot:\n",
+      "\n",
+      "Human: What is ChatGPT?\n",
+      "AI: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\n",
+      "Human: Who developed it?\n",
+      "AI: ChatGPT was developed by OpenAI.\n",
+      "\n",
+      "Write a summary of the conversation for My daughter 5 years old:\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "Observation: \u001b[33;1m\u001b[1;3m\n",
+      "The conversation was about ChatGPT, an artificial intelligence chatbot developed by OpenAI. It is designed to have conversations with humans and can also send and receive images.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: ChatGPT is an artificial intelligence chatbot developed by OpenAI that can have conversations with humans and send and receive images.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'ChatGPT is an artificial intelligence chatbot developed by OpenAI that can have conversations with humans and send and receive images.'"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent_chain.run(input=\"Thanks. Summarize the conversation, for my daughter 5 years old.\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "d07415da",
+   "metadata": {},
+   "source": [
+    "The final answer is not wrong, but we see the 3rd Human input is actually from the agent in the memory because the memory was modified by the summary tool."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "32f97b21",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Human: What is ChatGPT?\n",
+      "AI: ChatGPT is an artificial intelligence chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3 family of large language models and is optimized for dialogue by using Reinforcement Learning with Human-in-the-Loop. It is also capable of sending and receiving images during chatting.\n",
+      "Human: Who developed it?\n",
+      "AI: ChatGPT was developed by OpenAI.\n",
+      "Human: My daughter 5 years old\n",
+      "AI: \n",
+      "The conversation was about ChatGPT, an artificial intelligence chatbot developed by OpenAI. It is designed to have conversations with humans and can also send and receive images.\n",
+      "Human: Thanks. Summarize the conversation, for my daughter 5 years old.\n",
+      "AI: ChatGPT is an artificial intelligence chatbot developed by OpenAI that can have conversations with humans and send and receive images.\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(agent_chain.memory.buffer)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/implementations/mrkl_chat.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f1390152",
+   "metadata": {},
+   "source": [
+    "# MRKL Chat\n",
+    "\n",
+    "This notebook showcases using an agent to replicate the MRKL chain using an agent optimized for chat models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "39ea3638",
+   "metadata": {},
+   "source": [
+    "This uses the example Chinook database.\n",
+    "To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the `.db` file in a notebooks folder at the root of this repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ac561cc4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain import OpenAI, LLMMathChain, SerpAPIWrapper, SQLDatabase, SQLDatabaseChain\n",
+    "from langchain.agents import initialize_agent, Tool\n",
+    "from langchain.chat_models import ChatOpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "07e96d99",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = ChatOpenAI(temperature=0)\n",
+    "llm1 = OpenAI(temperature=0)\n",
+    "search = SerpAPIWrapper()\n",
+    "llm_math_chain = LLMMathChain(llm=llm1, verbose=True)\n",
+    "db = SQLDatabase.from_uri(\"sqlite:///../../../../notebooks/Chinook.db\")\n",
+    "db_chain = SQLDatabaseChain(llm=llm1, database=db, verbose=True)\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name = \"Search\",\n",
+    "        func=search.run,\n",
+    "        description=\"useful for when you need to answer questions about current events. You should ask targeted questions\"\n",
+    "    ),\n",
+    "    Tool(\n",
+    "        name=\"Calculator\",\n",
+    "        func=llm_math_chain.run,\n",
+    "        description=\"useful for when you need to answer questions about math\"\n",
+    "    ),\n",
+    "    Tool(\n",
+    "        name=\"FooBar DB\",\n",
+    "        func=db_chain.run,\n",
+    "        description=\"useful for when you need to answer questions about FooBar. Input should be in the form of a question containing full context\"\n",
+    "    )\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "a069c4b6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mrkl = initialize_agent(tools, llm, agent=\"chat-zero-shot-react-description\", verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "e603cd7d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mThought: The first question requires a search, while the second question requires a calculator.\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"Search\",\n",
+      "  \"action_input\": \"Who is Leo DiCaprio's girlfriend?\"\n",
+      "}\n",
+      "```\n",
+      "\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mFor the second question, I need to use the calculator tool to raise her current age to the 0.43 power.\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"Calculator\",\n",
+      "  \"action_input\": \"22.0^(0.43)\"\n",
+      "}\n",
+      "```\n",
+      "\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
+      "22.0^(0.43)\u001b[32;1m\u001b[1;3m\n",
+      "```python\n",
+      "import math\n",
+      "print(math.pow(22.0, 0.43))\n",
+      "```\n",
+      "\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m3.777824273683966\n",
+      "\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.777824273683966\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mI now know the final answer.\n",
+      "Final Answer: Camila Morrone, 3.777824273683966.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Camila Morrone, 3.777824273683966.'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "mrkl.run(\"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "a5c07010",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mQuestion: What is the full name of the artist who recently released an album called 'The Storm Before the Calm' and are they in the FooBar database? If so, what albums of theirs are in the FooBar database?\n",
+      "Thought: I should use the Search tool to find the answer to the first part of the question and then use the FooBar DB tool to find the answer to the second part of the question.\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"Search\",\n",
+      "  \"action_input\": \"Who recently released an album called 'The Storm Before the Calm'\"\n",
+      "}\n",
+      "```\n",
+      "\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAlanis Morissette\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mNow that I have the name of the artist, I can use the FooBar DB tool to find their albums in the database.\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"FooBar DB\",\n",
+      "  \"action_input\": \"What albums does Alanis Morissette have in the database?\"\n",
+      "}\n",
+      "```\n",
+      "\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
+      "What albums does Alanis Morissette have in the database? \n",
+      "SQLQuery:"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/harrisonchase/workplace/langchain/langchain/sql_database.py:141: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
+      "  sample_rows = connection.execute(command)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[32;1m\u001b[1;3m SELECT Title FROM Album WHERE ArtistId IN (SELECT ArtistId FROM Artist WHERE Name = 'Alanis Morissette') LIMIT 5;\u001b[0m\n",
+      "SQLResult: \u001b[33;1m\u001b[1;3m[('Jagged Little Pill',)]\u001b[0m\n",
+      "Answer:\u001b[32;1m\u001b[1;3m Alanis Morissette has the album 'Jagged Little Pill' in the database.\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "Observation: \u001b[38;5;200m\u001b[1;3m Alanis Morissette has the album 'Jagged Little Pill' in the database.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mI have found the answer to both parts of the question.\n",
+      "Final Answer: The artist who recently released an album called 'The Storm Before the Calm' is Alanis Morissette. The album 'Jagged Little Pill' is in the FooBar database.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"The artist who recently released an album called 'The Storm Before the Calm' is Alanis Morissette. The album 'Jagged Little Pill' is in the FooBar database.\""
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "mrkl.run(\"What is the full name of the artist who recently released an album called 'The Storm Before the Calm' and are they in the FooBar database? If so, what albums of theirs are in the FooBar database?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "af016a70",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/tools.md

@@ -136,3 +136,12 @@ Below is a list of all supported tools and relevant information:
 - Requires LLM: No
 - Extra Parameters: `serper_api_key`
 - For more information on this, see [this page](../../ecosystem/google_serper.md)
+
+**wikipedia**
+
+- Tool Name: Wikipedia
+- Tool Description: A wrapper around Wikipedia. Useful for when you need to answer general questions about people, places, companies, historical events, or other subjects. Input should be a search query.
+- Notes: Uses the [wikipedia](https://pypi.org/project/wikipedia/) Python package to call the MediaWiki API and then parses results.
+- Requires LLM: No
+- Extra Parameters: `top_k_results`
+

docs/modules/chains/examples/sqlite.ipynb

@@ -377,18 +377,19 @@
       "\tFOREIGN KEY(\"GenreId\") REFERENCES \"Genre\" (\"GenreId\"), \n",
       "\tFOREIGN KEY(\"AlbumId\") REFERENCES \"Album\" (\"AlbumId\")\n",
       ")\n",
-      "\n",
-      "SELECT * FROM 'Track' LIMIT 2;\n",
+      "/*\n",
+      "2 rows from Track table:\n",
       "TrackId\tName\tAlbumId\tMediaTypeId\tGenreId\tComposer\tMilliseconds\tBytes\tUnitPrice\n",
       "1\tFor Those About To Rock (We Salute You)\t1\t1\t1\tAngus Young, Malcolm Young, Brian Johnson\t343719\t11170334\t0.99\n",
-      "2\tBalls to the Wall\t2\t2\t1\tNone\t342562\t5510424\t0.99\n"
+      "2\tBalls to the Wall\t2\t2\t1\tNone\t342562\t5510424\t0.99\n",
+      "*/\n"
      ]
     },
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/home/jon/projects/langchain/langchain/sql_database.py:121: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
+      "/home/jon/projects/langchain/langchain/sql_database.py:135: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
       "  sample_rows = connection.execute(command)\n"
      ]
     }
@@ -467,12 +468,13 @@
     "\t\"Composer\" NVARCHAR(220),\n",
     "\tPRIMARY KEY (\"TrackId\")\n",
     ")\n",
-    "\n",
-    "SELECT * FROM 'Track' LIMIT 3;\n",
+    "/*\n",
+    "3 rows from Track table:\n",
     "TrackId\tName\tComposer\n",
     "1\tFor Those About To Rock (We Salute You)\tAngus Young, Malcolm Young, Brian Johnson\n",
     "2\tBalls to the Wall\tNone\n",
-    "3\tMy favorite song ever\tThe coolest composer of all time\"\"\"\n",
+    "3\tMy favorite song ever\tThe coolest composer of all time\n",
+    "*/\"\"\"\n",
     "}"
    ]
   },
@@ -492,11 +494,12 @@
       "\t\"Name\" NVARCHAR(120), \n",
       "\tPRIMARY KEY (\"PlaylistId\")\n",
       ")\n",
-      "\n",
-      "SELECT * FROM 'Playlist' LIMIT 2;\n",
+      "/*\n",
+      "2 rows from Playlist table:\n",
       "PlaylistId\tName\n",
       "1\tMusic\n",
       "2\tMovies\n",
+      "*/\n",
       "\n",
       "CREATE TABLE Track (\n",
       "\t\"TrackId\" INTEGER NOT NULL, \n",
@@ -504,12 +507,13 @@
       "\t\"Composer\" NVARCHAR(220),\n",
       "\tPRIMARY KEY (\"TrackId\")\n",
       ")\n",
-      "\n",
-      "SELECT * FROM 'Track' LIMIT 3;\n",
+      "/*\n",
+      "3 rows from Track table:\n",
       "TrackId\tName\tComposer\n",
       "1\tFor Those About To Rock (We Salute You)\tAngus Young, Malcolm Young, Brian Johnson\n",
       "2\tBalls to the Wall\tNone\n",
-      "3\tMy favorite song ever\tThe coolest composer of all time\n"
+      "3\tMy favorite song ever\tThe coolest composer of all time\n",
+      "*/\n"
      ]
     }
    ],
@@ -675,7 +679,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/chat/examples/promptlayer_chatopenai.ipynb

@@ -0,0 +1,154 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "959300d4",
+   "metadata": {},
+   "source": [
+    "# PromptLayer ChatOpenAI\n",
+    "\n",
+    "This example showcases how to connect to [PromptLayer](https://www.promptlayer.com) to start recording your ChatOpenAI requests."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "6a45943e",
+   "metadata": {},
+   "source": [
+    "## Install PromptLayer\n",
+    "The `promptlayer` package is required to use PromptLayer with OpenAI. Install `promptlayer` using pip."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dbe09bd8",
+   "metadata": {
+    "vscode": {
+     "languageId": "powershell"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "pip install promptlayer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "536c1dfa",
+   "metadata": {},
+   "source": [
+    "## Imports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c16da3b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from langchain.chat_models import PromptLayerChatOpenAI\n",
+    "from langchain.schema import HumanMessage"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "8564ce7d",
+   "metadata": {},
+   "source": [
+    "## Set the Environment API Key\n",
+    "You can create a PromptLayer API Key at [wwww.promptlayer.com](https://ww.promptlayer.com) by clicking the settings cog in the navbar.\n",
+    "\n",
+    "Set it as an environment variable called `PROMPTLAYER_API_KEY`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "46ba25dc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"PROMPTLAYER_API_KEY\"] = \"**********\""
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "bf0294de",
+   "metadata": {},
+   "source": [
+    "## Use the PromptLayerOpenAI LLM like normal\n",
+    "*You can optionally pass in `pl_tags` to track your requests with PromptLayer's tagging feature.*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "3acf0069",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content='to take a nap in a cozy spot. I search around for a suitable place and finally settle on a soft cushion on the window sill. I curl up into a ball and close my eyes, relishing the warmth of the sun on my fur. As I drift off to sleep, I can hear the birds chirping outside and feel the gentle breeze blowing through the window. This is the life of a contented cat.', additional_kwargs={})"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat = PromptLayerChatOpenAI(pl_tags=[\"langchain\"])\n",
+    "chat([HumanMessage(content=\"I am a cat and I want\")])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "a2d76826",
+   "metadata": {},
+   "source": [
+    "**The above request should now appear on your [PromptLayer dashboard](https://ww.promptlayer.com).**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "05e9e2fe",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "base",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.8"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "c4fe2cd85a8d9e8baaec5340ce66faff1c77581a9f43e6c45e85e09b6fced008"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/chat/examples/vector_db_qa_with_sources.ipynb

@@ -165,28 +165,6 @@
    "source": [
     "chain({\"question\": \"What did the president say about Justice Breyer\"}, return_only_outputs=True)"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "c91fdc8a",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'answer': ' The president honored Justice Stephen Breyer for his service.\\n',\n",
-       " 'sources': '30-pl'}"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "qa({\"question\": \"What did the president say about Justice Breyer\"}, return_only_outputs=True)"
-   ]
   }
  ],
  "metadata": {

docs/modules/document_loaders/examples/example_data/mlb_teams_2012.csv

@@ -0,0 +1,32 @@
+"Team", "Payroll (millions)", "Wins"
+"Nationals",     81.34, 98
+"Reds",          82.20, 97
+"Yankees",      197.96, 95
+"Giants",       117.62, 94
+"Braves",        83.31, 94
+"Athletics",     55.37, 94
+"Rangers",      120.51, 93
+"Orioles",       81.43, 93
+"Rays",          64.17, 90
+"Angels",       154.49, 89
+"Tigers",       132.30, 88
+"Cardinals",    110.30, 88
+"Dodgers",       95.14, 86
+"White Sox",     96.92, 85
+"Brewers",       97.65, 83
+"Phillies",     174.54, 81
+"Diamondbacks",  74.28, 81
+"Pirates",       63.43, 79
+"Padres",        55.24, 76
+"Mariners",      81.97, 75
+"Mets",          93.35, 74
+"Blue Jays",     75.48, 73
+"Royals",        60.91, 72
+"Marlins",      118.07, 69
+"Red Sox",      173.18, 69
+"Indians",       78.43, 68
+"Twins",         94.08, 66
+"Rockies",       78.06, 64
+"Cubs",          88.19, 61
+"Astros",        60.65, 55
+

docs/modules/document_loaders/examples/markdown.ipynb

@@ -0,0 +1,145 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "39af9ecd",
+   "metadata": {},
+   "source": [
+    "# Markdown\n",
+    "\n",
+    "This covers how to load markdown documents into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "721c48aa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredMarkdownLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "9d3d0e35",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredMarkdownLoader(\"../../../../README.md\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "06073f91",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "c9adc5cb",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content=\"ð\\x9f¦\\x9cï¸\\x8fð\\x9f”\\x97 LangChain\\n\\nâ\\x9a¡ Building applications with LLMs through composability â\\x9a¡\\n\\nProduction Support: As you move your LangChains into production, we'd love to offer more comprehensive support.\\nPlease fill out this form and we'll set up a dedicated support Slack channel.\\n\\nQuick Install\\n\\npip install langchain\\n\\nð\\x9f¤” What is this?\\n\\nLarge language models (LLMs) are emerging as a transformative technology, enabling\\ndevelopers to build applications that they previously could not.\\nBut using these LLMs in isolation is often not enough to\\ncreate a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.\\n\\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\\n\\nâ\\x9d“ Question Answering over specific documents\\n\\nDocumentation\\n\\nEnd-to-end Example: Question Answering over Notion Database\\n\\nð\\x9f’¬ Chatbots\\n\\nDocumentation\\n\\nEnd-to-end Example: Chat-LangChain\\n\\nð\\x9f¤\\x96 Agents\\n\\nDocumentation\\n\\nEnd-to-end Example: GPT+WolframAlpha\\n\\nð\\x9f“\\x96 Documentation\\n\\nPlease see here for full documentation on:\\n\\nGetting started (installation, setting up the environment, simple examples)\\n\\nHow-To examples (demos, integrations, helper functions)\\n\\nReference (full API docs)\\n  Resources (high-level explanation of core concepts)\\n\\nð\\x9f\\x9a\\x80 What can this help with?\\n\\nThere are six main areas that LangChain is designed to help with.\\nThese are, in increasing order of complexity:\\n\\nð\\x9f“\\x83 LLMs and Prompts:\\n\\nThis includes prompt management, prompt optimization, generic interface for all LLMs, and common utilities for working with LLMs.\\n\\nð\\x9f”\\x97 Chains:\\n\\nChains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\\n\\nð\\x9f“\\x9a Data Augmented Generation:\\n\\nData Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources.\\n\\nð\\x9f¤\\x96 Agents:\\n\\nAgents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.\\n\\nð\\x9f§\\xa0 Memory:\\n\\nMemory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\\n\\nð\\x9f§\\x90 Evaluation:\\n\\n[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\\n\\nFor more information on these concepts, please see our full documentation.\\n\\nð\\x9f’\\x81 Contributing\\n\\nAs an open source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infra, or better documentation.\\n\\nFor detailed information on how to contribute, see here.\", lookup_str='', metadata={'source': '../../../../README.md'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "525d6b67",
+   "metadata": {},
+   "source": [
+    "## Retain Elements\n",
+    "\n",
+    "Under the hood, Unstructured creates different \"elements\" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying `mode=\"elements\"`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "064f9162",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredMarkdownLoader(\"../../../../README.md\", mode=\"elements\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "abefbbdb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a547c534",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Document(page_content='ð\\x9f¦\\x9cï¸\\x8fð\\x9f”\\x97 LangChain', lookup_str='', metadata={'source': '../../../../README.md', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0)"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "381d4139",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/unstructured_file.ipynb

@@ -158,7 +158,72 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7874d01d",
+   "id": "672733fd",
+   "metadata": {},
+   "source": [
+    "## Define a Partitioning Strategy\n",
+    "\n",
+    "Unstructured document loader allow users to pass in a `strategy` parameter that lets `unstructured` know how to partitioning the document. Currently supported strategies are `\"hi_res\"` (the default) and `\"fast\"`. Hi res partitioning strategies are more accurate, but take longer to process. Fast strategies partition the document more quickly, but trade-off accuracy. Not all document types have separate hi res and fast partitioning strategies. For those document types, the `strategy` kwarg is ignored. In some cases, the high res strategy will fallback to fast if there is a dependency missing (i.e. a model for document partitioning). You can see how to apply a strategy to an `UnstructuredFileLoader` below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "767238a4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredFileLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "9518b425",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredFileLoader(\"layout-parser-paper-fast.pdf\", strategy=\"fast\", mode=\"elements\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "645f29e9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "60685353",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='1', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0),\n",
+       " Document(page_content='2', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0),\n",
+       " Document(page_content='0', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0),\n",
+       " Document(page_content='2', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0),\n",
+       " Document(page_content='n', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'Title'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "docs[:5]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8de9ef16",
    "metadata": {},
    "source": [
     "## PDF Example\n",
@@ -166,7 +231,6 @@
     "Processing PDF documents works exactly the same way. Unstructured detects the file type and extracts the same types of `elements`. "
    ]
   },
-
   {
    "cell_type": "code",
    "execution_count": 1,
@@ -225,7 +289,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8ca8a648",
+   "id": "f52b04cb",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -247,7 +311,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.8.13"
   }
  },
  "nbformat": 4,

docs/modules/document_loaders/examples/youtube.ipynb

@@ -7,22 +7,23 @@
    "source": [
     "# YouTube\n",
     "\n",
-    "How to load documents from YouTube transcripts."
+    "How to load documents from YouTube transcripts.\n",
+    "\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "id": "da4a867f",
+   "execution_count": null,
+   "id": "427d5745",
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.document_loaders import YoutubeLoader"
+    "from langchain.document_loaders import YoutubeLoader\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "id": "34a25b57",
    "metadata": {
     "scrolled": true
@@ -34,7 +35,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "id": "bc8b308a",
    "metadata": {},
    "outputs": [],
@@ -44,21 +45,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "d073dd36",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='LADIES AND GENTLEMEN, PEDRO PASCAL! [ CHEERS AND APPLAUSE ] >> THANK YOU, THANK YOU. THANK YOU VERY MUCH. I\\'M SO EXCITED TO BE HERE. THANK YOU. I SPENT THE LAST YEAR SHOOTING A SHOW CALLED \"THE LAST OF US\" ON HBO. FOR SOME HBO SHOES, YOU GET TO SHOOT IN A FIVE STAR ITALIAN RESORT SURROUNDED BY BEAUTIFUL PEOPLE, BUT I SAID, NO, THAT\\'S TOO EASY. I WANT TO SHOOT IN A FREEZING CANADIAN FOREST WHILE BEING CHASED AROUND BY A GUY WHOSE HEAD LOOKS LIKE A GENITAL WART. IT IS AN HONOR BEING A PART OF THESE HUGE FRANCHISEs LIKE \"GAME OF THRONES\" AND \"STAR WARS,\" BUT I\\'M STILL GETTING USED TO PEOPLE RECOGNIZING ME. THE OTHER DAY, A GUY STOPPED ME ON THE STREET AND SAYS, MY SON LOVES \"THE MANDALORIAN\" AND THE NEXT THING I KNOW, I\\'M FACE TIMING WITH A 6-YEAR-OLD WHO HAS NO IDEA WHO I AM BECAUSE MY CHARACTER WEARS A MASK THE ENTIRE SHOW. THE GUY IS LIKE, DO THE MANDO VOICE, BUT IT\\'S LIKE A BEDROOM VOICE. WITHOUT THE MASK, IT JUST SOUNDS PORNY. PEOPLE WALKING BY ON THE STREET SEE ME WHISPERING TO A 6-YEAR-OLD KID. I CAN BRING YOU IN WARM, OR I CAN BRING YOU IN COLD. EVEN THOUGH I CAME TO THE U.S. WHEN I WAS LITTLE, I WAS BORN IN CHILE, AND I HAVE 34 FIRST COUSINS WHO ARE STILL THERE. THEY\\'RE VERY PROUD OF ME. I KNOW THEY\\'RE PROUD BECAUSE THEY GIVE MY PHONE NUMBER TO EVERY PERSON THEY MEET, WHICH MEANS EVERY DAY, SOMEONE IN SANTIAGO WILL TEXT ME STUFF LIKE, CAN YOU COME TO MY WEDDING, OR CAN YOU SING MY PRIEST HAPPY BIRTHDAY, OR IS BABY YODA MEAN IN REAL LIFE. SO I HAVE TO BE LIKE NO, NO, AND HIS NAME IS GROGU. BUT MY COUSINS WEREN\\'T ALWAYS SO PROUD. EARLY IN MY CAREER, I PLAYED SMALL PARTS IN EVERY CRIME SHOW. I EVEN PLAYED TWO DIFFERENT CHARACTERS ON \"LAW AND ORDER.\" TITO CABASSA WHO LOOKED LIKE THIS. AND ONE YEAR LATER, I PLAYED REGGIE LUCKMAN WHO LOOKS LIKE THIS. AND THAT, MY FRIENDS, IS CALLED RANGE. BUT IT IS AMAZING TO BE HERE, LIKE I SAID. I WAS BORN IN CHILE, AND NINE MONTHS LATER, MY PARENTS FLED AND BROUGHT ME AND MY SISTER TO THE U.S. THEY WERE SO BRAVE, AND WITHOUT THEM, I WOULDN\\'T BE HERE IN THIS WONDERFUL COUNTRY, AND I CERTAINLY WOULDN\\'T BE STANDING HERE WITH YOU ALL TONIGHT. SO TO ALL MY FAMILY WATCHING IN CHILE, I WANT TO SAY [ SPEAKING NON-ENGLISH ] WHICH MEANS, I LOVE YOU, I MISS YOU, AND STOP GIVING OUT MY PHONE NUMBER. WE\\'VE GOT AN AMAZING SHOW FOR YOU TONIGHT. COLDPLAY IS HERE, SO STICK', lookup_str='', metadata={'source': 'QsYGlZkevEg', 'title': 'Pedro Pascal Monologue - SNL', 'description': 'First-time host Pedro Pascal talks about filming The Last of Us and being recognized by fans.\\n\\nSaturday Night Live. Stream now on Peacock: https://pck.tv/3uQxh4q\\n\\nSubscribe to SNL: https://goo.gl/tUsXwM\\nStream Current Full Episodes: http://www.nbc.com/saturday-night-live\\n\\nWATCH PAST SNL SEASONS\\nGoogle Play - http://bit.ly/SNLGooglePlay\\niTunes - http://bit.ly/SNLiTunes\\n\\nSNL ON SOCIAL\\nSNL Instagram: http://instagram.com/nbcsnl\\nSNL Facebook: https://www.facebook.com/snl\\nSNL Twitter: https://twitter.com/nbcsnl\\nSNL TikTok: https://www.tiktok.com/@nbcsnl\\n\\nGET MORE NBC\\nLike NBC: http://Facebook.com/NBC\\nFollow NBC: http://Twitter.com/NBC\\nNBC Tumblr: http://NBCtv.tumblr.com/\\nYouTube: http://www.youtube.com/nbc\\nNBC Instagram: http://instagram.com/nbc\\n\\n#SNL #PedroPascal #SNL48 #Coldplay', 'view_count': 1175057, 'thumbnail_url': 'https://i.ytimg.com/vi/QsYGlZkevEg/sddefault.jpg', 'publish_date': datetime.datetime(2023, 2, 4, 0, 0), 'length': 224, 'author': 'Saturday Night Live'}, lookup_index=0)]"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
     "loader.load()"
    ]
@@ -73,7 +63,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "id": "ba28af69",
    "metadata": {},
    "outputs": [],
@@ -83,7 +73,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "id": "9b8ea390",
    "metadata": {},
    "outputs": [],
@@ -93,24 +83,61 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": null,
    "id": "97b98e92",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='LADIES AND GENTLEMEN, PEDRO PASCAL! [ CHEERS AND APPLAUSE ] >> THANK YOU, THANK YOU. THANK YOU VERY MUCH. I\\'M SO EXCITED TO BE HERE. THANK YOU. I SPENT THE LAST YEAR SHOOTING A SHOW CALLED \"THE LAST OF US\" ON HBO. FOR SOME HBO SHOES, YOU GET TO SHOOT IN A FIVE STAR ITALIAN RESORT SURROUNDED BY BEAUTIFUL PEOPLE, BUT I SAID, NO, THAT\\'S TOO EASY. I WANT TO SHOOT IN A FREEZING CANADIAN FOREST WHILE BEING CHASED AROUND BY A GUY WHOSE HEAD LOOKS LIKE A GENITAL WART. IT IS AN HONOR BEING A PART OF THESE HUGE FRANCHISEs LIKE \"GAME OF THRONES\" AND \"STAR WARS,\" BUT I\\'M STILL GETTING USED TO PEOPLE RECOGNIZING ME. THE OTHER DAY, A GUY STOPPED ME ON THE STREET AND SAYS, MY SON LOVES \"THE MANDALORIAN\" AND THE NEXT THING I KNOW, I\\'M FACE TIMING WITH A 6-YEAR-OLD WHO HAS NO IDEA WHO I AM BECAUSE MY CHARACTER WEARS A MASK THE ENTIRE SHOW. THE GUY IS LIKE, DO THE MANDO VOICE, BUT IT\\'S LIKE A BEDROOM VOICE. WITHOUT THE MASK, IT JUST SOUNDS PORNY. PEOPLE WALKING BY ON THE STREET SEE ME WHISPERING TO A 6-YEAR-OLD KID. I CAN BRING YOU IN WARM, OR I CAN BRING YOU IN COLD. EVEN THOUGH I CAME TO THE U.S. WHEN I WAS LITTLE, I WAS BORN IN CHILE, AND I HAVE 34 FIRST COUSINS WHO ARE STILL THERE. THEY\\'RE VERY PROUD OF ME. I KNOW THEY\\'RE PROUD BECAUSE THEY GIVE MY PHONE NUMBER TO EVERY PERSON THEY MEET, WHICH MEANS EVERY DAY, SOMEONE IN SANTIAGO WILL TEXT ME STUFF LIKE, CAN YOU COME TO MY WEDDING, OR CAN YOU SING MY PRIEST HAPPY BIRTHDAY, OR IS BABY YODA MEAN IN REAL LIFE. SO I HAVE TO BE LIKE NO, NO, AND HIS NAME IS GROGU. BUT MY COUSINS WEREN\\'T ALWAYS SO PROUD. EARLY IN MY CAREER, I PLAYED SMALL PARTS IN EVERY CRIME SHOW. I EVEN PLAYED TWO DIFFERENT CHARACTERS ON \"LAW AND ORDER.\" TITO CABASSA WHO LOOKED LIKE THIS. AND ONE YEAR LATER, I PLAYED REGGIE LUCKMAN WHO LOOKS LIKE THIS. AND THAT, MY FRIENDS, IS CALLED RANGE. BUT IT IS AMAZING TO BE HERE, LIKE I SAID. I WAS BORN IN CHILE, AND NINE MONTHS LATER, MY PARENTS FLED AND BROUGHT ME AND MY SISTER TO THE U.S. THEY WERE SO BRAVE, AND WITHOUT THEM, I WOULDN\\'T BE HERE IN THIS WONDERFUL COUNTRY, AND I CERTAINLY WOULDN\\'T BE STANDING HERE WITH YOU ALL TONIGHT. SO TO ALL MY FAMILY WATCHING IN CHILE, I WANT TO SAY [ SPEAKING NON-ENGLISH ] WHICH MEANS, I LOVE YOU, I MISS YOU, AND STOP GIVING OUT MY PHONE NUMBER. WE\\'VE GOT AN AMAZING SHOW FOR YOU TONIGHT. COLDPLAY IS HERE, SO STICK', lookup_str='', metadata={'source': 'QsYGlZkevEg', 'title': 'Pedro Pascal Monologue - SNL', 'description': 'First-time host Pedro Pascal talks about filming The Last of Us and being recognized by fans.\\n\\nSaturday Night Live. Stream now on Peacock: https://pck.tv/3uQxh4q\\n\\nSubscribe to SNL: https://goo.gl/tUsXwM\\nStream Current Full Episodes: http://www.nbc.com/saturday-night-live\\n\\nWATCH PAST SNL SEASONS\\nGoogle Play - http://bit.ly/SNLGooglePlay\\niTunes - http://bit.ly/SNLiTunes\\n\\nSNL ON SOCIAL\\nSNL Instagram: http://instagram.com/nbcsnl\\nSNL Facebook: https://www.facebook.com/snl\\nSNL Twitter: https://twitter.com/nbcsnl\\nSNL TikTok: https://www.tiktok.com/@nbcsnl\\n\\nGET MORE NBC\\nLike NBC: http://Facebook.com/NBC\\nFollow NBC: http://Twitter.com/NBC\\nNBC Tumblr: http://NBCtv.tumblr.com/\\nYouTube: http://www.youtube.com/nbc\\nNBC Instagram: http://instagram.com/nbc\\n\\n#SNL #PedroPascal #SNL48 #Coldplay', 'view_count': 1175057, 'thumbnail_url': 'https://i.ytimg.com/vi/QsYGlZkevEg/sddefault.jpg', 'publish_date': datetime.datetime(2023, 2, 4, 0, 0), 'length': 224, 'author': 'Saturday Night Live'}, lookup_index=0)]"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
     "loader.load()"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "65796cc5",
+   "metadata": {},
+   "source": [
+    "## YouTube loader from Google Cloud\n",
+    "\n",
+    "### Prerequisites\n",
+    "\n",
+    "1. Create a Google Cloud project or use an existing project\n",
+    "1. Enable the [Youtube Api](https://console.cloud.google.com/apis/enableflow?apiid=youtube.googleapis.com&project=sixth-grammar-344520)\n",
+    "1. [Authorize credentials for desktop app](https://developers.google.com/drive/api/quickstart/python#authorize_credentials_for_a_desktop_application)\n",
+    "1. `pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib youtube-transcript-api`\n",
+    "\n",
+    "### 🧑 Instructions for ingesting your Google Docs data\n",
+    "By default, the `GoogleDriveLoader` expects the `credentials.json` file to be `~/.credentials/credentials.json`, but this is configurable using the `credentials_file` keyword argument. Same thing with `token.json`. Note that `token.json` will be created automatically the first time you use the loader.\n",
+    "\n",
+    "`GoogleApiYoutubeLoader` can load from a list of Google Docs document ids or a folder id. You can obtain your folder and document id from the URL:\n",
+    "Note depending on your set up, the `service_account_path` needs to be set up. See [here](https://developers.google.com/drive/api/v3/quickstart/python) for more details."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c345bc43",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import GoogleApiClient, GoogleApiYoutubeLoader\n",
+    "\n",
+    "# Init the GoogleApiClient \n",
+    "from pathlib import Path\n",
+    "\n",
+    "\n",
+    "google_api_client = GoogleApiClient(credentials_path=Path(\"your_path_creds.json\"))\n",
+    "\n",
+    "\n",
+    "# Use a Channel\n",
+    "youtube_loader_channel = GoogleApiYoutubeLoader(google_api_client=google_api_client, channel_name=\"Reducible\",captions_language=\"en\")\n",
+    "\n",
+    "# Use Youtube Ids\n",
+    "\n",
+    "youtube_loader_ids = GoogleApiYoutubeLoader(google_api_client=google_api_client, video_ids=[\"TrdevFK_am4\"], add_video_info=True)\n",
+    "\n",
+    "# returns a list of Documents\n",
+    "youtube_loader_channel.load()"
+   ]
   }
  ],
  "metadata": {
@@ -130,6 +157,11 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.9.1"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "604c1013f65d31a2eb1fca07aae054bedd5a5a0d272dbb31e502c81f0b254b99"
+   }
   }
  },
  "nbformat": 4,

docs/modules/document_loaders/how_to_guides.rst

@@ -55,8 +55,6 @@ There are a lot of different document loaders that LangChain supports. Below are
 
 `Airbyte Json <./examples/airbyte_json.html>`_: A walkthrough of how to load data from a local Airbyte JSON file.
 
-`Online PDF <./examples/online_pdf.html>`_: A walkthrough of how to load data from an online PDF.
-
 `CoNLL-U <./examples/CoNLL-U.html>`_: A walkthrough of how to load data from a ConLL-U file.
 
 `iFixit <./examples/ifixit.html>`_: A walkthrough of how to search and load data like guides, technical Q&A's, and device wikis from iFixit.com

docs/modules/indexes/chain_examples/chat_vector_db.ipynb

@@ -268,48 +268,44 @@
   },
   {
    "cell_type": "markdown",
+   "id": "4f49beab",
+   "metadata": {},
    "source": [
     "## Chat Vector DB with `search_distance`\n",
     "If you are using a vector store that supports filtering by search distance, you can add a threshold value parameter."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "5ed8d612",
+   "metadata": {},
    "outputs": [],
    "source": [
     "vectordbkwargs = {\"search_distance\": 0.9}"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "6a7b3459",
+   "metadata": {},
    "outputs": [],
    "source": [
     "qa = ChatVectorDBChain.from_llm(OpenAI(temperature=0), vectorstore, return_source_documents=True)\n",
     "chat_history = []\n",
     "query = \"What did the president say about Ketanji Brown Jackson\"\n",
     "result = qa({\"question\": query, \"chat_history\": chat_history, \"vectordbkwargs\": vectordbkwargs})"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "markdown",
+   "id": "99b96dae",
+   "metadata": {},
    "source": [
     "## Chat Vector DB with `map_reduce`\n",
     "We can also use different types of combine document chains with the Chat Vector DB chain."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
@@ -524,6 +520,71 @@
     "query = \"Did he mention who she suceeded\"\n",
     "result = qa({\"question\": query, \"chat_history\": chat_history})\n"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f793d56b",
+   "metadata": {},
+   "source": [
+    "## get_chat_history Function\n",
+    "You can also specify a `get_chat_history` function, which can be used to format the chat_history string."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "a7ba9d8c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_chat_history(inputs) -> str:\n",
+    "    res = []\n",
+    "    for human, ai in inputs:\n",
+    "        res.append(f\"Human:{human}\\nAI:{ai}\")\n",
+    "    return \"\\n\".join(res)\n",
+    "qa = ChatVectorDBChain.from_llm(OpenAI(temperature=0), vectorstore, get_chat_history=get_chat_history)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "a3e33c0d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chat_history = []\n",
+    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
+    "result = qa({\"question\": query, \"chat_history\": chat_history})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "936dc62f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\""
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result['answer']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b8c26901",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {

docs/modules/indexes/chain_examples/graph_qa.ipynb

@@ -178,16 +178,16 @@
      "text": [
       "\n",
       "\n",
-      "\u001B[1m> Entering new GraphQAChain chain...\u001B[0m\n",
+      "\u001b[1m> Entering new GraphQAChain chain...\u001b[0m\n",
       "Entities Extracted:\n",
-      "\u001B[32;1m\u001B[1;3m Intel\u001B[0m\n",
+      "\u001b[32;1m\u001b[1;3m Intel\u001b[0m\n",
       "Full Context:\n",
-      "\u001B[32;1m\u001B[1;3mIntel is going to build $20 billion semiconductor \"mega site\"\n",
+      "\u001b[32;1m\u001b[1;3mIntel is going to build $20 billion semiconductor \"mega site\"\n",
       "Intel is building state-of-the-art factories\n",
       "Intel is creating 10,000 new good-paying jobs\n",
-      "Intel is helping build Silicon Valley\u001B[0m\n",
+      "Intel is helping build Silicon Valley\u001b[0m\n",
       "\n",
-      "\u001B[1m> Finished chain.\u001B[0m\n"
+      "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
     },
     {
@@ -205,10 +205,76 @@
     "chain.run(\"what is Intel going to build?\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "410aafa0",
+   "metadata": {},
+   "source": [
+    "## Save the graph\n",
+    "We can also save and load the graph."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "bc72cca0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "graph.write_to_gml(\"graph.gml\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "652760ad",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes.graph import NetworkxEntityGraph"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "eae591fe",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loaded_graph = NetworkxEntityGraph.from_gml(\"graph.gml\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "9439d419",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[('Intel', '$20 billion semiconductor \"mega site\"', 'is going to build'),\n",
+       " ('Intel', 'state-of-the-art factories', 'is building'),\n",
+       " ('Intel', '10,000 new good-paying jobs', 'is creating'),\n",
+       " ('Intel', 'Silicon Valley', 'is helping build'),\n",
+       " ('Field of dreams',\n",
+       "  \"America's future will be built\",\n",
+       "  'is the ground on which')]"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loaded_graph.get_triples()"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f70b9ada",
+   "id": "045796cf",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/indexes/chain_examples/qa_with_sources.ipynb

@@ -635,7 +635,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts.base import RegexParser\n",
+    "from langchain.output_parsers import RegexParser\n",
     "\n",
     "output_parser = RegexParser(\n",
     "    regex=r\"(.*?)\\nScore: (.*)\",\n",
@@ -732,4 +732,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

docs/modules/indexes/chain_examples/question_answering.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Question Answering\n",
     "\n",
-    "This notebook walks through how to use LangChain for question answering over a list of documents. It covers four different types of chains: `stuff`, `map_reduce`, `refine`, `map-rerank`. For a more in depth explanation of what these chain types are, see [here](../combine_docs.md)."
+    "This notebook walks through how to use LangChain for question answering over a list of documents. It covers four different types of chains: `stuff`, `map_reduce`, `refine`, `map_rerank`. For a more in depth explanation of what these chain types are, see [here](../combine_docs.md)."
    ]
   },
   {
@@ -635,7 +635,7 @@
     }
    ],
    "source": [
-    "from langchain.prompts.base import RegexParser\n",
+    "from langchain.output_parsers import RegexParser\n",
     "\n",
     "output_parser = RegexParser(\n",
     "    regex=r\"(.*?)\\nScore: (.*)\",\n",

docs/modules/indexes/how_to_guides.rst

@@ -36,6 +36,8 @@ In the below guides, we cover different types of vectorstores and how to use the
 
 `Chroma <./vectorstore_examples/chroma.html>`_: A walkthrough of how to use the Chroma vectorstore wrapper.
 
+`AtlasDB <./vectorstore_examples/atlas.html>`_: A walkthrough of how to use the AtlasDB vectorstore and visualizer wrapper.
+
 `DeepLake <./vectorstore_examples/deeplake.html>`_: A walkthrough of how to use the Deep Lake, data lake, wrapper.
 
 `FAISS <./vectorstore_examples/faiss.html>`_: A walkthrough of how to use the FAISS vectorstore wrapper.
@@ -50,6 +52,8 @@ In the below guides, we cover different types of vectorstores and how to use the
 
 `Weaviate <./vectorstore_examples/weaviate.html>`_: A walkthrough of how to use the Weaviate vectorstore wrapper.
 
+`PGVector <./vectorstore_examples/pgvector.html>`_: A walkthrough of how to use the PGVector (Postgres Vector DB) vectorstore wrapper.
+
 
 .. toctree::
    :maxdepth: 1

docs/modules/indexes/vectorstore_examples/atlas.ipynb

@@ -2,11 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "metadata": {
-    "pycharm": {
-     "name": "#%% md\n"
-    }
-   },
+   "metadata": {},
    "source": [
     "# AtlasDB\n",
     "\n",
@@ -15,10 +11,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {
     "pycharm": {
-     "name": "#%%\n"
+     "is_executing": true
     }
    },
    "outputs": [],
@@ -32,56 +28,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {
-    "scrolled": true
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Collecting en-core-web-sm==3.5.0\n",
-      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0-py3-none-any.whl (12.8 MB)\n",
-      "\u001B[2K     \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m12.8/12.8 MB\u001B[0m \u001B[31m90.8 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m00:01\u001B[0m00:01\u001B[0m\n",
-      "\u001B[?25hRequirement already satisfied: spacy<3.6.0,>=3.5.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from en-core-web-sm==3.5.0) (3.5.0)\n",
-      "Requirement already satisfied: packaging>=20.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (23.0)\n",
-      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.1.1)\n",
-      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.3.0)\n",
-      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2.4.5)\n",
-      "Requirement already satisfied: pathy>=0.10.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (0.10.1)\n",
-      "Requirement already satisfied: setuptools in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (67.4.0)\n",
-      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (4.64.1)\n",
-      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.0.4)\n",
-      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (6.3.0)\n",
-      "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (8.1.7)\n",
-      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2.0.7)\n",
-      "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (0.7.0)\n",
-      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2.28.2)\n",
-      "Requirement already satisfied: jinja2 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.1.2)\n",
-      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.10.5)\n",
-      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2.0.8)\n",
-      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.0.12)\n",
-      "Requirement already satisfied: numpy>=1.15.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.24.2)\n",
-      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.0.9)\n",
-      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.0.8)\n",
-      "Requirement already satisfied: typing-extensions>=4.2.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (4.5.0)\n",
-      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.0.1)\n",
-      "Requirement already satisfied: idna<4,>=2.5 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (3.4)\n",
-      "Requirement already satisfied: certifi>=2017.4.17 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2022.12.7)\n",
-      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (1.26.14)\n",
-      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from thinc<8.2.0,>=8.1.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (0.7.9)\n",
-      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from thinc<8.2.0,>=8.1.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (0.0.4)\n",
-      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from typer<0.8.0,>=0.3.0->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (8.1.3)\n",
-      "Requirement already satisfied: MarkupSafe>=2.0 in /home/ubuntu/langchain/.venv/lib/python3.9/site-packages (from jinja2->spacy<3.6.0,>=3.5.0->en-core-web-sm==3.5.0) (2.1.2)\n",
-      "\n",
-      "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m A new release of pip is available: \u001B[0m\u001B[31;49m23.0\u001B[0m\u001B[39;49m -> \u001B[0m\u001B[32;49m23.0.1\u001B[0m\n",
-      "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpip install --upgrade pip\u001B[0m\n",
-      "\u001B[38;5;2m✔ Download and installation successful\u001B[0m\n",
-      "You can now load the package via spacy.load('en_core_web_sm')\n"
-     ]
+    "scrolled": true,
+    "pycharm": {
+     "is_executing": true
     }
-   ],
+   },
+   "outputs": [],
    "source": [
     "!python -m spacy download en_core_web_sm"
    ]
@@ -113,51 +67,31 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "2023-02-24 16:13:49.696 | INFO     | nomic.project:_create_project:884 - Creating project `test_index_1677255228.136989` in organization `Atlas Demo`\n",
-      "2023-02-24 16:13:51.087 | INFO     | nomic.project:wait_for_project_lock:993 - test_index_1677255228.136989: Project lock is released.\n",
-      "2023-02-24 16:13:51.225 | INFO     | nomic.project:wait_for_project_lock:993 - test_index_1677255228.136989: Project lock is released.\n",
-      "2023-02-24 16:13:51.481 | INFO     | nomic.project:add_text:1351 - Uploading text to Atlas.\n",
-      "1it [00:00,  1.20it/s]\n",
-      "2023-02-24 16:13:52.318 | INFO     | nomic.project:add_text:1422 - Text upload succeeded.\n",
-      "2023-02-24 16:13:52.628 | INFO     | nomic.project:wait_for_project_lock:993 - test_index_1677255228.136989: Project lock is released.\n",
-      "2023-02-24 16:13:53.380 | INFO     | nomic.project:create_index:1192 - Created map `test_index_1677255228.136989_index` in project `test_index_1677255228.136989`: https://atlas.nomic.ai/map/ee2354a3-7f9a-4c6b-af43-b0cda09d7198/db996d77-8981-48a0-897a-ff2c22bbf541\n"
-     ]
+   "execution_count": null,
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
     }
-   ],
+   },
+   "outputs": [],
    "source": [
     "db = AtlasDB.from_texts(texts=texts,\n",
-    "                        name='test_index_'+str(time.time()),\n",
-    "                        description='test_index',\n",
+    "                        name='test_index_'+str(time.time()), # unique name for your vector store\n",
+    "                        description='test_index', #a description for your vector store\n",
     "                        api_key=ATLAS_TEST_API_KEY,\n",
     "                        index_kwargs={'build_topic_model': True})"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {
-    "scrolled": false
-   },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "2023-02-24 16:14:09.106 | INFO     | nomic.project:wait_for_project_lock:993 - test_index_1677255228.136989: Project lock is released.\n"
-     ]
-    }
-   ],
+   "execution_count": null,
+   "outputs": [],
    "source": [
-    "with db.project.wait_for_project_lock():\n",
-    "    time.sleep(1)"
-   ]
+    "db.project.wait_for_project_lock()"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
   },
   {
    "cell_type": "code",
@@ -263,4 +197,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 1
-}
+}

docs/modules/indexes/vectorstore_examples/pgvector.ipynb

@@ -0,0 +1,194 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# PGVector\n",
+    "\n",
+    "This notebook shows how to use functionality related to the Postgres vector database (PGVector)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Loading Environment Variables\n",
+    "from typing import List, Tuple\n",
+    "from dotenv import load_dotenv\n",
+    "load_dotenv()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.vectorstores.pgvector import PGVector\n",
+    "from langchain.document_loaders import TextLoader\n",
+    "from langchain.docstore.document import Document"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = TextLoader('../../state_of_the_union.txt')\n",
+    "documents = loader.load()\n",
+    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+    "docs = text_splitter.split_documents(documents)\n",
+    "\n",
+    "embeddings = OpenAIEmbeddings()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## PGVector needs the connection string to the database.\n",
+    "## We will load it from the environment variables.\n",
+    "import os\n",
+    "CONNECTION_STRING = PGVector.connection_string_from_db_params(\n",
+    "    driver=os.environ.get(\"PGVECTOR_DRIVER\", \"psycopg2\"),\n",
+    "    host=os.environ.get(\"PGVECTOR_HOST\", \"localhost\"),\n",
+    "    port=int(os.environ.get(\"PGVECTOR_PORT\", \"5432\")),\n",
+    "    database=os.environ.get(\"PGVECTOR_DATABASE\", \"postgres\"),\n",
+    "    user=os.environ.get(\"PGVECTOR_USER\", \"postgres\"),\n",
+    "    password=os.environ.get(\"PGVECTOR_PASSWORD\", \"postgres\"),\n",
+    ")\n",
+    "\n",
+    "\n",
+    "## Example\n",
+    "# postgresql+psycopg2://username:password@localhost:5432/database_name"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Similarity search with score"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Similarity Search with Euclidean Distance (Default)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# The PGVector Module will try to create a table with the name of the collection. So, make sure that the collection name is unique and the user has the \n",
+    "# permission to create a table.\n",
+    "\n",
+    "db = PGVector.from_documents(\n",
+    "    embedding=embeddings,\n",
+    "    documents=docs,\n",
+    "    collection_name=\"state_of_the_union\",\n",
+    "    connection_string=CONNECTION_STRING,\n",
+    ")\n",
+    "\n",
+    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
+    "docs_with_score: List[Tuple[Document, float]] = db.similarity_search_with_score(query)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------------------------------------------\n",
+      "Score:  0.6076628081132506\n",
+      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
+      "\n",
+      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
+      "\n",
+      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
+      "\n",
+      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
+      "--------------------------------------------------------------------------------\n",
+      "--------------------------------------------------------------------------------\n",
+      "Score:  0.6076628081132506\n",
+      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
+      "\n",
+      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
+      "\n",
+      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
+      "\n",
+      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
+      "--------------------------------------------------------------------------------\n",
+      "--------------------------------------------------------------------------------\n",
+      "Score:  0.6076804780049968\n",
+      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
+      "\n",
+      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
+      "\n",
+      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
+      "\n",
+      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
+      "--------------------------------------------------------------------------------\n",
+      "--------------------------------------------------------------------------------\n",
+      "Score:  0.6076804780049968\n",
+      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
+      "\n",
+      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
+      "\n",
+      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
+      "\n",
+      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
+      "--------------------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "for doc, score in docs_with_score:\n",
+    "    print(\"-\" * 80)\n",
+    "    print(\"Score: \", score)\n",
+    "    print(doc.page_content)\n",
+    "    print(\"-\" * 80)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.10"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/modules/indexes/vectorstore_examples/redis.ipynb

@@ -0,0 +1,204 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "source": [
+    "# Redis\n",
+    "\n",
+    "This notebook shows how to use functionality related to the Redis database."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.vectorstores.redis import Redis"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "loader = TextLoader('../../state_of_the_union.txt')\n",
+    "documents = loader.load()\n",
+    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+    "docs = text_splitter.split_documents(documents)\n",
+    "\n",
+    "embeddings = OpenAIEmbeddings()"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "outputs": [],
+   "source": [
+    "rds = Redis.from_documents(docs, embeddings,redis_url=\"redis://localhost:6379\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "outputs": [
+    {
+     "data": {
+      "text/plain": "'b564189668a343648996bd5a1d353d4e'"
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "rds.index_name"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n",
+      "\n",
+      "We cannot let this happen. \n",
+      "\n",
+      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
+      "\n",
+      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
+      "\n",
+      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
+      "\n",
+      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
+     ]
+    }
+   ],
+   "source": [
+    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
+    "results = rds.similarity_search(query)\n",
+    "print(results[0].page_content)"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['doc:333eadf75bd74be393acafa8bca48669']\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(rds.add_texts([\"Ankush went to Princeton\"]))"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Ankush went to Princeton\n"
+     ]
+    }
+   ],
+   "source": [
+    "query = \"Princeton\"\n",
+    "results = rds.similarity_search(query)\n",
+    "print(results[0].page_content)"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

docs/modules/memory/key_concepts.md

@@ -9,8 +9,8 @@ both at a short term but also at a long term level. The concept of "Memory" exis
 One of the simpler forms of memory occurs in chatbots, where they remember previous conversations.
 There are a few different ways to accomplish this:
 - Buffer: This is just passing in the past `N` interactions in as context. `N` can be chosen based on a fixed number, the length of the interactions, or other!
-- Summary: This involves summarizing previous conversations and passing that summary in, instead of the raw dialouge itself. Compared to `Buffer`, this compresses information: meaning it is more lossy, but also less likely to run into context length limits.
-- Combination: A combination of the above two approaches, where you compute a summary but also pass in some previous interfactions directly!
+- Summary: This involves summarizing previous conversations and passing that summary in, instead of the raw dialogue itself. Compared to `Buffer`, this compresses information: meaning it is more lossy, but also less likely to run into context length limits.
+- Combination: A combination of the above two approaches, where you compute a summary but also pass in some previous interactions directly!
 
 ## Entity Memory
 A more complex form of memory is remembering information about specific entities in the conversation.

docs/modules/prompts/examples/output_parsers.ipynb

@@ -0,0 +1,334 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "084ee2f0",
+   "metadata": {},
+   "source": [
+    "# Output Parsers\n",
+    "\n",
+    "Language models output text. But many times you may want to get more structured information than just text back. This is where output parsers come in.\n",
+    "\n",
+    "Output parsers are classes that help structure language model responses. There are two main methods an output parser must implement:\n",
+    "\n",
+    "- `get_format_instructions() -> str`: A method which returns a string containing instructions for how the output of a language model should be formatted.\n",
+    "- `parse(str) -> Any`: A method which takes in a string (assumed to be the response from a language model) and parses it into some structure.\n",
+    "\n",
+    "Below we go over some examples of output parsers."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "91871002",
+   "metadata": {},
+   "source": [
+    "## Structured Output Parser\n",
+    "\n",
+    "This output parser can be used when you want to return multiple fields."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "b492997a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.output_parsers import StructuredOutputParser, ResponseSchema"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "ffb7fc57",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.chat_models import ChatOpenAI"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "09473dce",
+   "metadata": {},
+   "source": [
+    "Here we define the response schema we want to receive."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "432ac44a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response_schemas = [\n",
+    "    ResponseSchema(name=\"answer\", description=\"answer to the user's question\"),\n",
+    "    ResponseSchema(name=\"source\", description=\"source used to answer the user's question, should be a website.\")\n",
+    "]\n",
+    "output_parser = StructuredOutputParser.from_response_schemas(response_schemas)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7b92ce96",
+   "metadata": {},
+   "source": [
+    "We now get a string that contains instructions for how the response should be formatted, and we then insert that into our prompt."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "593cfc25",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "format_instructions = output_parser.get_format_instructions()\n",
+    "prompt = PromptTemplate(\n",
+    "    template=\"answer the users question as best as possible.\\n{format_instructions}\\n{question}\",\n",
+    "    input_variables=[\"question\"],\n",
+    "    partial_variables={\"format_instructions\": format_instructions}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0943e783",
+   "metadata": {},
+   "source": [
+    "We can now use this to format a prompt to send to the language model, and then parse the returned result."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "106f1ba6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = OpenAI(temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "86d9d24f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "_input = prompt.format_prompt(question=\"what's the capital of france\")\n",
+    "output = model(_input.to_string())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "956bdc99",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'answer': 'Paris', 'source': 'https://en.wikipedia.org/wiki/Paris'}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "output_parser.parse(output)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "da639285",
+   "metadata": {},
+   "source": [
+    "And here's an example of using this in a chat model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "8f483d7d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chat_model = ChatOpenAI(temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "f761cbf1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt = ChatPromptTemplate(\n",
+    "    messages=[\n",
+    "        HumanMessagePromptTemplate.from_template(\"answer the users question as best as possible.\\n{format_instructions}\\n{question}\")  \n",
+    "    ],\n",
+    "    input_variables=[\"question\"],\n",
+    "    partial_variables={\"format_instructions\": format_instructions}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "edd73ae3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "_input = prompt.format_prompt(question=\"what's the capital of france\")\n",
+    "output = chat_model(_input.to_messages())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "a3c8b91e",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'answer': 'Paris', 'source': 'https://en.wikipedia.org/wiki/Paris'}"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "output_parser.parse(output.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9936fa27",
+   "metadata": {},
+   "source": [
+    "## CommaSeparatedListOutputParser\n",
+    "\n",
+    "This output parser can be used to get a list of items as output."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "872246d7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.output_parsers import CommaSeparatedListOutputParser"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "c3f9aee6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "output_parser = CommaSeparatedListOutputParser()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "e77871b7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "format_instructions = output_parser.get_format_instructions()\n",
+    "prompt = PromptTemplate(\n",
+    "    template=\"List five {subject}.\\n{format_instructions}\",\n",
+    "    input_variables=[\"subject\"],\n",
+    "    partial_variables={\"format_instructions\": format_instructions}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "a71cb5d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = OpenAI(temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "783d7d98",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "_input = prompt.format(subject=\"ice cream flavors\")\n",
+    "output = model(_input)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "fcb81344",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['Vanilla',\n",
+       " 'Chocolate',\n",
+       " 'Strawberry',\n",
+       " 'Mint Chocolate Chip',\n",
+       " 'Cookies and Cream']"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "output_parser.parse(output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cba6d8e3",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/prompts/how_to_guides.rst

@@ -32,3 +32,4 @@ The user guide here shows more advanced workflows and how to use the library in
    ./examples/prompt_serialization.ipynb
    ./examples/few_shot_examples_data.ipynb
    ./examples/example_selectors.ipynb
+   ./examples/output_parsers.ipynb

docs/requirements.txt

@@ -9,3 +9,4 @@ sphinx-typlog-theme==0.8.0
 sphinx-panels
 toml
 myst_nb
+sphinx_copybutton

docs/tracing/agent_with_tracing.ipynb

@@ -35,12 +35,28 @@
     "\n",
     "import langchain\n",
     "from langchain.agents import Tool, initialize_agent, load_tools\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.llms import OpenAI"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 2,
+   "id": "1b62cd48",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Agent run with tracing. Ensure that OPENAI_API_KEY is set appropriately to run this example.\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "tools = load_tools([\"llm-math\"], llm=llm)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
    "id": "bfa16b79-aa4b-4d41-a067-70d1f593f667",
    "metadata": {
     "tags": []
@@ -70,16 +86,12 @@
        "'1.0891804557407723'"
       ]
      },
-     "execution_count": 2,
+     "execution_count": 3,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "# Agent run with tracing. Ensure that OPENAI_API_KEY is set appropriately to run this example.\n",
-    "\n",
-    "llm = OpenAI(temperature=0)\n",
-    "tools = load_tools([\"llm-math\"], llm=llm)\n",
     "agent = initialize_agent(\n",
     "    tools, llm, agent=\"zero-shot-react-description\", verbose=True\n",
     ")\n",
@@ -87,10 +99,94 @@
     "agent.run(\"What is 2 raised to .123243 power?\")"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "4829eb1d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mQuestion: What is 2 raised to .123243 power?\n",
+      "Thought: I need a calculator to solve this problem.\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"calculator\",\n",
+      "  \"action_input\": \"2^0.123243\"\n",
+      "}\n",
+      "```\n",
+      "\u001b[0m\n",
+      "Observation: calculator is not a valid tool, try another one.\n",
+      "\u001b[32;1m\u001b[1;3mI made a mistake, I need to use the correct tool for this question.\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"calculator\",\n",
+      "  \"action_input\": \"2^0.123243\"\n",
+      "}\n",
+      "```\n",
+      "\n",
+      "\u001b[0m\n",
+      "Observation: calculator is not a valid tool, try another one.\n",
+      "\u001b[32;1m\u001b[1;3mI made a mistake, the tool name is actually \"calc\" instead of \"calculator\".\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"calc\",\n",
+      "  \"action_input\": \"2^0.123243\"\n",
+      "}\n",
+      "```\n",
+      "\n",
+      "\u001b[0m\n",
+      "Observation: calc is not a valid tool, try another one.\n",
+      "\u001b[32;1m\u001b[1;3mI made another mistake, the tool name is actually \"Calculator\" instead of \"calc\".\n",
+      "Action:\n",
+      "```\n",
+      "{\n",
+      "  \"action\": \"Calculator\",\n",
+      "  \"action_input\": \"2^0.123243\"\n",
+      "}\n",
+      "```\n",
+      "\n",
+      "\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 1.0891804557407723\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mThe final answer is 1.0891804557407723.\n",
+      "Final Answer: 1.0891804557407723\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'1.0891804557407723'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Agent run with tracing using a chat model\n",
+    "agent = initialize_agent(\n",
+    "    tools, ChatOpenAI(temperature=0), agent=\"chat-zero-shot-react-description\", verbose=True\n",
+    ")\n",
+    "\n",
+    "agent.run(\"What is 2 raised to .123243 power?\")"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "25addd7f",
+   "id": "76abfd82",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -112,7 +208,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/use_cases/evaluation.rst

@@ -1,9 +1,85 @@
 Evaluation
 ==============
 
-Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.
+This section of documentation covers how we approach and think about evaluation in LangChain.
+Both evaluation of internal chains/agents, but also how we would recommend people building on top of LangChain approach evaluation.
 
-The examples here all highlight how to use language models to assist in evaluation of themselves.
+The Problem
+-----------
+
+It can be really hard to evaluate LangChain chains and agents.
+There are two main reasons for this:
+
+**# 1: Lack of data**
+
+You generally don't have a ton of data to evaluate your chains/agents over before starting a project.
+This is usually because Large Language Models (the core of most chains/agents) are terrific few-shot and zero shot learners,
+meaning you are almost always able to get started on a particular task (text-to-SQL, question answering, etc) without
+a large dataset of examples.
+This is in stark contrast to traditional machine learning where you had to first collect a bunch of datapoints
+before even getting started using a model.
+
+**# 2: Lack of metrics**
+
+Most chains/agents are performing tasks for which there are not very good metrics to evaluate performance.
+For example, one of the most common use cases is generating text of some form.
+Evaluating generated text is much more complicated than evaluating a classification prediction, or a numeric prediction.
+
+The Solution
+------------
+
+LangChain attempts to tackle both of those issues.
+What we have so far are initial passes at solutions - we do not think we have a perfect solution.
+So we very much welcome feedback, contributions, integrations, and thoughts on this.
+
+Here is what we have for each problem so far:
+
+**# 1: Lack of data**
+
+We have started `LangChainDatasets <https://huggingface.co/LangChainDatasets>`_ a Community space on Hugging Face.
+We intend this to be a collection of open source datasets for evaluating common chains and agents.
+We have contributed five datasets of our own to start, but we highly intend this to be a community effort.
+In order to contribute a dataset, you simply need to join the community and then you will be able to upload datasets.
+
+We're also aiming to make it as easy as possible for people to create their own datasets.
+As a first pass at this, we've added a QAGenerationChain, which given a document comes up
+with question-answer pairs that can be used to evaluate question-answering tasks over that document down the line.
+See `this notebook <./evaluation/qa_generation.html>`_ for an example of how to use this chain.
+
+**# 2: Lack of metrics**
+
+We have two solutions to the lack of metrics.
+
+The first solution is to use no metrics, and rather just rely on looking at results by eye to get a sense for how the chain/agent is performing.
+To assist in this, we have developed (and will continue to develop) `tracing <../tracing.html>`_, a UI-based visualizer of your chain and agent runs.
+
+The second solution we recommend is to use Language Models themselves to evaluate outputs.
+For this we have a few different chains and prompts aimed at tackling this issue.
+
+The Examples
+------------
+
+We have created a bunch of examples combining the above two solutions to show how we internally evaluate chains and agents when we are developing.
+In addition to the examples we've curated, we also highly welcome contributions here.
+To facilitate that, we've included a `template notebook <./evaluation/benchmarking_template.html>`_ for community members to use to build their own examples.
+
+The existing examples we have are:
+
+`Question Answering (State of Union) <./evaluation/qa_benchmarking_sota.html>`_: An notebook showing evaluation of a question-answering task over a State-of-the-Union address.
+
+`Question Answering (Paul Graham Essay) <./evaluation/qa_benchmarking_pg.html>`_: An notebook showing evaluation of a question-answering task over a Paul Graham essay.
+
+`SQL Question Answering (Chinook) <./evaluation/sql_qa_benchmarking_chinook.html>`_: An notebook showing evaluation of a question-answering task over a SQL database (the Chinook database).
+
+`Agent Vectorstore <./evaluation/agent_vectordb_sota_pg.html>`_: An notebook showing evaluation of an agent doing question answering while routing between two different vector databases.
+
+`Agent Search + Calculator <./evaluation/agent_benchmarking.html>`_: An notebook showing evaluation of an agent doing question answering using a Search engine and a Calculator as tools.
+
+
+Other Examples
+------------
+
+In addition, we also have some more generic resources for evaluation.
 
 `Question Answering <./evaluation/question_answering.html>`_: An overview of LLMs aimed at evaluating question answering systems in general.
 

docs/use_cases/evaluation/agent_benchmarking.ipynb

@@ -0,0 +1,343 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Agent Benchmarking: Search + Calculator\n",
+    "\n",
+    "Here we go over how to benchmark performance of an agent on tasks where it has access to a calculator and a search tool.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "46bf9205",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-search-calculator-8a025c0ce5fb99d2/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "3a275586643f4ccfba1a8d54be28c351",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "dataset = load_dataset(\"agent-search-calculator\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to load an agent capable of answering these questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.llms import OpenAI\n",
+    "from langchain.chains import LLMMathChain\n",
+    "from langchain.agents import initialize_agent, Tool, load_tools\n",
+    "\n",
+    "tools = load_tools(['serpapi', 'llm-math'], llm=OpenAI(temperature=0))\n",
+    "agent = initialize_agent(tools, OpenAI(temperature=0), agent=\"zero-shot-react-description\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "68504a8f",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "cbcafc92",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'38,630,316 people live in Canada as of 2023.'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(dataset[0]['question'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer')).\n"
+     ]
+    }
+   ],
+   "source": [
+    "predictions = []\n",
+    "predicted_dataset = []\n",
+    "error_dataset = []\n",
+    "for data in dataset:\n",
+    "    new_data = {\"input\": data[\"question\"], \"answer\": data[\"answer\"]}\n",
+    "    try:\n",
+    "        predictions.append(agent(new_data))\n",
+    "        predicted_dataset.append(new_data)\n",
+    "    except Exception:\n",
+    "        error_dataset.append(new_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': 'How many people live in canada as of 2023?',\n",
+       " 'answer': 'approximately 38,625,801',\n",
+       " 'output': '38,630,316 people live in Canada as of 2023.',\n",
+       " 'intermediate_steps': [(AgentAction(tool='Search', tool_input='Population of Canada 2023', log=' I need to find population data\\nAction: Search\\nAction Input: Population of Canada 2023'),\n",
+       "   '38,630,316')]}"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(dataset, predictions, question_key=\"question\", prediction_key=\"output\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction['grade'] = graded_outputs[i]['text']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 4, ' INCORRECT': 6})"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "Counter([pred['grade'] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred['grade'] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': \"who is dua lipa's boyfriend? what is his age raised to the .43 power?\",\n",
+       " 'answer': 'her boyfriend is Romain Gravas. his age raised to the .43 power is approximately 4.9373857399466665',\n",
+       " 'output': \"Isaac Carew, Dua Lipa's boyfriend, is 36 years old and his age raised to the .43 power is 4.6688516567750975.\",\n",
+       " 'intermediate_steps': [(AgentAction(tool='Search', tool_input=\"Dua Lipa's boyfriend\", log=' I need to find out who Dua Lipa\\'s boyfriend is and then calculate his age raised to the .43 power\\nAction: Search\\nAction Input: \"Dua Lipa\\'s boyfriend\"'),\n",
+       "   'Dua and Isaac, a model and a chef, dated on and off from 2013 to 2019. The two first split in early 2017, which is when Dua went on to date LANY ...'),\n",
+       "  (AgentAction(tool='Search', tool_input='Isaac Carew age', log=' I need to find out Isaac\\'s age\\nAction: Search\\nAction Input: \"Isaac Carew age\"'),\n",
+       "   '36 years'),\n",
+       "  (AgentAction(tool='Calculator', tool_input='36^.43', log=' I need to calculate 36 raised to the .43 power\\nAction: Calculator\\nAction Input: 36^.43'),\n",
+       "   'Answer: 4.6688516567750975\\n')],\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/use_cases/evaluation/agent_vectordb_sota_pg.ipynb

@@ -0,0 +1,516 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Agent VectorDB Question Answering Benchmarking\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task using an agent to route between multiple vectordatabases.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "id": "7b57a50f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-vectordb-qa-sota-pg-d3ae24016b514f92/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a7abbc20615d4c58b75a055a790d7212",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "dataset = load_dataset(\"agent-vectordb-qa-sota-pg\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "61375342",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'steps': [{'tool': 'State of Union QA System', 'tool_input': None},\n",
+       "  {'tool': None, 'tool_input': 'What is the purpose of the NATO Alliance?'}]}"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "02500304",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of YC?',\n",
+       " 'answer': 'The purpose of YC is to cause startups to be founded that would not otherwise have existed.',\n",
+       " 'steps': [{'tool': 'Paul Graham QA System', 'tool_input': None},\n",
+       "  {'tool': None, 'tool_input': 'What is the purpose of YC?'}]}"
+      ]
+     },
+     "execution_count": 22,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[-1]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to create some pipelines for doing question answering. Step one in that is creating indexes over the data in question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "7f0de2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes import VectorstoreIndexCreator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "ef84ff99",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore_sota = VectorstoreIndexCreator(vectorstore_kwargs={\"collection_name\":\"sota\"}).from_loaders([loader]).vectorstore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import VectorDBQA\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "573719a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_sota = VectorDBQA.from_chain_type(llm=OpenAI(temperature=0), chain_type=\"stuff\", vectorstore=vectorstore_sota, input_key=\"question\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e48b03d8",
+   "metadata": {},
+   "source": [
+    "Now we do the same for the Paul Graham data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "c2dbb014",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = TextLoader(\"../../modules/paul_graham_essay.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "98d16f08",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore_pg = VectorstoreIndexCreator(vectorstore_kwargs={\"collection_name\":\"paul_graham\"}).from_loaders([loader]).vectorstore"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "ec0aab02",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_pg = VectorDBQA.from_chain_type(llm=OpenAI(temperature=0), chain_type=\"stuff\", vectorstore=vectorstore_pg, input_key=\"question\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "76b5f8fb",
+   "metadata": {},
+   "source": [
+    "We can now set up an agent to route between them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "ade1aafa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import initialize_agent, Tool\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name = \"State of Union QA System\",\n",
+    "        func=chain_sota.run,\n",
+    "        description=\"useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question.\"\n",
+    "    ),\n",
+    "    Tool(\n",
+    "        name = \"Paul Graham System\",\n",
+    "        func=chain_pg.run,\n",
+    "        description=\"useful for when you need to answer questions about Paul Graham. Input should be a fully formed question.\"\n",
+    "    ),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "id": "104853f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = initialize_agent(tools, OpenAI(temperature=0), agent=\"zero-shot-react-description\", max_iterations=3)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f036641",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 48,
+   "id": "4664e79f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The purpose of the NATO Alliance is to promote peace and security in the North Atlantic region by providing a collective defense against potential threats.'"
+      ]
+     },
+     "execution_count": 48,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(dataset[0]['question'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = []\n",
+    "predicted_dataset = []\n",
+    "error_dataset = []\n",
+    "for data in dataset:\n",
+    "    new_data = {\"input\": data[\"question\"], \"answer\": data[\"answer\"]}\n",
+    "    try:\n",
+    "        predictions.append(agent(new_data))\n",
+    "        predicted_dataset.append(new_data)\n",
+    "    except Exception:\n",
+    "        error_dataset.append(new_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'output': 'The purpose of the NATO Alliance is to promote peace and security in the North Atlantic region by providing a collective defense against potential threats.'}"
+      ]
+     },
+     "execution_count": 36,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(predicted_dataset, predictions, question_key=\"input\", prediction_key=\"output\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction['grade'] = graded_outputs[i]['text']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 42,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 19, ' INCORRECT': 14})"
+      ]
+     },
+     "execution_count": 42,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "Counter([pred['grade'] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred['grade'] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': 'What is the purpose of the Bipartisan Innovation Act mentioned in the text?',\n",
+       " 'answer': 'The Bipartisan Innovation Act will make record investments in emerging technologies and American manufacturing to level the playing field with China and other competitors.',\n",
+       " 'output': 'The purpose of the Bipartisan Innovation Act is to promote innovation and entrepreneurship in the United States by providing tax incentives and other support for startups and small businesses.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 46,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/use_cases/evaluation/benchmarking_template.ipynb

@@ -0,0 +1,160 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a175c650",
+   "metadata": {},
+   "source": [
+    "# Benchmarking Template\n",
+    "\n",
+    "This is an example notebook that can be used to create a benchmarking notebook for a task of your choice. Evaluation is really hard, and so we greatly welcome any contributions that can make it easier for people to experiment"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "9fe4d1b4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f66405e",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "79402a8f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This notebook should so how to load the dataset from LangChainDatasets on Hugging Face\n",
+    "\n",
+    "# Please upload your dataset to https://huggingface.co/LangChainDatasets\n",
+    "\n",
+    "# The value passed into `load_dataset` should NOT have the `LangChainDatasets/` prefix\n",
+    "from langchain.evaluation.loading import load_dataset\n",
+    "dataset = load_dataset(\"TODO\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "\n",
+    "This next section should have an example of setting up a chain that can be run on this dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a2661ce0",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c0062e7",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "d28c5e7d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example of running the chain on a single datapoint (`dataset[0]`) goes here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example of running the chain on many predictions goes here\n",
+    "\n",
+    "# Sometimes its as simple as `chain.apply(dataset)`\n",
+    "\n",
+    "# Othertimes you may want to write a for loop to catch errors"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "\n",
+    "Any guide to evaluating performance in a more systematic manner goes here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/use_cases/evaluation/data_augmented_question_answering.ipynb

@@ -28,7 +28,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "id": "4fdc211d",
    "metadata": {},
    "outputs": [
@@ -67,7 +67,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 3,
    "id": "3459b001",
    "metadata": {},
    "outputs": [],
@@ -87,7 +87,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 4,
    "id": "b9c3fa75",
    "metadata": {},
    "outputs": [],
@@ -99,7 +99,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 5,
    "id": "c24543a9",
    "metadata": {},
    "outputs": [],
@@ -116,16 +116,16 @@
     {
      "data": {
       "text/plain": [
-       "[{'query': 'What did Vladimir Putin miscalculate when he sought to shake the foundations of the free world? ',\n",
-       "  'answer': 'He miscalculated that the world would roll over and that he could roll into Ukraine without facing resistance.'},\n",
-       " {'query': 'What is the purpose of NATO?',\n",
-       "  'answer': 'The purpose of NATO is to secure peace and stability in Europe after World War 2.'},\n",
-       " {'query': \"What did the author do to prepare for Putin's attack on Ukraine?\",\n",
-       "  'answer': \"The author spent months building a coalition of freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin, shared with the world in advance what they knew Putin was planning, and countered Russia's lies with truth.\"},\n",
-       " {'query': 'What are the US and its allies doing to isolate Russia from the world?',\n",
-       "  'answer': \"Enforcing powerful economic sanctions, cutting off Russia's largest banks from the international financial system, preventing Russia's central bank from defending the Russian Ruble, choking off Russia's access to technology, and joining with European allies to find and seize assets of Russian oligarchs.\"},\n",
-       " {'query': 'How much direct assistance is the U.S. providing to Ukraine?',\n",
-       "  'answer': 'The U.S. is providing more than $1 Billion in direct assistance to Ukraine.'}]"
+       "[{'query': 'According to the document, what did Vladimir Putin miscalculate?',\n",
+       "  'answer': 'He miscalculated that he could roll into Ukraine and the world would roll over.'},\n",
+       " {'query': 'Who is the Ukrainian Ambassador to the United States?',\n",
+       "  'answer': 'The Ukrainian Ambassador to the United States is here tonight.'},\n",
+       " {'query': 'How many countries were part of the coalition formed to confront Putin?',\n",
+       "  'answer': '27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.'},\n",
+       " {'query': 'What action is the U.S. Department of Justice taking to target Russian oligarchs?',\n",
+       "  'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.'},\n",
+       " {'query': 'How much direct assistance is the United States providing to Ukraine?',\n",
+       "  'answer': 'The United States is providing more than $1 Billion in direct assistance to Ukraine.'}]"
       ]
      },
      "execution_count": 6,
@@ -211,44 +211,43 @@
       "Example 0:\n",
       "Question: What did the president say about Ketanji Brown Jackson\n",
       "Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
-      "Predicted Answer:  The president said that Ketanji Brown Jackson is one of the nation's top legal minds and that she will continue Justice Breyer's legacy of excellence.\n",
+      "Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.\n",
       "Predicted Grade:  CORRECT\n",
       "\n",
       "Example 1:\n",
       "Question: What did the president say about Michael Jackson\n",
       "Real Answer: Nothing\n",
-      "Predicted Answer: \n",
-      "The president did not mention Michael Jackson in this context.\n",
+      "Predicted Answer:  The president did not mention Michael Jackson in this speech.\n",
       "Predicted Grade:  CORRECT\n",
       "\n",
       "Example 2:\n",
-      "Question: What did Vladimir Putin miscalculate when he sought to shake the foundations of the free world? \n",
-      "Real Answer: He miscalculated that the world would roll over and that he could roll into Ukraine without facing resistance.\n",
-      "Predicted Answer:  Putin miscalculated that the West and NATO wouldn't respond to his attack on Ukraine and that he could divide the US and its allies.\n",
+      "Question: According to the document, what did Vladimir Putin miscalculate?\n",
+      "Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.\n",
+      "Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.\n",
       "Predicted Grade:  CORRECT\n",
       "\n",
       "Example 3:\n",
-      "Question: What is the purpose of NATO?\n",
-      "Real Answer: The purpose of NATO is to secure peace and stability in Europe after World War 2.\n",
-      "Predicted Answer:  The purpose of NATO is to secure peace and stability in Europe after World War 2.\n",
-      "Predicted Grade:  CORRECT\n",
+      "Question: Who is the Ukrainian Ambassador to the United States?\n",
+      "Real Answer: The Ukrainian Ambassador to the United States is here tonight.\n",
+      "Predicted Answer:  I don't know.\n",
+      "Predicted Grade:  INCORRECT\n",
       "\n",
       "Example 4:\n",
-      "Question: What did the author do to prepare for Putin's attack on Ukraine?\n",
-      "Real Answer: The author spent months building a coalition of freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin, shared with the world in advance what they knew Putin was planning, and countered Russia's lies with truth.\n",
-      "Predicted Answer:  The author prepared extensively and carefully. They spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin, and they spent countless hours unifying their European allies. They also shared with the world in advance what they knew Putin was planning and precisely how he would try to falsely justify his aggression. They countered Russia’s lies with truth.\n",
-      "Predicted Grade:  CORRECT\n",
+      "Question: How many countries were part of the coalition formed to confront Putin?\n",
+      "Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Grade:  INCORRECT\n",
       "\n",
       "Example 5:\n",
-      "Question: What are the US and its allies doing to isolate Russia from the world?\n",
-      "Real Answer: Enforcing powerful economic sanctions, cutting off Russia's largest banks from the international financial system, preventing Russia's central bank from defending the Russian Ruble, choking off Russia's access to technology, and joining with European allies to find and seize assets of Russian oligarchs.\n",
-      "Predicted Answer:  The US and its allies are enforcing economic sanctions on Russia, cutting off its largest banks from the international financial system, preventing its central bank from defending the Russian Ruble, choking off Russia's access to technology, closing American airspace to all Russian flights, and providing support to Ukraine.\n",
-      "Predicted Grade:  CORRECT\n",
+      "Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?\n",
+      "Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Grade:  INCORRECT\n",
       "\n",
       "Example 6:\n",
-      "Question: How much direct assistance is the U.S. providing to Ukraine?\n",
-      "Real Answer: The U.S. is providing more than $1 Billion in direct assistance to Ukraine.\n",
-      "Predicted Answer:  The U.S. is providing more than $1 Billion in direct assistance to Ukraine.\n",
+      "Question: How much direct assistance is the United States providing to Ukraine?\n",
+      "Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.\n",
+      "Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.\n",
       "Predicted Grade:  CORRECT\n",
       "\n"
      ]
@@ -264,13 +263,159 @@
     "    print()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "50a9e845",
+   "metadata": {},
+   "source": [
+    "## Evaluate with Other Metrics\n",
+    "\n",
+    "In addition to predicting whether the answer is correct or incorrect using a language model, we can also use other metrics to get a more nuanced view on the quality of the answers. To do so, we can use the [Critique](https://docs.inspiredco.ai/critique/) library, which allows for simple calculation of various metrics over generated text.\n",
+    "\n",
+    "First you can get an API key from the [Inspired Cognition Dashboard](https://dashboard.inspiredco.ai) and do some setup:\n",
+    "\n",
+    "```bash\n",
+    "export INSPIREDCO_API_KEY=\"...\"\n",
+    "pip install inspiredco\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 13,
    "id": "bd0b01dc",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "import inspiredco.critique\n",
+    "import os\n",
+    "critique = inspiredco.critique.Critique(api_key=os.environ['INSPIREDCO_API_KEY'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4f52629e",
+   "metadata": {},
+   "source": [
+    "Then run the following code to set up the configuration and calculate the [ROUGE](https://docs.inspiredco.ai/critique/metric_rouge.html), [chrf](https://docs.inspiredco.ai/critique/metric_chrf.html), [BERTScore](https://docs.inspiredco.ai/critique/metric_bert_score.html), and [UniEval](https://docs.inspiredco.ai/critique/metric_uni_eval.html) (you can choose [other metrics](https://docs.inspiredco.ai/critique/metrics.html) too):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "84a0ba21",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = {\n",
+    "    \"rouge\": {\n",
+    "        \"metric\": \"rouge\",\n",
+    "        \"config\": {\"variety\": \"rouge_l\"},\n",
+    "    },\n",
+    "    \"chrf\": {\n",
+    "        \"metric\": \"chrf\",\n",
+    "        \"config\": {},\n",
+    "    },\n",
+    "    \"bert_score\": {\n",
+    "        \"metric\": \"bert_score\",\n",
+    "        \"config\": {\"model\": \"bert-base-uncased\"},\n",
+    "    },\n",
+    "    \"uni_eval\": {\n",
+    "        \"metric\": \"uni_eval\",\n",
+    "        \"config\": {\"task\": \"summarization\", \"evaluation_aspect\": \"relevance\"},\n",
+    "    },\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "3b9a4056",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "critique_data = [\n",
+    "    {\"target\": pred['result'], \"references\": [pred['answer']]} for pred in predictions\n",
+    "]\n",
+    "eval_results = {\n",
+    "    k: critique.evaluate(dataset=critique_data, metric=v[\"metric\"], config=v[\"config\"])\n",
+    "    for k, v in metrics.items()\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f0ae799",
+   "metadata": {},
+   "source": [
+    "Finally, we can print out the results. We can see that overall the scores are higher when the output is semantically correct, and also when the output closely matches with the gold-standard answer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "b51edcf4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Example 0:\n",
+      "Question: What did the president say about Ketanji Brown Jackson\n",
+      "Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
+      "Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.\n",
+      "Predicted Scores: rouge=0.0941, chrf=0.2001, bert_score=0.5219, uni_eval=0.9043\n",
+      "\n",
+      "Example 1:\n",
+      "Question: What did the president say about Michael Jackson\n",
+      "Real Answer: Nothing\n",
+      "Predicted Answer:  The president did not mention Michael Jackson in this speech.\n",
+      "Predicted Scores: rouge=0.0000, chrf=0.1087, bert_score=0.3486, uni_eval=0.7802\n",
+      "\n",
+      "Example 2:\n",
+      "Question: According to the document, what did Vladimir Putin miscalculate?\n",
+      "Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.\n",
+      "Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.\n",
+      "Predicted Scores: rouge=0.5185, chrf=0.6955, bert_score=0.8421, uni_eval=0.9578\n",
+      "\n",
+      "Example 3:\n",
+      "Question: Who is the Ukrainian Ambassador to the United States?\n",
+      "Real Answer: The Ukrainian Ambassador to the United States is here tonight.\n",
+      "Predicted Answer:  I don't know.\n",
+      "Predicted Scores: rouge=0.0000, chrf=0.0375, bert_score=0.3159, uni_eval=0.7493\n",
+      "\n",
+      "Example 4:\n",
+      "Question: How many countries were part of the coalition formed to confront Putin?\n",
+      "Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Scores: rouge=0.7419, chrf=0.8602, bert_score=0.8388, uni_eval=0.0669\n",
+      "\n",
+      "Example 5:\n",
+      "Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?\n",
+      "Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Scores: rouge=0.9412, chrf=0.8687, bert_score=0.9607, uni_eval=0.9718\n",
+      "\n",
+      "Example 6:\n",
+      "Question: How much direct assistance is the United States providing to Ukraine?\n",
+      "Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.\n",
+      "Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.\n",
+      "Predicted Scores: rouge=1.0000, chrf=0.9483, bert_score=1.0000, uni_eval=0.9734\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i, eg in enumerate(examples):\n",
+    "    score_string = \", \".join([f\"{k}={v['examples'][i]['value']:.4f}\" for k, v in eval_results.items()])\n",
+    "    print(f\"Example {i}:\")\n",
+    "    print(\"Question: \" + predictions[i]['query'])\n",
+    "    print(\"Real Answer: \" + predictions[i]['answer'])\n",
+    "    print(\"Predicted Answer: \" + predictions[i]['result'])\n",
+    "    print(\"Predicted Scores: \" + score_string)\n",
+    "    print()"
+   ]
   }
  ],
  "metadata": {
@@ -289,7 +434,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.9"
   }
  },
  "nbformat": 4,

docs/use_cases/evaluation/qa_benchmarking_pg.ipynb

@@ -0,0 +1,374 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Question Answering Benchmarking: Paul Graham Essay\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task over a Paul Graham essay.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "3bd13ab7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-paul-graham-76e8f711e038d742/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "63f434a42cba4739919333c75324acc9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "dataset = load_dataset(\"question-answering-paul-graham\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "loader = TextLoader(\"../../modules/paul_graham_essay.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "7f0de2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes import VectorstoreIndexCreator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "ef84ff99",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import VectorDBQA\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "573719a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = VectorDBQA.from_chain_type(llm=OpenAI(), chain_type=\"stuff\", vectorstore=vectorstore, input_key=\"question\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53b5aa23",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "3f81d951",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What were the two main things the author worked on before college?',\n",
+       " 'answer': 'The two main things the author worked on before college were writing and programming.',\n",
+       " 'result': ' Writing and programming.'}"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain(dataset[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = chain.apply(dataset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What were the two main things the author worked on before college?',\n",
+       " 'answer': 'The two main things the author worked on before college were writing and programming.',\n",
+       " 'result': ' Writing and programming.'}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(dataset, predictions, question_key=\"question\", prediction_key=\"result\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction['grade'] = graded_outputs[i]['text']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 12, ' INCORRECT': 10})"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "Counter([pred['grade'] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred['grade'] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What did the author write their dissertation on?',\n",
+       " 'answer': 'The author wrote their dissertation on applications of continuations.',\n",
+       " 'result': ' The author does not mention what their dissertation was on, so it is not known.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/use_cases/evaluation/qa_benchmarking_sota.ipynb

@@ -0,0 +1,451 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Question Answering Benchmarking: State of the Union Address\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task over a state of the union address.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "f127fb04",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "5d66c27b9b4744989843142f08f5c1b4",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading readme:   0%|          | 0.00/21.0 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Downloading and preparing dataset json/LangChainDatasets--question-answering-state-of-the-union to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-state-of-the-union-a7e5a3b2db4f440d/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "9e21e2ab96a0491ea5e252720d7dfa26",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "c883830e068c42d39da8406ab38574c4",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading data:   0%|          | 0.00/2.90k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "3b085715e52e49948d2a59d27e004eba",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Generating train split: 0 examples [00:00, ? examples/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Dataset json downloaded and prepared to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-state-of-the-union-a7e5a3b2db4f440d/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51. Subsequent calls will reuse this data.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ee900d35e27d4843b42b31811b43212b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "dataset = load_dataset(\"question-answering-state-of-the-union\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "7f0de2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes import VectorstoreIndexCreator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "ef84ff99",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import VectorDBQA\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "573719a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = VectorDBQA.from_chain_type(llm=OpenAI(), chain_type=\"stuff\", vectorstore=vectorstore, input_key=\"question\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "37d669e9",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "3089e409",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'result': ' The NATO Alliance was created to secure peace and stability in Europe after World War 2.'}"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain(dataset[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = chain.apply(dataset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'result': ' The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'}"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(dataset, predictions, question_key=\"question\", prediction_key=\"result\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction['grade'] = graded_outputs[i]['text']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 7, ' INCORRECT': 4})"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "Counter([pred['grade'] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred['grade'] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',\n",
+       " 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.',\n",
+       " 'result': ' The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and is naming a chief prosecutor for pandemic fraud.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/use_cases/evaluation/qa_generation.ipynb

@@ -0,0 +1,117 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "ee2a3a21",
+   "metadata": {},
+   "source": [
+    "# QA Generation\n",
+    "This notebook shows how to use the `QAGenerationChain` to come up with question-answer pairs over a specific document.\n",
+    "This is important because often times you may not have data to evaluate your question-answer system over, so this is a cheap and lightweight way to generate it!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "33d3f0b4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2029a29c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "87edb84c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "doc = loader.load()[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "04125b6d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains import QAGenerationChain\n",
+    "chain = QAGenerationChain.from_llm(ChatOpenAI(temperature = 0))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "4f1593e4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "qa = chain.run(doc.page_content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "ee831f92",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',\n",
+       " 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.'}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "qa[1]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7028754e",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/use_cases/evaluation/question_answering.ipynb

@@ -191,7 +191,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "782ae8c8",
    "metadata": {},
@@ -316,7 +315,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -330,7 +329,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7 (default, Sep 16 2021, 08:50:36) \n[Clang 10.0.0 ]"
+   "version": "3.9.1"
   },
   "vscode": {
    "interpreter": {

docs/use_cases/evaluation/sql_qa_benchmarking_chinook.ipynb

@@ -0,0 +1,423 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# SQL Question Answering Benchmarking: Chinook\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task over a SQL database.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "44874486",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f66405e",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "0df1393f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b220d07ee5d14909bc842b4545cdc0de",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading readme:   0%|          | 0.00/21.0 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Downloading and preparing dataset json/LangChainDatasets--sql-qa-chinook to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "e89e3c8ef76f49889c4b39c624828c71",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a8421df6c26045e8978c7086cb418222",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading data:   0%|          | 0.00/1.44k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d1fb6becc3324a85bf039a53caf30924",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Generating train split: 0 examples [00:00, ? examples/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Dataset json downloaded and prepared to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51. Subsequent calls will reuse this data.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "9d68ad1b3e4a4bd79f92597aac4d3cc9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "dataset = load_dataset(\"sql-qa-chinook\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "ab44d504",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'How many employees are there?', 'answer': '8'}"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "This uses the example Chinook database.\n",
+    "To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the `.db` file in a notebooks folder at the root of this repository.\n",
+    "\n",
+    "Note that here we load a simple chain. If you want to experiment with more complex chains, or an agent, just create the `chain` object in a different way."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain import OpenAI, SQLDatabase, SQLDatabaseChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "33cdcbfc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db = SQLDatabase.from_uri(\"sqlite:///../../../notebooks/Chinook.db\")\n",
+    "llm = OpenAI(temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a SQL database chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = SQLDatabaseChain(llm=llm, database=db, input_key=\"question\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c0062e7",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "id": "d28c5e7d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'How many employees are there?',\n",
+       " 'answer': '8',\n",
+       " 'result': ' There are 8 employees.'}"
+      ]
+     },
+     "execution_count": 27,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain(dataset[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions. Note that we add a try-except because this chain can sometimes error (if SQL is written incorrectly, etc)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = []\n",
+    "predicted_dataset = []\n",
+    "error_dataset = []\n",
+    "for data in dataset:\n",
+    "    try:\n",
+    "        predictions.append(chain(data))\n",
+    "        predicted_dataset.append(data)\n",
+    "    except:\n",
+    "        error_dataset.append(data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. We can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(predicted_dataset, predictions, question_key=\"question\", prediction_key=\"result\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction['grade'] = graded_outputs[i]['text']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 3, ' INCORRECT': 4})"
+      ]
+     },
+     "execution_count": 24,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "Counter([pred['grade'] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred['grade'] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'How many employees are also customers?',\n",
+       " 'answer': 'None',\n",
+       " 'result': ' 59 employees are also customers.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 26,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

langchain/__init__.py

@@ -48,6 +48,7 @@ from langchain.utilities.google_search import GoogleSearchAPIWrapper
 from langchain.utilities.google_serper import GoogleSerperAPIWrapper
 from langchain.utilities.searx_search import SearxSearchWrapper
 from langchain.utilities.serpapi import SerpAPIWrapper
+from langchain.utilities.wikipedia import WikipediaAPIWrapper
 from langchain.utilities.wolfram_alpha import WolframAlphaAPIWrapper
 from langchain.vectorstores import FAISS, ElasticVectorSearch
 
@@ -70,6 +71,7 @@ __all__ = [
     "GoogleSearchAPIWrapper",
     "GoogleSerperAPIWrapper",
     "WolframAlphaAPIWrapper",
+    "WikipediaAPIWrapper",
     "Anthropic",
     "Banana",
     "CerebriumAI",

langchain/agents/agent.py

@@ -19,7 +19,7 @@ from langchain.llms.base import BaseLLM
 from langchain.prompts.base import BasePromptTemplate
 from langchain.prompts.few_shot import FewShotPromptTemplate
 from langchain.prompts.prompt import PromptTemplate
-from langchain.schema import AgentAction, AgentFinish
+from langchain.schema import AgentAction, AgentFinish, BaseMessage
 from langchain.tools.base import BaseTool
 
 logger = logging.getLogger()
@@ -47,11 +47,14 @@ class Agent(BaseModel):
 
     @property
     def _stop(self) -> List[str]:
-        return [f"\n{self.observation_prefix}", f"\n\t{self.observation_prefix}"]
+        return [
+            f"\n{self.observation_prefix.rstrip()}",
+            f"\n\t{self.observation_prefix.rstrip()}",
+        ]
 
     def _construct_scratchpad(
         self, intermediate_steps: List[Tuple[AgentAction, str]]
-    ) -> str:
+    ) -> Union[str, List[BaseMessage]]:
         """Construct the scratchpad that lets the agent continue its thought process."""
         thoughts = ""
         for action, observation in intermediate_steps:
@@ -432,10 +435,6 @@ class AgentExecutor(Chain, BaseModel):
                 llm_prefix="",
                 observation_prefix=self.agent.observation_prefix,
             )
-            return_direct = False
-        if return_direct:
-            # Set the log to "" because we do not want to log it.
-            return AgentFinish({self.agent.return_values[0]: observation}, "")
         return output, observation
 
     async def _atake_next_step(
@@ -480,9 +479,6 @@ class AgentExecutor(Chain, BaseModel):
                 observation_prefix=self.agent.observation_prefix,
             )
             return_direct = False
-        if return_direct:
-            # Set the log to "" because we do not want to log it.
-            return AgentFinish({self.agent.return_values[0]: observation}, "")
         return output, observation
 
     def _call(self, inputs: Dict[str, str]) -> Dict[str, Any]:
@@ -507,6 +503,10 @@ class AgentExecutor(Chain, BaseModel):
                 return self._return(next_step_output, intermediate_steps)
 
             intermediate_steps.append(next_step_output)
+            # See if tool should return directly
+            tool_return = self._get_tool_return(next_step_output)
+            if tool_return is not None:
+                return self._return(tool_return, intermediate_steps)
             iterations += 1
         output = self.agent.return_stopped_response(
             self.early_stopping_method, intermediate_steps, **inputs
@@ -535,8 +535,28 @@ class AgentExecutor(Chain, BaseModel):
                 return await self._areturn(next_step_output, intermediate_steps)
 
             intermediate_steps.append(next_step_output)
+            # See if tool should return directly
+            tool_return = self._get_tool_return(next_step_output)
+            if tool_return is not None:
+                return await self._areturn(tool_return, intermediate_steps)
+
             iterations += 1
         output = self.agent.return_stopped_response(
             self.early_stopping_method, intermediate_steps, **inputs
         )
         return await self._areturn(output, intermediate_steps)
+
+    def _get_tool_return(
+        self, next_step_output: Tuple[AgentAction, str]
+    ) -> Optional[AgentFinish]:
+        """Check if the tool is a returning tool."""
+        agent_action, observation = next_step_output
+        name_to_tool_map = {tool.name: tool for tool in self.tools}
+        # Invalid tools won't be in the map, so we return False.
+        if agent_action.tool in name_to_tool_map:
+            if name_to_tool_map[agent_action.tool].return_direct:
+                return AgentFinish(
+                    {self.agent.return_values[0]: observation},
+                    "",
+                )
+        return None

langchain/agents/chat/base.py

@@ -0,0 +1,113 @@
+import json
+from typing import Any, List, Optional, Sequence, Tuple
+
+from langchain.agents.agent import Agent
+from langchain.agents.chat.prompt import FORMAT_INSTRUCTIONS, PREFIX, SUFFIX
+from langchain.callbacks.base import BaseCallbackManager
+from langchain.chains.llm import LLMChain
+from langchain.prompts.base import BasePromptTemplate
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    HumanMessagePromptTemplate,
+    SystemMessagePromptTemplate,
+)
+from langchain.schema import AgentAction, BaseLanguageModel
+from langchain.tools import BaseTool
+
+FINAL_ANSWER_ACTION = "Final Answer:"
+
+
+class ChatAgent(Agent):
+    @property
+    def observation_prefix(self) -> str:
+        """Prefix to append the observation with."""
+        return "Observation: "
+
+    @property
+    def llm_prefix(self) -> str:
+        """Prefix to append the llm call with."""
+        return "Thought:"
+
+    def _construct_scratchpad(
+        self, intermediate_steps: List[Tuple[AgentAction, str]]
+    ) -> str:
+        agent_scratchpad = super()._construct_scratchpad(intermediate_steps)
+        if not isinstance(agent_scratchpad, str):
+            raise ValueError("agent_scratchpad should be of type string.")
+        if agent_scratchpad:
+            return (
+                f"This was your previous work "
+                f"(but I haven't seen any of it! I only see what "
+                f"you return as final answer):\n{agent_scratchpad}"
+            )
+        else:
+            return agent_scratchpad
+
+    def _extract_tool_and_input(self, text: str) -> Optional[Tuple[str, str]]:
+        if FINAL_ANSWER_ACTION in text:
+            return "Final Answer", text.split(FINAL_ANSWER_ACTION)[-1].strip()
+        try:
+            _, action, _ = text.split("```")
+            response = json.loads(action.strip())
+            return response["action"], response["action_input"]
+
+        except Exception:
+            raise ValueError(f"Could not parse LLM output: {text}")
+
+    @property
+    def _stop(self) -> List[str]:
+        return ["Observation:"]
+
+    @classmethod
+    def create_prompt(
+        cls,
+        tools: Sequence[BaseTool],
+        prefix: str = PREFIX,
+        suffix: str = SUFFIX,
+        format_instructions: str = FORMAT_INSTRUCTIONS,
+        input_variables: Optional[List[str]] = None,
+    ) -> BasePromptTemplate:
+        tool_strings = "\n".join([f"{tool.name}: {tool.description}" for tool in tools])
+        tool_names = ", ".join([tool.name for tool in tools])
+        format_instructions = format_instructions.format(tool_names=tool_names)
+        template = "\n\n".join([prefix, tool_strings, format_instructions, suffix])
+        messages = [
+            SystemMessagePromptTemplate.from_template(template),
+            HumanMessagePromptTemplate.from_template("{input}\n\n{agent_scratchpad}"),
+        ]
+        if input_variables is None:
+            input_variables = ["input", "agent_scratchpad"]
+        return ChatPromptTemplate(input_variables=input_variables, messages=messages)
+
+    @classmethod
+    def from_llm_and_tools(
+        cls,
+        llm: BaseLanguageModel,
+        tools: Sequence[BaseTool],
+        callback_manager: Optional[BaseCallbackManager] = None,
+        prefix: str = PREFIX,
+        suffix: str = SUFFIX,
+        format_instructions: str = FORMAT_INSTRUCTIONS,
+        input_variables: Optional[List[str]] = None,
+        **kwargs: Any,
+    ) -> Agent:
+        """Construct an agent from an LLM and tools."""
+        cls._validate_tools(tools)
+        prompt = cls.create_prompt(
+            tools,
+            prefix=prefix,
+            suffix=suffix,
+            format_instructions=format_instructions,
+            input_variables=input_variables,
+        )
+        llm_chain = LLMChain(
+            llm=llm,
+            prompt=prompt,
+            callback_manager=callback_manager,
+        )
+        tool_names = [tool.name for tool in tools]
+        return cls(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)
+
+    @property
+    def _agent_type(self) -> str:
+        raise ValueError

langchain/agents/chat/prompt.py

@@ -0,0 +1,29 @@
+# flake8: noqa
+PREFIX = """Answer the following questions as best you can. You have access to the following tools:"""
+FORMAT_INSTRUCTIONS = """The way you use the tools is by specifying a json blob.
+Specifically, this json should have a `action` key (with the name of the tool to use) and a `action_input` key (with the input to the tool going here).
+
+The only values that should be in the "action" field are: {tool_names}
+
+The $JSON_BLOB should only contain a SINGLE action, do NOT return a list of multiple actions. Here is an example of a valid $JSON_BLOB:
+
+```
+{{{{
+  "action": $TOOL_NAME,
+  "action_input": $INPUT
+}}}}
+```
+
+ALWAYS use the following format:
+
+Question: the input question you must answer
+Thought: you should always think about what to do
+Action:
+```
+$JSON_BLOB
+```
+Observation: the result of the action
+... (this Thought/Action/Observation can repeat N times)
+Thought: I now know the final answer
+Final Answer: the final answer to the original input question"""
+SUFFIX = """Begin! Reminder to always use the exact characters `Final Answer` when responding."""

langchain/agents/conversational/base.py

@@ -78,7 +78,7 @@ class ConversationalAgent(Agent):
     def _extract_tool_and_input(self, llm_output: str) -> Optional[Tuple[str, str]]:
         if f"{self.ai_prefix}:" in llm_output:
             return self.ai_prefix, llm_output.split(f"{self.ai_prefix}:")[-1].strip()
-        regex = r"Action: (.*?)\nAction Input: (.*)"
+        regex = r"Action: (.*?)[\n]*Action Input: (.*)"
         match = re.search(regex, llm_output)
         if not match:
             raise ValueError(f"Could not parse LLM output: `{llm_output}`")

langchain/agents/conversational_chat/__init__.py

@@ -0,0 +1 @@
+"""An agent designed to hold a conversation in addition to using tools."""

langchain/agents/conversational_chat/base.py

@@ -0,0 +1,155 @@
+"""An agent designed to hold a conversation in addition to using tools."""
+from __future__ import annotations
+
+import json
+from typing import Any, List, Optional, Sequence, Tuple
+
+from langchain.agents.agent import Agent
+from langchain.agents.conversational_chat.prompt import (
+    FORMAT_INSTRUCTIONS,
+    PREFIX,
+    SUFFIX,
+    TEMPLATE_TOOL_RESPONSE,
+)
+from langchain.callbacks.base import BaseCallbackManager
+from langchain.chains import LLMChain
+from langchain.output_parsers.base import BaseOutputParser
+from langchain.prompts.base import BasePromptTemplate
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    HumanMessagePromptTemplate,
+    MessagesPlaceholder,
+    SystemMessagePromptTemplate,
+)
+from langchain.schema import (
+    AgentAction,
+    AIMessage,
+    BaseLanguageModel,
+    BaseMessage,
+    HumanMessage,
+)
+from langchain.tools.base import BaseTool
+
+
+class AgentOutputParser(BaseOutputParser):
+    def get_format_instructions(self) -> str:
+        return FORMAT_INSTRUCTIONS
+
+    def parse(self, text: str) -> Any:
+        cleaned_output = text.strip()
+        if "```json" in cleaned_output:
+            _, cleaned_output = cleaned_output.split("```json")
+        if cleaned_output.startswith("```json"):
+            cleaned_output = cleaned_output[len("```json") :]
+        if cleaned_output.startswith("```"):
+            cleaned_output = cleaned_output[len("```") :]
+        if cleaned_output.endswith("```"):
+            cleaned_output = cleaned_output[: -len("```")]
+        cleaned_output = cleaned_output.strip()
+        response = json.loads(cleaned_output)
+        return {"action": response["action"], "action_input": response["action_input"]}
+
+
+class ConversationalChatAgent(Agent):
+    """An agent designed to hold a conversation in addition to using tools."""
+
+    output_parser: BaseOutputParser
+
+    @property
+    def _agent_type(self) -> str:
+        raise NotImplementedError
+
+    @property
+    def observation_prefix(self) -> str:
+        """Prefix to append the observation with."""
+        return "Observation: "
+
+    @property
+    def llm_prefix(self) -> str:
+        """Prefix to append the llm call with."""
+        return "Thought:"
+
+    @classmethod
+    def create_prompt(
+        cls,
+        tools: Sequence[BaseTool],
+        system_message: str = PREFIX,
+        human_message: str = SUFFIX,
+        input_variables: Optional[List[str]] = None,
+        output_parser: Optional[BaseOutputParser] = None,
+    ) -> BasePromptTemplate:
+        tool_strings = "\n".join(
+            [f"> {tool.name}: {tool.description}" for tool in tools]
+        )
+        tool_names = ", ".join([tool.name for tool in tools])
+        _output_parser = output_parser or AgentOutputParser()
+        format_instructions = human_message.format(
+            format_instructions=_output_parser.get_format_instructions()
+        )
+        final_prompt = format_instructions.format(
+            tool_names=tool_names, tools=tool_strings
+        )
+        if input_variables is None:
+            input_variables = ["input", "chat_history", "agent_scratchpad"]
+        messages = [
+            SystemMessagePromptTemplate.from_template(system_message),
+            MessagesPlaceholder(variable_name="chat_history"),
+            HumanMessagePromptTemplate.from_template(final_prompt),
+            MessagesPlaceholder(variable_name="agent_scratchpad"),
+        ]
+        return ChatPromptTemplate(input_variables=input_variables, messages=messages)
+
+    def _extract_tool_and_input(self, llm_output: str) -> Optional[Tuple[str, str]]:
+        try:
+            response = self.output_parser.parse(llm_output)
+            return response["action"], response["action_input"]
+        except Exception:
+            raise ValueError(f"Could not parse LLM output: {llm_output}")
+
+    def _construct_scratchpad(
+        self, intermediate_steps: List[Tuple[AgentAction, str]]
+    ) -> List[BaseMessage]:
+        """Construct the scratchpad that lets the agent continue its thought process."""
+        thoughts: List[BaseMessage] = []
+        for action, observation in intermediate_steps:
+            thoughts.append(AIMessage(content=action.log))
+            human_message = HumanMessage(
+                content=TEMPLATE_TOOL_RESPONSE.format(observation=observation)
+            )
+            thoughts.append(human_message)
+        return thoughts
+
+    @classmethod
+    def from_llm_and_tools(
+        cls,
+        llm: BaseLanguageModel,
+        tools: Sequence[BaseTool],
+        callback_manager: Optional[BaseCallbackManager] = None,
+        system_message: str = PREFIX,
+        human_message: str = SUFFIX,
+        input_variables: Optional[List[str]] = None,
+        output_parser: Optional[BaseOutputParser] = None,
+        **kwargs: Any,
+    ) -> Agent:
+        """Construct an agent from an LLM and tools."""
+        cls._validate_tools(tools)
+        _output_parser = output_parser or AgentOutputParser()
+        prompt = cls.create_prompt(
+            tools,
+            system_message=system_message,
+            human_message=human_message,
+            input_variables=input_variables,
+            output_parser=_output_parser,
+        )
+        llm_chain = LLMChain(
+            llm=llm,
+            prompt=prompt,
+            callback_manager=callback_manager,
+        )
+        tool_names = [tool.name for tool in tools]
+        return cls(
+            llm_chain=llm_chain,
+            allowed_tools=tool_names,
+            output_parser=_output_parser,
+            **kwargs,
+        )

langchain/agents/conversational_chat/prompt.py

@@ -0,0 +1,57 @@
+# flake8: noqa
+PREFIX = """Assistant is a large language model trained by OpenAI.
+
+Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.
+
+Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.
+
+Overall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist."""
+
+FORMAT_INSTRUCTIONS = """RESPONSE FORMAT INSTRUCTIONS
+----------------------------
+
+When responding to me please, please output a response in one of two formats:
+
+**Option 1:**
+Use this if you want the human to use a tool.
+Markdown code snippet formatted in the following schema:
+
+```json
+{{{{
+    "action": string \\ The action to take. Must be one of {tool_names}
+    "action_input": string \\ The input to the action
+}}}}
+```
+
+**Option #2:**
+Use this if you want to respond directly to the human. Markdown code snippet formatted in the following schema:
+
+```json
+{{{{
+    "action": "Final Answer",
+    "action_input": string \\ You should put what you want to return to use here
+}}}}
+```"""
+
+SUFFIX = """TOOLS
+------
+Assistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:
+
+{{tools}}
+
+{format_instructions}
+
+USER'S INPUT
+--------------------
+Here is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):
+
+{{{{input}}}}"""
+
+TEMPLATE_TOOL_RESPONSE = """TOOL RESPONSE: 
+---------------------
+{observation}
+
+USER'S INPUT
+--------------------
+
+Okay, so what is the response to my original question? If using information from tools, you must say it explicitly - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else."""

langchain/agents/load_tools.py

@@ -9,12 +9,13 @@ from langchain.chains.api.base import APIChain
 from langchain.chains.llm_math.base import LLMMathChain
 from langchain.chains.pal.base import PALChain
 from langchain.llms.base import BaseLLM
-from langchain.tools.python.tool import PythonREPLTool
 from langchain.requests import RequestsWrapper
 from langchain.tools.base import BaseTool
 from langchain.tools.bing_search.tool import BingSearchRun
 from langchain.tools.google_search.tool import GoogleSearchResults, GoogleSearchRun
+from langchain.tools.python.tool import PythonREPLTool
 from langchain.tools.requests.tool import RequestsGetTool
+from langchain.tools.wikipedia.tool import WikipediaQueryRun
 from langchain.tools.wolfram_alpha.tool import WolframAlphaQueryRun
 from langchain.utilities.bash import BashProcess
 from langchain.utilities.bing_search import BingSearchAPIWrapper
@@ -22,6 +23,7 @@ from langchain.utilities.google_search import GoogleSearchAPIWrapper
 from langchain.utilities.google_serper import GoogleSerperAPIWrapper
 from langchain.utilities.searx_search import SearxSearchWrapper
 from langchain.utilities.serpapi import SerpAPIWrapper
+from langchain.utilities.wikipedia import WikipediaAPIWrapper
 from langchain.utilities.wolfram_alpha import WolframAlphaAPIWrapper
 
 
@@ -124,6 +126,10 @@ def _get_google_search(**kwargs: Any) -> BaseTool:
     return GoogleSearchRun(api_wrapper=GoogleSearchAPIWrapper(**kwargs))
 
 
+def _get_wikipedia(**kwargs: Any) -> BaseTool:
+    return WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper(**kwargs))
+
+
 def _get_google_serper(**kwargs: Any) -> BaseTool:
     return Tool(
         name="Serper Search",
@@ -173,6 +179,7 @@ _EXTRA_OPTIONAL_TOOLS = {
     "google-serper": (_get_google_serper, ["serper_api_key"]),
     "serpapi": (_get_serpapi, ["serpapi_api_key", "aiosession"]),
     "searx-search": (_get_searx_search, ["searx_host"]),
+    "wikipedia": (_get_wikipedia, ["top_k_results"]),
 }
 
 

langchain/agents/loading.py

@@ -6,7 +6,9 @@ from typing import Any, List, Optional, Union
 import yaml
 
 from langchain.agents.agent import Agent
+from langchain.agents.chat.base import ChatAgent
 from langchain.agents.conversational.base import ConversationalAgent
+from langchain.agents.conversational_chat.base import ConversationalChatAgent
 from langchain.agents.mrkl.base import ZeroShotAgent
 from langchain.agents.react.base import ReActDocstoreAgent
 from langchain.agents.self_ask_with_search.base import SelfAskWithSearchAgent
@@ -20,6 +22,8 @@ AGENT_TO_CLASS = {
     "react-docstore": ReActDocstoreAgent,
     "self-ask-with-search": SelfAskWithSearchAgent,
     "conversational-react-description": ConversationalAgent,
+    "chat-zero-shot-react-description": ChatAgent,
+    "chat-conversational-react-description": ConversationalChatAgent,
 }
 
 URL_BASE = "https://raw.githubusercontent.com/hwchase17/langchain-hub/master/agents/"

langchain/agents/mrkl/base.py

@@ -40,7 +40,7 @@ def get_action_and_input(llm_output: str) -> Tuple[str, str]:
     """
     if FINAL_ANSWER_ACTION in llm_output:
         return "Final Answer", llm_output.split(FINAL_ANSWER_ACTION)[-1].strip()
-    regex = r"Action: (.*?)\nAction Input: (.*)"
+    regex = r"Action: (.*?)[\n]*Action Input: (.*)"
     match = re.search(regex, llm_output, re.DOTALL)
     if not match:
         raise ValueError(f"Could not parse LLM output: `{llm_output}`")

langchain/chains/__init__.py

@@ -16,6 +16,7 @@ from langchain.chains.loading import load_chain
 from langchain.chains.mapreduce import MapReduceChain
 from langchain.chains.moderation import OpenAIModerationChain
 from langchain.chains.pal.base import PALChain
+from langchain.chains.qa_generation.base import QAGenerationChain
 from langchain.chains.qa_with_sources.base import QAWithSourcesChain
 from langchain.chains.qa_with_sources.vector_db import VectorDBQAWithSourcesChain
 from langchain.chains.sequential import SequentialChain, SimpleSequentialChain
@@ -52,4 +53,5 @@ __all__ = [
     "ChatVectorDBChain",
     "GraphQAChain",
     "ConstitutionalChain",
+    "QAGenerationChain",
 ]

langchain/chains/chat_vector_db/base.py

@@ -1,7 +1,8 @@
 """Chain for chatting with a vector database."""
 from __future__ import annotations
 
-from typing import Any, Dict, List, Optional, Tuple
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 from pydantic import BaseModel
 
@@ -33,6 +34,7 @@ class ChatVectorDBChain(Chain, BaseModel):
     output_key: str = "answer"
     return_source_documents: bool = False
     top_k_docs_for_context: int = 4
+    get_chat_history: Optional[Callable[[Tuple[str, str]], str]] = None
     """Return the source documents."""
 
     @property
@@ -81,7 +83,8 @@ class ChatVectorDBChain(Chain, BaseModel):
 
     def _call(self, inputs: Dict[str, Any]) -> Dict[str, Any]:
         question = inputs["question"]
-        chat_history_str = _get_chat_history(inputs["chat_history"])
+        get_chat_history = self.get_chat_history or _get_chat_history
+        chat_history_str = get_chat_history(inputs["chat_history"])
         vectordbkwargs = inputs.get("vectordbkwargs", {})
         if chat_history_str:
             new_question = self.question_generator.run(
@@ -103,7 +106,8 @@ class ChatVectorDBChain(Chain, BaseModel):
 
     async def _acall(self, inputs: Dict[str, Any]) -> Dict[str, Any]:
         question = inputs["question"]
-        chat_history_str = _get_chat_history(inputs["chat_history"])
+        get_chat_history = self.get_chat_history or _get_chat_history
+        chat_history_str = get_chat_history(inputs["chat_history"])
         vectordbkwargs = inputs.get("vectordbkwargs", {})
         if chat_history_str:
             new_question = await self.question_generator.arun(
@@ -123,3 +127,8 @@ class ChatVectorDBChain(Chain, BaseModel):
             return {self.output_key: answer, "source_documents": docs}
         else:
             return {self.output_key: answer}
+
+    def save(self, file_path: Union[Path, str]) -> None:
+        if self.get_chat_history:
+            raise ValueError("Chain not savable when `get_chat_history` is not None.")
+        super().save(file_path)

langchain/chains/combine_documents/map_rerank.py

@@ -9,7 +9,7 @@ from pydantic import BaseModel, Extra, root_validator
 from langchain.chains.combine_documents.base import BaseCombineDocumentsChain
 from langchain.chains.llm import LLMChain
 from langchain.docstore.document import Document
-from langchain.prompts.base import RegexParser
+from langchain.output_parsers.regex import RegexParser
 
 
 class MapRerankDocumentsChain(BaseCombineDocumentsChain, BaseModel):

langchain/chains/conversation/memory.py

@@ -1,6 +1,9 @@
 """Memory modules for conversation prompts."""
 
-from langchain.memory.buffer import ConversationBufferMemory
+from langchain.memory.buffer import (
+    ConversationBufferMemory,
+    ConversationStringBufferMemory,
+)
 from langchain.memory.buffer_window import ConversationBufferWindowMemory
 from langchain.memory.combined import CombinedMemory
 from langchain.memory.entity import ConversationEntityMemory
@@ -18,4 +21,5 @@ __all__ = [
     "ConversationEntityMemory",
     "ConversationBufferMemory",
     "CombinedMemory",
+    "ConversationStringBufferMemory",
 ]

langchain/chains/hyde/base.py

@@ -52,7 +52,7 @@ class HypotheticalDocumentEmbedder(Chain, Embeddings, BaseModel):
     def embed_query(self, text: str) -> List[float]:
         """Generate a hypothetical document and embedded it."""
         var_name = self.llm_chain.input_keys[0]
-        result = self.llm_chain.generate([{var_name: text}])
+        result, _ = self.llm_chain.generate([{var_name: text}])
         documents = [generation.text for generation in result.generations[0]]
         embeddings = self.embed_documents(documents)
         return self.combine_embeddings(embeddings)

langchain/chains/llm.py

@@ -3,13 +3,20 @@ from __future__ import annotations
 
 from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
 
-from pydantic import BaseModel, Extra
+from pydantic import BaseModel, Extra, Field
 
 from langchain.chains.base import Chain
 from langchain.input import get_colored_text
+from langchain.output_parsers.base import OutputGuardrail
 from langchain.prompts.base import BasePromptTemplate
 from langchain.prompts.prompt import PromptTemplate
-from langchain.schema import BaseLanguageModel, LLMResult, PromptValue
+from langchain.schema import (
+    BaseLanguageModel,
+    Guardrail,
+    LLMResult,
+    PromptValue,
+    ValidationError,
+)
 
 
 class LLMChain(Chain, BaseModel):
@@ -30,6 +37,8 @@ class LLMChain(Chain, BaseModel):
     """Prompt object to use."""
     llm: BaseLanguageModel
     output_key: str = "text"  #: :meta private:
+    output_parser: Optional[OutputGuardrail] = None
+    guardrails: List[Guardrail] = Field(default_factory=list)
 
     class Config:
         """Configuration for this pydantic object."""
@@ -56,15 +65,19 @@ class LLMChain(Chain, BaseModel):
     def _call(self, inputs: Dict[str, Any]) -> Dict[str, str]:
         return self.apply([inputs])[0]
 
-    def generate(self, input_list: List[Dict[str, Any]]) -> LLMResult:
+    def generate(
+        self, input_list: List[Dict[str, Any]]
+    ) -> Tuple[LLMResult, List[PromptValue]]:
         """Generate LLM result from inputs."""
         prompts, stop = self.prep_prompts(input_list)
-        return self.llm.generate_prompt(prompts, stop)
+        return self.llm.generate_prompt(prompts, stop), prompts
 
-    async def agenerate(self, input_list: List[Dict[str, Any]]) -> LLMResult:
+    async def agenerate(
+        self, input_list: List[Dict[str, Any]]
+    ) -> Tuple[LLMResult, List[PromptValue]]:
         """Generate LLM result from inputs."""
         prompts, stop = await self.aprep_prompts(input_list)
-        return await self.llm.agenerate_prompt(prompts, stop)
+        return await self.llm.agenerate_prompt(prompts, stop), prompts
 
     def prep_prompts(
         self, input_list: List[Dict[str, Any]]
@@ -115,20 +128,37 @@ class LLMChain(Chain, BaseModel):
 
     def apply(self, input_list: List[Dict[str, Any]]) -> List[Dict[str, str]]:
         """Utilize the LLM generate method for speed gains."""
-        response = self.generate(input_list)
-        return self.create_outputs(response)
+        response, prompts = self.generate(input_list)
+        return self.create_outputs(response, prompts)
 
     async def aapply(self, input_list: List[Dict[str, Any]]) -> List[Dict[str, str]]:
         """Utilize the LLM generate method for speed gains."""
-        response = await self.agenerate(input_list)
-        return self.create_outputs(response)
+        response, prompts = await self.agenerate(input_list)
+        return self.create_outputs(response, prompts)
 
-    def create_outputs(self, response: LLMResult) -> List[Dict[str, str]]:
+    def _get_final_output(self, text: str, prompt_value: PromptValue) -> Any:
+        result: Any = text
+        for guardrail in self.guardrails:
+            if isinstance(guardrail, OutputGuardrail):
+                try:
+                    result = guardrail.output_parser.parse(result)
+                    error = None
+                except Exception as e:
+                    error = ValidationError(text=e)
+            else:
+                error = guardrail.check(prompt_value, result)
+            if error is not None:
+                result = guardrail.fix(prompt_value, result, error)
+        return result
+
+    def create_outputs(
+        self, response: LLMResult, prompts: List[PromptValue]
+    ) -> List[Dict[str, str]]:
         """Create outputs from response."""
         return [
             # Get the text of the top generated string.
-            {self.output_key: generation[0].text}
-            for generation in response.generations
+            {self.output_key: self._get_final_output(generation[0].text, prompts[i])}
+            for i, generation in enumerate(response.generations)
         ]
 
     async def _acall(self, inputs: Dict[str, Any]) -> Dict[str, str]:

langchain/chains/qa_generation/base.py

@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+import json
+from typing import Any, Dict, List, Optional
+
+from pydantic import Field
+
+from langchain.chains.base import Chain
+from langchain.chains.llm import LLMChain
+from langchain.chains.qa_generation.prompt import PROMPT_SELECTOR
+from langchain.prompts.base import BasePromptTemplate
+from langchain.schema import BaseLanguageModel
+from langchain.text_splitter import RecursiveCharacterTextSplitter, TextSplitter
+
+
+class QAGenerationChain(Chain):
+    llm_chain: LLMChain
+    text_splitter: TextSplitter = Field(
+        default=RecursiveCharacterTextSplitter(chunk_overlap=500)
+    )
+    input_key: str = "text"
+    output_key: str = "questions"
+    k: Optional[int] = None
+
+    @classmethod
+    def from_llm(
+        cls,
+        llm: BaseLanguageModel,
+        prompt: Optional[BasePromptTemplate] = None,
+        **kwargs: Any,
+    ) -> QAGenerationChain:
+        _prompt = prompt or PROMPT_SELECTOR.get_prompt(llm)
+        chain = LLMChain(llm=llm, prompt=_prompt)
+        return cls(llm_chain=chain, **kwargs)
+
+    @property
+    def _chain_type(self) -> str:
+        raise NotImplementedError
+
+    @property
+    def input_keys(self) -> List[str]:
+        return [self.input_key]
+
+    @property
+    def output_keys(self) -> List[str]:
+        return [self.output_key]
+
+    def _call(self, inputs: Dict[str, str]) -> Dict[str, Any]:
+        docs = self.text_splitter.create_documents([inputs[self.input_key]])
+        results, _ = self.llm_chain.generate([{"text": d.page_content} for d in docs])
+        qa = [json.loads(res[0].text) for res in results.generations]
+        return {self.output_key: qa}
+
+    async def _acall(self, inputs: Dict[str, str]) -> Dict[str, str]:
+        raise NotImplementedError

langchain/chains/qa_generation/prompt.py

@@ -0,0 +1,50 @@
+# flake8: noqa
+from langchain.chains.prompt_selector import ConditionalPromptSelector, is_chat_model
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    HumanMessagePromptTemplate,
+    SystemMessagePromptTemplate,
+)
+from langchain.prompts.prompt import PromptTemplate
+
+templ1 = """You are a smart assistant designed to help high school teachers come up with reading comprehension questions.
+Given a piece of text, you must come up with a question and answer pair that can be used to test a student's reading comprehension abilities.
+When coming up with this question/answer pair, you must respond in the following format:
+```
+{{
+    "question": "$YOUR_QUESTION_HERE",
+    "answer": "$THE_ANSWER_HERE"
+}}
+```
+
+Everything between the ``` must be valid json.
+"""
+templ2 = """Please come up with a question/answer pair, in the specified JSON format, for the following text:
+----------------
+{text}"""
+CHAT_PROMPT = ChatPromptTemplate.from_messages(
+    [
+        SystemMessagePromptTemplate.from_template(templ1),
+        HumanMessagePromptTemplate.from_template(templ2),
+    ]
+)
+templ = """You are a smart assistant designed to help high school teachers come up with reading comprehension questions.
+Given a piece of text, you must come up with a question and answer pair that can be used to test a student's reading comprehension abilities.
+When coming up with this question/answer pair, you must respond in the following format:
+```
+{{
+    "question": "$YOUR_QUESTION_HERE",
+    "answer": "$THE_ANSWER_HERE"
+}}
+```
+
+Everything between the ``` must be valid json.
+
+Please come up with a question/answer pair, in the specified JSON format, for the following text:
+----------------
+{text}"""
+PROMPT = PromptTemplate.from_template(templ)
+
+PROMPT_SELECTOR = ConditionalPromptSelector(
+    default_prompt=PROMPT, conditionals=[(is_chat_model, CHAT_PROMPT)]
+)

langchain/chains/question_answering/__init__.py

@@ -61,7 +61,7 @@ def _load_stuff_chain(
 ) -> StuffDocumentsChain:
     _prompt = prompt or stuff_prompt.PROMPT_SELECTOR.get_prompt(llm)
     llm_chain = LLMChain(
-        llm=llm, prompt=prompt, verbose=verbose, callback_manager=callback_manager
+        llm=llm, prompt=_prompt, verbose=verbose, callback_manager=callback_manager
     )
     # TODO: document prompt
     return StuffDocumentsChain(

langchain/chains/question_answering/map_rerank_prompt.py

@@ -1,6 +1,6 @@
 # flake8: noqa
 from langchain.prompts import PromptTemplate
-from langchain.prompts.base import RegexParser
+from langchain.output_parsers.regex import RegexParser
 
 output_parser = RegexParser(
     regex=r"(.*?)\nScore: (.*)",

langchain/chains/sql_database/base.py

@@ -117,6 +117,8 @@ class SQLDatabaseSequentialChain(Chain, BaseModel):
     This is useful in cases where the number of tables in the database is large.
     """
 
+    return_intermediate_steps: bool = False
+
     @classmethod
     def from_llm(
         cls,
@@ -154,7 +156,10 @@ class SQLDatabaseSequentialChain(Chain, BaseModel):
 
         :meta private:
         """
-        return [self.output_key]
+        if not self.return_intermediate_steps:
+            return [self.output_key]
+        else:
+            return [self.output_key, "intermediate_steps"]
 
     def _call(self, inputs: Dict[str, str]) -> Dict[str, str]:
         _table_names = self.sql_chain.database.get_table_names()

langchain/chains/sql_database/prompt.py

@@ -1,5 +1,5 @@
 # flake8: noqa
-from langchain.prompts.base import CommaSeparatedListOutputParser
+from langchain.output_parsers.list import CommaSeparatedListOutputParser
 from langchain.prompts.prompt import PromptTemplate
 
 _DEFAULT_TEMPLATE = """Given an input question, first create a syntactically correct {dialect} query to run, then look at the results of the query and return the answer. Unless the user specifies in his question a specific number of examples he wishes to obtain, always limit your query to at most {top_k} results. You can order the results by a relevant column to return the most interesting examples in the database.

langchain/chat_models/__init__.py

@@ -1,3 +1,4 @@
 from langchain.chat_models.openai import ChatOpenAI
+from langchain.chat_models.promptlayer_openai import PromptLayerChatOpenAI
 
-__all__ = ["ChatOpenAI"]
+__all__ = ["ChatOpenAI", "PromptLayerChatOpenAI"]

langchain/chat_models/base.py

@@ -12,6 +12,7 @@ from langchain.schema import (
     BaseMessage,
     ChatGeneration,
     ChatResult,
+    HumanMessage,
     LLMResult,
     PromptValue,
 )
@@ -60,13 +61,44 @@ class BaseChatModel(BaseLanguageModel, BaseModel, ABC):
         self, prompts: List[PromptValue], stop: Optional[List[str]] = None
     ) -> LLMResult:
         prompt_messages = [p.to_messages() for p in prompts]
-        return self.generate(prompt_messages, stop=stop)
+        prompt_strings = [p.to_string() for p in prompts]
+        self.callback_manager.on_llm_start(
+            {"name": self.__class__.__name__}, prompt_strings, verbose=self.verbose
+        )
+        try:
+            output = self.generate(prompt_messages, stop=stop)
+        except (KeyboardInterrupt, Exception) as e:
+            self.callback_manager.on_llm_error(e, verbose=self.verbose)
+            raise e
+        self.callback_manager.on_llm_end(output, verbose=self.verbose)
+        return output
 
     async def agenerate_prompt(
         self, prompts: List[PromptValue], stop: Optional[List[str]] = None
     ) -> LLMResult:
         prompt_messages = [p.to_messages() for p in prompts]
-        return await self.agenerate(prompt_messages, stop=stop)
+        prompt_strings = [p.to_string() for p in prompts]
+        if self.callback_manager.is_async:
+            await self.callback_manager.on_llm_start(
+                {"name": self.__class__.__name__}, prompt_strings, verbose=self.verbose
+            )
+        else:
+            self.callback_manager.on_llm_start(
+                {"name": self.__class__.__name__}, prompt_strings, verbose=self.verbose
+            )
+        try:
+            output = await self.agenerate(prompt_messages, stop=stop)
+        except (KeyboardInterrupt, Exception) as e:
+            if self.callback_manager.is_async:
+                await self.callback_manager.on_llm_error(e, verbose=self.verbose)
+            else:
+                self.callback_manager.on_llm_error(e, verbose=self.verbose)
+            raise e
+        if self.callback_manager.is_async:
+            await self.callback_manager.on_llm_end(output, verbose=self.verbose)
+        else:
+            self.callback_manager.on_llm_end(output, verbose=self.verbose)
+        return output
 
     @abstractmethod
     def _generate(
@@ -85,6 +117,10 @@ class BaseChatModel(BaseLanguageModel, BaseModel, ABC):
     ) -> BaseMessage:
         return self._generate(messages, stop=stop).generations[0].message
 
+    def call_as_llm(self, message: str, stop: Optional[List[str]] = None) -> str:
+        result = self([HumanMessage(content=message)], stop=stop)
+        return result.content
+
 
 class SimpleChatModel(BaseChatModel):
     def _generate(

langchain/chat_models/promptlayer_openai.py

@@ -0,0 +1,84 @@
+"""PromptLayer wrapper."""
+import datetime
+from typing import List, Optional
+
+from pydantic import BaseModel
+
+from langchain.chat_models import ChatOpenAI
+from langchain.schema import BaseMessage, ChatResult
+
+
+class PromptLayerChatOpenAI(ChatOpenAI, BaseModel):
+    """Wrapper around OpenAI Chat large language models and PromptLayer.
+
+    To use, you should have the ``openai`` and ``promptlayer`` python
+    package installed, and the environment variable ``OPENAI_API_KEY``
+    and ``PROMPTLAYER_API_KEY`` set with your openAI API key and
+    promptlayer key respectively.
+
+    All parameters that can be passed to the OpenAI LLM can also
+    be passed here. The PromptLayerChatOpenAI LLM adds an extra
+    ``pl_tags`` parameter that can be used to tag the request.
+
+    Example:
+        .. code-block:: python
+
+            from langchain.chat_models import PromptLayerChatOpenAI
+            openai = PromptLayerChatOpenAI(model_name="gpt-3.5-turbo")
+    """
+
+    pl_tags: Optional[List[str]]
+
+    def _generate(
+        self, messages: List[BaseMessage], stop: Optional[List[str]] = None
+    ) -> ChatResult:
+        """Call ChatOpenAI generate and then call PromptLayer API to log the request."""
+        from promptlayer.utils import get_api_key, promptlayer_api_request
+
+        request_start_time = datetime.datetime.now().timestamp()
+        generated_responses = super()._generate(messages, stop)
+        request_end_time = datetime.datetime.now().timestamp()
+        message_dicts, params = super()._create_message_dicts(messages, stop)
+        for i, generation in enumerate(generated_responses.generations):
+            response_dict, params = super()._create_message_dicts(
+                [generation.message], stop
+            )
+            promptlayer_api_request(
+                "langchain.PromptLayerChatOpenAI",
+                "langchain",
+                message_dicts,
+                params,
+                self.pl_tags,
+                response_dict,
+                request_start_time,
+                request_end_time,
+                get_api_key(),
+            )
+        return generated_responses
+
+    async def _agenerate(
+        self, messages: List[BaseMessage], stop: Optional[List[str]] = None
+    ) -> ChatResult:
+        """Call ChatOpenAI agenerate and then call PromptLayer to log."""
+        from promptlayer.utils import get_api_key, promptlayer_api_request
+
+        request_start_time = datetime.datetime.now().timestamp()
+        generated_responses = await super()._agenerate(messages, stop)
+        request_end_time = datetime.datetime.now().timestamp()
+        message_dicts, params = super()._create_message_dicts(messages, stop)
+        for i, generation in enumerate(generated_responses.generations):
+            response_dict, params = super()._create_message_dicts(
+                [generation.message], stop
+            )
+            promptlayer_api_request(
+                "langchain.PromptLayerChatOpenAI.async",
+                "langchain",
+                message_dicts,
+                params,
+                self.pl_tags,
+                response_dict,
+                request_start_time,
+                request_end_time,
+                get_api_key(),
+            )
+        return generated_responses

langchain/document_loaders/__init__.py

@@ -4,6 +4,7 @@ from langchain.document_loaders.airbyte_json import AirbyteJSONLoader
 from langchain.document_loaders.azlyrics import AZLyricsLoader
 from langchain.document_loaders.college_confidential import CollegeConfidentialLoader
 from langchain.document_loaders.conllu import CoNLLULoader
+from langchain.document_loaders.csv import CSVLoader
 from langchain.document_loaders.directory import DirectoryLoader
 from langchain.document_loaders.docx import UnstructuredDocxLoader
 from langchain.document_loaders.email import UnstructuredEmailLoader
@@ -19,14 +20,15 @@ from langchain.document_loaders.html import UnstructuredHTMLLoader
 from langchain.document_loaders.ifixit import IFixitLoader
 from langchain.document_loaders.image import UnstructuredImageLoader
 from langchain.document_loaders.imsdb import IMSDbLoader
+from langchain.document_loaders.markdown import UnstructuredMarkdownLoader
 from langchain.document_loaders.notebook import NotebookLoader
 from langchain.document_loaders.notion import NotionDirectoryLoader
 from langchain.document_loaders.obsidian import ObsidianLoader
-from langchain.document_loaders.online_pdf import OnlinePDFLoader
-from langchain.document_loaders.paged_pdf import PagedPDFSplitter
 from langchain.document_loaders.pdf import (
+    OnlinePDFLoader,
     PDFMinerLoader,
     PyMuPDFLoader,
+    PyPDFLoader,
     UnstructuredPDFLoader,
 )
 from langchain.document_loaders.powerpoint import UnstructuredPowerPointLoader
@@ -44,7 +46,14 @@ from langchain.document_loaders.unstructured import (
 from langchain.document_loaders.url import UnstructuredURLLoader
 from langchain.document_loaders.web_base import WebBaseLoader
 from langchain.document_loaders.word_document import UnstructuredWordDocumentLoader
-from langchain.document_loaders.youtube import YoutubeLoader
+from langchain.document_loaders.youtube import (
+    GoogleApiClient,
+    GoogleApiYoutubeLoader,
+    YoutubeLoader,
+)
+
+"""Legacy: only for backwards compat. use PyPDFLoader instead"""
+PagedPDFSplitter = PyPDFLoader
 
 __all__ = [
     "UnstructuredFileLoader",
@@ -62,6 +71,7 @@ __all__ = [
     "ObsidianLoader",
     "UnstructuredDocxLoader",
     "UnstructuredEmailLoader",
+    "UnstructuredMarkdownLoader",
     "RoamLoader",
     "YoutubeLoader",
     "S3FileLoader",
@@ -78,6 +88,7 @@ __all__ = [
     "IFixitLoader",
     "GutenbergLoader",
     "PagedPDFSplitter",
+    "PyPDFLoader",
     "EverNoteLoader",
     "AirbyteJSONLoader",
     "OnlinePDFLoader",
@@ -88,4 +99,7 @@ __all__ = [
     "FacebookChatLoader",
     "NotebookLoader",
     "CoNLLULoader",
+    "GoogleApiYoutubeLoader",
+    "GoogleApiClient",
+    "CSVLoader",
 ]

langchain/document_loaders/csv.py

@@ -0,0 +1,47 @@
+from csv import DictReader
+from typing import Dict, List, Optional
+
+from langchain.docstore.document import Document
+from langchain.document_loaders.base import BaseLoader
+
+
+class CSVLoader(BaseLoader):
+    """Loads a CSV file into a list of documents.
+
+    Each document represents one row of the CSV file. Every row is converted into a
+    key/value pair and outputted to a new line in the document's page_content.
+
+    Output Example:
+        .. code-block:: txt
+
+            column1: value1
+            column2: value2
+            column3: value3
+    """
+
+    def __init__(self, file_path: str, csv_args: Optional[Dict] = None):
+        self.file_path = file_path
+        if csv_args is None:
+            self.csv_args = {
+                "delimiter": ",",
+                "quotechar": '"',
+            }
+        else:
+            self.csv_args = csv_args
+
+    def load(self) -> List[Document]:
+        docs = []
+
+        with open(self.file_path, newline="") as csvfile:
+            csv = DictReader(csvfile, **self.csv_args)  # type: ignore
+            for i, row in enumerate(csv):
+                docs.append(
+                    Document(
+                        page_content="\n".join(
+                            f"{k.strip()}: {v.strip()}" for k, v in row.items()
+                        ),
+                        metadata={"source": self.file_path, "row": i},
+                    )
+                )
+
+        return docs

langchain/document_loaders/gitbook.py

@@ -12,9 +12,26 @@ class GitbookLoader(WebBaseLoader):
     2. load all (relative) paths in the navbar.
     """
 
-    def __init__(self, web_page: str, load_all_paths: bool = False):
-        """Initialize with web page and whether to load all paths."""
+    def __init__(
+        self,
+        web_page: str,
+        load_all_paths: bool = False,
+        base_url: Optional[str] = None,
+    ):
+        """Initialize with web page and whether to load all paths.
+
+        Args:
+            web_page: The web page to load or the starting point from where
+                relative paths are discovered.
+            load_all_paths: If set to True, all relative paths in the navbar
+                are loaded instead of only `web_page`.
+            base_url: If `load_all_paths` is True, the relative paths are
+                appended to this base url. Defaults to `web_page` if not set.
+        """
         super().__init__(web_page)
+        self.base_url = base_url or web_page
+        if self.base_url.endswith("/"):
+            self.base_url = self.base_url[:-1]
         self.load_all_paths = load_all_paths
 
     def load(self) -> List[Document]:
@@ -24,7 +41,7 @@ class GitbookLoader(WebBaseLoader):
             relative_paths = self._get_paths(soup_info)
             documents = []
             for path in relative_paths:
-                url = self.web_path + path
+                url = self.base_url + path
                 print(f"Fetching text from {url}")
                 soup_info = self._scrape(url)
                 documents.append(self._get_document(soup_info, url))

langchain/document_loaders/googledrive.py

@@ -26,16 +26,26 @@ class GoogleDriveLoader(BaseLoader, BaseModel):
     token_path: Path = Path.home() / ".credentials" / "token.json"
     folder_id: Optional[str] = None
     document_ids: Optional[List[str]] = None
+    file_ids: Optional[List[str]] = None
 
     @root_validator
     def validate_folder_id_or_document_ids(
         cls, values: Dict[str, Any]
     ) -> Dict[str, Any]:
         """Validate that either folder_id or document_ids is set, but not both."""
-        if values.get("folder_id") and values.get("document_ids"):
-            raise ValueError("Cannot specify both folder_id and document_ids")
-        if not values.get("folder_id") and not values.get("document_ids"):
-            raise ValueError("Must specify either folder_id or document_ids")
+        if values.get("folder_id") and (
+            values.get("document_ids") or values.get("file_ids")
+        ):
+            raise ValueError(
+                "Cannot specify both folder_id and document_ids nor "
+                "folder_id and file_ids"
+            )
+        if (
+            not values.get("folder_id")
+            and not values.get("document_ids")
+            and not values.get("file_ids")
+        ):
+            raise ValueError("Must specify either folder_id, document_ids, or file_ids")
         return values
 
     @validator("credentials_path")
@@ -115,13 +125,16 @@ class GoogleDriveLoader(BaseLoader, BaseModel):
             .execute()
         )
         items = results.get("files", [])
+        returns = []
+        for item in items:
+            if item["mimeType"] == "application/vnd.google-apps.document":
+                returns.append(self._load_document_from_id(item["id"]))
+            elif item["mimeType"] == "application/pdf":
+                returns.extend(self._load_file_from_id(item["id"]))
+            else:
+                pass
 
-        return [
-            self._load_document_from_id(item["id"])
-            for item in items
-            # Only support Google Docs for now
-            if item["mimeType"] == "application/vnd.google-apps.document"
-        ]
+        return returns
 
     def _load_documents_from_ids(self) -> List[Document]:
         """Load documents from a list of IDs."""
@@ -130,9 +143,53 @@ class GoogleDriveLoader(BaseLoader, BaseModel):
 
         return [self._load_document_from_id(doc_id) for doc_id in self.document_ids]
 
+    def _load_file_from_id(self, id: str) -> List[Document]:
+        """Load a file from an ID."""
+        from io import BytesIO
+
+        from googleapiclient.discovery import build
+        from googleapiclient.http import MediaIoBaseDownload
+
+        creds = self._load_credentials()
+        service = build("drive", "v3", credentials=creds)
+
+        request = service.files().get_media(fileId=id)
+        fh = BytesIO()
+        downloader = MediaIoBaseDownload(fh, request)
+        done = False
+        while done is False:
+            status, done = downloader.next_chunk()
+        content = fh.getvalue()
+
+        from PyPDF2 import PdfReader
+
+        pdf_reader = PdfReader(BytesIO(content))
+
+        return [
+            Document(
+                page_content=page.extract_text(),
+                metadata={
+                    "source": f"https://drive.google.com/file/d/{id}/view",
+                    "page": i,
+                },
+            )
+            for i, page in enumerate(pdf_reader.pages)
+        ]
+
+    def _load_file_from_ids(self) -> List[Document]:
+        """Load files from a list of IDs."""
+        if not self.file_ids:
+            raise ValueError("file_ids must be set")
+        docs = []
+        for file_id in self.file_ids:
+            docs.extend(self._load_file_from_id(file_id))
+        return docs
+
     def load(self) -> List[Document]:
         """Load documents."""
         if self.folder_id:
             return self._load_documents_from_folder()
-        else:
+        elif self.document_ids:
             return self._load_documents_from_ids()
+        else:
+            return self._load_file_from_ids()

langchain/document_loaders/html.py

@@ -1,4 +1,4 @@
-"""Loader that loads PDF files."""
+"""Loader that uses unstructured to load HTML files."""
 from typing import List
 
 from langchain.document_loaders.unstructured import UnstructuredFileLoader
@@ -10,4 +10,4 @@ class UnstructuredHTMLLoader(UnstructuredFileLoader):
     def _get_elements(self) -> List:
         from unstructured.partition.html import partition_html
 
-        return partition_html(filename=self.file_path)
+        return partition_html(filename=self.file_path, **self.unstructured_kwargs)

langchain/document_loaders/ifixit.py

@@ -99,7 +99,10 @@ class IFixitLoader(BaseLoader):
         output.append("# " + title)
         output.append(soup.select_one(".post-content .post-text").text.strip())
 
-        output.append("\n## " + soup.find("div", "post-answers-header").text.strip())
+        answersHeader = soup.find("div", "post-answers-header")
+        if answersHeader:
+            output.append("\n## " + answersHeader.text.strip())
+
         for answer in soup.select(".js-answers-list .post.post-answer"):
             if answer.has_attr("itemprop") and "acceptedAnswer" in answer["itemprop"]:
                 output.append("\n### Accepted Answer")

langchain/document_loaders/image.py

@@ -10,4 +10,4 @@ class UnstructuredImageLoader(UnstructuredFileLoader):
     def _get_elements(self) -> List:
         from unstructured.partition.image import partition_image
 
-        return partition_image(filename=self.file_path)
+        return partition_image(filename=self.file_path, **self.unstructured_kwargs)

langchain/document_loaders/markdown.py

@@ -0,0 +1,25 @@
+"""Loader that loads Markdown files."""
+from typing import List
+
+from langchain.document_loaders.unstructured import UnstructuredFileLoader
+
+
+class UnstructuredMarkdownLoader(UnstructuredFileLoader):
+    """Loader that uses unstructured to load markdown files."""
+
+    def _get_elements(self) -> List:
+        from unstructured.__version__ import __version__ as __unstructured_version__
+        from unstructured.partition.md import partition_md
+
+        # NOTE(MthwRobinson) - enables the loader to work when you're using pre-release
+        # versions of unstructured like 0.4.17-dev1
+        _unstructured_version = __unstructured_version__.split("-")[0]
+        unstructured_version = tuple([int(x) for x in _unstructured_version.split(".")])
+
+        if unstructured_version < (0, 4, 16):
+            raise ValueError(
+                f"You are on unstructured version {__unstructured_version__}. "
+                "Partitioning markdown files is only supported in unstructured>=0.4.16."
+            )
+
+        return partition_md(filename=self.file_path)

langchain/document_loaders/online_pdf.py

@@ -1,30 +0,0 @@
-"""Loader that loads online PDF files."""
-
-import tempfile
-from pathlib import Path
-from typing import List
-
-import requests
-
-from langchain.docstore.document import Document
-from langchain.document_loaders.base import BaseLoader
-from langchain.document_loaders.pdf import UnstructuredPDFLoader
-
-
-class OnlinePDFLoader(BaseLoader):
-    """Loader that loads online PDFs."""
-
-    def __init__(self, web_path: str):
-        """Initialize with file path."""
-        self.web_path = web_path
-
-    def load(self) -> List[Document]:
-        """Load documents."""
-        r = requests.get(self.web_path)
-        with tempfile.TemporaryDirectory() as temp_dir:
-            file_path = Path(temp_dir) / "online_file.pdf"
-            file = open(file_path, "wb")
-            file.write(r.content)
-            file.close()
-            loader = UnstructuredPDFLoader(str(file_path))
-            return loader.load()

langchain/document_loaders/paged_pdf.py

@@ -1,36 +0,0 @@
-"""Loads a PDF with pypdf and chunks at character level."""
-from typing import List
-
-from langchain.docstore.document import Document
-from langchain.document_loaders.base import BaseLoader
-
-
-class PagedPDFSplitter(BaseLoader):
-    """Loads a PDF with pypdf and chunks at character level.
-
-    Loader also stores page numbers in metadatas.
-    """
-
-    def __init__(self, file_path: str):
-        """Initialize with file path."""
-        try:
-            import pypdf  # noqa:F401
-        except ImportError:
-            raise ValueError(
-                "pypdf package not found, please install it with " "`pip install pypdf`"
-            )
-        self._file_path = file_path
-
-    def load(self) -> List[Document]:
-        """Load given path as pages."""
-        import pypdf
-
-        with open(self._file_path, "rb") as pdf_file_obj:
-            pdf_reader = pypdf.PdfReader(pdf_file_obj)
-            return [
-                Document(
-                    page_content=page.extract_text(),
-                    metadata={"source": self._file_path, "page": i},
-                )
-                for i, page in enumerate(pdf_reader.pages)
-            ]

langchain/document_loaders/pdf.py

@@ -1,5 +1,11 @@
 """Loader that loads PDF files."""
+import os
+import tempfile
+from abc import ABC
 from typing import Any, List, Optional
+from urllib.parse import urlparse
+
+import requests
 
 from langchain.docstore.document import Document
 from langchain.document_loaders.base import BaseLoader
@@ -12,10 +18,94 @@ class UnstructuredPDFLoader(UnstructuredFileLoader):
     def _get_elements(self) -> List:
         from unstructured.partition.pdf import partition_pdf
 
-        return partition_pdf(filename=self.file_path)
+        return partition_pdf(filename=self.file_path, **self.unstructured_kwargs)
 
 
-class PDFMinerLoader(BaseLoader):
+class BasePDFLoader(BaseLoader, ABC):
+    """Base loader class for PDF files.
+
+    Defaults to check for local file, but if the file is a web path, it will download it
+    to a temporary file, and use that, then clean up the temporary file after completion
+    """
+
+    file_path: str
+    web_path: Optional[str] = None
+
+    def __init__(self, file_path: str):
+        """Initialize with file path."""
+        self.file_path = file_path
+        if "~" in self.file_path:
+            self.file_path = os.path.expanduser(self.file_path)
+
+        # If the file is a web path, download it to a temporary file, and use that
+        if not os.path.isfile(self.file_path) and self._is_valid_url(self.file_path):
+            r = requests.get(self.file_path)
+
+            if r.status_code != 200:
+                raise ValueError(
+                    "Check the url of your file; returned status code %s"
+                    % r.status_code
+                )
+
+            self.web_path = self.file_path
+            self.temp_file = tempfile.NamedTemporaryFile()
+            self.temp_file.write(r.content)
+            self.file_path = self.temp_file.name
+        elif not os.path.isfile(self.file_path):
+            raise ValueError("File path %s is not a valid file or url" % self.file_path)
+
+    def __del__(self) -> None:
+        if hasattr(self, "temp_file"):
+            self.temp_file.close()
+
+    @staticmethod
+    def _is_valid_url(url: str) -> bool:
+        """Check if the url is valid."""
+        parsed = urlparse(url)
+        return bool(parsed.netloc) and bool(parsed.scheme)
+
+
+class OnlinePDFLoader(BasePDFLoader):
+    """Loader that loads online PDFs."""
+
+    def load(self) -> List[Document]:
+        """Load documents."""
+        loader = UnstructuredPDFLoader(str(self.file_path))
+        return loader.load()
+
+
+class PyPDFLoader(BasePDFLoader):
+    """Loads a PDF with pypdf and chunks at character level.
+
+    Loader also stores page numbers in metadatas.
+    """
+
+    def __init__(self, file_path: str):
+        """Initialize with file path."""
+        try:
+            import pypdf  # noqa:F401
+        except ImportError:
+            raise ValueError(
+                "pypdf package not found, please install it with " "`pip install pypdf`"
+            )
+        super().__init__(file_path)
+
+    def load(self) -> List[Document]:
+        """Load given path as pages."""
+        import pypdf
+
+        with open(self.file_path, "rb") as pdf_file_obj:
+            pdf_reader = pypdf.PdfReader(pdf_file_obj)
+            return [
+                Document(
+                    page_content=page.extract_text(),
+                    metadata={"source": self.file_path, "page": i},
+                )
+                for i, page in enumerate(pdf_reader.pages)
+            ]
+
+
+class PDFMinerLoader(BasePDFLoader):
     """Loader that uses PDFMiner to load PDF files."""
 
     def __init__(self, file_path: str):
@@ -28,7 +118,7 @@ class PDFMinerLoader(BaseLoader):
                 "`pip install pdfminer.six`"
             )
 
-        self.file_path = file_path
+        super().__init__(file_path)
 
     def load(self) -> List[Document]:
         """Load file."""
@@ -39,7 +129,7 @@ class PDFMinerLoader(BaseLoader):
         return [Document(page_content=text, metadata=metadata)]
 
 
-class PyMuPDFLoader(BaseLoader):
+class PyMuPDFLoader(BasePDFLoader):
     """Loader that uses PyMuPDF to load PDF files."""
 
     def __init__(self, file_path: str):
@@ -52,22 +142,30 @@ class PyMuPDFLoader(BaseLoader):
                 "`pip install pymupdf`"
             )
 
-        self.file_path = file_path
+        super().__init__(file_path)
 
     def load(self, **kwargs: Optional[Any]) -> List[Document]:
         """Load file."""
         import fitz
 
         doc = fitz.open(self.file_path)  # open document
+        file_path = self.file_path if self.web_path is None else self.web_path
+
         return [
             Document(
                 page_content=page.get_text(**kwargs).encode("utf-8"),
-                metadata={
-                    "file_path": self.file_path,
-                    "page_number": page.number + 1,
-                    "total_pages": len(doc),
-                }
-                | doc.metadata,
+                metadata=dict(
+                    {
+                        "file_path": file_path,
+                        "page_number": page.number + 1,
+                        "total_pages": len(doc),
+                    },
+                    **{
+                        k: doc.metadata[k]
+                        for k in doc.metadata
+                        if type(doc.metadata[k]) in [str, int]
+                    }
+                ),
             )
             for page in doc
         ]

langchain/document_loaders/powerpoint.py

@@ -36,8 +36,8 @@ class UnstructuredPowerPointLoader(UnstructuredFileLoader):
         if is_ppt:
             from unstructured.partition.ppt import partition_ppt
 
-            return partition_ppt(filename=self.file_path)
+            return partition_ppt(filename=self.file_path, **self.unstructured_kwargs)
         else:
             from unstructured.partition.pptx import partition_pptx
 
-            return partition_pptx(filename=self.file_path)
+            return partition_pptx(filename=self.file_path, **self.unstructured_kwargs)

langchain/document_loaders/s3_file.py

@@ -1,4 +1,5 @@
 """Loading logic for loading documents from an s3 file."""
+import os
 import tempfile
 from typing import List
 
@@ -27,6 +28,7 @@ class S3FileLoader(BaseLoader):
         s3 = boto3.client("s3")
         with tempfile.TemporaryDirectory() as temp_dir:
             file_path = f"{temp_dir}/{self.key}"
+            os.makedirs(os.path.dirname(file_path), exist_ok=True)
             s3.download_file(self.bucket, self.key, file_path)
             loader = UnstructuredFileLoader(file_path)
             return loader.load()

langchain/document_loaders/unstructured.py

@@ -1,15 +1,32 @@
 """Loader that uses unstructured to load files."""
 from abc import ABC, abstractmethod
-from typing import IO, List
+from typing import IO, Any, List
 
 from langchain.docstore.document import Document
 from langchain.document_loaders.base import BaseLoader
 
 
+def satisfies_min_unstructured_version(min_version: str) -> bool:
+    """Checks to see if the installed unstructured version exceeds the minimum version
+    for the feature in question."""
+    from unstructured.__version__ import __version__ as __unstructured_version__
+
+    min_version_tuple = tuple([int(x) for x in min_version.split(".")])
+
+    # NOTE(MthwRobinson) - enables the loader to work when you're using pre-release
+    # versions of unstructured like 0.4.17-dev1
+    _unstructured_version = __unstructured_version__.split("-")[0]
+    unstructured_version_tuple = tuple(
+        [int(x) for x in _unstructured_version.split(".")]
+    )
+
+    return unstructured_version_tuple >= min_version_tuple
+
+
 class UnstructuredBaseLoader(BaseLoader, ABC):
     """Loader that uses unstructured to load files."""
 
-    def __init__(self, mode: str = "single"):
+    def __init__(self, mode: str = "single", **unstructured_kwargs: Any):
         """Initialize with file path."""
         try:
             import unstructured  # noqa:F401
@@ -25,6 +42,12 @@ class UnstructuredBaseLoader(BaseLoader, ABC):
             )
         self.mode = mode
 
+        if not satisfies_min_unstructured_version("0.5.4"):
+            if "strategy" in unstructured_kwargs:
+                unstructured_kwargs.pop("strategy")
+
+        self.unstructured_kwargs = unstructured_kwargs
+
     @abstractmethod
     def _get_elements(self) -> List:
         """Get elements."""
@@ -59,15 +82,17 @@ class UnstructuredBaseLoader(BaseLoader, ABC):
 class UnstructuredFileLoader(UnstructuredBaseLoader):
     """Loader that uses unstructured to load files."""
 
-    def __init__(self, file_path: str, mode: str = "single"):
+    def __init__(
+        self, file_path: str, mode: str = "single", **unstructured_kwargs: Any
+    ):
         """Initialize with file path."""
         self.file_path = file_path
-        super().__init__(mode=mode)
+        super().__init__(mode=mode, **unstructured_kwargs)
 
     def _get_elements(self) -> List:
         from unstructured.partition.auto import partition
 
-        return partition(filename=self.file_path)
+        return partition(filename=self.file_path, **self.unstructured_kwargs)
 
     def _get_metadata(self) -> dict:
         return {"source": self.file_path}
@@ -76,15 +101,15 @@ class UnstructuredFileLoader(UnstructuredBaseLoader):
 class UnstructuredFileIOLoader(UnstructuredBaseLoader):
     """Loader that uses unstructured to load file IO objects."""
 
-    def __init__(self, file: IO, mode: str = "single"):
+    def __init__(self, file: IO, mode: str = "single", **unstructured_kwargs: Any):
         """Initialize with file path."""
         self.file = file
-        super().__init__(mode=mode)
+        super().__init__(mode=mode, **unstructured_kwargs)
 
     def _get_elements(self) -> List:
         from unstructured.partition.auto import partition
 
-        return partition(file=self.file)
+        return partition(file=self.file, **self.unstructured_kwargs)
 
     def _get_metadata(self) -> dict:
         return {}

langchain/document_loaders/url.py

@@ -1,4 +1,4 @@
-"""Loader that loads PDF files."""
+"""Loader that uses unstructured to load HTML files."""
 from typing import List
 
 from langchain.docstore.document import Document

langchain/document_loaders/word_document.py

@@ -36,8 +36,8 @@ class UnstructuredWordDocumentLoader(UnstructuredFileLoader):
         if is_doc:
             from unstructured.partition.doc import partition_doc
 
-            return partition_doc(filename=self.file_path)
+            return partition_doc(filename=self.file_path, **self.unstructured_kwargs)
         else:
             from unstructured.partition.docx import partition_docx
 
-            return partition_docx(filename=self.file_path)
+            return partition_docx(filename=self.file_path, **self.unstructured_kwargs)

langchain/document_loaders/youtube.py

@@ -1,11 +1,98 @@
 """Loader that loads YouTube transcript."""
 from __future__ import annotations
 
-from typing import Any, List
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from pydantic import root_validator
+from pydantic.dataclasses import dataclass
 
 from langchain.docstore.document import Document
 from langchain.document_loaders.base import BaseLoader
 
+SCOPES = ["https://www.googleapis.com/auth/drive.readonly"]
+
+
+@dataclass
+class GoogleApiClient:
+    """A Generic Google Api Client.
+
+    To use, you should have the ``google_auth_oauthlib,youtube_transcript_api,google``
+    python package installed.
+    As the google api expects credentials you need to set up a google account and
+    register your Service. "https://developers.google.com/docs/api/quickstart/python"
+
+
+
+    Example:
+        .. code-block:: python
+
+            from langchain.document_loaders import GoogleApiClient
+            google_api_client = GoogleApiClient(
+                service_account_path=Path("path_to_your_sec_file.json")
+            )
+
+    """
+
+    credentials_path: Path = Path.home() / ".credentials" / "credentials.json"
+    service_account_path: Path = Path.home() / ".credentials" / "credentials.json"
+    token_path: Path = Path.home() / ".credentials" / "token.json"
+
+    def __post_init__(self) -> None:
+        self.creds = self._load_credentials()
+
+    @root_validator
+    def validate_channel_or_videoIds_is_set(
+        cls, values: Dict[str, Any]
+    ) -> Dict[str, Any]:
+        """Validate that either folder_id or document_ids is set, but not both."""
+
+        if not values.get("credentials_path") and not values.get(
+            "service_account_path"
+        ):
+            raise ValueError("Must specify either channel_name or video_ids")
+        return values
+
+    def _load_credentials(self) -> Any:
+        """Load credentials."""
+        # Adapted from https://developers.google.com/drive/api/v3/quickstart/python
+        try:
+            from google.auth.transport.requests import Request
+            from google.oauth2 import service_account
+            from google.oauth2.credentials import Credentials
+            from google_auth_oauthlib.flow import InstalledAppFlow
+            from youtube_transcript_api import YouTubeTranscriptApi  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "You must run"
+                "`pip install --upgrade "
+                "google-api-python-client google-auth-httplib2 "
+                "google-auth-oauthlib"
+                "youtube-transcript-api`"
+                "to use the Google Drive loader"
+            )
+
+        creds = None
+        if self.service_account_path.exists():
+            return service_account.Credentials.from_service_account_file(
+                str(self.service_account_path)
+            )
+        if self.token_path.exists():
+            creds = Credentials.from_authorized_user_file(str(self.token_path), SCOPES)
+
+        if not creds or not creds.valid:
+            if creds and creds.expired and creds.refresh_token:
+                creds.refresh(Request())
+            else:
+                flow = InstalledAppFlow.from_client_secrets_file(
+                    str(self.credentials_path), SCOPES
+                )
+                creds = flow.run_local_server(port=0)
+            with open(self.token_path, "w") as token:
+                token.write(creds.to_json())
+
+        return creds
+
 
 class YoutubeLoader(BaseLoader):
     """Loader that loads Youtube transcripts."""
@@ -19,8 +106,8 @@ class YoutubeLoader(BaseLoader):
         self.language = language
 
     @classmethod
-    def from_youtube_url(cls, youtube_url: str, **kwargs: Any) -> YoutubeLoader:
-        """Parse out video id from YouTube url."""
+    def from_youtube_channel(cls, youtube_url: str, **kwargs: Any) -> YoutubeLoader:
+        """Given a channel name, load all videos."""
         video_id = youtube_url.split("youtube.com/watch?v=")[-1]
         return cls(video_id, **kwargs)
 
@@ -43,7 +130,7 @@ class YoutubeLoader(BaseLoader):
             metadata.update(video_info)
 
         transcript_pieces = YouTubeTranscriptApi.get_transcript(
-            self.video_id, languages=(self.language,)
+            self.video_id, languages=[self.language]
         )
         transcript = " ".join([t["text"].strip(" ") for t in transcript_pieces])
 
@@ -79,3 +166,147 @@ class YoutubeLoader(BaseLoader):
             "author": yt.author,
         }
         return video_info
+
+
+@dataclass
+class GoogleApiYoutubeLoader(BaseLoader):
+    """Loader that loads all Videos from a Channel
+
+    To use, you should have the ``googleapiclient,youtube_transcript_api``
+    python package installed.
+    As the service needs a google_api_client, you first have to initialize
+    the GoogleApiClient.
+
+    Additonali you have to either provide a channel name or a list of videoids
+    "https://developers.google.com/docs/api/quickstart/python"
+
+
+
+    Example:
+        .. code-block:: python
+
+            from langchain.document_loaders import GoogleApiClient
+            from langchain.document_loaders import GoogleApiYoutubeLoader
+            google_api_client = GoogleApiClient(
+                service_account_path=Path("path_to_your_sec_file.json")
+            )
+            loader = GoogleApiYoutubeLoader(
+                google_api_client=google_api_client,
+                channel_name = "CodeAesthetic"
+            )
+            load.load()
+
+    """
+
+    google_api_client: GoogleApiClient
+    channel_name: Optional[str] = None
+    video_ids: Optional[List[str]] = None
+    add_video_info: bool = True
+    captions_language: str = "en"
+
+    def __post_init__(self) -> None:
+        self.youtube_client = self._build_youtube_client(self.google_api_client.creds)
+
+    def _build_youtube_client(self, creds: Any) -> Any:
+        try:
+            from googleapiclient.discovery import build
+            from youtube_transcript_api import YouTubeTranscriptApi  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "You must run"
+                "`pip install --upgrade "
+                "google-api-python-client google-auth-httplib2 "
+                "google-auth-oauthlib"
+                "youtube-transcript-api`"
+                "to use the Google Drive loader"
+            )
+
+        return build("youtube", "v3", credentials=creds)
+
+    @root_validator
+    def validate_channel_or_videoIds_is_set(
+        cls, values: Dict[str, Any]
+    ) -> Dict[str, Any]:
+        """Validate that either folder_id or document_ids is set, but not both."""
+        if not values.get("channel_name") and not values.get("video_ids"):
+            raise ValueError("Must specify either channel_name or video_ids")
+        return values
+
+    def _get_transcripe_for_video_id(self, video_id: str) -> str:
+        from youtube_transcript_api import YouTubeTranscriptApi
+
+        transcript_pieces = YouTubeTranscriptApi.get_transcript(video_id)
+        return " ".join([t["text"].strip(" ") for t in transcript_pieces])
+
+    def _get_document_for_video_id(self, video_id: str, **kwargs: Any) -> Document:
+        captions = self._get_transcripe_for_video_id(video_id)
+        video_response = (
+            self.youtube_client.videos()
+            .list(
+                part="id,snippet",
+                id=video_id,
+            )
+            .execute()
+        )
+        return Document(
+            page_content=captions,
+            metadata=video_response.get("items")[0],
+        )
+
+    def _get_channel_id(self, channel_name: str) -> str:
+        request = self.youtube_client.search().list(
+            part="id",
+            q=channel_name,
+            type="channel",
+            maxResults=1,  # we only need one result since channel names are unique
+        )
+        response = request.execute()
+        channel_id = response["items"][0]["id"]["channelId"]
+        return channel_id
+
+    def _get_document_for_channel(self, channel: str, **kwargs: Any) -> List[Document]:
+        channel_id = self._get_channel_id(channel)
+        request = self.youtube_client.search().list(
+            part="id,snippet",
+            channelId=channel_id,
+            maxResults=50,  # adjust this value to retrieve more or fewer videos
+        )
+        video_ids = []
+        while request is not None:
+            response = request.execute()
+
+            # Add each video ID to the list
+            for item in response["items"]:
+                if not item["id"].get("videoId"):
+                    continue
+                meta_data = {"videoId": item["id"]["videoId"]}
+                if self.add_video_info:
+                    item["snippet"].pop("thumbnails")
+                    meta_data.update(item["snippet"])
+                video_ids.append(
+                    Document(
+                        page_content=self._get_transcripe_for_video_id(
+                            item["id"]["videoId"]
+                        ),
+                        metadata=meta_data,
+                    )
+                )
+            request = self.youtube_client.search().list_next(request, response)
+
+        return video_ids
+
+    def load(self) -> List[Document]:
+        """Load documents."""
+        document_list = []
+        if self.channel_name:
+            document_list.extend(self._get_document_for_channel(self.channel_name))
+        elif self.video_ids:
+            document_list.extend(
+                [
+                    self._get_document_for_video_id(video_id)
+                    for video_id in self.video_ids
+                ]
+            )
+        else:
+            raise ValueError("Must specify either channel_name or video_ids")
+        return document_list

langchain/embeddings/openai.py

@@ -1,12 +1,57 @@
 """Wrapper around OpenAI embedding models."""
-from typing import Any, Dict, List, Optional
+from __future__ import annotations
+
+import logging
+from typing import Any, Callable, Dict, List, Optional
 
 import numpy as np
 from pydantic import BaseModel, Extra, root_validator
+from tenacity import (
+    before_sleep_log,
+    retry,
+    retry_if_exception_type,
+    stop_after_attempt,
+    wait_exponential,
+)
 
 from langchain.embeddings.base import Embeddings
 from langchain.utils import get_from_dict_or_env
 
+logger = logging.getLogger(__name__)
+
+
+def _create_retry_decorator(embeddings: OpenAIEmbeddings) -> Callable[[Any], Any]:
+    import openai
+
+    min_seconds = 4
+    max_seconds = 10
+    # Wait 2^x * 1 second between each retry starting with
+    # 4 seconds, then up to 10 seconds, then 10 seconds afterwards
+    return retry(
+        reraise=True,
+        stop=stop_after_attempt(embeddings.max_retries),
+        wait=wait_exponential(multiplier=1, min=min_seconds, max=max_seconds),
+        retry=(
+            retry_if_exception_type(openai.error.Timeout)
+            | retry_if_exception_type(openai.error.APIError)
+            | retry_if_exception_type(openai.error.APIConnectionError)
+            | retry_if_exception_type(openai.error.RateLimitError)
+            | retry_if_exception_type(openai.error.ServiceUnavailableError)
+        ),
+        before_sleep=before_sleep_log(logger, logging.WARNING),
+    )
+
+
+def embed_with_retry(embeddings: OpenAIEmbeddings, **kwargs: Any) -> Any:
+    """Use tenacity to retry the completion call."""
+    retry_decorator = _create_retry_decorator(embeddings)
+
+    @retry_decorator
+    def _completion_with_retry(**kwargs: Any) -> Any:
+        return embeddings.client.create(**kwargs)
+
+    return _completion_with_retry(**kwargs)
+
 
 class OpenAIEmbeddings(BaseModel, Embeddings):
     """Wrapper around OpenAI embedding models.
@@ -27,6 +72,10 @@ class OpenAIEmbeddings(BaseModel, Embeddings):
     query_model_name: str = "text-embedding-ada-002"
     embedding_ctx_length: int = -1
     openai_api_key: Optional[str] = None
+    chunk_size: int = 1000
+    """Maximum number of texts to embed in each batch"""
+    max_retries: int = 6
+    """Maximum number of retries to make when generating."""
 
     class Config:
         """Configuration for this pydantic object."""
@@ -74,7 +123,7 @@ class OpenAIEmbeddings(BaseModel, Embeddings):
     # please refer to
     # https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb
     def _get_len_safe_embeddings(
-        self, texts: List[str], *, engine: str, chunk_size: int = 1000
+        self, texts: List[str], *, engine: str, chunk_size: Optional[int] = None
     ) -> List[List[float]]:
         embeddings: List[List[float]] = [[] for i in range(len(texts))]
         try:
@@ -92,9 +141,12 @@ class OpenAIEmbeddings(BaseModel, Embeddings):
                     indices += [i]
 
             batched_embeddings = []
-            for i in range(0, len(tokens), chunk_size):
-                response = self.client.create(
-                    input=tokens[i : i + chunk_size], engine=self.document_model_name
+            _chunk_size = chunk_size or self.chunk_size
+            for i in range(0, len(tokens), _chunk_size):
+                response = embed_with_retry(
+                    self,
+                    input=tokens[i : i + _chunk_size],
+                    engine=self.document_model_name,
                 )
                 batched_embeddings += [r["embedding"] for r in response["data"]]
 
@@ -124,33 +176,34 @@ class OpenAIEmbeddings(BaseModel, Embeddings):
             return self._get_len_safe_embeddings([text], engine=engine)[0]
         else:
             text = text.replace("\n", " ")
-            return self.client.create(input=[text], engine=engine)["data"][0][
+            return embed_with_retry(self, input=[text], engine=engine)["data"][0][
                 "embedding"
             ]
 
     def embed_documents(
-        self, texts: List[str], chunk_size: int = 1000
+        self, texts: List[str], chunk_size: Optional[int] = 0
     ) -> List[List[float]]:
         """Call out to OpenAI's embedding endpoint for embedding search docs.
 
         Args:
             texts: The list of texts to embed.
-            chunk_size: The maximum number of texts to send to OpenAI at once
-                (max 1000).
+            chunk_size: The chunk size of embeddings. If None, will use the chunk size
+                specified by the class.
 
         Returns:
             List of embeddings, one for each text.
         """
         # handle large batches of texts
         if self.embedding_ctx_length > 0:
-            return self._get_len_safe_embeddings(
-                texts, engine=self.document_model_name, chunk_size=chunk_size
-            )
+            return self._get_len_safe_embeddings(texts, engine=self.document_model_name)
         else:
             results = []
-            for i in range(0, len(texts), chunk_size):
-                response = self.client.create(
-                    input=texts[i : i + chunk_size], engine=self.document_model_name
+            _chunk_size = chunk_size or self.chunk_size
+            for i in range(0, len(texts), _chunk_size):
+                response = embed_with_retry(
+                    self,
+                    input=texts[i : i + _chunk_size],
+                    engine=self.document_model_name,
                 )
                 results += [r["embedding"] for r in response["data"]]
             return results

langchain/evaluation/loading.py

@@ -0,0 +1,8 @@
+from typing import Dict, List
+
+
+def load_dataset(uri: str) -> List[Dict]:
+    from datasets import load_dataset
+
+    dataset = load_dataset(f"LangChainDatasets/{uri}")
+    return [d for d in dataset["train"]]

langchain/evaluation/qa/generate_prompt.py

@@ -1,6 +1,6 @@
 # flake8: noqa
 from langchain.prompts import PromptTemplate
-from langchain.prompts.base import RegexParser
+from langchain.output_parsers.regex import RegexParser
 
 template = """You are a teacher coming up with questions to ask on a quiz. 
 Given the following document, please generate a question and answer based on that document.

langchain/graphs/networkx_graph.py

@@ -1,6 +1,7 @@
 """Networkx wrapper for graph operations."""
+from __future__ import annotations
 
-from typing import List, NamedTuple, Tuple
+from typing import Any, List, NamedTuple, Optional, Tuple
 
 KG_TRIPLE_DELIMITER = "<|>"
 
@@ -48,7 +49,7 @@ def get_entities(entity_str: str) -> List[str]:
 class NetworkxEntityGraph:
     """Networkx wrapper for entity graph operations."""
 
-    def __init__(self) -> None:
+    def __init__(self, graph: Optional[Any] = None) -> None:
         """Create a new graph."""
         try:
             import networkx as nx
@@ -57,8 +58,24 @@ class NetworkxEntityGraph:
                 "Could not import networkx python package. "
                 "Please it install it with `pip install networkx`."
             )
+        if graph is not None:
+            if not isinstance(graph, nx.DiGraph):
+                raise ValueError("Passed in graph is not of correct shape")
+            self._graph = graph
+        else:
+            self._graph = nx.DiGraph()
 
-        self._graph = nx.DiGraph()
+    @classmethod
+    def from_gml(cls, gml_path: str) -> NetworkxEntityGraph:
+        try:
+            import networkx as nx
+        except ImportError:
+            raise ValueError(
+                "Could not import networkx python package. "
+                "Please it install it with `pip install networkx`."
+            )
+        graph = nx.read_gml(gml_path)
+        return cls(graph)
 
     def add_triple(self, knowledge_triple: KnowledgeTriple) -> None:
         """Add a triple to the graph."""
@@ -97,6 +114,11 @@ class NetworkxEntityGraph:
             results.append(f"{src} {relation} {sink}")
         return results
 
+    def write_to_gml(self, path: str) -> None:
+        import networkx as nx
+
+        nx.write_gml(self._graph, path)
+
     def clear(self) -> None:
         """Clear the graph."""
         self._graph.clear()

langchain/indexes/vectorstore.py

@@ -52,6 +52,7 @@ class VectorstoreIndexCreator(BaseModel):
     vectorstore_cls: Type[VectorStore] = Chroma
     embedding: Embeddings = Field(default_factory=OpenAIEmbeddings)
     text_splitter: TextSplitter = Field(default_factory=_get_default_text_splitter)
+    vectorstore_kwargs: dict = Field(default_factory=dict)
 
     class Config:
         """Configuration for this pydantic object."""
@@ -65,5 +66,7 @@ class VectorstoreIndexCreator(BaseModel):
         for loader in loaders:
             docs.extend(loader.load())
         sub_docs = self.text_splitter.split_documents(docs)
-        vectorstore = self.vectorstore_cls.from_documents(sub_docs, self.embedding)
+        vectorstore = self.vectorstore_cls.from_documents(
+            sub_docs, self.embedding, **self.vectorstore_kwargs
+        )
         return VectorStoreIndexWrapper(vectorstore=vectorstore)

langchain/llms/__init__.py

@@ -18,7 +18,7 @@ from langchain.llms.modal import Modal
 from langchain.llms.nlpcloud import NLPCloud
 from langchain.llms.openai import AzureOpenAI, OpenAI, OpenAIChat
 from langchain.llms.petals import Petals
-from langchain.llms.promptlayer_openai import PromptLayerOpenAI
+from langchain.llms.promptlayer_openai import PromptLayerOpenAI, PromptLayerOpenAIChat
 from langchain.llms.self_hosted import SelfHostedPipeline
 from langchain.llms.self_hosted_hugging_face import SelfHostedHuggingFaceLLM
 from langchain.llms.stochasticai import StochasticAI
@@ -46,6 +46,7 @@ __all__ = [
     "SelfHostedPipeline",
     "SelfHostedHuggingFaceLLM",
     "PromptLayerOpenAI",
+    "PromptLayerOpenAIChat",
     "StochasticAI",
     "Writer",
 ]