add notebook

Co-authored-by: Stefan Keselj <skeselj@princeton.edu>
Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>

.coveragerc

@@ -1,2 +0,0 @@
-[run]
-omit = tests/*

CITATION.cff

@@ -0,0 +1,8 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+- family-names: "Chase"
+  given-names: "Harrison"
+title: "LangChain"
+date-released: 2022-10-17
+url: "https://github.com/hwchase17/langchain"

CONTRIBUTING.md

@@ -47,7 +47,7 @@ good code into the codebase.
 ### 🏭Release process
 
 As of now, LangChain has an ad hoc release process: releases are cut with high frequency via by
-a developer and published to [PyPI](https://pypi.org/project/ruff/).
+a developer and published to [PyPI](https://pypi.org/project/langchain/).
 
 LangChain follows the [semver](https://semver.org/) versioning standard. However, as pre-1.0 software,
 even patch releases may contain [non-backwards-compatible changes](https://semver.org/#spec-item-4).

README.md

@@ -4,6 +4,9 @@
 
 [![lint](https://github.com/hwchase17/langchain/actions/workflows/lint.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/lint.yml) [![test](https://github.com/hwchase17/langchain/actions/workflows/test.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/test.yml) [![linkcheck](https://github.com/hwchase17/langchain/actions/workflows/linkcheck.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/linkcheck.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40LangChainAI)](https://twitter.com/langchainai) [![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true&style=flat)](https://discord.gg/6adMQxSpJS)
 
+**Production Support:** As you move your LangChains into production, we'd love to offer more comprehensive support.
+Please fill out [this form](https://forms.gle/57d8AmXBYp8PP8tZA) and we'll set up a dedicated support Slack channel.
+
 ## Quick Install
 
 `pip install langchain`
@@ -15,7 +18,22 @@ developers to build applications that they previously could not.
 But using these LLMs in isolation is often not enough to
 create a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.
 
-This library is aimed at assisting in the development of those types of applications.
+This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:
+
+**❓ Question Answering over specific documents**
+
+- [Documentation](https://langchain.readthedocs.io/en/latest/use_cases/question_answering.html)
+- End-to-end Example: [Question Answering over Notion Database](https://github.com/hwchase17/notion-qa)
+
+**💬 Chatbots**
+
+- [Documentation](https://langchain.readthedocs.io/en/latest/use_cases/chatbots.html)
+- End-to-end Example: [Chat-LangChain](https://github.com/hwchase17/chat-langchain)
+
+**🤖 Agents**
+
+- [Documentation](https://langchain.readthedocs.io/en/latest/use_cases/agents.html)
+- End-to-end Example: [GPT+WolframAlpha](https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain)
 
 ## 📖 Documentation
 

docs/deployments.md

@@ -22,3 +22,18 @@ This repo serves as a template for how deploy a LangChain with Gradio.
 It implements a chatbot interface, with a "Bring-Your-Own-Token" approach (nice for not wracking up big bills).
 It also contains instructions for how to deploy this app on the Hugging Face platform.
 This is heavily influenced by James Weaver's [excellent examples](https://huggingface.co/JavaFXpert).
+
+## [Beam](https://github.com/slai-labs/get-beam/tree/main/examples/langchain-question-answering)
+
+This repo serves as a template for how deploy a LangChain with [Beam](https://beam.cloud).
+
+It implements a Question Answering app and contains instructions for deploying the app as a serverless REST API.
+
+## [Vercel](https://github.com/homanp/vercel-langchain)
+
+A minimal example on how to run LangChain on Vercel using Flask.
+
+
+## [SteamShip](https://github.com/steamship-core/steamship-langchain/)
+This repository contains LangChain adapters for Steamship, enabling LangChain developers to rapidly deploy their apps on Steamship.
+This includes: production ready endpoints, horizontal scaling across dependencies, persistant storage of app state, multi-tenancy support, etc.

docs/ecosystem/google_search.md

@@ -1,7 +1,7 @@
 # Google Search Wrapper
 
 This page covers how to use the Google Search API within LangChain.
-It is broken into two parts: installation and setup, and then references to specific Pinecone wrappers.
+It is broken into two parts: installation and setup, and then references to the specific Google Search wrapper.
 
 ## Installation and Setup
 - Install requirements with `pip install google-api-python-client`

docs/ecosystem/serpapi.md

@@ -1,7 +1,7 @@
 # SerpAPI
 
 This page covers how to use the SerpAPI search APIs within LangChain.
-It is broken into two parts: installation and setup, and then references to specific Pinecone wrappers.
+It is broken into two parts: installation and setup, and then references to the specific SerpAPI wrapper.
 
 ## Installation and Setup
 - Install requirements with `pip install google-search-results`

docs/gallery.rst

@@ -77,6 +77,17 @@ Open Source
 
     +++
 
+    A jupyter notebook demonstrating how you could create a semantic search engine on documents in one of your Google Folders
+
+    ---
+
+    .. link-button:: https://github.com/venuv/langchain_semantic_search
+        :type: url
+        :text: Google Folder Semantic Search
+        :classes: stretched-link btn-lg
+
+    +++
+
     Build a GitHub support bot with GPT3, LangChain, and Python.
 
     ---
@@ -188,6 +199,17 @@ Open Source
     +++
 
     This repo is a simple demonstration of using LangChain to do fact-checking with prompt chaining.
+    
+    ---
+
+    .. link-button:: https://github.com/arc53/docsgpt
+        :type: url
+        :text: DocsGPT
+        :classes: stretched-link btn-lg
+    
+    +++
+
+    Answer questions about the documentation of any project    
 
 Misc. Colab Notebooks
 ~~~~~~~~~~~~~~~

docs/getting_started/getting_started.md

@@ -162,7 +162,7 @@ This is one of the simpler types of chains, but understanding how it works will
 
 `````{dropdown} Agents: Dynamically call chains based on user input
 
-So for the chains we've looked at run in a predetermined order.
+So far the chains we've looked at run in a predetermined order.
 
 Agents no longer do: they use an LLM to determine which actions to take and in what order. An action can either be using a tool and observing its output, or returning to the user.
 
@@ -179,6 +179,20 @@ In order to load agents, you should understand the following concepts:
 
 **Tools**: For a list of predefined tools and their specifications, see [here](../modules/agents/tools.md).
 
+For this example, you will also need to install the SerpAPI Python package.
+
+```bash
+pip install google-search-results
+```
+
+And set the appropriate environment variables.
+
+```python
+import os
+os.environ["SERPAPI_API_KEY"] = "..."
+```
+
+Now we can get started!
 
 ```python
 from langchain.agents import load_tools

docs/index.rst

@@ -7,7 +7,22 @@ But using these LLMs in isolation is often not enough to
 create a truly powerful app - the real power comes when you are able to
 combine them with other sources of computation or knowledge.
 
-This library is aimed at assisting in the development of those types of applications.
+This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:
+
+**❓ Question Answering over specific documents**
+
+- `Documentation <./use_cases/question_answering.html>`_
+- End-to-end Example: `Question Answering over Notion Database <https://github.com/hwchase17/notion-qa>`_
+
+**💬 Chatbots**
+
+- `Documentation <./use_cases/chatbots.html>`_
+- End-to-end Example: `Chat-LangChain <https://github.com/hwchase17/chat-langchain>`_
+
+**🤖 Agents**
+
+- `Documentation <./use_cases/agents.html>`_
+- End-to-end Example: `GPT+WolframAlpha <https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain>`_
 
 Getting Started
 ----------------
@@ -36,6 +51,8 @@ These modules are, in increasing order of complexity:
 
 - `LLMs <./modules/llms.html>`_: This includes a generic interface for all LLMs, and common utilities for working with LLMs.
 
+- `Document Loaders <./modules/document_loaders.html>`_: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.
+
 - `Utils <./modules/utils.html>`_: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.
 
 - `Chains <./modules/chains.html>`_: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.
@@ -53,6 +70,7 @@ These modules are, in increasing order of complexity:
 
    ./modules/prompts.md
    ./modules/llms.md
+   ./modules/document_loaders.md
    ./modules/utils.md
    ./modules/chains.md
    ./modules/agents.md
@@ -137,6 +155,8 @@ Additional Resources
 
 Additional collection of resources we think may be useful as you develop your application!
 
+- `LangChainHub <https://github.com/hwchase17/langchain-hub>`_: The LangChainHub is a place to share and explore other prompts, chains, and agents.
+
 - `Glossary <./glossary.html>`_: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not!
 
 - `Gallery <./gallery.html>`_: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications.
@@ -145,6 +165,10 @@ Additional collection of resources we think may be useful as you develop your ap
 
 - `Discord <https://discord.gg/6adMQxSpJS>`_: Join us on our Discord to discuss all things LangChain!
 
+- `Tracing <./tracing.html>`_: A guide on using tracing in LangChain to visualize the execution of chains and agents.
+
+- `Production Support <https://forms.gle/57d8AmXBYp8PP8tZA>`_: As you move your LangChains into production, we'd love to offer more comprehensive support. Please fill out this form and we'll set up a dedicated support Slack channel.
+
 
 .. toctree::
    :maxdepth: 1
@@ -152,6 +176,10 @@ Additional collection of resources we think may be useful as you develop your ap
    :name: resources
    :hidden:
 
+   LangChainHub <https://github.com/hwchase17/langchain-hub>
    ./glossary.md
    ./gallery.rst
    ./deployments.md
+   ./tracing.md
+   Discord <https://discord.gg/6adMQxSpJS>
+   Production Support <https://forms.gle/57d8AmXBYp8PP8tZA>

docs/modules/agents/examples/async_agent.ipynb

@@ -0,0 +1,423 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "6fb92deb-d89e-439b-855d-c7f2607d794b",
+   "metadata": {},
+   "source": [
+    "# Async API for Agent\n",
+    "\n",
+    "LangChain provides async support for Agents by leveraging the [asyncio](https://docs.python.org/3/library/asyncio.html) library.\n",
+    "\n",
+    "Async methods are currently supported for the following `Tools`: [`SerpAPIWrapper`](https://github.com/hwchase17/langchain/blob/master/langchain/serpapi.py) and [`LLMMathChain`](https://github.com/hwchase17/langchain/blob/master/langchain/chains/llm_math/base.py). Async support for other agent tools are on the roadmap.\n",
+    "\n",
+    "For `Tool`s that have a `coroutine` implemented (the two mentioned above), the `AgentExecutor` will `await` them directly. Otherwise, the `AgentExecutor` will call the `Tool`'s `func` via `asyncio.get_event_loop().run_in_executor` to avoid blocking the main runloop.\n",
+    "\n",
+    "You can use `arun` to call an `AgentExecutor` asynchronously."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97800378-cc34-4283-9bd0-43f336bc914c",
+   "metadata": {},
+   "source": [
+    "## Serial vs. Concurrent Execution\n",
+    "\n",
+    "In this example, we kick off agents to answer some questions serially vs. concurrently. You can see that concurrent execution significantly speeds this up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "da5df06c-af6f-4572-b9f5-0ab971c16487",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "import time\n",
+    "\n",
+    "from langchain.agents import initialize_agent, load_tools\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.callbacks.stdout import StdOutCallbackHandler\n",
+    "from langchain.callbacks.base import CallbackManager\n",
+    "from langchain.callbacks.tracers import LangChainTracer\n",
+    "from aiohttp import ClientSession\n",
+    "\n",
+    "questions = [\n",
+    "    \"Who won the US Open men's final in 2019? What is his age raised to the 0.334 power?\",\n",
+    "    \"Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?\",\n",
+    "    \"Who won the most recent formula 1 grand prix? What is their age raised to the 0.23 power?\",\n",
+    "    \"Who won the US Open women's final in 2019? What is her age raised to the 0.34 power?\",\n",
+    "    \"Who is Beyonce's husband? What is his age raised to the 0.19 power?\"\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "fd4c294e-b1d6-44b8-b32e-2765c017e503",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who won the US Open men's final in 2019 and then calculate his age raised to the 0.334 power.\n",
+      "Action: Search\n",
+      "Action Input: \"US Open men's final 2019 winner\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mRafael Nadal\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Rafael Nadal's age\n",
+      "Action: Search\n",
+      "Action Input: \"Rafael Nadal age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m36 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 36 raised to the 0.334 power\n",
+      "Action: Calculator\n",
+      "Action Input: 36^0.334\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 3.3098250249682484\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Rafael Nadal, aged 36, won the US Open men's final in 2019 and his age raised to the 0.334 power is 3.3098250249682484.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who Olivia Wilde's boyfriend is and then calculate his age raised to the 0.23 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Olivia Wilde boyfriend\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mJason Sudeikis\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Jason Sudeikis' age\n",
+      "Action: Search\n",
+      "Action Input: \"Jason Sudeikis age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mDaniel Jason Sudeikis is an American actor, comedian, writer, and producer. In the 1990s, he began his career in improv comedy and performed with ComedySportz, iO Chicago, and The Second City.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Jason Sudeikis' exact age\n",
+      "Action: Search\n",
+      "Action Input: \"Jason Sudeikis age exact\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mDaniel Jason Sudeikis. (1975-09-18) September 18, 1975 (age 47). Fairfax, Virginia, U.S. · Fort Scott Community College · Actor; comedian; producer; writer · 1997– ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now have the information I need to calculate the age raised to the 0.23 power\n",
+      "Action: Calculator\n",
+      "Action Input: 47^0.23\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.4242784855673896\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Jason Sudeikis, Olivia Wilde's boyfriend, is 47 years old and his age raised to the 0.23 power is 2.4242784855673896.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who won the grand prix and then calculate their age raised to the 0.23 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Formula 1 Grand Prix Winner\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mMax Emilian Verstappen is a Belgian-Dutch racing driver and the 2021 and 2022 Formula One World Champion. He competes under the Dutch flag in Formula One with Red Bull Racing. Verstappen is the son of racing drivers Jos Verstappen, who also competed in Formula One, and Sophie Kumpen.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Max Emilian Verstappen's age.\n",
+      "Action: Search\n",
+      "Action Input: \"Max Emilian Verstappen age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m25 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now need to calculate 25 raised to the 0.23 power.\n",
+      "Action: Calculator\n",
+      "Action Input: 25^0.23\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.096651272316035\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: Max Emilian Verstappen, who is 25 years old, won the most recent Formula 1 Grand Prix and his age raised to the 0.23 power is 2.096651272316035.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who won the US Open women's final in 2019 and then calculate her age raised to the 0.34 power.\n",
+      "Action: Search\n",
+      "Action Input: \"US Open women's final 2019 winner\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mBianca Andreescu defeated Serena Williams in the final, 6–3, 7–5 to win the women's singles tennis title at the 2019 US Open. It was her first major title, and she became the first Canadian, as well as the first player born in the 2000s, to win a major singles title.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Bianca Andreescu's age.\n",
+      "Action: Search\n",
+      "Action Input: \"Bianca Andreescu age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mBianca Vanessa Andreescu is a Canadian-Romanian professional tennis player. She has a career-high ranking of No. 4 in the world, and is the highest-ranked Canadian in the history of the Women's Tennis Association.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the age of Bianca Andreescu.\n",
+      "Action: Calculator\n",
+      "Action Input: 19^0.34\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.7212987634680084\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: Bianca Andreescu, aged 19, won the US Open women's final in 2019. Her age raised to the 0.34 power is 2.7212987634680084.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who Beyonce's husband is and then calculate his age raised to the 0.19 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Who is Beyonce's husband?\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mJay-Z\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Jay-Z's age\n",
+      "Action: Search\n",
+      "Action Input: \"How old is Jay-Z?\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m53 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 53 raised to the 0.19 power\n",
+      "Action: Calculator\n",
+      "Action Input: 53^0.19\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.12624064206896\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Jay-Z is Beyonce's husband and his age raised to the 0.19 power is 2.12624064206896.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "Serial executed in 94.83 seconds.\n"
+     ]
+    }
+   ],
+   "source": [
+    "def generate_serially():\n",
+    "    for q in questions:\n",
+    "        llm = OpenAI(temperature=0)\n",
+    "        tools = load_tools([\"llm-math\", \"serpapi\"], llm=llm)\n",
+    "        agent = initialize_agent(\n",
+    "            tools, llm, agent=\"zero-shot-react-description\", verbose=True\n",
+    "        )\n",
+    "        agent.run(q)\n",
+    "\n",
+    "s = time.perf_counter()\n",
+    "generate_serially()\n",
+    "elapsed = time.perf_counter() - s\n",
+    "print(f\"Serial executed in {elapsed:0.2f} seconds.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "076d7b85-45ec-465d-8b31-c2ad119c3438",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[33;1m\u001b[1;3m I need to find out who Beyonce's husband is and then calculate his age raised to the 0.19 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Who is Beyonce's husband?\"\u001b[0m\u001b[31;1m\u001b[1;3m I need to find out who won the grand prix and then calculate their age raised to the 0.23 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Formula 1 Grand Prix Winner\"\u001b[0m\u001b[32;1m\u001b[1;3m I need to find out who Olivia Wilde's boyfriend is and then calculate his age raised to the 0.23 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Olivia Wilde boyfriend\"\u001b[0m\u001b[38;5;200m\u001b[1;3m I need to find out who won the US Open women's final in 2019 and then calculate her age raised to the 0.34 power.\n",
+      "Action: Search\n",
+      "Action Input: \"US Open women's final 2019 winner\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mJay-Z\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[33;1m\u001b[1;3mMax Emilian Verstappen is a Belgian-Dutch racing driver and the 2021 and 2022 Formula One World Champion. He competes under the Dutch flag in Formula One with Red Bull Racing. Verstappen is the son of racing drivers Jos Verstappen, who also competed in Formula One, and Sophie Kumpen.\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[33;1m\u001b[1;3mJason Sudeikis\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[33;1m\u001b[1;3mBianca Andreescu defeated Serena Williams in the final, 6–3, 7–5 to win the women's singles tennis title at the 2019 US Open. It was her first major title, and she became the first Canadian, as well as the first player born in the 2000s, to win a major singles title.\u001b[0m\n",
+      "Thought:\u001b[31;1m\u001b[1;3m I need to find out Max Emilian Verstappen's age.\n",
+      "Action: Search\n",
+      "Action Input: \"Max Emilian Verstappen age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m25 years\u001b[0m\n",
+      "Thought:\u001b[38;5;200m\u001b[1;3m I need to find out Bianca Andreescu's age.\n",
+      "Action: Search\n",
+      "Action Input: \"Bianca Andreescu age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mBianca Vanessa Andreescu is a Canadian-Romanian professional tennis player. She has a career-high ranking of No. 4 in the world, and is the highest-ranked Canadian in the history of the Women's Tennis Association.\u001b[0m\n",
+      "Thought:\u001b[36;1m\u001b[1;3m I need to find out who won the US Open men's final in 2019 and then calculate his age raised to the 0.334 power.\n",
+      "Action: Search\n",
+      "Action Input: \"US Open men's final 2019 winner\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mRafael Nadal\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Jason Sudeikis' age\n",
+      "Action: Search\n",
+      "Action Input: \"Jason Sudeikis age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mDaniel Jason Sudeikis is an American actor, comedian, writer, and producer. In the 1990s, he began his career in improv comedy and performed with ComedySportz, iO Chicago, and The Second City.\u001b[0m\n",
+      "Thought:\u001b[33;1m\u001b[1;3m I need to find out Jay-Z's age\n",
+      "Action: Search\n",
+      "Action Input: \"How old is Jay-Z?\"\u001b[0m\u001b[36;1m\u001b[1;3m I need to find out Rafael Nadal's age\n",
+      "Action: Search\n",
+      "Action Input: \"Rafael Nadal age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m36 years\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[33;1m\u001b[1;3m53 years\u001b[0m\n",
+      "Thought:\u001b[38;5;200m\u001b[1;3m I now know the age of Bianca Andreescu.\n",
+      "Action: Calculator\n",
+      "Action Input: 19^0.34\u001b[0m\u001b[31;1m\u001b[1;3m I now need to calculate 25 raised to the 0.23 power.\n",
+      "Action: Calculator\n",
+      "Action Input: 25^0.23\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.7212987634680084\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Jason Sudeikis' exact age\n",
+      "Action: Search\n",
+      "Action Input: \"Jason Sudeikis age exact\"\u001b[0m\u001b[33;1m\u001b[1;3m I need to calculate 53 raised to the 0.19 power\n",
+      "Action: Calculator\n",
+      "Action Input: 53^0.19\u001b[0m\u001b[36;1m\u001b[1;3m I need to calculate 36 raised to the 0.334 power\n",
+      "Action: Calculator\n",
+      "Action Input: 36^0.334\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mDaniel Jason Sudeikis. (1975-09-18) September 18, 1975 (age 47). Fairfax, Virginia, U.S. · Fort Scott Community College · Actor; comedian; producer; writer · 1997– ...\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.096651272316035\n",
+      "\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.12624064206896\n",
+      "\u001b[0m\n",
+      "Thought:\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 3.3098250249682484\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now have the information I need to calculate the age raised to the 0.23 power\n",
+      "Action: Calculator\n",
+      "Action Input: 47^0.23\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 2.4242784855673896\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: Bianca Andreescu, aged 19, won the US Open women's final in 2019. Her age raised to the 0.34 power is 2.7212987634680084.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Jay-Z is Beyonce's husband and his age raised to the 0.19 power is 2.12624064206896.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Rafael Nadal, aged 36, won the US Open men's final in 2019 and his age raised to the 0.334 power is 3.3098250249682484.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Jason Sudeikis, Olivia Wilde's boyfriend, is 47 years old and his age raised to the 0.23 power is 2.4242784855673896.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: Max Emilian Verstappen, who is 25 years old, won the most recent Formula 1 Grand Prix and his age raised to the 0.23 power is 2.096651272316035.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "Concurrent executed in 25.06 seconds.\n"
+     ]
+    }
+   ],
+   "source": [
+    "async def generate_concurrently():\n",
+    "    agents = []\n",
+    "    # To make async requests in Tools more efficient, you can pass in your own aiohttp.ClientSession, \n",
+    "    # but you must manually close the client session at the end of your program/event loop\n",
+    "    aiosession = ClientSession()\n",
+    "    colors = [\"blue\", \"green\", \"red\", \"pink\", \"yellow\"]\n",
+    "    for color in colors:\n",
+    "        # Use a custom CallbackManager to print in different colors.\n",
+    "        manager = CallbackManager([StdOutCallbackHandler(color=color)])\n",
+    "        llm = OpenAI(temperature=0, callback_manager=manager)\n",
+    "        async_tools = load_tools([\"llm-math\", \"serpapi\"], llm=llm, aiosession=aiosession)\n",
+    "        agents.append(\n",
+    "            initialize_agent(async_tools, llm, agent=\"zero-shot-react-description\", verbose=True, callback_manager=manager)\n",
+    "        )\n",
+    "    tasks = [async_agent.arun(q) for async_agent, q in zip(agents, questions)]\n",
+    "    await asyncio.gather(*tasks)\n",
+    "    await aiosession.close()\n",
+    "\n",
+    "s = time.perf_counter()\n",
+    "# If running this outside of Jupyter, use asyncio.run(generate_concurrently())\n",
+    "await generate_concurrently()\n",
+    "elapsed = time.perf_counter() - s\n",
+    "print(f\"Concurrent executed in {elapsed:0.2f} seconds.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97ef285c-4a43-4a4e-9698-cd52a1bc56c9",
+   "metadata": {},
+   "source": [
+    "## Using Tracing with Asynchronous Agents\n",
+    "\n",
+    "To use tracing with async agents, you must pass in a custom `CallbackManager` with `LangChainTracer` to each agent running asynchronously. This way, you avoid collisions while the trace is being collected."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "44bda05a-d33e-4e91-9a71-a0f3f96aae95",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who won the US Open men's final in 2019 and then calculate his age raised to the 0.334 power.\n",
+      "Action: Search\n",
+      "Action Input: \"US Open men's final 2019 winner\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mRafael Nadal\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Rafael Nadal's age\n",
+      "Action: Search\n",
+      "Action Input: \"Rafael Nadal age\"\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m36 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 36 raised to the 0.334 power\n",
+      "Action: Calculator\n",
+      "Action Input: 36^0.334\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 3.3098250249682484\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Rafael Nadal, aged 36, won the US Open men's final in 2019 and his age raised to the 0.334 power is 3.3098250249682484.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# To make async requests in Tools more efficient, you can pass in your own aiohttp.ClientSession, \n",
+    "# but you must manually close the client session at the end of your program/event loop\n",
+    "aiosession = ClientSession()\n",
+    "tracer = LangChainTracer()\n",
+    "tracer.load_default_session()\n",
+    "manager = CallbackManager([StdOutCallbackHandler(), tracer])\n",
+    "\n",
+    "# Pass the manager into the llm if you want llm calls traced.\n",
+    "llm = OpenAI(temperature=0, callback_manager=manager)\n",
+    "\n",
+    "async_tools = load_tools([\"llm-math\", \"serpapi\"], llm=llm, aiosession=aiosession)\n",
+    "async_agent = initialize_agent(async_tools, llm, agent=\"zero-shot-react-description\", verbose=True, callback_manager=manager)\n",
+    "await async_agent.arun(questions[0])\n",
+    "await aiosession.close()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/examples/custom_agent.ipynb

@@ -53,7 +53,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 2,
    "id": "becda2a1",
    "metadata": {},
    "outputs": [],
@@ -70,7 +70,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 3,
    "id": "339b1bb8",
    "metadata": {},
    "outputs": [],
@@ -99,7 +99,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 4,
    "id": "e21d2098",
    "metadata": {},
    "outputs": [
@@ -134,7 +134,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "5e028e6d",
    "metadata": {},
@@ -146,7 +145,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 5,
    "id": "9b1cc2a2",
    "metadata": {},
    "outputs": [],
@@ -156,17 +155,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 7,
    "id": "e4f5092f",
    "metadata": {},
    "outputs": [],
    "source": [
-    "agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools)"
+    "tool_names = [tool.name for tool in tools]\n",
+    "agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 8,
    "id": "490604e9",
    "metadata": {},
    "outputs": [],
@@ -176,7 +176,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 9,
    "id": "653b1617",
    "metadata": {},
    "outputs": [
@@ -191,22 +191,23 @@
       "Action: Search\n",
       "Action Input: Population of Canada\u001b[0m\n",
       "Observation: \u001b[36;1m\u001b[1;3mCanada is a country in North America. Its ten provinces and three territories extend from the Atlantic Ocean to the Pacific Ocean and northward into the Arctic Ocean, covering over 9.98 million square kilometres, making it the world's second-largest country by total area.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to find out the exact population of Canada\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out the population of Canada\n",
       "Action: Search\n",
-      "Action Input: Population of Canada 2020\u001b[0m\n",
+      "Action Input: Population of Canada\u001b[0m\n",
       "Observation: \u001b[36;1m\u001b[1;3mCanada is a country in North America. Its ten provinces and three territories extend from the Atlantic Ocean to the Pacific Ocean and northward into the Arctic Ocean, covering over 9.98 million square kilometres, making it the world's second-largest country by total area.\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m I now know the population of Canada\n",
-      "Final Answer: Arrr, Canada be home to 37.59 million people!\u001b[0m\n",
-      "\u001b[1m> Finished AgentExecutor chain.\u001b[0m\n"
+      "Final Answer: Arrr, Canada be home to over 37 million people!\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
     },
     {
      "data": {
       "text/plain": [
-       "'Arrr, Canada be home to 37.59 million people!'"
+       "'Arrr, Canada be home to over 37 million people!'"
       ]
      },
-     "execution_count": 19,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -361,7 +362,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -375,7 +376,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.12 (default, Feb 15 2022, 17:41:09) \n[Clang 12.0.5 (clang-1205.0.22.11)]"
+   "version": "3.10.9"
   },
   "vscode": {
    "interpreter": {

docs/modules/agents/examples/custom_tools.ipynb

@@ -10,15 +10,17 @@
     "When constructing your own agent, you will need to provide it with a list of Tools that it can use. A Tool is defined as below.\n",
     "\n",
     "```python\n",
-    "class Tool(NamedTuple):\n",
+    "@dataclass \n",
+    "class Tool:\n",
     "    \"\"\"Interface for tools.\"\"\"\n",
     "\n",
     "    name: str\n",
     "    func: Callable[[str], str]\n",
     "    description: Optional[str] = None\n",
+    "    return_direct: bool = True\n",
     "```\n",
     "\n",
-    "The two required components of a Tool are the name and then the tool itself. A tool description is optional, as it is needed for some agents but not all."
+    "The two required components of a Tool are the name and then the tool itself. A tool description is optional, as it is needed for some agents but not all. You can create these tools directly, but we also provide a decorator to easily convert any function into a tool."
    ]
   },
   {
@@ -151,6 +153,94 @@
     "agent.run(\"Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "824eaf74",
+   "metadata": {},
+   "source": [
+    "## Using the `tool` decorator\n",
+    "\n",
+    "To make it easier to define custom tools, a `@tool` decorator is provided. This decorator can be used to quickly create a `Tool` from a simple function. The decorator uses the function name as the tool name by default, but this can be overridden by passing a string as the first argument. Additionally, the decorator will use the function's docstring as the tool's description."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "8f15307d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import tool\n",
+    "\n",
+    "@tool\n",
+    "def search_api(query: str) -> str:\n",
+    "    \"\"\"Searches the API for the query.\"\"\"\n",
+    "    return \"Results\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "0a23b91b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Tool(name='search_api', func=<function search_api at 0x10dad7d90>, description='search_api(query: str) -> str - Searches the API for the query.', return_direct=False)"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search_api"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cc6ee8c1",
+   "metadata": {},
+   "source": [
+    "You can also provide arguments like the tool name and whether to return directly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "28cdf04d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@tool(\"search\", return_direct=True)\n",
+    "def search_api(query: str) -> str:\n",
+    "    \"\"\"Searches the API for the query.\"\"\"\n",
+    "    return \"Results\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "1085a4bd",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Tool(name='search', func=<function search_api at 0x112301bd0>, description='search(query: str) -> str - Searches the API for the query.', return_direct=True)"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search_api"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "1d0430d6",
@@ -432,7 +522,7 @@
   },
   "vscode": {
    "interpreter": {
-    "hash": "cb23c3a7a387ab03496baa08507270f8e0861b23170e79d5edc545893cdca840"
+    "hash": "e90c8aa204a57276aa905271aff2d11799d0acb3547adabc5892e639a5e45e34"
    }
   }
  },

docs/modules/agents/examples/load_from_hub.ipynb

@@ -0,0 +1,108 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "991b1cc1",
+   "metadata": {},
+   "source": [
+    "# Loading from LangChainHub\n",
+    "\n",
+    "This notebook covers how to load agents from [LangChainHub](https://github.com/hwchase17/langchain-hub)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "bd4450a2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m Yes.\n",
+      "Follow up: Who is the reigning men's U.S. Open champion?\u001b[0m\n",
+      "Intermediate answer: \u001b[36;1m\u001b[1;3m2016 · SUI · Stan Wawrinka ; 2017 · ESP · Rafael Nadal ; 2018 · SRB · Novak Djokovic ; 2019 · ESP · Rafael Nadal.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mSo the reigning men's U.S. Open champion is Rafael Nadal.\n",
+      "Follow up: What is Rafael Nadal's hometown?\u001b[0m\n",
+      "Intermediate answer: \u001b[36;1m\u001b[1;3mIn 2016, he once again showed his deep ties to Mallorca and opened the Rafa Nadal Academy in his hometown of Manacor.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mSo the final answer is: Manacor, Mallorca, Spain.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Manacor, Mallorca, Spain.'"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain import OpenAI, SerpAPIWrapper\n",
+    "from langchain.agents import initialize_agent, Tool\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "search = SerpAPIWrapper()\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name=\"Intermediate Answer\",\n",
+    "        func=search.run\n",
+    "    )\n",
+    "]\n",
+    "\n",
+    "self_ask_with_search = initialize_agent(tools, llm, agent_path=\"lc://agents/self-ask-with-search/agent.json\", verbose=True)\n",
+    "self_ask_with_search.run(\"What is the hometown of the reigning men's U.S. Open champion?\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3aede965",
+   "metadata": {},
+   "source": [
+    "# Pinning Dependencies\n",
+    "\n",
+    "Specific versions of LangChainHub agents can be pinned with the `lc@<ref>://` syntax."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e679f7b6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "self_ask_with_search = initialize_agent(tools, llm, agent_path=\"lc@2826ef9e8acdf88465e1e5fc8a7bf59e0f9d0a85://agents/self-ask-with-search/agent.json\", verbose=True)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/examples/serialization.ipynb

@@ -0,0 +1,148 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bfe18e28",
+   "metadata": {},
+   "source": [
+    "# Serialization\n",
+    "\n",
+    "This notebook goes over how to serialize agents. For this notebook, it is important to understand the distinction we draw between `agents` and `tools`. An agent is the LLM powered decision maker that decides which actions to take and in which order. Tools are various instruments (functions) an agent has access to, through which an agent can interact with the outside world. When people generally use agents, they primarily talk about using an agent WITH tools. However, when we talk about serialization of agents, we are talking about the agent by itself. We plan to add support for serializing an agent WITH tools sometime in the future.\n",
+    "\n",
+    "Let's start by creating an agent with tools as we normally do:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "eb729f16",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import load_tools\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.llms import OpenAI\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
+    "agent = initialize_agent(tools, llm, agent=\"zero-shot-react-description\", verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0578f566",
+   "metadata": {},
+   "source": [
+    "Let's now serialize the agent. To be explicit that we are serializing ONLY the agent, we will call the `save_agent` method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "dc544de6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent.save_agent('agent.json')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "62dd45bf",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      "    \"llm_chain\": {\r\n",
+      "        \"memory\": null,\r\n",
+      "        \"verbose\": false,\r\n",
+      "        \"prompt\": {\r\n",
+      "            \"input_variables\": [\r\n",
+      "                \"input\",\r\n",
+      "                \"agent_scratchpad\"\r\n",
+      "            ],\r\n",
+      "            \"output_parser\": null,\r\n",
+      "            \"template\": \"Answer the following questions as best you can. You have access to the following tools:\\n\\nSearch: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.\\nCalculator: Useful for when you need to answer questions about math.\\n\\nUse the following format:\\n\\nQuestion: the input question you must answer\\nThought: you should always think about what to do\\nAction: the action to take, should be one of [Search, Calculator]\\nAction Input: the input to the action\\nObservation: the result of the action\\n... (this Thought/Action/Action Input/Observation can repeat N times)\\nThought: I now know the final answer\\nFinal Answer: the final answer to the original input question\\n\\nBegin!\\n\\nQuestion: {input}\\nThought:{agent_scratchpad}\",\r\n",
+      "            \"template_format\": \"f-string\"\r\n",
+      "        },\r\n",
+      "        \"llm\": {\r\n",
+      "            \"model_name\": \"text-davinci-003\",\r\n",
+      "            \"temperature\": 0.0,\r\n",
+      "            \"max_tokens\": 256,\r\n",
+      "            \"top_p\": 1,\r\n",
+      "            \"frequency_penalty\": 0,\r\n",
+      "            \"presence_penalty\": 0,\r\n",
+      "            \"n\": 1,\r\n",
+      "            \"best_of\": 1,\r\n",
+      "            \"request_timeout\": null,\r\n",
+      "            \"logit_bias\": {},\r\n",
+      "            \"_type\": \"openai\"\r\n",
+      "        },\r\n",
+      "        \"output_key\": \"text\",\r\n",
+      "        \"_type\": \"llm_chain\"\r\n",
+      "    },\r\n",
+      "    \"return_values\": [\r\n",
+      "        \"output\"\r\n",
+      "    ],\r\n",
+      "    \"_type\": \"zero-shot-react-description\"\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "!cat agent.json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0eb72510",
+   "metadata": {},
+   "source": [
+    "We can now load the agent back in"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "eb660b76",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = initialize_agent(tools, llm, agent_path=\"agent.json\", verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "aa624ea5",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/getting_started.ipynb

@@ -152,7 +152,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.0 64-bit ('llm-env')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -166,7 +166,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/agents/how_to_guides.rst

@@ -3,6 +3,8 @@ How-To Guides
 
 The first category of how-to guides here cover specific parts of working with agents.
 
+`Load From Hub <./examples/load_from_hub.html>`_: This notebook covers how to load agents from `LangChainHub <https://github.com/hwchase17/langchain-hub>`_.
+
 `Custom Tools <./examples/custom_tools.html>`_: How to create custom tools that an agent can use.
 
 `Intermediate Steps <./examples/intermediate_steps.html>`_: How to access and use intermediate steps to get more visibility into the internals of an agent.
@@ -15,6 +17,7 @@ The first category of how-to guides here cover specific parts of working with ag
 
 `Max Iterations <./examples/max_iterations.html>`_: How to restrict an agent to a certain number of iterations.
 
+`Asynchronous <./examples/async_agent.html>`_: Covering asynchronous functionality.
 
 The next set of examples are all end-to-end agents for specific applications.
 In all examples there is an Agent with a particular set of tools.

docs/modules/agents/implementations/natbot.py

@@ -2,7 +2,7 @@
 import time
 
 from langchain.chains.natbot.base import NatBotChain
-from langchain.chains.natbot.crawler import Crawler  # type: ignore
+from langchain.chains.natbot.crawler import Crawler
 
 
 def run_cmd(cmd: str, _crawler: Crawler) -> None:
@@ -33,7 +33,6 @@ def run_cmd(cmd: str, _crawler: Crawler) -> None:
 
 
 if __name__ == "__main__":
-
     objective = "Make a reservation for 2 at 7pm at bistro vida in menlo park"
     print("\nWelcome to natbot! What is your objective?")
     i = input()

docs/modules/agents/tools.md

@@ -22,6 +22,7 @@ tools = load_tools(tool_names, llm=llm)
 ```
 
 Below is a list of all supported tools and relevant information:
+
 - Tool Name: The name the LLM refers to the tool by.
 - Tool Description: The description of the tool that is passed to the LLM.
 - Notes: Notes about the tool that are NOT passed to the LLM.
@@ -31,61 +32,71 @@ Below is a list of all supported tools and relevant information:
 ## List of Tools
 
 **python_repl**
+
 - Tool Name: Python REPL
 - Tool Description: A Python shell. Use this to execute python commands. Input should be a valid python command. If you expect output it should be printed out.
 - Notes: Maintains state.
 - Requires LLM: No
 
-
 **serpapi**
+
 - Tool Name: Search
 - Tool Description: A search engine. Useful for when you need to answer questions about current events. Input should be a search query.
 - Notes: Calls the Serp API and then parses results.
 - Requires LLM: No
 
 **wolfram-alpha**
+
 - Tool Name: Wolfram Alpha
 - Tool Description: A wolfram alpha search engine. Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Input should be a search query.
 - Notes: Calls the Wolfram Alpha API and then parses results.
 - Requires LLM: No
+- Extra Parameters: `wolfram_alpha_appid`: The Wolfram Alpha app id.
 
 **requests**
+
 - Tool Name: Requests
 - Tool Description: A portal to the internet. Use this when you need to get specific content from a site. Input should be a specific url, and the output will be all the text on that page.
 - Notes: Uses the Python requests module.
 - Requires LLM: No
 
 **terminal**
+
 - Tool Name: Terminal
 - Tool Description: Executes commands in a terminal. Input should be valid commands, and the output will be any output from running that command.
 - Notes: Executes commands with subprocess.
 - Requires LLM: No
 
 **pal-math**
+
 - Tool Name: PAL-MATH
 - Tool Description: A language model that is excellent at solving complex word math problems. Input should be a fully worded hard word math problem.
 - Notes: Based on [this paper](https://arxiv.org/pdf/2211.10435.pdf).
 - Requires LLM: Yes
 
 **pal-colored-objects**
+
 - Tool Name: PAL-COLOR-OBJ
 - Tool Description: A language model that is wonderful at reasoning about position and the color attributes of objects. Input should be a fully worded hard reasoning problem. Make sure to include all information about the objects AND the final question you want to answer.
 - Notes: Based on [this paper](https://arxiv.org/pdf/2211.10435.pdf).
 - Requires LLM: Yes
 
 **llm-math**
+
 - Tool Name: Calculator
 - Tool Description: Useful for when you need to answer questions about math.
 - Notes: An instance of the `LLMMath` chain.
 - Requires LLM: Yes
 
 **open-meteo-api**
+
 - Tool Name: Open Meteo API
 - Tool Description: Useful for when you want to get weather information from the OpenMeteo API. The input should be a question in natural language that this API can answer.
 - Notes: A natural language connection to the Open Meteo API (`https://api.open-meteo.com/`), specifically the `/v1/forecast` endpoint.
 - Requires LLM: Yes
 
 **news-api**
+
 - Tool Name: News API
 - Tool Description: Use this when you want to get information about the top headlines of current news stories. The input should be a question in natural language that this API can answer.
 - Notes: A natural language connection to the News API (`https://newsapi.org`), specifically the `/v2/top-headlines` endpoint.
@@ -93,8 +104,18 @@ Below is a list of all supported tools and relevant information:
 - Extra Parameters: `news_api_key` (your API key to access this endpoint)
 
 **tmdb-api**
+
 - Tool Name: TMDB API
 - Tool Description: Useful for when you want to get information from The Movie Database. The input should be a question in natural language that this API can answer.
 - Notes: A natural language connection to the TMDB API (`https://api.themoviedb.org/3`), specifically the `/search/movie` endpoint.
 - Requires LLM: Yes
 - Extra Parameters: `tmdb_bearer_token` (your Bearer Token to access this endpoint - note that this is different from the API key)
+
+**google-search**
+
+- Tool Name: Search
+- Tool Description: A wrapper around Google Search. Useful for when you need to answer questions about current events. Input should be a search query.
+- Notes: Uses the Google Custom Search API
+- Requires LLM: No
+- Extra Parameters: `google_api_key`, `google_cse_id`
+- For more information on this, see [this page](../../ecosystem/google_search.md)

docs/modules/chains/async_chain.ipynb

@@ -0,0 +1,132 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "593f7553-7038-498e-96d4-8255e5ce34f0",
+   "metadata": {},
+   "source": [
+    "# Async API for Chain\n",
+    "\n",
+    "LangChain provides async support for Chains by leveraging the [asyncio](https://docs.python.org/3/library/asyncio.html) library.\n",
+    "\n",
+    "Async methods are currently supported in `LLMChain` (through `arun`, `apredict`, `acall`) and `LLMMathChain` (through `arun` and `acall`). Async support for other chains is on the roadmap."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "c19c736e-ca74-4726-bb77-0a849bcc2960",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "BrightSmile Toothpaste Company\n",
+      "\n",
+      "\n",
+      "BrightSmile Toothpaste Co.\n",
+      "\n",
+      "\n",
+      "BrightSmile Toothpaste\n",
+      "\n",
+      "\n",
+      "Gleaming Smile Inc.\n",
+      "\n",
+      "\n",
+      "SparkleSmile Toothpaste\n",
+      "\u001b[1mConcurrent executed in 1.54 seconds.\u001b[0m\n",
+      "\n",
+      "\n",
+      "BrightSmile Toothpaste Co.\n",
+      "\n",
+      "\n",
+      "MintyFresh Toothpaste Co.\n",
+      "\n",
+      "\n",
+      "SparkleSmile Toothpaste.\n",
+      "\n",
+      "\n",
+      "Pearly Whites Toothpaste Co.\n",
+      "\n",
+      "\n",
+      "BrightSmile Toothpaste.\n",
+      "\u001b[1mSerial executed in 6.38 seconds.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "import asyncio\n",
+    "import time\n",
+    "\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
+    "\n",
+    "\n",
+    "def generate_serially():\n",
+    "    llm = OpenAI(temperature=0.9)\n",
+    "    prompt = PromptTemplate(\n",
+    "        input_variables=[\"product\"],\n",
+    "        template=\"What is a good name for a company that makes {product}?\",\n",
+    "    )\n",
+    "    chain = LLMChain(llm=llm, prompt=prompt)\n",
+    "    for _ in range(5):\n",
+    "        resp = chain.run(product=\"toothpaste\")\n",
+    "        print(resp)\n",
+    "\n",
+    "\n",
+    "async def async_generate(chain):\n",
+    "    resp = await chain.arun(product=\"toothpaste\")\n",
+    "    print(resp)\n",
+    "\n",
+    "\n",
+    "async def generate_concurrently():\n",
+    "    llm = OpenAI(temperature=0.9)\n",
+    "    prompt = PromptTemplate(\n",
+    "        input_variables=[\"product\"],\n",
+    "        template=\"What is a good name for a company that makes {product}?\",\n",
+    "    )\n",
+    "    chain = LLMChain(llm=llm, prompt=prompt)\n",
+    "    tasks = [async_generate(chain) for _ in range(5)]\n",
+    "    await asyncio.gather(*tasks)\n",
+    "\n",
+    "s = time.perf_counter()\n",
+    "# If running this outside of Jupyter, use asyncio.run(generate_concurrently())\n",
+    "await generate_concurrently()\n",
+    "elapsed = time.perf_counter() - s\n",
+    "print('\\033[1m' + f\"Concurrent executed in {elapsed:0.2f} seconds.\" + '\\033[0m')\n",
+    "\n",
+    "s = time.perf_counter()\n",
+    "generate_serially()\n",
+    "elapsed = time.perf_counter() - s\n",
+    "print('\\033[1m' + f\"Serial executed in {elapsed:0.2f} seconds.\" + '\\033[0m')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/chains/combine_docs_examples/analyze_document.ipynb

@@ -0,0 +1,178 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "ad719b65",
+   "metadata": {},
+   "source": [
+    "# Analyze Document\n",
+    "\n",
+    "The AnalyzeDocumentChain is more of an end to chain. This chain takes in a single document, splits it up, and then runs it through a CombineDocumentsChain. This can be used as more of an end-to-end chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "15e1a8a2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('../../state_of_the_union.txt') as f:\n",
+    "    state_of_the_union = f.read()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14da4012",
+   "metadata": {},
+   "source": [
+    "## Summarize\n",
+    "Let's take a look at it in action below, using it summarize a long document."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "765d6326",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain import OpenAI\n",
+    "from langchain.chains.summarize import load_summarize_chain\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "summary_chain = load_summarize_chain(llm, chain_type=\"map_reduce\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "3a3d3ebc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import AnalyzeDocumentChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "97178aad",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "summarize_document_chain = AnalyzeDocumentChain(combine_docs_chain=summary_chain)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "2e5a7bf7",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\" In this speech, President Biden addresses the American people and the world, discussing the recent aggression of Russia's Vladimir Putin in Ukraine and the US response. He outlines economic sanctions and other measures taken to hold Putin accountable, and announces the US Department of Justice's task force to go after the crimes of Russian oligarchs. He also announces plans to fight inflation and lower costs for families, invest in American manufacturing, and provide military, economic, and humanitarian assistance to Ukraine. He calls for immigration reform, protecting the rights of women, and advancing the rights of LGBTQ+ Americans, and pays tribute to military families. He concludes with optimism for the future of America.\""
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "summarize_document_chain.run(state_of_the_union)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "35739404",
+   "metadata": {},
+   "source": [
+    "## Question Answering\n",
+    "Let's take a look at this using a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "8b9b7705",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains.question_answering import load_qa_chain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "60c309a8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "qa_chain = load_qa_chain(llm, chain_type=\"map_reduce\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "ba1fc940",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "qa_document_chain = AnalyzeDocumentChain(combine_docs_chain=qa_chain)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "9aa1fbde",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' The president thanked Justice Breyer for his service.'"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "qa_document_chain.run(input_document=state_of_the_union, question=\"what did the president say about justice breyer?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7eb02f1e",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/chains/combine_docs_examples/chat_vector_db.ipynb

@@ -0,0 +1,220 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "134a0785",
+   "metadata": {},
+   "source": [
+    "# Chat Vector DB\n",
+    "\n",
+    "This notebook goes over how to set up a chain to chat with a vector database. The only difference because this chain and the [VectorDBQAChain](./vector_db_qa.ipynb) is that this allows for passing in of a chat history which can be used to allow for follow up questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "70c4e529",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.vectorstores.faiss import FAISS\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.chains import ChatVectorDBChain\n",
+    "from langchain.document_loaders import TextLoader"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cdff94be",
+   "metadata": {},
+   "source": [
+    "Load in documents. You can replace this with a loader for whatever type of data you want"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "01c46e92",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = TextLoader('../../state_of_the_union.txt')\n",
+    "documents = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9be4779",
+   "metadata": {},
+   "source": [
+    "If you had multiple loaders that you wanted to combine, you do something like:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "433363a5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# loaders = [....]\n",
+    "# docs = []\n",
+    "# for loader in loaders:\n",
+    "#     docs.extend(loader.load())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "239475d2",
+   "metadata": {},
+   "source": [
+    "We now split the documents, create embeddings for them, and put them in a vectorstore. This allows us to do semantic search over them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "a8930cf7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+    "documents = text_splitter.split_documents(documents)\n",
+    "\n",
+    "embeddings = OpenAIEmbeddings()\n",
+    "vectorstore = FAISS.from_documents(documents, embeddings)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c96b118",
+   "metadata": {},
+   "source": [
+    "We now initialize the ChatVectorDBChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "7b4110f3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "qa = ChatVectorDBChain.from_llm(OpenAI(temperature=0), vectorstore)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3872432d",
+   "metadata": {},
+   "source": [
+    "Here's an example of asking a question with no chat history"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "7fe3e730",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chat_history = []\n",
+    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
+    "result = qa({\"question\": query, \"chat_history\": chat_history})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "bfff9cc8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\""
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result[\"answer\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9e46edf7",
+   "metadata": {},
+   "source": [
+    "Here's an example of asking a question with some chat history"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "00b4cf00",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chat_history = [(query, result[\"answer\"])]\n",
+    "query = \"Did he mention who she suceeded\"\n",
+    "result = qa({\"question\": query, \"chat_history\": chat_history})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "f01828d1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' Justice Stephen Breyer'"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result['answer']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d0f869c6",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/chains/combine_docs_examples/vector_db_qa_with_sources.ipynb

@@ -187,7 +187,7 @@
     }
    ],
    "source": [
-    "chain({\"question\": \"What did the president say about Justice Breyer\"}, return_only_outputs=True)"
+    "qa({\"question\": \"What did the president say about Justice Breyer\"}, return_only_outputs=True)"
    ]
   }
  ],

docs/modules/chains/combine_docs_examples/vector_db_text_generation.ipynb

@@ -0,0 +1,199 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Vector DB Text Generation\n",
+    "\n",
+    "This notebook walks through how to use LangChain for text generation over a vector index. This is useful if we want to generate text that is able to draw from a large body of custom text, for example, generating blog posts that have an understanding of previous blog posts written, or product tutorials that can refer to product documentation."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prepare Data\n",
+    "\n",
+    "First, we prepare the data. For this example, we fetch a documentation site that consists of markdown files hosted on Github and split them into small enough Documents."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.llms import OpenAI\n",
+    "from langchain.docstore.document import Document\n",
+    "import requests\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.vectorstores.faiss import FAISS\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "import pathlib\n",
+    "import subprocess\n",
+    "import tempfile"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Cloning into '.'...\n"
+     ]
+    }
+   ],
+   "source": [
+    "def get_github_docs(repo_owner, repo_name):\n",
+    "    with tempfile.TemporaryDirectory() as d:\n",
+    "        subprocess.check_call(\n",
+    "            f\"git clone --depth 1 https://github.com/{repo_owner}/{repo_name}.git .\",\n",
+    "            cwd=d,\n",
+    "            shell=True,\n",
+    "        )\n",
+    "        git_sha = (\n",
+    "            subprocess.check_output(\"git rev-parse HEAD\", shell=True, cwd=d)\n",
+    "            .decode(\"utf-8\")\n",
+    "            .strip()\n",
+    "        )\n",
+    "        repo_path = pathlib.Path(d)\n",
+    "        markdown_files = list(repo_path.glob(\"*/*.md\")) + list(\n",
+    "            repo_path.glob(\"*/*.mdx\")\n",
+    "        )\n",
+    "        for markdown_file in markdown_files:\n",
+    "            with open(markdown_file, \"r\") as f:\n",
+    "                relative_path = markdown_file.relative_to(repo_path)\n",
+    "                github_url = f\"https://github.com/{repo_owner}/{repo_name}/blob/{git_sha}/{relative_path}\"\n",
+    "                yield Document(page_content=f.read(), metadata={\"source\": github_url})\n",
+    "\n",
+    "sources = get_github_docs(\"yirenlu92\", \"deno-manual-forked\")\n",
+    "\n",
+    "source_chunks = []\n",
+    "splitter = CharacterTextSplitter(separator=\" \", chunk_size=1024, chunk_overlap=0)\n",
+    "for source in sources:\n",
+    "    for chunk in splitter.split_text(source.page_content):\n",
+    "        source_chunks.append(Document(page_content=chunk, metadata=source.metadata))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set Up Vector DB\n",
+    "\n",
+    "Now that we have the documentation content in chunks, let's put all this information in a vector index for easy retrieval."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search_index = FAISS.from_documents(source_chunks, OpenAIEmbeddings())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set Up LLM Chain with Custom Prompt\n",
+    "\n",
+    "Next, let's set up a simple LLM chain but give it a custom prompt for blog post generation. Note that the custom prompt is parameterized and takes two inputs: `context`, which will be the documents fetched from the vector search, and `topic`, which is given by the user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import LLMChain\n",
+    "prompt_template = \"\"\"Use the context below to write a 400 word blog post about the topic below:\n",
+    "    Context: {context}\n",
+    "    Topic: {topic}\n",
+    "    Blog post:\"\"\"\n",
+    "\n",
+    "PROMPT = PromptTemplate(\n",
+    "    template=prompt_template, input_variables=[\"context\", \"topic\"]\n",
+    ")\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "\n",
+    "chain = LLMChain(llm=llm, prompt=PROMPT)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generate Text\n",
+    "\n",
+    "Finally, we write a function to apply our inputs to the chain. The function takes an input parameter `topic`. We find the documents in the vector index that correspond to that `topic`, and use them as additional context in our simple LLM chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def generate_blog_post(topic):\n",
+    "    docs = search_index.similarity_search(topic, k=4)\n",
+    "    inputs = [{\"context\": doc.page_content, \"topic\": topic} for doc in docs]\n",
+    "    print(chain.apply(inputs))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[{'text': '\\n\\nEnvironment variables are a great way to store and access sensitive information in your Deno applications. Deno offers built-in support for environment variables with `Deno.env`, and you can also use a `.env` file to store and access environment variables.\\n\\nUsing `Deno.env` is simple. It has getter and setter methods, so you can easily set and retrieve environment variables. For example, you can set the `FIREBASE_API_KEY` and `FIREBASE_AUTH_DOMAIN` environment variables like this:\\n\\n```ts\\nDeno.env.set(\"FIREBASE_API_KEY\", \"examplekey123\");\\nDeno.env.set(\"FIREBASE_AUTH_DOMAIN\", \"firebasedomain.com\");\\n\\nconsole.log(Deno.env.get(\"FIREBASE_API_KEY\")); // examplekey123\\nconsole.log(Deno.env.get(\"FIREBASE_AUTH_DOMAIN\")); // firebasedomain.com\\n```\\n\\nYou can also store environment variables in a `.env` file. This is a great'}, {'text': '\\n\\nEnvironment variables are a powerful tool for managing configuration settings in a program. They allow us to set values that can be used by the program, without having to hard-code them into the code. This makes it easier to change settings without having to modify the code.\\n\\nIn Deno, environment variables can be set in a few different ways. The most common way is to use the `VAR=value` syntax. This will set the environment variable `VAR` to the value `value`. This can be used to set any number of environment variables before running a command. For example, if we wanted to set the environment variable `VAR` to `hello` before running a Deno command, we could do so like this:\\n\\n```\\nVAR=hello deno run main.ts\\n```\\n\\nThis will set the environment variable `VAR` to `hello` before running the command. We can then access this variable in our code using the `Deno.env.get()` function. For example, if we ran the following command:\\n\\n```\\nVAR=hello && deno eval \"console.log(\\'Deno: \\' + Deno.env.get(\\'VAR'}, {'text': '\\n\\nEnvironment variables are a powerful tool for developers, allowing them to store and access data without having to hard-code it into their applications. In Deno, you can access environment variables using the `Deno.env.get()` function.\\n\\nFor example, if you wanted to access the `HOME` environment variable, you could do so like this:\\n\\n```js\\n// env.js\\nDeno.env.get(\"HOME\");\\n```\\n\\nWhen running this code, you\\'ll need to grant the Deno process access to environment variables. This can be done by passing the `--allow-env` flag to the `deno run` command. You can also specify which environment variables you want to grant access to, like this:\\n\\n```shell\\n# Allow access to only the HOME env var\\ndeno run --allow-env=HOME env.js\\n```\\n\\nIt\\'s important to note that environment variables are case insensitive on Windows, so Deno also matches them case insensitively (on Windows only).\\n\\nAnother thing to be aware of when using environment variables is subprocess permissions. Subprocesses are powerful and can access system resources regardless of the permissions you granted to the Den'}, {'text': '\\n\\nEnvironment variables are an important part of any programming language, and Deno is no exception. Deno is a secure JavaScript and TypeScript runtime built on the V8 JavaScript engine, and it recently added support for environment variables. This feature was added in Deno version 1.6.0, and it is now available for use in Deno applications.\\n\\nEnvironment variables are used to store information that can be used by programs. They are typically used to store configuration information, such as the location of a database or the name of a user. In Deno, environment variables are stored in the `Deno.env` object. This object is similar to the `process.env` object in Node.js, and it allows you to access and set environment variables.\\n\\nThe `Deno.env` object is a read-only object, meaning that you cannot directly modify the environment variables. Instead, you must use the `Deno.env.set()` function to set environment variables. This function takes two arguments: the name of the environment variable and the value to set it to. For example, if you wanted to set the `FOO` environment variable to `bar`, you would use the following code:\\n\\n```'}]\n"
+     ]
+    }
+   ],
+   "source": [
+    "generate_blog_post(\"environment variables\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/modules/chains/combine_docs_how_to.rst

@@ -11,6 +11,8 @@ The examples here are all end-to-end chains for working with documents.
 
 `Summarization <./combine_docs_examples/summarize.html>`_: A walkthrough of how to use LangChain for summarization over specific documents.
 
+`Vector DB Text Generation <./combine_docs_examples/vector_db_text_generation.html>`_: A walkthrough of how to use LangChain for text generation over a vector database.
+
 `Vector DB Question Answering <./combine_docs_examples/vector_db_qa.html>`_: A walkthrough of how to use LangChain for question answering over a vector database.
 
 `Vector DB Question Answering with Sources <./combine_docs_examples/vector_db_qa_with_sources.html>`_: A walkthrough of how to use LangChain for question answering (with sources) over a vector database.

docs/modules/chains/examples/pal.ipynb

@@ -21,6 +21,24 @@
     "from langchain import OpenAI"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a58e15e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(model_name='code-davinci-002', temperature=0, max_tokens=512)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "095adc76",
+   "metadata": {},
+   "source": [
+    "## Math Prompt"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -28,7 +46,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "llm = OpenAI(model_name='code-davinci-002', temperature=0, max_tokens=512)\n",
     "pal_chain = PALChain.from_math_prompt(llm, verbose=True)"
    ]
   },
@@ -64,7 +81,7 @@
       "    result = total_pets\n",
       "    return result\u001b[0m\n",
       "\n",
-      "\u001b[1m> Finished PALChain chain.\u001b[0m\n"
+      "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
     },
     {
@@ -82,6 +99,14 @@
     "pal_chain.run(question)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "0269d20a",
+   "metadata": {},
+   "source": [
+    "## Colored Objects"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 5,
@@ -89,7 +114,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "llm = OpenAI(model_name='code-davinci-002', temperature=0, max_tokens=512)\n",
     "pal_chain = PALChain.from_colored_object_prompt(llm, verbose=True)"
    ]
   },
@@ -147,10 +171,94 @@
     "pal_chain.run(question)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "fc3d7f10",
+   "metadata": {},
+   "source": [
+    "## Intermediate Steps\n",
+    "You can also use the intermediate steps flag to return the code executed that generates the answer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "9d2d9c61",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pal_chain = PALChain.from_colored_object_prompt(llm, verbose=True, return_intermediate_steps=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "b29b971b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "question = \"On the desk, you see two blue booklets, two purple booklets, and two yellow pairs of sunglasses. If I remove all the pairs of sunglasses from the desk, how many purple items remain on it?\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "a2c40c28",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new PALChain chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m# Put objects into a list to record ordering\n",
+      "objects = []\n",
+      "objects += [('booklet', 'blue')] * 2\n",
+      "objects += [('booklet', 'purple')] * 2\n",
+      "objects += [('sunglasses', 'yellow')] * 2\n",
+      "\n",
+      "# Remove all pairs of sunglasses\n",
+      "objects = [object for object in objects if object[0] != 'sunglasses']\n",
+      "\n",
+      "# Count number of purple objects\n",
+      "num_purple = len([object for object in objects if object[1] == 'purple'])\n",
+      "answer = num_purple\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "result = pal_chain({\"question\": question})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "efddd033",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"# Put objects into a list to record ordering\\nobjects = []\\nobjects += [('booklet', 'blue')] * 2\\nobjects += [('booklet', 'purple')] * 2\\nobjects += [('sunglasses', 'yellow')] * 2\\n\\n# Remove all pairs of sunglasses\\nobjects = [object for object in objects if object[0] != 'sunglasses']\\n\\n# Count number of purple objects\\nnum_purple = len([object for object in objects if object[1] == 'purple'])\\nanswer = num_purple\""
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result['intermediate_steps']"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "4ab20fec",
+   "id": "dfd88594",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/chains/examples/sqlite.ipynb

@@ -56,9 +56,17 @@
     "llm = OpenAI(temperature=0)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "3d1e692e",
+   "metadata": {},
+   "source": [
+    "**NOTE:** For data-sensitive projects, you can specify `return_direct=True` in the `SQLDatabaseChain` initialization to directly return the output of the SQL query without any additional formatting. This prevents the LLM from seeing any contents within the database. Note, however, the LLM still has access to the database scheme (i.e. dialect, table and key names) by default."
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "id": "a8fc8f23",
    "metadata": {},
    "outputs": [],
@@ -68,7 +76,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "id": "15ff81df",
    "metadata": {
     "pycharm": {
@@ -85,18 +93,18 @@
       "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
       "How many employees are there? \n",
       "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT COUNT(*) FROM Employee;\u001b[0m\n",
-      "SQLResult: \u001b[33;1m\u001b[1;3m[(9,)]\u001b[0m\n",
-      "Answer:\u001b[32;1m\u001b[1;3m There are 9 employees.\u001b[0m\n",
+      "SQLResult: \u001b[33;1m\u001b[1;3m[(8,)]\u001b[0m\n",
+      "Answer:\u001b[32;1m\u001b[1;3m There are 8 employees.\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
     },
     {
      "data": {
       "text/plain": [
-       "' There are 9 employees.'"
+       "' There are 8 employees.'"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -168,15 +176,15 @@
       "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
       "How many employees are there in the foobar table? \n",
       "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT COUNT(*) FROM Employee;\u001b[0m\n",
-      "SQLResult: \u001b[33;1m\u001b[1;3m[(9,)]\u001b[0m\n",
-      "Answer:\u001b[32;1m\u001b[1;3m There are 9 employees in the foobar table.\u001b[0m\n",
+      "SQLResult: \u001b[33;1m\u001b[1;3m[(8,)]\u001b[0m\n",
+      "Answer:\u001b[32;1m\u001b[1;3m There are 8 employees in the foobar table.\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
     },
     {
      "data": {
       "text/plain": [
-       "' There are 9 employees in the foobar table.'"
+       "' There are 8 employees in the foobar table.'"
       ]
      },
      "execution_count": 7,
@@ -188,6 +196,62 @@
     "db_chain.run(\"How many employees are there in the foobar table?\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "88d8b969",
+   "metadata": {},
+   "source": [
+    "## Return Intermediate Steps\n",
+    "\n",
+    "You can also return the intermediate steps of the SQLDatabaseChain. This allows you to access the SQL statement that was generated, as well as the result of running that against the SQL Database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "38559487",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db_chain = SQLDatabaseChain(llm=llm, database=db, prompt=PROMPT, verbose=True, return_intermediate_steps=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "78b6af4d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
+      "How many employees are there in the foobar table? \n",
+      "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT COUNT(*) FROM Employee;\u001b[0m\n",
+      "SQLResult: \u001b[33;1m\u001b[1;3m[(8,)]\u001b[0m\n",
+      "Answer:\u001b[32;1m\u001b[1;3m There are 8 employees in the foobar table.\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[' SELECT COUNT(*) FROM Employee;', '[(8,)]']"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result = db_chain(\"How many employees are there in the foobar table?\")\n",
+    "result[\"intermediate_steps\"]"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "b408f800",
@@ -199,7 +263,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 10,
    "id": "6adaa799",
    "metadata": {},
    "outputs": [],
@@ -209,7 +273,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 11,
    "id": "edfc8a8e",
    "metadata": {},
    "outputs": [
@@ -221,8 +285,8 @@
       "\n",
       "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
       "What are some example tracks by composer Johann Sebastian Bach? \n",
-      "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT Name FROM Track WHERE Composer = 'Johann Sebastian Bach' LIMIT 3;\u001b[0m\n",
-      "SQLResult: \u001b[33;1m\u001b[1;3m[('Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace',), ('Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria',), ('Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude',)]\u001b[0m\n",
+      "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT Name, Composer FROM Track WHERE Composer = 'Johann Sebastian Bach' LIMIT 3;\u001b[0m\n",
+      "SQLResult: \u001b[33;1m\u001b[1;3m[('Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace', 'Johann Sebastian Bach'), ('Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria', 'Johann Sebastian Bach'), ('Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude', 'Johann Sebastian Bach')]\u001b[0m\n",
       "Answer:\u001b[32;1m\u001b[1;3m Examples of tracks by Johann Sebastian Bach include 'Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace', 'Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria', and 'Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude'.\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -233,7 +297,7 @@
        "' Examples of tracks by Johann Sebastian Bach include \\'Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace\\', \\'Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria\\', and \\'Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude\\'.'"
       ]
      },
-     "execution_count": 8,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -242,6 +306,101 @@
     "db_chain.run(\"What are some example tracks by composer Johann Sebastian Bach?\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "bcc5e936",
+   "metadata": {},
+   "source": [
+    "## Adding example rows from each table\n",
+    "Sometimes, the format of the data is not obvious and it is optimal to include a sample of rows from the tables in the prompt to allow the LLM to understand the data before providing a final query. Here we will use this feature to let the LLM know that artists are saved with their full names by providing two rows from the `Track` table."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "9a22ee47",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db = SQLDatabase.from_uri(\n",
+    "    \"sqlite:///../../../../notebooks/Chinook.db\", \n",
+    "    include_tables=['Track'], # we include only one table to save tokens in the prompt :)\n",
+    "    sample_rows_in_table_info=2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "952c0b4d",
+   "metadata": {},
+   "source": [
+    "The sample rows are added to the prompt after each corresponding table's column information:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "9de86267",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Table 'Track' has columns: TrackId (INTEGER), Name (NVARCHAR(200)), AlbumId (INTEGER), MediaTypeId (INTEGER), GenreId (INTEGER), Composer (NVARCHAR(220)), Milliseconds (INTEGER), Bytes (INTEGER), UnitPrice (NUMERIC(10, 2)). Here is an example of 2 rows from this table (long strings are truncated):\n",
+      "1 For Those About To Rock (We Salute You) 1 1 1 Angus Young, Malcolm Young, Brian Johnson 343719 11170334 0.99\n",
+      "2 Balls to the Wall 2 2 1 None 342562 5510424 0.99\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(db.table_info)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "bcb7a489",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db_chain = SQLDatabaseChain(llm=llm, database=db, verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "81e05d82",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
+      "What are some example tracks by Bach? \n",
+      "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT Name, Composer FROM Track WHERE Composer LIKE '%Bach%' LIMIT 5;\u001b[0m\n",
+      "SQLResult: \u001b[33;1m\u001b[1;3m[('American Woman', 'B. Cummings/G. Peterson/M.J. Kale/R. Bachman'), ('Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace', 'Johann Sebastian Bach'), ('Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria', 'Johann Sebastian Bach'), ('Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude', 'Johann Sebastian Bach'), ('Toccata and Fugue in D Minor, BWV 565: I. Toccata', 'Johann Sebastian Bach')]\u001b[0m\n",
+      "Answer:\u001b[32;1m\u001b[1;3m Some example tracks by Bach are 'American Woman', 'Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace', 'Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria', 'Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude', and 'Toccata and Fugue in D Minor, BWV 565: I. Toccata'.\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' Some example tracks by Bach are \\'American Woman\\', \\'Concerto for 2 Violins in D Minor, BWV 1043: I. Vivace\\', \\'Aria Mit 30 Veränderungen, BWV 988 \"Goldberg Variations\": Aria\\', \\'Suite for Solo Cello No. 1 in G Major, BWV 1007: I. Prélude\\', and \\'Toccata and Fugue in D Minor, BWV 565: I. Toccata\\'.'"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "db_chain.run(\"What are some example tracks by Bach?\")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "c12ae15a",
@@ -319,17 +478,13 @@
    "source": [
     "chain.run(\"How many employees are also customers?\")"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b2998b03",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
+  "@webio": {
+   "lastCommId": null,
+   "lastKernelId": null
+  },
   "kernelspec": {
    "display_name": "Python 3 (ipykernel)",
    "language": "python",
@@ -345,7 +500,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.9.2"
   }
  },
  "nbformat": 4,

docs/modules/chains/generic/from_hub.ipynb

@@ -0,0 +1,157 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "25c90e9e",
+   "metadata": {},
+   "source": [
+    "# Loading from LangChainHub\n",
+    "\n",
+    "This notebook covers how to load chains from [LangChainHub](https://github.com/hwchase17/langchain-hub)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "8b54479e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import load_chain\n",
+    "\n",
+    "chain = load_chain(\"lc://chains/llm-math/chain.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "4828f31f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
+      "whats 2 raised to .12\u001b[32;1m\u001b[1;3m\n",
+      "Answer: 1.0791812460476249\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Answer: 1.0791812460476249'"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"whats 2 raised to .12\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8db72cda",
+   "metadata": {},
+   "source": [
+    "Sometimes chains will require extra arguments that were not serialized with the chain. For example, a chain that does question answering over a vector database will require a vector database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "aab39528",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.vectorstores.faiss import FAISS\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain import OpenAI, VectorDBQA"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "16a85d5e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('../../state_of_the_union.txt') as f:\n",
+    "    state_of_the_union = f.read()\n",
+    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+    "texts = text_splitter.split_text(state_of_the_union)\n",
+    "\n",
+    "embeddings = OpenAIEmbeddings()\n",
+    "vectorstore = FAISS.from_texts(texts, embeddings)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "6a82e91e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = load_chain(\"lc://chains/vector-db-qa/stuff/chain.json\", vectorstore=vectorstore)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "efe9b25b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\" The president said that Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers, and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\""
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
+    "chain.run(query)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f910a32f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/chains/generic/llm.json

@@ -0,0 +1,13 @@
+{
+    "model_name": "text-davinci-003",
+    "temperature": 0.0,
+    "max_tokens": 256,
+    "top_p": 1,
+    "frequency_penalty": 0,
+    "presence_penalty": 0,
+    "n": 1,
+    "best_of": 1,
+    "request_timeout": null,
+    "logit_bias": {},
+    "_type": "openai"
+}

docs/modules/chains/generic/llm_chain.ipynb

@@ -121,10 +121,51 @@
     "llm_chain.predict(adjective=\"sad\", subject=\"ducks\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "672f59d4",
+   "metadata": {},
+   "source": [
+    "## From string\n",
+    "You can also construct an LLMChain from a string template directly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "f8bc262e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "template = \"\"\"Write a {adjective} poem about {subject}.\"\"\"\n",
+    "llm_chain = LLMChain.from_string(llm=OpenAI(temperature=0), template=template)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "cb164a76",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"\\n\\nThe ducks swim in the pond,\\nTheir feathers so soft and warm,\\nBut they can't help but feel so forlorn.\\n\\nTheir quacks echo in the air,\\nBut no one is there to hear,\\nFor they have no one to share.\\n\\nThe ducks paddle around in circles,\\nTheir heads hung low in despair,\\nFor they have no one to care.\\n\\nThe ducks look up to the sky,\\nBut no one is there to see,\\nFor they have no one to be.\\n\\nThe ducks drift away in the night,\\nTheir hearts filled with sorrow and pain,\\nFor they have no one to gain.\""
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "llm_chain.predict(adjective=\"sad\", subject=\"ducks\")"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8310cdaa",
+   "id": "9f0adbc7",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/chains/generic/llm_chain.json

@@ -0,0 +1,27 @@
+{
+    "memory": null,
+    "verbose": true,
+    "prompt": {
+        "input_variables": [
+            "question"
+        ],
+        "output_parser": null,
+        "template": "Question: {question}\n\nAnswer: Let's think step by step.",
+        "template_format": "f-string"
+    },
+    "llm": {
+        "model_name": "text-davinci-003",
+        "temperature": 0.0,
+        "max_tokens": 256,
+        "top_p": 1,
+        "frequency_penalty": 0,
+        "presence_penalty": 0,
+        "n": 1,
+        "best_of": 1,
+        "request_timeout": null,
+        "logit_bias": {},
+        "_type": "openai"
+    },
+    "output_key": "text",
+    "_type": "llm_chain"
+}

docs/modules/chains/generic/llm_chain_separate.json

@@ -0,0 +1,8 @@
+{
+  "memory": null,
+  "verbose": true,
+  "prompt_path": "prompt.json",
+  "llm_path": "llm.json",
+  "output_key": "text",
+  "_type": "llm_chain"
+}

docs/modules/chains/generic/prompt.json

@@ -0,0 +1,8 @@
+{
+    "input_variables": [
+        "question"
+    ],
+    "output_parser": null,
+    "template": "Question: {question}\n\nAnswer: Let's think step by step.",
+    "template_format": "f-string"
+}

docs/modules/chains/generic/serialization.ipynb

@@ -0,0 +1,376 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cbe47c3a",
+   "metadata": {},
+   "source": [
+    "# Serialization\n",
+    "This notebook covers how to serialize chains to and from disk. The serialization format we use is json or yaml. Currently, only some chains support this type of serialization. We will grow the number of supported chains over time.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4a8a447",
+   "metadata": {},
+   "source": [
+    "## Saving a chain to disk\n",
+    "First, let's go over how to save a chain to disk. This can be done with the `.save` method, and specifying a file path with a json or yaml extension."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "26e28451",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain import PromptTemplate, OpenAI, LLMChain\n",
+    "template = \"\"\"Question: {question}\n",
+    "\n",
+    "Answer: Let's think step by step.\"\"\"\n",
+    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=OpenAI(temperature=0), verbose=True)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "bfa18e1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_chain.save(\"llm_chain.json\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ea82665d",
+   "metadata": {},
+   "source": [
+    "Let's now take a look at what's inside this saved file"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "0fd33328",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      "    \"memory\": null,\r\n",
+      "    \"verbose\": true,\r\n",
+      "    \"prompt\": {\r\n",
+      "        \"input_variables\": [\r\n",
+      "            \"question\"\r\n",
+      "        ],\r\n",
+      "        \"output_parser\": null,\r\n",
+      "        \"template\": \"Question: {question}\\n\\nAnswer: Let's think step by step.\",\r\n",
+      "        \"template_format\": \"f-string\"\r\n",
+      "    },\r\n",
+      "    \"llm\": {\r\n",
+      "        \"model_name\": \"text-davinci-003\",\r\n",
+      "        \"temperature\": 0.0,\r\n",
+      "        \"max_tokens\": 256,\r\n",
+      "        \"top_p\": 1,\r\n",
+      "        \"frequency_penalty\": 0,\r\n",
+      "        \"presence_penalty\": 0,\r\n",
+      "        \"n\": 1,\r\n",
+      "        \"best_of\": 1,\r\n",
+      "        \"request_timeout\": null,\r\n",
+      "        \"logit_bias\": {},\r\n",
+      "        \"_type\": \"openai\"\r\n",
+      "    },\r\n",
+      "    \"output_key\": \"text\",\r\n",
+      "    \"_type\": \"llm_chain\"\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "!cat llm_chain.json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2012c724",
+   "metadata": {},
+   "source": [
+    "## Loading a chain from disk\n",
+    "We can load a chain from disk by using the `load_chain` method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "342a1974",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import load_chain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "394b7da8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = load_chain(\"llm_chain.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "20d99787",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mQuestion: whats 2 + 2\n",
+      "\n",
+      "Answer: Let's think step by step.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' 2 + 2 = 4'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"whats 2 + 2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14449679",
+   "metadata": {},
+   "source": [
+    "## Saving components separately\n",
+    "In the above example, we can see that the prompt and llm configuration information is saved in the same json as the overall chain. Alternatively, we can split them up and save them separately. This is often useful to make the saved components more modular. In order to do this, we just need to specify `llm_path` instead of the `llm` component, and `prompt_path` instead of the `prompt` component."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "50ec35ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_chain.prompt.save(\"prompt.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "c48b39aa",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      "    \"input_variables\": [\r\n",
+      "        \"question\"\r\n",
+      "    ],\r\n",
+      "    \"output_parser\": null,\r\n",
+      "    \"template\": \"Question: {question}\\n\\nAnswer: Let's think step by step.\",\r\n",
+      "    \"template_format\": \"f-string\"\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "!cat prompt.json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "13c92944",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_chain.llm.save(\"llm.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1b815f89",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      "    \"model_name\": \"text-davinci-003\",\r\n",
+      "    \"temperature\": 0.0,\r\n",
+      "    \"max_tokens\": 256,\r\n",
+      "    \"top_p\": 1,\r\n",
+      "    \"frequency_penalty\": 0,\r\n",
+      "    \"presence_penalty\": 0,\r\n",
+      "    \"n\": 1,\r\n",
+      "    \"best_of\": 1,\r\n",
+      "    \"request_timeout\": null,\r\n",
+      "    \"logit_bias\": {},\r\n",
+      "    \"_type\": \"openai\"\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "!cat llm.json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "7e6aa9ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config = {\n",
+    "    \"memory\": None,\n",
+    "    \"verbose\": True,\n",
+    "    \"prompt_path\": \"prompt.json\",\n",
+    "    \"llm_path\": \"llm.json\",\n",
+    "    \"output_key\": \"text\",\n",
+    "    \"_type\": \"llm_chain\"\n",
+    "}\n",
+    "import json\n",
+    "with open(\"llm_chain_separate.json\", \"w\") as f:\n",
+    "    json.dump(config, f, indent=2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "8e959ca6",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\r\n",
+      "  \"memory\": null,\r\n",
+      "  \"verbose\": true,\r\n",
+      "  \"prompt_path\": \"prompt.json\",\r\n",
+      "  \"llm_path\": \"llm.json\",\r\n",
+      "  \"output_key\": \"text\",\r\n",
+      "  \"_type\": \"llm_chain\"\r\n",
+      "}"
+     ]
+    }
+   ],
+   "source": [
+    "!cat llm_chain_separate.json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "662731c0",
+   "metadata": {},
+   "source": [
+    "We can then load it in the same way"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "d69ceb93",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = load_chain(\"llm_chain_separate.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "a99d61b9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mQuestion: whats 2 + 2\n",
+      "\n",
+      "Answer: Let's think step by step.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' 2 + 2 = 4'"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"whats 2 + 2\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "822b7c12",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/chains/how_to_guides.rst

@@ -9,6 +9,7 @@ They are broken up into three categories:
 1. `Generic Chains <./generic_how_to.html>`_: Generic chains, that are meant to help build other chains rather than serve a particular purpose.
 2. `CombineDocuments Chains <./combine_docs_how_to.html>`_: Chains aimed at making it easy to work with documents (question answering, summarization, etc).
 3. `Utility Chains <./utility_how_to.html>`_: Chains consisting of an LLMChain interacting with a specific util.
+4. `Asynchronous <./async_chain.html>`_: Covering asynchronous functionality.
 
 .. toctree::
    :maxdepth: 1
@@ -18,3 +19,7 @@ They are broken up into three categories:
    ./generic_how_to.rst
    ./combine_docs_how_to.rst
    ./utility_how_to.rst
+
+In addition to different types of chains, we also have the following how-to guides for working with chains in general:
+
+`Load From Hub <./generic/from_hub.html>`_: This notebook covers how to load chains from `LangChainHub <https://github.com/hwchase17/langchain-hub>`_.

docs/modules/document_loaders.rst

@@ -0,0 +1,29 @@
+Document Loaders
+==========================
+
+Combining language models with your own text data is a powerful way to differentiate them.
+The first step in doing this is to load the data into "documents" - a fancy way of say some pieces of text.
+This module is aimed at making this easy.
+
+A primary driver of a lot of this is the `Unstructured <https://github.com/Unstructured-IO/unstructured>`_ python package.
+This package is a great way to transform all types of files - text, powerpoint, images, html, pdf, etc - into text data.
+
+For detailed instructions on how to get set up with Unstructured, see installation guidelines `here <https://github.com/Unstructured-IO/unstructured#coffee-getting-started>`_.
+
+The following sections of documentation are provided:
+
+- `Key Concepts <./document_loaders/key_concepts.html>`_: A conceptual guide going over the various concepts related to loading documents.
+
+- `How-To Guides <./document_loaders/how_to_guides.html>`_: A collection of how-to guides. These highlight different types of loaders.
+
+
+
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Document Loaders
+   :name: Document Loaders
+   :hidden:
+
+   ./document_loaders/key_concepts.md
+   ./document_loaders/how_to_guides.rst

docs/modules/document_loaders/examples/airbyte_json.ipynb

@@ -0,0 +1,171 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1f3a5ebf",
+   "metadata": {},
+   "source": [
+    "# Airbyte JSON\n",
+    "This covers how to load any source from Airbyte into a local JSON file that can be read in as a document\n",
+    "\n",
+    "Prereqs:\n",
+    "Have docker desktop installed\n",
+    "\n",
+    "Steps:\n",
+    "\n",
+    "1) clone Airbyte from GitHub - `git clone https://github.com/airbytehq/airbyte.git`\n",
+    "\n",
+    "2) switch into Airbyte directory - `cd airbyte`\n",
+    "\n",
+    "3) start Airbyte - `docker compose up`\n",
+    "\n",
+    "4) In your browser, just visit http://localhost:8000. You will be asked for a username and password. By default, that's username `airbyte` and password `password`.\n",
+    "\n",
+    "5) Setup any source you wish\n",
+    "\n",
+    "6) Set destination as Local JSON, with specified destination path - lets say `/json_data`. Set up manual sync.\n",
+    "\n",
+    "7) Run the connection!\n",
+    "\n",
+    "7) To see what files are create, you can navigate to: `file:///tmp/airbyte_local`\n",
+    "\n",
+    "8) Find your data and copy path. That path should be saved in the file variable below. It should start with `/tmp/airbyte_local`\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "180c8b74",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import AirbyteJSONLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "4af10665",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "_airbyte_raw_pokemon.jsonl\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!ls /tmp/airbyte_local/json_data/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "721d9316",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = AirbyteJSONLoader('/tmp/airbyte_local/json_data/_airbyte_raw_pokemon.jsonl')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "9858b946",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "fca024cb",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "abilities: \n",
+      "ability: \n",
+      "name: blaze\n",
+      "url: https://pokeapi.co/api/v2/ability/66/\n",
+      "\n",
+      "is_hidden: False\n",
+      "slot: 1\n",
+      "\n",
+      "\n",
+      "ability: \n",
+      "name: solar-power\n",
+      "url: https://pokeapi.co/api/v2/ability/94/\n",
+      "\n",
+      "is_hidden: True\n",
+      "slot: 3\n",
+      "\n",
+      "base_experience: 267\n",
+      "forms: \n",
+      "name: charizard\n",
+      "url: https://pokeapi.co/api/v2/pokemon-form/6/\n",
+      "\n",
+      "game_indices: \n",
+      "game_index: 180\n",
+      "version: \n",
+      "name: red\n",
+      "url: https://pokeapi.co/api/v2/version/1/\n",
+      "\n",
+      "\n",
+      "\n",
+      "game_index: 180\n",
+      "version: \n",
+      "name: blue\n",
+      "url: https://pokeapi.co/api/v2/version/2/\n",
+      "\n",
+      "\n",
+      "\n",
+      "game_index: 180\n",
+      "version: \n",
+      "n\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(data[0].page_content[:500])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9fa002a5",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/azlyrics.ipynb

@@ -0,0 +1,93 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9c31caff",
+   "metadata": {},
+   "source": [
+    "# AZLyrics\n",
+    "This covers how to load AZLyrics webpages into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "7e6f5726",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import AZLyricsLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a0df4c24",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = AZLyricsLoader(\"https://www.azlyrics.com/lyrics/mileycyrus/flowers.html\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "8cd61b6e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "162fd286",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content=\"Miley Cyrus - Flowers Lyrics | AZLyrics.com\\n\\r\\nWe were good, we were gold\\nKinda dream that can't be sold\\nWe were right till we weren't\\nBuilt a home and watched it burn\\n\\nI didn't wanna leave you\\nI didn't wanna lie\\nStarted to cry but then remembered I\\n\\nI can buy myself flowers\\nWrite my name in the sand\\nTalk to myself for hours\\nSay things you don't understand\\nI can take myself dancing\\nAnd I can hold my own hand\\nYeah, I can love me better than you can\\n\\nCan love me better\\nI can love me better, baby\\nCan love me better\\nI can love me better, baby\\n\\nPaint my nails, cherry red\\nMatch the roses that you left\\nNo remorse, no regret\\nI forgive every word you said\\n\\nI didn't wanna leave you, baby\\nI didn't wanna fight\\nStarted to cry but then remembered I\\n\\nI can buy myself flowers\\nWrite my name in the sand\\nTalk to myself for hours, yeah\\nSay things you don't understand\\nI can take myself dancing\\nAnd I can hold my own hand\\nYeah, I can love me better than you can\\n\\nCan love me better\\nI can love me better, baby\\nCan love me better\\nI can love me better, baby\\nCan love me better\\nI can love me better, baby\\nCan love me better\\nI\\n\\nI didn't wanna wanna leave you\\nI didn't wanna fight\\nStarted to cry but then remembered I\\n\\nI can buy myself flowers\\nWrite my name in the sand\\nTalk to myself for hours (Yeah)\\nSay things you don't understand\\nI can take myself dancing\\nAnd I can hold my own hand\\nYeah, I can love me better than\\nYeah, I can love me better than you can, uh\\n\\nCan love me better\\nI can love me better, baby\\nCan love me better\\nI can love me better, baby (Than you can)\\nCan love me better\\nI can love me better, baby\\nCan love me better\\nI\\n\", lookup_str='', metadata={'source': 'https://www.azlyrics.com/lyrics/mileycyrus/flowers.html'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6358000c",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/directory_loader.ipynb

@@ -0,0 +1,101 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "79f24a6b",
+   "metadata": {},
+   "source": [
+    "# Directory Loader\n",
+    "This covers how to use the DirectoryLoader to load all documents in a directory. Under the hood, this uses the [UnstructuredLoader](./unstructured_file.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "019d8520",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import DirectoryLoader"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0c76cdc5",
+   "metadata": {},
+   "source": [
+    "We can use the `glob` parameter to control which files to load. Note that here it doesn't load the `.rst` file or the `.ipynb` files."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "891fe56f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = DirectoryLoader('../', glob=\"**/*.md\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "addfe9cf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "b042086d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "1"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(docs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cbc8256b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/email.ipynb

@@ -0,0 +1,94 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9fdbd55d",
+   "metadata": {},
+   "source": [
+    "# Email\n",
+    "\n",
+    "This notebook shows how to load email (`.eml`) files."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "40cd9806",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredEmailLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2d20b852",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredEmailLoader('example_data/fake-email.eml')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "579fa702",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "90c1d899",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='This is a test email to use for unit tests.\\n\\nImportant points:\\n\\nRoses are red\\n\\nViolets are blue', lookup_str='', metadata={'source': 'example_data/fake-email.eml'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4ef9a5f4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/everynote.ipynb

@@ -0,0 +1,80 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "56ac1584",
+   "metadata": {},
+   "source": [
+    "# EveryNote\n",
+    "\n",
+    "How to load EveryNote file from disk."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "1a53ece0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# !pip install pypandoc\n",
+    "# import pypandoc\n",
+    "\n",
+    "# pypandoc.download_pandoc()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "88df766f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='testing this\\n\\nwhat happens?\\n\\nto the world?\\n', lookup_str='', metadata={'source': 'example_data/testing.enex'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain.document_loaders import EveryNoteLoader\n",
+    "\n",
+    "loader = EveryNoteLoader(\"example_data/testing.enex\")\n",
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c1329905",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/example_data/fake-content.html

@@ -0,0 +1,9 @@
+<!DOCTYPE html>
+<html>
+<body>
+
+<h1>My First Heading</h1>
+<p>My first paragraph.</p>
+
+</body>
+</html>

docs/modules/document_loaders/examples/example_data/fake-email.eml

@@ -0,0 +1,20 @@
+MIME-Version: 1.0
+Date: Fri, 16 Dec 2022 17:04:16 -0500
+Message-ID: <CADc-_xaLB2FeVQ7mNsoX+NJb_7hAJhBKa_zet-rtgPGenj0uVw@mail.gmail.com>
+Subject: Test Email
+From: Matthew Robinson <mrobinson@unstructured.io>
+To: Matthew Robinson <mrobinson@unstructured.io>
+Content-Type: multipart/alternative; boundary="00000000000095c9b205eff92630"
+
+--00000000000095c9b205eff92630
+Content-Type: text/plain; charset="UTF-8"
+This is a test email to use for unit tests.
+Important points:
+   - Roses are red
+   - Violets are blue
+--00000000000095c9b205eff92630
+Content-Type: text/html; charset="UTF-8"
+
+<div dir="ltr"><div>This is a test email to use for unit tests.</div><div><br></div><div>Important points:</div><div><ul><li>Roses are red</li><li>Violets are blue</li></ul></div></div>
+
+--00000000000095c9b205eff92630--

docs/modules/document_loaders/examples/example_data/testing.enex

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE en-export SYSTEM "http://xml.evernote.com/pub/evernote-export4.dtd">
+<en-export export-date="20230309T035336Z" application="Evernote" version="10.53.2">
+  <note>
+    <title>testing</title>
+    <created>20230209T034746Z</created>
+    <updated>20230209T035328Z</updated>
+    <note-attributes>
+      <author>Harrison Chase</author>
+    </note-attributes>
+    <content>
+      <![CDATA[<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"><en-note><div>testing this</div><div>what happens?</div><div>to the world?</div></en-note>      ]]>
+    </content>
+  </note>
+</en-export>

docs/modules/document_loaders/examples/gcs_directory.ipynb

@@ -0,0 +1,156 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0ef41fd4",
+   "metadata": {},
+   "source": [
+    "# GCS Directory\n",
+    "\n",
+    "This covers how to load document objects from an Google Cloud Storage (GCS) directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5cfb25c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import GCSDirectoryLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "93a4d0f1",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "# !pip install google-cloud-storage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "633dc839",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = GCSDirectoryLoader(project_name=\"aist\", bucket=\"testing-hwc\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "a863467d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
+      "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n",
+      "/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
+      "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpz37njh7u/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "17c0dcbb",
+   "metadata": {},
+   "source": [
+    "## Specifying a prefix\n",
+    "You can also specify a prefix for more finegrained control over what files to load."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "b3143c89",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = GCSDirectoryLoader(project_name=\"aist\", bucket=\"testing-hwc\", prefix=\"fake\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "226ac6f5",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
+      "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n",
+      "/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
+      "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpylg6291i/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f9c0734f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/gcs_file.ipynb

@@ -0,0 +1,104 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0ef41fd4",
+   "metadata": {},
+   "source": [
+    "# GCS File Storage\n",
+    "\n",
+    "This covers how to load document objects from an Google Cloud Storage (GCS) file object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5cfb25c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import GCSFileLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "93a4d0f1",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "# !pip install google-cloud-storage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "633dc839",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = GCSFileLoader(project_name=\"aist\", bucket=\"testing-hwc\", blob=\"fake.docx\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "a863467d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/harrisonchase/workplace/langchain/.venv/lib/python3.10/site-packages/google/auth/_default.py:83: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
+      "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmp3srlf8n8/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "eba3002d",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/googledrive.ipynb

@@ -0,0 +1,84 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "b0ed136e-6983-4893-ae1b-b75753af05f8",
+   "metadata": {},
+   "source": [
+    "# Google Drive\n",
+    "This notebook covers how to load documents from Google Drive. Currently, only Google Docs are supported.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "1. Create a Google Cloud project or use an existing project\n",
+    "1. Enable the [Google Drive API](https://console.cloud.google.com/flows/enableapi?apiid=drive.googleapis.com)\n",
+    "1. [Authorize credentials for desktop app](https://developers.google.com/drive/api/quickstart/python#authorize_credentials_for_a_desktop_application)\n",
+    "1. `pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib`\n",
+    "\n",
+    "## 🧑 Instructions for ingesting your Google Docs data\n",
+    "By default, the `GoogleDriveLoader` expects the `credentials.json` file to be `~/.credentials/credentials.json`, but this is configurable using the `credentials_file` keyword argument. Same thing with `token.json`. Note that `token.json` will be created automatically the first time you use the loader.\n",
+    "\n",
+    "`GoogleDriveLoader` can load from a list of Google Docs document ids or a folder id. You can obtain your folder and document id from the URL:\n",
+    "* Folder: https://drive.google.com/drive/u/0/folders/1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5 -> folder id is `\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\"`\n",
+    "* Document: https://docs.google.com/document/d/1bfaMQ18_i56204VaQDVeAFpqEijJTgvurupdEDiaUQw/edit -> document id is `\"1bfaMQ18_i56204VaQDVeAFpqEijJTgvurupdEDiaUQw\"`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "878928a6-a5ae-4f74-b351-64e3b01733fe",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import GoogleDriveLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2216c83f-68e4-4d2f-8ea2-5878fb18bbe7",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "loader = GoogleDriveLoader(folder_id=\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "8f3b6aa0-b45d-4e37-8c50-5bebe70fdb9d",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/gutenberg.ipynb

@@ -0,0 +1,83 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bda1f3f5",
+   "metadata": {},
+   "source": [
+    "# Gutenberg\n",
+    "\n",
+    "This covers how to load links to Gutenberg e-books into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9bfd5e46",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import GutenbergLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "700e4ef2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = GutenbergLoader('https://www.gutenberg.org/cache/epub/69972/pg69972.txt')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "b6f28930",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7d436441",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3b74d755",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/html.ipynb

@@ -0,0 +1,94 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "2dfc4698",
+   "metadata": {},
+   "source": [
+    "# HTML\n",
+    "\n",
+    "This covers how to load HTML documents into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "24b434b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredHTMLLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "00f46fda",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredHTMLLoader(\"example_data/fake-content.html\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "b68a26b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "34de48fa",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='My First Heading\\n\\nMy first paragraph.', lookup_str='', metadata={'source': 'example_data/fake-content.html'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "79b1bce4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/microsoft_word.ipynb

@@ -0,0 +1,94 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "34c90eed",
+   "metadata": {},
+   "source": [
+    "# Microsoft Word\n",
+    "\n",
+    "This notebook shows how to load text from Microsoft word documents."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "28ded768",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredDocxLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "f1f26035",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredDocxLoader('example_data/fake.docx')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "2c87dde9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "0e4a884c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': 'example_data/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "61953c83",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/notion.ipynb

@@ -0,0 +1,82 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1dc7df1d",
+   "metadata": {},
+   "source": [
+    "# Notion\n",
+    "This notebook covers how to load documents from a Notion database dump.\n",
+    "\n",
+    "In order to get this notion dump, follow these instructions:\n",
+    "\n",
+    "## 🧑 Instructions for ingesting your own dataset\n",
+    "\n",
+    "Export your dataset from Notion. You can do this by clicking on the three dots in the upper right hand corner and then clicking `Export`.\n",
+    "\n",
+    "When exporting, make sure to select the `Markdown & CSV` format option.\n",
+    "\n",
+    "This will produce a `.zip` file in your Downloads folder. Move the `.zip` file into this repository.\n",
+    "\n",
+    "Run the following command to unzip the zip file (replace the `Export...` with your own file name as needed).\n",
+    "\n",
+    "```shell\n",
+    "unzip Export-d3adfe0f-3131-4bf3-8987-a52017fc1bae.zip -d Notion_DB\n",
+    "```\n",
+    "\n",
+    "Run the following command to ingest the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "007c5cbf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import NotionDirectoryLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1caec59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = NotionDirectoryLoader(\"Notion_DB\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1c30ff7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/obsidian.ipynb

@@ -0,0 +1,66 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1dc7df1d",
+   "metadata": {},
+   "source": [
+    "# Obsidian\n",
+    "This notebook covers how to load documents from an Obsidian database.\n",
+    "\n",
+    "Since Obsidian is just stored on disk as a folder of Markdown files, the loader just takes a path to this directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "007c5cbf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import ObsidianLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1caec59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = ObsidianLoader(\"<path-to-obsidian>\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1c30ff7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/pdf.ipynb

@@ -0,0 +1,200 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f70e6118",
+   "metadata": {},
+   "source": [
+    "# PDF\n",
+    "\n",
+    "This covers how to load pdfs into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "743f9413",
+   "metadata": {},
+   "source": [
+    "## Using PyPDF\n",
+    "\n",
+    "Allows for tracking of page numbers as well."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "c428b0c5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import PagedPDFSplitter\n",
+    "\n",
+    "loader = PagedPDFSplitter(\"example_data/layout-parser-paper.pdf\")\n",
+    "pages = loader.load_and_split()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ebd895e4",
+   "metadata": {},
+   "source": [
+    "An advantage of this approach is that documents can be retrieved with page numbers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "87fa7b3a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "9: 10 Z. Shen et al.\n",
+      "Fig. 4: Illustration of (a) the original historical Japanese document with layout\n",
+      "detection results and (b) a recreated version of the document image that achieves\n",
+      "much better character recognition recall. The reorganization algorithm rearranges\n",
+      "the tokens based on the their detected bounding boxes given a maximum allowed\n",
+      "height.\n",
+      "4LayoutParser Community Platform\n",
+      "Another focus of LayoutParser is promoting the reusability of layout detection\n",
+      "models and full digitization pipelines. Similar to many existing deep learning\n",
+      "libraries, LayoutParser comes with a community model hub for distributing\n",
+      "layout models. End-users can upload their self-trained models to the model hub,\n",
+      "and these models can be loaded into a similar interface as the currently available\n",
+      "LayoutParser pre-trained models. For example, the model trained on the News\n",
+      "Navigator dataset [17] has been incorporated in the model hub.\n",
+      "Beyond DL models, LayoutParser also promotes the sharing of entire doc-\n",
+      "ument digitization pipelines. For example, sometimes the pipeline requires the\n",
+      "combination of multiple DL models to achieve better accuracy. Currently, pipelines\n",
+      "are mainly described in academic papers and implementations are often not pub-\n",
+      "licly available. To this end, the LayoutParser community platform also enables\n",
+      "the sharing of layout pipelines to promote the discussion and reuse of techniques.\n",
+      "For each shared pipeline, it has a dedicated project page, with links to the source\n",
+      "code, documentation, and an outline of the approaches. A discussion panel is\n",
+      "provided for exchanging ideas. Combined with the core LayoutParser library,\n",
+      "users can easily build reusable components based on the shared pipelines and\n",
+      "apply them to solve their unique problems.\n",
+      "5 Use Cases\n",
+      "The core objective of LayoutParser is to make it easier to create both large-scale\n",
+      "and light-weight document digitization pipelines. Large-scale document processing\n",
+      "3: 4 Z. Shen et al.\n",
+      "Efficient Data AnnotationC u s t o m i z e d  M o d e l  T r a i n i n gModel Cust omizationDI A Model HubDI A Pipeline SharingCommunity PlatformLa y out Detection ModelsDocument Images \n",
+      "T h e  C o r e  L a y o u t P a r s e r  L i b r a r yOCR ModuleSt or age & VisualizationLa y out Data Structur e\n",
+      "Fig. 1: The overall architecture of LayoutParser . For an input document image,\n",
+      "the core LayoutParser library provides a set of o\u000b",
+      "-the-shelf tools for layout\n",
+      "detection, OCR, visualization, and storage, backed by a carefully designed layout\n",
+      "data structure. LayoutParser also supports high level customization via e\u000ecient\n",
+      "layout annotation and model training functions. These improve model accuracy\n",
+      "on the target samples. The community platform enables the easy sharing of DIA\n",
+      "models and whole digitization pipelines to promote reusability and reproducibility.\n",
+      "A collection of detailed documentation, tutorials and exemplar projects make\n",
+      "LayoutParser easy to learn and use.\n",
+      "AllenNLP [ 8] and transformers [ 34] have provided the community with complete\n",
+      "DL-based support for developing and deploying models for general computer\n",
+      "vision and natural language processing problems. LayoutParser , on the other\n",
+      "hand, specializes speci\f",
+      "cally in DIA tasks. LayoutParser is also equipped with a\n",
+      "community platform inspired by established model hubs such as Torch Hub [23]\n",
+      "andTensorFlow Hub [1]. It enables the sharing of pretrained models as well as\n",
+      "full document processing pipelines that are unique to DIA tasks.\n",
+      "There have been a variety of document data collections to facilitate the\n",
+      "development of DL models. Some examples include PRImA [ 3](magazine layouts),\n",
+      "PubLayNet [ 38](academic paper layouts), Table Bank [ 18](tables in academic\n",
+      "papers), Newspaper Navigator Dataset [ 16,17](newspaper \f",
+      "gure layouts) and\n",
+      "HJDataset [31](historical Japanese document layouts). A spectrum of models\n",
+      "trained on these datasets are currently available in the LayoutParser model zoo\n",
+      "to support di\u000b",
+      "erent use cases.\n",
+      "3 The Core LayoutParser Library\n",
+      "At the core of LayoutParser is an o\u000b",
+      "-the-shelf toolkit that streamlines DL-\n",
+      "based document image analysis. Five components support a simple interface\n",
+      "with comprehensive functionalities: 1) The layout detection models enable using\n",
+      "pre-trained or self-trained DL models for layout detection with just four lines\n",
+      "of code. 2) The detected layout information is stored in carefully engineered\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.vectorstores import FAISS\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "\n",
+    "faiss_index = FAISS.from_documents(pages, OpenAIEmbeddings())\n",
+    "docs = faiss_index.similarity_search(\"How will the community be engaged?\", k=2)\n",
+    "for doc in docs:\n",
+    "    print(str(doc.metadata[\"page\"]) + \":\", doc.page_content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "09d64998",
+   "metadata": {},
+   "source": [
+    "## Using Unstructured"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "0cc0cd42",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredPDFLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "082d557c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredPDFLoader(\"example_data/layout-parser-paper.pdf\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "5c41106f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "54fb6b62",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/powerpoint.ipynb

@@ -0,0 +1,94 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "39af9ecd",
+   "metadata": {},
+   "source": [
+    "# PowerPoint\n",
+    "\n",
+    "This covers how to load PowerPoint documents into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "721c48aa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredPowerPointLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "9d3d0e35",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredPowerPointLoader(\"example_data/fake-power-point.pptx\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "06073f91",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "c9adc5cb",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Adding a Bullet Slide\\n\\nFind the bullet slide layout\\n\\nUse _TextFrame.text for first bullet\\n\\nUse _TextFrame.add_paragraph() for subsequent bullets\\n\\nHere is a lot of text!\\n\\nHere is some text in a text box!', lookup_str='', metadata={'source': 'example_data/fake-power-point.pptx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0c55f1cf",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/readthedocs_documentation.ipynb

@@ -0,0 +1,78 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "17812129",
+   "metadata": {},
+   "source": [
+    "# ReadTheDocs Documentation\n",
+    "This notebook covers how to load content from html that was generated as part of a Read-The-Docs build.\n",
+    "\n",
+    "For an example of this in the wild, see [here](https://github.com/hwchase17/chat-langchain).\n",
+    "\n",
+    "This assumes that the html has already been scraped into a folder. This can be done by uncommenting and running the following command"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "84696e27",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!wget -r -A.html -P rtdocs https://langchain.readthedocs.io/en/latest/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "92dd950b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import ReadTheDocsLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "494567c3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = ReadTheDocsLoader(\"rtdocs\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e2e6d6f0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/roam.ipynb

@@ -0,0 +1,78 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1dc7df1d",
+   "metadata": {},
+   "source": [
+    "# Roam\n",
+    "This notebook covers how to load documents from a Roam database. This takes a lot of inspiration from the example repo [here](https://github.com/JimmyLv/roam-qa).\n",
+    "\n",
+    "## 🧑 Instructions for ingesting your own dataset\n",
+    "\n",
+    "Export your dataset from Roam Research. You can do this by clicking on the three dots in the upper right hand corner and then clicking `Export`.\n",
+    "\n",
+    "When exporting, make sure to select the `Markdown & CSV` format option.\n",
+    "\n",
+    "This will produce a `.zip` file in your Downloads folder. Move the `.zip` file into this repository.\n",
+    "\n",
+    "Run the following command to unzip the zip file (replace the `Export...` with your own file name as needed).\n",
+    "\n",
+    "```shell\n",
+    "unzip Roam-Export-1675782732639.zip -d Roam_DB\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "007c5cbf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import RoamLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1caec59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = ObsidianLoader(\"Roam_DB\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1c30ff7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/s3_directory.ipynb

@@ -0,0 +1,134 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a634365e",
+   "metadata": {},
+   "source": [
+    "# s3 Directory\n",
+    "\n",
+    "This covers how to load document objects from an s3 directory object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "2f0cd6a5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import S3DirectoryLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "49815096",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install boto3"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "321cc7f1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = S3DirectoryLoader(\"testing-hwc\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "2b11d155",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpaa9xl6ch/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0690c40a",
+   "metadata": {},
+   "source": [
+    "## Specifying a prefix\n",
+    "You can also specify a prefix for more finegrained control over what files to load."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "72d44781",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = S3DirectoryLoader(\"testing-hwc\", prefix=\"fake\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "2d3c32db",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpujbkzf_l/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "885dc280",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/s3_file.ipynb

@@ -0,0 +1,94 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "66a7777e",
+   "metadata": {},
+   "source": [
+    "# s3 File\n",
+    "\n",
+    "This covers how to load document objects from an s3 file object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9ec8a3b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import S3FileLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "43128d8d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install boto3"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "35d6809a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = S3FileLoader(\"testing-hwc\", \"fake.docx\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "efd6be84",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpxvave6wl/fake.docx'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "93689594",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/unstructured_file.ipynb

@@ -0,0 +1,72 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "20deed05",
+   "metadata": {},
+   "source": [
+    "# Unstructured File Loader\n",
+    "This notebook covers how to use Unstructured to load files of many types. Unstructured currently supports loading of text files, powerpoints, html, pdfs, images, and more."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "79d3e549",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import UnstructuredFileLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2593d1dc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredFileLoader(\"../../state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "fe34e941",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "24e577e5",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/url.ipynb

@@ -0,0 +1,78 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "2dfc4698",
+   "metadata": {},
+   "source": [
+    "# URL\n",
+    "\n",
+    "This covers how to load HTML documents from a list of URLs into a document format that we can use downstream."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "16c3699e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    " from langchain.document_loaders import UnstructuredURLLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "836fbac1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "urls = [\n",
+    "    \"https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023\",\n",
+    "    \"https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023\"\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "00f46fda",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = UnstructuredURLLoader(urls=urls)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "b68a26b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/examples/youtube.ipynb

@@ -0,0 +1,137 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "df770c72",
+   "metadata": {},
+   "source": [
+    "# YouTube\n",
+    "\n",
+    "How to load documents from YouTube transcripts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "da4a867f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import YoutubeLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "34a25b57",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "# !pip install youtube-transcript-api"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "bc8b308a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = YoutubeLoader.from_youtube_url(\"https://www.youtube.com/watch?v=QsYGlZkevEg\", add_video_info=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "d073dd36",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='LADIES AND GENTLEMEN, PEDRO PASCAL! [ CHEERS AND APPLAUSE ] >> THANK YOU, THANK YOU. THANK YOU VERY MUCH. I\\'M SO EXCITED TO BE HERE. THANK YOU. I SPENT THE LAST YEAR SHOOTING A SHOW CALLED \"THE LAST OF US\" ON HBO. FOR SOME HBO SHOES, YOU GET TO SHOOT IN A FIVE STAR ITALIAN RESORT SURROUNDED BY BEAUTIFUL PEOPLE, BUT I SAID, NO, THAT\\'S TOO EASY. I WANT TO SHOOT IN A FREEZING CANADIAN FOREST WHILE BEING CHASED AROUND BY A GUY WHOSE HEAD LOOKS LIKE A GENITAL WART. IT IS AN HONOR BEING A PART OF THESE HUGE FRANCHISEs LIKE \"GAME OF THRONES\" AND \"STAR WARS,\" BUT I\\'M STILL GETTING USED TO PEOPLE RECOGNIZING ME. THE OTHER DAY, A GUY STOPPED ME ON THE STREET AND SAYS, MY SON LOVES \"THE MANDALORIAN\" AND THE NEXT THING I KNOW, I\\'M FACE TIMING WITH A 6-YEAR-OLD WHO HAS NO IDEA WHO I AM BECAUSE MY CHARACTER WEARS A MASK THE ENTIRE SHOW. THE GUY IS LIKE, DO THE MANDO VOICE, BUT IT\\'S LIKE A BEDROOM VOICE. WITHOUT THE MASK, IT JUST SOUNDS PORNY. PEOPLE WALKING BY ON THE STREET SEE ME WHISPERING TO A 6-YEAR-OLD KID. I CAN BRING YOU IN WARM, OR I CAN BRING YOU IN COLD. EVEN THOUGH I CAME TO THE U.S. WHEN I WAS LITTLE, I WAS BORN IN CHILE, AND I HAVE 34 FIRST COUSINS WHO ARE STILL THERE. THEY\\'RE VERY PROUD OF ME. I KNOW THEY\\'RE PROUD BECAUSE THEY GIVE MY PHONE NUMBER TO EVERY PERSON THEY MEET, WHICH MEANS EVERY DAY, SOMEONE IN SANTIAGO WILL TEXT ME STUFF LIKE, CAN YOU COME TO MY WEDDING, OR CAN YOU SING MY PRIEST HAPPY BIRTHDAY, OR IS BABY YODA MEAN IN REAL LIFE. SO I HAVE TO BE LIKE NO, NO, AND HIS NAME IS GROGU. BUT MY COUSINS WEREN\\'T ALWAYS SO PROUD. EARLY IN MY CAREER, I PLAYED SMALL PARTS IN EVERY CRIME SHOW. I EVEN PLAYED TWO DIFFERENT CHARACTERS ON \"LAW AND ORDER.\" TITO CABASSA WHO LOOKED LIKE THIS. AND ONE YEAR LATER, I PLAYED REGGIE LUCKMAN WHO LOOKS LIKE THIS. AND THAT, MY FRIENDS, IS CALLED RANGE. BUT IT IS AMAZING TO BE HERE, LIKE I SAID. I WAS BORN IN CHILE, AND NINE MONTHS LATER, MY PARENTS FLED AND BROUGHT ME AND MY SISTER TO THE U.S. THEY WERE SO BRAVE, AND WITHOUT THEM, I WOULDN\\'T BE HERE IN THIS WONDERFUL COUNTRY, AND I CERTAINLY WOULDN\\'T BE STANDING HERE WITH YOU ALL TONIGHT. SO TO ALL MY FAMILY WATCHING IN CHILE, I WANT TO SAY [ SPEAKING NON-ENGLISH ] WHICH MEANS, I LOVE YOU, I MISS YOU, AND STOP GIVING OUT MY PHONE NUMBER. WE\\'VE GOT AN AMAZING SHOW FOR YOU TONIGHT. COLDPLAY IS HERE, SO STICK', lookup_str='', metadata={'source': 'QsYGlZkevEg', 'title': 'Pedro Pascal Monologue - SNL', 'description': 'First-time host Pedro Pascal talks about filming The Last of Us and being recognized by fans.\\n\\nSaturday Night Live. Stream now on Peacock: https://pck.tv/3uQxh4q\\n\\nSubscribe to SNL: https://goo.gl/tUsXwM\\nStream Current Full Episodes: http://www.nbc.com/saturday-night-live\\n\\nWATCH PAST SNL SEASONS\\nGoogle Play - http://bit.ly/SNLGooglePlay\\niTunes - http://bit.ly/SNLiTunes\\n\\nSNL ON SOCIAL\\nSNL Instagram: http://instagram.com/nbcsnl\\nSNL Facebook: https://www.facebook.com/snl\\nSNL Twitter: https://twitter.com/nbcsnl\\nSNL TikTok: https://www.tiktok.com/@nbcsnl\\n\\nGET MORE NBC\\nLike NBC: http://Facebook.com/NBC\\nFollow NBC: http://Twitter.com/NBC\\nNBC Tumblr: http://NBCtv.tumblr.com/\\nYouTube: http://www.youtube.com/nbc\\nNBC Instagram: http://instagram.com/nbc\\n\\n#SNL #PedroPascal #SNL48 #Coldplay', 'view_count': 1175057, 'thumbnail_url': 'https://i.ytimg.com/vi/QsYGlZkevEg/sddefault.jpg', 'publish_date': datetime.datetime(2023, 2, 4, 0, 0), 'length': 224, 'author': 'Saturday Night Live'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6b278a1b",
+   "metadata": {},
+   "source": [
+    "## Add video info"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "ba28af69",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ! pip install pytube"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "9b8ea390",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = YoutubeLoader.from_youtube_url(\"https://www.youtube.com/watch?v=QsYGlZkevEg\", add_video_info=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "97b98e92",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='LADIES AND GENTLEMEN, PEDRO PASCAL! [ CHEERS AND APPLAUSE ] >> THANK YOU, THANK YOU. THANK YOU VERY MUCH. I\\'M SO EXCITED TO BE HERE. THANK YOU. I SPENT THE LAST YEAR SHOOTING A SHOW CALLED \"THE LAST OF US\" ON HBO. FOR SOME HBO SHOES, YOU GET TO SHOOT IN A FIVE STAR ITALIAN RESORT SURROUNDED BY BEAUTIFUL PEOPLE, BUT I SAID, NO, THAT\\'S TOO EASY. I WANT TO SHOOT IN A FREEZING CANADIAN FOREST WHILE BEING CHASED AROUND BY A GUY WHOSE HEAD LOOKS LIKE A GENITAL WART. IT IS AN HONOR BEING A PART OF THESE HUGE FRANCHISEs LIKE \"GAME OF THRONES\" AND \"STAR WARS,\" BUT I\\'M STILL GETTING USED TO PEOPLE RECOGNIZING ME. THE OTHER DAY, A GUY STOPPED ME ON THE STREET AND SAYS, MY SON LOVES \"THE MANDALORIAN\" AND THE NEXT THING I KNOW, I\\'M FACE TIMING WITH A 6-YEAR-OLD WHO HAS NO IDEA WHO I AM BECAUSE MY CHARACTER WEARS A MASK THE ENTIRE SHOW. THE GUY IS LIKE, DO THE MANDO VOICE, BUT IT\\'S LIKE A BEDROOM VOICE. WITHOUT THE MASK, IT JUST SOUNDS PORNY. PEOPLE WALKING BY ON THE STREET SEE ME WHISPERING TO A 6-YEAR-OLD KID. I CAN BRING YOU IN WARM, OR I CAN BRING YOU IN COLD. EVEN THOUGH I CAME TO THE U.S. WHEN I WAS LITTLE, I WAS BORN IN CHILE, AND I HAVE 34 FIRST COUSINS WHO ARE STILL THERE. THEY\\'RE VERY PROUD OF ME. I KNOW THEY\\'RE PROUD BECAUSE THEY GIVE MY PHONE NUMBER TO EVERY PERSON THEY MEET, WHICH MEANS EVERY DAY, SOMEONE IN SANTIAGO WILL TEXT ME STUFF LIKE, CAN YOU COME TO MY WEDDING, OR CAN YOU SING MY PRIEST HAPPY BIRTHDAY, OR IS BABY YODA MEAN IN REAL LIFE. SO I HAVE TO BE LIKE NO, NO, AND HIS NAME IS GROGU. BUT MY COUSINS WEREN\\'T ALWAYS SO PROUD. EARLY IN MY CAREER, I PLAYED SMALL PARTS IN EVERY CRIME SHOW. I EVEN PLAYED TWO DIFFERENT CHARACTERS ON \"LAW AND ORDER.\" TITO CABASSA WHO LOOKED LIKE THIS. AND ONE YEAR LATER, I PLAYED REGGIE LUCKMAN WHO LOOKS LIKE THIS. AND THAT, MY FRIENDS, IS CALLED RANGE. BUT IT IS AMAZING TO BE HERE, LIKE I SAID. I WAS BORN IN CHILE, AND NINE MONTHS LATER, MY PARENTS FLED AND BROUGHT ME AND MY SISTER TO THE U.S. THEY WERE SO BRAVE, AND WITHOUT THEM, I WOULDN\\'T BE HERE IN THIS WONDERFUL COUNTRY, AND I CERTAINLY WOULDN\\'T BE STANDING HERE WITH YOU ALL TONIGHT. SO TO ALL MY FAMILY WATCHING IN CHILE, I WANT TO SAY [ SPEAKING NON-ENGLISH ] WHICH MEANS, I LOVE YOU, I MISS YOU, AND STOP GIVING OUT MY PHONE NUMBER. WE\\'VE GOT AN AMAZING SHOW FOR YOU TONIGHT. COLDPLAY IS HERE, SO STICK', lookup_str='', metadata={'source': 'QsYGlZkevEg', 'title': 'Pedro Pascal Monologue - SNL', 'description': 'First-time host Pedro Pascal talks about filming The Last of Us and being recognized by fans.\\n\\nSaturday Night Live. Stream now on Peacock: https://pck.tv/3uQxh4q\\n\\nSubscribe to SNL: https://goo.gl/tUsXwM\\nStream Current Full Episodes: http://www.nbc.com/saturday-night-live\\n\\nWATCH PAST SNL SEASONS\\nGoogle Play - http://bit.ly/SNLGooglePlay\\niTunes - http://bit.ly/SNLiTunes\\n\\nSNL ON SOCIAL\\nSNL Instagram: http://instagram.com/nbcsnl\\nSNL Facebook: https://www.facebook.com/snl\\nSNL Twitter: https://twitter.com/nbcsnl\\nSNL TikTok: https://www.tiktok.com/@nbcsnl\\n\\nGET MORE NBC\\nLike NBC: http://Facebook.com/NBC\\nFollow NBC: http://Twitter.com/NBC\\nNBC Tumblr: http://NBCtv.tumblr.com/\\nYouTube: http://www.youtube.com/nbc\\nNBC Instagram: http://instagram.com/nbc\\n\\n#SNL #PedroPascal #SNL48 #Coldplay', 'view_count': 1175057, 'thumbnail_url': 'https://i.ytimg.com/vi/QsYGlZkevEg/sddefault.jpg', 'publish_date': datetime.datetime(2023, 2, 4, 0, 0), 'length': 224, 'author': 'Saturday Night Live'}, lookup_index=0)]"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader.load()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/document_loaders/how_to_guides.rst

@@ -0,0 +1,61 @@
+How To Guides
+====================================
+
+There are a lot of different document loaders that LangChain supports. Below are how-to guides for working with them
+
+`File Loader <./examples/unstructured_file.html>`_: A walkthrough of how to use Unstructured to load files of arbitrary types (pdfs, txt, html, etc).
+
+`Directory Loader <./examples/directory_loader.html>`_: A walkthrough of how to use Unstructured load files from a given directory.
+
+`Notion <./examples/notion.html>`_: A walkthrough of how to load data for an arbitrary Notion DB.
+
+`ReadTheDocs <./examples/readthedocs_documentation.html>`_: A walkthrough of how to load data for documentation generated by ReadTheDocs.
+
+`HTML <./examples/html.html>`_: A walkthrough of how to load data from an html file.
+
+`PDF <./examples/pdf.html>`_: A walkthrough of how to load data from a PDF file.
+
+`PowerPoint <./examples/powerpoint.html>`_: A walkthrough of how to load data from a powerpoint file.
+
+`Email <./examples/email.html>`_: A walkthrough of how to load data from an email (`.eml`) file.
+
+`GoogleDrive <./examples/googledrive.html>`_: A walkthrough of how to load data from Google drive.
+
+`Microsoft Word <./examples/microsoft_word.html>`_: A walkthrough of how to load data from Microsoft Word files.
+
+`Obsidian <./examples/obsidian.html>`_: A walkthrough of how to load data from an Obsidian file dump.
+
+`Roam <./examples/roam.html>`_: A walkthrough of how to load data from a Roam file export.
+
+`EveryNote <./examples/everynote.html>`_: A walkthrough of how to load data from a EveryNote (`.enex`) file.
+
+`YouTube <./examples/youtube.html>`_: A walkthrough of how to load the transcript from a YouTube video.
+
+`s3 File <./examples/s3_file.html>`_: A walkthrough of how to load a file from s3.
+
+`s3 Directory <./examples/s3_directory.html>`_: A walkthrough of how to load all files in a directory from s3.
+
+`GCS File <./examples/gcs_file.html>`_: A walkthrough of how to load a file from Google Cloud Storage (GCS).
+
+`GCS Directory <./examples/gcs_directory.html>`_: A walkthrough of how to load all files in a directory from Google Cloud Storage (GCS).
+
+`Web Base <./examples/web_base.html>`_: A walkthrough of how to load all text data from webpages.
+
+`IMSDb <./examples/imsdb.html>`_: A walkthrough of how to load all text data from IMSDb webpage.
+
+`AZLyrics <./examples/azlyrics.html>`_: A walkthrough of how to load all text data from AZLyrics webpage.
+
+`College Confidential <./examples/college_confidential.html>`_: A walkthrough of how to load all text data from College Confidential webpage.
+
+`Gutenberg <./examples/gutenberg.html>`_: A walkthrough of how to load data from a Gutenberg ebook text.
+
+`Airbyte Json <./examples/airbyte_json.html>`_: A walkthrough of how to load data from a local Airbyte JSON file.
+
+`Online PDF <./examples/online_pdf.html>`_: A walkthrough of how to load data from an online PDF.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+   :hidden:
+
+   examples/*

docs/modules/document_loaders/key_concepts.md

@@ -0,0 +1,12 @@
+# Key Concepts
+
+## Document
+This class is a container for document information. This contains two parts:
+- `page_content`: The content of the actual page itself.
+- `metadata`: The metadata associated with the document. This can be things like the file path, the url, etc.
+
+## Loader
+This base class is a way to load documents. It exposes a `load` method that returns `Document` objects.
+
+## [Unstructured](https://github.com/Unstructured-IO/unstructured)
+Unstructured is a python package specifically focused on transformations from raw documents to text.

docs/modules/llms/async_llm.ipynb

@@ -0,0 +1,150 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f6574496-b360-4ffa-9523-7fd34a590164",
+   "metadata": {},
+   "source": [
+    "# Async API for LLM\n",
+    "\n",
+    "LangChain provides async support for LLMs by leveraging the [asyncio](https://docs.python.org/3/library/asyncio.html) library.\n",
+    "\n",
+    "Async support is particularly useful for calling multiple LLMs concurrently, as these calls are network-bound. Currently, only `OpenAI` is supported, but async support for other LLMs is on the roadmap.\n",
+    "\n",
+    "You can use the `agenerate` method to call an OpenAI LLM asynchronously."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "5e49e96c-0f88-466d-b3d3-ea0966bdf19e",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "I'm doing well. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "I am doing quite well. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing great, thank you! How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thanks for asking. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\u001b[1mConcurrent executed in 1.93 seconds.\u001b[0m\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing well, thank you. How about you?\n",
+      "\n",
+      "\n",
+      "I'm doing great, thank you. How about you?\n",
+      "\u001b[1mSerial executed in 10.54 seconds.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "import time\n",
+    "import asyncio\n",
+    "\n",
+    "from langchain.llms import OpenAI\n",
+    "\n",
+    "def generate_serially():\n",
+    "    llm = OpenAI(temperature=0.9)\n",
+    "    for _ in range(10):\n",
+    "        resp = llm.generate([\"Hello, how are you?\"])\n",
+    "        print(resp.generations[0][0].text)\n",
+    "\n",
+    "\n",
+    "async def async_generate(llm):\n",
+    "    resp = await llm.agenerate([\"Hello, how are you?\"])\n",
+    "    print(resp.generations[0][0].text)\n",
+    "\n",
+    "\n",
+    "async def generate_concurrently():\n",
+    "    llm = OpenAI(temperature=0.9)\n",
+    "    tasks = [async_generate(llm) for _ in range(10)]\n",
+    "    await asyncio.gather(*tasks)\n",
+    "\n",
+    "\n",
+    "s = time.perf_counter()\n",
+    "# If running this outside of Jupyter, use asyncio.run(generate_concurrently())\n",
+    "await generate_concurrently() \n",
+    "elapsed = time.perf_counter() - s\n",
+    "print('\\033[1m' + f\"Concurrent executed in {elapsed:0.2f} seconds.\" + '\\033[0m')\n",
+    "\n",
+    "s = time.perf_counter()\n",
+    "generate_serially()\n",
+    "elapsed = time.perf_counter() - s\n",
+    "print('\\033[1m' + f\"Serial executed in {elapsed:0.2f} seconds.\" + '\\033[0m')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/llms/examples/fake_llm.ipynb

@@ -0,0 +1,138 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "052dfe58",
+   "metadata": {},
+   "source": [
+    "# Fake LLM\n",
+    "We expose a fake LLM class that can be used for testing. This allows you to mock out calls to the LLM and simulate what would happen if the LLM responded in a certain way.\n",
+    "\n",
+    "In this notebook we go over how to use this.\n",
+    "\n",
+    "We start this with using the FakeLLM in an agent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ef97ac4d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.llms.fake import FakeListLLM"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "9a0a160f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import load_tools\n",
+    "from langchain.agents import initialize_agent"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "b272258c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tools = load_tools([\"python_repl\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "94096c4c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "responses=[\n",
+    "    \"Action: Python REPL\\nAction Input: print(2 + 2)\",\n",
+    "    \"Final Answer: 4\"\n",
+    "]\n",
+    "llm = FakeListLLM(responses=responses)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "da226d02",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = initialize_agent(tools, llm, agent=\"zero-shot-react-description\", verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "44c13426",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mAction: Python REPL\n",
+      "Action Input: print(2 + 2)\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3m4\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mFinal Answer: 4\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'4'"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"whats 2 + 2\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "814c2858",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/llms/examples/token_usage_tracking.ipynb

@@ -0,0 +1,179 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e5715368",
+   "metadata": {},
+   "source": [
+    "# Token Usage Tracking\n",
+    "\n",
+    "This notebook goes over how to track your token usage for specific calls. It is currently only implemented for the OpenAI API.\n",
+    "\n",
+    "Let's first look at an extremely simple example of tracking token usage for a single LLM call."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9455db35",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.llms import OpenAI\n",
+    "from langchain.callbacks import get_openai_callback"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "d1c55cc9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(model_name=\"text-davinci-002\", n=2, best_of=2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "31667d54",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "42\n"
+     ]
+    }
+   ],
+   "source": [
+    "with get_openai_callback() as cb:\n",
+    "    result = llm(\"Tell me a joke\")\n",
+    "    print(cb.total_tokens)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c0ab6d27",
+   "metadata": {},
+   "source": [
+    "Anything inside the context manager will get tracked. Here's an example of using it to track multiple calls in sequence."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "e09420f4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "83\n"
+     ]
+    }
+   ],
+   "source": [
+    "with get_openai_callback() as cb:\n",
+    "    result = llm(\"Tell me a joke\")\n",
+    "    result2 = llm(\"Tell me a joke\")\n",
+    "    print(cb.total_tokens)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d8186e7b",
+   "metadata": {},
+   "source": [
+    "If a chain or agent with multiple steps in it is used, it will track all those steps."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "5d1125c6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import load_tools\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.llms import OpenAI\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
+    "agent = initialize_agent(tools, llm, agent=\"zero-shot-react-description\", verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "2f98c536",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who Olivia Wilde's boyfriend is and then calculate his age raised to the 0.23 power.\n",
+      "Action: Search\n",
+      "Action Input: \"Olivia Wilde boyfriend\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mJason Sudeikis\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Jason Sudeikis' age\n",
+      "Action: Search\n",
+      "Action Input: \"Jason Sudeikis age\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3m47 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 47 raised to the 0.23 power\n",
+      "Action: Calculator\n",
+      "Action Input: 47^0.23\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 2.4242784855673896\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Jason Sudeikis, Olivia Wilde's boyfriend, is 47 years old and his age raised to the 0.23 power is 2.4242784855673896.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "1465\n"
+     ]
+    }
+   ],
+   "source": [
+    "with get_openai_callback() as cb:\n",
+    "    response = agent.run(\"Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?\")\n",
+    "    print(cb.total_tokens)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "80ca77a3",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/llms/generic_how_to.rst

@@ -9,6 +9,10 @@ The examples here all address certain "how-to" guides for working with LLMs.
 
 `Custom LLM <./examples/custom_llm.html>`_: How to create and use a custom LLM class, in case you have an LLM not from one of the standard providers (including one that you host yourself).
 
+`Token Usage Tracking <./examples/token_usage_tracking.html>`_: How to track the token usage of various chains/agents/LLM calls.
+
+`Fake LLM <./examples/fake_llm.html>`_: How to create and use a fake LLM for testing and debugging purposes.
+
 
 .. toctree::
    :maxdepth: 1

docs/modules/llms/how_to_guides.rst

@@ -7,6 +7,7 @@ They are split into two categories:
 
 1. `Generic Functionality <./generic_how_to.html>`_: Covering generic functionality all LLMs should have.
 2. `Integrations <./integrations.html>`_: Covering integrations with various LLM providers.
+3. `Asynchronous <./async_llm.html>`_: Covering asynchronous functionality.
 
 .. toctree::
    :maxdepth: 1

docs/modules/llms/integrations/huggingface_hub.ipynb

@@ -5,9 +5,9 @@
    "id": "959300d4",
    "metadata": {},
    "source": [
-    "# HuggingFace Hub\n",
+    "# Hugging Face Hub\n",
     "\n",
-    "This example showcases how to connect to the HuggingFace Hub."
+    "This example showcases how to connect to the Hugging Face Hub."
    ]
   },
   {
@@ -20,7 +20,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "The Seattle Seahawks won the Super Bowl in 2010. Justin Beiber was born in 2010. The\n"
+      "The Seattle Seahawks won the Super Bowl in 2010. Justin Beiber was born in 2010. The final answer: Seattle Seahawks.\n"
      ]
     }
    ],
@@ -31,7 +31,7 @@
     "\n",
     "Answer: Let's think step by step.\"\"\"\n",
     "prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=HuggingFaceHub(repo_id=\"google/flan-t5-xl\", model_kwargs={\"temperature\":1e-10}))\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=HuggingFaceHub(repo_id=\"google/flan-t5-xl\", model_kwargs={\"temperature\":0, \"max_length\":64}))\n",
     "\n",
     "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
     "\n",

docs/modules/memory/examples/chatgpt_clone.ipynb

@@ -77,7 +77,7 @@
     "    memory=ConversationalBufferWindowMemory(k=2),\n",
     ")\n",
     "\n",
-    "output = chatgpt_chain.predict(human_input=\"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply wiht the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\")\n",
+    "output = chatgpt_chain.predict(human_input=\"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\")\n",
     "print(output)"
    ]
   },
@@ -103,7 +103,7 @@
       "\n",
       "Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.\n",
       "\n",
-      "Human: I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply wiht the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\n",
+      "Human: I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\n",
       "AI: \n",
       "```\n",
       "$ pwd\n",
@@ -148,7 +148,7 @@
       "\n",
       "Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.\n",
       "\n",
-      "Human: I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply wiht the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\n",
+      "Human: I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\n",
       "AI: \n",
       "```\n",
       "$ pwd\n",
@@ -915,14 +915,14 @@
       "  \"response\": \"Artificial intelligence (AI) is the simulation of human intelligence processes by machines, especially computer systems. These processes include learning (the acquisition of information and rules for using the information), reasoning (using the rules to reach approximate or definite conclusions) and self-correction. AI is used to develop computer systems that can think and act like humans.\"\n",
       "}\n",
       "```\n",
-      "Human: curl --header \"Content-Type:application/json\" --request POST --data '{\"message\": \"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply wiht the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\"}' https://chat.openai.com/chat\n",
+      "Human: curl --header \"Content-Type:application/json\" --request POST --data '{\"message\": \"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\"}' https://chat.openai.com/chat\n",
       "Assistant:\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished LLMChain chain.\u001b[0m\n",
       " \n",
       "\n",
       "```\n",
-      "$ curl --header \"Content-Type:application/json\" --request POST --data '{\"message\": \"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply wiht the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\"}' https://chat.openai.com/chat\n",
+      "$ curl --header \"Content-Type:application/json\" --request POST --data '{\"message\": \"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\"}' https://chat.openai.com/chat\n",
       "\n",
       "{\n",
       "  \"response\": \"```\\n/current/working/directory\\n```\"\n",
@@ -932,7 +932,7 @@
     }
    ],
    "source": [
-    "output = chatgpt_chain.predict(human_input=\"\"\"curl --header \"Content-Type:application/json\" --request POST --data '{\"message\": \"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply wiht the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\"}' https://chat.openai.com/chat\"\"\")\n",
+    "output = chatgpt_chain.predict(human_input=\"\"\"curl --header \"Content-Type:application/json\" --request POST --data '{\"message\": \"I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curly brackets {like this}. My first command is pwd.\"}' https://chat.openai.com/chat\"\"\")\n",
     "print(output)"
    ]
   },

docs/modules/memory/examples/conversational_agent.ipynb

@@ -9,7 +9,7 @@
     "\n",
     "This notebook walks through using an agent optimized for conversation. Other agents are often optimized for using tools to figure out the best response, which is not ideal in a conversational setting where you may want the agent to be able to chat with the user as well.\n",
     "\n",
-    "This is accomplisehd with a specific type of agent (`conversational-react-description`) which expects to be used with a memory component."
+    "This is accomplished with a specific type of agent (`conversational-react-description`) which expects to be used with a memory component."
    ]
   },
   {

docs/modules/memory/examples/entity_summary_memory.ipynb

@@ -0,0 +1,459 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "ff31084d",
+   "metadata": {},
+   "source": [
+    "# Entity Memory\n",
+    "This notebook shows how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "13471fbd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain import OpenAI, ConversationChain\n",
+    "from langchain.chains.conversation.memory import ConversationEntityMemory\n",
+    "from langchain.chains.conversation.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATE\n",
+    "from pydantic import BaseModel\n",
+    "from typing import List, Dict, Any"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "183346e2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "conversation = ConversationChain(\n",
+    "    llm=llm, \n",
+    "    verbose=True,\n",
+    "    prompt=ENTITY_MEMORY_CONVERSATION_TEMPLATE,\n",
+    "    memory=ConversationEntityMemory(llm=llm)\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "7eb1460a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are an assistant to a human, powered by a large language model trained by OpenAI.\n",
+      "\n",
+      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
+      "\n",
+      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
+      "\n",
+      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
+      "\n",
+      "Context:\n",
+      "{'Deven': '', 'Sam': ''}\n",
+      "\n",
+      "Current conversation:\n",
+      "\n",
+      "Last line:\n",
+      "Human: Deven & Sam are working on a hackathon project\n",
+      "You:\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' That sounds like a great project! What kind of project are they working on?'"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation.predict(input=\"Deven & Sam are working on a hackathon project\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "46324ca8",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are an assistant to a human, powered by a large language model trained by OpenAI.\n",
+      "\n",
+      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
+      "\n",
+      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
+      "\n",
+      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
+      "\n",
+      "Context:\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam.', 'Sam': 'Sam is working on a hackathon project with Deven.', 'Langchain': ''}\n",
+      "\n",
+      "Current conversation:\n",
+      "Human: Deven & Sam are working on a hackathon project\n",
+      "AI:  That sounds like a great project! What kind of project are they working on?\n",
+      "Last line:\n",
+      "Human: They are trying to add more complex memory structures to Langchain\n",
+      "You:\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' That sounds like an interesting project! What kind of memory structures are they trying to add?'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation.predict(input=\"They are trying to add more complex memory structures to Langchain\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "ff2ebf6b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are an assistant to a human, powered by a large language model trained by OpenAI.\n",
+      "\n",
+      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
+      "\n",
+      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
+      "\n",
+      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
+      "\n",
+      "Context:\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam to add more complex memory structures to Langchain.', 'Sam': 'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain.', 'Langchain': 'Langchain is a project that seeks to add more complex memory structures.', 'Key-Value Store': ''}\n",
+      "\n",
+      "Current conversation:\n",
+      "Human: Deven & Sam are working on a hackathon project\n",
+      "AI:  That sounds like a great project! What kind of project are they working on?\n",
+      "Human: They are trying to add more complex memory structures to Langchain\n",
+      "AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?\n",
+      "Last line:\n",
+      "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
+      "You:\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' That sounds like a great idea! How will the key-value store work?'"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation.predict(input=\"They are adding in a key-value store for entities mentioned so far in the conversation.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "56cfd4ba",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are an assistant to a human, powered by a large language model trained by OpenAI.\n",
+      "\n",
+      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
+      "\n",
+      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
+      "\n",
+      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
+      "\n",
+      "Context:\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.', 'Sam': 'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'}\n",
+      "\n",
+      "Current conversation:\n",
+      "Human: Deven & Sam are working on a hackathon project\n",
+      "AI:  That sounds like a great project! What kind of project are they working on?\n",
+      "Human: They are trying to add more complex memory structures to Langchain\n",
+      "AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?\n",
+      "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
+      "AI:  That sounds like a great idea! How will the key-value store work?\n",
+      "Last line:\n",
+      "Human: What do you know about Deven & Sam?\n",
+      "You:\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success.'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation.predict(input=\"What do you know about Deven & Sam?\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e6df549",
+   "metadata": {},
+   "source": [
+    "## Inspecting the memory store\n",
+    "We can also inspect the memory store directly. In the following examaples, we look at it directly, and then go through some examples of adding information and watch how it changes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "038b4d3f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'Deven': 'Deven is working on a hackathon project with Sam to add more '\n",
+      "          'complex memory structures to Langchain, including a key-value store '\n",
+      "          'for entities mentioned so far in the conversation.',\n",
+      " 'Key-Value Store': 'Key-Value Store: A data structure that stores values '\n",
+      "                    'associated with a unique key, allowing for efficient '\n",
+      "                    'retrieval of values. Deven and Sam are adding a key-value '\n",
+      "                    'store for entities mentioned so far in the conversation.',\n",
+      " 'Langchain': 'Langchain is a project that seeks to add more complex memory '\n",
+      "              'structures, including a key-value store for entities mentioned '\n",
+      "              'so far in the conversation.',\n",
+      " 'Sam': 'Sam is working on a hackathon project with Deven to add more complex '\n",
+      "        'memory structures to Langchain, including a key-value store for '\n",
+      "        'entities mentioned so far in the conversation.'}\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pprint import pprint\n",
+    "pprint(conversation.memory.store)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "2df4800e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are an assistant to a human, powered by a large language model trained by OpenAI.\n",
+      "\n",
+      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
+      "\n",
+      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
+      "\n",
+      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
+      "\n",
+      "Context:\n",
+      "{'Daimon': '', 'Sam': 'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'}\n",
+      "\n",
+      "Current conversation:\n",
+      "Human: They are trying to add more complex memory structures to Langchain\n",
+      "AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?\n",
+      "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
+      "AI:  That sounds like a great idea! How will the key-value store work?\n",
+      "Human: What do you know about Deven & Sam?\n",
+      "AI:  Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success.\n",
+      "Last line:\n",
+      "Human: Sam is the founder of a company called Daimon.\n",
+      "You:\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"\\nThat's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\""
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation.predict(input=\"Sam is the founder of a company called Daimon.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "ebe9e36f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'Daimon': 'Daimon is a company founded by Sam.',\n",
+      " 'Deven': 'Deven is working on a hackathon project with Sam to add more '\n",
+      "          'complex memory structures to Langchain, including a key-value store '\n",
+      "          'for entities mentioned so far in the conversation.',\n",
+      " 'Key-Value Store': 'Key-Value Store: A data structure that stores values '\n",
+      "                    'associated with a unique key, allowing for efficient '\n",
+      "                    'retrieval of values. Deven and Sam are adding a key-value '\n",
+      "                    'store for entities mentioned so far in the conversation.',\n",
+      " 'Langchain': 'Langchain is a project that seeks to add more complex memory '\n",
+      "              'structures, including a key-value store for entities mentioned '\n",
+      "              'so far in the conversation.',\n",
+      " 'Sam': 'Sam is working on a hackathon project with Deven to add more complex '\n",
+      "        'memory structures to Langchain, including a key-value store for '\n",
+      "        'entities mentioned so far in the conversation. He is also the founder '\n",
+      "        'of a company called Daimon.'}\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pprint import pprint\n",
+    "pprint(conversation.memory.store)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "dd547144",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are an assistant to a human, powered by a large language model trained by OpenAI.\n",
+      "\n",
+      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
+      "\n",
+      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
+      "\n",
+      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
+      "\n",
+      "Context:\n",
+      "{'Sam': 'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He is also the founder of a company called Daimon.', 'Daimon': 'Daimon is a company founded by Sam.'}\n",
+      "\n",
+      "Current conversation:\n",
+      "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
+      "AI:  That sounds like a great idea! How will the key-value store work?\n",
+      "Human: What do you know about Deven & Sam?\n",
+      "AI:  Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success.\n",
+      "Human: Sam is the founder of a company called Daimon.\n",
+      "AI: \n",
+      "That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\n",
+      "Last line:\n",
+      "Human: What do you know about Sam?\n",
+      "You:\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "' Sam is the founder of a company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He seems to be very motivated and passionate about his project, and is working hard to make it a success.'"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation.predict(input=\"What do you know about Sam?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e00463b5",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/memory/how_to_guides.rst

@@ -7,6 +7,9 @@ The examples here all highlight how to use memory in different ways.
 
 `ChatGPT Clone <./examples/chatgpt_clone.html>`_: How to recreate ChatGPT with LangChain prompting + memory components.
 
+`Entity Memory <./examples/entity_summary_memory.html>`_: How to use a type of memory that organizes information by entity.
+
+
 `Adding Memory to Multi-Input Chain <./examples/adding_memory_chain_multiple_inputs.html>`_: How to add a memory component to any multiple input chain.
 
 `Conversational Memory Customization <./examples/conversational_customization.html>`_: How to customize existing conversation memory components.

docs/modules/memory/key_concepts.md

@@ -12,3 +12,8 @@ There are a few different ways to accomplish this:
 - Summary: This involves summarizing previous conversations and passing that summary in, instead of the raw dialouge itself. Compared to `Buffer`, this compresses information: meaning it is more lossy, but also less likely to run into context length limits.
 - Combination: A combination of the above two approaches, where you compute a summary but also pass in some previous interfactions directly!
 
+## Entity Memory
+A more complex form of memory is remembering information about specific entities in the conversation.
+This is a more direct and organized way of remembering information over time.
+Putting it a more structured form also has the benefit of allowing easy inspection of what is known about specific entities.
+For a guide on how to use this type of memory, see [this notebook](./examples/entity_summary_memory.ipynb).

docs/modules/prompts/examples/custom_prompt_template.ipynb

@@ -0,0 +1,168 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "c75efab3",
+   "metadata": {},
+   "source": [
+    "# Create a custom prompt template\n",
+    "\n",
+    "Let's suppose we want the LLM to generate English language explanations of a function given its name. To achieve this task, we will create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.\n",
+    "\n",
+    "## Why are custom prompt templates needed?\n",
+    "\n",
+    "LangChain provides a set of default prompt templates that can be used to generate prompts for a variety of tasks. However, there may be cases where the default prompt templates do not meet your needs. For example, you may want to create a prompt template with specific dynamic instructions for your language model. In such cases, you can create a custom prompt template.\n",
+    "\n",
+    "Take a look at the current set of default prompt templates [here](../getting_started.md)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d56ce86",
+   "metadata": {},
+   "source": [
+    "## Create a custom prompt template\n",
+    "\n",
+    "The only two requirements for all prompt templates are:\n",
+    "\n",
+    "1. They have a input_variables attribute that exposes what input variables this prompt template expects.\n",
+    "2. They expose a format method which takes in keyword arguments corresponding to the expected input_variables and returns the formatted prompt.\n",
+    "\n",
+    "Let's create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.\n",
+    "\n",
+    "First, let's create a function that will return the source code of a function given its name."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c831e1ce",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import inspect\n",
+    "\n",
+    "def get_source_code(function_name):\n",
+    "    # Get the source code of the function\n",
+    "    return inspect.getsource(function_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c2c8f4ea",
+   "metadata": {},
+   "source": [
+    "Next, we'll create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "3ad1efdc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import BasePromptTemplate\n",
+    "from pydantic import BaseModel, validator\n",
+    "\n",
+    "\n",
+    "class FunctionExplainerPromptTemplate(BasePromptTemplate, BaseModel):\n",
+    "    \"\"\" A custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function. \"\"\"\n",
+    "\n",
+    "    @validator(\"input_variables\")\n",
+    "    def validate_input_variables(cls, v):\n",
+    "        \"\"\" Validate that the input variables are correct. \"\"\"\n",
+    "        if len(v) != 1 or \"function_name\" not in v:\n",
+    "            raise ValueError(\"function_name must be the only input_variable.\")\n",
+    "        return v\n",
+    "\n",
+    "    def format(self, **kwargs) -> str:\n",
+    "        # Get the source code of the function\n",
+    "        source_code = get_source_code(kwargs[\"function_name\"])\n",
+    "\n",
+    "        # Generate the prompt to be sent to the language model\n",
+    "        prompt = f\"\"\"\n",
+    "        Given the function name and source code, generate an English language explanation of the function.\n",
+    "        Function Name: {kwargs[\"function_name\"].__name__}\n",
+    "        Source Code:\n",
+    "        {source_code}\n",
+    "        Explanation:\n",
+    "        \"\"\"\n",
+    "        return prompt\n",
+    "    \n",
+    "    def _prompt_type(self):\n",
+    "        return \"function-explainer\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7fcbf6ef",
+   "metadata": {},
+   "source": [
+    "## Use the custom prompt template\n",
+    "\n",
+    "Now that we have created a custom prompt template, we can use it to generate prompts for our task."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "bd836cda",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "        Given the function name and source code, generate an English language explanation of the function.\n",
+      "        Function Name: get_source_code\n",
+      "        Source Code:\n",
+      "        def get_source_code(function_name):\n",
+      "    # Get the source code of the function\n",
+      "    return inspect.getsource(function_name)\n",
+      "\n",
+      "        Explanation:\n",
+      "        \n"
+     ]
+    }
+   ],
+   "source": [
+    "fn_explainer = FunctionExplainerPromptTemplate(input_variables=[\"function_name\"])\n",
+    "\n",
+    "# Generate a prompt for the function \"get_source_code\"\n",
+    "prompt = fn_explainer.format(function_name=get_source_code)\n",
+    "print(prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7f3161c6",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/prompts/examples/custom_prompt_template.md

@@ -1,75 +0,0 @@
-# Create a custom prompt template
-
-Let's suppose we want the LLM to generate English language explanations of a function given its name. To achieve this task, we will create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.
-
-## Why are custom prompt templates needed?
-
-LangChain provides a set of default prompt templates that can be used to generate prompts for a variety of tasks. However, there may be cases where the default prompt templates do not meet your needs. For example, you may want to create a prompt template with specific dynamic instructions for your language model. In such cases, you can create a custom prompt template.
-
-:::{note}
-Take a look at the current set of default prompt templates [here](../prompt_templates.md).
-:::
-<!-- TODO(shreya): Add correct link here. -->
-
-## Create a custom prompt template
-
-The only two requirements for all prompt templates are:
-
-1. They have a input_variables attribute that exposes what input variables this prompt template expects.
-2. They expose a format method which takes in keyword arguments corresponding to the expected input_variables and returns the formatted prompt.
-
-Let's create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.
-
-First, let's create a function that will return the source code of a function given its name.
-
-```python
-import inspect
-
-def get_source_code(function_name):
-    # Get the source code of the function
-    return inspect.getsource(function_name)
-```
-
-Next, we'll create a custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function.
-
-```python
-from langchain.prompts import BasePromptTemplate
-from pydantic import BaseModel
-
-
-class FunctionExplainerPromptTemplate(BasePromptTemplate, BaseModel):
-    """ A custom prompt template that takes in the function name as input, and formats the prompt template to provide the source code of the function. """
-
-    @validator("input_variables")
-    def validate_input_variables(cls, v):
-        """ Validate that the input variables are correct. """
-        if len(v) != 1 or "function_name" not in v:
-            raise ValueError("function_name must be the only input_variable.")
-        return v
-
-    def format(self, **kwargs) -> str:
-        # Get the source code of the function
-        source_code = get_source_code(kwargs["function_name"])
-
-        # Generate the prompt to be sent to the language model
-        prompt = f"""
-        Given the function name and source code, generate an English language explanation of the function.
-        Function Name: {kwargs["function_name"]}
-        Source Code:
-        {source_code}
-        Explanation:
-        """
-        return prompt
-```
-
-## Use the custom prompt template
-
-Now that we have created a custom prompt template, we can use it to generate prompts for our task.
-
-```python
-fn_explainer = FunctionExplainerPromptTemplate(input_variables=["function_name"])
-
-# Generate a prompt for the function "get_source_code"
-prompt = fn_explainer.format(function_name=get_source_code)
-print(prompt)
-```

docs/modules/prompts/examples/example_selectors.ipynb

@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "8244ff60",
    "metadata": {},
    "outputs": [],
@@ -48,6 +48,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain.prompts.example_selector import LengthBasedExampleSelector"
    ]
   },
@@ -75,8 +76,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "example_prompt = PromptTemplate(\n",
+    "    input_variables=[\"input\", \"output\"],\n",
+    "    template=\"Input: {input}\\nOutput: {output}\",\n",
+    ")\n",
     "example_selector = LengthBasedExampleSelector(\n",
-    "    # These are the examples is has available to choose from.\n",
+    "    # These are the examples it has available to choose from.\n",
     "    examples=examples, \n",
     "    # This is the PromptTemplate being used to format the examples.\n",
     "    example_prompt=example_prompt, \n",
@@ -434,10 +439,242 @@
     "print(similar_prompt.format(adjective=\"worried\"))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "4aaeed2f",
+   "metadata": {},
+   "source": [
+    "## NGram Overlap ExampleSelector\n",
+    "\n",
+    "The NGramOverlapExampleSelector selects and orders examples based on which examples are most similar to the input, according to an ngram overlap score. The ngram overlap score is a float between 0.0 and 1.0, inclusive. \n",
+    "\n",
+    "The selector allows for a threshold score to be set. Examples with an ngram overlap score less than or equal to the threshold are excluded. The threshold is set to -1.0, by default, so will not exclude any examples, only reorder them. Setting the threshold to 0.0 will exclude examples that have no ngram overlaps with the input.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "9cbc0acc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.prompts.example_selector.ngram_overlap import NGramOverlapExampleSelector"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "4f318f4b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# These are examples of a fictional translation task.\n",
+    "examples = [\n",
+    "    {\"input\": \"See Spot run.\", \"output\": \"Ver correr a Spot.\"},\n",
+    "    {\"input\": \"My dog barks.\", \"output\": \"Mi perro ladra.\"},\n",
+    "    {\"input\": \"Spot can run.\", \"output\": \"Spot puede correr.\"},\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "bf75e0fe",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "example_prompt = PromptTemplate(\n",
+    "    input_variables=[\"input\", \"output\"],\n",
+    "    template=\"Input: {input}\\nOutput: {output}\",\n",
+    ")\n",
+    "example_selector = NGramOverlapExampleSelector(\n",
+    "    # These are the examples it has available to choose from.\n",
+    "    examples=examples, \n",
+    "    # This is the PromptTemplate being used to format the examples.\n",
+    "    example_prompt=example_prompt, \n",
+    "    # This is the threshold, at which selector stops.\n",
+    "    # It is set to -1.0 by default.\n",
+    "    threshold=-1.0,\n",
+    "    # For negative threshold:\n",
+    "    # Selector sorts examples by ngram overlap score, and excludes none.\n",
+    "    # For threshold greater than 1.0:\n",
+    "    # Selector excludes all examples, and returns an empty list.\n",
+    "    # For threshold equal to 0.0:\n",
+    "    # Selector sorts examples by ngram overlap score,\n",
+    "    # and excludes those with no ngram overlap with input.\n",
+    ")\n",
+    "dynamic_prompt = FewShotPromptTemplate(\n",
+    "    # We provide an ExampleSelector instead of examples.\n",
+    "    example_selector=example_selector,\n",
+    "    example_prompt=example_prompt,\n",
+    "    prefix=\"Give the Spanish translation of every input\",\n",
+    "    suffix=\"Input: {sentence}\\nOutput:\", \n",
+    "    input_variables=[\"sentence\"],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "83fb218a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Give the Spanish translation of every input\n",
+      "\n",
+      "Input: Spot can run.\n",
+      "Output: Spot puede correr.\n",
+      "\n",
+      "Input: See Spot run.\n",
+      "Output: Ver correr a Spot.\n",
+      "\n",
+      "Input: My dog barks.\n",
+      "Output: Mi perro ladra.\n",
+      "\n",
+      "Input: Spot can run fast.\n",
+      "Output:\n"
+     ]
+    }
+   ],
+   "source": [
+    "# An example input with large ngram overlap with \"Spot can run.\"\n",
+    "# and no overlap with \"My dog barks.\"\n",
+    "print(dynamic_prompt.format(sentence=\"Spot can run fast.\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "485f5307",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Give the Spanish translation of every input\n",
+      "\n",
+      "Input: Spot can run.\n",
+      "Output: Spot puede correr.\n",
+      "\n",
+      "Input: See Spot run.\n",
+      "Output: Ver correr a Spot.\n",
+      "\n",
+      "Input: Spot plays fetch.\n",
+      "Output: Spot juega a buscar.\n",
+      "\n",
+      "Input: My dog barks.\n",
+      "Output: Mi perro ladra.\n",
+      "\n",
+      "Input: Spot can run fast.\n",
+      "Output:\n"
+     ]
+    }
+   ],
+   "source": [
+    "# You can add examples to NGramOverlapExampleSelector as well.\n",
+    "new_example = {\"input\": \"Spot plays fetch.\", \"output\": \"Spot juega a buscar.\"}\n",
+    "\n",
+    "example_selector.add_example(new_example)\n",
+    "print(dynamic_prompt.format(sentence=\"Spot can run fast.\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "606ce697",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Give the Spanish translation of every input\n",
+      "\n",
+      "Input: Spot can run.\n",
+      "Output: Spot puede correr.\n",
+      "\n",
+      "Input: See Spot run.\n",
+      "Output: Ver correr a Spot.\n",
+      "\n",
+      "Input: Spot plays fetch.\n",
+      "Output: Spot juega a buscar.\n",
+      "\n",
+      "Input: Spot can run fast.\n",
+      "Output:\n"
+     ]
+    }
+   ],
+   "source": [
+    "# You can set a threshold at which examples are excluded.\n",
+    "# For example, setting threshold equal to 0.0\n",
+    "# excludes examples with no ngram overlaps with input.\n",
+    "# Since \"My dog barks.\" has no ngram overlaps with \"Spot can run fast.\"\n",
+    "# it is excluded.\n",
+    "example_selector.threshold=0.0\n",
+    "print(dynamic_prompt.format(sentence=\"Spot can run fast.\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 87,
+   "id": "7f8d72f7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Give the Spanish translation of every input\n",
+      "\n",
+      "Input: Spot can run.\n",
+      "Output: Spot puede correr.\n",
+      "\n",
+      "Input: Spot plays fetch.\n",
+      "Output: Spot juega a buscar.\n",
+      "\n",
+      "Input: Spot can play fetch.\n",
+      "Output:\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Setting small nonzero threshold\n",
+    "example_selector.threshold=0.09\n",
+    "print(dynamic_prompt.format(sentence=\"Spot can play fetch.\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 88,
+   "id": "09633aa8",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Give the Spanish translation of every input\n",
+      "\n",
+      "Input: Spot can play fetch.\n",
+      "Output:\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Setting threshold greater than 1.0\n",
+    "example_selector.threshold=1.0+1e-9\n",
+    "print(dynamic_prompt.format(sentence=\"Spot can play fetch.\"))"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c746d6f4",
+   "id": "39f30097",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/prompts/examples/examples.yaml

@@ -0,0 +1,4 @@
+- input: happy
+  output: sad
+- input: tall
+  output: short

docs/modules/prompts/examples/few_shot_prompt_yaml_examples.yaml

@@ -0,0 +1,14 @@
+_type: few_shot
+input_variables:
+    ["adjective"]
+prefix: 
+    Write antonyms for the following words.
+example_prompt:
+    input_variables:
+        ["input", "output"]
+    template:
+        "Input: {input}\nOutput: {output}"
+examples:
+    examples.yaml
+suffix:
+    "Input: {adjective}\nOutput:"

docs/modules/prompts/examples/prompt_management.ipynb

@@ -151,6 +151,47 @@
     "multiple_input_prompt.format(adjective=\"funny\", content=\"chickens\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "cc991ad2",
+   "metadata": {},
+   "source": [
+    "## From Template\n",
+    "You can also easily load a prompt template by just specifying the template, and not worrying about the input variables."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "d0a0756c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "template = \"Tell me a {adjective} joke about {content}.\"\n",
+    "multiple_input_prompt = PromptTemplate.from_template(template)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "59046640",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "PromptTemplate(input_variables=['adjective', 'content'], output_parser=None, template='Tell me a {adjective} joke about {content}.', template_format='f-string', validate_template=True)"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "multiple_input_prompt"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "b2dd6154",
@@ -291,6 +332,69 @@
     "print(prompt_from_string_examples.format(adjective=\"big\"))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "874b7575",
+   "metadata": {},
+   "source": [
+    "## Few Shot Prompts with Templates\n",
+    "We can also construct few shot prompt templates where the prefix and suffix themselves are prompt templates"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "e710115f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import FewShotPromptWithTemplates"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "5bf23a65",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prefix = PromptTemplate(input_variables=[\"content\"], template=\"This is a test about {content}.\")\n",
+    "suffix = PromptTemplate(input_variables=[\"new_content\"], template=\"Now you try to talk about {new_content}.\")\n",
+    "\n",
+    "prompt = FewShotPromptWithTemplates(\n",
+    "    suffix=suffix,\n",
+    "    prefix=prefix,\n",
+    "    input_variables=[\"content\", \"new_content\"],\n",
+    "    examples=examples,\n",
+    "    example_prompt=example_prompt,\n",
+    "    example_separator=\"\\n\",\n",
+    ")\n",
+    "output = prompt.format(content=\"animals\", new_content=\"party\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "d4036351",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "This is a test about animals.\n",
+      "Input: happy\n",
+      "Output: sad\n",
+      "Input: tall\n",
+      "Output: short\n",
+      "Now you try to talk about party.\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(output)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "bf038596",

docs/modules/prompts/examples/prompt_serialization.ipynb

@@ -225,6 +225,35 @@
     "!cat examples.json"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "d3052850",
+   "metadata": {},
+   "source": [
+    "And here is what the same examples stored as yaml might look like."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "901385d1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "- input: happy\r\n",
+      "  output: sad\r\n",
+      "- input: tall\r\n",
+      "  output: short\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!cat examples.yaml"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "8e300335",
@@ -236,7 +265,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 10,
    "id": "e2bec0fc",
    "metadata": {},
    "outputs": [
@@ -267,7 +296,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 11,
    "id": "98c8f356",
    "metadata": {},
    "outputs": [
@@ -293,6 +322,73 @@
     "print(prompt.format(adjective=\"funny\"))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "13620324",
+   "metadata": {},
+   "source": [
+    "The same would work if you loaded examples from the yaml file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "831e5e4a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "_type: few_shot\r\n",
+      "input_variables:\r\n",
+      "    [\"adjective\"]\r\n",
+      "prefix: \r\n",
+      "    Write antonyms for the following words.\r\n",
+      "example_prompt:\r\n",
+      "    input_variables:\r\n",
+      "        [\"input\", \"output\"]\r\n",
+      "    template:\r\n",
+      "        \"Input: {input}\\nOutput: {output}\"\r\n",
+      "examples:\r\n",
+      "    examples.yaml\r\n",
+      "suffix:\r\n",
+      "    \"Input: {adjective}\\nOutput:\"\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!cat few_shot_prompt_yaml_examples.yaml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "6f0a7eaa",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Write antonyms for the following words.\n",
+      "\n",
+      "Input: happy\n",
+      "Output: sad\n",
+      "\n",
+      "Input: tall\n",
+      "Output: short\n",
+      "\n",
+      "Input: funny\n",
+      "Output:\n"
+     ]
+    }
+   ],
+   "source": [
+    "prompt = load_prompt(\"few_shot_prompt_yaml_examples.yaml\")\n",
+    "print(prompt.format(adjective=\"funny\"))"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "4870aa9d",
@@ -304,7 +400,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 14,
    "id": "9d996a86",
    "metadata": {},
    "outputs": [
@@ -332,7 +428,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 15,
    "id": "dd2c10bb",
    "metadata": {},
    "outputs": [
@@ -369,7 +465,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 16,
    "id": "6cd781ef",
    "metadata": {},
    "outputs": [
@@ -400,7 +496,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 17,
    "id": "533ab8a7",
    "metadata": {},
    "outputs": [
@@ -437,7 +533,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 18,
    "id": "0b6dd7b8",
    "metadata": {},
    "outputs": [
@@ -458,7 +554,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 19,
    "id": "76a1065d",
    "metadata": {},
    "outputs": [
@@ -483,7 +579,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 20,
    "id": "744d275d",
    "metadata": {},
    "outputs": [
@@ -530,7 +626,7 @@
   },
   "vscode": {
    "interpreter": {
-    "hash": "b1677b440931f40d89ef8be7bf03acb108ce003de0ac9b18e8d43753ea2e7103"
+    "hash": "8eb71adebe840dca1185e9603533462bc47eb1b1a73bf7dab2d0a8a4c932882e"
    }
   }
  },

docs/modules/prompts/getting_started.md

@@ -80,6 +80,20 @@ Currently, the template should be formatted as a Python f-string. We also suppor
 :::
 
 
+## Load a prompt template from LangChainHub
+
+LangChainHub contains a collection of prompts which can be loaded directly via LangChain.
+
+
+```python
+from langchain.prompts import load_prompt
+
+prompt = load_prompt("lc://prompts/conversation/prompt.json")
+prompt.format(history="", input="What is 1 + 1?")
+```
+
+You can read more about LangChainHub and the prompts available with it [here](https://github.com/hwchase17/langchain-hub).
+
 ## Pass few shot examples to a prompt template
 
 Few shot examples are a set of examples that can be used to help the language model generate a better response.
@@ -155,11 +169,11 @@ from langchain.prompts.example_selector import LengthBasedExampleSelector
 
 # These are a lot of examples of a pretend task of creating antonyms.
 examples = [
-    {"input": "happy", "output": "sad"},
-    {"input": "tall", "output": "short"},
-    {"input": "energetic", "output": "lethargic"},
-    {"input": "sunny", "output": "gloomy"},
-    {"input": "windy", "output": "calm"},
+    {"word": "happy", "antonym": "sad"},
+    {"word": "tall", "antonym": "short"},
+    {"word": "energetic", "antonym": "lethargic"},
+    {"word": "sunny", "antonym": "gloomy"},
+    {"word": "windy", "antonym": "calm"},
 ]
 
 # We'll use the `LengthBasedExampleSelector` to select the examples.
@@ -174,7 +188,7 @@ example_selector = LengthBasedExampleSelector(
 )
 
 # We can now use the `example_selector` to create a `FewShotPromptTemplate`.
-few_shot_prompt = FewShotPromptTemplate(
+dynamic_prompt = FewShotPromptTemplate(
     # We provide an ExampleSelector instead of examples.
     example_selector=example_selector,
     example_prompt=example_prompt,
@@ -185,7 +199,7 @@ few_shot_prompt = FewShotPromptTemplate(
 )
 
 # We can now generate a prompt using the `format` method.
-print(few_shot_prompt.format(input="big"))
+print(dynamic_prompt.format(input="big"))
 # -> Give the antonym of every input
 # ->
 # -> Word: happy
@@ -211,7 +225,7 @@ In contrast, if we provide a very long input, the `LengthBasedExampleSelector` w
 
 ```python
 long_string = "big and huge and massive and large and gigantic and tall and much much much much much bigger than everything else"
-print(dynamic_prompt.format(adjective=long_string))
+print(dynamic_prompt.format(input=long_string))
 # -> Give the antonym of every input
 
 # -> Word: happy
@@ -224,4 +238,4 @@ print(dynamic_prompt.format(adjective=long_string))
 <!-- TODO(shreya): Add correct link here. -->
 LangChain comes with a few example selectors that you can use. For more details on how to use them, see [Example Selectors](./examples/example_selectors.ipynb).
 
-You can create custom example selectors that select examples based on any criteria you want. For more details on how to do this, see [Creating a custom example selector](examples/custom_example_selector.ipynb).
+You can create custom example selectors that select examples based on any criteria you want. For more details on how to do this, see [Creating a custom example selector](examples/custom_example_selector.ipynb).

docs/modules/prompts/how_to_guides.rst

@@ -19,11 +19,6 @@ The user guide here shows more advanced workflows and how to use the library in
 
 
 
-
-
-
-
-
 .. toctree::
    :maxdepth: 1
    :glob:

docs/modules/utils/combine_docs_examples/embeddings.ipynb

@@ -77,7 +77,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "42f76e43",
    "metadata": {},
@@ -138,7 +137,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "ed47bb62",
    "metadata": {},
@@ -196,11 +194,137 @@
    "source": [
     "doc_result = embeddings.embed_documents([text])"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fff4734f",
+   "metadata": {},
+   "source": [
+    "## TensorflowHub\n",
+    "Let's load the TensorflowHub Embedding class."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f822104b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings import TensorflowHubEmbeddings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "bac84e46",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "2023-01-30 23:53:01.652176: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
+      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
+      "2023-01-30 23:53:34.362802: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
+      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
+     ]
+    }
+   ],
+   "source": [
+    "embeddings = TensorflowHubEmbeddings()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "4790d770",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = \"This is a test document.\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "f556dcdb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query_result = embeddings.embed_query(text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "59428e05",
+   "metadata": {},
+   "source": [
+    "## InstructEmbeddings\n",
+    "Let's load the HuggingFace instruct Embeddings class."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "92c5b61e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings import HuggingFaceInstructEmbeddings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "062547b9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "load INSTRUCTOR_Transformer\n",
+      "max_seq_length  512\n"
+     ]
+    }
+   ],
+   "source": [
+    "embeddings = HuggingFaceInstructEmbeddings(query_instruction=\"Represent the query for retrieval: \")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "e1dcc4bd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = \"This is a test document.\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "90f0db94",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query_result = embeddings.embed_query(text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a961cdb5",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "cohere",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -214,7 +338,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.8"
+   "version": "3.10.9"
   },
   "vscode": {
    "interpreter": {

docs/modules/utils/combine_docs_examples/hyde.ipynb

@@ -10,7 +10,7 @@
     "\n",
     "At a high level, HyDE is an embedding technique that takes queries, generates a hypothetical answer, and then embeds that generated document and uses that as the final example. \n",
     "\n",
-    "In order to use HyDE, we therefor need to provide a base embedding model, as well as an LLMChain that can be used to generate those documents. By default, the HyDE class comes with some default prompts to use (see the paper for more details on them), but we can also create our own."
+    "In order to use HyDE, we therefore need to provide a base embedding model, as well as an LLMChain that can be used to generate those documents. By default, the HyDE class comes with some default prompts to use (see the paper for more details on them), but we can also create our own."
    ]
   },
   {
@@ -21,8 +21,8 @@
    "outputs": [],
    "source": [
     "from langchain.llms import OpenAI\n",
-    "from langchain.embeddings import OpenAIEmbeddings, HypotheticalDocumentEmbedder\n",
-    "from langchain.chains import LLMChain\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.chains import LLMChain, HypotheticalDocumentEmbedder\n",
     "from langchain.prompts import PromptTemplate"
    ]
   },
@@ -220,7 +220,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "llm-env",
    "language": "python",
    "name": "python3"
   },
@@ -234,7 +234,12 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.9.0 (default, Nov 15 2020, 06:25:35) \n[Clang 10.0.0 ]"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "9dd01537e9ab68cf47cb0398488d182358f774f73101197b3bd1b5502c6ec7f9"
+   }
   }
  },
  "nbformat": 4,

docs/modules/utils/combine_docs_examples/textsplitter.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Text Splitter\n",
     "\n",
-    "When you want to deal wit long pieces of text, it is necessary to split up that text into chunks.\n",
+    "When you want to deal with long pieces of text, it is necessary to split up that text into chunks.\n",
     "This notebook showcases several ways to do that.\n",
     "\n",
     "At a high level, text splitters work as following:\n",
@@ -151,7 +151,7 @@
    "metadata": {},
    "source": [
     "## Document creation\n",
-    "We can also use the text splitter to create \"Documents\" directly. Documents a way of bundling pieces of text with associated metadata so that chains can interact with them. We can also create documents with empty metadata though!\n",
+    "We can also use the text splitter to create \"Documents\" directly. Documents are a way of bundling pieces of text with associated metadata so that chains can interact with them. We can also create documents with empty metadata though!\n",
     "\n",
     "In the below example, we pass two pieces of text to get split up (we pass two just to show off the interface of splitting multiple pieces of text)."
    ]
@@ -475,10 +475,59 @@
     "print(texts[0])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "53049ff5",
+   "metadata": {},
+   "source": [
+    "## Token Text Splitter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a1a118b1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.text_splitter import TokenTextSplitter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "ef37c5d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "5750228a",
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Madam Speaker, Madam Vice President, our\n"
+     ]
+    }
+   ],
+   "source": [
+    "texts = text_splitter.split_text(state_of_the_union)\n",
+    "print(texts[0])"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a1a118b1",
+   "id": "0905c1de",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -501,6 +550,11 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.10.9"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
+   }
   }
  },
  "nbformat": 4,

docs/modules/utils/combine_docs_examples/vectorstores.ipynb

@@ -16,7 +16,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 1,
    "id": "965eecee",
    "metadata": {
     "pycharm": {
@@ -27,12 +27,12 @@
    "source": [
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain.vectorstores import ElasticVectorSearch, Pinecone, Weaviate, FAISS"
+    "from langchain.vectorstores import ElasticVectorSearch, Pinecone, Weaviate, FAISS, Qdrant"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 2,
    "id": "68481687",
    "metadata": {
     "pycharm": {
@@ -51,7 +51,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 3,
    "id": "015f4ff5",
    "metadata": {
     "pycharm": {
@@ -68,7 +68,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 4,
    "id": "67baf32e",
    "metadata": {
     "pycharm": {
@@ -98,6 +98,68 @@
     "print(docs[0].page_content)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "fb6baaf8",
+   "metadata": {},
+   "source": [
+    "## Add texts\n",
+    "You can easily add text to a vectorstore with the `add_texts` method. It will return a list of document IDs (in case you need to use them downstream)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "70758e4f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['64108bd0-4d91-485c-9743-1e18debdd59e']"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "docsearch.add_texts([\"Ankush went to Princeton\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "4edeb88f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"Where did Ankush go to college?\"\n",
+    "docs = docsearch.similarity_search(query)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "1cba64a2",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Document(page_content='Ankush went to Princeton', lookup_str='', metadata={}, lookup_index=0)"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "docs[0]"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "bbf5ec44",
@@ -210,39 +272,27 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 4,
    "id": "b58b3955",
    "metadata": {},
    "outputs": [],
    "source": [
-    "import pickle"
+    "docsearch.save_local(\"faiss_index\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
-   "id": "1897e23d",
+   "execution_count": 5,
+   "id": "ca72c650",
    "metadata": {},
    "outputs": [],
    "source": [
-    "with open(\"foo.pkl\", 'wb') as f:\n",
-    "    pickle.dump(docsearch, f)"
+    "new_docsearch = FAISS.load_local(\"faiss_index\", embeddings)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
-   "id": "bf3732f1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with open(\"foo.pkl\", 'rb') as f:\n",
-    "    new_docsearch = pickle.load(f)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 6,
    "id": "5bf2ee24",
    "metadata": {},
    "outputs": [],
@@ -252,7 +302,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 7,
    "id": "edc2aad1",
    "metadata": {},
    "outputs": [
@@ -262,7 +312,7 @@
        "Document(page_content='In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \\n\\nWe cannot let this happen. \\n\\nTonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', lookup_str='', metadata={}, lookup_index=0)"
       ]
      },
-     "execution_count": 18,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -483,7 +533,10 @@
     "import pinecone \n",
     "\n",
     "# initialize pinecone\n",
-    "pinecone.init(api_key=\"\", environment=\"us-west1-gcp\")\n",
+    "pinecone.init(\n",
+    "    api_key=\"YOUR_API_KEY\",  # find at app.pinecone.io\n",
+    "    environment=\"YOUR_ENV\"  # next to api key in console\n",
+    ")\n",
     "\n",
     "index_name = \"langchain-demo\"\n",
     "\n",
@@ -514,10 +567,126 @@
     "docs[0]"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "9b852079",
+   "metadata": {},
+   "source": [
+    "## Qdrant"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e7d74bd2",
+   "id": "e5ec70ce",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "host = \"<---host name here --->\"\n",
+    "api_key = \"<---api key here--->\"\n",
+    "qdrant = Qdrant.from_texts(texts, embeddings, host=host, prefer_grpc=True, api_key=api_key)\n",
+    "query = \"What did the president say about Ketanji Brown Jackson\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "9805ad1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = qdrant.similarity_search(query)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "bd097a0e",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Document(page_content='In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \\n\\nWe cannot let this happen. \\n\\nTonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', lookup_str='', metadata={}, lookup_index=0)"
+      ]
+     },
+     "execution_count": 22,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "docs[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c3ec797",
+   "metadata": {},
+   "source": [
+    "## Milvus\n",
+    "To run, you should have a Milvus instance up and running: https://milvus.io/docs/install_standalone-docker.md"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "be347313",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.vectorstores import Milvus"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "f2eee23f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "vector_db = Milvus.from_texts(\n",
+    "    texts,\n",
+    "    embeddings,\n",
+    "    connection_args={\"host\": \"127.0.0.1\", \"port\": \"19530\"},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "06bdb701",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = vector_db.similarity_search(query)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "7b3e94aa",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Document(page_content='In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \\n\\nWe cannot let this happen. \\n\\nTonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', lookup_str='', metadata={}, lookup_index=0)"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "docs[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4af5a071",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -539,7 +708,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/utils/examples/bing_search.ipynb

@@ -0,0 +1,192 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Bing Search"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook goes over how to use the bing search component.\n",
+    "\n",
+    "First, you need to set up the proper API keys and environment variables. To set it up, follow the instructions found [here](https://levelup.gitconnected.com/api-tutorial-how-to-use-bing-web-search-api-in-python-4165d5592a7e).\n",
+    "\n",
+    "Then we will need to set some environment variables."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ[\"BING_SUBSCRIPTION_KEY\"] = \"\"\n",
+    "os.environ[\"BING_SEARCH_URL\"] = \"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.utilities import BingSearchAPIWrapper"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search = BingSearchAPIWrapper()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Thanks to the flexibility of <b>Python</b> and the powerful ecosystem of packages, the Azure CLI supports features such as autocompletion (in shells that support it), persistent credentials, JMESPath result parsing, lazy initialization, network-less unit tests, and more. Building an open-source and cross-platform Azure CLI with <b>Python</b> by Dan Taylor. <b>Python</b> releases by version number: Release version Release date Click for more. <b>Python</b> 3.11.1 Dec. 6, 2022 Download Release Notes. <b>Python</b> 3.10.9 Dec. 6, 2022 Download Release Notes. <b>Python</b> 3.9.16 Dec. 6, 2022 Download Release Notes. <b>Python</b> 3.8.16 Dec. 6, 2022 Download Release Notes. <b>Python</b> 3.7.16 Dec. 6, 2022 Download Release Notes. In this lesson, we will look at the += operator in <b>Python</b> and see how it works with several simple examples.. The operator ‘+=’ is a shorthand for the addition assignment operator.It adds two values and assigns the sum to a variable (left operand). W3Schools offers free online tutorials, references and exercises in all the major languages of the web. Covering popular subjects like HTML, CSS, JavaScript, <b>Python</b>, SQL, Java, and many, many more. This tutorial introduces the reader informally to the basic concepts and features of the <b>Python</b> language and system. It helps to have a <b>Python</b> interpreter handy for hands-on experience, but all examples are self-contained, so the tutorial can be read off-line as well. For a description of standard objects and modules, see The <b>Python</b> Standard ... <b>Python</b> is a general-purpose, versatile, and powerful programming language. It&#39;s a great first language because <b>Python</b> code is concise and easy to read. Whatever you want to do, <b>python</b> can do it. From web development to machine learning to data science, <b>Python</b> is the language for you. To install <b>Python</b> using the Microsoft Store: Go to your Start menu (lower left Windows icon), type &quot;Microsoft Store&quot;, select the link to open the store. Once the store is open, select Search from the upper-right menu and enter &quot;<b>Python</b>&quot;. Select which version of <b>Python</b> you would like to use from the results under Apps. Under the “<b>Python</b> Releases for Mac OS X” heading, click the link for the Latest <b>Python</b> 3 Release - <b>Python</b> 3.x.x. As of this writing, the latest version was <b>Python</b> 3.8.4. Scroll to the bottom and click macOS 64-bit installer to start the download. When the installer is finished downloading, move on to the next step. Step 2: Run the Installer'"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search.run(\"python\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Number of results\n",
+    "You can use the `k` parameter to set the number of results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search = BingSearchAPIWrapper(k=1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Thanks to the flexibility of <b>Python</b> and the powerful ecosystem of packages, the Azure CLI supports features such as autocompletion (in shells that support it), persistent credentials, JMESPath result parsing, lazy initialization, network-less unit tests, and more. Building an open-source and cross-platform Azure CLI with <b>Python</b> by Dan Taylor.'"
+      ]
+     },
+     "execution_count": 25,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search.run(\"python\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Metadata Results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run query through BingSearch and return snippet, title, and link metadata.\n",
+    "\n",
+    "- Snippet: The description of the result.\n",
+    "- Title: The title of the result.\n",
+    "- Link: The link to the result."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search = BingSearchAPIWrapper()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'snippet': 'Lady Alice. Pink Lady <b>apples</b> aren’t the only lady in the apple family. Lady Alice <b>apples</b> were discovered growing, thanks to bees pollinating, in Washington. They are smaller and slightly more stout in appearance than other varieties. Their skin color appears to have red and yellow stripes running from stem to butt.',\n",
+       "  'title': '25 Types of Apples - Jessica Gavin',\n",
+       "  'link': 'https://www.jessicagavin.com/types-of-apples/'},\n",
+       " {'snippet': '<b>Apples</b> can do a lot for you, thanks to plant chemicals called flavonoids. And they have pectin, a fiber that breaks down in your gut. If you take off the apple’s skin before eating it, you won ...',\n",
+       "  'title': 'Apples: Nutrition &amp; Health Benefits - WebMD',\n",
+       "  'link': 'https://www.webmd.com/food-recipes/benefits-apples'},\n",
+       " {'snippet': '<b>Apples</b> boast many vitamins and minerals, though not in high amounts. However, <b>apples</b> are usually a good source of vitamin C. Vitamin C. Also called ascorbic acid, this vitamin is a common ...',\n",
+       "  'title': 'Apples 101: Nutrition Facts and Health Benefits',\n",
+       "  'link': 'https://www.healthline.com/nutrition/foods/apples'},\n",
+       " {'snippet': 'Weight management. The fibers in <b>apples</b> can slow digestion, helping one to feel greater satisfaction after eating. After following three large prospective cohorts of 133,468 men and women for 24 years, researchers found that higher intakes of fiber-rich fruits with a low glycemic load, particularly <b>apples</b> and pears, were associated with the least amount of weight gain over time.',\n",
+       "  'title': 'Apples | The Nutrition Source | Harvard T.H. Chan School of Public Health',\n",
+       "  'link': 'https://www.hsph.harvard.edu/nutritionsource/food-features/apples/'}]"
+      ]
+     },
+     "execution_count": 27,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search.results(\"apples\", 5)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/modules/utils/examples/google_search.ipynb

@@ -16,19 +16,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 1,
    "id": "34bb5968",
    "metadata": {},
    "outputs": [],
    "source": [
     "import os\n",
-    "os.environ[\"GOOGLE_CSE_ID\"] = \n",
-    "os.environ[\"GOOGLE_API_KEY\"] = "
+    "os.environ[\"GOOGLE_CSE_ID\"] = \"\"\n",
+    "os.environ[\"GOOGLE_API_KEY\"] = \"\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 2,
    "id": "ac4910f8",
    "metadata": {},
    "outputs": [],
@@ -38,7 +38,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 3,
    "id": "84b8f773",
    "metadata": {},
    "outputs": [],
@@ -48,17 +48,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 4,
    "id": "068991a6",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "'STATE OF HAWAII. 1 Child\\'s First Name. (Type or print). 2. Sex. BARACK. 3. This Birth. CERTIFICATE OF LIVE BIRTH. FILE. NUMBER 151 le. lb. Middle Name. Barack Hussein Obama II is an American politician who served as the 44th president of the United States from 2009 to 2017. A member of the Democratic Party,\\xa0... First Lady Michelle LaVaughn Robinson Obama is a lawyer, writer, and the wife of the 44th President, Barack Obama. She is the first African-American First\\xa0... Barack Obama, in full Barack Hussein Obama II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009–17) and the first\\xa0... Aug 18, 2017 ... It took him several seconds and multiple clues to remember former President Barack Obama\\'s first name. Miller knew that every answer had to\\xa0... Feb 9, 2015 ... Michael Jordan misspelled Barack Obama\\'s first name on 50th-birthday gift ... Knowing Obama is a Chicagoan and huge basketball fan,\\xa0... His full name is Barack Hussein Obama II. Since the “II” is simply because he was named for his father, his last name is Obama. Jan 16, 2007 ... 4, 1961, in Honolulu. His first name means \"one who is blessed\" in Swahili. While Obama\\'s father, Barack Hussein Obama Sr., was from Kenya, his\\xa0... Jan 19, 2017 ... Hopeful parents named their sons for the first Black president, whose name is a variation of the Hebrew name Baruch, which means “blessed”\\xa0... Feb 27, 2020 ... President Barack Obama was born Barack Hussein Obama, II, as shown here on his birth certificate here . As reported by Reuters here , his\\xa0...'"
+       "'1 Child\\'s First Name. 2. 6. 7d. Street Address. 71. (Type or print). BARACK. Sex. 3. This Birth. 4. If Twin or Triplet,. Was Child Born. Barack Hussein Obama II is an American retired politician who served as the 44th president of the United States from 2009 to 2017. His full name is Barack Hussein Obama II. Since the “II” is simply because he was named for his father, his last name is Obama. Feb 9, 2015 ... Michael Jordan misspelled Barack Obama\\'s first name on 50th-birthday gift ... Knowing Obama is a Chicagoan and huge basketball fan,\\xa0... Aug 18, 2017 ... It took him several seconds and multiple clues to remember former President Barack Obama\\'s first name. Miller knew that every answer had to end\\xa0... First Lady Michelle LaVaughn Robinson Obama is a lawyer, writer, and the wife of the 44th President, Barack Obama. She is the first African-American First\\xa0... Barack Obama, in full Barack Hussein Obama II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009–17) and the first\\xa0... When Barack Obama was elected president in 2008, he became the first African American to hold ... The Middle East remained a key foreign policy challenge. Feb 27, 2020 ... President Barack Obama was born Barack Hussein Obama, II, as shown here on his birth certificate here . As reported by Reuters here , his\\xa0... Jan 16, 2007 ... 4, 1961, in Honolulu. His first name means \"one who is blessed\" in Swahili. While Obama\\'s father, Barack Hussein Obama Sr., was from Kenya, his\\xa0...'"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -67,13 +67,118 @@
     "search.run(\"Obama's first name?\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "074b7f07",
+   "metadata": {},
+   "source": [
+    "## Number of Results\n",
+    "You can use the `k` parameter to set the number of results"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 5,
+   "id": "5083fbdd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search = GoogleSearchAPIWrapper(k=1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "77aaa857",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The official home of the Python Programming Language.'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search.run(\"python\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11c8d94f",
+   "metadata": {},
+   "source": [
+    "'The official home of the Python Programming Language.'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "73473110",
+   "metadata": {},
+   "source": [
+    "## Metadata Results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "109fe796",
+   "metadata": {},
+   "source": [
+    "Run query through GoogleSearch and return snippet, title, and link metadata.\n",
+    "\n",
+    "- Snippet: The description of the result.\n",
+    "- Title: The title of the result.\n",
+    "- Link: The link to the result."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
    "id": "028f4cba",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "search = GoogleSearchAPIWrapper()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "4d8f734f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'snippet': 'Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment,\\xa0...',\n",
+       "  'title': 'Apple',\n",
+       "  'link': 'https://www.apple.com/'},\n",
+       " {'snippet': \"Jul 10, 2022 ... Whether or not you're up on your apple trivia, no doubt you know how delicious this popular fruit is, and how nutritious. Apples are rich in\\xa0...\",\n",
+       "  'title': '25 Types of Apples and What to Make With Them - Parade ...',\n",
+       "  'link': 'https://parade.com/1330308/bethlipton/types-of-apples/'},\n",
+       " {'snippet': 'An apple is an edible fruit produced by an apple tree (Malus domestica). Apple trees are cultivated worldwide and are the most widely grown species in the\\xa0...',\n",
+       "  'title': 'Apple - Wikipedia',\n",
+       "  'link': 'https://en.wikipedia.org/wiki/Apple'},\n",
+       " {'snippet': 'Apples are a popular fruit. They contain antioxidants, vitamins, dietary fiber, and a range of other nutrients. Due to their varied nutrient content,\\xa0...',\n",
+       "  'title': 'Apples: Benefits, nutrition, and tips',\n",
+       "  'link': 'https://www.medicalnewstoday.com/articles/267290'},\n",
+       " {'snippet': \"An apple is a crunchy, bright-colored fruit, one of the most popular in the United States. You've probably heard the age-old saying, “An apple a day keeps\\xa0...\",\n",
+       "  'title': 'Apples: Nutrition & Health Benefits',\n",
+       "  'link': 'https://www.webmd.com/food-recipes/benefits-apples'}]"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "search.results(\"apples\", 5)"
+   ]
   }
  ],
  "metadata": {
@@ -93,6 +198,11 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.10.9"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
+   }
   }
  },
  "nbformat": 4,

docs/tracing.md

@@ -0,0 +1,57 @@
+# Tracing
+
+By enabling tracing in your LangChain runs, you’ll be able to more effectively visualize, step through, and debug your chains and agents.
+
+First, you should install tracing and set up your environment properly.
+You can use either a locally hosted version of this (uses Docker) or a cloud hosted version (in closed alpha).
+If you're interested in using the hosted platform, please fill out the form [here](https://forms.gle/tRCEMSeopZf6TE3b6).
+
+
+- [Locally Hosted Setup](./tracing/local_installation.md)
+- [Cloud Hosted Setup](./tracing/hosted_installation.md)
+
+## Tracing Walkthrough
+
+When you first access the UI, you should see a page with your tracing sessions. 
+An initial one "default" should already be created for you. 
+A session is just a way to group traces together. 
+If you click on a session, it will take you to a page with no recorded traces that says "No Runs." 
+You can create a new session with the new session form.
+
+![](tracing/homepage.png)
+
+If we click on the `default` session, we can see that to start we have no traces stored.
+
+![](tracing/default_empty.png)
+
+If we now start running chains and agents with tracing enabled, we will see data show up here.
+To do so, we can run [this notebook](tracing/agent_with_tracing.ipynb) as an example.
+After running it, we will see an initial trace show up.
+
+![](tracing/first_trace.png)
+
+From here we can explore the trace at a high level by clicking on the arrow to show nested runs.
+We can keep on clicking further and further down to explore deeper and deeper.
+
+![](tracing/explore.png)
+
+We can also click on the "Explore" button of the top level run to dive even deeper. 
+Here, we can see the inputs and outputs in full, as well as all the nested traces.
+
+![](tracing/explore_trace.png)
+
+We can keep on exploring each of these nested traces in more detail.
+For example, here is the lowest level trace with the exact inputs/outputs to the LLM.
+
+![](tracing/explore_llm.png)
+
+## Changing Sessions
+1. To initially record traces to a session other than `"default"`, you can set the `LANGCHAIN_SESSION` environment variable to the name of the session you want to record to:
+
+```python
+import os
+os.environ["LANGCHAIN_HANDLER"] = "langchain"
+os.environ["LANGCHAIN_SESSION"] = "my_session" # Make sure this session actually exists. You can create a new session in the UI.
+```
+
+2. To switch sessions mid-script or mid-notebook, do NOT set the `LANGCHAIN_SESSION` environment variable. Instead: `langchain.set_tracing_callback_manager(session_name="my_session")`

docs/tracing/agent_with_tracing.ipynb

@@ -0,0 +1,116 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "5371a9bb",
+   "metadata": {},
+   "source": [
+    "# Tracing Walkthrough"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "17c04cc6-c93d-4b6c-a033-e897577f4ed1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\"\n",
+    "\n",
+    "## Uncomment this if using hosted setup.\n",
+    "\n",
+    "# os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://langchain-api-gateway-57eoxz8z.uc.gateway.dev\" \n",
+    "\n",
+    "## Uncomment this if you want traces to be recorded to \"my_session\" instead of default.\n",
+    "\n",
+    "# os.environ[\"LANGCHAIN_SESSION\"] = \"my_session\"  \n",
+    "\n",
+    "## Better to set this environment variable in the terminal\n",
+    "## Uncomment this if using hosted version. Replace \"my_api_key\" with your actual API Key.\n",
+    "\n",
+    "# os.environ[\"LANGCHAIN_API_KEY\"] = \"my_api_key\"  \n",
+    "\n",
+    "import langchain\n",
+    "from langchain.agents import Tool, initialize_agent, load_tools\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "bfa16b79-aa4b-4d41-a067-70d1f593f667",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to use a calculator to solve this.\n",
+      "Action: Calculator\n",
+      "Action Input: 2^.123243\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 1.0891804557407723\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
+      "Final Answer: 1.0891804557407723\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'1.0891804557407723'"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Agent run with tracing. Ensure that OPENAI_API_KEY is set appropriately to run this example.\n",
+    "\n",
+    "llm = OpenAI(temperature=0)\n",
+    "tools = load_tools([\"llm-math\"], llm=llm)\n",
+    "agent = initialize_agent(\n",
+    "    tools, llm, agent=\"zero-shot-react-description\", verbose=True\n",
+    ")\n",
+    "\n",
+    "agent.run(\"What is 2 raised to .123243 power?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "25addd7f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}