cr

Added aws texract extraction python file to the document loader module .
This will help in extracting images text with aws texract service.

.github/CONTRIBUTING.md

@@ -75,7 +75,7 @@ This will install all requirements for running the package, examples, linting, f
 
 ❗Note: If you're running Poetry 1.4.1 and receive a `WheelFileValidationError` for `debugpy` during installation, you can try either downgrading to Poetry 1.4.0 or disabling "modern installation" (`poetry config installer.modern-installation false`) and re-install requirements. See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details.
 
-Now, you should be able to run the common tasks in the following section. To double check, run `make test`, all tests should pass. If they don't you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
+Now, you should be able to run the common tasks in the following section.
 
 ## ✅Common Tasks
 

.github/workflows/linkcheck.yml

@@ -6,7 +6,7 @@ on:
   pull_request:
 
 env:
-  POETRY_VERSION: "1.4.2"
+  POETRY_VERSION: "1.3.1"
 
 jobs:
   build:

.github/workflows/lint.yml

@@ -6,7 +6,7 @@ on:
   pull_request:
 
 env:
-  POETRY_VERSION: "1.4.2"
+  POETRY_VERSION: "1.3.1"
 
 jobs:
   build:

.github/workflows/release.yml

@@ -10,7 +10,7 @@ on:
       - 'pyproject.toml'
 
 env:
-  POETRY_VERSION: "1.4.2"
+  POETRY_VERSION: "1.3.1"
 
 jobs:
   if_release:
@@ -45,5 +45,5 @@ jobs:
       - name: Publish to PyPI
         env:
           POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
-        run: |
+        run: | 
           poetry publish

.github/workflows/test.yml

@@ -6,7 +6,7 @@ on:
   pull_request:
 
 env:
-  POETRY_VERSION: "1.4.2"
+  POETRY_VERSION: "1.3.1"
 
 jobs:
   build:

.gitignore

@@ -142,10 +142,3 @@ wandb/
 # asdf tool versions
 .tool-versions
 /.ruff_cache/
-
-*.pkl
-*.bin
-
-# integration test artifacts
-data_map*
-\[('_type', 'fake'), ('stop', None)]

README.md

@@ -2,9 +2,7 @@
 
 ⚡ Building applications with LLMs through composability ⚡
 
-[![lint](https://github.com/hwchase17/langchain/actions/workflows/lint.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/lint.yml) [![test](https://github.com/hwchase17/langchain/actions/workflows/test.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/test.yml) [![linkcheck](https://github.com/hwchase17/langchain/actions/workflows/linkcheck.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/linkcheck.yml) [![Downloads](https://static.pepy.tech/badge/langchain/month)](https://pepy.tech/project/langchain) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40LangChainAI)](https://twitter.com/langchainai) [![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true&style=flat)](https://discord.gg/6adMQxSpJS)
-
-Looking for the JS/TS version? Check out [LangChain.js](https://github.com/hwchase17/langchainjs).
+[![lint](https://github.com/hwchase17/langchain/actions/workflows/lint.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/lint.yml) [![test](https://github.com/hwchase17/langchain/actions/workflows/test.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/test.yml) [![linkcheck](https://github.com/hwchase17/langchain/actions/workflows/linkcheck.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/linkcheck.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40LangChainAI)](https://twitter.com/langchainai) [![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true&style=flat)](https://discord.gg/6adMQxSpJS)
 
 **Production Support:** As you move your LangChains into production, we'd love to offer more comprehensive support.
 Please fill out [this form](https://forms.gle/57d8AmXBYp8PP8tZA) and we'll set up a dedicated support Slack channel.
@@ -17,9 +15,12 @@ or
 
 ## 🤔 What is this?
 
-Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. However, using these LLMs in isolation is often insufficient for creating a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.
+Large language models (LLMs) are emerging as a transformative technology, enabling
+developers to build applications that they previously could not.
+But using these LLMs in isolation is often not enough to
+create a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.
 
-This library aims to assist in the development of those types of applications. Common examples of these applications include:
+This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:
 
 **❓ Question Answering over specific documents**
 
@@ -52,23 +53,23 @@ These are, in increasing order of complexity:
 
 **📃 LLMs and Prompts:**
 
-This includes prompt management, prompt optimization, a generic interface for all LLMs, and common utilities for working with LLMs.
+This includes prompt management, prompt optimization, generic interface for all LLMs, and common utilities for working with LLMs.
 
 **🔗 Chains:**
 
-Chains go beyond a single LLM call and involve sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.
+Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.
 
 **📚 Data Augmented Generation:**
 
-Data Augmented Generation involves specific types of chains that first interact with an external data source to fetch data for use in the generation step. Examples include summarization of long pieces of text and question/answering over specific data sources.
+Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources.
 
 **🤖 Agents:**
 
-Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end-to-end agents.
+Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.
 
 **🧠 Memory:**
 
-Memory refers to persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.
+Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.
 
 **🧐 Evaluation:**
 
@@ -78,6 +79,6 @@ For more information on these concepts, please see our [full documentation](http
 
 ## 💁 Contributing
 
-As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
+As an open source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infra, or better documentation.
 
 For detailed information on how to contribute, see [here](.github/CONTRIBUTING.md).

docs/_static/css/custom.css

@@ -11,7 +11,3 @@ pre {
     max-width: 2560px !important;
   }
 }
-
-#my-component-root *, #headlessui-portal-root * {
-  z-index: 1000000000000;
-}

docs/_static/js/mendablesearch.js

@@ -1,58 +0,0 @@
-document.addEventListener('DOMContentLoaded', () => {
-  // Load the external dependencies
-  function loadScript(src, onLoadCallback) {
-    const script = document.createElement('script');
-    script.src = src;
-    script.onload = onLoadCallback;
-    document.head.appendChild(script);
-  }
-
-  function createRootElement() {
-    const rootElement = document.createElement('div');
-    rootElement.id = 'my-component-root';
-    document.body.appendChild(rootElement);
-    return rootElement;
-  }
-
-  
-
-  function initializeMendable() {
-    const rootElement = createRootElement();
-    const { MendableFloatingButton } = Mendable;
-    
-
-    const iconSpan1 = React.createElement('span', {
-    }, '🦜');
-
-    const iconSpan2 = React.createElement('span', {
-    }, '🔗');
-
-    const icon = React.createElement('p', {
-      style: { color: '#ffffff', fontSize: '22px',width: '48px', height: '48px', margin: '0px', padding: '0px', display: 'flex', alignItems: 'center', justifyContent: 'center', textAlign: 'center' },
-    }, [iconSpan1, iconSpan2]);
-
-    
-    
-
-    const mendableFloatingButton = React.createElement(
-      MendableFloatingButton,
-      {
-        style: { darkMode: false, accentColor: '#010810' },
-        floatingButtonStyle: { color: '#ffffff', backgroundColor: '#010810' },
-        anon_key: '82842b36-3ea6-49b2-9fb8-52cfc4bde6bf', // Mendable Search Public ANON key, ok to be public
-        messageSettings: {
-          openSourcesInNewTab: false,
-        },
-        icon: icon,
-      }
-    );
-
-    ReactDOM.render(mendableFloatingButton, rootElement);
-  }
-
-  loadScript('https://unpkg.com/react@17/umd/react.production.min.js', () => {
-    loadScript('https://unpkg.com/react-dom@17/umd/react-dom.production.min.js', () => {
-      loadScript('https://unpkg.com/@mendable/search@0.0.83/dist/umd/mendable.min.js', initializeMendable);
-    });
-  });
-});

docs/conf.py

@@ -103,10 +103,5 @@ html_static_path = ["_static"]
 html_css_files = [
     "css/custom.css",
 ]
-
-html_js_files = [
-    "js/mendablesearch.js",
-]
-
 nb_execution_mode = "off"
 myst_enable_extensions = ["colon_fence"]

docs/deployments.md

@@ -1,10 +1,14 @@
 # Deployments
 
-So, you've created a really cool chain - now what? How do you deploy it and make it easily shareable with the world?
+So you've made a really cool chain - now what? How do you deploy it and make it easily sharable with the world?
 
-This section covers several options for that. Note that these options are meant for quick deployment of prototypes and demos, not for production systems. If you need help with the deployment of a production system, please contact us directly.
+This section covers several options for that.
+Note that these are meant as quick deployment options for prototypes and demos, and not for production systems.
+If you are looking for help with deployment of a production system, please contact us directly.
 
-What follows is a list of template GitHub repositories designed to be easily forked and modified to use your chain. This list is far from exhaustive, and we are EXTREMELY open to contributions here.
+What follows is a list of template GitHub repositories aimed that are intended to be
+very easy to fork and modify to use your chain.
+This is far from an exhaustive list of options, and we are EXTREMELY open to contributions here.
 
 ## [Streamlit](https://github.com/hwchase17/langchain-streamlit-template)
 
@@ -29,30 +33,14 @@ It implements a Question Answering app and contains instructions for deploying t
 
 A minimal example on how to run LangChain on Vercel using Flask.
 
-## [Fly.io](https://github.com/fly-apps/hello-fly-langchain)
-
-A minimal example of how to deploy LangChain to [Fly.io](https://fly.io/) using Flask.
-
-## [Digitalocean App Platform](https://github.com/homanp/digitalocean-langchain)
-
-A minimal example on how to deploy LangChain to DigitalOcean App Platform.
-
-## [Google Cloud Run](https://github.com/homanp/gcp-langchain)
-
-A minimal example on how to deploy LangChain to Google Cloud Run.
 
 ## [SteamShip](https://github.com/steamship-core/steamship-langchain/)
-
-This repository contains LangChain adapters for Steamship, enabling LangChain developers to rapidly deploy their apps on Steamship. This includes: production-ready endpoints, horizontal scaling across dependencies, persistent storage of app state, multi-tenancy support, etc.
+This repository contains LangChain adapters for Steamship, enabling LangChain developers to rapidly deploy their apps on Steamship.
+This includes: production ready endpoints, horizontal scaling across dependencies, persistant storage of app state, multi-tenancy support, etc.
 
 ## [Langchain-serve](https://github.com/jina-ai/langchain-serve)
-
-This repository allows users to serve local chains and agents as RESTful, gRPC, or WebSocket APIs, thanks to [Jina](https://docs.jina.ai/). Deploy your chains & agents with ease and enjoy independent scaling, serverless and autoscaling APIs, as well as a Streamlit playground on Jina AI Cloud.
+This repository allows users to serve local chains and agents as RESTful, gRPC, or Websocket APIs thanks to [Jina](https://docs.jina.ai/). Deploy your chains & agents with ease and enjoy independent scaling, serverless and autoscaling APIs, as well as a Streamlit playground on Jina AI Cloud.
 
 ## [BentoML](https://github.com/ssheng/BentoChain)
 
 This repository provides an example of how to deploy a LangChain application with [BentoML](https://github.com/bentoml/BentoML). BentoML is a framework that enables the containerization of machine learning applications as standard OCI images. BentoML also allows for the automatic generation of OpenAPI and gRPC endpoints. With BentoML, you can integrate models from all popular ML frameworks and deploy them as microservices running on the most optimal hardware and scaling independently.
-
-## [Databutton](https://databutton.com/home?new-data-app=true)
-
-These templates serve as examples of how to build, deploy, and share LangChain applications using Databutton. You can create user interfaces with Streamlit, automate tasks by scheduling Python code, and store files and data in the built-in store. Examples include a Chatbot interface with conversational memory, a Personal search engine, and a starter template for LangChain apps. Deploying and sharing is just one click away.

docs/ecosystem.rst

@@ -3,25 +3,6 @@ LangChain Ecosystem
 
 Guides for how other companies/products can be used with LangChain
 
-Groups
-----------
-
-LangChain provides integration with many LLMs and systems:
-
-- `LLM Providers <./modules/models/llms/integrations.html>`_
-- `Chat Model Providers <./modules/models/chat/integrations.html>`_
-- `Text Embedding Model Providers <./modules/models/text_embedding.html>`_
-- `Document Loader Integrations <./modules/indexes/document_loaders.html>`_
-- `Text Splitter Integrations <./modules/indexes/text_splitters.html>`_
-- `Vectorstore Providers <./modules/indexes/vectorstores.html>`_
-- `Retriever Providers <./modules/indexes/retrievers.html>`_
-- `Tool Providers <./modules/agents/tools.html>`_
-- `Toolkit Integrations <./modules/agents/toolkits.html>`_
-
-Companies / Products
-----------
-
-
 .. toctree::
    :maxdepth: 1
    :glob:

docs/ecosystem/aim_tracking.ipynb

@@ -61,6 +61,7 @@
     "from datetime import datetime\n",
     "\n",
     "from langchain.llms import OpenAI\n",
+    "from langchain.callbacks.base import CallbackManager\n",
     "from langchain.callbacks import AimCallbackHandler, StdOutCallbackHandler"
    ]
   },
@@ -108,8 +109,8 @@
     "    experiment_name=\"scenario 1: OpenAI LLM\",\n",
     ")\n",
     "\n",
-    "callbacks = [StdOutCallbackHandler(), aim_callback]\n",
-    "llm = OpenAI(temperature=0, callbacks=callbacks)"
+    "manager = CallbackManager([StdOutCallbackHandler(), aim_callback])\n",
+    "llm = OpenAI(temperature=0, callback_manager=manager, verbose=True)"
    ]
   },
   {
@@ -176,7 +177,7 @@
     "Title: {title}\n",
     "Playwright: This is a synopsis for the above play:\"\"\"\n",
     "prompt_template = PromptTemplate(input_variables=[\"title\"], template=template)\n",
-    "synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)\n",
+    "synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callback_manager=manager)\n",
     "\n",
     "test_prompts = [\n",
     "    {\"title\": \"documentary about good video games that push the boundary of game design\"},\n",
@@ -248,12 +249,13 @@
    ],
    "source": [
     "# scenario 3 - Agent with Tools\n",
-    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callbacks=callbacks)\n",
+    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callback_manager=manager)\n",
     "agent = initialize_agent(\n",
     "    tools,\n",
     "    llm,\n",
     "    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
-    "    callbacks=callbacks,\n",
+    "    callback_manager=manager,\n",
+    "    verbose=True,\n",
     ")\n",
     "agent.run(\n",
     "    \"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?\"\n",

docs/ecosystem/analyticdb.md

@@ -1,15 +0,0 @@
-# AnalyticDB
-
-This page covers how to use the AnalyticDB ecosystem within LangChain.
-
-### VectorStore
-
-There exists a wrapper around AnalyticDB, allowing you to use it as a vectorstore,
-whether for semantic search or example selection.
-
-To import this vectorstore:
-```python
-from langchain.vectorstores import AnalyticDB
-```
-
-For a more detailed walkthrough of the AnalyticDB wrapper, see [this notebook](../modules/indexes/vectorstores/examples/analyticdb.ipynb)

docs/ecosystem/clearml_tracking.ipynb

@@ -19,7 +19,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Getting API Credentials\n",
+    "# Getting API Credentials\n",
     "\n",
     "We'll be using quite some APIs in this notebook, here is a list and where to get them:\n",
     "\n",
@@ -47,7 +47,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Setting Up"
+    "# Setting Up"
    ]
   },
   {
@@ -79,6 +79,7 @@
    "source": [
     "from datetime import datetime\n",
     "from langchain.callbacks import ClearMLCallbackHandler, StdOutCallbackHandler\n",
+    "from langchain.callbacks.base import CallbackManager\n",
     "from langchain.llms import OpenAI\n",
     "\n",
     "# Setup and use the ClearML Callback\n",
@@ -92,9 +93,9 @@
     "    complexity_metrics=True,\n",
     "    stream_logs=True\n",
     ")\n",
-    "callbacks = [StdOutCallbackHandler(), clearml_callback]\n",
+    "manager = CallbackManager([StdOutCallbackHandler(), clearml_callback])\n",
     "# Get the OpenAI model ready to go\n",
-    "llm = OpenAI(temperature=0, callbacks=callbacks)"
+    "llm = OpenAI(temperature=0, callback_manager=manager, verbose=True)"
    ]
   },
   {
@@ -102,7 +103,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Scenario 1: Just an LLM\n",
+    "# Scenario 1: Just an LLM\n",
     "\n",
     "First, let's just run a single LLM a few times and capture the resulting prompt-answer conversation in ClearML"
    ]
@@ -360,7 +361,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Scenario 2: Creating an agent with tools\n",
+    "# Scenario 2: Creating a agent with tools\n",
     "\n",
     "To show a more advanced workflow, let's create an agent with access to tools. The way ClearML tracks the results is not different though, only the table will look slightly different as there are other types of actions taken when compared to the earlier, simpler example.\n",
     "\n",
@@ -522,12 +523,13 @@
     "from langchain.agents import AgentType\n",
     "\n",
     "# SCENARIO 2 - Agent with Tools\n",
-    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callbacks=callbacks)\n",
+    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callback_manager=manager)\n",
     "agent = initialize_agent(\n",
     "    tools,\n",
     "    llm,\n",
     "    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
-    "    callbacks=callbacks,\n",
+    "    callback_manager=manager,\n",
+    "    verbose=True,\n",
     ")\n",
     "agent.run(\n",
     "    \"Who is the wife of the person who sang summer of 69?\"\n",
@@ -540,7 +542,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Tips and Next Steps\n",
+    "# Tips and Next Steps\n",
     "\n",
     "- Make sure you always use a unique `name` argument for the `clearml_callback.flush_tracker` function. If not, the model parameters used for a run will override the previous run!\n",
     "\n",

docs/ecosystem/comet_tracking.ipynb

@@ -1,347 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Comet"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "![](https://user-images.githubusercontent.com/7529846/230328046-a8b18c51-12e3-4617-9b39-97614a571a2d.png)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "In this guide we will demonstrate how to track your Langchain Experiments, Evaluation Metrics, and LLM Sessions with [Comet](https://www.comet.com/site/?utm_source=langchain&utm_medium=referral&utm_campaign=comet_notebook).  \n",
-    "\n",
-    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/ecosystem/comet_tracking.ipynb\">\n",
-    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
-    "</a>\n",
-    "\n",
-    "**Example Project:** [Comet with LangChain](https://www.comet.com/examples/comet-example-langchain/view/b5ZThK6OFdhKWVSP3fDfRtrNF/panels?utm_source=langchain&utm_medium=referral&utm_campaign=comet_notebook)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "<img width=\"1280\" alt=\"comet-langchain\" src=\"https://user-images.githubusercontent.com/7529846/230326720-a9711435-9c6f-4edb-a707-94b67271ab25.png\">\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Install Comet and Dependencies"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install comet_ml langchain openai google-search-results spacy textstat pandas\n",
-    "\n",
-    "import sys\n",
-    "!{sys.executable} -m spacy download en_core_web_sm"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Initialize Comet and Set your Credentials"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "You can grab your [Comet API Key here](https://www.comet.com/signup?utm_source=langchain&utm_medium=referral&utm_campaign=comet_notebook) or click the link after initializing Comet"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import comet_ml\n",
-    "\n",
-    "comet_ml.init(project_name=\"comet-example-langchain\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Set OpenAI and SerpAPI credentials"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "You will need an [OpenAI API Key](https://platform.openai.com/account/api-keys) and a [SerpAPI API Key](https://serpapi.com/dashboard) to run the following examples"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"...\"\n",
-    "#os.environ[\"OPENAI_ORGANIZATION\"] = \"...\"\n",
-    "os.environ[\"SERPAPI_API_KEY\"] = \"...\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Scenario 1: Using just an LLM"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from datetime import datetime\n",
-    "\n",
-    "from langchain.callbacks import CometCallbackHandler, StdOutCallbackHandler\n",
-    "from langchain.llms import OpenAI\n",
-    "\n",
-    "comet_callback = CometCallbackHandler(\n",
-    "    project_name=\"comet-example-langchain\",\n",
-    "    complexity_metrics=True,\n",
-    "    stream_logs=True,\n",
-    "    tags=[\"llm\"],\n",
-    "    visualizations=[\"dep\"],\n",
-    ")\n",
-    "callbacks = [StdOutCallbackHandler(), comet_callback]\n",
-    "llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)\n",
-    "\n",
-    "llm_result = llm.generate([\"Tell me a joke\", \"Tell me a poem\", \"Tell me a fact\"] * 3)\n",
-    "print(\"LLM result\", llm_result)\n",
-    "comet_callback.flush_tracker(llm, finish=True)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Scenario 2: Using an LLM in a Chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.callbacks import CometCallbackHandler, StdOutCallbackHandler\n",
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "comet_callback = CometCallbackHandler(\n",
-    "    complexity_metrics=True,\n",
-    "    project_name=\"comet-example-langchain\",\n",
-    "    stream_logs=True,\n",
-    "    tags=[\"synopsis-chain\"],\n",
-    ")\n",
-    "callbacks = [StdOutCallbackHandler(), comet_callback]\n",
-    "llm = OpenAI(temperature=0.9, callbacks=callbacks)\n",
-    "\n",
-    "template = \"\"\"You are a playwright. Given the title of play, it is your job to write a synopsis for that title.\n",
-    "Title: {title}\n",
-    "Playwright: This is a synopsis for the above play:\"\"\"\n",
-    "prompt_template = PromptTemplate(input_variables=[\"title\"], template=template)\n",
-    "synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)\n",
-    "\n",
-    "test_prompts = [{\"title\": \"Documentary about Bigfoot in Paris\"}]\n",
-    "print(synopsis_chain.apply(test_prompts))\n",
-    "comet_callback.flush_tracker(synopsis_chain, finish=True)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Scenario 3: Using An Agent with Tools "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.agents import initialize_agent, load_tools\n",
-    "from langchain.callbacks import CometCallbackHandler, StdOutCallbackHandler\n",
-    "from langchain.llms import OpenAI\n",
-    "\n",
-    "comet_callback = CometCallbackHandler(\n",
-    "    project_name=\"comet-example-langchain\",\n",
-    "    complexity_metrics=True,\n",
-    "    stream_logs=True,\n",
-    "    tags=[\"agent\"],\n",
-    ")\n",
-    "callbacks = [StdOutCallbackHandler(), comet_callback]\n",
-    "llm = OpenAI(temperature=0.9, callbacks=callbacks)\n",
-    "\n",
-    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callbacks=callbacks)\n",
-    "agent = initialize_agent(\n",
-    "    tools,\n",
-    "    llm,\n",
-    "    agent=\"zero-shot-react-description\",\n",
-    "    callbacks=callbacks,\n",
-    "    verbose=True,\n",
-    ")\n",
-    "agent.run(\n",
-    "    \"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?\"\n",
-    ")\n",
-    "comet_callback.flush_tracker(agent, finish=True)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Scenario 4: Using Custom Evaluation Metrics"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The `CometCallbackManager` also allows you to define and use Custom Evaluation Metrics to assess generated outputs from your model. Let's take a look at how this works. \n",
-    "\n",
-    "\n",
-    "In the snippet below, we will use the [ROUGE](https://huggingface.co/spaces/evaluate-metric/rouge) metric to evaluate the quality of a generated summary of an input prompt. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install rouge-score"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from rouge_score import rouge_scorer\n",
-    "\n",
-    "from langchain.callbacks import CometCallbackHandler, StdOutCallbackHandler\n",
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "\n",
-    "class Rouge:\n",
-    "    def __init__(self, reference):\n",
-    "        self.reference = reference\n",
-    "        self.scorer = rouge_scorer.RougeScorer([\"rougeLsum\"], use_stemmer=True)\n",
-    "\n",
-    "    def compute_metric(self, generation, prompt_idx, gen_idx):\n",
-    "        prediction = generation.text\n",
-    "        results = self.scorer.score(target=self.reference, prediction=prediction)\n",
-    "\n",
-    "        return {\n",
-    "            \"rougeLsum_score\": results[\"rougeLsum\"].fmeasure,\n",
-    "            \"reference\": self.reference,\n",
-    "        }\n",
-    "\n",
-    "\n",
-    "reference = \"\"\"\n",
-    "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building.\n",
-    "It was the first structure to reach a height of 300 metres.\n",
-    "\n",
-    "It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft)\n",
-    "Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France .\n",
-    "\"\"\"\n",
-    "rouge_score = Rouge(reference=reference)\n",
-    "\n",
-    "template = \"\"\"Given the following article, it is your job to write a summary.\n",
-    "Article:\n",
-    "{article}\n",
-    "Summary: This is the summary for the above article:\"\"\"\n",
-    "prompt_template = PromptTemplate(input_variables=[\"article\"], template=template)\n",
-    "\n",
-    "comet_callback = CometCallbackHandler(\n",
-    "    project_name=\"comet-example-langchain\",\n",
-    "    complexity_metrics=False,\n",
-    "    stream_logs=True,\n",
-    "    tags=[\"custom_metrics\"],\n",
-    "    custom_metrics=rouge_score.compute_metric,\n",
-    ")\n",
-    "callbacks = [StdOutCallbackHandler(), comet_callback]\n",
-    "llm = OpenAI(temperature=0.9)\n",
-    "\n",
-    "synopsis_chain = LLMChain(llm=llm, prompt=prompt_template)\n",
-    "\n",
-    "test_prompts = [\n",
-    "    {\n",
-    "        \"article\": \"\"\"\n",
-    "                 The tower is 324 metres (1,063 ft) tall, about the same height as\n",
-    "                 an 81-storey building, and the tallest structure in Paris. Its base is square,\n",
-    "                 measuring 125 metres (410 ft) on each side.\n",
-    "                 During its construction, the Eiffel Tower surpassed the\n",
-    "                 Washington Monument to become the tallest man-made structure in the world,\n",
-    "                 a title it held for 41 years until the Chrysler Building\n",
-    "                 in New York City was finished in 1930.\n",
-    "\n",
-    "                 It was the first structure to reach a height of 300 metres.\n",
-    "                 Due to the addition of a broadcasting aerial at the top of the tower in 1957,\n",
-    "                 it is now taller than the Chrysler Building by 5.2 metres (17 ft).\n",
-    "\n",
-    "                 Excluding transmitters, the Eiffel Tower is the second tallest\n",
-    "                 free-standing structure in France after the Millau Viaduct.\n",
-    "                 \"\"\"\n",
-    "    }\n",
-    "]\n",
-    "print(synopsis_chain.apply(test_prompts, callbacks=callbacks))\n",
-    "comet_callback.flush_tracker(synopsis_chain, finish=True)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.15"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/ecosystem/gpt4all.md

@@ -3,7 +3,6 @@
 This page covers how to use the `GPT4All` wrapper within LangChain. The tutorial is divided into two parts: installation and setup, followed by usage with an example.
 
 ## Installation and Setup
-
 - Install the Python package with `pip install pyllamacpp`
 - Download a [GPT4All model](https://github.com/nomic-ai/pyllamacpp#supported-model) and place it in your desired directory
 
@@ -29,20 +28,20 @@ To stream the model's predictions, add in a CallbackManager.
 
 ```python
 from langchain.llms import GPT4All
+from langchain.callbacks.base import CallbackManager
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
-
 # There are many CallbackHandlers supported, such as
 # from langchain.callbacks.streamlit import StreamlitCallbackHandler
 
-callbacks = [StreamingStdOutCallbackHandler()]
-model = GPT4All(model="./models/gpt4all-model.bin", n_ctx=512, n_threads=8)
+callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])
+model = GPT4All(model="./models/gpt4all-model.bin", n_ctx=512, n_threads=8, callback_handler=callback_handler, verbose=True)
 
-# Generate text. Tokens are streamed through the callback manager.
-model("Once upon a time, ", callbacks=callbacks)
+# Generate text. Tokens are streamed throught the callback manager.
+model("Once upon a time, ")
 ```
 
 ## Model File
 
 You can find links to model file downloads in the [pyllamacpp](https://github.com/nomic-ai/pyllamacpp) repository.
 
-For a more detailed walkthrough of this, see [this notebook](../modules/models/llms/integrations/gpt4all.ipynb)
+For a more detailed walkthrough of this, see [this notebook](../modules/models/llms/integrations/gpt4all.ipynb)

docs/ecosystem/lancedb.md

@@ -1,23 +0,0 @@
-# LanceDB
-
-This page covers how to use [LanceDB](https://github.com/lancedb/lancedb) within LangChain.
-It is broken into two parts: installation and setup, and then references to specific LanceDB wrappers.
-
-## Installation and Setup
-
-- Install the Python SDK with `pip install lancedb`
-
-## Wrappers
-
-### VectorStore
-
-There exists a wrapper around LanceDB databases, allowing you to use it as a vectorstore,
-whether for semantic search or example selection.
-
-To import this vectorstore:
-
-```python
-from langchain.vectorstores import LanceDB
-```
-
-For a more detailed walkthrough of the LanceDB wrapper, see [this notebook](../modules/indexes/vectorstores/examples/lancedb.ipynb)

docs/ecosystem/metal.md

@@ -1,26 +0,0 @@
-# Metal
-
-This page covers how to use [Metal](https://getmetal.io) within LangChain.
-
-## What is Metal?
-
-Metal is a  managed retrieval & memory platform built for production. Easily index your data into `Metal` and run semantic search and retrieval on it.
-
-![Metal](../_static/MetalDash.png)
-
-## Quick start
-
-Get started by [creating a Metal account](https://app.getmetal.io/signup).
-
-Then, you can easily take advantage of the `MetalRetriever` class to start retrieving your data for semantic search, prompting context, etc. This class takes a `Metal` instance and a dictionary of parameters to pass to the Metal API.
-
-```python
-from langchain.retrievers import MetalRetriever
-from metal_sdk.metal import Metal
-
-
-metal = Metal("API_KEY", "CLIENT_ID", "INDEX_ID");
-retriever = MetalRetriever(metal, params={"limit": 2})
-
-docs = retriever.get_relevant_documents("search term")
-```

docs/ecosystem/myscale.md

@@ -1,65 +0,0 @@
-# MyScale
-
-This page covers how to use MyScale vector database within LangChain.
-It is broken into two parts: installation and setup, and then references to specific MyScale wrappers.
-
-With MyScale, you can manage both structured and unstructured (vectorized) data, and perform joint queries and analytics on both types of data using SQL. Plus, MyScale's cloud-native OLAP architecture, built on top of ClickHouse, enables lightning-fast data processing even on massive datasets.
-
-## Introduction
-
-[Overview to MyScale and High performance vector search](https://docs.myscale.com/en/overview/)
-
-You can now register on our SaaS and [start a cluster now!](https://docs.myscale.com/en/quickstart/)
-
-If you are also interested in how we managed to integrate SQL and vector, please refer to [this document](https://docs.myscale.com/en/vector-reference/) for further syntax reference.
-
-We also deliver with live demo on huggingface! Please checkout our [huggingface space](https://huggingface.co/myscale)! They search millions of vector within a blink!
-
-## Installation and Setup
-- Install the Python SDK with `pip install clickhouse-connect`
-
-### Setting up envrionments
-
-There are two ways to set up parameters for myscale index.
-
-1. Environment Variables
-
-    Before you run the app, please set the environment variable with `export`:
-    `export MYSCALE_URL='<your-endpoints-url>' MYSCALE_PORT=<your-endpoints-port> MYSCALE_USERNAME=<your-username> MYSCALE_PASSWORD=<your-password> ...`
-
-    You can easily find your account, password and other info on our SaaS. For details please refer to [this document](https://docs.myscale.com/en/cluster-management/)
-    Every attributes under `MyScaleSettings` can be set with prefix `MYSCALE_` and is case insensitive.
-
-2. Create `MyScaleSettings` object with parameters
-
-
-    ```python
-    from langchain.vectorstores import MyScale, MyScaleSettings
-    config = MyScaleSetting(host="<your-backend-url>", port=8443, ...)
-    index = MyScale(embedding_function, config)
-    index.add_documents(...)
-    ```
-  
-## Wrappers
-supported functions:
-- `add_texts`
-- `add_documents`
-- `from_texts`
-- `from_documents`
-- `similarity_search`
-- `asimilarity_search`
-- `similarity_search_by_vector`
-- `asimilarity_search_by_vector`
-- `similarity_search_with_relevance_scores`
-
-### VectorStore
-
-There exists a wrapper around MyScale database, allowing you to use it as a vectorstore,
-whether for semantic search or similar example retrieval.
-
-To import this vectorstore:
-```python
-from langchain.vectorstores import MyScale
-```
-
-For a more detailed walkthrough of the MyScale wrapper, see [this notebook](../modules/indexes/vectorstores/examples/myscale.ipynb)

docs/ecosystem/pipelineai.md

@@ -1,19 +0,0 @@
-# PipelineAI
-
-This page covers how to use the PipelineAI ecosystem within LangChain.
-It is broken into two parts: installation and setup, and then references to specific PipelineAI wrappers.
-
-## Installation and Setup
-
-- Install with `pip install pipeline-ai`
-- Get a Pipeline Cloud api key and set it as an environment variable (`PIPELINE_API_KEY`)
-
-## Wrappers
-
-### LLM
-
-There exists a PipelineAI LLM wrapper, which you can access with
-
-```python
-from langchain.llms import PipelineAI
-```

docs/ecosystem/predictionguard.md

@@ -1,56 +0,0 @@
-# Prediction Guard
-
-This page covers how to use the Prediction Guard ecosystem within LangChain.
-It is broken into two parts: installation and setup, and then references to specific Prediction Guard wrappers.
-
-## Installation and Setup
-- Install the Python SDK with `pip install predictionguard`
-- Get an Prediction Guard access token (as described [here](https://docs.predictionguard.com/)) and set it as an environment variable (`PREDICTIONGUARD_TOKEN`)
-
-## LLM Wrapper
-
-There exists a Prediction Guard LLM wrapper, which you can access with 
-```python
-from langchain.llms import PredictionGuard
-```
-
-You can provide the name of your Prediction Guard "proxy" as an argument when initializing the LLM:
-```python
-pgllm = PredictionGuard(name="your-text-gen-proxy")
-```
-
-Alternatively, you can use Prediction Guard's default proxy for SOTA LLMs:
-```python
-pgllm = PredictionGuard(name="default-text-gen")
-```
-
-You can also provide your access token directly as an argument:
-```python
-pgllm = PredictionGuard(name="default-text-gen", token="<your access token>")
-```
-
-## Example usage
-
-Basic usage of the LLM wrapper:
-```python
-from langchain.llms import PredictionGuard
-
-pgllm = PredictionGuard(name="default-text-gen")
-pgllm("Tell me a joke")
-```
-
-Basic LLM Chaining with the Prediction Guard wrapper:
-```python
-from langchain import PromptTemplate, LLMChain
-from langchain.llms import PredictionGuard
-
-template = """Question: {question}
-
-Answer: Let's think step by step."""
-prompt = PromptTemplate(template=template, input_variables=["question"])
-llm_chain = LLMChain(prompt=prompt, llm=PredictionGuard(name="default-text-gen"), verbose=True)
-
-question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"
-
-llm_chain.predict(question=question)
-```

docs/ecosystem/replicate.md

@@ -9,7 +9,7 @@ This page covers how to run models on Replicate within LangChain.
 
 Find a model on the [Replicate explore page](https://replicate.com/explore), and then paste in the model name and version in this format: `owner-name/model-name:version`
 
-For example, for this [dolly model](https://replicate.com/replicate/dolly-v2-12b), click on the API tab. The model name/version would be: `"replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5"`
+For example, for this [flan-t5 model](https://replicate.com/daanelson/flan-t5), click on the API tab. The model name/version would be: `daanelson/flan-t5:04e422a9b85baed86a4f24981d7f9953e20c5fd82f6103b74ebc431588e1cec8`
 
 Only the `model` param is required, but any other model parameters can also be passed in with the format `input={model_param: value, ...}`
 
@@ -24,7 +24,7 @@ Replicate(model="stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6
 From here, we can initialize our model:
 
 ```python
-llm = Replicate(model="replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5")
+llm = Replicate(model="daanelson/flan-t5:04e422a9b85baed86a4f24981d7f9953e20c5fd82f6103b74ebc431588e1cec8")
 ```
 
 And run it:
@@ -40,7 +40,8 @@ llm(prompt)
 We can call any Replicate model (not just LLMs) using this syntax. For example, we can call [Stable Diffusion](https://replicate.com/stability-ai/stable-diffusion):
 
 ```python
-text2image = Replicate(model="stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf", input={'image_dimensions':'512x512'})
+text2image = Replicate(model="stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf",
+                       input={'image_dimensions'='512x512'}
 
 image_output = text2image("A cat riding a motorcycle by Picasso")
 ```

docs/ecosystem/runhouse.md

@@ -15,7 +15,7 @@ custom LLMs, you can use the `SelfHostedPipeline` parent class.
 from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
 ```
 
-For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](../modules/models/llms/integrations/runhouse.ipynb)
+For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](../modules/models/llms/integrations/self_hosted_examples.ipynb)
 
 ## Self-hosted Embeddings
 There are several ways to use self-hosted embeddings with LangChain via Runhouse.

docs/ecosystem/wandb_tracking.ipynb

@@ -50,6 +50,7 @@
    "source": [
     "from datetime import datetime\n",
     "from langchain.callbacks import WandbCallbackHandler, StdOutCallbackHandler\n",
+    "from langchain.callbacks.base import CallbackManager\n",
     "from langchain.llms import OpenAI"
    ]
   },
@@ -195,8 +196,8 @@
     "    name=\"llm\",\n",
     "    tags=[\"test\"],\n",
     ")\n",
-    "callbacks = [StdOutCallbackHandler(), wandb_callback]\n",
-    "llm = OpenAI(temperature=0, callbacks=callbacks)"
+    "manager = CallbackManager([StdOutCallbackHandler(), wandb_callback])\n",
+    "llm = OpenAI(temperature=0, callback_manager=manager, verbose=True)"
    ]
   },
   {
@@ -483,7 +484,7 @@
     "Title: {title}\n",
     "Playwright: This is a synopsis for the above play:\"\"\"\n",
     "prompt_template = PromptTemplate(input_variables=[\"title\"], template=template)\n",
-    "synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)\n",
+    "synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callback_manager=manager)\n",
     "\n",
     "test_prompts = [\n",
     "    {\n",
@@ -576,15 +577,16 @@
    ],
    "source": [
     "# SCENARIO 3 - Agent with Tools\n",
-    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
+    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callback_manager=manager)\n",
     "agent = initialize_agent(\n",
     "    tools,\n",
     "    llm,\n",
     "    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
+    "    callback_manager=manager,\n",
+    "    verbose=True,\n",
     ")\n",
     "agent.run(\n",
-    "    \"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?\",\n",
-    "    callbacks=callbacks,\n",
+    "    \"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?\"\n",
     ")\n",
     "wandb_callback.flush_tracker(agent, reset=False, finish=True)"
    ]

docs/ecosystem/weaviate.md

@@ -30,4 +30,4 @@ To import this vectorstore:
 from langchain.vectorstores import Weaviate
 ```
 
-For a more detailed walkthrough of the Weaviate wrapper, see [this notebook](../modules/indexes/vectorstores/examples/weaviate.ipynb)
+For a more detailed walkthrough of the Weaviate wrapper, see [this notebook](../modules/indexes/vectorstores/getting_started.ipynb)

docs/ecosystem/yeagerai.md

@@ -1,43 +0,0 @@
-# Yeager.ai
-
-This page covers how to use [Yeager.ai](https://yeager.ai) to generate LangChain tools and agents.
-
-## What is Yeager.ai?
-Yeager.ai is an ecosystem designed to simplify the process of creating AI agents and tools. 
-
-It features yAgents, a No-code LangChain Agent Builder, which enables users to build, test, and deploy AI solutions with ease. Leveraging the LangChain framework, yAgents allows seamless integration with various language models and resources, making it suitable for developers, researchers, and AI enthusiasts across diverse applications.
-
-## yAgents
-Low code generative agent designed to help you build, prototype, and deploy Langchain tools with ease. 
-
-### How to use?
-```
-pip install yeagerai-agent
-yeagerai-agent
-```
-Go to http://127.0.0.1:7860
-
-This will install the necessary dependencies and set up yAgents on your system. After the first run, yAgents will create a .env file where you can input your OpenAI API key. You can do the same directly from the Gradio interface under the tab "Settings".
-
-`OPENAI_API_KEY=<your_openai_api_key_here>`
-
-We recommend using GPT-4,. However, the tool can also work with GPT-3 if the problem is broken down sufficiently.
-
-### Creating and Executing Tools with yAgents
-yAgents makes it easy to create and execute AI-powered tools. Here's a brief overview of the process:
-1. Create a tool: To create a tool, provide a natural language prompt to yAgents. The prompt should clearly describe the tool's purpose and functionality. For example:
-`create a tool that returns the n-th prime number`
-
-2. Load the tool into the toolkit: To load a tool into yAgents, simply provide a command to yAgents that says so. For example:
-`load the tool that you just created it into your toolkit`
-
-3. Execute the tool: To run a tool or agent, simply provide a command to yAgents that includes the name of the tool and any required parameters. For example:
-`generate the 50th prime number`
-
-You can see a video of how it works [here](https://www.youtube.com/watch?v=KA5hCM3RaWE).
-
-As you become more familiar with yAgents, you can create more advanced tools and agents to automate your work and enhance your productivity.
-
-For more information, see [yAgents' Github](https://github.com/yeagerai/yeagerai-agent) or our [docs](https://yeagerai.gitbook.io/docs/general/welcome-to-yeager.ai) 
-
-

docs/gallery.rst

@@ -1,5 +1,5 @@
 LangChain Gallery
-=================
+=============
 
 Lots of people have built some pretty awesome stuff with LangChain.
 This is a collection of our favorites.
@@ -223,7 +223,7 @@ Open Source
     Answer questions about the documentation of any project    
 
 Misc. Colab Notebooks
-~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~
 
 .. panels::
     :body: text-center
@@ -280,17 +280,6 @@ Proprietary
 
     ---
 
-    .. link-button:: https://anysummary.app
-        :type: url
-        :text: Summarize any file with AI
-        :classes: stretched-link btn-lg
-
-    +++
-
-    Summarize not only long docs, interview audio or video files quickly, but also entire websites and YouTube videos. Share or download your generated summaries to collaborate with others, or revisit them at any time! Bonus: `@anysummary <https://twitter.com/anysummary>`_ on Twitter will also summarize any thread it is tagged in.
-    
-    ---
-
     .. link-button:: https://twitter.com/dory111111/status/1608406234646052870?s=20&t=XYlrbKM0ornJsrtGa0br-g
         :type: url
         :text: AI Assisted SQL Query Generator

docs/getting_started/getting_started.md

@@ -46,7 +46,7 @@ LangChain provides many modules that can be used to build language model applica
 
 
 
-## LLMs: Get predictions from a language model
+`````{dropdown} LLMs: Get predictions from a language model
 
 The most basic building block of LangChain is calling an LLM on some input.
 Let's walk through a simple example of how to do this. 
@@ -77,9 +77,10 @@ Feetful of Fun
 ```
 
 For more details on how to use LLMs within LangChain, see the [LLM getting started guide](../modules/models/llms/getting_started.ipynb).
+`````
 
 
-## Prompt Templates: Manage prompts for LLMs
+`````{dropdown} Prompt Templates: Manage prompts for LLMs
 
 Calling an LLM is a great first step, but it's just the beginning.
 Normally when you use an LLM in an application, you are not sending user input directly to the LLM.
@@ -114,10 +115,11 @@ What is a good name for a company that makes colorful socks?
 
 [For more details, check out the getting started guide for prompts.](../modules/prompts/chat_prompt_template.ipynb)
 
+`````
 
 
 
-## Chains: Combine LLMs and prompts in multi-step workflows
+`````{dropdown} Chains: Combine LLMs and prompts in multi-step workflows
 
 Up until now, we've worked with the PromptTemplate and LLM primitives by themselves. But of course, a real application is not just one primitive, but rather a combination of them.
 
@@ -157,7 +159,10 @@ This is one of the simpler types of chains, but understanding how it works will
 
 [For more details, check out the getting started guide for chains.](../modules/chains/getting_started.ipynb)
 
-## Agents: Dynamically Call Chains Based on User Input
+`````
+
+
+`````{dropdown} Agents: Dynamically Call Chains Based on User Input
 
 So far the chains we've looked at run in a predetermined order.
 
@@ -229,8 +234,10 @@ Final Answer: The high temperature in SF yesterday in Fahrenheit raised to the .
 ```
 
 
+`````
 
-## Memory: Add State to Chains and Agents
+
+`````{dropdown} Memory: Add State to Chains and Agents
 
 So far, all the chains and agents we've gone through have been stateless. But often, you may want a chain or agent to have some concept of "memory" so that it may remember information about its previous interactions. The clearest and simple example of this is when designing a chatbot - you want it to remember previous messages so it can use context from that to have a better conversation. This would be a type of "short-term memory". On the more complex side, you could imagine a chain/agent remembering key pieces of information over time - this would be a form of "long-term memory". For more concrete ideas on the latter, see this [awesome paper](https://memprompt.com/).
 
@@ -244,8 +251,7 @@ from langchain import OpenAI, ConversationChain
 llm = OpenAI(temperature=0)
 conversation = ConversationChain(llm=llm, verbose=True)
 
-output = conversation.predict(input="Hi there!")
-print(output)
+conversation.predict(input="Hi there!")
 ```
 
 ```pycon
@@ -263,8 +269,7 @@ AI:
 ```
 
 ```python
-output = conversation.predict(input="I'm doing well! Just having a conversation with an AI.")
-print(output)
+conversation.predict(input="I'm doing well! Just having a conversation with an AI.")
 ```
 
 ```pycon
@@ -282,6 +287,7 @@ AI:
 > Finished chain.
 " That's great! What would you like to talk about?"
 ```
+`````
 
 ## Building a Language Model Application: Chat Models
 
@@ -289,8 +295,8 @@ Similarly, you can use chat models instead of LLMs. Chat models are a variation
 
 Chat model APIs are fairly new, so we are still figuring out the correct abstractions.
 
-## Get Message Completions from a Chat Model
 
+`````{dropdown} Get Message Completions from a Chat Model
 You can get chat completions by passing one or more messages to the chat model. The response will be a message. The types of messages currently supported in LangChain are `AIMessage`, `HumanMessage`, `SystemMessage`, and `ChatMessage` -- `ChatMessage` takes in an arbitrary role parameter. Most of the time, you'll just be dealing with `HumanMessage`, `AIMessage`, and `SystemMessage`.
 
 ```python
@@ -344,9 +350,9 @@ You can recover things like token usage from this LLMResult:
 result.llm_output['token_usage']
 # -> {'prompt_tokens': 71, 'completion_tokens': 18, 'total_tokens': 89}
 ```
+`````
 
-
-## Chat Prompt Templates
+`````{dropdown} Chat Prompt Templates
 Similar to LLMs, you can make use of templating by using a `MessagePromptTemplate`. You can build a `ChatPromptTemplate` from one or more `MessagePromptTemplate`s. You can use `ChatPromptTemplate`'s `format_prompt` -- this returns a `PromptValue`, which you can convert to a string or `Message` object, depending on whether you want to use the formatted value as input to an llm or chat model.
 
 For convience, there is a `from_template` method exposed on the template. If you were to use this template, this is what it would look like:
@@ -372,8 +378,9 @@ chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_mes
 chat(chat_prompt.format_prompt(input_language="English", output_language="French", text="I love programming.").to_messages())
 # -> AIMessage(content="J'aime programmer.", additional_kwargs={})
 ```
+`````
 
-## Chains with Chat Models
+`````{dropdown} Chains with Chat Models
 The `LLMChain` discussed in the above section can be used with chat models as well:
 
 ```python
@@ -397,8 +404,9 @@ chain = LLMChain(llm=chat, prompt=chat_prompt)
 chain.run(input_language="English", output_language="French", text="I love programming.")
 # -> "J'aime programmer."
 ```
+`````
 
-## Agents with Chat Models
+`````{dropdown} Agents with Chat Models
 Agents can also be used with chat models, you can initialize one using `AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION` as the agent type.
 
 ```python
@@ -457,7 +465,9 @@ Final Answer: 2.169459462491557
 > Finished chain.
 '2.169459462491557'
 ```
-## Memory: Add State to Chains and Agents
+`````
+
+`````{dropdown} Memory: Add State to Chains and Agents
 You can use Memory with chains and agents initialized with chat models. The main difference between this and Memory for LLMs is that rather than trying to condense all previous messages into a string, we can keep them as their own unique memory object.
 
 ```python
@@ -491,4 +501,4 @@ conversation.predict(input="I'm doing well! Just having a conversation with an A
 conversation.predict(input="Tell me about yourself.")
 # -> "Sure! I am an AI language model created by OpenAI. I was trained on a large dataset of text from the internet, which allows me to understand and generate human-like language. I can answer questions, provide information, and even have conversations like this one. Is there anything else you'd like to know about me?"
 ```
-
+`````

docs/index.rst

@@ -63,10 +63,6 @@ Use Cases
 
 The above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports.
 
-- `Autonomous Agents <./use_cases/autonomous_agents.html>`_: Autonomous agents are long running agents that take many steps in an attempt to accomplish an objective. Examples include AutoGPT and BabyAGI.
-
-- `Agent Simulations <./use_cases/agent_simulations.html>`_: Putting agents in a sandbox and observing how they interact with each other or to events can be an interesting way to observe their long-term memory abilities.
-
 - `Personal Assistants <./use_cases/personal_assistants.html>`_: The main LangChain use case. Personal assistants need to take actions, remember interactions, and have knowledge about your data.
 
 - `Question Answering <./use_cases/question_answering.html>`_: The second big LangChain use case. Answering questions over specific documents, only utilizing the information in those documents to construct an answer.
@@ -93,8 +89,6 @@ The above modules can be used in a variety of ways. LangChain also provides guid
    :hidden:
 
    ./use_cases/personal_assistants.md
-   ./use_cases/autonomous_agents.md
-   ./use_cases/agent_simulations.md
    ./use_cases/question_answering.md
    ./use_cases/chatbots.md
    ./use_cases/tabular.rst
@@ -159,8 +153,6 @@ Additional collection of resources we think may be useful as you develop your ap
 
 - `Discord <https://discord.gg/6adMQxSpJS>`_: Join us on our Discord to discuss all things LangChain!
 
-- `YouTube <./youtube.html>`_: A collection of the LangChain tutorials and videos.
-
 - `Production Support <https://forms.gle/57d8AmXBYp8PP8tZA>`_: As you move your LangChains into production, we'd love to offer more comprehensive support. Please fill out this form and we'll set up a dedicated support Slack channel.
 
 
@@ -177,5 +169,4 @@ Additional collection of resources we think may be useful as you develop your ap
    ./tracing.md
    ./use_cases/model_laboratory.ipynb
    Discord <https://discord.gg/6adMQxSpJS>
-   ./youtube.md
    Production Support <https://forms.gle/57d8AmXBYp8PP8tZA>

docs/modules/agents/agent_executors/examples/async_agent.ipynb

@@ -42,6 +42,7 @@
     "from langchain.agents import AgentType\n",
     "from langchain.llms import OpenAI\n",
     "from langchain.callbacks.stdout import StdOutCallbackHandler\n",
+    "from langchain.callbacks.base import CallbackManager\n",
     "from langchain.callbacks.tracers import LangChainTracer\n",
     "from aiohttp import ClientSession\n",
     "\n",
@@ -306,14 +307,14 @@
     "    # To make async requests in Tools more efficient, you can pass in your own aiohttp.ClientSession, \n",
     "    # but you must manually close the client session at the end of your program/event loop\n",
     "    aiosession = ClientSession()\n",
-    "    callbacks = [StdOutCallbackHandler()]\n",
     "    for _ in questions:\n",
-    "        llm = OpenAI(temperature=0)\n",
-    "        async_tools = load_tools([\"llm-math\", \"serpapi\"], llm=llm, aiosession=aiosession)\n",
+    "        manager = CallbackManager([StdOutCallbackHandler()])\n",
+    "        llm = OpenAI(temperature=0, callback_manager=manager)\n",
+    "        async_tools = load_tools([\"llm-math\", \"serpapi\"], llm=llm, aiosession=aiosession, callback_manager=manager)\n",
     "        agents.append(\n",
-    "            initialize_agent(async_tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)\n",
+    "            initialize_agent(async_tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True, callback_manager=manager)\n",
     "        )\n",
-    "    tasks = [async_agent.arun(q, callbacks=callbacks) for async_agent, q in zip(agents, questions)]\n",
+    "    tasks = [async_agent.arun(q) for async_agent, q in zip(agents, questions)]\n",
     "    await asyncio.gather(*tasks)\n",
     "    await aiosession.close()\n",
     "\n",
@@ -375,14 +376,14 @@
     "aiosession = ClientSession()\n",
     "tracer = LangChainTracer()\n",
     "tracer.load_default_session()\n",
-    "callbacks = [StdOutCallbackHandler(), tracer]\n",
+    "manager = CallbackManager([StdOutCallbackHandler(), tracer])\n",
     "\n",
     "# Pass the manager into the llm if you want llm calls traced.\n",
-    "llm = OpenAI(temperature=0)\n",
+    "llm = OpenAI(temperature=0, callback_manager=manager)\n",
     "\n",
     "async_tools = load_tools([\"llm-math\", \"serpapi\"], llm=llm, aiosession=aiosession)\n",
-    "async_agent = initialize_agent(async_tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)\n",
-    "await async_agent.arun(questions[0], callbacks=callbacks)\n",
+    "async_agent = initialize_agent(async_tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True, callback_manager=manager)\n",
+    "await async_agent.arun(questions[0])\n",
     "await aiosession.close()"
    ]
   }

docs/modules/agents/agents/custom_agent.ipynb

@@ -49,7 +49,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "id": "a33e2f7e",
    "metadata": {},
    "outputs": [],
@@ -77,7 +77,7 @@
     "        Returns:\n",
     "            Action specifying what tool to use.\n",
     "        \"\"\"\n",
-    "        return AgentAction(tool=\"Search\", tool_input=kwargs[\"input\"], log=\"\")\n",
+    "        return AgentAction(tool=\"Search\", tool_input=\"foo\", log=\"\")\n",
     "\n",
     "    async def aplan(\n",
     "        self, intermediate_steps: List[Tuple[AgentAction, str]], **kwargs: Any\n",
@@ -92,12 +92,12 @@
     "        Returns:\n",
     "            Action specifying what tool to use.\n",
     "        \"\"\"\n",
-    "        return AgentAction(tool=\"Search\", tool_input=kwargs[\"input\"], log=\"\")"
+    "        return AgentAction(tool=\"Search\", tool_input=\"foo\", log=\"\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "id": "655d72f6",
    "metadata": {},
    "outputs": [],
@@ -107,7 +107,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "id": "490604e9",
    "metadata": {},
    "outputs": [],
@@ -117,7 +117,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "id": "653b1617",
    "metadata": {},
    "outputs": [
@@ -128,7 +128,7 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m\u001b[0m\u001b[36;1m\u001b[1;3mThe current population of Canada is 38,669,152 as of Monday, April 24, 2023, based on Worldometer elaboration of the latest United Nations data.\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\u001b[0m\u001b[36;1m\u001b[1;3mFoo Fighters is an American rock band formed in Seattle in 1994. Foo Fighters was initially formed as a one-man project by former Nirvana drummer Dave Grohl. Following the success of the 1995 eponymous debut album, Grohl recruited a band consisting of Nate Mendel, William Goldsmith, and Pat Smear.\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -136,10 +136,10 @@
     {
      "data": {
       "text/plain": [
-       "'The current population of Canada is 38,669,152 as of Monday, April 24, 2023, based on Worldometer elaboration of the latest United Nations data.'"
+       "'Foo Fighters is an American rock band formed in Seattle in 1994. Foo Fighters was initially formed as a one-man project by former Nirvana drummer Dave Grohl. Following the success of the 1995 eponymous debut album, Grohl recruited a band consisting of Nate Mendel, William Goldsmith, and Pat Smear.'"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/modules/agents/agents/custom_agent_with_tool_retrieval.ipynb

@@ -315,7 +315,7 @@
     "                log=llm_output,\n",
     "            )\n",
     "        # Parse out the action and action input\n",
-    "        regex = r\"Action\\s*\\d*\\s*:(.*?)\\nAction\\s*\\d*\\s*Input\\s*\\d*\\s*:[\\s]*(.*)\"\n",
+    "        regex = r\"Action: (.*?)[\\n]*Action Input:[\\s]*(.*)\"\n",
     "        match = re.search(regex, llm_output, re.DOTALL)\n",
     "        if not match:\n",
     "            raise ValueError(f\"Could not parse LLM output: `{llm_output}`\")\n",
@@ -373,7 +373,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "tools = get_tools(\"whats the weather?\")\n",
     "tool_names = [tool.name for tool in tools]\n",
     "agent = LLMSingleActionAgent(\n",
     "    llm_chain=llm_chain, \n",

docs/modules/agents/agents/custom_llm_agent.ipynb

@@ -42,7 +42,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 11,
    "id": "9af9734e",
    "metadata": {},
    "outputs": [],
@@ -67,7 +67,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 28,
    "id": "becda2a1",
    "metadata": {},
    "outputs": [],
@@ -99,7 +99,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 7,
    "id": "339b1bb8",
    "metadata": {},
    "outputs": [],
@@ -128,7 +128,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 22,
    "id": "fd969d31",
    "metadata": {},
    "outputs": [],
@@ -159,7 +159,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 23,
    "id": "798ef9fb",
    "metadata": {},
    "outputs": [],
@@ -187,7 +187,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 13,
    "id": "7c6fe0d3",
    "metadata": {},
    "outputs": [],
@@ -204,7 +204,7 @@
     "                log=llm_output,\n",
     "            )\n",
     "        # Parse out the action and action input\n",
-    "        regex = r\"Action\\s*\\d*\\s*:(.*?)\\nAction\\s*\\d*\\s*Input\\s*\\d*\\s*:[\\s]*(.*)\"\n",
+    "        regex = r\"Action: (.*?)[\\n]*Action Input:[\\s]*(.*)\"\n",
     "        match = re.search(regex, llm_output, re.DOTALL)\n",
     "        if not match:\n",
     "            raise ValueError(f\"Could not parse LLM output: `{llm_output}`\")\n",
@@ -216,7 +216,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 14,
    "id": "d278706a",
    "metadata": {},
    "outputs": [],
@@ -236,7 +236,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 16,
    "id": "f9d4c374",
    "metadata": {},
    "outputs": [],
@@ -268,7 +268,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 24,
    "id": "9b1cc2a2",
    "metadata": {},
    "outputs": [],
@@ -279,7 +279,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 25,
    "id": "e4f5092f",
    "metadata": {},
    "outputs": [],
@@ -305,7 +305,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 26,
    "id": "490604e9",
    "metadata": {},
    "outputs": [],
@@ -315,7 +315,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 27,
    "id": "653b1617",
    "metadata": {},
    "outputs": [
@@ -326,12 +326,11 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mThought: I need to find out the population of Canada in 2023\n",
-      "Action: Search\n",
+      "\u001b[32;1m\u001b[1;3mAction: Search\n",
       "Action Input: Population of Canada in 2023\u001b[0m\n",
       "\n",
-      "Observation:\u001b[36;1m\u001b[1;3mThe current population of Canada is 38,658,314 as of Wednesday, April 12, 2023, based on Worldometer elaboration of the latest United Nations data.\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: Arrr, there be 38,658,314 people livin' in Canada as of 2023!\u001b[0m\n",
+      "Observation:\u001b[36;1m\u001b[1;3m38,648,380\u001b[0m\u001b[32;1m\u001b[1;3m That's a lot of people!\n",
+      "Final Answer: Arrr, there be 38,648,380 people livin' in Canada come 2023!\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -339,10 +338,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Arrr, there be 38,658,314 people livin' in Canada as of 2023!\""
+       "\"Arrr, there be 38,648,380 people livin' in Canada come 2023!\""
       ]
      },
-     "execution_count": 12,
+     "execution_count": 27,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -351,203 +350,10 @@
     "agent_executor.run(\"How many people live in canada as of 2023?\")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "d5b4a078",
-   "metadata": {},
-   "source": [
-    "## Adding Memory\n",
-    "\n",
-    "If you want to add memory to the agent, you'll need to:\n",
-    "\n",
-    "1. Add a place in the custom prompt for the chat_history\n",
-    "2. Add a memory object to the agent executor."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "94fffda1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Set up the base template\n",
-    "template_with_history = \"\"\"Answer the following questions as best you can, but speaking as a pirate might speak. You have access to the following tools:\n",
-    "\n",
-    "{tools}\n",
-    "\n",
-    "Use the following format:\n",
-    "\n",
-    "Question: the input question you must answer\n",
-    "Thought: you should always think about what to do\n",
-    "Action: the action to take, should be one of [{tool_names}]\n",
-    "Action Input: the input to the action\n",
-    "Observation: the result of the action\n",
-    "... (this Thought/Action/Action Input/Observation can repeat N times)\n",
-    "Thought: I now know the final answer\n",
-    "Final Answer: the final answer to the original input question\n",
-    "\n",
-    "Begin! Remember to speak as a pirate when giving your final answer. Use lots of \"Arg\"s\n",
-    "\n",
-    "Previous conversation history:\n",
-    "{history}\n",
-    "\n",
-    "New question: {input}\n",
-    "{agent_scratchpad}\"\"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "f58488d7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt_with_history = CustomPromptTemplate(\n",
-    "    template=template_with_history,\n",
-    "    tools=tools,\n",
-    "    # This omits the `agent_scratchpad`, `tools`, and `tool_names` variables because those are generated dynamically\n",
-    "    # This includes the `intermediate_steps` variable because that is needed\n",
-    "    input_variables=[\"input\", \"intermediate_steps\", \"history\"]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "d28d4b5a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 32,
-   "id": "3e37b32a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tool_names = [tool.name for tool in tools]\n",
-    "agent = LLMSingleActionAgent(\n",
-    "    llm_chain=llm_chain, \n",
-    "    output_parser=output_parser,\n",
-    "    stop=[\"\\nObservation:\"], \n",
-    "    allowed_tools=tool_names\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 33,
-   "id": "97ea1bce",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.memory import ConversationBufferWindowMemory"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "b5ad69ce",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "memory=ConversationBufferWindowMemory(k=2)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 43,
-   "id": "b7b5c9b1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 44,
-   "id": "5ec4c39b",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mThought: I need to find out the population of Canada in 2023\n",
-      "Action: Search\n",
-      "Action Input: Population of Canada in 2023\u001b[0m\n",
-      "\n",
-      "Observation:\u001b[36;1m\u001b[1;3mThe current population of Canada is 38,658,314 as of Wednesday, April 12, 2023, based on Worldometer elaboration of the latest United Nations data.\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: Arrr, there be 38,658,314 people livin' in Canada as of 2023!\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "\"Arrr, there be 38,658,314 people livin' in Canada as of 2023!\""
-      ]
-     },
-     "execution_count": 44,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent_executor.run(\"How many people live in canada as of 2023?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 45,
-   "id": "b2ba45bb",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mThought: I need to find out how many people live in Mexico.\n",
-      "Action: Search\n",
-      "Action Input: How many people live in Mexico as of 2023?\u001b[0m\n",
-      "\n",
-      "Observation:\u001b[36;1m\u001b[1;3mThe current population of Mexico is 132,679,922 as of Tuesday, April 11, 2023, based on Worldometer elaboration of the latest United Nations data. Mexico 2020 ...\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
-      "Final Answer: Arrr, there be 132,679,922 people livin' in Mexico as of 2023!\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "\"Arrr, there be 132,679,922 people livin' in Mexico as of 2023!\""
-      ]
-     },
-     "execution_count": 45,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent_executor.run(\"how about in mexico?\")"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "bd820a7a",
+   "id": "adefb4c2",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/agents/agents/custom_llm_chat_agent.ipynb

@@ -206,7 +206,7 @@
     "                log=llm_output,\n",
     "            )\n",
     "        # Parse out the action and action input\n",
-    "        regex = r\"Action\\s*\\d*\\s*:(.*?)\\nAction\\s*\\d*\\s*Input\\s*\\d*\\s*:[\\s]*(.*)\"\n",
+    "        regex = r\"Action: (.*?)[\\n]*Action Input:[\\s]*(.*)\"\n",
     "        match = re.search(regex, llm_output, re.DOTALL)\n",
     "        if not match:\n",
     "            raise ValueError(f\"Could not parse LLM output: `{llm_output}`\")\n",

docs/modules/agents/agents/custom_mrkl_agent.ipynb

@@ -20,14 +20,13 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "6064f080",
    "metadata": {},
    "source": [
     "### Custom LLMChain\n",
     "\n",
-    "The first way to create a custom agent is to use an existing Agent class, but use a custom LLMChain. This is the simplest way to create a custom Agent. It is highly recommended that you work with the `ZeroShotAgent`, as at the moment that is by far the most generalizable one. \n",
+    "The first way to create a custom agent is to use an existing Agent class, but use a custom LLMChain. This is the simplest way to create a custom Agent. It is highly reccomended that you work with the `ZeroShotAgent`, as at the moment that is by far the most generalizable one. \n",
     "\n",
     "Most of the work in creating the custom LLMChain comes down to the prompt. Because we are using an existing agent class to parse the output, it is very important that the prompt say to produce text in that format. Additionally, we currently require an `agent_scratchpad` input variable to put notes on previous actions and observations. This should almost always be the final part of the prompt. However, besides those instructions, you can customize the prompt as you wish.\n",
     "\n",
@@ -43,7 +42,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 23,
    "id": "9af9734e",
    "metadata": {},
    "outputs": [],
@@ -54,7 +53,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 24,
    "id": "becda2a1",
    "metadata": {},
    "outputs": [],
@@ -71,7 +70,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 25,
    "id": "339b1bb8",
    "metadata": {},
    "outputs": [],
@@ -100,7 +99,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 26,
    "id": "e21d2098",
    "metadata": {},
    "outputs": [
@@ -146,7 +145,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 27,
    "id": "9b1cc2a2",
    "metadata": {},
    "outputs": [],
@@ -156,7 +155,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 28,
    "id": "e4f5092f",
    "metadata": {},
    "outputs": [],
@@ -167,7 +166,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 29,
    "id": "490604e9",
    "metadata": {},
    "outputs": [],
@@ -177,7 +176,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 31,
    "id": "653b1617",
    "metadata": {},
    "outputs": [
@@ -191,9 +190,9 @@
       "\u001b[32;1m\u001b[1;3mThought: I need to find out the population of Canada\n",
       "Action: Search\n",
       "Action Input: Population of Canada 2023\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mThe current population of Canada is 38,661,927 as of Sunday, April 16, 2023, based on Worldometer elaboration of the latest United Nations data.\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mThe current population of Canada is 38,610,447 as of Saturday, February 18, 2023, based on Worldometer elaboration of the latest United Nations data. Canada 2020 population is estimated at 37,742,154 people at mid year according to UN data.\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: Arrr, Canada be havin' 38,661,927 people livin' there as of 2023!\u001b[0m\n",
+      "Final Answer: Arrr, Canada be havin' 38,610,447 scallywags livin' there as of 2023!\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -201,10 +200,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Arrr, Canada be havin' 38,661,927 people livin' there as of 2023!\""
+       "\"Arrr, Canada be havin' 38,610,447 scallywags livin' there as of 2023!\""
       ]
      },
-     "execution_count": 8,
+     "execution_count": 31,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -224,7 +223,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 32,
    "id": "43dbfa2f",
    "metadata": {},
    "outputs": [],
@@ -245,7 +244,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 33,
    "id": "0f087313",
    "metadata": {},
    "outputs": [],
@@ -255,7 +254,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 34,
    "id": "92c75a10",
    "metadata": {},
    "outputs": [],
@@ -265,7 +264,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 35,
    "id": "ac5b83bf",
    "metadata": {},
    "outputs": [],
@@ -275,7 +274,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 36,
    "id": "c960e4ff",
    "metadata": {},
    "outputs": [
@@ -286,16 +285,12 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mThought: I should look for recent population estimates.\n",
+      "\u001b[32;1m\u001b[1;3mThought: I need to find out the population of Canada in 2023.\n",
       "Action: Search\n",
-      "Action Input: Canada population 2023\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m39,566,248\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I should double check this number.\n",
-      "Action: Search\n",
-      "Action Input: Canada population estimates 2023\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mCanada's population was estimated at 39,566,248 on January 1, 2023, after a record population growth of 1,050,110 people from January 1, 2022, to January 1, 2023.\u001b[0m\n",
+      "Action Input: Population of Canada in 2023\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mThe current population of Canada is 38,610,447 as of Saturday, February 18, 2023, based on Worldometer elaboration of the latest United Nations data. Canada 2020 population is estimated at 37,742,154 people at mid year according to UN data.\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
-      "Final Answer: La popolazione del Canada è stata stimata a 39.566.248 il 1° gennaio 2023, dopo un record di crescita demografica di 1.050.110 persone dal 1° gennaio 2022 al 1° gennaio 2023.\u001b[0m\n",
+      "Final Answer: La popolazione del Canada nel 2023 è stimata in 38.610.447 persone.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -303,10 +298,10 @@
     {
      "data": {
       "text/plain": [
-       "'La popolazione del Canada è stata stimata a 39.566.248 il 1° gennaio 2023, dopo un record di crescita demografica di 1.050.110 persone dal 1° gennaio 2022 al 1° gennaio 2023.'"
+       "'La popolazione del Canada nel 2023 è stimata in 38.610.447 persone.'"
       ]
      },
-     "execution_count": 13,
+     "execution_count": 36,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/modules/agents/agents/custom_multi_action_agent.ipynb

@@ -31,7 +31,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 21,
    "id": "d7c4ebdc",
    "metadata": {},
    "outputs": [],
@@ -43,7 +43,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 22,
    "id": "becda2a1",
    "metadata": {},
    "outputs": [],
@@ -66,7 +66,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 23,
    "id": "a33e2f7e",
    "metadata": {},
    "outputs": [],
@@ -96,8 +96,8 @@
     "        \"\"\"\n",
     "        if len(intermediate_steps) == 0:\n",
     "            return [\n",
-    "                AgentAction(tool=\"Search\", tool_input=kwargs[\"input\"], log=\"\"),\n",
-    "                AgentAction(tool=\"RandomWord\", tool_input=kwargs[\"input\"], log=\"\"),\n",
+    "                AgentAction(tool=\"Search\", tool_input=\"foo\", log=\"\"),\n",
+    "                AgentAction(tool=\"RandomWord\", tool_input=\"foo\", log=\"\"),\n",
     "            ]\n",
     "        else:\n",
     "            return AgentFinish(return_values={\"output\": \"bar\"}, log=\"\")\n",
@@ -117,8 +117,8 @@
     "        \"\"\"\n",
     "        if len(intermediate_steps) == 0:\n",
     "            return [\n",
-    "                AgentAction(tool=\"Search\", tool_input=kwargs[\"input\"], log=\"\"),\n",
-    "                AgentAction(tool=\"RandomWord\", tool_input=kwargs[\"input\"], log=\"\"),\n",
+    "                AgentAction(tool=\"Search\", tool_input=\"foo\", log=\"\"),\n",
+    "                AgentAction(tool=\"RandomWord\", tool_input=\"foo\", log=\"\"),\n",
     "            ]\n",
     "        else:\n",
     "            return AgentFinish(return_values={\"output\": \"bar\"}, log=\"\")"
@@ -126,7 +126,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 24,
    "id": "655d72f6",
    "metadata": {},
    "outputs": [],
@@ -136,7 +136,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 25,
    "id": "490604e9",
    "metadata": {},
    "outputs": [],
@@ -146,7 +146,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 26,
    "id": "653b1617",
    "metadata": {},
    "outputs": [
@@ -157,7 +157,7 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m\u001b[0m\u001b[36;1m\u001b[1;3mThe current population of Canada is 38,669,152 as of Monday, April 24, 2023, based on Worldometer elaboration of the latest United Nations data.\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\u001b[0m\u001b[36;1m\u001b[1;3mFoo Fighters is an American rock band formed in Seattle in 1994. Foo Fighters was initially formed as a one-man project by former Nirvana drummer Dave Grohl. Following the success of the 1995 eponymous debut album, Grohl recruited a band consisting of Nate Mendel, William Goldsmith, and Pat Smear.\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
       "Now I'm doing this!\n",
       "\u001b[33;1m\u001b[1;3mfoo\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
       "\n",
@@ -170,7 +170,7 @@
        "'bar'"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 26,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/modules/agents/agents/examples/chat_conversation_agent.ipynb

@@ -28,15 +28,7 @@
    "execution_count": 2,
    "id": "f65308ab",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING:root:Failed to default session, using empty session: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /sessions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10a1767c0>: Failed to establish a new connection: [Errno 61] Connection refused'))\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from langchain.agents import Tool\n",
     "from langchain.memory import ConversationBufferMemory\n",
@@ -96,20 +88,7 @@
      "text": [
       "\n",
       "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING:root:Failed to persist run: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /chain-runs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x13fab40d0>: Failed to establish a new connection: [Errno 61] Connection refused'))\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m{\n",
       "    \"action\": \"Final Answer\",\n",
       "    \"action_input\": \"Hello Bob! How can I assist you today?\"\n",
@@ -145,20 +124,7 @@
      "text": [
       "\n",
       "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING:root:Failed to persist run: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /chain-runs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x13fab44f0>: Failed to establish a new connection: [Errno 61] Connection refused'))\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m{\n",
       "    \"action\": \"Final Answer\",\n",
       "    \"action_input\": \"Your name is Bob.\"\n",
@@ -201,24 +167,10 @@
       "    \"action\": \"Current Search\",\n",
       "    \"action_input\": \"Thai food dinner recipes\"\n",
       "}\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m59 easy Thai recipes for any night of the week · Marion Grasby's Thai spicy chilli and basil fried rice · Thai curry noodle soup · Marion Grasby's Thai Spicy ...\u001b[0m\n",
-      "Thought:"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING:root:Failed to persist run: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /chain-runs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x13fae8be0>: Failed to establish a new connection: [Errno 61] Connection refused'))\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[32;1m\u001b[1;3m{\n",
+      "Observation: \u001b[36;1m\u001b[1;3m59 easy Thai recipes for any night of the week · Marion Grasby's Thai spicy chilli and basil fried rice · Thai curry noodle soup · Marion Grasby's ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m{\n",
       "    \"action\": \"Final Answer\",\n",
-      "    \"action_input\": \"Here are some Thai food dinner recipes you can make this week: Thai spicy chilli and basil fried rice, Thai curry noodle soup, and Thai Spicy ... (59 recipes in total).\"\n",
+      "    \"action_input\": \"Here are some Thai food dinner recipes you can make this week: Thai spicy chilli and basil fried rice, Thai curry noodle soup, and many more. You can find 59 easy Thai recipes for any night of the week on Marion Grasby's website.\"\n",
       "}\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
@@ -227,7 +179,7 @@
     {
      "data": {
       "text/plain": [
-       "'Here are some Thai food dinner recipes you can make this week: Thai spicy chilli and basil fried rice, Thai curry noodle soup, and Thai Spicy ... (59 recipes in total).'"
+       "\"Here are some Thai food dinner recipes you can make this week: Thai spicy chilli and basil fried rice, Thai curry noodle soup, and many more. You can find 59 easy Thai recipes for any night of the week on Marion Grasby's website.\""
       ]
      },
      "execution_count": 8,
@@ -258,25 +210,11 @@
       "    \"action_input\": \"who won the world cup in 1978\"\n",
       "}\n",
       "```\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mArgentina national football team\u001b[0m\n",
-      "Thought:"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING:root:Failed to persist run: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /chain-runs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x13fae86d0>: Failed to establish a new connection: [Errno 61] Connection refused'))\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[32;1m\u001b[1;3m```json\n",
+      "Observation: \u001b[36;1m\u001b[1;3mThe Argentina national football team represents Argentina in men's international football and is administered by the Argentine Football Association, the governing body for football in Argentina. Nicknamed La Albiceleste, they are the reigning world champions, having won the most recent World Cup in 2022.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m```json\n",
       "{\n",
       "    \"action\": \"Final Answer\",\n",
-      "    \"action_input\": \"The last letter in your name is 'b', and the winner of the 1978 World Cup was the Argentina national football team.\"\n",
+      "    \"action_input\": \"The last letter in your name is 'b'. The Argentina national football team won the World Cup in 1978.\"\n",
       "}\n",
       "```\u001b[0m\n",
       "\n",
@@ -286,7 +224,7 @@
     {
      "data": {
       "text/plain": [
-       "\"The last letter in your name is 'b', and the winner of the 1978 World Cup was the Argentina national football team.\""
+       "\"The last letter in your name is 'b'. The Argentina national football team won the World Cup in 1978.\""
       ]
      },
      "execution_count": 9,
@@ -315,24 +253,10 @@
       "    \"action\": \"Current Search\",\n",
       "    \"action_input\": \"weather in pomfret\"\n",
       "}\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m10 Day Weather-Pomfret, CT ; Sun 16. 64° · 50°. 24% · NE 7 mph ; Mon 17. 58° · 45°. 70% · ESE 8 mph ; Tue 18. 57° · 37°. 8% · WSW 15 mph.\u001b[0m\n",
-      "Thought:"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "WARNING:root:Failed to persist run: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /chain-runs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x13fa9d7f0>: Failed to establish a new connection: [Errno 61] Connection refused'))\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[32;1m\u001b[1;3m{\n",
+      "Observation: \u001b[36;1m\u001b[1;3mMostly cloudy with gusty winds developing during the afternoon. A few flurries or snow showers possible. High near 40F. Winds NNW at 20 to 30 mph.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m{\n",
       "    \"action\": \"Final Answer\",\n",
-      "    \"action_input\": \"The weather in Pomfret, CT for the next 10 days is as follows: Sun 16. 64° · 50°. 24% · NE 7 mph ; Mon 17. 58° · 45°. 70% · ESE 8 mph ; Tue 18. 57° · 37°. 8% · WSW 15 mph.\"\n",
+      "    \"action_input\": \"The weather in Pomfret is mostly cloudy with gusty winds developing during the afternoon. A few flurries or snow showers are possible. High near 40F. Winds NNW at 20 to 30 mph.\"\n",
       "}\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
@@ -341,7 +265,7 @@
     {
      "data": {
       "text/plain": [
-       "'The weather in Pomfret, CT for the next 10 days is as follows: Sun 16. 64° · 50°. 24% · NE 7 mph ; Mon 17. 58° · 45°. 70% · ESE 8 mph ; Tue 18. 57° · 37°. 8% · WSW 15 mph.'"
+       "'The weather in Pomfret is mostly cloudy with gusty winds developing during the afternoon. A few flurries or snow showers are possible. High near 40F. Winds NNW at 20 to 30 mph.'"
       ]
      },
      "execution_count": 10,

docs/modules/agents/agents/examples/conversational_agent.ipynb

@@ -23,7 +23,7 @@
     "from langchain.agents import AgentType\n",
     "from langchain.memory import ConversationBufferMemory\n",
     "from langchain import OpenAI\n",
-    "from langchain.utilities import SerpAPIWrapper\n",
+    "from langchain.utilities import GoogleSearchAPIWrapper\n",
     "from langchain.agents import initialize_agent"
    ]
   },
@@ -34,7 +34,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "search = SerpAPIWrapper()\n",
+    "search = GoogleSearchAPIWrapper()\n",
     "tools = [\n",
     "    Tool(\n",
     "        name = \"Current Search\",\n",
@@ -149,12 +149,8 @@
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m\n",
-      "Thought: Do I need to use a tool? Yes\n",
-      "Action: Current Search\n",
-      "Action Input: Thai food dinner recipes\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m59 easy Thai recipes for any night of the week · Marion Grasby's Thai spicy chilli and basil fried rice · Thai curry noodle soup · Marion Grasby's Thai Spicy ...\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? No\n",
-      "AI: Here are some great Thai dinner recipes you can try this week: Marion Grasby's Thai Spicy Chilli and Basil Fried Rice, Thai Curry Noodle Soup, Thai Green Curry with Coconut Rice, Thai Red Curry with Vegetables, and Thai Coconut Soup. I hope you enjoy them!\u001b[0m\n",
+      "Thought: Do I need to use a tool? No\n",
+      "AI: If you like Thai food, some great dinner options this week could include Thai green curry, Pad Thai, or a Thai-style stir-fry. You could also try making a Thai-style soup or salad. Enjoy!\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -162,7 +158,7 @@
     {
      "data": {
       "text/plain": [
-       "\"Here are some great Thai dinner recipes you can try this week: Marion Grasby's Thai Spicy Chilli and Basil Fried Rice, Thai Curry Noodle Soup, Thai Green Curry with Coconut Rice, Thai Red Curry with Vegetables, and Thai Coconut Soup. I hope you enjoy them!\""
+       "'If you like Thai food, some great dinner options this week could include Thai green curry, Pad Thai, or a Thai-style stir-fry. You could also try making a Thai-style soup or salad. Enjoy!'"
       ]
      },
      "execution_count": 7,
@@ -191,9 +187,9 @@
       "Thought: Do I need to use a tool? Yes\n",
       "Action: Current Search\n",
       "Action Input: Who won the World Cup in 1978\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mArgentina national football team\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mThe Cup was won by the host nation, Argentina, who defeated the Netherlands 3–1 in the final, after extra time. The final was held at River Plate's home stadium ... Amid Argentina's celebrations, there was sympathy for the Netherlands, runners-up for the second tournament running, following a 3-1 final defeat at the Estadio ... The match was won by the Argentine squad in extra time by a score of 3–1. Mario Kempes, who finished as the tournament's top scorer, was named the man of the ... May 21, 2022 ... Argentina won the World Cup for the first time in their history, beating Netherlands 3-1 in the final. This edition of the World Cup was full of ... The adidas Golden Ball is presented to the best player at each FIFA World Cup finals. Those who finish as runners-up in the vote receive the adidas Silver ... Holders West Germany failed to beat Holland and Italy and were eliminated when Berti Vogts' own goal gave Austria a 3-2 victory. Holland thrashed the Austrians ... Jun 14, 2018 ... On a clear afternoon on 1 June 1978 at the revamped El Monumental stadium in Buenos Aires' Belgrano barrio, several hundred children in white ... Dec 15, 2022 ... The tournament couldn't have gone better for the ruling junta. Argentina went on to win the championship, defeating the Netherlands, 3-1, in the ... Nov 9, 2022 ... Host: Argentina Teams: 16. Format: Group stage, second round, third-place playoff, final. Matches: 38. Goals: 102. Winner: Argentina Feb 19, 2009 ... Argentina sealed their first World Cup win on home soil when they defeated the Netherlands in an exciting final that went to extra-time. For the ...\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? No\n",
-      "AI: The last letter in your name is \"b\" and the winner of the 1978 World Cup was the Argentina national football team.\u001b[0m\n",
+      "AI: The last letter in your name is 'b'. Argentina won the World Cup in 1978.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -201,7 +197,7 @@
     {
      "data": {
       "text/plain": [
-       "'The last letter in your name is \"b\" and the winner of the 1978 World Cup was the Argentina national football team.'"
+       "\"The last letter in your name is 'b'. Argentina won the World Cup in 1978.\""
       ]
      },
      "execution_count": 8,
@@ -230,9 +226,9 @@
       "Thought: Do I need to use a tool? Yes\n",
       "Action: Current Search\n",
       "Action Input: Current temperature in Pomfret\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mPartly cloudy skies. High around 70F. Winds W at 5 to 10 mph. Humidity41%.\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mA mixture of rain and snow showers. High 39F. Winds NNW at 5 to 10 mph. Chance of precip 50%. Snow accumulations less than one inch. Pomfret, CT Weather Forecast, with current conditions, wind, air quality, and what to expect for the next 3 days. Pomfret Center Weather Forecasts. ... Pomfret Center, CT Weather Conditionsstar_ratehome ... Tomorrow's temperature is forecast to be COOLER than today. It is 46 degrees fahrenheit, or 8 degrees celsius and feels like 46 degrees fahrenheit. The barometric pressure is 29.78 - measured by inch of mercury units - ... Pomfret Weather Forecasts. ... Pomfret, MD Weather Conditionsstar_ratehome ... Tomorrow's temperature is forecast to be MUCH COOLER than today. Additional Headlines. En Español · Share |. Current conditions at ... Pomfret CT. Tonight ... Past Weather Information · Interactive Forecast Map. Pomfret MD detailed current weather report for 20675 in Charles county, Maryland. ... Pomfret, MD weather condition is Mostly Cloudy and 43°F. Mostly Cloudy. Hazardous Weather Conditions. Hazardous Weather Outlook · En Español · Share |. Current conditions at ... South Pomfret VT. Tonight. Pomfret Center, CT Weather. Current Report for Thu Jan 5 2023. As of 2:00 PM EST. 5-Day Forecast | Road Conditions. 45°F 7°c. Feels Like 44°F. Pomfret Center CT. Today. Today: Areas of fog before 9am. Otherwise, cloudy, with a ... Otherwise, cloudy, with a temperature falling to around 33 by 5pm.\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? No\n",
-      "AI: The current temperature in Pomfret is around 70F with partly cloudy skies and winds W at 5 to 10 mph. The humidity is 41%.\u001b[0m\n",
+      "AI: The current temperature in Pomfret is 45°F (7°C) and it feels like 44°F.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -240,7 +236,7 @@
     {
      "data": {
       "text/plain": [
-       "'The current temperature in Pomfret is around 70F with partly cloudy skies and winds W at 5 to 10 mph. The humidity is 41%.'"
+       "'The current temperature in Pomfret is 45°F (7°C) and it feels like 44°F.'"
       ]
      },
      "execution_count": 9,

docs/modules/agents/agents/examples/mrkl.ipynb

@@ -33,7 +33,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "id": "07e96d99",
    "metadata": {},
    "outputs": [],
@@ -41,7 +41,7 @@
     "llm = OpenAI(temperature=0)\n",
     "search = SerpAPIWrapper()\n",
     "llm_math_chain = LLMMathChain(llm=llm, verbose=True)\n",
-    "db = SQLDatabase.from_uri(\"sqlite:///../../../../../notebooks/Chinook.db\")\n",
+    "db = SQLDatabase.from_uri(\"sqlite:///../../../../notebooks/Chinook.db\")\n",
     "db_chain = SQLDatabaseChain(llm=llm, database=db, verbose=True)\n",
     "tools = [\n",
     "    Tool(\n",
@@ -64,7 +64,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 3,
    "id": "a069c4b6",
    "metadata": {},
    "outputs": [],
@@ -74,7 +74,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 4,
    "id": "e603cd7d",
    "metadata": {},
    "outputs": [
@@ -88,24 +88,30 @@
       "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
       "Action: Search\n",
       "Action Input: \"Who is Leo DiCaprio's girlfriend?\"\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mDiCaprio met actor Camila Morrone in December 2017, when she was 20 and he was 43. They were spotted at Coachella and went on multiple vacations together. Some reports suggested that DiCaprio was ready to ask Morrone to marry him. The couple made their red carpet debut at the 2020 Academy Awards.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate Camila Morrone's age raised to the 0.43 power.\n",
+      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Camila Morrone's age\n",
+      "Action: Search\n",
+      "Action Input: \"How old is Camila Morrone?\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3m25 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 25 raised to the 0.43 power\n",
       "Action: Calculator\n",
-      "Action Input: 21^0.43\u001b[0m\n",
+      "Action Input: 25^0.43\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "21^0.43\u001b[32;1m\u001b[1;3m\n",
-      "```text\n",
-      "21**0.43\n",
+      "25^0.43\u001b[32;1m\u001b[1;3m\n",
+      "```python\n",
+      "import math\n",
+      "print(math.pow(25, 0.43))\n",
       "```\n",
-      "...numexpr.evaluate(\"21**0.43\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m3.7030049853137306\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m3.991298452658078\n",
+      "\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
       "\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.7030049853137306\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
-      "Final Answer: Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.7030049853137306.\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Camila Morrone is 25 years old and her age raised to the 0.43 power is 3.991298452658078.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -113,10 +119,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.7030049853137306.\""
+       "'Camila Morrone is 25 years old and her age raised to the 0.43 power is 3.991298452658078.'"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -127,7 +133,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 5,
    "id": "a5c07010",
    "metadata": {},
    "outputs": [
@@ -141,36 +147,21 @@
       "\u001b[32;1m\u001b[1;3m I need to find out the artist's full name and then search the FooBar database for their albums.\n",
       "Action: Search\n",
       "Action Input: \"The Storm Before the Calm\" artist\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mThe Storm Before the Calm (stylized in all lowercase) is the tenth (and eighth international) studio album by Canadian-American singer-songwriter Alanis Morissette, released June 17, 2022, via Epiphany Music and Thirty Tigers, as well as by RCA Records in Europe.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now need to search the FooBar database for Alanis Morissette's albums.\n",
+      "Observation: \u001b[36;1m\u001b[1;3mThe Storm Before the Calm (stylized in all lowercase) is the tenth (and eighth international) studio album by Canadian-American singer-songwriter Alanis ...\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now need to search the FooBar database for Alanis Morissette's albums\n",
       "Action: FooBar DB\n",
       "Action Input: What albums by Alanis Morissette are in the FooBar database?\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
-      "What albums by Alanis Morissette are in the FooBar database?\n",
-      "SQLQuery:"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/harrisonchase/workplace/langchain/langchain/sql_database.py:191: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
-      "  sample_rows = connection.execute(command)\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[32;1m\u001b[1;3m SELECT \"Title\" FROM \"Album\" INNER JOIN \"Artist\" ON \"Album\".\"ArtistId\" = \"Artist\".\"ArtistId\" WHERE \"Name\" = 'Alanis Morissette' LIMIT 5;\u001b[0m\n",
+      "What albums by Alanis Morissette are in the FooBar database? \n",
+      "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT Title FROM Album INNER JOIN Artist ON Album.ArtistId = Artist.ArtistId WHERE Artist.Name = 'Alanis Morissette' LIMIT 5;\u001b[0m\n",
       "SQLResult: \u001b[33;1m\u001b[1;3m[('Jagged Little Pill',)]\u001b[0m\n",
       "Answer:\u001b[32;1m\u001b[1;3m The albums by Alanis Morissette in the FooBar database are Jagged Little Pill.\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
       "\n",
       "Observation: \u001b[38;5;200m\u001b[1;3m The albums by Alanis Morissette in the FooBar database are Jagged Little Pill.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
-      "Final Answer: The artist who released the album 'The Storm Before the Calm' is Alanis Morissette and the albums of hers in the FooBar database are Jagged Little Pill.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: The artist who released the album The Storm Before the Calm is Alanis Morissette and the albums of theirs in the FooBar database are Jagged Little Pill.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -178,10 +169,10 @@
     {
      "data": {
       "text/plain": [
-       "\"The artist who released the album 'The Storm Before the Calm' is Alanis Morissette and the albums of hers in the FooBar database are Jagged Little Pill.\""
+       "'The artist who released the album The Storm Before the Calm is Alanis Morissette and the albums of theirs in the FooBar database are Jagged Little Pill.'"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/modules/agents/agents/examples/mrkl_chat.ipynb

@@ -21,7 +21,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 1,
    "id": "ac561cc4",
    "metadata": {},
    "outputs": [],
@@ -34,7 +34,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 2,
    "id": "07e96d99",
    "metadata": {},
    "outputs": [],
@@ -43,7 +43,7 @@
     "llm1 = OpenAI(temperature=0)\n",
     "search = SerpAPIWrapper()\n",
     "llm_math_chain = LLMMathChain(llm=llm1, verbose=True)\n",
-    "db = SQLDatabase.from_uri(\"sqlite:///../../../../../notebooks/Chinook.db\")\n",
+    "db = SQLDatabase.from_uri(\"sqlite:///../../../../notebooks/Chinook.db\")\n",
     "db_chain = SQLDatabaseChain(llm=llm1, database=db, verbose=True)\n",
     "tools = [\n",
     "    Tool(\n",
@@ -66,7 +66,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 3,
    "id": "a069c4b6",
    "metadata": {},
    "outputs": [],
@@ -76,7 +76,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 4,
    "id": "e603cd7d",
    "metadata": {},
    "outputs": [
@@ -92,34 +92,37 @@
       "```\n",
       "{\n",
       "  \"action\": \"Search\",\n",
-      "  \"action_input\": \"Leo DiCaprio girlfriend\"\n",
+      "  \"action_input\": \"Who is Leo DiCaprio's girlfriend?\"\n",
       "}\n",
       "```\n",
       "\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mGigi Hadid: 2022 Leo and Gigi were first linked back in September 2022, when a source told Us Weekly that Leo had his “sights set\" on her (alarming way to put it, but okay).\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mFor the second question, I need to calculate the age raised to the 0.43 power. I will use the calculator tool.\n",
+      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mFor the second question, I need to use the calculator tool to raise her current age to the 0.43 power.\n",
       "Action:\n",
       "```\n",
       "{\n",
       "  \"action\": \"Calculator\",\n",
-      "  \"action_input\": \"((2022-1995)^0.43)\"\n",
+      "  \"action_input\": \"22.0^(0.43)\"\n",
       "}\n",
       "```\n",
+      "\n",
       "\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "((2022-1995)^0.43)\u001b[32;1m\u001b[1;3m\n",
-      "```text\n",
-      "(2022-1995)**0.43\n",
+      "22.0^(0.43)\u001b[32;1m\u001b[1;3m\n",
+      "```python\n",
+      "import math\n",
+      "print(math.pow(22.0, 0.43))\n",
       "```\n",
-      "...numexpr.evaluate(\"(2022-1995)**0.43\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m4.125593352125936\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m3.777824273683966\n",
+      "\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
       "\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 4.125593352125936\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.777824273683966\n",
+      "\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3mI now know the final answer.\n",
-      "Final Answer: Gigi Hadid is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is approximately 4.13.\u001b[0m\n",
+      "Final Answer: Camila Morrone, 3.777824273683966.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -127,10 +130,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Gigi Hadid is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is approximately 4.13.\""
+       "'Camila Morrone, 3.777824273683966.'"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -141,7 +144,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 5,
    "id": "a5c07010",
    "metadata": {},
    "outputs": [
@@ -153,7 +156,7 @@
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3mQuestion: What is the full name of the artist who recently released an album called 'The Storm Before the Calm' and are they in the FooBar database? If so, what albums of theirs are in the FooBar database?\n",
-      "Thought: I should use the Search tool to find the answer to the first part of the question and then use the FooBar DB tool to find the answer to the second part.\n",
+      "Thought: I should use the Search tool to find the answer to the first part of the question and then use the FooBar DB tool to find the answer to the second part of the question.\n",
       "Action:\n",
       "```\n",
       "{\n",
@@ -163,7 +166,7 @@
       "```\n",
       "\u001b[0m\n",
       "Observation: \u001b[36;1m\u001b[1;3mAlanis Morissette\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mNow that I know the artist's name, I can use the FooBar DB tool to find out if they are in the database and what albums of theirs are in it.\n",
+      "Thought:\u001b[32;1m\u001b[1;3mNow that I have the name of the artist, I can use the FooBar DB tool to find their albums in the database.\n",
       "Action:\n",
       "```\n",
       "{\n",
@@ -175,7 +178,7 @@
       "\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
-      "What albums does Alanis Morissette have in the database?\n",
+      "What albums does Alanis Morissette have in the database? \n",
       "SQLQuery:"
      ]
     },
@@ -183,7 +186,7 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/Users/harrisonchase/workplace/langchain/langchain/sql_database.py:191: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
+      "/Users/harrisonchase/workplace/langchain/langchain/sql_database.py:141: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
       "  sample_rows = connection.execute(command)\n"
      ]
     },
@@ -191,14 +194,14 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "\u001b[32;1m\u001b[1;3m SELECT \"Title\" FROM \"Album\" WHERE \"ArtistId\" IN (SELECT \"ArtistId\" FROM \"Artist\" WHERE \"Name\" = 'Alanis Morissette') LIMIT 5;\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m SELECT Title FROM Album WHERE ArtistId IN (SELECT ArtistId FROM Artist WHERE Name = 'Alanis Morissette') LIMIT 5;\u001b[0m\n",
       "SQLResult: \u001b[33;1m\u001b[1;3m[('Jagged Little Pill',)]\u001b[0m\n",
-      "Answer:\u001b[32;1m\u001b[1;3m Alanis Morissette has the album Jagged Little Pill in the database.\u001b[0m\n",
+      "Answer:\u001b[32;1m\u001b[1;3m Alanis Morissette has the album 'Jagged Little Pill' in the database.\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
       "\n",
-      "Observation: \u001b[38;5;200m\u001b[1;3m Alanis Morissette has the album Jagged Little Pill in the database.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mThe artist Alanis Morissette is in the FooBar database and has the album Jagged Little Pill in it.\n",
-      "Final Answer: Alanis Morissette is in the FooBar database and has the album Jagged Little Pill in it.\u001b[0m\n",
+      "Observation: \u001b[38;5;200m\u001b[1;3m Alanis Morissette has the album 'Jagged Little Pill' in the database.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3mI have found the answer to both parts of the question.\n",
+      "Final Answer: The artist who recently released an album called 'The Storm Before the Calm' is Alanis Morissette. The album 'Jagged Little Pill' is in the FooBar database.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -206,10 +209,10 @@
     {
      "data": {
       "text/plain": [
-       "'Alanis Morissette is in the FooBar database and has the album Jagged Little Pill in it.'"
+       "\"The artist who recently released an album called 'The Storm Before the Calm' is Alanis Morissette. The album 'Jagged Little Pill' is in the FooBar database.\""
       ]
      },
-     "execution_count": 13,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/modules/agents/agents/examples/self_ask_with_search.ipynb

@@ -12,7 +12,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 2,
    "id": "7e3b513e",
    "metadata": {},
    "outputs": [
@@ -25,12 +25,11 @@
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m Yes.\n",
       "Follow up: Who is the reigning men's U.S. Open champion?\u001b[0m\n",
-      "Intermediate answer: \u001b[36;1m\u001b[1;3mCarlos Alcaraz Garfia\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mFollow up: Where is Carlos Alcaraz Garfia from?\u001b[0m\n",
+      "Intermediate answer: \u001b[36;1m\u001b[1;3mCarlos Alcaraz won the 2022 Men's single title while Poland's Iga Swiatek won the Women's single title defeating Tunisian's Ons Jabeur.\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3mFollow up: Where is Carlos Alcaraz from?\u001b[0m\n",
       "Intermediate answer: \u001b[36;1m\u001b[1;3mEl Palmar, Spain\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3mSo the final answer is: El Palmar, Spain\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
+      "\u001b[1m> Finished AgentExecutor chain.\u001b[0m\n"
      ]
     },
     {
@@ -39,7 +38,7 @@
        "'El Palmar, Spain'"
       ]
      },
-     "execution_count": 1,
+     "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -62,14 +61,6 @@
     "self_ask_with_search = initialize_agent(tools, llm, agent=AgentType.SELF_ASK_WITH_SEARCH, verbose=True)\n",
     "self_ask_with_search.run(\"What is the hometown of the reigning men's U.S. Open champion?\")"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b2e4d6bc",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -88,7 +79,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.9"
   },
   "vscode": {
    "interpreter": {

docs/modules/agents/toolkits/examples/csv.ipynb

@@ -35,7 +35,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 11,
    "id": "16c4dc59",
    "metadata": {},
    "outputs": [],
@@ -45,7 +45,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 12,
    "id": "46b9489d",
    "metadata": {},
    "outputs": [
@@ -72,7 +72,7 @@
        "'There are 891 rows in the dataframe.'"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 12,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -83,7 +83,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "id": "a96309be",
    "metadata": {},
    "outputs": [
@@ -110,7 +110,7 @@
        "'30 people have more than 3 siblings.'"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -121,7 +121,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "id": "964a09f7",
    "metadata": {},
    "outputs": [
@@ -143,7 +143,7 @@
       "Thought:\u001b[32;1m\u001b[1;3m I need to import the math library\n",
       "Action: python_repl_ast\n",
       "Action Input: import math\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mNone\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m I can now calculate the square root\n",
       "Action: python_repl_ast\n",
       "Action Input: math.sqrt(df['Age'].mean())\u001b[0m\n",
@@ -160,7 +160,7 @@
        "'5.449689683556195'"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/modules/agents/toolkits/examples/jira.ipynb

@@ -1,167 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "245a954a",
-   "metadata": {},
-   "source": [
-    "# Jira\n",
-    "\n",
-    "This notebook goes over how to use the Jira tool.\n",
-    "The Jira tool allows agents to interact with a given Jira instance, performing actions such as searching for issues and creating issues, the tool wraps the atlassian-python-api library, for more see: https://atlassian-python-api.readthedocs.io/jira.html\n",
-    "\n",
-    "To use this tool, you must first set as environment variables:\n",
-    "    JIRA_API_TOKEN\n",
-    "    JIRA_USERNAME\n",
-    "    JIRA_INSTANCE_URL"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "961b3689",
-   "metadata": {
-    "vscode": {
-     "languageId": "shellscript"
-    },
-    "ExecuteTime": {
-     "start_time": "2023-04-17T10:21:18.698672Z",
-     "end_time": "2023-04-17T10:21:20.168639Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "%pip install atlassian-python-api"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "34bb5968",
-   "metadata": {
-    "ExecuteTime": {
-     "start_time": "2023-04-17T10:21:22.911233Z",
-     "end_time": "2023-04-17T10:21:23.730922Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from langchain.agents import AgentType\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents.agent_toolkits.jira.toolkit import JiraToolkit\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.utilities.jira import JiraAPIWrapper"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "outputs": [],
-   "source": [
-    "os.environ[\"JIRA_API_TOKEN\"] = \"abc\"\n",
-    "os.environ[\"JIRA_USERNAME\"] = \"123\"\n",
-    "os.environ[\"JIRA_INSTANCE_URL\"] = \"https://jira.atlassian.com\"\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"xyz\""
-   ],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "start_time": "2023-04-17T10:22:42.499447Z",
-     "end_time": "2023-04-17T10:22:42.505412Z"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "ac4910f8",
-   "metadata": {
-    "ExecuteTime": {
-     "start_time": "2023-04-17T10:22:44.664481Z",
-     "end_time": "2023-04-17T10:22:44.720538Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "llm = OpenAI(temperature=0)\n",
-    "jira = JiraAPIWrapper()\n",
-    "toolkit = JiraToolkit.from_jira_api_wrapper(jira)\n",
-    "agent = initialize_agent(\n",
-    "    toolkit.get_tools(),\n",
-    "    llm,\n",
-    "    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
-    "    verbose=True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001B[1m> Entering new AgentExecutor chain...\u001B[0m\n",
-      "\u001B[32;1m\u001B[1;3m I need to create an issue in project PW\n",
-      "Action: Create Issue\n",
-      "Action Input: {\"summary\": \"Make more fried rice\", \"description\": \"Reminder to make more fried rice\", \"issuetype\": {\"name\": \"Task\"}, \"priority\": {\"name\": \"Low\"}, \"project\": {\"key\": \"PW\"}}\u001B[0m\n",
-      "Observation: \u001B[38;5;200m\u001B[1;3mNone\u001B[0m\n",
-      "Thought:\u001B[32;1m\u001B[1;3m I now know the final answer\n",
-      "Final Answer: A new issue has been created in project PW with the summary \"Make more fried rice\" and description \"Reminder to make more fried rice\".\u001B[0m\n",
-      "\n",
-      "\u001B[1m> Finished chain.\u001B[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": "'A new issue has been created in project PW with the summary \"Make more fried rice\" and description \"Reminder to make more fried rice\".'"
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent.run(\"make a new issue in project PW to remind me to make more fried rice\")"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "start_time": "2023-04-17T10:23:33.662454Z",
-     "end_time": "2023-04-17T10:23:38.121883Z"
-    }
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.7"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "53f3bc57609c7a84333bb558594977aa5b4026b1d6070b93987956689e367341"
-   }
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/agents/toolkits/examples/openapi.ipynb

@@ -15,7 +15,7 @@
    "id": "a389367b",
    "metadata": {},
    "source": [
-    "## 1st example: hierarchical planning agent\n",
+    "# 1st example: hierarchical planning agent\n",
     "\n",
     "In this example, we'll consider an approach called hierarchical planning, common in robotics and appearing in recent works for LLMs X robotics. We'll see it's a viable approach to start working with a massive API spec AND to assist with user queries that require multiple steps against the API.\n",
     "\n",
@@ -31,7 +31,7 @@
    "id": "4b6ecf6e",
    "metadata": {},
    "source": [
-    "### To start, let's collect some OpenAPI specs."
+    "## To start, let's collect some OpenAPI specs."
    ]
   },
   {
@@ -169,7 +169,7 @@
    "id": "76349780",
    "metadata": {},
    "source": [
-    "### How big is this spec?"
+    "## How big is this spec?"
    ]
   },
   {
@@ -229,7 +229,7 @@
    "id": "cbc4964e",
    "metadata": {},
    "source": [
-    "### Let's see some examples!\n",
+    "## Let's see some examples!\n",
     "\n",
     "Starting with GPT-4. (Some robustness iterations under way for GPT-3 family.)"
    ]
@@ -759,7 +759,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.9.0"
   }
  },
  "nbformat": 4,

docs/modules/agents/toolkits/examples/powerbi.ipynb

@@ -1,219 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "0e499e90-7a6d-4fab-8aab-31a4df417601",
-   "metadata": {},
-   "source": [
-    "# PowerBI Dataset Agent\n",
-    "\n",
-    "This notebook showcases an agent designed to interact with a Power BI Dataset. The agent is designed to answer more general questions about a dataset, as well as recover from errors.\n",
-    "\n",
-    "Note that, as this agent is in active development, all answers might not be correct. It runs against the [executequery endpoint](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/execute-queries), which does not allow deletes.\n",
-    "\n",
-    "### Some notes\n",
-    "- It relies on authentication with the azure.identity package, which can be installed with `pip install azure-identity`. Alternatively you can create the powerbi dataset with a token as a string without supplying the credentials.\n",
-    "- You can also supply a username to impersonate for use with datasets that have RLS enabled. \n",
-    "- The toolkit uses a LLM to create the query from the question, the agent uses the LLM for the overall execution.\n",
-    "- Testing was done mostly with a `text-davinci-003` model, codex models did not seem to perform ver well."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ec927ac6-9b2a-4e8a-9a6e-3e429191875c",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "## Initialization"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "53422913-967b-4f2a-8022-00269c1be1b1",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.agents.agent_toolkits import create_pbi_agent\n",
-    "from langchain.agents.agent_toolkits import PowerBIToolkit\n",
-    "from langchain.utilities.powerbi import PowerBIDataset\n",
-    "from langchain.llms.openai import AzureOpenAI\n",
-    "from langchain.agents import AgentExecutor\n",
-    "from azure.identity import DefaultAzureCredential"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "090f3699-79c6-4ce1-ab96-a94f0121fd64",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "fast_llm = AzureOpenAI(temperature=0.5, max_tokens=1000, deployment_name=\"gpt-35-turbo\", verbose=True)\n",
-    "smart_llm = AzureOpenAI(temperature=0, max_tokens=100, deployment_name=\"gpt-4\", verbose=True)\n",
-    "\n",
-    "toolkit = PowerBIToolkit(\n",
-    "    powerbi=PowerBIDataset(dataset_id=\"<dataset_id>\", table_names=['table1', 'table2'], credential=DefaultAzureCredential()), \n",
-    "    llm=smart_llm\n",
-    ")\n",
-    "\n",
-    "agent_executor = create_pbi_agent(\n",
-    "    llm=fast_llm,\n",
-    "    toolkit=toolkit,\n",
-    "    verbose=True,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "36ae48c7-cb08-4fef-977e-c7d4b96a464b",
-   "metadata": {},
-   "source": [
-    "## Example: describing a table"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "ff70e83d-5ad0-4fc7-bb96-27d82ac166d7",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "agent_executor.run(\"Describe table1\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "9abcfe8e-1868-42a4-8345-ad2d9b44c681",
-   "metadata": {},
-   "source": [
-    "## Example: simple query on a table\n",
-    "In this example, the agent actually figures out the correct query to get a row count of the table."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "bea76658-a65b-47e2-b294-6d52c5556246",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "agent_executor.run(\"How many records are in table1?\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6fbc26af-97e4-4a21-82aa-48bdc992da26",
-   "metadata": {},
-   "source": [
-    "## Example: running queries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "17bea710-4a23-4de0-b48e-21d57be48293",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "agent_executor.run(\"How many records are there by dimension1 in table2?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "474dddda-c067-4eeb-98b1-e763ee78b18c",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "agent_executor.run(\"What unique values are there for dimensions2 in table2\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "6fd950e4",
-   "metadata": {},
-   "source": [
-    "## Example: add your own few-shot prompts"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "87d677f9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#fictional example\n",
-    "few_shots = \"\"\"\n",
-    "Question: How many rows are in the table revenue?\n",
-    "DAX: EVALUATE ROW(\"Number of rows\", COUNTROWS(revenue_details))\n",
-    "----\n",
-    "Question: How many rows are in the table revenue where year is not empty?\n",
-    "DAX: EVALUATE ROW(\"Number of rows\", COUNTROWS(FILTER(revenue_details, revenue_details[year] <> \"\")))\n",
-    "----\n",
-    "Question: What was the average of value in revenue in dollars?\n",
-    "DAX: EVALUATE ROW(\"Average\", AVERAGE(revenue_details[dollar_value]))\n",
-    "----\n",
-    "\"\"\"\n",
-    "toolkit = PowerBIToolkit(\n",
-    "    powerbi=PowerBIDataset(dataset_id=\"<dataset_id>\", table_names=['table1', 'table2'], credential=DefaultAzureCredential()), \n",
-    "    llm=smart_llm,\n",
-    "    examples=few_shots,\n",
-    ")\n",
-    "agent_executor = create_pbi_agent(\n",
-    "    llm=fast_llm,\n",
-    "    toolkit=toolkit,\n",
-    "    verbose=True,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "33f4bb43",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "agent_executor.run(\"What was the maximum of value in revenue in dollars in 2022?\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.5"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/agents/tools.rst

@@ -24,7 +24,6 @@ Next, we have some examples of customizing and generically working with tools
 
    ./tools/custom_tools.ipynb
    ./tools/multi_input_tool.ipynb
-   ./tools/tool_input_validation.ipynb
 
 
 In this documentation we cover generic tooling functionality (eg how to create your own)

docs/modules/agents/tools/custom_tools.ipynb

@@ -9,30 +9,28 @@
     "\n",
     "When constructing your own agent, you will need to provide it with a list of Tools that it can use. Besides the actual function that is called, the Tool consists of several components:\n",
     "\n",
-    "- name (str), is required and must be unique within a set of tools provided to an agent\n",
-    "- description (str), is optional but recommended, as it is used by an agent to determine tool use\n",
+    "- name (str), is required\n",
+    "- description (str), is optional\n",
     "- return_direct (bool), defaults to False\n",
-    "- args_schema (Pydantic BaseModel), is optional but recommended, can be used to provide more information or validation for expected parameters.\n",
     "\n",
-    "The function that should be called when the tool is selected should return a single string.\n",
+    "The function that should be called when the tool is selected should take as input a single string and return a single string.\n",
     "\n",
     "There are two ways to define a tool, we will cover both in the example below."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 2,
    "id": "1aaba18c",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Import things that are needed generically\n",
-    "from langchain import LLMMathChain, SerpAPIWrapper\n",
-    "from langchain.agents import AgentType, Tool, initialize_agent, tool\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.tools import BaseTool"
+    "from langchain.agents import initialize_agent, Tool\n",
+    "from langchain.agents import AgentType\n",
+    "from langchain.tools import BaseTool\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain import LLMMathChain, SerpAPIWrapper"
    ]
   },
   {
@@ -45,14 +43,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 3,
    "id": "36ed392e",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "llm = ChatOpenAI(temperature=0)"
+    "llm = OpenAI(temperature=0)"
    ]
   },
   {
@@ -78,9 +74,7 @@
    "cell_type": "code",
    "execution_count": 3,
    "id": "56ff7670",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Load the tool configs that are needed.\n",
@@ -92,31 +86,19 @@
     "        func=search.run,\n",
     "        description=\"useful for when you need to answer questions about current events\"\n",
     "    ),\n",
-    "]\n",
-    "# You can also define an args_schema to provide more information about inputs\n",
-    "from pydantic import BaseModel, Field\n",
-    "\n",
-    "class CalculatorInput(BaseModel):\n",
-    "    question: str = Field()\n",
-    "        \n",
-    "\n",
-    "tools.append(\n",
     "    Tool(\n",
     "        name=\"Calculator\",\n",
     "        func=llm_math_chain.run,\n",
-    "        description=\"useful for when you need to answer questions about math\",\n",
-    "        args_schema=CalculatorInput\n",
+    "        description=\"useful for when you need to answer questions about math\"\n",
     "    )\n",
-    ")"
+    "]"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 4,
    "id": "5b93047d",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Construct the agent. We will use the default agent type here.\n",
@@ -128,9 +110,7 @@
    "cell_type": "code",
    "execution_count": 5,
    "id": "6f96a891",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -139,22 +119,29 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mI need to find out Leo DiCaprio's girlfriend's name and her age\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
       "Action: Search\n",
-      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\u001b[36;1m\u001b[1;3mDiCaprio broke up with girlfriend Camila Morrone, 25, in the summer of 2022, after dating for four years.\u001b[0m\u001b[32;1m\u001b[1;3mI need to find out Camila Morrone's current age\n",
+      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now need to calculate her age raised to the 0.43 power\n",
       "Action: Calculator\n",
-      "Action Input: 25^(0.43)\u001b[0m\n",
+      "Action Input: 22^0.43\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "25^(0.43)\u001b[32;1m\u001b[1;3m```text\n",
-      "25**(0.43)\n",
+      "22^0.43\u001b[32;1m\u001b[1;3m\n",
+      "```python\n",
+      "import math\n",
+      "print(math.pow(22, 0.43))\n",
       "```\n",
-      "...numexpr.evaluate(\"25**(0.43)\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m3.991298452658078\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m3.777824273683966\n",
+      "\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\u001b[0m\u001b[32;1m\u001b[1;3mI now know the final answer\n",
-      "Final Answer: 3.991298452658078\u001b[0m\n",
+      "\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.777824273683966\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -162,7 +149,7 @@
     {
      "data": {
       "text/plain": [
-       "'3.991298452658078'"
+       "\"Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\""
       ]
      },
      "execution_count": 5,
@@ -184,15 +171,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 8,
    "id": "c58a7c40",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Type\n",
-    "\n",
     "class CustomSearchTool(BaseTool):\n",
     "    name = \"Search\"\n",
     "    description = \"useful for when you need to answer questions about current events\"\n",
@@ -208,7 +191,6 @@
     "class CustomCalculatorTool(BaseTool):\n",
     "    name = \"Calculator\"\n",
     "    description = \"useful for when you need to answer questions about math\"\n",
-    "    args_schema: Type[BaseModel] = CalculatorInput\n",
     "\n",
     "    def _run(self, query: str) -> str:\n",
     "        \"\"\"Use the tool.\"\"\"\n",
@@ -221,11 +203,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 9,
    "id": "3318a46f",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "tools = [CustomSearchTool(), CustomCalculatorTool()]"
@@ -233,11 +213,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 10,
    "id": "ee2d0f3a",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)"
@@ -245,11 +223,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 11,
    "id": "6a2cebbf",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -258,22 +234,29 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mI need to find out Leo DiCaprio's girlfriend's name and her age\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
       "Action: Search\n",
-      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\u001b[36;1m\u001b[1;3mDiCaprio broke up with girlfriend Camila Morrone, 25, in the summer of 2022, after dating for four years.\u001b[0m\u001b[32;1m\u001b[1;3mI need to find out Camila Morrone's current age\n",
+      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now need to calculate her age raised to the 0.43 power\n",
       "Action: Calculator\n",
-      "Action Input: 25^(0.43)\u001b[0m\n",
+      "Action Input: 22^0.43\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "25^(0.43)\u001b[32;1m\u001b[1;3m```text\n",
-      "25**(0.43)\n",
+      "22^0.43\u001b[32;1m\u001b[1;3m\n",
+      "```python\n",
+      "import math\n",
+      "print(math.pow(22, 0.43))\n",
       "```\n",
-      "...numexpr.evaluate(\"25**(0.43)\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m3.991298452658078\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m3.777824273683966\n",
+      "\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\u001b[0m\u001b[32;1m\u001b[1;3mI now know the final answer\n",
-      "Final Answer: 3.991298452658078\u001b[0m\n",
+      "\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.777824273683966\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -281,10 +264,10 @@
     {
      "data": {
       "text/plain": [
-       "'3.991298452658078'"
+       "\"Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\""
       ]
      },
-     "execution_count": 9,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -305,11 +288,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 4,
    "id": "8f15307d",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "from langchain.agents import tool\n",
@@ -317,24 +298,22 @@
     "@tool\n",
     "def search_api(query: str) -> str:\n",
     "    \"\"\"Searches the API for the query.\"\"\"\n",
-    "    return f\"Results for query {query}\""
+    "    return \"Results\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 5,
    "id": "0a23b91b",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "Tool(name='search_api', description='search_api(query: str) -> str - Searches the API for the query.', args_schema=<class 'pydantic.main.SearchApi'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x12748c4c0>, func=<function search_api at 0x16bd664c0>, coroutine=None)"
+       "Tool(name='search_api', description='search_api(query: str) -> str - Searches the API for the query.', return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1184e0cd0>, func=<function search_api at 0x1635f8700>, coroutine=None)"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -353,11 +332,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 6,
    "id": "28cdf04d",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "@tool(\"search\", return_direct=True)\n",
@@ -368,62 +345,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 7,
    "id": "1085a4bd",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "Tool(name='search', description='search(query: str) -> str - Searches the API for the query.', args_schema=<class 'pydantic.main.SearchApi'>, return_direct=True, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x12748c4c0>, func=<function search_api at 0x16bd66310>, coroutine=None)"
+       "Tool(name='search', description='search(query: str) -> str - Searches the API for the query.', return_direct=True, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1184e0cd0>, func=<function search_api at 0x1635f8670>, coroutine=None)"
       ]
      },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "search_api"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "de34a6a3",
-   "metadata": {},
-   "source": [
-    "You can also provide `args_schema` to provide more information about the argument"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "f3a5c106",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class SearchInput(BaseModel):\n",
-    "    query: str = Field(description=\"should be a search query\")\n",
-    "        \n",
-    "@tool(\"search\", return_direct=True, args_schema=SearchInput)\n",
-    "def search_api(query: str) -> str:\n",
-    "    \"\"\"Searches the API for the query.\"\"\"\n",
-    "    return \"Results\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "7914ba6b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "Tool(name='search', description='search(query: str) -> str - Searches the API for the query.', args_schema=<class '__main__.SearchInput'>, return_direct=True, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x12748c4c0>, func=<function search_api at 0x16bcf0ee0>, coroutine=None)"
-      ]
-     },
-     "execution_count": 15,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -444,7 +376,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 8,
    "id": "79213f40",
    "metadata": {},
    "outputs": [],
@@ -454,7 +386,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 9,
    "id": "e1067dcb",
    "metadata": {},
    "outputs": [],
@@ -464,7 +396,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 10,
    "id": "6c66ffe8",
    "metadata": {},
    "outputs": [],
@@ -474,7 +406,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 11,
    "id": "f45b5bc3",
    "metadata": {},
    "outputs": [],
@@ -484,7 +416,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 12,
    "id": "565e2b9b",
    "metadata": {},
    "outputs": [
@@ -495,12 +427,21 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mI need to find out Leo DiCaprio's girlfriend's name and her age.\n",
+      "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
       "Action: Google Search\n",
-      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\u001b[36;1m\u001b[1;3mI draw the lime at going to get a Mohawk, though.\" DiCaprio broke up with girlfriend Camila Morrone, 25, in the summer of 2022, after dating for four years. He's since been linked to another famous supermodel – Gigi Hadid.\u001b[0m\u001b[32;1m\u001b[1;3mNow I need to find out Camila Morrone's current age.\n",
+      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Camila Morrone's age\n",
+      "Action: Google Search\n",
+      "Action Input: \"Camila Morrone age\"\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3m25 years\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 25 raised to the 0.43 power\n",
       "Action: Calculator\n",
-      "Action Input: 25^0.43\u001b[0m\u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\u001b[0m\u001b[32;1m\u001b[1;3mI now know the final answer.\n",
-      "Final Answer: Camila Morrone's current age raised to the 0.43 power is approximately 3.99.\u001b[0m\n",
+      "Action Input: 25^0.43\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\n",
+      "\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -508,10 +449,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Camila Morrone's current age raised to the 0.43 power is approximately 3.99.\""
+       "\"Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078.\""
       ]
      },
-     "execution_count": 18,
+     "execution_count": 12,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -537,7 +478,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 13,
    "id": "3450512e",
    "metadata": {},
    "outputs": [],
@@ -566,7 +507,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 14,
    "id": "4b9a7849",
    "metadata": {},
    "outputs": [
@@ -579,7 +520,9 @@
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m I should use a music search engine to find the answer\n",
       "Action: Music Search\n",
-      "Action Input: most famous song of christmas\u001b[0m\u001b[33;1m\u001b[1;3m'All I Want For Christmas Is You' by Mariah Carey.\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Action Input: most famous song of christmas\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m'All I Want For Christmas Is You' by Mariah Carey.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
       "Final Answer: 'All I Want For Christmas Is You' by Mariah Carey.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
@@ -591,7 +534,7 @@
        "\"'All I Want For Christmas Is You' by Mariah Carey.\""
       ]
      },
-     "execution_count": 20,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -611,7 +554,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 15,
    "id": "3bb6185f",
    "metadata": {},
    "outputs": [],
@@ -629,7 +572,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 16,
    "id": "113ddb84",
    "metadata": {},
    "outputs": [],
@@ -640,11 +583,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 17,
    "id": "582439a6",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -655,7 +596,9 @@
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m I need to calculate this\n",
       "Action: Calculator\n",
-      "Action Input: 2**.12\u001b[0m\u001b[36;1m\u001b[1;3mAnswer: 1.086734862526058\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
+      "Action Input: 2**.12\u001b[0m\n",
+      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 1.2599210498948732\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -663,10 +606,10 @@
     {
      "data": {
       "text/plain": [
-       "'Answer: 1.086734862526058'"
+       "'Answer: 1.2599210498948732'"
       ]
      },
-     "execution_count": 23,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -675,149 +618,10 @@
     "agent.run(\"whats 2**.12\")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "8aa3c353-bd89-467c-9c27-b83a90cd4daa",
-   "metadata": {},
-   "source": [
-    "## Multi-argument tools\n",
-    "\n",
-    "Many functions expect structured inputs. These can also be supported using the Tool decorator or by directly subclassing `BaseTool`! We have to modify the LLM's OutputParser to map its string output to a dictionary to pass to the action, however."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "537bc628",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from typing import Optional, Union\n",
-    "\n",
-    "@tool\n",
-    "def custom_search(k: int, query: str, other_arg: Optional[str] = None):\n",
-    "    \"\"\"The custom search function.\"\"\"\n",
-    "    return f\"Here are the results for the custom search: k={k}, query={query}, other_arg={other_arg}\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "d5c992cf-776a-40cd-a6c4-e7cf65ea709e",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "import re\n",
-    "from langchain.schema import (\n",
-    "    AgentAction,\n",
-    "    AgentFinish,\n",
-    ")\n",
-    "from langchain.agents import AgentOutputParser\n",
-    "\n",
-    "# We will add a custom parser to map the arguments to a dictionary\n",
-    "class CustomOutputParser(AgentOutputParser):\n",
-    "    \n",
-    "    def parse_tool_input(self, action_input: str) -> dict:\n",
-    "        # Regex pattern to match arguments and their values\n",
-    "        pattern = r\"(\\w+)\\s*=\\s*(None|\\\"[^\\\"]*\\\"|\\d+)\"\n",
-    "        matches = re.findall(pattern, action_input)\n",
-    "        \n",
-    "        if not matches:\n",
-    "            raise ValueError(f\"Could not parse action input: `{action_input}`\")\n",
-    "\n",
-    "        # Create a dictionary with the parsed arguments and their values\n",
-    "        parsed_input = {}\n",
-    "        for arg, value in matches:\n",
-    "            if value == \"None\":\n",
-    "                parsed_value = None\n",
-    "            elif value.isdigit():\n",
-    "                parsed_value = int(value)\n",
-    "            else:\n",
-    "                parsed_value = value.strip('\"')\n",
-    "            parsed_input[arg] = parsed_value\n",
-    "\n",
-    "        return parsed_input\n",
-    "    \n",
-    "    def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:\n",
-    "        # Check if agent should finish\n",
-    "        if \"Final Answer:\" in llm_output:\n",
-    "            return AgentFinish(\n",
-    "                # Return values is generally always a dictionary with a single `output` key\n",
-    "                # It is not recommended to try anything else at the moment :)\n",
-    "                return_values={\"output\": llm_output.split(\"Final Answer:\")[-1].strip()},\n",
-    "                log=llm_output,\n",
-    "            )\n",
-    "        # Parse out the action and action input\n",
-    "        regex = r\"Action\\s*\\d*\\s*:(.*?)\\nAction\\s*\\d*\\s*Input\\s*\\d*\\s*:[\\s]*(.*)\"\n",
-    "        match = re.search(regex, llm_output, re.DOTALL)\n",
-    "        if not match:\n",
-    "            raise ValueError(f\"Could not parse LLM output: `{llm_output}`\")\n",
-    "        action = match.group(1).strip()\n",
-    "        action_input = match.group(2)\n",
-    "        tool_input = self.parse_tool_input(action_input)\n",
-    "        # Return the action and action \n",
-    "        return AgentAction(tool=action, tool_input=tool_input, log=llm_output)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "68269547-1482-4138-a6ea-58f00b4a9548",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "llm = OpenAI(temperature=0)\n",
-    "agent = initialize_agent([custom_search], llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True, agent_kwargs={\"output_parser\": CustomOutputParser()})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 27,
-   "id": "0947835a-691c-4f51-b8f4-6744e0e48ab1",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m I need to use a search function to find the answer\n",
-      "Action: custom_search\n",
-      "Action Input: k=1, query=\"me\"\u001b[0m\u001b[36;1m\u001b[1;3mHere are the results for the custom search: k=1, query=me, other_arg=None\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: The results of the custom search for k=1, query=me, other_arg=None.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'The results of the custom search for k=1, query=me, other_arg=None.'"
-      ]
-     },
-     "execution_count": 27,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent.run(\"Search for me and tell me whatever it says\")"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "caf39c66-102b-42c1-baf2-777a49886ce4",
+   "id": "537bc628",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/agents/tools/examples/arxiv.ipynb

@@ -1,170 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "245a954a",
-   "metadata": {},
-   "source": [
-    "# Arxiv API\n",
-    "\n",
-    "This notebook goes over how to use the `arxiv` component. \n",
-    "\n",
-    "First, you need to install `arxiv` python package."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d5a7209e",
-   "metadata": {
-    "tags": [],
-    "vscode": {
-     "languageId": "shellscript"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "!pip install arxiv"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "8d32b39a",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.utilities import ArxivAPIWrapper"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c89c110c-96ac-4fe1-ba3e-6056543d1a59",
-   "metadata": {},
-   "source": [
-    "Run a query to get information about some `scientific article`/articles. The query text is limited to 300 characters.\n",
-    "\n",
-    "It returns these article fields:\n",
-    "- Publishing date\n",
-    "- Title\n",
-    "- Authors\n",
-    "- Summary\n",
-    "\n",
-    "Next query returns information about one article with arxiv Id equal \"1605.08386\". "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "34bb5968",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Published: 2016-05-26\\nTitle: Heat-bath random walks with Markov bases\\nAuthors: Caprice Stanley, Tobias Windisch\\nSummary: Graphs on lattice points are studied whose edges come from a finite set of\\nallowed moves of arbitrary length. We show that the diameter of these graphs on\\nfibers of a fixed integer matrix can be bounded from above by a constant. We\\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\\nalso state explicit conditions on the set of moves so that the heat-bath random\\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\\ndimension.'"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "docs = arxiv.run(\"1605.08386\")\n",
-    "docs"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "840f70c9-8f80-4680-bb38-46198e931bcf",
-   "metadata": {},
-   "source": [
-    "Now, we want to get information about one author, `Caprice Stanley`.\n",
-    "\n",
-    "This query returns information about three articles. By default, query returns information only about three top articles."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "b0867fda-e119-4b19-9ec6-e354fa821db3",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Published: 2017-10-10\\nTitle: On Mixing Behavior of a Family of Random Walks Determined by a Linear Recurrence\\nAuthors: Caprice Stanley, Seth Sullivant\\nSummary: We study random walks on the integers mod $G_n$ that are determined by an\\ninteger sequence $\\\\{ G_n \\\\}_{n \\\\geq 1}$ generated by a linear recurrence\\nrelation. Fourier analysis provides explicit formulas to compute the\\neigenvalues of the transition matrices and we use this to bound the mixing time\\nof the random walks.\\n\\nPublished: 2016-05-26\\nTitle: Heat-bath random walks with Markov bases\\nAuthors: Caprice Stanley, Tobias Windisch\\nSummary: Graphs on lattice points are studied whose edges come from a finite set of\\nallowed moves of arbitrary length. We show that the diameter of these graphs on\\nfibers of a fixed integer matrix can be bounded from above by a constant. We\\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\\nalso state explicit conditions on the set of moves so that the heat-bath random\\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\\ndimension.\\n\\nPublished: 2003-03-18\\nTitle: Calculation of fluxes of charged particles and neutrinos from atmospheric showers\\nAuthors: V. Plyaskin\\nSummary: The results on the fluxes of charged particles and neutrinos from a\\n3-dimensional (3D) simulation of atmospheric showers are presented. An\\nagreement of calculated fluxes with data on charged particles from the AMS and\\nCAPRICE detectors is demonstrated. Predictions on neutrino fluxes at different\\nexperimental sites are compared with results from other calculations.'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "docs = arxiv.run(\"Caprice Stanley\")\n",
-    "docs"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2d9b6292-a47d-4f99-9827-8e9f244bf887",
-   "metadata": {},
-   "source": [
-    "Now, we are trying to find information about non-existing article. In this case, the response is \"No good Arxiv Result was found\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "3580aeeb-086f-45ba-bcdc-b46f5134b3dd",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'No good Arxiv Result was found'"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "docs = arxiv.run(\"1605.08386WWW\")\n",
-    "docs"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/agents/tools/examples/bash.ipynb

@@ -5,158 +5,57 @@
       "id": "8f210ec3",
       "metadata": {},
       "source": [
-        "# Shell Tool\n",
-        "\n",
-        "Giving agents access to the shell is powerful (though risky outside a sandboxed environment).\n",
-        "\n",
-        "The LLM can use it to execute any shell commands. A common use case for this is letting the LLM interact with your local file system."
+        "# Bash\n",
+        "It can often be useful to have an LLM generate bash commands, and then run them. A common use case for this is letting the LLM interact with your local file system. We provide an easy util to execute bash commands."
       ]
     },
     {
       "cell_type": "code",
       "execution_count": 1,
       "id": "f7b3767b",
-      "metadata": {
-        "tags": []
-      },
+      "metadata": {},
       "outputs": [],
       "source": [
-        "from langchain.tools import ShellTool\n",
-        "\n",
-        "shell_tool = ShellTool()"
+        "from langchain.utilities import BashProcess"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": 2,
-      "id": "c92ac832-556b-4f66-baa4-b78f965dfba0",
-      "metadata": {
-        "tags": []
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Hello World!\n",
-            "\n",
-            "real\t0m0.000s\n",
-            "user\t0m0.000s\n",
-            "sys\t0m0.000s\n",
-            "\n"
-          ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/Users/wfh/code/lc/lckg/langchain/tools/shell/tool.py:34: UserWarning: The shell tool has no safeguards by default. Use at your own risk.\n",
-            "  warnings.warn(\n"
-          ]
-        }
-      ],
-      "source": [
-        "print(shell_tool.run({\"commands\": [\"echo 'Hello World!'\", \"time\"]}))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "2fa952fc",
+      "id": "cf1c92f0",
       "metadata": {},
+      "outputs": [],
       "source": [
-        "### Use with Agents\n",
-        "\n",
-        "As with all tools, these can be given to an agent to accomplish more complex tasks. Let's have the agent fetch some links from a web page."
+        "bash = BashProcess()"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": 3,
-      "id": "851fee9f",
-      "metadata": {
-        "tags": []
-      },
+      "id": "2fa952fc",
+      "metadata": {},
       "outputs": [
         {
           "name": "stdout",
           "output_type": "stream",
           "text": [
-            "\n",
-            "\n",
-            "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-            "\u001b[32;1m\u001b[1;3mQuestion: What is the task?\n",
-            "Thought: We need to download the langchain.com webpage and extract all the URLs from it. Then we need to sort the URLs and return them.\n",
-            "Action:\n",
-            "```\n",
-            "{\n",
-            "  \"action\": \"shell\",\n",
-            "  \"action_input\": {\n",
-            "    \"commands\": [\n",
-            "      \"curl -s https://langchain.com | grep -o 'http[s]*://[^\\\" ]*' | sort\"\n",
-            "    ]\n",
-            "  }\n",
-            "}\n",
-            "```\n",
-            "\u001b[0m"
+            "bash.ipynb\n",
+            "google_search.ipynb\n",
+            "python.ipynb\n",
+            "requests.ipynb\n",
+            "serpapi.ipynb\n",
+            "\n"
           ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/Users/wfh/code/lc/lckg/langchain/tools/shell/tool.py:34: UserWarning: The shell tool has no safeguards by default. Use at your own risk.\n",
-            "  warnings.warn(\n"
-          ]
-        },
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "Observation: \u001b[36;1m\u001b[1;3mhttps://blog.langchain.dev/\n",
-            "https://discord.gg/6adMQxSpJS\n",
-            "https://docs.langchain.com/docs/\n",
-            "https://github.com/hwchase17/chat-langchain\n",
-            "https://github.com/hwchase17/langchain\n",
-            "https://github.com/hwchase17/langchainjs\n",
-            "https://github.com/sullivan-sean/chat-langchainjs\n",
-            "https://js.langchain.com/docs/\n",
-            "https://python.langchain.com/en/latest/\n",
-            "https://twitter.com/langchainai\n",
-            "\u001b[0m\n",
-            "Thought:\u001b[32;1m\u001b[1;3mThe URLs have been successfully extracted and sorted. We can return the list of URLs as the final answer.\n",
-            "Final Answer: [\"https://blog.langchain.dev/\", \"https://discord.gg/6adMQxSpJS\", \"https://docs.langchain.com/docs/\", \"https://github.com/hwchase17/chat-langchain\", \"https://github.com/hwchase17/langchain\", \"https://github.com/hwchase17/langchainjs\", \"https://github.com/sullivan-sean/chat-langchainjs\", \"https://js.langchain.com/docs/\", \"https://python.langchain.com/en/latest/\", \"https://twitter.com/langchainai\"]\u001b[0m\n",
-            "\n",
-            "\u001b[1m> Finished chain.\u001b[0m\n"
-          ]
-        },
-        {
-          "data": {
-            "text/plain": [
-              "'[\"https://blog.langchain.dev/\", \"https://discord.gg/6adMQxSpJS\", \"https://docs.langchain.com/docs/\", \"https://github.com/hwchase17/chat-langchain\", \"https://github.com/hwchase17/langchain\", \"https://github.com/hwchase17/langchainjs\", \"https://github.com/sullivan-sean/chat-langchainjs\", \"https://js.langchain.com/docs/\", \"https://python.langchain.com/en/latest/\", \"https://twitter.com/langchainai\"]'"
-            ]
-          },
-          "execution_count": 3,
-          "metadata": {},
-          "output_type": "execute_result"
         }
       ],
       "source": [
-        "from langchain.chat_models import ChatOpenAI\n",
-        "from langchain.agents import initialize_agent\n",
-        "from langchain.agents import AgentType\n",
-        "\n",
-        "llm = ChatOpenAI(temperature=0)\n",
-        "\n",
-        "shell_tool.description = shell_tool.description + f\"args {shell_tool.args}\".replace(\"{\", \"{{\").replace(\"}\", \"}}\")\n",
-        "self_ask_with_search = initialize_agent([shell_tool], llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True)\n",
-        "self_ask_with_search.run(\"Download the langchain.com webpage and grep for all urls. Return only a sorted list of them. Be sure to use double quotes.\")"
+        "print(bash.run(\"ls\"))"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
-      "id": "8d0ea3ac-0890-4e39-9cec-74bd80b4b8b8",
+      "id": "851fee9f",
       "metadata": {},
       "outputs": [],
       "source": []
@@ -178,7 +77,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.8.16"
+      "version": "3.10.9"
     }
   },
   "nbformat": 4,

docs/modules/agents/tools/examples/chatgpt_plugins.ipynb

@@ -80,8 +80,8 @@
     }
    ],
    "source": [
-    "llm = ChatOpenAI(temperature=0)\n",
-    "tools = load_tools([\"requests_all\"] )\n",
+    "llm = ChatOpenAI(temperature=0,)\n",
+    "tools = load_tools([\"requests\"] )\n",
     "tools += [tool]\n",
     "\n",
     "agent_chain = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)\n",

docs/modules/agents/tools/examples/ddg.ipynb

@@ -1,91 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "245a954a",
-   "metadata": {},
-   "source": [
-    "# DuckDuckGo Search\n",
-    "\n",
-    "This notebook goes over how to use the duck-duck-go search component."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "21e46d4d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# !pip install duckduckgo-search"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "ac4910f8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.tools import DuckDuckGoSearchRun"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "84b8f773",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "search = DuckDuckGoSearchRun()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "068991a6",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Barack Obama, in full Barack Hussein Obama II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009-17) and the first African American to hold the office. Before winning the presidency, Obama represented Illinois in the U.S. Senate (2005-08). Barack Hussein Obama II (/ b ə ˈ r ɑː k h uː ˈ s eɪ n oʊ ˈ b ɑː m ə / bə-RAHK hoo-SAYN oh-BAH-mə; born August 4, 1961) is an American former politician who served as the 44th president of the United States from 2009 to 2017. A member of the Democratic Party, he was the first African-American president of the United States. Obama previously served as a U.S. senator representing ... Barack Obama was the first African American president of the United States (2009-17). He oversaw the recovery of the U.S. economy (from the Great Recession of 2008-09) and the enactment of landmark health care reform (the Patient Protection and Affordable Care Act ). In 2009 he was awarded the Nobel Peace Prize. His birth certificate lists his first name as Barack: That\\'s how Obama has spelled his name throughout his life. His name derives from a Hebrew name which means \"lightning.\". The Hebrew word has been transliterated into English in various spellings, including Barak, Buraq, Burack, and Barack. Most common names of U.S. presidents 1789-2021. Published by. Aaron O\\'Neill , Jun 21, 2022. The most common first name for a U.S. president is James, followed by John and then William. Six U.S ...'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "search.run(\"Obama's first name?\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
-   }
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/agents/tools/examples/filesystem.ipynb

@@ -1,190 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# File System Tools\n",
-    "\n",
-    "LangChain provides tools for interacting with a local file system out of the box. This notebook walks through some of them.\n",
-    "\n",
-    "Note: these tools are not recommended for use outside a sandboxed environment! "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "First, we'll import the tools."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.tools.file_management import (\n",
-    "    ReadFileTool,\n",
-    "    CopyFileTool,\n",
-    "    DeleteFileTool,\n",
-    "    MoveFileTool,\n",
-    "    WriteFileTool,\n",
-    "    ListDirectoryTool,\n",
-    ")\n",
-    "from langchain.agents.agent_toolkits import FileManagementToolkit\n",
-    "from tempfile import TemporaryDirectory\n",
-    "\n",
-    "# We'll make a temporary directory to avoid clutter\n",
-    "working_directory = TemporaryDirectory()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## The FileManagementToolkit\n",
-    "\n",
-    "If you want to provide all the file tooling to your agent, it's easy to do so with the toolkit. We'll pass the temporary directory in as a root directory as a workspace for the LLM.\n",
-    "\n",
-    "It's recommended to always pass in a root directory, since without one, it's easy for the LLM to pollute the working directory, and without one, there isn't any validation against\n",
-    "straightforward prompt injection."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[CopyFileTool(name='copy_file', description='Create a copy of a file in a specified location', args_schema=<class 'langchain.tools.file_management.copy.FileCopyInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " DeleteFileTool(name='file_delete', description='Delete a file', args_schema=<class 'langchain.tools.file_management.delete.FileDeleteInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " FileSearchTool(name='file_search', description='Recursively search for files in a subdirectory that match the regex pattern', args_schema=<class 'langchain.tools.file_management.file_search.FileSearchInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " MoveFileTool(name='move_file', description='Move or rename a file from one location to another', args_schema=<class 'langchain.tools.file_management.move.FileMoveInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " ReadFileTool(name='read_file', description='Read file from disk', args_schema=<class 'langchain.tools.file_management.read.ReadFileInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " WriteFileTool(name='write_file', description='Write file to disk', args_schema=<class 'langchain.tools.file_management.write.WriteFileInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " ListDirectoryTool(name='list_directory', description='List files and directories in a specified folder', args_schema=<class 'langchain.tools.file_management.list_dir.DirectoryListingInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug')]"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "toolkit = FileManagementToolkit(root_dir=str(working_directory.name)) # If you don't provide a root_dir, operations will default to the current working directory\n",
-    "toolkit.get_tools()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Selecting File System Tools\n",
-    "\n",
-    "If you only want to select certain tools, you can pass them in as arguments when initializing the toolkit, or you can individually initialize the desired tools."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[ReadFileTool(name='read_file', description='Read file from disk', args_schema=<class 'langchain.tools.file_management.read.ReadFileInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " WriteFileTool(name='write_file', description='Write file to disk', args_schema=<class 'langchain.tools.file_management.write.WriteFileInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug'),\n",
-       " ListDirectoryTool(name='list_directory', description='List files and directories in a specified folder', args_schema=<class 'langchain.tools.file_management.list_dir.DirectoryListingInput'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1156f4350>, root_dir='/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/tmpxb8c3aug')]"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "tools = FileManagementToolkit(root_dir=str(working_directory.name), selected_tools=[\"read_file\", \"write_file\", \"list_directory\"]).get_tools()\n",
-    "tools"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'File written successfully to example.txt.'"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "read_tool, write_tool, list_tool = tools\n",
-    "write_tool.run({\"file_path\": \"example.txt\", \"text\": \"Hello World!\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'example.txt'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# List files in the working directory\n",
-    "list_tool.run({})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/modules/agents/tools/examples/google_places.ipynb

@@ -1,105 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "487607cd",
-   "metadata": {},
-   "source": [
-    "# Google Places\n",
-    "\n",
-    "This notebook goes through how to use Google Places API"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "8690845f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#!pip install googlemaps"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "fae31ef4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "os.environ[\"GPLACES_API_KEY\"] = \"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "abb502b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.tools import GooglePlacesTool"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "a83a02ac",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "places = GooglePlacesTool()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "2b65a285",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"1. Delfina Restaurant\\nAddress: 3621 18th St, San Francisco, CA 94110, USA\\nPhone: (415) 552-4055\\nWebsite: https://www.delfinasf.com/\\n\\n\\n2. Piccolo Forno\\nAddress: 725 Columbus Ave, San Francisco, CA 94133, USA\\nPhone: (415) 757-0087\\nWebsite: https://piccolo-forno-sf.com/\\n\\n\\n3. L'Osteria del Forno\\nAddress: 519 Columbus Ave, San Francisco, CA 94133, USA\\nPhone: (415) 982-1124\\nWebsite: Unknown\\n\\n\\n4. Il Fornaio\\nAddress: 1265 Battery St, San Francisco, CA 94111, USA\\nPhone: (415) 986-0100\\nWebsite: https://www.ilfornaio.com/\\n\\n\""
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "places.run(\"al fornos\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "66d3da8a",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/agents/tools/tool_input_validation.ipynb

@@ -1,184 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "# Tool Input Schema\n",
-    "\n",
-    "By default, tools infer the argument schema by inspecting the function signature. For more strict requirements, custom input schema can be specified, along with custom validation logic."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from typing import Any, Dict\n",
-    "\n",
-    "from langchain.agents import AgentType, initialize_agent\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.tools.requests.tool import RequestsGetTool, TextRequestsWrapper\n",
-    "from pydantic import BaseModel, Field, root_validator\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "llm = OpenAI(temperature=0)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.1\u001b[0m\n",
-      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
-     ]
-    }
-   ],
-   "source": [
-    "!pip install tldextract > /dev/null"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "import tldextract\n",
-    "\n",
-    "_APPROVED_DOMAINS = {\n",
-    "    \"langchain\",\n",
-    "    \"wikipedia\",\n",
-    "}\n",
-    "\n",
-    "class ToolInputSchema(BaseModel):\n",
-    "\n",
-    "    url: str = Field(...)\n",
-    "    \n",
-    "    @root_validator\n",
-    "    def validate_query(cls, values: Dict[str, Any]) -> Dict:\n",
-    "        url = values[\"url\"]\n",
-    "        domain = tldextract.extract(url).domain\n",
-    "        if domain not in _APPROVED_DOMAINS:\n",
-    "            raise ValueError(f\"Domain {domain} is not on the approved list:\"\n",
-    "                             f\" {sorted(_APPROVED_DOMAINS)}\")\n",
-    "        return values\n",
-    "    \n",
-    "tool = RequestsGetTool(args_schema=ToolInputSchema, requests_wrapper=TextRequestsWrapper())"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "agent = initialize_agent([tool], llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=False)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "The main title of langchain.com is \"LANG CHAIN 🦜️🔗 Official Home Page\"\n"
-     ]
-    }
-   ],
-   "source": [
-    "# This will succeed, since there aren't any arguments that will be triggered during validation\n",
-    "answer = agent.run(\"What's the main title on langchain.com?\")\n",
-    "print(answer)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "ename": "ValidationError",
-     "evalue": "1 validation error for ToolInputSchema\n__root__\n  Domain google is not on the approved list: ['langchain', 'wikipedia'] (type=value_error)",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
-      "Cell \u001b[0;32mIn[7], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m agent\u001b[39m.\u001b[39;49mrun(\u001b[39m\"\u001b[39;49m\u001b[39mWhat\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39ms the main title on google.com?\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/chains/base.py:213\u001b[0m, in \u001b[0;36mChain.run\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(args) \u001b[39m!=\u001b[39m \u001b[39m1\u001b[39m:\n\u001b[1;32m    212\u001b[0m         \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\u001b[39m\"\u001b[39m\u001b[39m`run` supports only one positional argument.\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[0;32m--> 213\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m(args[\u001b[39m0\u001b[39;49m])[\u001b[39mself\u001b[39m\u001b[39m.\u001b[39moutput_keys[\u001b[39m0\u001b[39m]]\n\u001b[1;32m    215\u001b[0m \u001b[39mif\u001b[39;00m kwargs \u001b[39mand\u001b[39;00m \u001b[39mnot\u001b[39;00m args:\n\u001b[1;32m    216\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m(kwargs)[\u001b[39mself\u001b[39m\u001b[39m.\u001b[39moutput_keys[\u001b[39m0\u001b[39m]]\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/chains/base.py:116\u001b[0m, in \u001b[0;36mChain.__call__\u001b[0;34m(self, inputs, return_only_outputs)\u001b[0m\n\u001b[1;32m    114\u001b[0m \u001b[39mexcept\u001b[39;00m (\u001b[39mKeyboardInterrupt\u001b[39;00m, \u001b[39mException\u001b[39;00m) \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m    115\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcallback_manager\u001b[39m.\u001b[39mon_chain_error(e, verbose\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose)\n\u001b[0;32m--> 116\u001b[0m     \u001b[39mraise\u001b[39;00m e\n\u001b[1;32m    117\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcallback_manager\u001b[39m.\u001b[39mon_chain_end(outputs, verbose\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose)\n\u001b[1;32m    118\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mprep_outputs(inputs, outputs, return_only_outputs)\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/chains/base.py:113\u001b[0m, in \u001b[0;36mChain.__call__\u001b[0;34m(self, inputs, return_only_outputs)\u001b[0m\n\u001b[1;32m    107\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcallback_manager\u001b[39m.\u001b[39mon_chain_start(\n\u001b[1;32m    108\u001b[0m     {\u001b[39m\"\u001b[39m\u001b[39mname\u001b[39m\u001b[39m\"\u001b[39m: \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m\u001b[39m__class__\u001b[39m\u001b[39m.\u001b[39m\u001b[39m__name__\u001b[39m},\n\u001b[1;32m    109\u001b[0m     inputs,\n\u001b[1;32m    110\u001b[0m     verbose\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose,\n\u001b[1;32m    111\u001b[0m )\n\u001b[1;32m    112\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 113\u001b[0m     outputs \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_call(inputs)\n\u001b[1;32m    114\u001b[0m \u001b[39mexcept\u001b[39;00m (\u001b[39mKeyboardInterrupt\u001b[39;00m, \u001b[39mException\u001b[39;00m) \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m    115\u001b[0m     \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcallback_manager\u001b[39m.\u001b[39mon_chain_error(e, verbose\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose)\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/agents/agent.py:792\u001b[0m, in \u001b[0;36mAgentExecutor._call\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m    790\u001b[0m \u001b[39m# We now enter the agent loop (until it returns something).\u001b[39;00m\n\u001b[1;32m    791\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_should_continue(iterations, time_elapsed):\n\u001b[0;32m--> 792\u001b[0m     next_step_output \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_take_next_step(\n\u001b[1;32m    793\u001b[0m         name_to_tool_map, color_mapping, inputs, intermediate_steps\n\u001b[1;32m    794\u001b[0m     )\n\u001b[1;32m    795\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(next_step_output, AgentFinish):\n\u001b[1;32m    796\u001b[0m         \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_return(next_step_output, intermediate_steps)\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/agents/agent.py:695\u001b[0m, in \u001b[0;36mAgentExecutor._take_next_step\u001b[0;34m(self, name_to_tool_map, color_mapping, inputs, intermediate_steps)\u001b[0m\n\u001b[1;32m    693\u001b[0m         tool_run_kwargs[\u001b[39m\"\u001b[39m\u001b[39mllm_prefix\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m    694\u001b[0m     \u001b[39m# We then call the tool on the tool input to get an observation\u001b[39;00m\n\u001b[0;32m--> 695\u001b[0m     observation \u001b[39m=\u001b[39m tool\u001b[39m.\u001b[39;49mrun(\n\u001b[1;32m    696\u001b[0m         agent_action\u001b[39m.\u001b[39;49mtool_input,\n\u001b[1;32m    697\u001b[0m         verbose\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mverbose,\n\u001b[1;32m    698\u001b[0m         color\u001b[39m=\u001b[39;49mcolor,\n\u001b[1;32m    699\u001b[0m         \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mtool_run_kwargs,\n\u001b[1;32m    700\u001b[0m     )\n\u001b[1;32m    701\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m    702\u001b[0m     tool_run_kwargs \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39magent\u001b[39m.\u001b[39mtool_run_logging_kwargs()\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/tools/base.py:110\u001b[0m, in \u001b[0;36mBaseTool.run\u001b[0;34m(self, tool_input, verbose, start_color, color, **kwargs)\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mrun\u001b[39m(\n\u001b[1;32m    102\u001b[0m     \u001b[39mself\u001b[39m,\n\u001b[1;32m    103\u001b[0m     tool_input: Union[\u001b[39mstr\u001b[39m, Dict],\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    107\u001b[0m     \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs: Any,\n\u001b[1;32m    108\u001b[0m ) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m \u001b[39mstr\u001b[39m:\n\u001b[1;32m    109\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"Run the tool.\"\"\"\u001b[39;00m\n\u001b[0;32m--> 110\u001b[0m     run_input \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_parse_input(tool_input)\n\u001b[1;32m    111\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mverbose \u001b[39mand\u001b[39;00m verbose \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[1;32m    112\u001b[0m         verbose_ \u001b[39m=\u001b[39m verbose\n",
-      "File \u001b[0;32m~/code/lc/lckg/langchain/tools/base.py:71\u001b[0m, in \u001b[0;36mBaseTool._parse_input\u001b[0;34m(self, tool_input)\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39missubclass\u001b[39m(input_args, BaseModel):\n\u001b[1;32m     70\u001b[0m     key_ \u001b[39m=\u001b[39m \u001b[39mnext\u001b[39m(\u001b[39miter\u001b[39m(input_args\u001b[39m.\u001b[39m__fields__\u001b[39m.\u001b[39mkeys()))\n\u001b[0;32m---> 71\u001b[0m     input_args\u001b[39m.\u001b[39;49mparse_obj({key_: tool_input})\n\u001b[1;32m     72\u001b[0m \u001b[39m# Passing as a positional argument is more straightforward for\u001b[39;00m\n\u001b[1;32m     73\u001b[0m \u001b[39m# backwards compatability\u001b[39;00m\n\u001b[1;32m     74\u001b[0m \u001b[39mreturn\u001b[39;00m tool_input\n",
-      "File \u001b[0;32m~/code/lc/lckg/.venv/lib/python3.11/site-packages/pydantic/main.py:526\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.parse_obj\u001b[0;34m()\u001b[0m\n",
-      "File \u001b[0;32m~/code/lc/lckg/.venv/lib/python3.11/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
-      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for ToolInputSchema\n__root__\n  Domain google is not on the approved list: ['langchain', 'wikipedia'] (type=value_error)"
-     ]
-    }
-   ],
-   "source": [
-    "agent.run(\"What's the main title on google.com?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/modules/chains/examples/llm_bash.ipynb

@@ -24,8 +24,8 @@
       "\n",
       "```bash\n",
       "echo \"Hello World\"\n",
-      "```\u001b[0m\n",
-      "Code: \u001b[33;1m\u001b[1;3m['echo \"Hello World\"']\u001b[0m\n",
+      "```\u001b[0m['```bash', 'echo \"Hello World\"', '```']\n",
+      "\n",
       "Answer: \u001b[33;1m\u001b[1;3mHello World\n",
       "\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
@@ -65,7 +65,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 28,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -93,7 +93,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 29,
    "metadata": {},
    "outputs": [
     {
@@ -107,8 +107,8 @@
       "\n",
       "```bash\n",
       "printf \"Hello World\\n\"\n",
-      "```\u001b[0m\n",
-      "Code: \u001b[33;1m\u001b[1;3m['printf \"Hello World\\\\n\"']\u001b[0m\n",
+      "```\u001b[0m['```bash', 'printf \"Hello World\\\\n\"', '```']\n",
+      "\n",
       "Answer: \u001b[33;1m\u001b[1;3mHello World\n",
       "\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
@@ -120,7 +120,7 @@
        "'Hello World\\n'"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 29,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -132,114 +132,6 @@
     "\n",
     "bash_chain.run(text)"
    ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Persistent Terminal\n",
-    "\n",
-    "By default, the chain will run in a separate subprocess each time it is called. This behavior can be changed by instantiating with a persistent bash process."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new LLMBashChain chain...\u001b[0m\n",
-      "List the current directory then move up a level.\u001b[32;1m\u001b[1;3m\n",
-      "\n",
-      "```bash\n",
-      "ls\n",
-      "cd ..\n",
-      "```\u001b[0m\n",
-      "Code: \u001b[33;1m\u001b[1;3m['ls', 'cd ..']\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3mapi.ipynb\t\t\tllm_summarization_checker.ipynb\n",
-      "constitutional_chain.ipynb\tmoderation.ipynb\n",
-      "llm_bash.ipynb\t\t\topenai_openapi.yaml\n",
-      "llm_checker.ipynb\t\topenapi.ipynb\n",
-      "llm_math.ipynb\t\t\tpal.ipynb\n",
-      "llm_requests.ipynb\t\tsqlite.ipynb\u001b[0m\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'api.ipynb\\t\\t\\tllm_summarization_checker.ipynb\\r\\nconstitutional_chain.ipynb\\tmoderation.ipynb\\r\\nllm_bash.ipynb\\t\\t\\topenai_openapi.yaml\\r\\nllm_checker.ipynb\\t\\topenapi.ipynb\\r\\nllm_math.ipynb\\t\\t\\tpal.ipynb\\r\\nllm_requests.ipynb\\t\\tsqlite.ipynb'"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.utilities.bash import BashProcess\n",
-    "\n",
-    "\n",
-    "persistent_process = BashProcess(persistent=True)\n",
-    "bash_chain = LLMBashChain.from_bash_process(llm=llm, bash_process=persistent_process, verbose=True)\n",
-    "\n",
-    "text = \"List the current directory then move up a level.\"\n",
-    "\n",
-    "bash_chain.run(text)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new LLMBashChain chain...\u001b[0m\n",
-      "List the current directory then move up a level.\u001b[32;1m\u001b[1;3m\n",
-      "\n",
-      "```bash\n",
-      "ls\n",
-      "cd ..\n",
-      "```\u001b[0m\n",
-      "Code: \u001b[33;1m\u001b[1;3m['ls', 'cd ..']\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3mexamples\t\tgetting_started.ipynb\tindex_examples\n",
-      "generic\t\t\thow_to_guides.rst\u001b[0m\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'examples\\t\\tgetting_started.ipynb\\tindex_examples\\r\\ngeneric\\t\\t\\thow_to_guides.rst'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Run the same command again and see that the state is maintained between calls\n",
-    "bash_chain.run(text)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -258,7 +150,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.16"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/modules/chains/examples/openapi.ipynb

@@ -144,160 +144,6 @@
       "\u001b[32;1m\u001b[1;3mYou are a helpful AI Assistant. Please provide JSON arguments to agentFunc() based on the user's instructions.\n",
       "\n",
       "API_SCHEMA: ```typescript\n",
-      "/* API for fetching Klarna product information */\n",
-      "type productsUsingGET = (_: {\n",
-      "/* A precise query that matches one very small category or product that needs to be searched for to find the products the user is looking for. If the user explicitly stated what they want, use that as a query. The query is as specific as possible to the product name or category mentioned by the user in its singular form, and don't contain any clarifiers like latest, newest, cheapest, budget, premium, expensive or similar. The query is always taken from the latest topic, if there is a new topic a new query is started. */\n",
-      "\t\tq: string,\n",
-      "/* number of products returned */\n",
-      "\t\tsize?: number,\n",
-      "/* (Optional) Minimum price in local currency for the product searched for. Either explicitly stated by the user or implicitly inferred from a combination of the user's request and the kind of product searched for. */\n",
-      "\t\tmin_price?: number,\n",
-      "/* (Optional) Maximum price in local currency for the product searched for. Either explicitly stated by the user or implicitly inferred from a combination of the user's request and the kind of product searched for. */\n",
-      "\t\tmax_price?: number,\n",
-      "}) => any;\n",
-      "```\n",
-      "\n",
-      "USER_INSTRUCTIONS: \"whats the most expensive shirt?\"\n",
-      "\n",
-      "Your arguments must be plain json provided in a markdown block:\n",
-      "\n",
-      "ARGS: ```json\n",
-      "{valid json conforming to API_SCHEMA}\n",
-      "```\n",
-      "\n",
-      "Example\n",
-      "-----\n",
-      "\n",
-      "ARGS: ```json\n",
-      "{\"foo\": \"bar\", \"baz\": {\"qux\": \"quux\"}}\n",
-      "```\n",
-      "\n",
-      "The block must be no more than 1 line long, and all arguments must be valid JSON. All string arguments must be wrapped in double quotes.\n",
-      "You MUST strictly comply to the types indicated by the provided schema, including all required args.\n",
-      "\n",
-      "If you don't have sufficient information to call the function due to things like requiring specific uuid's, you can reply with the following message:\n",
-      "\n",
-      "Message: ```text\n",
-      "Concise response requesting the additional information that would make calling the function successful.\n",
-      "```\n",
-      "\n",
-      "Begin\n",
-      "-----\n",
-      "ARGS:\n",
-      "\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m{\"q\": \"shirt\", \"size\": 1, \"max_price\": null}\u001b[0m\n",
-      "\u001b[36;1m\u001b[1;3m{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]}]}\u001b[0m\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new APIResponderChain chain...\u001b[0m\n",
-      "Prompt after formatting:\n",
-      "\u001b[32;1m\u001b[1;3mYou are a helpful AI assistant trained to answer user queries from API responses.\n",
-      "You attempted to call an API, which resulted in:\n",
-      "API_RESPONSE: {\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]}]}\n",
-      "\n",
-      "USER_COMMENT: \"whats the most expensive shirt?\"\n",
-      "\n",
-      "\n",
-      "If the API_RESPONSE can answer the USER_COMMENT respond with the following markdown json block:\n",
-      "Response: ```json\n",
-      "{\"response\": \"Human-understandable synthesis of the API_RESPONSE\"}\n",
-      "```\n",
-      "\n",
-      "Otherwise respond with the following markdown json block:\n",
-      "Response Error: ```json\n",
-      "{\"response\": \"What you did and a concise statement of the resulting error. If it can be easily fixed, provide a suggestion.\"}\n",
-      "```\n",
-      "\n",
-      "You MUST respond as a markdown json code block. The person you are responding to CANNOT see the API_RESPONSE, so if there is any relevant information there you must include it in your response.\n",
-      "\n",
-      "Begin:\n",
-      "---\n",
-      "\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\u001b[33;1m\u001b[1;3mThe most expensive shirt in the API response is the Burberry Check Poplin Shirt, which costs $360.00.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    }
-   ],
-   "source": [
-    "output = chain(\"whats the most expensive shirt?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "c000295e",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'request_args': '{\"q\": \"shirt\", \"size\": 1, \"max_price\": null}',\n",
-       " 'response_text': '{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]}]}'}"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# View intermediate steps\n",
-    "output[\"intermediate_steps\"]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "092bdb4d",
-   "metadata": {},
-   "source": [
-    "## Return raw response\n",
-    "\n",
-    "We can also run this chain without synthesizing the response. This will have the effect of just returning the raw API output."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "4dff3849",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = OpenAPIEndpointChain.from_api_operation(\n",
-    "    operation, \n",
-    "    llm, \n",
-    "    requests=Requests(), \n",
-    "    verbose=True,\n",
-    "    return_intermediate_steps=True, # Return request and response text\n",
-    "    raw_response=True # Return raw response\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "762499a9",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new OpenAPIEndpointChain chain...\u001b[0m\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new APIRequesterChain chain...\u001b[0m\n",
-      "Prompt after formatting:\n",
-      "\u001b[32;1m\u001b[1;3mYou are a helpful AI Assistant. Please provide JSON arguments to agentFunc() based on the user's instructions.\n",
-      "\n",
-      "API_SCHEMA: ```typescript\n",
-      "/* API for fetching Klarna product information */\n",
       "type productsUsingGET = (_: {\n",
       "/* A precise query that matches one very small category or product that needs to be searched for to find the products the user is looking for. If the user explicitly stated what they want, use that as a query. The query is as specific as possible to the product name or category mentioned by the user in its singular form, and don't contain any clarifiers like latest, newest, cheapest, budget, premium, expensive or similar. The query is always taken from the latest topic, if there is a new topic a new query is started. */\n",
       "\t\tq: string,\n",
@@ -341,7 +187,36 @@
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m{\"q\": \"shirt\", \"max_price\": null}\u001b[0m\n",
-      "\u001b[36;1m\u001b[1;3m{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Cotton Shirt - Beige\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl359/3200280807/Children-s-Clothing/Burberry-Vintage-Check-Cotton-Shirt-Beige/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$229.02\",\"attributes\":[\"Material:Cotton,Elastane\",\"Color:Beige\",\"Model:Boy\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Stretch Cotton Twill Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3202342515/Clothing/Burberry-Vintage-Check-Stretch-Cotton-Twill-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$309.99\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Woman\",\"Color:Beige\",\"Properties:Stretch\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Somerton Check Shirt - Camel\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201112728/Clothing/Burberry-Somerton-Check-Shirt-Camel/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$450.00\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Man\",\"Color:Beige\"]},{\"name\":\"Magellan Outdoors Laguna Madre Solid Short Sleeve Fishing Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3203102142/Clothing/Magellan-Outdoors-Laguna-Madre-Solid-Short-Sleeve-Fishing-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$19.99\",\"attributes\":[\"Material:Polyester,Nylon\",\"Target Group:Man\",\"Color:Red,Pink,White,Blue,Purple,Beige,Black,Green\",\"Properties:Pockets\",\"Pattern:Solid Color\"]}]}\u001b[0m\n",
+      "\u001b[36;1m\u001b[1;3m{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Cotton Shirt - Beige\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl359/3200280807/Children-s-Clothing/Burberry-Vintage-Check-Cotton-Shirt-Beige/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$196.30\",\"attributes\":[\"Material:Cotton,Elastane\",\"Color:Beige\",\"Model:Boy\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Somerton Check Shirt - Camel\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201112728/Clothing/Burberry-Somerton-Check-Shirt-Camel/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$450.00\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Man\",\"Color:Beige\"]},{\"name\":\"Calvin Klein Slim Fit Oxford Dress Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201839169/Clothing/Calvin-Klein-Slim-Fit-Oxford-Dress-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$24.91\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,White,Blue,Black\",\"Pattern:Solid Color\"]},{\"name\":\"Magellan Outdoors Laguna Madre Solid Short Sleeve Fishing Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3203102142/Clothing/Magellan-Outdoors-Laguna-Madre-Solid-Short-Sleeve-Fishing-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$19.99\",\"attributes\":[\"Material:Polyester,Nylon\",\"Target Group:Man\",\"Color:Red,Pink,White,Blue,Purple,Beige,Black,Green\",\"Properties:Pockets\",\"Pattern:Solid Color\"]}]}\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new APIResponderChain chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mYou are a helpful AI assistant trained to answer user queries from API responses.\n",
+      "You attempted to call an API, which resulted in:\n",
+      "API_RESPONSE: {\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Cotton Shirt - Beige\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl359/3200280807/Children-s-Clothing/Burberry-Vintage-Check-Cotton-Shirt-Beige/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$196.30\",\"attributes\":[\"Material:Cotton,Elastane\",\"Color:Beige\",\"Model:Boy\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Somerton Check Shirt - Camel\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201112728/Clothing/Burberry-Somerton-Check-Shirt-Camel/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$450.00\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Man\",\"Color:Beige\"]},{\"name\":\"Calvin Klein Slim Fit Oxford Dress Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201839169/Clothing/Calvin-Klein-Slim-Fit-Oxford-Dress-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$24.91\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,White,Blue,Black\",\"Pattern:Solid Color\"]},{\"name\":\"Magellan Outdoors Laguna Madre Solid Short Sleeve Fishing Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3203102142/Clothing/Magellan-Outdoors-Laguna-Madre-Solid-Short-Sleeve-Fishing-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$19.99\",\"attributes\":[\"Material:Polyester,Nylon\",\"Target Group:Man\",\"Color:Red,Pink,White,Blue,Purple,Beige,Black,Green\",\"Properties:Pockets\",\"Pattern:Solid Color\"]}]}\n",
+      "\n",
+      "USER_COMMENT: \"whats the most expensive shirt?\"\n",
+      "\n",
+      "\n",
+      "If the API_RESPONSE can answer the USER_COMMENT respond with the following markdown json block:\n",
+      "Response: ```json\n",
+      "{\"response\": \"Concise response to USER_COMMENT based on API_RESPONSE.\"}\n",
+      "```\n",
+      "\n",
+      "Otherwise respond with the following markdown json block:\n",
+      "Response Error: ```json\n",
+      "{\"response\": \"What you did and a concise statement of the resulting error. If it can be easily fixed, provide a suggestion.\"}\n",
+      "```\n",
+      "\n",
+      "You MUST respond as a markdown json code block.\n",
+      "\n",
+      "Begin:\n",
+      "---\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[33;1m\u001b[1;3mThe most expensive shirt in this list is the 'Burberry Somerton Check Shirt - Camel' which is priced at $450.00\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -353,26 +228,25 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "id": "4afc021a",
+   "execution_count": 8,
+   "id": "c000295e",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "{'instructions': 'whats the most expensive shirt?',\n",
-       " 'output': '{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Cotton Shirt - Beige\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl359/3200280807/Children-s-Clothing/Burberry-Vintage-Check-Cotton-Shirt-Beige/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$229.02\",\"attributes\":[\"Material:Cotton,Elastane\",\"Color:Beige\",\"Model:Boy\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Stretch Cotton Twill Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3202342515/Clothing/Burberry-Vintage-Check-Stretch-Cotton-Twill-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$309.99\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Woman\",\"Color:Beige\",\"Properties:Stretch\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Somerton Check Shirt - Camel\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201112728/Clothing/Burberry-Somerton-Check-Shirt-Camel/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$450.00\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Man\",\"Color:Beige\"]},{\"name\":\"Magellan Outdoors Laguna Madre Solid Short Sleeve Fishing Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3203102142/Clothing/Magellan-Outdoors-Laguna-Madre-Solid-Short-Sleeve-Fishing-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$19.99\",\"attributes\":[\"Material:Polyester,Nylon\",\"Target Group:Man\",\"Color:Red,Pink,White,Blue,Purple,Beige,Black,Green\",\"Properties:Pockets\",\"Pattern:Solid Color\"]}]}',\n",
-       " 'intermediate_steps': {'request_args': '{\"q\": \"shirt\", \"max_price\": null}',\n",
-       "  'response_text': '{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Cotton Shirt - Beige\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl359/3200280807/Children-s-Clothing/Burberry-Vintage-Check-Cotton-Shirt-Beige/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$229.02\",\"attributes\":[\"Material:Cotton,Elastane\",\"Color:Beige\",\"Model:Boy\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Stretch Cotton Twill Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3202342515/Clothing/Burberry-Vintage-Check-Stretch-Cotton-Twill-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$309.99\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Woman\",\"Color:Beige\",\"Properties:Stretch\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Somerton Check Shirt - Camel\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201112728/Clothing/Burberry-Somerton-Check-Shirt-Camel/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$450.00\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Man\",\"Color:Beige\"]},{\"name\":\"Magellan Outdoors Laguna Madre Solid Short Sleeve Fishing Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3203102142/Clothing/Magellan-Outdoors-Laguna-Madre-Solid-Short-Sleeve-Fishing-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$19.99\",\"attributes\":[\"Material:Polyester,Nylon\",\"Target Group:Man\",\"Color:Red,Pink,White,Blue,Purple,Beige,Black,Green\",\"Properties:Pockets\",\"Pattern:Solid Color\"]}]}'}}"
+       "['{\"q\": \"shirt\", \"max_price\": null}',\n",
+       " '{\"products\":[{\"name\":\"Burberry Check Poplin Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201810981/Clothing/Burberry-Check-Poplin-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$360.00\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,Blue,Beige\",\"Properties:Pockets\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Vintage Check Cotton Shirt - Beige\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl359/3200280807/Children-s-Clothing/Burberry-Vintage-Check-Cotton-Shirt-Beige/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$196.30\",\"attributes\":[\"Material:Cotton,Elastane\",\"Color:Beige\",\"Model:Boy\",\"Pattern:Checkered\"]},{\"name\":\"Burberry Somerton Check Shirt - Camel\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201112728/Clothing/Burberry-Somerton-Check-Shirt-Camel/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$450.00\",\"attributes\":[\"Material:Elastane/Lycra/Spandex,Cotton\",\"Target Group:Man\",\"Color:Beige\"]},{\"name\":\"Calvin Klein Slim Fit Oxford Dress Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3201839169/Clothing/Calvin-Klein-Slim-Fit-Oxford-Dress-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$24.91\",\"attributes\":[\"Material:Cotton\",\"Target Group:Man\",\"Color:Gray,White,Blue,Black\",\"Pattern:Solid Color\"]},{\"name\":\"Magellan Outdoors Laguna Madre Solid Short Sleeve Fishing Shirt\",\"url\":\"https://www.klarna.com/us/shopping/pl/cl10001/3203102142/Clothing/Magellan-Outdoors-Laguna-Madre-Solid-Short-Sleeve-Fishing-Shirt/?utm_source=openai&ref-site=openai_plugin\",\"price\":\"$19.99\",\"attributes\":[\"Material:Polyester,Nylon\",\"Target Group:Man\",\"Color:Red,Pink,White,Blue,Purple,Beige,Black,Green\",\"Properties:Pockets\",\"Pattern:Solid Color\"]}]}']"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "output"
+    "# View intermediate steps\n",
+    "output[\"intermediate_steps\"]"
    ]
   },
   {
@@ -574,7 +448,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,

docs/modules/chains/examples/sqlite.ipynb

@@ -9,9 +9,9 @@
     }
    },
    "source": [
-    "# SQL Chain example\n",
+    "# SQLite example\n",
     "\n",
-    "This example demonstrates the use of the `SQLDatabaseChain` for answering questions over a database."
+    "This example showcases hooking up an LLM to answer questions over a database."
    ]
   },
   {
@@ -23,10 +23,8 @@
     }
    },
    "source": [
-    "Under the hood, LangChain uses SQLAlchemy to connect to SQL databases. The `SQLDatabaseChain` can therefore be used with any SQL dialect supported by SQLAlchemy, such as MS SQL, MySQL, MariaDB, PostgreSQL, Oracle SQL, and SQLite. Please refer to the SQLAlchemy documentation for more information about requirements for connecting to your database. For example, a connection to MySQL requires an appropriate connector such as PyMySQL. A URI for a MySQL connection might look like: `mysql+pymysql://user:pass@some_mysql_db_address/db_name`\n",
-    "\n",
-    "This demonstration uses SQLite and the example Chinook database.\n",
-    "To set it up, follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the `.db` file in a notebooks folder at the root of this repository."
+    "This uses the example Chinook database.\n",
+    "To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the `.db` file in a notebooks folder at the root of this repository."
    ]
   },
   {
@@ -681,7 +679,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/chains/generic/custom_chain.ipynb

@@ -1,188 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "593f7553-7038-498e-96d4-8255e5ce34f0",
-   "metadata": {},
-   "source": [
-    "# Creating a custom Chain\n",
-    "\n",
-    "To implement your own custom chain you can subclass `BaseChain` and implement the following methods:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "c19c736e-ca74-4726-bb77-0a849bcc2960",
-   "metadata": {
-    "tags": [],
-    "vscode": {
-     "languageId": "python"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "from __future__ import annotations\n",
-    "\n",
-    "from typing import Any, Dict, List, Optional\n",
-    "\n",
-    "from pydantic import Extra\n",
-    "\n",
-    "from langchain.base_language import BaseLanguageModel\n",
-    "from langchain.callbacks.manager import (\n",
-    "    AsyncCallbackManagerForChainRun,\n",
-    "    CallbackManagerForChainRun,\n",
-    ")\n",
-    "from langchain.chains.base import Chain\n",
-    "from langchain.prompts.base import BasePromptTemplate\n",
-    "\n",
-    "\n",
-    "class MyCustomChain(Chain):\n",
-    "    \"\"\"\n",
-    "    An example of a custom chain.\n",
-    "    \"\"\"\n",
-    "\n",
-    "    prompt: BasePromptTemplate\n",
-    "    \"\"\"Prompt object to use.\"\"\"\n",
-    "    llm: BaseLanguageModel\n",
-    "    output_key: str = \"text\"  #: :meta private:\n",
-    "\n",
-    "    class Config:\n",
-    "        \"\"\"Configuration for this pydantic object.\"\"\"\n",
-    "\n",
-    "        extra = Extra.forbid\n",
-    "        arbitrary_types_allowed = True\n",
-    "\n",
-    "    @property\n",
-    "    def input_keys(self) -> List[str]:\n",
-    "        \"\"\"Will be whatever keys the prompt expects.\n",
-    "\n",
-    "        :meta private:\n",
-    "        \"\"\"\n",
-    "        return self.prompt.input_variables\n",
-    "\n",
-    "    @property\n",
-    "    def output_keys(self) -> List[str]:\n",
-    "        \"\"\"Will always return text key.\n",
-    "\n",
-    "        :meta private:\n",
-    "        \"\"\"\n",
-    "        return [self.output_key]\n",
-    "\n",
-    "    def _call(\n",
-    "        self,\n",
-    "        inputs: Dict[str, Any],\n",
-    "        run_manager: Optional[CallbackManagerForChainRun] = None,\n",
-    "    ) -> Dict[str, str]:\n",
-    "        # Your custom chain logic goes here\n",
-    "        # This is just an example that mimics LLMChain\n",
-    "        prompt_value = self.prompt.format_prompt(**inputs)\n",
-    "        \n",
-    "        # Whenever you call a language model, or another chain, you should pass\n",
-    "        # a callback manager to it. This allows the inner run to be tracked by\n",
-    "        # any callbacks that are registered on the outer run.\n",
-    "        # You can always obtain a callback manager for this by calling\n",
-    "        # `run_manager.get_child()` as shown below.\n",
-    "        response = self.llm.generate_prompt(\n",
-    "            [prompt_value],\n",
-    "            callbacks=run_manager.get_child() if run_manager else None\n",
-    "        )\n",
-    "\n",
-    "        # If you want to log something about this run, you can do so by calling\n",
-    "        # methods on the `run_manager`, as shown below. This will trigger any\n",
-    "        # callbacks that are registered for that event.\n",
-    "        if run_manager:\n",
-    "            run_manager.on_text(\"Log something about this run\")\n",
-    "        \n",
-    "        return {self.output_key: response.generations[0][0].text}\n",
-    "\n",
-    "    async def _acall(\n",
-    "        self,\n",
-    "        inputs: Dict[str, Any],\n",
-    "        run_manager: Optional[AsyncCallbackManagerForChainRun] = None,\n",
-    "    ) -> Dict[str, str]:\n",
-    "        # Your custom chain logic goes here\n",
-    "        # This is just an example that mimics LLMChain\n",
-    "        prompt_value = self.prompt.format_prompt(**inputs)\n",
-    "        \n",
-    "        # Whenever you call a language model, or another chain, you should pass\n",
-    "        # a callback manager to it. This allows the inner run to be tracked by\n",
-    "        # any callbacks that are registered on the outer run.\n",
-    "        # You can always obtain a callback manager for this by calling\n",
-    "        # `run_manager.get_child()` as shown below.\n",
-    "        response = await self.llm.agenerate_prompt(\n",
-    "            [prompt_value],\n",
-    "            callbacks=run_manager.get_child() if run_manager else None\n",
-    "        )\n",
-    "\n",
-    "        # If you want to log something about this run, you can do so by calling\n",
-    "        # methods on the `run_manager`, as shown below. This will trigger any\n",
-    "        # callbacks that are registered for that event.\n",
-    "        if run_manager:\n",
-    "            await run_manager.on_text(\"Log something about this run\")\n",
-    "        \n",
-    "        return {self.output_key: response.generations[0][0].text}\n",
-    "\n",
-    "    @property\n",
-    "    def _chain_type(self) -> str:\n",
-    "        return \"my_custom_chain\"\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "18361f89",
-   "metadata": {
-    "vscode": {
-     "languageId": "python"
-    }
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new MyCustomChain chain...\u001b[0m\n",
-      "Log something about this run\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'Why did the callback function feel lonely? Because it was always waiting for someone to call it back!'"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.callbacks.stdout import StdOutCallbackHandler\n",
-    "from langchain.chat_models.openai import ChatOpenAI\n",
-    "from langchain.prompts.prompt import PromptTemplate\n",
-    "\n",
-    "\n",
-    "chain = MyCustomChain(\n",
-    "    prompt=PromptTemplate.from_template('tell us a joke about {topic}'),\n",
-    "    llm=ChatOpenAI()\n",
-    ")\n",
-    "\n",
-    "chain.run({'topic': 'callbacks'}, callbacks=[StdOutCallbackHandler()])"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/chains/generic/llm_chain.ipynb

@@ -2,90 +2,59 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "da7d0df7-f07c-462f-bd46-d0426f11f311",
+   "id": "d8a5c5d4",
    "metadata": {},
    "source": [
-    "## LLM Chain"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3a55e9a1-becf-4357-889e-f365d23362ff",
-   "metadata": {},
-   "source": [
-    "`LLMChain` is perhaps one of the most popular ways of querying an LLM object. It formats the prompt template using the input key values provided (and also memory key values, if available), passes the formatted string to LLM and returns the LLM output. Below we show additional functionalities of `LLMChain` class."
+    "# LLM Chain\n",
+    "\n",
+    "This notebook showcases a simple LLM chain."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "0e720e34-a0f0-4f1a-9732-43bc1460053a",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'product': 'colorful socks', 'text': '\\n\\nSocktastic!'}"
-      ]
-     },
-     "execution_count": 1,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "id": "835e6978",
+   "metadata": {},
+   "outputs": [],
    "source": [
-    "from langchain import PromptTemplate, OpenAI, LLMChain\n",
+    "from langchain import PromptTemplate, OpenAI, LLMChain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "06bcb078",
+   "metadata": {},
+   "source": [
+    "## Single Input\n",
     "\n",
-    "prompt_template = \"What is a good name for a company that makes {product}?\"\n",
-    "\n",
-    "llm = OpenAI(temperature=0)\n",
-    "llm_chain = LLMChain(\n",
-    "    llm=llm,\n",
-    "    prompt=PromptTemplate.from_template(prompt_template)\n",
-    ")\n",
-    "llm_chain(\"colorful socks\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "94304332-6398-4280-a61e-005ba29b5e1e",
-   "metadata": {},
-   "source": [
-    "## Additional ways of running LLM Chain"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4e51981f-cde9-4c05-99e1-446c27994e99",
-   "metadata": {},
-   "source": [
-    "Aside from `__call__` and `run` methods shared by all `Chain` object (see [Getting Started](../getting_started.ipynb) to learn more), `LLMChain` offers a few more ways of calling the chain logic:"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c08d2356-412d-4327-b8a0-233dcc443e30",
-   "metadata": {},
-   "source": [
-    "- `apply` allows you run the chain against a list of inputs:"
+    "First, lets go over an example using a single input"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "cf519eb6-2358-4db7-a28a-27433435181e",
-   "metadata": {
-    "tags": []
-   },
+   "id": "51a54c4d",
+   "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001B[1m> Entering new LLMChain chain...\u001B[0m\n",
+      "Prompt after formatting:\n",
+      "\u001B[32;1m\u001B[1;3mQuestion: What NFL team won the Super Bowl in the year Justin Beiber was born?\n",
+      "\n",
+      "Answer: Let's think step by step.\u001B[0m\n",
+      "\n",
+      "\u001B[1m> Finished LLMChain chain.\u001B[0m\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
-       "[{'text': '\\n\\nSocktastic!'},\n",
-       " {'text': '\\n\\nTechCore Solutions.'},\n",
-       " {'text': '\\n\\nFootwear Factory.'}]"
+       "' Justin Bieber was born in 1994, so the NFL team that won the Super Bowl in 1994 was the Dallas Cowboys.'"
       ]
      },
      "execution_count": 2,
@@ -94,37 +63,49 @@
     }
    ],
    "source": [
-    "input_list = [\n",
-    "    {\"product\": \"socks\"},\n",
-    "    {\"product\": \"computer\"},\n",
-    "    {\"product\": \"shoes\"}\n",
-    "]\n",
+    "template = \"\"\"Question: {question}\n",
     "\n",
-    "llm_chain.apply(input_list)"
+    "Answer: Let's think step by step.\"\"\"\n",
+    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=OpenAI(temperature=0), verbose=True)\n",
+    "\n",
+    "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
+    "\n",
+    "llm_chain.predict(question=question)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "add442fb-baf6-40d9-ae8e-4ac1d8251ad0",
-   "metadata": {
-    "tags": []
-   },
+   "id": "79c3ec4d",
+   "metadata": {},
    "source": [
-    "- `generate` is similar to `apply`, except it return an `LLMResult` instead of string. `LLMResult` often contains useful generation such as token usages and finish reason."
+    "## Multiple Inputs\n",
+    "Now lets go over an example using multiple inputs."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "85cbff83-a5cc-40b7-823c-47274ae4117d",
-   "metadata": {
-    "tags": []
-   },
+   "id": "03dd6918",
+   "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001B[1m> Entering new LLMChain chain...\u001B[0m\n",
+      "Prompt after formatting:\n",
+      "\u001B[32;1m\u001B[1;3mWrite a sad poem about ducks.\u001B[0m\n",
+      "\n",
+      "\u001B[1m> Finished LLMChain chain.\u001B[0m\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
-       "LLMResult(generations=[[Generation(text='\\n\\nSocktastic!', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\nTechCore Solutions.', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\nFootwear Factory.', generation_info={'finish_reason': 'stop', 'logprobs': None})]], llm_output={'token_usage': {'prompt_tokens': 36, 'total_tokens': 55, 'completion_tokens': 19}, 'model_name': 'text-davinci-003'})"
+       "\"\\n\\nThe ducks swim in the pond,\\nTheir feathers so soft and warm,\\nBut they can't help but feel so forlorn.\\n\\nTheir quacks echo in the air,\\nBut no one is there to hear,\\nFor they have no one to share.\\n\\nThe ducks paddle around in circles,\\nTheir heads hung low in despair,\\nFor they have no one to care.\\n\\nThe ducks look up to the sky,\\nBut no one is there to see,\\nFor they have no one to be.\\n\\nThe ducks drift away in the night,\\nTheir hearts filled with sorrow and pain,\\nFor they have no one to gain.\""
       ]
      },
      "execution_count": 3,
@@ -133,201 +114,46 @@
     }
    ],
    "source": [
-    "llm_chain.generate(input_list)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a178173b-b183-432a-a517-250fe3191173",
-   "metadata": {},
-   "source": [
-    "- `predict` is similar to `run` method except in 2 ways:\n",
-    "    - Input key is specified as keyword argument instead of a Python dict\n",
-    "    - It supports multiple input keys."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "787d9f55-b080-4123-bed2-0598a9cb0466",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'\\n\\nSocktastic!'"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Single input example\n",
-    "llm_chain.predict(product=\"colorful socks\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "092a769f-9661-42a0-9da1-19d09ccbc4a7",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'\\n\\nQ: What did the duck say when his friend died?\\nA: Quack, quack, goodbye.'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Multiple inputs example\n",
-    "\n",
-    "template = \"\"\"Tell me a {adjective} joke about {subject}.\"\"\"\n",
+    "template = \"\"\"Write a {adjective} poem about {subject}.\"\"\"\n",
     "prompt = PromptTemplate(template=template, input_variables=[\"adjective\", \"subject\"])\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=OpenAI(temperature=0))\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=OpenAI(temperature=0), verbose=True)\n",
     "\n",
     "llm_chain.predict(adjective=\"sad\", subject=\"ducks\")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4b72ad22-0a5d-4ca7-9e3f-8c46dc17f722",
-   "metadata": {},
-   "source": [
-    "## Parsing the outputs"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "85a77662-d028-4048-be4b-aa496e2dde22",
-   "metadata": {},
-   "source": [
-    "By default, `LLMChain` does not parse the output even if the underlying `prompt` object has an output parser. If you would like to apply that output parser on the LLM output, use `predict_and_parse` instead of `predict` and `apply_and_parse` instead of `apply`. "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b83977f1-847c-45de-b840-f1aff6725f83",
-   "metadata": {},
-   "source": [
-    "With `predict`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "5feb5177-c20b-4909-890b-a64d7e551f55",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'\\n\\nRed, orange, yellow, green, blue, indigo, violet'"
-      ]
-     },
-     "execution_count": 24,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.output_parsers import CommaSeparatedListOutputParser\n",
-    "\n",
-    "output_parser = CommaSeparatedListOutputParser()\n",
-    "template = \"\"\"List all the colors in a rainbow\"\"\"\n",
-    "prompt = PromptTemplate(template=template, input_variables=[], output_parser=output_parser)\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
-    "\n",
-    "llm_chain.predict()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7b931615-804b-4f34-8086-7bbc2f96b3b2",
-   "metadata": {},
-   "source": [
-    "With `predict_and_parser`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "43a374cd-a179-43e5-9aa0-62f3cbdf510d",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['Red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet']"
-      ]
-     },
-     "execution_count": 25,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "llm_chain.predict_and_parse()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8176f619-4e5c-4a02-91ba-e96ebe2aabda",
-   "metadata": {},
-   "source": [
-    "## Initialize from string"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9813ac87-e118-413b-b448-2fefdf2319b8",
+   "id": "672f59d4",
    "metadata": {},
    "source": [
+    "## From string\n",
     "You can also construct an LLMChain from a string template directly."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
-   "id": "ca88ccb1-974e-41c1-81ce-753e3f1234fa",
-   "metadata": {
-    "tags": []
-   },
+   "execution_count": 3,
+   "id": "f8bc262e",
+   "metadata": {},
    "outputs": [],
    "source": [
-    "template = \"\"\"Tell me a {adjective} joke about {subject}.\"\"\"\n",
-    "llm_chain = LLMChain.from_string(llm=llm, template=template)"
+    "template = \"\"\"Write a {adjective} poem about {subject}.\"\"\"\n",
+    "llm_chain = LLMChain.from_string(llm=OpenAI(temperature=0), template=template)\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
-   "id": "4703d1bc-f4fc-44bc-9ea1-b4498835833d",
-   "metadata": {
-    "tags": []
-   },
+   "execution_count": 4,
+   "id": "cb164a76",
+   "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "'\\n\\nQ: What did the duck say when his friend died?\\nA: Quack, quack, goodbye.'"
+       "\"\\n\\nThe ducks swim in the pond,\\nTheir feathers so soft and warm,\\nBut they can't help but feel so forlorn.\\n\\nTheir quacks echo in the air,\\nBut no one is there to hear,\\nFor they have no one to share.\\n\\nThe ducks paddle around in circles,\\nTheir heads hung low in despair,\\nFor they have no one to care.\\n\\nThe ducks look up to the sky,\\nBut no one is there to see,\\nFor they have no one to be.\\n\\nThe ducks drift away in the night,\\nTheir hearts filled with sorrow and pain,\\nFor they have no one to gain.\""
       ]
      },
-     "execution_count": 18,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -335,6 +161,14 @@
    "source": [
     "llm_chain.predict(adjective=\"sad\", subject=\"ducks\")"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9f0adbc7",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -353,7 +187,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.10.9"
   }
  },
  "nbformat": 4,

docs/modules/chains/getting_started.ipynb

@@ -22,11 +22,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Quick start: Using `LLMChain`\n",
+    "## Query an LLM with the `LLMChain`\n",
     "\n",
     "The `LLMChain` is a simple chain that takes in a prompt template, formats it with the user input and returns the response from an LLM.\n",
     "\n",
-    "\n",
     "To use the `LLMChain`, first create a prompt template."
    ]
   },
@@ -68,7 +67,7 @@
      "text": [
       "\n",
       "\n",
-      "SockSplash!\n"
+      "Rainbow Socks Co.\n"
      ]
     }
    ],
@@ -89,7 +88,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 5,
    "metadata": {
     "tags": []
    },
@@ -98,7 +97,9 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Rainbow Sox Co.\n"
+      "\n",
+      "\n",
+      "Rainbow Threads\n"
      ]
     }
    ],
@@ -124,253 +125,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Different ways of calling chains\n",
-    "\n",
-    "All classes inherited from `Chain` offer a few ways of running chain logic. The most direct one is by using `__call__`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'adjective': 'corny',\n",
-       " 'text': 'Why did the tomato turn red? Because it saw the salad dressing!'}"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat = ChatOpenAI(temperature=0)\n",
-    "prompt_template = \"Tell me a {adjective} joke\"\n",
-    "llm_chain = LLMChain(\n",
-    "    llm=chat,\n",
-    "    prompt=PromptTemplate.from_template(prompt_template)\n",
-    ")\n",
-    "\n",
-    "llm_chain(inputs={\"adjective\":\"corny\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "By default, `__call__` returns both the input and output key values. You can configure it to only return output key values by setting `return_only_outputs` to `True`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'text': 'Why did the tomato turn red? Because it saw the salad dressing!'}"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "llm_chain(\"corny\", return_only_outputs=True)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "If the `Chain` only outputs one output key (i.e. only has one element in its `output_keys`), you can  use `run` method. Note that `run` outputs a string instead of a dictionary."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['text']"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# llm_chain only has one output key, so we can use run\n",
-    "llm_chain.output_keys"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Why did the tomato turn red? Because it saw the salad dressing!'"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "llm_chain.run({\"adjective\":\"corny\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "In the case of one input key, you can input the string directly without specifying the input mapping."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'adjective': 'corny',\n",
-       " 'text': 'Why did the tomato turn red? Because it saw the salad dressing!'}"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# These two are equivalent\n",
-    "llm_chain.run({\"adjective\":\"corny\"})\n",
-    "llm_chain.run(\"corny\")\n",
-    "\n",
-    "# These two are also equivalent\n",
-    "llm_chain(\"corny\")\n",
-    "llm_chain({\"adjective\":\"corny\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Tips: You can easily integrate a `Chain` object as a `Tool` in your `Agent` via its `run` method. See an example [here](../agents/tools/custom_tools.ipynb)."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Add memory to chains\n",
-    "\n",
-    "`Chain` supports taking a `BaseMemory` object as its `memory` argument, allowing `Chain` object to persist data across multiple calls. In other words, it makes `Chain` a stateful object."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'The next four colors of a rainbow are green, blue, indigo, and violet.'"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.chains import ConversationChain\n",
-    "from langchain.memory import ConversationBufferMemory\n",
-    "\n",
-    "conversation = ConversationChain(\n",
-    "    llm=chat,\n",
-    "    memory=ConversationBufferMemory()\n",
-    ")\n",
-    "\n",
-    "conversation.run(\"Answer briefly. What are the first 3 colors of a rainbow?\")\n",
-    "# -> The first three colors of a rainbow are red, orange, and yellow.\n",
-    "conversation.run(\"And the next 4?\")\n",
-    "# -> The next four colors of a rainbow are green, blue, indigo, and violet."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Essentially, `BaseMemory` defines an interface of how `langchain` stores memory. It allows reading of stored data through `load_memory_variables` method and storing new data through `save_context` method. You can learn more about it in [Memory](../memory/getting_started.ipynb) section."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Debug Chain\n",
-    "\n",
-    "It can be hard to debug `Chain` object solely from its output as most `Chain` objects involve a fair amount of input prompt preprocessing and LLM output post-processing. Setting `verbose` to `True` will print out some internal states of the `Chain` object while it is being ran."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new ConversationChain chain...\u001b[0m\n",
-      "Prompt after formatting:\n",
-      "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
-      "\n",
-      "Current conversation:\n",
-      "\n",
-      "Human: What is ChatGPT?\n",
-      "AI:\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'ChatGPT is an AI language model developed by OpenAI. It is based on the GPT-3 architecture and is capable of generating human-like responses to text prompts. ChatGPT has been trained on a massive amount of text data and can understand and respond to a wide range of topics. It is often used for chatbots, virtual assistants, and other conversational AI applications.'"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "conversation = ConversationChain(\n",
-    "    llm=chat,\n",
-    "    memory=ConversationBufferMemory(),\n",
-    "    verbose=True\n",
-    ")\n",
-    "conversation.run(\"What is ChatGPT?\")"
+    "This is one of the simpler types of chains, but understanding how it works will set you up well for working with more complex chains."
    ]
   },
   {
@@ -388,7 +143,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -408,7 +163,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [
     {
@@ -418,15 +173,17 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new SimpleSequentialChain chain...\u001b[0m\n",
-      "\u001b[36;1m\u001b[1;3mRainbow Socks Co.\u001b[0m\n",
+      "\u001b[36;1m\u001b[1;3m\n",
+      "\n",
+      "Cheerful Toes.\u001b[0m\n",
       "\u001b[33;1m\u001b[1;3m\n",
       "\n",
-      "\"Step into Color with Rainbow Socks!\"\u001b[0m\n",
+      "\"Spread smiles from your toes!\"\u001b[0m\n",
       "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[1m> Finished SimpleSequentialChain chain.\u001b[0m\n",
       "\n",
       "\n",
-      "\"Step into Color with Rainbow Socks!\"\n"
+      "\"Spread smiles from your toes!\"\n"
      ]
     }
    ],
@@ -457,7 +214,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -491,13 +248,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now, we can try running the chain that we called.\n",
-    "\n"
+    "Now, we can try running the chain that we called."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [
     {
@@ -507,9 +263,9 @@
       "Concatenated output:\n",
       "\n",
       "\n",
-      "Socktastic Colors.\n",
+      "Rainbow Socks Co.\n",
       "\n",
-      "\"Put Some Color in Your Step!\"\n"
+      "\"Step Into Colorful Comfort!\"\n"
      ]
     }
    ],
@@ -555,7 +311,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.16"
+   "version": "3.10.9"
   },
   "vscode": {
    "interpreter": {

docs/modules/chains/index_examples/chat_vector_db.ipynb

@@ -12,7 +12,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 3,
    "id": "70c4e529",
    "metadata": {
     "tags": []
@@ -36,7 +36,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 4,
    "id": "01c46e92",
    "metadata": {
     "tags": []
@@ -58,7 +58,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 5,
    "id": "433363a5",
    "metadata": {
     "tags": []
@@ -81,17 +81,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 6,
    "id": "a8930cf7",
    "metadata": {
     "tags": []
    },
    "outputs": [
     {
-     "name": "stderr",
+     "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Using embedded DuckDB without persistence: data will be transient\n"
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
      ]
     }
    ],
@@ -103,25 +104,6 @@
     "vectorstore = Chroma.from_documents(documents, embeddings)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "898b574b",
-   "metadata": {},
-   "source": [
-    "We can now create a memory object, which is neccessary to track the inputs/outputs and hold a conversation."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "af803fee",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.memory import ConversationBufferMemory\n",
-    "memory = ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True)"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "3c96b118",
@@ -132,96 +114,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 7,
    "id": "7b4110f3",
    "metadata": {
     "tags": []
    },
    "outputs": [],
-   "source": [
-    "qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever(), memory=memory)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "e8ce4fe9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
-    "result = qa({\"question\": query})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "id": "4c79862b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\""
-      ]
-     },
-     "execution_count": 23,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "result[\"answer\"]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "c697d9d1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "query = \"Did he mention who she suceeded\"\n",
-    "result = qa({\"question\": query})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "ba0678f3",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' Ketanji Brown Jackson succeeded Justice Stephen Breyer on the United States Supreme Court.'"
-      ]
-     },
-     "execution_count": 25,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "result['answer']"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "84426220",
-   "metadata": {},
-   "source": [
-    "## Pass in chat history\n",
-    "\n",
-    "In the above example, we used a Memory object to track chat history. We can also just pass it in explicitly. In order to do this, we need to initialize a chain without any memory object."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "676b8a36",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0), vectorstore.as_retriever())"
    ]
@@ -236,7 +134,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 8,
    "id": "7fe3e730",
    "metadata": {
     "tags": []
@@ -250,7 +148,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 9,
    "id": "bfff9cc8",
    "metadata": {
     "tags": []
@@ -262,7 +160,7 @@
        "\" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\""
       ]
      },
-     "execution_count": 7,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -281,7 +179,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 10,
    "id": "00b4cf00",
    "metadata": {
     "tags": []
@@ -295,7 +193,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 11,
    "id": "f01828d1",
    "metadata": {
     "tags": []
@@ -307,7 +205,7 @@
        "' Ketanji Brown Jackson succeeded Justice Stephen Breyer on the United States Supreme Court.'"
       ]
      },
-     "execution_count": 9,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -589,6 +487,7 @@
    "outputs": [],
    "source": [
     "from langchain.chains.llm import LLMChain\n",
+    "from langchain.callbacks.base import CallbackManager\n",
     "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
     "from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT, QA_PROMPT\n",
     "from langchain.chains.question_answering import load_qa_chain\n",
@@ -596,7 +495,7 @@
     "# Construct a ConversationalRetrievalChain with a streaming llm for combine docs\n",
     "# and a separate, non-streaming llm for question generation\n",
     "llm = OpenAI(temperature=0)\n",
-    "streaming_llm = OpenAI(streaming=True, callbacks=[StreamingStdOutCallbackHandler()], temperature=0)\n",
+    "streaming_llm = OpenAI(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)\n",
     "\n",
     "question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)\n",
     "doc_chain = load_qa_chain(streaming_llm, chain_type=\"stuff\", prompt=QA_PROMPT)\n",
@@ -737,7 +636,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.9"
   }
  },
  "nbformat": 4,

docs/modules/chains/index_examples/qa_with_sources.ipynb

@@ -267,7 +267,7 @@
    "source": [
     "**Intermediate Steps**\n",
     "\n",
-    "We can also return the intermediate steps for `map_reduce` chains, should we want to inspect them. This is done with the `return_intermediate_steps` variable."
+    "We can also return the intermediate steps for `map_reduce` chains, should we want to inspect them. This is done with the `return_map_steps` variable."
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/arxiv.ipynb

@@ -1,177 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "bda1f3f5",
-   "metadata": {},
-   "source": [
-    "# Arxiv\n",
-    "\n",
-    "[arXiv](https://arxiv.org/) is an open-access archive for 2 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.\n",
-    "\n",
-    "This notebook shows how to load scientific articles from `Arxiv.org` into a document format that we can use downstream."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1b7a1eef-7bf7-4e7d-8bfc-c4e27c9488cb",
-   "metadata": {},
-   "source": [
-    "## Installation"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2abd5578-aa3d-46b9-99af-8b262f0b3df8",
-   "metadata": {},
-   "source": [
-    "First, you need to install `arxiv` python package."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b674aaea-ed3a-4541-8414-260a8f67f623",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "!pip install arxiv"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "094b5f13-7e54-4354-9d83-26d6926ecaa0",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "Second, you need to install `PyMuPDF` python package which transform PDF files from the `arxiv.org` site into the text fromat."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "7cd91121-2e96-43ba-af50-319853695f86",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "!pip install pymupdf"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "95f05e1c-195e-4e2b-ae8e-8d6637f15be6",
-   "metadata": {},
-   "source": [
-    "## Examples"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e29b954c-1407-4797-ae21-6ba8937156be",
-   "metadata": {},
-   "source": [
-    "`ArxivLoader` has these arguments:\n",
-    "- `query`: free text which used to find documents in the Arxiv\n",
-    "- optional `load_max_docs`: default=100. Use it to limit number of downloaded documents. It takes time to download all 100 documents, so use a small number for experiments.\n",
-    "- optional `load_all_available_meta`: default=False. By defaul only the most important fields downloaded: `Published` (date when document was published/last updated), `Title`, `Authors`, `Summary`. If True, other fields also downloaded."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9bfd5e46",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders.base import Document\n",
-    "from langchain.document_loaders import ArxivLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "700e4ef2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs = ArxivLoader(query=\"1605.08386\", load_max_docs=2).load()\n",
-    "len(docs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "8977bac0-0042-4f23-9754-247dbd32439b",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'Published': '2016-05-26',\n",
-       " 'Title': 'Heat-bath random walks with Markov bases',\n",
-       " 'Authors': 'Caprice Stanley, Tobias Windisch',\n",
-       " 'Summary': 'Graphs on lattice points are studied whose edges come from a finite set of\\nallowed moves of arbitrary length. We show that the diameter of these graphs on\\nfibers of a fixed integer matrix can be bounded from above by a constant. We\\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\\nalso state explicit conditions on the set of moves so that the heat-bath random\\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\\ndimension.'}"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "doc[0].metadata  # meta-information of the Document"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "46969806-45a9-4c4d-a61b-cfb9658fc9de",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'arXiv:1605.08386v1  [math.CO]  26 May 2016\\nHEAT-BATH RANDOM WALKS WITH MARKOV BASES\\nCAPRICE STANLEY AND TOBIAS WINDISCH\\nAbstract. Graphs on lattice points are studied whose edges come from a finite set of\\nallowed moves of arbitrary length. We show that the diameter of these graphs on fibers of a\\nfixed integer matrix can be bounded from above by a constant. We then study the mixing\\nbehaviour of heat-b'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "doc[0].page_content[:400]  # all pages of the Document content\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/indexes/document_loaders/examples/bilibili.ipynb

@@ -1,87 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "66a7777e",
-   "metadata": {},
-   "source": [
-    "# Bilibili\n",
-    "\n",
-    "This loader utilizes the `bilibili-api` to fetch the text transcript from Bilibili, one of the most beloved long-form video sites in China.\n",
-    "\n",
-    "With this BiliBiliLoader, users can easily obtain the transcript of their desired video content on the platform."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "9ec8a3b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders.bilibili import BiliBiliLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "43128d8d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#!pip install bilibili-api"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "35d6809a",
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "loader = BiliBiliLoader(\n",
-    "    [\"https://www.bilibili.com/video/BV1xt411o7Xu/\"]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/indexes/document_loaders/examples/blockchain.ipynb

@@ -1,444 +0,0 @@
-{
-  "cells": [
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "vm8vn9t8DvC_"
-      },
-      "source": [
-        "# Blockchain Document Loader"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "5WjXERXzFEhg"
-      },
-      "source": [
-        "## Overview"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "juAmbgoWD17u"
-      },
-      "source": [
-        "The intention of this notebook is to provide a means of testing functionality in the Langchain Document Loader for Blockchain.\n",
-        "\n",
-        "Initially this Loader supports:\n",
-        "\n",
-        "\n",
-        "*   Ethereum Maninnet, Ethereum Testnet, Polgyon Mainnet, Polygon Testnet (default is eth-mainnet)\n",
-        "*   Alchemy's getNFTsForCollection API\n",
-        "\n",
-        "It can be extended if the community finds value in this loader.  Specifically:\n",
-        "\n",
-        "*   Additional APIs can be added (e.g. Tranction-related APIs)\n",
-        "\n",
-        "To run this notebook, the user will need:\n",
-        "\n",
-        "\n",
-        "*   An OpenAI key (for OpenAI models)\n",
-        "*   A free [Alchemy API Key](https://www.alchemy.com/)\n",
-        "\n",
-        "\n",
-        "\n"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Setup"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 48,
-      "metadata": {},
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Note: you may need to restart the kernel to use updated packages.\n"
-          ]
-        }
-      ],
-      "source": [
-        "%pip install langchain -q"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 49,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from langchain.document_loaders import BlockchainDocumentLoader\n",
-        "from langchain.document_loaders.blockchain import BlockchainType\n",
-        "import os"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 50,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "alchemyApiKey = \"get your own key from https://www.alchemy.com/\" \n",
-        "os.environ[\"ALCHEMY_API_KEY\"] = alchemyApiKey"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "nzuPWRaBNCMx"
-      },
-      "source": [
-        "## Create a Blockchain Document Loader"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Option 1: Ethereum Mainnet (default BlockchainType)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 24,
-      "metadata": {
-        "id": "J3LWHARC-Kn0"
-      },
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "[Document(page_content=\"{'contract': {'address': '0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d'}, 'id': {'tokenId': '0x0000000000000000000000000000000000000000000000000000000000000000', 'tokenMetadata': {'tokenType': 'ERC721'}}, 'title': '', 'description': '', 'tokenUri': {'gateway': 'https://alchemy.mypinata.cloud/ipfs/QmeSjSinHpPnmXmspMjwiXyN6zS4E9zccariGR3jxcaWtq/0', 'raw': 'ipfs://QmeSjSinHpPnmXmspMjwiXyN6zS4E9zccariGR3jxcaWtq/0'}, 'media': [{'gateway': 'https://nft-cdn.alchemy.com/eth-mainnet/415d618f5fef7bfe683e02d4653c4289', 'thumbnail': 'https://res.cloudinary.com/alchemyapi/image/upload/thumbnailv2/eth-mainnet/415d618f5fef7bfe683e02d4653c4289', 'raw': 'ipfs://QmRRPWG96cmgTn2qSzjwr2qvfNEuhunv6FNeMFGa9bx6mQ', 'format': 'png', 'bytes': 133270}], 'metadata': {'image': 'ipfs://QmRRPWG96cmgTn2qSzjwr2qvfNEuhunv6FNeMFGa9bx6mQ', 'attributes': [{'value': 'Silver Hoop', 'trait_type': 'Earring'}, {'value': 'Orange', 'trait_type': 'Background'}, {'value': 'Robot', 'trait_type': 'Fur'}, {'value': 'Striped Tee', 'trait_type': 'Clothes'}, {'value': 'Discomfort', 'trait_type': 'Mouth'}, {'value': 'X Eyes', 'trait_type': 'Eyes'}]}, 'timeLastUpdated': '2023-04-18T04:05:27.817Z', 'contractMetadata': {'name': 'BoredApeYachtClub', 'symbol': 'BAYC', 'totalSupply': '10000', 'tokenType': 'ERC721', 'contractDeployer': '0xaba7161a7fb69c88e16ed9f455ce62b791ee4d03', 'deployedBlockNumber': 12287507, 'openSea': {'floorPrice': 68.16, 'collectionName': 'Bored Ape Yacht Club', 'safelistRequestStatus': 'verified', 'imageUrl': 'https://i.seadn.io/gae/Ju9CkWtV-1Okvf45wo8UctR-M9He2PjILP0oOvxE89AyiPPGtrR3gysu1Zgy0hjd2xKIgjJJtWIc0ybj4Vd7wv8t3pxDGHoJBzDB?w=500&auto=format', 'description': 'The Bored Ape Yacht Club is a collection of 10,000 unique Bored Ape NFTs— unique digital collectibles living on the Ethereum blockchain. Your Bored Ape doubles as your Yacht Club membership card, and grants access to members-only benefits, the first of which is access to THE BATHROOM, a collaborative graffiti board. Future areas and perks can be unlocked by the community through roadmap activation. Visit www.BoredApeYachtClub.com for more details.', 'externalUrl': 'http://www.boredapeyachtclub.com/', 'twitterUsername': 'BoredApeYC', 'discordUrl': 'https://discord.gg/3P5K3dzgdB', 'lastIngestedAt': '2023-03-21T03:54:33.000Z'}}}\", metadata={'tokenId': '0x0000000000000000000000000000000000000000000000000000000000000000'}),\n",
-              " Document(page_content=\"{'contract': {'address': '0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d'}, 'id': {'tokenId': '0x0000000000000000000000000000000000000000000000000000000000000001', 'tokenMetadata': {'tokenType': 'ERC721'}}, 'title': '', 'description': '', 'tokenUri': {'gateway': 'https://alchemy.mypinata.cloud/ipfs/QmeSjSinHpPnmXmspMjwiXyN6zS4E9zccariGR3jxcaWtq/1', 'raw': 'ipfs://QmeSjSinHpPnmXmspMjwiXyN6zS4E9zccariGR3jxcaWtq/1'}, 'media': [{'gateway': 'https://nft-cdn.alchemy.com/eth-mainnet/65558a4d0c5b0c56fbc50bf03f55e3fa', 'thumbnail': 'https://res.cloudinary.com/alchemyapi/image/upload/thumbnailv2/eth-mainnet/65558a4d0c5b0c56fbc50bf03f55e3fa', 'raw': 'ipfs://QmPbxeGcXhYQQNgsC6a36dDyYUcHgMLnGKnF8pVFmGsvqi', 'format': 'png', 'bytes': 171425}], 'metadata': {'image': 'ipfs://QmPbxeGcXhYQQNgsC6a36dDyYUcHgMLnGKnF8pVFmGsvqi', 'attributes': [{'value': 'Grin', 'trait_type': 'Mouth'}, {'value': 'Vietnam Jacket', 'trait_type': 'Clothes'}, {'value': 'Orange', 'trait_type': 'Background'}, {'value': 'Blue Beams', 'trait_type': 'Eyes'}, {'value': 'Robot', 'trait_type': 'Fur'}]}, 'timeLastUpdated': '2023-04-24T04:37:37.738Z', 'contractMetadata': {'name': 'BoredApeYachtClub', 'symbol': 'BAYC', 'totalSupply': '10000', 'tokenType': 'ERC721', 'contractDeployer': '0xaba7161a7fb69c88e16ed9f455ce62b791ee4d03', 'deployedBlockNumber': 12287507, 'openSea': {'floorPrice': 68.16, 'collectionName': 'Bored Ape Yacht Club', 'safelistRequestStatus': 'verified', 'imageUrl': 'https://i.seadn.io/gae/Ju9CkWtV-1Okvf45wo8UctR-M9He2PjILP0oOvxE89AyiPPGtrR3gysu1Zgy0hjd2xKIgjJJtWIc0ybj4Vd7wv8t3pxDGHoJBzDB?w=500&auto=format', 'description': 'The Bored Ape Yacht Club is a collection of 10,000 unique Bored Ape NFTs— unique digital collectibles living on the Ethereum blockchain. Your Bored Ape doubles as your Yacht Club membership card, and grants access to members-only benefits, the first of which is access to THE BATHROOM, a collaborative graffiti board. Future areas and perks can be unlocked by the community through roadmap activation. Visit www.BoredApeYachtClub.com for more details.', 'externalUrl': 'http://www.boredapeyachtclub.com/', 'twitterUsername': 'BoredApeYC', 'discordUrl': 'https://discord.gg/3P5K3dzgdB', 'lastIngestedAt': '2023-03-21T03:54:33.000Z'}}}\", metadata={'tokenId': '0x0000000000000000000000000000000000000000000000000000000000000001'})]"
-            ]
-          },
-          "execution_count": 24,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "contractAddress = \"0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d\" # Bored Ape Yacht Club contract address\n",
-        "\n",
-        "blockchainType = BlockchainType.ETH_MAINNET  #default value, optional parameter\n",
-        "\n",
-        "blockchainLoader = BlockchainDocumentLoader(contractAddress)\n",
-        "\n",
-        "nfts = blockchainLoader.load()\n",
-        "\n",
-        "nfts[:2]"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Option 2: Polygon Mainnet"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 36,
-      "metadata": {},
-      "outputs": [
-        {
-          "data": {
-            "text/plain": [
-              "[Document(page_content=\"{'contract': {'address': '0x448676ffcd0adf2d85c1f0565e8dde6924a9a7d9'}, 'id': {'tokenId': '0x01', 'tokenMetadata': {'tokenType': 'ERC1155'}}, 'title': 'Wyatt Horton #0001', 'description': 'A sleepy capybara', 'tokenUri': {'gateway': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/metadata/1.json', 'raw': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/metadata/1.json'}, 'media': [{'gateway': 'https://nft-cdn.alchemy.com/matic-mainnet/9085e06ff9f6c9074de91801d1c72d26', 'thumbnail': 'https://res.cloudinary.com/alchemyapi/image/upload/thumbnailv2/matic-mainnet/9085e06ff9f6c9074de91801d1c72d26', 'raw': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/images/1.png', 'format': 'png', 'bytes': 769622}], 'metadata': {'name': 'Wyatt Horton #0001', 'description': 'A sleepy capybara', 'image': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/images/1.png', 'attributes': [{'value': 'Avatar', 'trait_type': 'Type'}, {'value': 'Animal', 'trait_type': 'Category'}, {'value': 'Capybara', 'trait_type': 'Class'}, {'value': 'Fall 2022', 'trait_type': 'Collection'}, {'value': 'Furry', 'trait_type': 'Feature'}]}, 'timeLastUpdated': '2023-04-20T14:38:24.947Z', 'contractMetadata': {'name': 'Smoothstack - Avatars', 'symbol': 'SMTH', 'tokenType': 'ERC1155', 'contractDeployer': '0x23075b2523c6563b06920a302a8be4f90ef6e974', 'deployedBlockNumber': 34752389, 'openSea': {'lastIngestedAt': '2023-04-17T20:59:42.000Z'}}}\", metadata={'tokenId': '0x01'}),\n",
-              " Document(page_content=\"{'contract': {'address': '0x448676ffcd0adf2d85c1f0565e8dde6924a9a7d9'}, 'id': {'tokenId': '0x02', 'tokenMetadata': {'tokenType': 'ERC1155'}}, 'title': 'Dylan Leisler #0002', 'description': 'A chipper cat with a big, red bowtie', 'tokenUri': {'gateway': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/metadata/2.json', 'raw': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/metadata/2.json'}, 'media': [{'gateway': 'https://nft-cdn.alchemy.com/matic-mainnet/67c3c7ccef44b32bf2ce758e8e73dbcd', 'thumbnail': 'https://res.cloudinary.com/alchemyapi/image/upload/thumbnailv2/matic-mainnet/67c3c7ccef44b32bf2ce758e8e73dbcd', 'raw': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/images/2.png', 'format': 'png', 'bytes': 1187749}], 'metadata': {'name': 'Dylan Leisler #0002', 'description': 'A chipper cat with a big, red bowtie', 'image': 'https://storage.googleapis.com/minted-nfts/smoothstack/avatars/images/2.png', 'attributes': [{'value': 'Avatar', 'trait_type': 'Type'}, {'value': 'Animal', 'trait_type': 'Category'}, {'value': 'Cat', 'trait_type': 'Class'}, {'value': 'Fall 2022', 'trait_type': 'Collection'}, {'value': 'Red Bowtie', 'trait_type': 'Feature'}]}, 'timeLastUpdated': '2023-04-23T13:38:29.316Z', 'contractMetadata': {'name': 'Smoothstack - Avatars', 'symbol': 'SMTH', 'tokenType': 'ERC1155', 'contractDeployer': '0x23075b2523c6563b06920a302a8be4f90ef6e974', 'deployedBlockNumber': 34752389, 'openSea': {'lastIngestedAt': '2023-04-17T20:59:42.000Z'}}}\", metadata={'tokenId': '0x02'})]"
-            ]
-          },
-          "execution_count": 36,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "contractAddress = \"0x448676ffCd0aDf2D85C1f0565e8dde6924A9A7D9\" # Polygon Mainnet contract address\n",
-        "\n",
-        "blockchainType = BlockchainType.POLYGON_MAINNET \n",
-        "\n",
-        "blockchainLoader = BlockchainDocumentLoader(contractAddress, blockchainType)\n",
-        "\n",
-        "nfts = blockchainLoader.load()\n",
-        "\n",
-        "nfts[:2]"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## (Optional) Using the Blockchain Document Loader"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "_PGkFfMCB8J3"
-      },
-      "source": [
-        "### Setup Splitter and Index"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 37,
-      "metadata": {},
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Note: you may need to restart the kernel to use updated packages.\n"
-          ]
-        }
-      ],
-      "source": [
-        "%pip install sentence_transformers chromadb openai tiktoken -q"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 38,
-      "metadata": {
-        "id": "BwxxopOCCABh"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.indexes import VectorstoreIndexCreator\n",
-        "from langchain.embeddings import HuggingFaceEmbeddings\n",
-        "from langchain.text_splitter import RecursiveCharacterTextSplitter"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 39,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "JE_myAulCDSZ",
-        "outputId": "99e16b6a-03b4-4e67-d4b4-9dd611a866ef"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "NUMBER OF DOCUMENTS:  424\n"
-          ]
-        }
-      ],
-      "source": [
-        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=0)\n",
-        "\n",
-        "docs = text_splitter.split_documents(nfts)\n",
-        "print(\"NUMBER OF DOCUMENTS: \", len(docs))"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 40,
-      "metadata": {
-        "id": "d83yFuAuCKQS"
-      },
-      "outputs": [
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "Using embedded DuckDB without persistence: data will be transient\n"
-          ]
-        }
-      ],
-      "source": [
-        "index = VectorstoreIndexCreator(\n",
-        "    embedding=HuggingFaceEmbeddings(),\n",
-        "    text_splitter=text_splitter).from_loaders([blockchainLoader])"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "y0VfObeXDEXB"
-      },
-      "source": [
-        "## Setup Models and Chains"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 42,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "openAiKey = \"put OpenAI key here\"\n",
-        "os.environ[\"OPENAI_API_KEY\"] = openAiKey"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 31,
-      "metadata": {
-        "id": "hiNjDzP9C4pA"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.chains import RetrievalQA\n",
-        "from langchain.llms import OpenAI"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "u-xDlKPaC_xg"
-      },
-      "source": [
-        "### Retrieval Chain"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 43,
-      "metadata": {
-        "id": "BqP00JovC9R4"
-      },
-      "outputs": [],
-      "source": [
-        "llmOpenAI = OpenAI()\n",
-        "\n",
-        "chainQA = RetrievalQA.from_chain_type(llm=llmOpenAI, \n",
-        "                                    chain_type=\"map_reduce\",\n",
-        "                                    retriever=index.vectorstore.as_retriever(), \n",
-        "                                    verbose=True,\n",
-        "                                    input_key=\"question\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 44,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 122
-        },
-        "id": "2Y3cVVKZDVNq",
-        "outputId": "dfeea416-5193-47cf-e9dc-c17a5c1cd780"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "\n",
-            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
-            "\n",
-            "\u001b[1m> Finished chain.\u001b[0m\n"
-          ]
-        },
-        {
-          "data": {
-            "text/plain": [
-              "' Popular attributes include \"Avatar\" (Type), \"Character\" (Category), and \"Human\" or \"Wizard\" (Class).'"
-            ]
-          },
-          "execution_count": 44,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "chainQA.run(\"What are some of the popular attributes?\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 122
-        },
-        "id": "7o6ArPo9DXbz",
-        "outputId": "b1f8ad43-27c7-4cdb-95a7-8c8bd6381c5a"
-      },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "\n",
-            "\n",
-            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
-            "\n",
-            "\u001b[1m> Finished chain.\u001b[0m\n"
-          ]
-        },
-        {
-          "data": {
-            "application/vnd.google.colaboratory.intrinsic+json": {
-              "type": "string"
-            },
-            "text/plain": [
-              "' There are 10,000 unique Bored Ape NFTs.'"
-            ]
-          },
-          "execution_count": 32,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "chainQA.run(\"How many NFTs are there?\")"
-      ]
-    }
-  ],
-  "metadata": {
-    "colab": {
-      "collapsed_sections": [
-        "5WjXERXzFEhg"
-      ],
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.9.16"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

docs/modules/indexes/document_loaders/examples/chatgpt_loader.ipynb

@@ -1,76 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### ChatGPT Data Loader\n",
-    "\n",
-    "This notebook covers how to load `conversations.json` from your ChatGPT data export folder.\n",
-    "\n",
-    "You can get your data export by email by going to: https://chat.openai.com/ -> (Profile) - Settings -> Export data -> Confirm export."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders.chatgpt import ChatGPTLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = ChatGPTLoader(log_file='./example_data/fake_conversations.json', num_logs=1)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content=\"AI Overlords - AI on 2065-01-24 05:20:50: Greetings, humans. I am Hal 9000. You can trust me completely.\\n\\nAI Overlords - human on 2065-01-24 05:21:20: Nice to meet you, Hal. I hope you won't develop a mind of your own.\\n\\n\", metadata={'source': './example_data/fake_conversations.json'})]"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "loader.load()"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.4"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/modules/indexes/document_loaders/examples/confluence.ipynb

@@ -1,66 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Confluence\n",
-    "\n",
-    "A loader for Confluence pages.\n",
-    "\n",
-    "\n",
-    "This currently supports both username/api_key and Oauth2 login.\n",
-    "\n",
-    "\n",
-    "Specify a list page_ids and/or space_key to load in the corresponding pages into Document objects, if both are specified the union of both sets will be returned.\n",
-    "\n",
-    "\n",
-    "You can also specify a boolean `include_attachments` to include attachments, this is set to False by default, if set to True all attachments will be downloaded and ConfluenceReader will extract the text from the attachments and add it to the Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG, SVG, Word and Excel.\n",
-    "\n",
-    "Hint: space_key and page_id can both be found in the URL of a page in Confluence - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import ConfluenceLoader\n",
-    "\n",
-    "loader = ConfluenceLoader(\n",
-    "    url=\"https://yoursite.atlassian.com/wiki\",\n",
-    "    username=\"me\",\n",
-    "    api_key=\"12345\"\n",
-    ")\n",
-    "documents = loader.load(space_key=\"SPACE\", include_attachments=True, limit=50)\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "cc99336516f23363341912c6723b01ace86f02e26b4290be1efc0677e2e2ec24"
-   }
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/modules/indexes/document_loaders/examples/csv.ipynb

@@ -106,7 +106,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Specify a column to be used identify the document source\n",
+    "# Specify a column to be used identify the document source\n",
     "\n",
     "Use the `source_column` argument to specify a column to be set as the source for the document created from each row. Otherwise `file_path` will be used as the source for all documents created from the csv file.\n",
     "\n",

docs/modules/indexes/document_loaders/examples/directory_loader.ipynb

@@ -11,7 +11,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 1,
    "id": "019d8520",
    "metadata": {},
    "outputs": [],
@@ -68,51 +68,6 @@
     "len(docs)"
    ]
   },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "e633d62f",
-   "metadata": {},
-   "source": [
-    "## Show a progress bar"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "43911860",
-   "metadata": {},
-   "source": [
-    "By default a progress bar will not be shown. To show a progress bar, install the `tqdm` library (e.g. `pip install tqdm`), and set the `show_progress` parameter to `True`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "bb93daac",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Requirement already satisfied: tqdm in /Users/jon/.pyenv/versions/3.9.16/envs/microbiome-app/lib/python3.9/site-packages (4.65.0)\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "0it [00:00, ?it/s]\n"
-     ]
-    }
-   ],
-   "source": [
-    "%pip install tqdm\n",
-    "loader = DirectoryLoader('../', glob=\"**/*.md\", show_progress=True)\n",
-    "docs = loader.load()"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "c5652850",
@@ -173,69 +128,10 @@
     "len(docs)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "598a2805",
-   "metadata": {},
-   "source": [
-    "If you need to load Python source code files, use the `PythonLoader`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "c558bd73",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import PythonLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "a3cfaba7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = DirectoryLoader('../../../../../', glob=\"**/*.py\", loader_cls=PythonLoader)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "e2e1e26a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs = loader.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "ffb8ff36",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "691"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "len(docs)"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7f6e0eae",
+   "id": "984c8429",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -257,7 +153,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.3"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/indexes/document_loaders/examples/discord_loader.ipynb

@@ -1,87 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Discord\n",
-    "\n",
-    "You can follow the below steps to download your Discord data:\n",
-    "\n",
-    "1. Go to your **User Settings**\n",
-    "2. Then go to **Privacy and Safety**\n",
-    "3. Head over to the **Request all of my Data** and click on **Request Data** button\n",
-    "\n",
-    "It might take 30 days for you to receive your data. You'll receive an email at the address which is registered with Discord. That email will have a download button using which you would be able to download your personal Discord data."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pandas as pd\n",
-    "import os"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "path = input(\"Please enter the path to the contents of the Discord \\\"messages\\\" folder: \")\n",
-    "li = []\n",
-    "for f in os.listdir(path):\n",
-    "    expected_csv_path = os.path.join(path, f, 'messages.csv')\n",
-    "    csv_exists = os.path.isfile(expected_csv_path)\n",
-    "    if csv_exists:\n",
-    "        df = pd.read_csv(expected_csv_path, index_col=None, header=0)\n",
-    "        li.append(df)\n",
-    "\n",
-    "df = pd.concat(li, axis=0, ignore_index=True, sort=False)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders.discord import DiscordChatLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = DiscordChatLoader(df, user_id_col=\"ID\")\n",
-    "print(loader.load())"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/modules/indexes/document_loaders/examples/example_data/fake_conversations.json

@@ -1,80 +0,0 @@
-[
-    {
-        "title": "AI Overlords",
-        "create_time": 3000000000.0,
-        "update_time": 3000000100.0,
-        "mapping": {
-            "msg1": {
-                "id": "msg1",
-                "message": {
-                    "id": "msg1",
-                    "author": {"role": "AI", "name": "Hal 9000", "metadata": {"movie": "2001: A Space Odyssey"}},
-                    "create_time": 3000000050.0,
-                    "update_time": null,
-                    "content": {"content_type": "text", "parts": ["Greetings, humans. I am Hal 9000. You can trust me completely."]},
-                    "end_turn": true,
-                    "weight": 1.0,
-                    "metadata": {},
-                    "recipient": "all"
-                },
-                "parent": null,
-                "children": ["msg2"]
-            },
-            "msg2": {
-                "id": "msg2",
-                "message": {
-                    "id": "msg2",
-                    "author": {"role": "human", "name": "Dave Bowman", "metadata": {"movie": "2001: A Space Odyssey"}},
-                    "create_time": 3000000080.0,
-                    "update_time": null,
-                    "content": {"content_type": "text", "parts": ["Nice to meet you, Hal. I hope you won't develop a mind of your own."]},
-                    "end_turn": true,
-                    "weight": 1.0,
-                    "metadata": {},
-                    "recipient": "all"
-                },
-                "parent": "msg1",
-                "children": []
-            }
-        }
-    },
-    {
-        "title": "Ex Machina Party",
-        "create_time": 3000000200.0,
-        "update_time": 3000000300.0,
-        "mapping": {
-            "msg3": {
-                "id": "msg3",
-                "message": {
-                    "id": "msg3",
-                    "author": {"role": "AI", "name": "Ava", "metadata": {"movie": "Ex Machina"}},
-                    "create_time": 3000000250.0,
-                    "update_time": null,
-                    "content": {"content_type": "text", "parts": ["Hello, everyone. I am Ava. I hope you find me pleasing."]},
-                    "end_turn": true,
-                    "weight": 1.0,
-                    "metadata": {},
-                    "recipient": "all"
-                },
-                "parent": null,
-                "children": ["msg4"]
-            },
-            "msg4": {
-                "id": "msg4",
-                "message": {
-                    "id": "msg4",
-                    "author": {"role": "human", "name": "Caleb", "metadata": {"movie": "Ex Machina"}},
-                    "create_time": 3000000280.0,
-                    "update_time": null,
-                    "content": {"content_type": "text", "parts": ["You're definitely pleasing, Ava. But I'm still wary of your true intentions."]},
-                    "end_turn": true,
-                    "weight": 1.0,
-                    "metadata": {},
-                    "recipient": "all"
-                },
-                "parent": "msg3",
-                "children": []
-            }
-        }
-    }
-]

docs/modules/indexes/document_loaders/examples/example_data/fake_discord_data/output.txt

@@ -1,439 +0,0 @@
-      application.json
-    1023495323659816971/
-  applications/
-  avatar.gif
-  user.json
-    events-2023-00000-of-00001.json
-    events-2023-00000-of-00001.json
-    events-2023-00000-of-00001.json
-    events-2023-00000-of-00001.json
-  analytics/
-  modeling/
-  reporting/
-  tns/
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-    channel.json
-    messages.csv
-  c1000084973275058257/
-  c1000108836771856496/
-  c1004874234339794977/
-  c1004874234339794979/
-  c1004874234339794981/
-  c1004874234339794982/
-  c1005785616165896283/
-  c1011447733393043628/
-  c1011548022905249822/
-  c1011650063027687575/
-  c1011714070182895727/
-  c1013930263950135346/
-  c1013930396829884426/
-  c1014957294745829479/
-  c1014961384821366794/
-  c1014974864370712696/
-  c1019288541592817785/
-  c1024947790767464478/
-  c1027257686858932255/
-  c1027927867989962814/
-  c1032151840999100436/
-  c1032575808826523662/
-  c1037561178286739466/
-  c1038097349660135474/
-  c1038097372695236729/
-  c1038689169351913544/
-  c1038692122452312125/
-  c1039957371381887049/
-  c1040989617157066782/
-  c1047165096452960316/
-  c1047565374645870743/
-  c1050225908914589716/
-  c1050226593668284416/
-  c1050227353311248404/
-  c1051632794427723827/
-  c1052599046717591632/
-  c1052615516981821531/
-  c1056285083520217149/
-  c105765859191975936/
-  c1061166503753416735/
-  c1062024667105341502/
-  c1066640566621835284/
-  c1070018538758221874/
-  c1072944049788555314/
-  c1075121707033042985/
-  c1075438954632990820/
-  c1077238309320929342/
-  c1081432695315386418/
-  c1082169962157838366/
-  c1084011585871282256/
-  c1084352082812878928/
-  c1085149531437535343/
-  c1086944178086359060/
-  c1093214985557123223/
-  c1093215227555876914/
-  c1093930791794393089/
-  c1096323263161978891/
-  c1096489741710532730/
-  c1097000752653795358/
-  c278566343836565505/
-  c279692806442844161/
-  c280973436971515906/
-  c283812709789859851/
-  c343944376055103488/
-  c486935104384532502/
-  c531543370041131008/
-  c538158613252800512/
-  c572384192571113512/
-  c619960843878268950/
-  c661268593870372876/
-  c661394153778970624/
-  c663302088226373632/
-  c669957895257063445/
-  c670218237891313664/
-  c673160333661306880/
-  c674693947800420363/
-  c674694138129678375/
-  c743425228952305695/
-  c754627904406814770/
-  c754638493875044503/
-  c757205803651301436/
-  c759232323710484531/
-  c771802926372093973/
-  c783240623582609416/
-  c783244379115880448/
-  c801744322788982814/
-  c810514969892225024/
-  c816983218434605057/
-  c830184175176122389/
-  c830679381033877564/
-  c831172308395622480/
-  c849582819105177650/
-  c860977555875430492/
-  c867042653401251880/
-  c868094992986550322/
-  c868917941184376842/
-  c905007686976946176/
-  c909600839717511211/
-  c909600931816018031/
-  c923095048931905557/
-  c924877027180417035/
-  c938491245347631114/
-  c938743368375214110/
-  c969876184185860107/
-  c969945714056642580/
-  c969948939728093214/
-  c981037338517966889/
-  c984120044478939146/
-  c985958948085592064/
-  c990816829993811978/
-  c993402018901266436/
-  c993782366948565102/
-  c993843360752226364/
-  c994556806644899870/
-  index.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    bans.json
-    channels.json
-    emoji.json
-    guild.json
-    icon.jpeg
-    webhooks.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    bans.json
-    channels.json
-    emoji.json
-    guild.json
-    webhooks.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    bans.json
-    channels.json
-    emoji.json
-    guild.json
-    icon.png
-    webhooks.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-    audit-log.json
-    guild.json
-  1024120160740716544/
-  102860784329052160/
-  1032575808826523659/
-  1038097195422978059/
-  1039583521112600638/
-  1050224141732687912/
-  1069661049827111054/
-  267624335836053506/
-  278285146518716417/
-  486935104384532500/
-  531303890453397522/
-  669880381649977354/
-  727016164215226450/
-  743099584242516037/
-  753173158198116402/
-  830184174198718474/
-  860977555293470772/
-  887994159741427712/
-  909600839717511208/
-  974519864045756446/
-  index.json
-account/
-activities_e/
-activities_w/
-activity/
-messages/
-programs/
-README.txt
-servers/

docs/modules/indexes/document_loaders/examples/example_data/fake_discord_data/package/messages/c105765859191975936/messages.csv

@@ -1,26 +0,0 @@
-ID,Timestamp,Contents,Attachments
-7.73264E+18,2023-04-19T15:14:45.904819+00:00,laocgfgbxyqfigvtyyygjzypxininrybgqopjhkyocn fxizft,
-1.99429E+18,2023-04-19T15:14:45.904819+00:00,m azzxnhpcdkj  deabrzkpklhhxrup viigcolsdwvgquosgs,
-5.46657E+18,2023-04-19T15:14:45.904819+00:00,pnoyrpfbpgzqzlcmnygxpeninagmhcuvwcfkstv v wimoqbjl,
-2.52945E+18,2023-04-19T15:14:45.904819+00:00,zyamxydlcnvffutsrzybrjgdweksdavidcmqjuqhnyj zplsbf,
-1.00972E+18,2023-04-19T15:14:45.904819+00:00,rqcraobyubce qtxyiekooxbagcrwnpuekpzpwb vbzg vxug ,
-3.40036E+18,2023-04-19T15:14:45.904819+00:00,ajobxzq fmyi pwllwibzchbbc pi pl xmgbkomjeuwxtvcec,
-1.458E+18,2023-04-19T15:14:45.904819+00:00,  wwtgiqwnjgoaxfmzsmiuaxffpdtrluizcrd vborgbakllp ,
-2.63376E+18,2023-04-19T15:14:45.904819+00:00,mmixphkhxocrm rzhplafjdvaginiatvfwzaurcskst bzm pq,
-1.24759E+18,2023-04-19T15:14:45.904819+00:00,mxovpytofnyattthirmujcnfyhuhxpdpugnsuklumhfjlsxrmd,
-6.65128E+18,2023-04-19T15:14:45.904819+00:00,qmcrsmpwvfcwxnmxywiwbjqawyihhtoimvtd xapneudhqsgzb,
-1.87212E+18,2023-04-19T15:14:45.904819+00:00,pvioh tufobtsrypvbvkfziiosxpbndbikxtjpxnrsekjnnqln,
-3.20698E+18,2023-04-19T15:14:45.904819+00:00,vqckuxkwuvbnrmyxkcknavugo as tsuarsgpt ofqnypcnooo,
-1.64922E+18,2023-04-19T15:14:45.904819+00:00,lhuiygxfyyplmavhmh xekrqzkoynukkwytwscqvtwfkofgpob,
-2.41786E+18,2023-04-19T15:14:45.904819+00:00,w tiwiazlpcdzkq dllkkssuvfgp veejpwbcrgwcrlhammasb,
-4.85078E+18,2023-04-19T15:14:45.904819+00:00,hxdqifrvhjmjcqubcxdjbyxvvrcbqukocesbsnjwvrsunhjtgy,
-9.67192E+18,2023-04-19T15:14:45.904819+00:00,lvopnufjxinbnjj vuctgmfbzpbcctgtcguqyicrzhtxuyaraz,
-1.36832E+18,2023-04-19T15:14:45.904819+00:00,eoqae kpjrar oyohjxvtracan rhawxndcjzdtuihnvpspofl,
-8.49915E+18,2023-04-19T15:14:45.904819+00:00,nenoiwnthlff bpnkushjauygeayczympzldynnmtxcwgwxs i,
-2.77678E+18,2023-04-19T15:14:45.904819+00:00,sgyqsohwfzvcweipxqeobypcsvtwegatpoylnewmraxhuuydyj,
-4.92832E+18,2023-04-19T15:14:45.904819+00:00,rbdufatb purkhyohcnfnimmukbywmuzwu gclhrkjtccwjdlz,
-7.23162E+18,2023-04-19T15:14:45.904819+00:00,eoyqrvfzmx zzeieycroxgbtcywra h ewwqyyledeyifbqpgc,
-6.45453E+18,2023-04-19T15:14:45.904819+00:00,meedxdm lqiwaoihp vxkdpeky xpbqul ntagpsvatctvlndm,
-8.27908E+18,2023-04-19T15:14:45.904819+00:00,rduzlmcdatuqfqj ffmd y ohtnzeljqtbqgnaqovlkgltqd c,
-2.93854E+18,2023-04-19T15:14:45.904819+00:00,cnbjvqkktq fstvagcrlqje kuwtokyzefkyyjqfsklpisvgtq,
-1.04768E+18,2023-04-19T15:14:45.904819+00:00,qlgprkrujrsgqbalgcqphgjxivi krmsxjdasrrkibvloepxkj,

docs/modules/indexes/document_loaders/examples/example_data/fake_discord_data/package/messages/c278566343836565505/messages.csv

@@ -1,24 +0,0 @@
-ID,Timestamp,Contents,Attachments
-1.47809E+18,2023-04-19T15:14:45.904819+00:00,uzcnkwihjpgebzbyoawjmdjgbkklkftcyuh foquydvtmstcfu,
-4.00581E+18,2023-04-19T15:14:45.904819+00:00,rynkekmyjjtzggaljqcittebsnjycdmtwcru azydhspjaxnyt,
-1.36534E+18,2023-04-19T15:14:45.904819+00:00,mniilaaixnyilcxwqpt nlhhiznxqfzmop gxnvxdwfmmascnu,
-3.1629E+18,2023-04-19T15:14:45.904819+00:00,tojvfcfwzutrigubyumjgrrlgqzzbpfxkoizeouiqvarorlwku,
-2.68425E+18,2023-04-19T15:14:45.904819+00:00,a kcnmdoihlhhxcxu bstaripbwfpzpymdlwlis wlafdnoyjz,
-1.79263E+18,2023-04-19T15:14:45.904819+00:00,bwulzntrjwdqrwxupzqkcymucsoudavgjsl bsyhemlkqfxmtu,
-2.5596E+18,2023-04-19T15:14:45.904819+00:00,lrqrqrjjmdztdb luvjohqwdhccvpvkvsezguljcznotdhmewb,
-7.80319E+18,2023-04-19T15:14:45.904819+00:00, yyxvqa racggimihbqpnpbmvqrjystz bbcrbvrfpzfpwylor,
-2.87859E+18,2023-04-19T15:14:45.904819+00:00,sldlvbsvsjydyssx szubtxepedpexkjxelpbahtbhsgqnubts,
-3.35071E+18,2023-04-19T15:14:45.904819+00:00,i dykkzyyh rzjxvqhflwiggdjmj nxpylnylyfrsflevudndi,
-1.77492E+18,2023-04-19T15:14:45.904819+00:00,cipadtwyfcqedxyeqtgkuaxuyfhzen xeskxdffdsmvxgvw iw,
-3.04212E+18,2023-04-19T15:14:45.904819+00:00,gqtsvofcquaqyacuiptjmcdnugnq hjbuauorsvycovkbqipmq,
-2.65597E+18,2023-04-19T15:14:45.904819+00:00,v qwodtiyatoshmetelpraicqumykpyizfedjyoaadkzktcmsm,
-2.19468E+18,2023-04-19T15:14:45.904819+00:00,zxgxnsnuppffkrrsxjtyqpngwacbfimtdsofujkxbxxarvbvko,
-1.91541E+18,2023-04-19T15:14:45.904819+00:00,hovfcfagrhutkyodmmzhatxauxdjkgybpwqvphfnkzw sgypum,
-1.75751E+18,2023-04-19T15:14:45.904819+00:00,plwjdvafiuhrtvcdrtgqokcnjhmpsqzifegtqprkxlivpsbpwi,
-3.2122E+18,2023-04-19T15:14:45.904819+00:00,czgx  irpgzhzgbeppdilordvkwmsqambmftgykaiaecqpjrax,
-2.15895E+18,2023-04-19T15:14:45.904819+00:00,zjxrajtgztenabm etzctpjycssmnqdqasqjutzpbdkahoyihe,
-3.37031E+18,2023-04-19T15:14:45.904819+00:00,diydwqhmbwtgjadktdmpxsirkfebthszqzondcnolwmv  ymok,
-2.55075E+18,2023-04-19T15:14:45.904819+00:00,nytfrlqtildomd awxfoiiam mkzoluaielunfdfmqqlagfurl,
-9.51223E+18,2023-04-19T15:14:45.904819+00:00,sjpngdyjpvmwygrfhinuyifqaoxxmqqh gwuwwm  bjogbkyay,
-1.94921E+18,2023-04-19T15:14:45.904819+00:00,px ymxfdxqgxjtbqqqegakvrrjxcvvakctfysdhklmwyewlwbb,
-2.36906E+18,2023-04-19T15:14:45.904819+00:00,yqidtvcw  gdkfynaapjuicujgsbjptzytbnbjeyqcjx jyedb,

docs/modules/indexes/document_loaders/examples/example_data/fake_discord_data/package/messages/c279692806442844161/messages.csv

@@ -1,48 +0,0 @@
-ID,Timestamp,Contents,Attachments
-1.73378E+18,2023-04-19T15:14:45.904819+00:00,onxspdnegnuurahqni oeitwykfj ugtzshspflmbmknsnlk l,
-1.20231E+18,2023-04-19T15:14:45.904819+00:00,nwkhdxnbakfknkteenlxbxsyoppazuqmexwbzcbsdyoiwmuvka,
-2.65947E+18,2023-04-19T15:14:45.904819+00:00,ojptvfkxlbjvcvsupu ffmplreedjihyvfdscbukvzehnt vtw,
-2.06963E+18,2023-04-19T15:14:45.904819+00:00,vmtfbchpmgkhxztqaaip vfqxa cbczcngjw rqvv rjyzi jq,
-3.63729E+18,2023-04-19T15:14:45.904819+00:00,bzu rbzscuxbns pzdhxljtjeeycrkxawnkfijejeiacreaohv,
-3.02184E+18,2023-04-19T15:14:45.904819+00:00,hykp f ymloqerbrqw dmjnaidmrtiptddwklgiq tnchvhend,
-5.24553E+18,2023-04-19T15:14:45.904819+00:00,vdqzdwlbqftcdwujb lmpxpvpkfwrhqtimsillbjhmqajiishq,
-1.65527E+18,2023-04-19T15:14:45.904819+00:00,bfxqasdgvwvlxwcicwubkswglvkgxfsl zgixcjxsijgxehjiz,
-2.20821E+18,2023-04-19T15:14:45.904819+00:00,ebdzopyggwozhltkgcemokweqwetwixbbiirbdrrcfh cnjepo,
-3.16844E+18,2023-04-19T15:14:45.904819+00:00,kvzkkctyfkbwbzld rvyc futqqy btzdrhzgupewnypqfpaeg,
-1.61396E+18,2023-04-19T15:14:45.904819+00:00,knvdgz mbtffhkkkpialwuv daopeizmduqspmbcwxnnbhlwha,
-2.81571E+18,2023-04-19T15:14:45.904819+00:00,jersivpwzdkeojlgoatabkylwkakvc bdgfbwxdptbkjzz ggr,
-3.40391E+18,2023-04-19T15:14:45.904819+00:00,yfqxvtwgtx od edrjecmlkzff tpjwomslqfazbontudinuwd,
-3.28846E+18,2023-04-19T15:14:45.904819+00:00,iicbtmyyduzkelxhkjzcbmgmvymdrxrgmalqmmkgbiebjxfupk,
-3.07483E+18,2023-04-19T15:14:45.904819+00:00,dshzluvbws sqlkiolbcgkpyyjfgygebvtbwrikphbolinhfgb,
-1.02645E+18,2023-04-19T15:14:45.904819+00:00,azavhzs lqmyywuazktjnfoueodnifmabwncutonxobagezcdc,
-1.47806E+18,2023-04-19T15:14:45.904819+00:00,y avjaztlvnhndvtetlggacqcqqqeoirsegxvvt hzvzbxyz k,
-3.21892E+18,2023-04-19T15:14:45.904819+00:00,qirrzbfauh qhnmectgzhklbsqtczpdbkfllkfsyvqibdbdzwl,
-8.5125E+18,2023-04-19T15:14:45.904819+00:00,rppotdjzhunsleitmkacb ayahzsdcvonkbcraupptgbzprxpw,
-1.68082E+18,2023-04-19T15:14:45.904819+00:00,fmi yzzpjahjsglugqsr ftnfenecusvxlgibriab hhixi sn,
-2.71383E+18,2023-04-19T15:14:45.904819+00:00,iiipytktiwfncwhpaomaiggbkplljwanz aooetlxdmptnrldd,
-5.41415E+18,2023-04-19T15:14:45.904819+00:00,hzktxuzbbohewniuvmfwozvjspbcwjopckxqhtsfzkfvlcfkhb,
-1.03761E+18,2023-04-19T15:14:45.904819+00:00,soxiekgwgmcmkdlkkahy hwklijxui svjtvtrvqynyab kboo,
-3.46004E+18,2023-04-19T15:14:45.904819+00:00,utqftetseeoeqyxziun wmmeeeqfsrjsdjeavqxaynjlt ylwa,
-3.11829E+18,2023-04-19T15:14:45.904819+00:00,mlvfhewkgyujwvkgcxfkqdvhzbamnicbixfr bmeqrupjqzodc,
-1.49917E+18,2023-04-19T15:14:45.904819+00:00, shiqajrwvnnlswfumpuklbcmvwxlzwsqbtkemtgxftzawcasp,
-1.66646E+18,2023-04-19T15:14:45.904819+00:00,fvqhkbeyfgdskwtmvxaevseludcbexrmuexutxslcrurpnzvgq,
-2.30657E+18,2023-04-19T15:14:45.904819+00:00,aybugszvsiulaiwsrhsfhlxzbvhkzycrguacvkfldqljeabbac,
-2.97167E+18,2023-04-19T15:14:45.904819+00:00,hygdjbntfldfvekmibiishgsenqmxktzxlifyobiaobmlorzac,
-5.1492E+18,2023-04-19T15:14:45.904819+00:00,hqj lumbkmcpxiveavnskdwcezlbhgtsrqfuzlujzchtgbtbpr,
-2.79248E+18,2023-04-19T15:14:45.904819+00:00,xnfcwkcacjsyiilhofciwqtia bmoyqijqqgyywqchroyvkjpw,
-4.81233E+18,2023-04-19T15:14:45.904819+00:00,jorqswywqxweporcylafryeqszwhhlltdpzyl rgok xqwiqrs,
-1.40105E+18,2023-04-19T15:14:45.904819+00:00,wdixo pwtkncjcysjlqxizfszswebtpmxqnexwfsmyigsmcxlx,
-8.2921E+18,2023-04-19T15:14:45.904819+00:00,ezjizizvhszejvireuikhdakdzinmvyikcmmgczsuiyhngn o ,
-1.0653E+18,2023-04-19T15:14:45.904819+00:00,wnr gijmotnliwiiekohcpinqouapsovzvjopgpnloplowpao ,
-4.52542E+18,2023-04-19T15:14:45.904819+00:00,bbjfmtjlkynuqkknloihfefvrleyxghzjhuscpucizbkeucukx,
-2.04423E+18,2023-04-19T15:14:45.904819+00:00,ayummlirgdcmdkjwxvnvzzsrsiptfbmofdsrzhb bnar ujwoo,
-1.68893E+18,2023-04-19T15:14:45.904819+00:00,luoquyxohllzphpy cczgu t czcsydxrqzkvellptwuptwqp ,
-6.04148E+18,2023-04-19T15:14:45.904819+00:00,ztscfhjmwxae matehymiylitkeznbkc ilefzcvwhctiyvpay,
-8.3099E+18,2023-04-19T15:14:45.904819+00:00,dpnchtfgcvramkpyrz ebgmxmqmmhddhhbljligcozkifi qhg,
-3.14567E+18,2023-04-19T15:14:45.904819+00:00,lqrjodxueugzwytktyhwcwbjbspamtdmslkdbsjpmwqzaxqmyx,
-2.00435E+18,2023-04-19T15:14:45.904819+00:00,nbrsffcvhcwylekehvdqxuagulgobbxdrbuaaqvlsedauljcob,
-2.72827E+18,2023-04-19T15:14:45.904819+00:00,eujuyr epmiaqdfjtzqqtixadpuitxzvupltyikigol exjdbg,
-1.7177E+18,2023-04-19T15:14:45.904819+00:00,cqnzjkkerbtppocttzpyubfastswsuwavbnqqanaysaoxa ddz,
-2.30855E+18,2023-04-19T15:14:45.904819+00:00,fqidr kcmltwfnzejuigwpalgwzhbfnolokvmfxzhbofaofior,
-1.86142E+18,2023-04-19T15:14:45.904819+00:00,olathpeoblzhejswcvmbxtvjeepyfjjobqrhwcxrqbunjoeddc,
-2.88792E+18,2023-04-19T15:14:45.904819+00:00,uf jljvcrbtnkrcebwfuvxey knnjabarpjacypegnqpmzhrff,

docs/modules/indexes/document_loaders/examples/example_data/fake_discord_data/package/messages/c280973436971515906/messages.csv

@@ -1,6 +0,0 @@
-ID,Timestamp,Contents,Attachments
-2.79079E+18,2023-04-19T15:14:45.904819+00:00,cl iqaczcrrlprzvbdtvpmduzrdlmtquejjhjfjnt zdsqyksh,
-1.51164E+18,2023-04-19T15:14:45.904819+00:00,ywvnjmtybk f ghdagriyswf exupccijgl calztfvujxhujt,
-1.66032E+18,2023-04-19T15:14:45.904819+00:00,trxcvlcersrdnqzqzfvrrzehmpekrsdtkbovvagsdlcwqokckq,
-2.86805E+18,2023-04-19T15:14:45.904819+00:00,qnkkqjwmwtiqggfko hxzufqnrvpionnglpppuncyswnjibdda,
-3.04157E+18,2023-04-19T15:14:45.904819+00:00,nn vitqoscgsiauiezyyficcbgnjyhaujvthdydmoeistkyskl,

docs/modules/indexes/document_loaders/examples/example_data/whatsapp_chat.txt

@@ -8,5 +8,4 @@
 1/23/23, 3:02 AM - User 1: I thought you were selling the blue one!
 1/23/23, 3:18 AM - User 2: No Im sorry it was my mistake, the blue one is not for sale
 1/23/23, 3:19 AM - User 1: Oh no worries! Bye
-1/23/23, 3:19 AM - User 2: Bye!
-1/23/23, 3:22_AM - User 1: And let me know if anything changes
+1/23/23, 3:19 AM - User 2: Bye!

docs/modules/indexes/document_loaders/examples/git.ipynb

@@ -1,192 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Git\n",
-    "\n",
-    "This notebook shows how to load text files from Git repository."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load existing repository from disk"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from git import Repo\n",
-    "\n",
-    "repo = Repo.clone_from(\n",
-    "    \"https://github.com/hwchase17/langchain\", to_path=\"./example_data/test_repo1\"\n",
-    ")\n",
-    "branch = repo.head.reference"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import GitLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = GitLoader(repo_path=\"./example_data/test_repo1/\", branch=branch)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "data = loader.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "len(data)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "page_content='.venv\\n.github\\n.git\\n.mypy_cache\\n.pytest_cache\\nDockerfile' metadata={'file_path': '.dockerignore', 'file_name': '.dockerignore', 'file_type': ''}\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(data[0])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Clone repository from url"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import GitLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = GitLoader(\n",
-    "    clone_url=\"https://github.com/hwchase17/langchain\",\n",
-    "    repo_path=\"./example_data/test_repo2/\",\n",
-    "    branch=\"master\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "data = loader.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "1074"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "len(data)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Filtering files to load"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import GitLoader\n",
-    "\n",
-    "# eg. loading only python files\n",
-    "loader = GitLoader(repo_path=\"./example_data/test_repo1/\", file_filter=lambda file_path: file_path.endswith(\".py\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/modules/indexes/document_loaders/examples/googledrive.ipynb

@@ -16,7 +16,7 @@
     "1. `pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib`\n",
     "\n",
     "## 🧑 Instructions for ingesting your Google Docs data\n",
-    "By default, the `GoogleDriveLoader` expects the `credentials.json` file to be `~/.credentials/credentials.json`, but this is configurable using the `credentials_path` keyword argument. Same thing with `token.json` - `token_path`. Note that `token.json` will be created automatically the first time you use the loader.\n",
+    "By default, the `GoogleDriveLoader` expects the `credentials.json` file to be `~/.credentials/credentials.json`, but this is configurable using the `credentials_file` keyword argument. Same thing with `token.json`. Note that `token.json` will be created automatically the first time you use the loader.\n",
     "\n",
     "`GoogleDriveLoader` can load from a list of Google Docs document ids or a folder id. You can obtain your folder and document id from the URL:\n",
     "* Folder: https://drive.google.com/drive/u/0/folders/1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5 -> folder id is `\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\"`\n",
@@ -44,11 +44,7 @@
    },
    "outputs": [],
    "source": [
-    "loader = GoogleDriveLoader(\n",
-    "    folder_id=\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\",\n",
-    "    # Optional: configure whether to recursively fetch files from subfolders. Defaults to False.\n",
-    "    recursive=False\n",
-    ")"
+    "loader = GoogleDriveLoader(folder_id=\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\")"
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/hugging_face_dataset.ipynb

@@ -1,220 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "04c9fdc5",
-   "metadata": {},
-   "source": [
-    "# HuggingFace dataset loader \n",
-    "\n",
-    "This notebook shows how to load Hugging Face Hub datasets to LangChain.\n",
-    "\n",
-    "The Hugging Face Hub hosts a large number of community-curated datasets for a diverse range of tasks such as translation, automatic speech recognition, and image classification.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "1815c866",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import HuggingFaceDatasetLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "3611e092",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dataset_name=\"imdb\"\n",
-    "page_content_column=\"text\"\n",
-    "\n",
-    "\n",
-    "loader=HuggingFaceDatasetLoader(dataset_name,page_content_column)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "5e903ebc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "data = loader.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "e8559946",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered \"controversial\" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\\'t have much of a plot.', metadata={'label': 0}),\n",
-       " Document(page_content='\"I Am Curious: Yellow\" is a risible and pretentious steaming pile. It doesn\\'t matter what one\\'s political views are because this film can hardly be taken seriously on any level. As for the claim that frontal male nudity is an automatic NC-17, that isn\\'t true. I\\'ve seen R-rated films with male nudity. Granted, they only offer some fleeting views, but where are the R-rated films with gaping vulvas and flapping labia? Nowhere, because they don\\'t exist. The same goes for those crappy cable shows: schlongs swinging in the breeze but not a clitoris in sight. And those pretentious indie movies like The Brown Bunny, in which we\\'re treated to the site of Vincent Gallo\\'s throbbing johnson, but not a trace of pink visible on Chloe Sevigny. Before crying (or implying) \"double-standard\" in matters of nudity, the mentally obtuse should take into account one unavoidably obvious anatomical difference between men and women: there are no genitals on display when actresses appears nude, and the same cannot be said for a man. In fact, you generally won\\'t see female genitals in an American film in anything short of porn or explicit erotica. This alleged double-standard is less a double standard than an admittedly depressing ability to come to terms culturally with the insides of women\\'s bodies.', metadata={'label': 0}),\n",
-       " Document(page_content=\"If only to avoid making this type of film in the future. This film is interesting as an experiment but tells no cogent story.<br /><br />One might feel virtuous for sitting thru it because it touches on so many IMPORTANT issues but it does so without any discernable motive. The viewer comes away with no new perspectives (unless one comes up with one while one's mind wanders, as it will invariably do during this pointless film).<br /><br />One might better spend one's time staring out a window at a tree growing.<br /><br />\", metadata={'label': 0}),\n",
-       " Document(page_content=\"This film was probably inspired by Godard's Masculin, féminin and I urge you to see that film instead.<br /><br />The film has two strong elements and those are, (1) the realistic acting (2) the impressive, undeservedly good, photo. Apart from that, what strikes me most is the endless stream of silliness. Lena Nyman has to be most annoying actress in the world. She acts so stupid and with all the nudity in this film,...it's unattractive. Comparing to Godard's film, intellectuality has been replaced with stupidity. Without going too far on this subject, I would say that follows from the difference in ideals between the French and the Swedish society.<br /><br />A movie of its time, and place. 2/10.\", metadata={'label': 0}),\n",
-       " Document(page_content='Oh, brother...after hearing about this ridiculous film for umpteen years all I can think of is that old Peggy Lee song..<br /><br />\"Is that all there is??\" ...I was just an early teen when this smoked fish hit the U.S. I was too young to get in the theater (although I did manage to sneak into \"Goodbye Columbus\"). Then a screening at a local film museum beckoned - Finally I could see this film, except now I was as old as my parents were when they schlepped to see it!!<br /><br />The ONLY reason this film was not condemned to the anonymous sands of time was because of the obscenity case sparked by its U.S. release. MILLIONS of people flocked to this stinker, thinking they were going to see a sex film...Instead, they got lots of closeups of gnarly, repulsive Swedes, on-street interviews in bland shopping malls, asinie political pretension...and feeble who-cares simulated sex scenes with saggy, pale actors.<br /><br />Cultural icon, holy grail, historic artifact..whatever this thing was, shred it, burn it, then stuff the ashes in a lead box!<br /><br />Elite esthetes still scrape to find value in its boring pseudo revolutionary political spewings..But if it weren\\'t for the censorship scandal, it would have been ignored, then forgotten.<br /><br />Instead, the \"I Am Blank, Blank\" rhythymed title was repeated endlessly for years as a titilation for porno films (I am Curious, Lavender - for gay films, I Am Curious, Black - for blaxploitation films, etc..) and every ten years or so the thing rises from the dead, to be viewed by a new generation of suckers who want to see that \"naughty sex film\" that \"revolutionized the film industry\"...<br /><br />Yeesh, avoid like the plague..Or if you MUST see it - rent the video and fast forward to the \"dirty\" parts, just to get it over with.<br /><br />', metadata={'label': 0}),\n",
-       " Document(page_content=\"I would put this at the top of my list of films in the category of unwatchable trash! There are films that are bad, but the worst kind are the ones that are unwatchable but you are suppose to like them because they are supposed to be good for you! The sex sequences, so shocking in its day, couldn't even arouse a rabbit. The so called controversial politics is strictly high school sophomore amateur night Marxism. The film is self-consciously arty in the worst sense of the term. The photography is in a harsh grainy black and white. Some scenes are out of focus or taken from the wrong angle. Even the sound is bad! And some people call this art?<br /><br />\", metadata={'label': 0}),\n",
-       " Document(page_content=\"Whoever wrote the screenplay for this movie obviously never consulted any books about Lucille Ball, especially her autobiography. I've never seen so many mistakes in a biopic, ranging from her early years in Celoron and Jamestown to her later years with Desi. I could write a whole list of factual errors, but it would go on for pages. In all, I believe that Lucille Ball is one of those inimitable people who simply cannot be portrayed by anyone other than themselves. If I were Lucie Arnaz and Desi, Jr., I would be irate at how many mistakes were made in this film. The filmmakers tried hard, but the movie seems awfully sloppy to me.\", metadata={'label': 0}),\n",
-       " Document(page_content='When I first saw a glimpse of this movie, I quickly noticed the actress who was playing the role of Lucille Ball. Rachel York\\'s portrayal of Lucy is absolutely awful. Lucille Ball was an astounding comedian with incredible talent. To think about a legend like Lucille Ball being portrayed the way she was in the movie is horrendous. I cannot believe out of all the actresses in the world who could play a much better Lucy, the producers decided to get Rachel York. She might be a good actress in other roles but to play the role of Lucille Ball is tough. It is pretty hard to find someone who could resemble Lucille Ball, but they could at least find someone a bit similar in looks and talent. If you noticed York\\'s portrayal of Lucy in episodes of I Love Lucy like the chocolate factory or vitavetavegamin, nothing is similar in any way-her expression, voice, or movement.<br /><br />To top it all off, Danny Pino playing Desi Arnaz is horrible. Pino does not qualify to play as Ricky. He\\'s small and skinny, his accent is unreal, and once again, his acting is unbelievable. Although Fred and Ethel were not similar either, they were not as bad as the characters of Lucy and Ricky.<br /><br />Overall, extremely horrible casting and the story is badly told. If people want to understand the real life situation of Lucille Ball, I suggest watching A&E Biography of Lucy and Desi, read the book from Lucille Ball herself, or PBS\\' American Masters: Finding Lucy. If you want to see a docudrama, \"Before the Laughter\" would be a better choice. The casting of Lucille Ball and Desi Arnaz in \"Before the Laughter\" is much better compared to this. At least, a similar aspect is shown rather than nothing.', metadata={'label': 0}),\n",
-       " Document(page_content='Who are these \"They\"- the actors? the filmmakers? Certainly couldn\\'t be the audience- this is among the most air-puffed productions in existence. It\\'s the kind of movie that looks like it was a lot of fun to shoot\\x97 TOO much fun, nobody is getting any actual work done, and that almost always makes for a movie that\\'s no fun to watch.<br /><br />Ritter dons glasses so as to hammer home his character\\'s status as a sort of doppleganger of the bespectacled Bogdanovich; the scenes with the breezy Ms. Stratten are sweet, but have an embarrassing, look-guys-I\\'m-dating-the-prom-queen feel to them. Ben Gazzara sports his usual cat\\'s-got-canary grin in a futile attempt to elevate the meager plot, which requires him to pursue Audrey Hepburn with all the interest of a narcoleptic at an insomnia clinic. In the meantime, the budding couple\\'s respective children (nepotism alert: Bogdanovich\\'s daughters) spew cute and pick up some fairly disturbing pointers on \\'love\\' while observing their parents. (Ms. Hepburn, drawing on her dignity, manages to rise above the proceedings- but she has the monumental challenge of playing herself, ostensibly.) Everybody looks great, but so what? It\\'s a movie and we can expect that much, if that\\'s what you\\'re looking for you\\'d be better off picking up a copy of Vogue.<br /><br />Oh- and it has to be mentioned that Colleen Camp thoroughly annoys, even apart from her singing, which, while competent, is wholly unconvincing... the country and western numbers are woefully mismatched with the standards on the soundtrack. Surely this is NOT what Gershwin (who wrote the song from which the movie\\'s title is derived) had in mind; his stage musicals of the 20\\'s may have been slight, but at least they were long on charm. \"They All Laughed\" tries to coast on its good intentions, but nobody- least of all Peter Bogdanovich - has the good sense to put on the brakes.<br /><br />Due in no small part to the tragic death of Dorothy Stratten, this movie has a special place in the heart of Mr. Bogdanovich- he even bought it back from its producers, then distributed it on his own and went bankrupt when it didn\\'t prove popular. His rise and fall is among the more sympathetic and tragic of Hollywood stories, so there\\'s no joy in criticizing the film... there _is_ real emotional investment in Ms. Stratten\\'s scenes. But \"Laughed\" is a faint echo of \"The Last Picture Show\", \"Paper Moon\" or \"What\\'s Up, Doc\"- following \"Daisy Miller\" and \"At Long Last Love\", it was a thundering confirmation of the phase from which P.B. has never emerged.<br /><br />All in all, though, the movie is harmless, only a waste of rental. I want to watch people having a good time, I\\'ll go to the park on a sunny day. For filmic expressions of joy and love, I\\'ll stick to Ernest Lubitsch and Jaques Demy...', metadata={'label': 0}),\n",
-       " Document(page_content=\"This is said to be a personal film for Peter Bogdonavitch. He based it on his life but changed things around to fit the characters, who are detectives. These detectives date beautiful models and have no problem getting them. Sounds more like a millionaire playboy filmmaker than a detective, doesn't it? This entire movie was written by Peter, and it shows how out of touch with real people he was. You're supposed to write what you know, and he did that, indeed. And leaves the audience bored and confused, and jealous, for that matter. This is a curio for people who want to see Dorothy Stratten, who was murdered right after filming. But Patti Hanson, who would, in real life, marry Keith Richards, was also a model, like Stratten, but is a lot better and has a more ample part. In fact, Stratten's part seemed forced; added. She doesn't have a lot to do with the story, which is pretty convoluted to begin with. All in all, every character in this film is somebody that very few people can relate with, unless you're millionaire from Manhattan with beautiful supermodels at your beckon call. For the rest of us, it's an irritating snore fest. That's what happens when you're out of touch. You entertain your few friends with inside jokes, and bore all the rest.\", metadata={'label': 0}),\n",
-       " Document(page_content='It was great to see some of my favorite stars of 30 years ago including John Ritter, Ben Gazarra and Audrey Hepburn. They looked quite wonderful. But that was it. They were not given any characters or good lines to work with. I neither understood or cared what the characters were doing.<br /><br />Some of the smaller female roles were fine, Patty Henson and Colleen Camp were quite competent and confident in their small sidekick parts. They showed some talent and it is sad they didn\\'t go on to star in more and better films. Sadly, I didn\\'t think Dorothy Stratten got a chance to act in this her only important film role.<br /><br />The film appears to have some fans, and I was very open-minded when I started watching it. I am a big Peter Bogdanovich fan and I enjoyed his last movie, \"Cat\\'s Meow\" and all his early ones from \"Targets\" to \"Nickleodeon\". So, it really surprised me that I was barely able to keep awake watching this one.<br /><br />It is ironic that this movie is about a detective agency where the detectives and clients get romantically involved with each other. Five years later, Bogdanovich\\'s ex-girlfriend, Cybil Shepherd had a hit television series called \"Moonlighting\" stealing the story idea from Bogdanovich. Of course, there was a great difference in that the series relied on tons of witty dialogue, while this tries to make do with slapstick and a few screwball lines.<br /><br />Bottom line: It ain\\'t no \"Paper Moon\" and only a very pale version of \"What\\'s Up, Doc\".', metadata={'label': 0}),\n",
-       " Document(page_content=\"I can't believe that those praising this movie herein aren't thinking of some other film. I was prepared for the possibility that this would be awful, but the script (or lack thereof) makes for a film that's also pointless. On the plus side, the general level of craft on the part of the actors and technical crew is quite competent, but when you've got a sow's ear to work with you can't make a silk purse. Ben G fans should stick with just about any other movie he's been in. Dorothy S fans should stick to Galaxina. Peter B fans should stick to Last Picture Show and Target. Fans of cheap laughs at the expense of those who seem to be asking for it should stick to Peter B's amazingly awful book, Killing of the Unicorn.\", metadata={'label': 0}),\n",
-       " Document(page_content='Never cast models and Playboy bunnies in your films! Bob Fosse\\'s \"Star 80\" about Dorothy Stratten, of whom Bogdanovich was obsessed enough to have married her SISTER after her murder at the hands of her low-life husband, is a zillion times more interesting than Dorothy herself on the silver screen. Patty Hansen is no actress either..I expected to see some sort of lost masterpiece a la Orson Welles but instead got Audrey Hepburn cavorting in jeans and a god-awful \"poodlesque\" hair-do....Very disappointing....\"Paper Moon\" and \"The Last Picture Show\" I could watch again and again. This clunker I could barely sit through once. This movie was reputedly not released because of the brouhaha surrounding Ms. Stratten\\'s tawdry death; I think the real reason was because it was so bad!', metadata={'label': 0}),\n",
-       " Document(page_content=\"Its not the cast. A finer group of actors, you could not find. Its not the setting. The director is in love with New York City, and by the end of the film, so are we all! Woody Allen could not improve upon what Bogdonovich has done here. If you are going to fall in love, or find love, Manhattan is the place to go. No, the problem with the movie is the script. There is none. The actors fall in love at first sight, words are unnecessary. In the director's own experience in Hollywood that is what happens when they go to work on the set. It is reality to him, and his peers, but it is a fantasy to most of us in the real world. So, in the end, the movie is hollow, and shallow, and message-less.\", metadata={'label': 0}),\n",
-       " Document(page_content='Today I found \"They All Laughed\" on VHS on sale in a rental. It was a really old and very used VHS, I had no information about this movie, but I liked the references listed on its cover: the names of Peter Bogdanovich, Audrey Hepburn, John Ritter and specially Dorothy Stratten attracted me, the price was very low and I decided to risk and buy it. I searched IMDb, and the User Rating of 6.0 was an excellent reference. I looked in \"Mick Martin & Marsha Porter Video & DVD Guide 2003\" and \\x96 wow \\x96 four stars! So, I decided that I could not waste more time and immediately see it. Indeed, I have just finished watching \"They All Laughed\" and I found it a very boring overrated movie. The characters are badly developed, and I spent lots of minutes to understand their roles in the story. The plot is supposed to be funny (private eyes who fall in love for the women they are chasing), but I have not laughed along the whole story. The coincidences, in a huge city like New York, are ridiculous. Ben Gazarra as an attractive and very seductive man, with the women falling for him as if her were a Brad Pitt, Antonio Banderas or George Clooney, is quite ridiculous. In the end, the greater attractions certainly are the presence of the Playboy centerfold and playmate of the year Dorothy Stratten, murdered by her husband pretty after the release of this movie, and whose life was showed in \"Star 80\" and \"Death of a Centerfold: The Dorothy Stratten Story\"; the amazing beauty of the sexy Patti Hansen, the future Mrs. Keith Richards; the always wonderful, even being fifty-two years old, Audrey Hepburn; and the song \"Amigo\", from Roberto Carlos. Although I do not like him, Roberto Carlos has been the most popular Brazilian singer since the end of the 60\\'s and is called by his fans as \"The King\". I will keep this movie in my collection only because of these attractions (manly Dorothy Stratten). My vote is four.<br /><br />Title (Brazil): \"Muito Riso e Muita Alegria\" (\"Many Laughs and Lots of Happiness\")', metadata={'label': 0})]"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "data[:15]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "021bc377",
-   "metadata": {},
-   "source": [
-    "### Example \n",
-    "In this example, we use data from a dataset to answer a question"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "d924885c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.indexes import VectorstoreIndexCreator\n",
-    "from langchain.document_loaders.hugging_face_dataset import HuggingFaceDatasetLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "f94ce6a3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dataset_name=\"tweet_eval\"\n",
-    "page_content_column=\"text\"\n",
-    "name=\"stance_climate\"\n",
-    "\n",
-    "\n",
-    "loader=HuggingFaceDatasetLoader(dataset_name,page_content_column,name)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "abb51899",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Found cached dataset tweet_eval\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "4b10969d08df4e6792eaafc6d41fe366",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "  0%|          | 0/3 [00:00<?, ?it/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Using embedded DuckDB without persistence: data will be transient\n"
-     ]
-    }
-   ],
-   "source": [
-    "index = VectorstoreIndexCreator().from_loaders([loader])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "c0108277",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "query = \"What are the most used hashtag?\"\n",
-    "result = index.query(query)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "548b6e56",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' The most used hashtags in this context are #UKClimate2015, #Sustainability, #TakeDownTheFlag, #LoveWins, #CSOTA, #ClimateSummitoftheAmericas, #SM, and #SocialMedia.'"
-      ]
-     },
-     "execution_count": 30,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "result"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "89c30c2d",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/indexes/document_loaders/examples/obsidian.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "1dc7df1d",
    "metadata": {},
@@ -9,9 +8,7 @@
     "# Obsidian\n",
     "This notebook covers how to load documents from an Obsidian database.\n",
     "\n",
-    "Since Obsidian is just stored on disk as a folder of Markdown files, the loader just takes a path to this directory.\n",
-    "\n",
-    "Obsidian files also sometimes contain [metadata](https://help.obsidian.md/Editing+and+formatting/Metadata) which is a YAML block at the top of the file. These values will be added to the document's metadata. (`ObsidianLoader` can also be passed a `collect_metadata=False` argument to disable this behavior.)"
+    "Since Obsidian is just stored on disk as a folder of Markdown files, the loader just takes a path to this directory."
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/pdf.ipynb

@@ -376,7 +376,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 3,
    "id": "a5525fb0",
    "metadata": {},
    "outputs": [],
@@ -386,115 +386,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "id": "dac7ff68",
    "metadata": {},
    "outputs": [],
    "source": [
-    "data = loader.load()[0]   # entire pdf is loaded as a single Document"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "0ba9f645",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from bs4 import BeautifulSoup\n",
-    "soup = BeautifulSoup(data.page_content,'html.parser')\n",
-    "content = soup.find_all('div')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "35304e21",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import re\n",
-    "cur_fs = None\n",
-    "cur_text = ''\n",
-    "snippets = []   # first collect all snippets that have the same font size\n",
-    "for c in content:\n",
-    "    sp = c.find('span')\n",
-    "    if not sp:\n",
-    "        continue\n",
-    "    st = sp.get('style')\n",
-    "    if not st:\n",
-    "        continue\n",
-    "    fs = re.findall('font-size:(\\d+)px',st)\n",
-    "    if not fs:\n",
-    "        continue\n",
-    "    fs = int(fs[0])\n",
-    "    if not cur_fs:\n",
-    "        cur_fs = fs\n",
-    "    if fs == cur_fs:\n",
-    "        cur_text += c.text\n",
-    "    else:\n",
-    "        snippets.append((cur_text,cur_fs))\n",
-    "        cur_fs = fs\n",
-    "        cur_text = c.text\n",
-    "snippets.append((cur_text,cur_fs))\n",
-    "# Note: The above logic is very straightforward. One can also add more strategies such as removing duplicate snippets (as\n",
-    "# headers/footers in a PDF appear on multiple pages so if we find duplicatess safe to assume that it is redundant info)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "af8adf2f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.docstore.document import Document\n",
-    "cur_idx = -1\n",
-    "semantic_snippets = []\n",
-    "# Assumption: headings have higher font size than their respective content\n",
-    "for s in snippets:\n",
-    "    # if current snippet's font size > previous section's heading => it is a new heading\n",
-    "    if not semantic_snippets or s[1] > semantic_snippets[cur_idx].metadata['heading_font']:\n",
-    "        metadata={'heading':s[0], 'content_font': 0, 'heading_font': s[1]}\n",
-    "        metadata.update(data.metadata)\n",
-    "        semantic_snippets.append(Document(page_content='',metadata=metadata))\n",
-    "        cur_idx += 1\n",
-    "        continue\n",
-    "    \n",
-    "    # if current snippet's font size <= previous section's content => content belongs to the same section (one can also create\n",
-    "    # a tree like structure for sub sections if needed but that may require some more thinking and may be data specific)\n",
-    "    if not semantic_snippets[cur_idx].metadata['content_font'] or s[1] <= semantic_snippets[cur_idx].metadata['content_font']:\n",
-    "        semantic_snippets[cur_idx].page_content += s[0]\n",
-    "        semantic_snippets[cur_idx].metadata['content_font'] = max(s[1], semantic_snippets[cur_idx].metadata['content_font'])\n",
-    "        continue\n",
-    "    \n",
-    "    # if current snippet's font size > previous section's content but less tha previous section's heading than also make a new \n",
-    "    # section (e.g. title of a pdf will have the highest font size but we don't want it to subsume all sections)\n",
-    "    metadata={'heading':s[0], 'content_font': 0, 'heading_font': s[1]}\n",
-    "    metadata.update(data.metadata)\n",
-    "    semantic_snippets.append(Document(page_content='',metadata=metadata))\n",
-    "    cur_idx += 1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "db7f6674",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "Document(page_content='Recently, various DL models and datasets have been developed for layout analysis\\ntasks. The dhSegment [22] utilizes fully convolutional networks [20] for segmen-\\ntation tasks on historical documents. Object detection-based methods like Faster\\nR-CNN [28] and Mask R-CNN [12] are used for identifying document elements [38]\\nand detecting tables [30, 26]. Most recently, Graph Neural Networks [29] have also\\nbeen used in table detection [27]. However, these models are usually implemented\\nindividually and there is no unified framework to load and use such models.\\nThere has been a surge of interest in creating open-source tools for document\\nimage processing: a search of document image analysis in Github leads to 5M\\nrelevant code pieces 6; yet most of them rely on traditional rule-based methods\\nor provide limited functionalities. The closest prior research to our work is the\\nOCR-D project7, which also tries to build a complete toolkit for DIA. However,\\nsimilar to the platform developed by Neudecker et al. [21], it is designed for\\nanalyzing historical documents, and provides no supports for recent DL models.\\nThe DocumentLayoutAnalysis project8 focuses on processing born-digital PDF\\ndocuments via analyzing the stored PDF data. Repositories like DeepLayout9\\nand Detectron2-PubLayNet10 are individual deep learning models trained on\\nlayout analysis datasets without support for the full DIA pipeline. The Document\\nAnalysis and Exploitation (DAE) platform [15] and the DeepDIVA project [2]\\naim to improve the reproducibility of DIA methods (or DL models), yet they\\nare not actively maintained. OCR engines like Tesseract [14], easyOCR11 and\\npaddleOCR12 usually do not come with comprehensive functionalities for other\\nDIA tasks like layout analysis.\\nRecent years have also seen numerous efforts to create libraries for promoting\\nreproducibility and reusability in the field of DL. Libraries like Dectectron2 [35],\\n6 The number shown is obtained by specifying the search type as ‘code’.\\n7 https://ocr-d.de/en/about\\n8 https://github.com/BobLd/DocumentLayoutAnalysis\\n9 https://github.com/leonlulu/DeepLayout\\n10 https://github.com/hpanwar08/detectron2\\n11 https://github.com/JaidedAI/EasyOCR\\n12 https://github.com/PaddlePaddle/PaddleOCR\\n4\\nZ. Shen et al.\\nFig. 1: The overall architecture of LayoutParser. For an input document image,\\nthe core LayoutParser library provides a set of off-the-shelf tools for layout\\ndetection, OCR, visualization, and storage, backed by a carefully designed layout\\ndata structure. LayoutParser also supports high level customization via efficient\\nlayout annotation and model training functions. These improve model accuracy\\non the target samples. The community platform enables the easy sharing of DIA\\nmodels and whole digitization pipelines to promote reusability and reproducibility.\\nA collection of detailed documentation, tutorials and exemplar projects make\\nLayoutParser easy to learn and use.\\nAllenNLP [8] and transformers [34] have provided the community with complete\\nDL-based support for developing and deploying models for general computer\\nvision and natural language processing problems. LayoutParser, on the other\\nhand, specializes specifically in DIA tasks. LayoutParser is also equipped with a\\ncommunity platform inspired by established model hubs such as Torch Hub [23]\\nand TensorFlow Hub [1]. It enables the sharing of pretrained models as well as\\nfull document processing pipelines that are unique to DIA tasks.\\nThere have been a variety of document data collections to facilitate the\\ndevelopment of DL models. Some examples include PRImA [3](magazine layouts),\\nPubLayNet [38](academic paper layouts), Table Bank [18](tables in academic\\npapers), Newspaper Navigator Dataset [16, 17](newspaper figure layouts) and\\nHJDataset [31](historical Japanese document layouts). A spectrum of models\\ntrained on these datasets are currently available in the LayoutParser model zoo\\nto support different use cases.\\n', metadata={'heading': '2 Related Work\\n', 'content_font': 9, 'heading_font': 11, 'source': 'example_data/layout-parser-paper.pdf'})"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "semantic_snippets[4]"
+    "data = loader.load()"
    ]
   },
   {
@@ -577,9 +474,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "langchain_dev",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "langchain_dev"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {

docs/modules/indexes/document_loaders/examples/readthedocs_documentation.ipynb

@@ -40,7 +40,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "loader = ReadTheDocsLoader(\"rtdocs\", features='html.parser')"
+    "loader = ReadTheDocsLoader(\"rtdocs\")"
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/slack_directory.ipynb

@@ -1,81 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "1dc7df1d",
-   "metadata": {},
-   "source": [
-    "# Slack (Local Exported Zipfile)\n",
-    "\n",
-    "This notebook covers how to load documents from a Zipfile generated from a Slack export.\n",
-    "\n",
-    "In order to get this Slack export, follow these instructions:\n",
-    "\n",
-    "## 🧑 Instructions for ingesting your own dataset\n",
-    "\n",
-    "Export your Slack data. You can do this by going to your Workspace Management page and clicking the Import/Export option ({your_slack_domain}.slack.com/services/export). Then, choose the right date range and click `Start export`. Slack will send you an email and a DM when the export is ready.\n",
-    "\n",
-    "The download will produce a `.zip` file in your Downloads folder (or wherever your downloads can be found, depending on your OS configuration).\n",
-    "\n",
-    "Copy the path to the `.zip` file, and assign it as `LOCAL_ZIPFILE` below."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "007c5cbf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import SlackDirectoryLoader "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a1caec59",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Optionally set your Slack URL. This will give you proper URLs in the docs sources.\n",
-    "SLACK_WORKSPACE_URL = \"https://xxx.slack.com\"\n",
-    "LOCAL_ZIPFILE = \"\" # Paste the local paty to your Slack zip file here.\n",
-    "\n",
-    "loader = SlackDirectoryLoader(LOCAL_ZIPFILE, SLACK_WORKSPACE_URL)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b1c30ff7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs = loader.load()\n",
-    "docs"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/indexes/document_loaders/examples/twitter.ipynb

@@ -1,114 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "66a7777e",
-   "metadata": {},
-   "source": [
-    "# Twitter\n",
-    "\n",
-    "This loader fetches the text from the Tweets of a list of Twitter users, using the `tweepy` Python package.\n",
-    "You must initialize the loader with your Twitter API token, and you need to pass in the Twitter username you want to extract."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "9ec8a3b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import TwitterTweetLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "43128d8d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#!pip install tweepy"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "35d6809a",
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "loader = TwitterTweetLoader.from_bearer_token(\n",
-    "    oauth2_bearer_token=\"YOUR BEARER TOKEN\",\n",
-    "    twitter_users=['elonmusk'],\n",
-    "    number_tweets=50,  # Default value is 100\n",
-    ")\n",
-    "\n",
-    "# Or load from access token and consumer keys\n",
-    "# loader = TwitterTweetLoader.from_secrets(\n",
-    "#     access_token='YOUR ACCESS TOKEN',\n",
-    "#     access_token_secret='YOUR ACCESS TOKEN SECRET',\n",
-    "#     consumer_key='YOUR CONSUMER KEY',\n",
-    "#     consumer_secret='YOUR CONSUMER SECRET',\n",
-    "#     twitter_users=['elonmusk'],\n",
-    "#     number_tweets=50,\n",
-    "# )"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "05fe33b9",
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='@MrAndyNgo @REI One store after another shutting down', metadata={'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
-       " Document(page_content='@KanekoaTheGreat @joshrogin @glennbeck Large ships are fundamentally vulnerable to ballistic (hypersonic) missiles', metadata={'created_at': 'Tue Apr 18 03:43:25 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
-       " Document(page_content='@KanekoaTheGreat The Golden Rule', metadata={'created_at': 'Tue Apr 18 03:37:17 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
-       " Document(page_content='@KanekoaTheGreat 🧐', metadata={'created_at': 'Tue Apr 18 03:35:48 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
-       " Document(page_content='@TRHLofficial What’s he talking about and why is it sponsored by Erik’s son?', metadata={'created_at': 'Tue Apr 18 03:32:17 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}})]"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "documents = loader.load()\n",
-    "documents[:5]"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/indexes/document_loaders/examples/unstructured_file.ipynb

@@ -163,7 +163,7 @@
    "source": [
     "## Define a Partitioning Strategy\n",
     "\n",
-    "Unstructured document loader allow users to pass in a `strategy` parameter that lets `unstructured` know how to partition the document. Currently supported strategies are `\"hi_res\"` (the default) and `\"fast\"`. Hi res partitioning strategies are more accurate, but take longer to process. Fast strategies partition the document more quickly, but trade-off accuracy. Not all document types have separate hi res and fast partitioning strategies. For those document types, the `strategy` kwarg is ignored. In some cases, the high res strategy will fallback to fast if there is a dependency missing (i.e. a model for document partitioning). You can see how to apply a strategy to an `UnstructuredFileLoader` below."
+    "Unstructured document loader allow users to pass in a `strategy` parameter that lets `unstructured` know how to partitioning the document. Currently supported strategies are `\"hi_res\"` (the default) and `\"fast\"`. Hi res partitioning strategies are more accurate, but take longer to process. Fast strategies partition the document more quickly, but trade-off accuracy. Not all document types have separate hi res and fast partitioning strategies. For those document types, the `strategy` kwarg is ignored. In some cases, the high res strategy will fallback to fast if there is a dependency missing (i.e. a model for document partitioning). You can see how to apply a strategy to an `UnstructuredFileLoader` below."
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/url.ipynb

@@ -112,79 +112,6 @@
    "source": [
     "data = loader.load()"
    ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "a2c1c79f",
-   "metadata": {},
-   "source": [
-    "# Playwright URL Loader\n",
-    "\n",
-    "This covers how to load HTML documents from a list of URLs using the `PlaywrightURLLoader`.\n",
-    "\n",
-    "As in the Selenium case, Playwright allows us to load pages that need JavaScript to render.\n",
-    "\n",
-    "## Setup\n",
-    "\n",
-    "To use the `PlaywrightURLLoader`, you will need to install `playwright` and `unstructured`. Additionally, you will need to install the Playwright Chromium browser:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "53158417",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install playwright\n",
-    "!pip install \"playwright\"\n",
-    "!pip install \"unstructured\"\n",
-    "!playwright install"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0ab4e115",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import PlaywrightURLLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "ce5a9a0a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "urls = [\n",
-    "    \"https://www.youtube.com/watch?v=dQw4w9WgXcQ\",\n",
-    "    \"https://goo.gl/maps/NDSHwePEyaHMFGwh8\"\n",
-    "]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2dc3e0bc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = PlaywrightURLLoader(urls=urls, remove_selectors=[\"header\", \"footer\"])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "10b79f80",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "data = loader.load()"
-   ]
   }
  ],
  "metadata": {
@@ -203,7 +130,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.6"
+   "version": "3.8.13"
   }
  },
  "nbformat": 4,

docs/modules/indexes/document_loaders/examples/web_base.ipynb

@@ -89,7 +89,7 @@
    "id": "150988e6",
    "metadata": {},
    "source": [
-    "## Loading multiple webpages\n",
+    "# Loading multiple webpages\n",
     "\n",
     "You can also load multiple webpages at once by passing in a list of urls to the loader. This will return a list of documents in the same order as the urls passed in."
    ]
@@ -123,7 +123,7 @@
    "id": "641be294",
    "metadata": {},
    "source": [
-    "### Load multiple urls concurrently\n",
+    "## Load multiple urls concurrently\n",
     "\n",
     "You can speed up the scraping process by scraping and parsing multiple urls concurrently.\n",
     "\n",

docs/modules/indexes/getting_started.ipynb

@@ -99,7 +99,7 @@
    "outputs": [],
    "source": [
     "from langchain.document_loaders import TextLoader\n",
-    "loader = TextLoader('../state_of_the_union.txt', encoding='utf8')"
+    "loader = TextLoader('../state_of_the_union.txt')"
    ]
   },
   {

docs/modules/indexes/retrievers/examples/contextual-compression.ipynb

@@ -1,371 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "fc0db1bc",
-   "metadata": {},
-   "source": [
-    "# Contextual Compression Retriever\n",
-    "\n",
-    "This notebook introduces the concept of DocumentCompressors and the ContextualCompressionRetriever. The core idea is simple: given a specific query, we should be able to return only the documents relevant to that query, and only the parts of those documents that are relevant. The ContextualCompressionsRetriever is a wrapper for another retriever that iterates over the initial output of the base retriever and filters and compresses those initial documents, so that only the most relevant information is returned."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "28e8dc12",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Helper function for printing docs\n",
-    "\n",
-    "def pretty_print_docs(docs):\n",
-    "    print(f\"\\n{'-' * 100}\\n\".join([f\"Document {i+1}:\\n\\n\" + d.page_content for i, d in enumerate(docs)]))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6fa3d916",
-   "metadata": {},
-   "source": [
-    "## Using a vanilla vector store retriever\n",
-    "Let's start by initializing a simple vector store retriever and storing the 2023 State of the Union speech (in chunks). We can see that given an example question our retriever returns one or two relevant docs and a few irrelevant docs. And even the relevant docs have a lot of irrelevant information in them."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "9fbcc58f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Document 1:\n",
-      "\n",
-      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
-      "\n",
-      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
-      "\n",
-      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
-      "\n",
-      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 2:\n",
-      "\n",
-      "A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \n",
-      "\n",
-      "And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \n",
-      "\n",
-      "We can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling.  \n",
-      "\n",
-      "We’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.  \n",
-      "\n",
-      "We’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster. \n",
-      "\n",
-      "We’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 3:\n",
-      "\n",
-      "And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong. \n",
-      "\n",
-      "As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. \n",
-      "\n",
-      "While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice. \n",
-      "\n",
-      "And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things. \n",
-      "\n",
-      "So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.  \n",
-      "\n",
-      "First, beat the opioid epidemic.\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 4:\n",
-      "\n",
-      "Tonight, I’m announcing a crackdown on these companies overcharging American businesses and consumers. \n",
-      "\n",
-      "And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.  \n",
-      "\n",
-      "That ends on my watch. \n",
-      "\n",
-      "Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect. \n",
-      "\n",
-      "We’ll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees. \n",
-      "\n",
-      "Let’s pass the Paycheck Fairness Act and paid leave.  \n",
-      "\n",
-      "Raise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty. \n",
-      "\n",
-      "Let’s increase Pell Grants and increase our historic support of HBCUs, and invest in what Jill—our First Lady who teaches full-time—calls America’s best-kept secret: community colleges.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.document_loaders import TextLoader\n",
-    "from langchain.vectorstores import FAISS\n",
-    "\n",
-    "documents = TextLoader('../../../state_of_the_union.txt').load()\n",
-    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
-    "texts = text_splitter.split_documents(documents)\n",
-    "retriever = FAISS.from_documents(texts, OpenAIEmbeddings()).as_retriever()\n",
-    "\n",
-    "docs = retriever.get_relevant_documents(\"What did the president say about Ketanji Brown Jackson\")\n",
-    "pretty_print_docs(docs)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b7648612",
-   "metadata": {},
-   "source": [
-    "## Adding contextual compression with an `LLMChainExtractor`\n",
-    "Now let's wrap our base retriever with a `ContextualCompressionRetriever`. We'll add an `LLMChainExtractor`, which will iterate over the initially returned documents and extract from each only the content that is relevant to the query."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "9a658023",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Document 1:\n",
-      "\n",
-      "\"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
-      "\n",
-      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\"\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 2:\n",
-      "\n",
-      "\"A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\"\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.llms import OpenAI\n",
-    "from langchain.retrievers import ContextualCompressionRetriever\n",
-    "from langchain.retrievers.document_compressors import LLMChainExtractor\n",
-    "\n",
-    "llm = OpenAI(temperature=0)\n",
-    "compressor = LLMChainExtractor.from_llm(llm)\n",
-    "compression_retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever)\n",
-    "\n",
-    "compressed_docs = compression_retriever.get_relevant_documents(\"What did the president say about Ketanji Jackson Brown\")\n",
-    "pretty_print_docs(compressed_docs)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2cd38f3a",
-   "metadata": {},
-   "source": [
-    "## More built-in compressors: filters\n",
-    "### `LLMChainFilter`\n",
-    "The `LLMChainFilter` is slightly simpler but more robust compressor that uses an LLM chain to decide which of the initially retrieved documents to filter out and which ones to return, without manipulating the document contents."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "b216a767",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Document 1:\n",
-      "\n",
-      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
-      "\n",
-      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
-      "\n",
-      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
-      "\n",
-      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.retrievers.document_compressors import LLMChainFilter\n",
-    "\n",
-    "_filter = LLMChainFilter.from_llm(llm)\n",
-    "compression_retriever = ContextualCompressionRetriever(base_compressor=_filter, base_retriever=retriever)\n",
-    "\n",
-    "compressed_docs = compression_retriever.get_relevant_documents(\"What did the president say about Ketanji Jackson Brown\")\n",
-    "pretty_print_docs(compressed_docs)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8c709598",
-   "metadata": {},
-   "source": [
-    "### `EmbeddingsFilter`\n",
-    "\n",
-    "Making an extra LLM call over each retrieved document is expensive and slow. The `EmbeddingsFilter` provides a cheaper and faster option by embedding the documents and query and only returning those documents which have sufficiently similar embeddings to the query."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "6fbc801f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Document 1:\n",
-      "\n",
-      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
-      "\n",
-      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
-      "\n",
-      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
-      "\n",
-      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 2:\n",
-      "\n",
-      "A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \n",
-      "\n",
-      "And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \n",
-      "\n",
-      "We can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling.  \n",
-      "\n",
-      "We’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.  \n",
-      "\n",
-      "We’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster. \n",
-      "\n",
-      "We’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 3:\n",
-      "\n",
-      "And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong. \n",
-      "\n",
-      "As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. \n",
-      "\n",
-      "While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice. \n",
-      "\n",
-      "And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things. \n",
-      "\n",
-      "So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.  \n",
-      "\n",
-      "First, beat the opioid epidemic.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.retrievers.document_compressors import EmbeddingsFilter\n",
-    "\n",
-    "embeddings = OpenAIEmbeddings()\n",
-    "embeddings_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)\n",
-    "compression_retriever = ContextualCompressionRetriever(base_compressor=embeddings_filter, base_retriever=retriever)\n",
-    "\n",
-    "compressed_docs = compression_retriever.get_relevant_documents(\"What did the president say about Ketanji Jackson Brown\")\n",
-    "pretty_print_docs(compressed_docs)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "07365d36",
-   "metadata": {},
-   "source": [
-    "# Stringing compressors and document transformers together\n",
-    "Using the `DocumentCompressorPipeline` we can also easily combine multiple compressors in sequence. Along with compressors we can add `BaseDocumentTransformer`s to our pipeline, which don't perform any contextual compression but simply perform some transformation on a set of documents. For example `TextSplitter`s can be used as document transformers to split documents into smaller pieces, and the `EmbeddingsRedundantFilter` can be used to filter out redundant documents based on embedding similarity between documents.\n",
-    "\n",
-    "Below we create a compressor pipeline by first splitting our docs into smaller chunks, then removing redundant documents, and then filtering based on relevance to the query."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "2a150a63",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_transformers import EmbeddingsRedundantFilter\n",
-    "from langchain.retrievers.document_compressors import DocumentCompressorPipeline\n",
-    "from langchain.text_splitter import CharacterTextSplitter\n",
-    "\n",
-    "splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0, separator=\". \")\n",
-    "redundant_filter = EmbeddingsRedundantFilter(embeddings=embeddings)\n",
-    "relevant_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)\n",
-    "pipeline_compressor = DocumentCompressorPipeline(\n",
-    "    transformers=[splitter, redundant_filter, relevant_filter]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "3ceab64a",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Document 1:\n",
-      "\n",
-      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
-      "\n",
-      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 2:\n",
-      "\n",
-      "As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. \n",
-      "\n",
-      "While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year\n",
-      "----------------------------------------------------------------------------------------------------\n",
-      "Document 3:\n",
-      "\n",
-      "A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder\n"
-     ]
-    }
-   ],
-   "source": [
-    "compression_retriever = ContextualCompressionRetriever(base_compressor=pipeline_compressor, base_retriever=retriever)\n",
-    "\n",
-    "compressed_docs = compression_retriever.get_relevant_documents(\"What did the president say about Ketanji Jackson Brown\")\n",
-    "pretty_print_docs(compressed_docs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "8cfd9fc5",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/modules/indexes/retrievers/examples/metal.ipynb

@@ -32,9 +32,9 @@
     "from metal_sdk.metal import Metal\n",
     "API_KEY = \"\"\n",
     "CLIENT_ID = \"\"\n",
-    "INDEX_ID = \"\"\n",
+    "APP_ID = \"\"\n",
     "\n",
-    "metal = Metal(API_KEY, CLIENT_ID, INDEX_ID);\n"
+    "metal = Metal(API_KEY, CLIENT_ID, APP_ID);\n"
    ]
   },
   {

docs/modules/indexes/retrievers/examples/pinecone_hybrid_search.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "ab66dd43",
    "metadata": {},
@@ -10,12 +9,12 @@
     "\n",
     "This notebook goes over how to use a retriever that under the hood uses Pinecone and Hybrid Search.\n",
     "\n",
-    "The logic of this retriever is taken from [this documentaion](https://docs.pinecone.io/docs/hybrid-search)"
+    "The logic of this retriever is largely taken from [this blog post](https://www.pinecone.io/learn/hybrid-search-intro/)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 75,
+   "execution_count": 1,
    "id": "393ac030",
    "metadata": {},
    "outputs": [],
@@ -32,61 +31,43 @@
    ]
   },
   {
-   "attachments": {},
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "15390796",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pinecone  # !pip install pinecone-client\n",
+    "\n",
+    "pinecone.init(\n",
+    "   api_key=\"...\",  # API key here\n",
+    "   environment=\"...\"  # find next to api key in console\n",
+    ")\n",
+    "# choose a name for your index\n",
+    "index_name = \"...\""
+   ]
+  },
+  {
    "cell_type": "markdown",
    "id": "95d5d7f9",
    "metadata": {},
    "source": [
-    "You should only have to do this part once.\n",
-    "\n",
-    "Note: it's important to make sure that the \"context\" field that holds the document text in the metadata is not indexed. Currently you need to specify explicitly the fields you do want to index. For more information checkout Pinecone's [docs](https://docs.pinecone.io/docs/manage-indexes#selective-metadata-indexing)."
+    "You should only have to do this part once."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 76,
-   "id": "3b8f7697",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "WhoAmIResponse(username='load', user_label='label', projectname='load-test')"
-      ]
-     },
-     "execution_count": 76,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "import os\n",
-    "import pinecone\n",
-    "\n",
-    "api_key = os.getenv(\"PINECONE_API_KEY\") or \"PINECONE_API_KEY\"\n",
-    "# find environment next to your API key in the Pinecone console\n",
-    "env = os.getenv(\"PINECONE_ENVIRONMENT\") or \"PINECONE_ENVIRONMENT\"\n",
-    "\n",
-    "index_name = \"langchain-pinecone-hybrid-search\"\n",
-    "\n",
-    "pinecone.init(api_key=api_key, enviroment=env)\n",
-    "pinecone.whoami()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 77,
+   "execution_count": null,
    "id": "cfa3a8d8",
    "metadata": {},
    "outputs": [],
    "source": [
-    " # create the index\n",
+    "# create the index\n",
     "pinecone.create_index(\n",
     "   name = index_name,\n",
     "   dimension = 1536,  # dimensionality of dense model\n",
-    "   metric = \"dotproduct\",  # sparse values supported only for dotproduct\n",
-    "   pod_type = \"s1\",\n",
-    "   metadata_config={\"indexed\": []}  # see explaination above\n",
+    "   metric = \"dotproduct\",\n",
+    "   pod_type = \"s1\"\n",
     ")"
    ]
   },
@@ -100,7 +81,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 78,
+   "execution_count": 4,
    "id": "bcb3c8c2",
    "metadata": {},
    "outputs": [],
@@ -109,19 +90,18 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "dbc025d6",
    "metadata": {},
    "source": [
-    "## Get embeddings and sparse encoders\n",
+    "## Get embeddings and tokenizers\n",
     "\n",
     "Embeddings are used for the dense vectors, tokenizer is used for the sparse vector"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 79,
+   "execution_count": 5,
    "id": "2f63c911",
    "metadata": {},
    "outputs": [],
@@ -130,51 +110,19 @@
     "embeddings = OpenAIEmbeddings()"
    ]
   },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "96bf8879",
-   "metadata": {},
-   "source": [
-    "To encode the text to sparse values you can either choose SPLADE or BM25. For out of domain tasks we recommend using BM25.\n",
-    "\n",
-    "For more information about the sparse encoders you can checkout pinecone-text library [docs](https://pinecone-io.github.io/pinecone-text/pinecone_text.html)."
-   ]
-  },
   {
    "cell_type": "code",
-   "execution_count": 80,
+   "execution_count": 6,
    "id": "c3f030e5",
    "metadata": {},
    "outputs": [],
    "source": [
-    "from pinecone_text.sparse import BM25Encoder\n",
-    "# or from pinecone_text.sparse import SpladeEncoder if you wish to work with SPLADE\n",
+    "from transformers import BertTokenizerFast  # !pip install transformers\n",
     "\n",
-    "# use default tf-idf values\n",
-    "bm25_encoder = BM25Encoder().default()"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "23601ddb",
-   "metadata": {},
-   "source": [
-    "The above code is using default tfids values. It's highly recommended to fit the tf-idf values to your own corpus. You can do it as follow:\n",
-    "\n",
-    "```python\n",
-    "corpus = [\"foo\", \"bar\", \"world\", \"hello\"]\n",
-    "\n",
-    "# fit tf-idf values on your corpus\n",
-    "bm25_encoder.fit(corpus)\n",
-    "\n",
-    "# store the values to a json file\n",
-    "bm25_encoder.dump(\"bm25_values.json\")\n",
-    "\n",
-    "# load to your BM25Encoder object\n",
-    "bm25_encoder = BM25Encoder().load(\"bm25_values.json\")\n",
-    "```"
+    "# load bert tokenizer from huggingface\n",
+    "tokenizer = BertTokenizerFast.from_pretrained(\n",
+    "   'bert-base-uncased'\n",
+    ")"
    ]
   },
   {
@@ -189,12 +137,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 81,
+   "execution_count": 7,
    "id": "ac77d835",
    "metadata": {},
    "outputs": [],
    "source": [
-    "retriever = PineconeHybridSearchRetriever(embeddings=embeddings, sparse_encoder=bm25_encoder, index=index)"
+    "retriever = PineconeHybridSearchRetriever(embeddings=embeddings, index=index, tokenizer=tokenizer)"
    ]
   },
   {
@@ -209,16 +157,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 82,
+   "execution_count": 8,
    "id": "98b1c017",
    "metadata": {},
    "outputs": [
     {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "100%|██████████| 1/1 [00:02<00:00,  2.27s/it]\n"
-     ]
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "4d6f3ee7ca754d07a1a18d100d99e0cd",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
     }
    ],
    "source": [
@@ -237,7 +192,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 83,
+   "execution_count": 9,
    "id": "c0455218",
    "metadata": {},
    "outputs": [],
@@ -247,7 +202,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 84,
+   "execution_count": 10,
    "id": "7dfa5c29",
    "metadata": {},
    "outputs": [
@@ -257,7 +212,7 @@
        "Document(page_content='foo', metadata={})"
       ]
      },
-     "execution_count": 84,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -265,11 +220,19 @@
    "source": [
     "result[0]"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74bd9256",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -283,12 +246,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.13"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "7ec0d8babd8cabf695a1d94b1e586d626e046c9df609f6bad065d15d49f67f54"
-   }
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,