bump version to 143 (#3095)

Co-authored-by: yummydum <sumita@nowcast.co.jp>

.github/CONTRIBUTING.md

@@ -75,7 +75,7 @@ This will install all requirements for running the package, examples, linting, f
 
 ❗Note: If you're running Poetry 1.4.1 and receive a `WheelFileValidationError` for `debugpy` during installation, you can try either downgrading to Poetry 1.4.0 or disabling "modern installation" (`poetry config installer.modern-installation false`) and re-install requirements. See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details.
 
-Now, you should be able to run the common tasks in the following section.
+Now, you should be able to run the common tasks in the following section. To double check, run `make test`, all tests should pass. If they don't you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
 
 ## ✅Common Tasks
 

docs/ecosystem/runhouse.md

@@ -15,7 +15,7 @@ custom LLMs, you can use the `SelfHostedPipeline` parent class.
 from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
 ```
 
-For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](../modules/models/llms/integrations/self_hosted_examples.ipynb)
+For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](../modules/models/llms/integrations/runhouse.ipynb)
 
 ## Self-hosted Embeddings
 There are several ways to use self-hosted embeddings with LangChain via Runhouse.

docs/getting_started/getting_started.md

@@ -46,7 +46,7 @@ LangChain provides many modules that can be used to build language model applica
 
 
 
-`````{dropdown} LLMs: Get predictions from a language model
+## LLMs: Get predictions from a language model
 
 The most basic building block of LangChain is calling an LLM on some input.
 Let's walk through a simple example of how to do this. 
@@ -77,10 +77,9 @@ Feetful of Fun
 ```
 
 For more details on how to use LLMs within LangChain, see the [LLM getting started guide](../modules/models/llms/getting_started.ipynb).
-`````
 
 
-`````{dropdown} Prompt Templates: Manage prompts for LLMs
+## Prompt Templates: Manage prompts for LLMs
 
 Calling an LLM is a great first step, but it's just the beginning.
 Normally when you use an LLM in an application, you are not sending user input directly to the LLM.
@@ -115,11 +114,10 @@ What is a good name for a company that makes colorful socks?
 
 [For more details, check out the getting started guide for prompts.](../modules/prompts/chat_prompt_template.ipynb)
 
-`````
 
 
 
-`````{dropdown} Chains: Combine LLMs and prompts in multi-step workflows
+## Chains: Combine LLMs and prompts in multi-step workflows
 
 Up until now, we've worked with the PromptTemplate and LLM primitives by themselves. But of course, a real application is not just one primitive, but rather a combination of them.
 
@@ -159,10 +157,7 @@ This is one of the simpler types of chains, but understanding how it works will
 
 [For more details, check out the getting started guide for chains.](../modules/chains/getting_started.ipynb)
 
-`````
-
-
-`````{dropdown} Agents: Dynamically Call Chains Based on User Input
+## Agents: Dynamically Call Chains Based on User Input
 
 So far the chains we've looked at run in a predetermined order.
 
@@ -234,10 +229,8 @@ Final Answer: The high temperature in SF yesterday in Fahrenheit raised to the .
 ```
 
 
-`````
 
-
-`````{dropdown} Memory: Add State to Chains and Agents
+## Memory: Add State to Chains and Agents
 
 So far, all the chains and agents we've gone through have been stateless. But often, you may want a chain or agent to have some concept of "memory" so that it may remember information about its previous interactions. The clearest and simple example of this is when designing a chatbot - you want it to remember previous messages so it can use context from that to have a better conversation. This would be a type of "short-term memory". On the more complex side, you could imagine a chain/agent remembering key pieces of information over time - this would be a form of "long-term memory". For more concrete ideas on the latter, see this [awesome paper](https://memprompt.com/).
 
@@ -251,7 +244,8 @@ from langchain import OpenAI, ConversationChain
 llm = OpenAI(temperature=0)
 conversation = ConversationChain(llm=llm, verbose=True)
 
-conversation.predict(input="Hi there!")
+output = conversation.predict(input="Hi there!")
+print(output)
 ```
 
 ```pycon
@@ -269,7 +263,8 @@ AI:
 ```
 
 ```python
-conversation.predict(input="I'm doing well! Just having a conversation with an AI.")
+output = conversation.predict(input="I'm doing well! Just having a conversation with an AI.")
+print(output)
 ```
 
 ```pycon
@@ -287,7 +282,6 @@ AI:
 > Finished chain.
 " That's great! What would you like to talk about?"
 ```
-`````
 
 ## Building a Language Model Application: Chat Models
 
@@ -295,8 +289,8 @@ Similarly, you can use chat models instead of LLMs. Chat models are a variation
 
 Chat model APIs are fairly new, so we are still figuring out the correct abstractions.
 
+## Get Message Completions from a Chat Model
 
-`````{dropdown} Get Message Completions from a Chat Model
 You can get chat completions by passing one or more messages to the chat model. The response will be a message. The types of messages currently supported in LangChain are `AIMessage`, `HumanMessage`, `SystemMessage`, and `ChatMessage` -- `ChatMessage` takes in an arbitrary role parameter. Most of the time, you'll just be dealing with `HumanMessage`, `AIMessage`, and `SystemMessage`.
 
 ```python
@@ -350,9 +344,9 @@ You can recover things like token usage from this LLMResult:
 result.llm_output['token_usage']
 # -> {'prompt_tokens': 71, 'completion_tokens': 18, 'total_tokens': 89}
 ```
-`````
 
-`````{dropdown} Chat Prompt Templates
+
+## Chat Prompt Templates
 Similar to LLMs, you can make use of templating by using a `MessagePromptTemplate`. You can build a `ChatPromptTemplate` from one or more `MessagePromptTemplate`s. You can use `ChatPromptTemplate`'s `format_prompt` -- this returns a `PromptValue`, which you can convert to a string or `Message` object, depending on whether you want to use the formatted value as input to an llm or chat model.
 
 For convience, there is a `from_template` method exposed on the template. If you were to use this template, this is what it would look like:
@@ -378,9 +372,8 @@ chat_prompt = ChatPromptTemplate.from_messages([system_message_prompt, human_mes
 chat(chat_prompt.format_prompt(input_language="English", output_language="French", text="I love programming.").to_messages())
 # -> AIMessage(content="J'aime programmer.", additional_kwargs={})
 ```
-`````
 
-`````{dropdown} Chains with Chat Models
+## Chains with Chat Models
 The `LLMChain` discussed in the above section can be used with chat models as well:
 
 ```python
@@ -404,9 +397,8 @@ chain = LLMChain(llm=chat, prompt=chat_prompt)
 chain.run(input_language="English", output_language="French", text="I love programming.")
 # -> "J'aime programmer."
 ```
-`````
 
-`````{dropdown} Agents with Chat Models
+## Agents with Chat Models
 Agents can also be used with chat models, you can initialize one using `AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION` as the agent type.
 
 ```python
@@ -465,9 +457,7 @@ Final Answer: 2.169459462491557
 > Finished chain.
 '2.169459462491557'
 ```
-`````
-
-`````{dropdown} Memory: Add State to Chains and Agents
+## Memory: Add State to Chains and Agents
 You can use Memory with chains and agents initialized with chat models. The main difference between this and Memory for LLMs is that rather than trying to condense all previous messages into a string, we can keep them as their own unique memory object.
 
 ```python
@@ -501,4 +491,4 @@ conversation.predict(input="I'm doing well! Just having a conversation with an A
 conversation.predict(input="Tell me about yourself.")
 # -> "Sure! I am an AI language model created by OpenAI. I was trained on a large dataset of text from the internet, which allows me to understand and generate human-like language. I can answer questions, provide information, and even have conversations like this one. Is there anything else you'd like to know about me?"
 ```
-`````
+

docs/modules/agents/toolkits/examples/jira.ipynb

@@ -0,0 +1,167 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "245a954a",
+   "metadata": {},
+   "source": [
+    "# Jira\n",
+    "\n",
+    "This notebook goes over how to use the Jira tool.\n",
+    "The Jira tool allows agents to interact with a given Jira instance, performing actions such as searching for issues and creating issues, the tool wraps the atlassian-python-api library, for more see: https://atlassian-python-api.readthedocs.io/jira.html\n",
+    "\n",
+    "To use this tool, you must first set as environment variables:\n",
+    "    JIRA_API_TOKEN\n",
+    "    JIRA_USERNAME\n",
+    "    JIRA_INSTANCE_URL"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "961b3689",
+   "metadata": {
+    "vscode": {
+     "languageId": "shellscript"
+    },
+    "ExecuteTime": {
+     "start_time": "2023-04-17T10:21:18.698672Z",
+     "end_time": "2023-04-17T10:21:20.168639Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "%pip install atlassian-python-api"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "34bb5968",
+   "metadata": {
+    "ExecuteTime": {
+     "start_time": "2023-04-17T10:21:22.911233Z",
+     "end_time": "2023-04-17T10:21:23.730922Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from langchain.agents import AgentType\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.agents.agent_toolkits.jira.toolkit import JiraToolkit\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.utilities.jira import JiraAPIWrapper"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "outputs": [],
+   "source": [
+    "os.environ[\"JIRA_API_TOKEN\"] = \"abc\"\n",
+    "os.environ[\"JIRA_USERNAME\"] = \"123\"\n",
+    "os.environ[\"JIRA_INSTANCE_URL\"] = \"https://jira.atlassian.com\"\n",
+    "os.environ[\"OPENAI_API_KEY\"] = \"xyz\""
+   ],
+   "metadata": {
+    "collapsed": false,
+    "ExecuteTime": {
+     "start_time": "2023-04-17T10:22:42.499447Z",
+     "end_time": "2023-04-17T10:22:42.505412Z"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "ac4910f8",
+   "metadata": {
+    "ExecuteTime": {
+     "start_time": "2023-04-17T10:22:44.664481Z",
+     "end_time": "2023-04-17T10:22:44.720538Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "jira = JiraAPIWrapper()\n",
+    "toolkit = JiraToolkit.from_jira_api_wrapper(jira)\n",
+    "agent = initialize_agent(\n",
+    "    toolkit.get_tools(),\n",
+    "    llm,\n",
+    "    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
+    "    verbose=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001B[1m> Entering new AgentExecutor chain...\u001B[0m\n",
+      "\u001B[32;1m\u001B[1;3m I need to create an issue in project PW\n",
+      "Action: Create Issue\n",
+      "Action Input: {\"summary\": \"Make more fried rice\", \"description\": \"Reminder to make more fried rice\", \"issuetype\": {\"name\": \"Task\"}, \"priority\": {\"name\": \"Low\"}, \"project\": {\"key\": \"PW\"}}\u001B[0m\n",
+      "Observation: \u001B[38;5;200m\u001B[1;3mNone\u001B[0m\n",
+      "Thought:\u001B[32;1m\u001B[1;3m I now know the final answer\n",
+      "Final Answer: A new issue has been created in project PW with the summary \"Make more fried rice\" and description \"Reminder to make more fried rice\".\u001B[0m\n",
+      "\n",
+      "\u001B[1m> Finished chain.\u001B[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": "'A new issue has been created in project PW with the summary \"Make more fried rice\" and description \"Reminder to make more fried rice\".'"
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"make a new issue in project PW to remind me to make more fried rice\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "ExecuteTime": {
+     "start_time": "2023-04-17T10:23:33.662454Z",
+     "end_time": "2023-04-17T10:23:38.121883Z"
+    }
+   }
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.7"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "53f3bc57609c7a84333bb558594977aa5b4026b1d6070b93987956689e367341"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/agents/toolkits/examples/openapi.ipynb

@@ -15,7 +15,7 @@
    "id": "a389367b",
    "metadata": {},
    "source": [
-    "# 1st example: hierarchical planning agent\n",
+    "## 1st example: hierarchical planning agent\n",
     "\n",
     "In this example, we'll consider an approach called hierarchical planning, common in robotics and appearing in recent works for LLMs X robotics. We'll see it's a viable approach to start working with a massive API spec AND to assist with user queries that require multiple steps against the API.\n",
     "\n",
@@ -31,7 +31,7 @@
    "id": "4b6ecf6e",
    "metadata": {},
    "source": [
-    "## To start, let's collect some OpenAPI specs."
+    "### To start, let's collect some OpenAPI specs."
    ]
   },
   {
@@ -169,7 +169,7 @@
    "id": "76349780",
    "metadata": {},
    "source": [
-    "## How big is this spec?"
+    "### How big is this spec?"
    ]
   },
   {
@@ -229,7 +229,7 @@
    "id": "cbc4964e",
    "metadata": {},
    "source": [
-    "## Let's see some examples!\n",
+    "### Let's see some examples!\n",
     "\n",
     "Starting with GPT-4. (Some robustness iterations under way for GPT-3 family.)"
    ]
@@ -759,7 +759,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.0"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/agents/tools.rst

@@ -24,6 +24,7 @@ Next, we have some examples of customizing and generically working with tools
 
    ./tools/custom_tools.ipynb
    ./tools/multi_input_tool.ipynb
+   ./tools/tool_input_validation.ipynb
 
 
 In this documentation we cover generic tooling functionality (eg how to create your own)

docs/modules/agents/tools/custom_tools.ipynb

@@ -9,28 +9,29 @@
     "\n",
     "When constructing your own agent, you will need to provide it with a list of Tools that it can use. Besides the actual function that is called, the Tool consists of several components:\n",
     "\n",
-    "- name (str), is required\n",
-    "- description (str), is optional\n",
+    "- name (str), is required and must be unique within a set of tools provided to an agent\n",
+    "- description (str), is optional but recommended, as it is used by an agent to determine tool use\n",
     "- return_direct (bool), defaults to False\n",
     "\n",
-    "The function that should be called when the tool is selected should take as input a single string and return a single string.\n",
+    "The function that should be called when the tool is selected should return a single string.\n",
     "\n",
     "There are two ways to define a tool, we will cover both in the example below."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 1,
    "id": "1aaba18c",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Import things that are needed generically\n",
-    "from langchain.agents import initialize_agent, Tool\n",
-    "from langchain.agents import AgentType\n",
-    "from langchain.tools import BaseTool\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain import LLMMathChain, SerpAPIWrapper"
+    "from langchain import LLMMathChain, SerpAPIWrapper\n",
+    "from langchain.agents import AgentType, Tool, initialize_agent, tool\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.tools import BaseTool"
    ]
   },
   {
@@ -43,12 +44,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "id": "36ed392e",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "llm = OpenAI(temperature=0)"
+    "llm = ChatOpenAI(temperature=0)"
    ]
   },
   {
@@ -74,7 +77,9 @@
    "cell_type": "code",
    "execution_count": 3,
    "id": "56ff7670",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Load the tool configs that are needed.\n",
@@ -98,7 +103,9 @@
    "cell_type": "code",
    "execution_count": 4,
    "id": "5b93047d",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Construct the agent. We will use the default agent type here.\n",
@@ -110,7 +117,9 @@
    "cell_type": "code",
    "execution_count": 5,
    "id": "6f96a891",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -119,29 +128,24 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
+      "\u001b[32;1m\u001b[1;3mI need to find out Leo DiCaprio's girlfriend's name and her age\n",
       "Action: Search\n",
-      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now need to calculate her age raised to the 0.43 power\n",
+      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\u001b[36;1m\u001b[1;3mI draw the lime at going to get a Mohawk, though.\" DiCaprio broke up with girlfriend Camila Morrone, 25, in the summer of 2022, after dating for four years. He's since been linked to another famous supermodel – Gigi Hadid.\u001b[0m\u001b[32;1m\u001b[1;3mI need to find out Gigi Hadid's age\n",
+      "Action: Search\n",
+      "Action Input: \"Gigi Hadid age\"\u001b[0m\u001b[36;1m\u001b[1;3m27 years\u001b[0m\u001b[32;1m\u001b[1;3mI need to calculate her age raised to the 0.43 power\n",
       "Action: Calculator\n",
-      "Action Input: 22^0.43\u001b[0m\n",
+      "Action Input: 27^(0.43)\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "22^0.43\u001b[32;1m\u001b[1;3m\n",
-      "```python\n",
-      "import math\n",
-      "print(math.pow(22, 0.43))\n",
+      "27^(0.43)\u001b[32;1m\u001b[1;3m```text\n",
+      "27**(0.43)\n",
       "```\n",
+      "...numexpr.evaluate(\"27**(0.43)\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m3.777824273683966\n",
-      "\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m4.125593352125936\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.777824273683966\n",
-      "\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\u001b[0m\n",
+      "\u001b[33;1m\u001b[1;3mAnswer: 4.125593352125936\u001b[0m\u001b[32;1m\u001b[1;3mI now know the final answer\n",
+      "Final Answer: 4.125593352125936\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -149,7 +153,7 @@
     {
      "data": {
       "text/plain": [
-       "\"Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\""
+       "'4.125593352125936'"
       ]
      },
      "execution_count": 5,
@@ -171,9 +175,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 6,
    "id": "c58a7c40",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "class CustomSearchTool(BaseTool):\n",
@@ -203,9 +209,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 7,
    "id": "3318a46f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "tools = [CustomSearchTool(), CustomCalculatorTool()]"
@@ -213,9 +221,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 8,
    "id": "ee2d0f3a",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)"
@@ -223,9 +233,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 9,
    "id": "6a2cebbf",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -234,29 +246,24 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
+      "\u001b[32;1m\u001b[1;3mI need to find out Leo DiCaprio's girlfriend's name and her age\n",
       "Action: Search\n",
-      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now need to calculate her age raised to the 0.43 power\n",
+      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\u001b[36;1m\u001b[1;3mI draw the lime at going to get a Mohawk, though.\" DiCaprio broke up with girlfriend Camila Morrone, 25, in the summer of 2022, after dating for four years. He's since been linked to another famous supermodel – Gigi Hadid.\u001b[0m\u001b[32;1m\u001b[1;3mI now know Leo DiCaprio's girlfriend's name and that he's currently linked to Gigi Hadid. I need to find out Camila Morrone's age.\n",
+      "Action: Search\n",
+      "Action Input: \"Camila Morrone age\"\u001b[0m\u001b[36;1m\u001b[1;3m25 years\u001b[0m\u001b[32;1m\u001b[1;3mI have Camila Morrone's age. I need to calculate her age raised to the 0.43 power.\n",
       "Action: Calculator\n",
-      "Action Input: 22^0.43\u001b[0m\n",
+      "Action Input: 25^(0.43)\u001b[0m\n",
       "\n",
       "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "22^0.43\u001b[32;1m\u001b[1;3m\n",
-      "```python\n",
-      "import math\n",
-      "print(math.pow(22, 0.43))\n",
+      "25^(0.43)\u001b[32;1m\u001b[1;3m```text\n",
+      "25**(0.43)\n",
       "```\n",
+      "...numexpr.evaluate(\"25**(0.43)\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m3.777824273683966\n",
-      "\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m3.991298452658078\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.777824273683966\n",
-      "\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\u001b[0m\n",
+      "\u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\u001b[0m\u001b[32;1m\u001b[1;3mI now know the answer to the original question.\n",
+      "Final Answer: Camila Morrone's current age raised to the 0.43 power is approximately 3.99.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -264,10 +271,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Camila Morrone's age raised to the 0.43 power is 3.777824273683966.\""
+       "\"Camila Morrone's current age raised to the 0.43 power is approximately 3.99.\""
       ]
      },
-     "execution_count": 11,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -288,9 +295,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 10,
    "id": "8f15307d",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.agents import tool\n",
@@ -298,22 +307,24 @@
     "@tool\n",
     "def search_api(query: str) -> str:\n",
     "    \"\"\"Searches the API for the query.\"\"\"\n",
-    "    return \"Results\""
+    "    return f\"Results for query {query}\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 11,
    "id": "0a23b91b",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "Tool(name='search_api', description='search_api(query: str) -> str - Searches the API for the query.', return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1184e0cd0>, func=<function search_api at 0x1635f8700>, coroutine=None)"
+       "Tool(name='search_api', description='search_api(query: str) -> str - Searches the API for the query.', args_schema=<class 'pydantic.main.ArgsModel'>, return_direct=False, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x124346f10>, func=<function search_api at 0x16ad6e020>, coroutine=None)"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -332,9 +343,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 12,
    "id": "28cdf04d",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "@tool(\"search\", return_direct=True)\n",
@@ -345,17 +358,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 13,
    "id": "1085a4bd",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "Tool(name='search', description='search(query: str) -> str - Searches the API for the query.', return_direct=True, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x1184e0cd0>, func=<function search_api at 0x1635f8670>, coroutine=None)"
+       "Tool(name='search', description='search(query: str) -> str - Searches the API for the query.', args_schema=<class 'pydantic.main.ArgsModel'>, return_direct=True, verbose=False, callback_manager=<langchain.callbacks.shared.SharedCallbackManager object at 0x124346f10>, func=<function search_api at 0x16ad6d3a0>, coroutine=None)"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -376,7 +389,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 14,
    "id": "79213f40",
    "metadata": {},
    "outputs": [],
@@ -386,7 +399,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 15,
    "id": "e1067dcb",
    "metadata": {},
    "outputs": [],
@@ -396,7 +409,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 16,
    "id": "6c66ffe8",
    "metadata": {},
    "outputs": [],
@@ -406,7 +419,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 17,
    "id": "f45b5bc3",
    "metadata": {},
    "outputs": [],
@@ -416,7 +429,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 18,
    "id": "565e2b9b",
    "metadata": {},
    "outputs": [
@@ -427,21 +440,12 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m I need to find out who Leo DiCaprio's girlfriend is and then calculate her age raised to the 0.43 power.\n",
+      "\u001b[32;1m\u001b[1;3mI need to find out Leo DiCaprio's girlfriend's name and her age.\n",
       "Action: Google Search\n",
-      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mCamila Morrone\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to find out Camila Morrone's age\n",
-      "Action: Google Search\n",
-      "Action Input: \"Camila Morrone age\"\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m25 years\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to calculate 25 raised to the 0.43 power\n",
+      "Action Input: \"Leo DiCaprio girlfriend\"\u001b[0m\u001b[36;1m\u001b[1;3mI draw the lime at going to get a Mohawk, though.\" DiCaprio broke up with girlfriend Camila Morrone, 25, in the summer of 2022, after dating for four years. He's since been linked to another famous supermodel – Gigi Hadid.\u001b[0m\u001b[32;1m\u001b[1;3mNow I need to find out Camila Morrone's current age.\n",
       "Action: Calculator\n",
-      "Action Input: 25^0.43\u001b[0m\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\n",
-      "\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
-      "Final Answer: Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078.\u001b[0m\n",
+      "Action Input: 25^0.43\u001b[0m\u001b[33;1m\u001b[1;3mAnswer: 3.991298452658078\u001b[0m\u001b[32;1m\u001b[1;3mI now know the final answer.\n",
+      "Final Answer: Camila Morrone's current age raised to the 0.43 power is approximately 3.99.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -449,10 +453,10 @@
     {
      "data": {
       "text/plain": [
-       "\"Camila Morrone is Leo DiCaprio's girlfriend and her current age raised to the 0.43 power is 3.991298452658078.\""
+       "\"Camila Morrone's current age raised to the 0.43 power is approximately 3.99.\""
       ]
      },
-     "execution_count": 12,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -478,7 +482,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 19,
    "id": "3450512e",
    "metadata": {},
    "outputs": [],
@@ -507,7 +511,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 20,
    "id": "4b9a7849",
    "metadata": {},
    "outputs": [
@@ -520,9 +524,7 @@
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m I should use a music search engine to find the answer\n",
       "Action: Music Search\n",
-      "Action Input: most famous song of christmas\u001b[0m\n",
-      "Observation: \u001b[33;1m\u001b[1;3m'All I Want For Christmas Is You' by Mariah Carey.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Action Input: most famous song of christmas\u001b[0m\u001b[33;1m\u001b[1;3m'All I Want For Christmas Is You' by Mariah Carey.\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
       "Final Answer: 'All I Want For Christmas Is You' by Mariah Carey.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
@@ -534,7 +536,7 @@
        "\"'All I Want For Christmas Is You' by Mariah Carey.\""
       ]
      },
-     "execution_count": 14,
+     "execution_count": 20,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -554,7 +556,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 21,
    "id": "3bb6185f",
    "metadata": {},
    "outputs": [],
@@ -572,7 +574,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 22,
    "id": "113ddb84",
    "metadata": {},
    "outputs": [],
@@ -583,9 +585,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 23,
    "id": "582439a6",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -596,9 +600,7 @@
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
       "\u001b[32;1m\u001b[1;3m I need to calculate this\n",
       "Action: Calculator\n",
-      "Action Input: 2**.12\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mAnswer: 1.2599210498948732\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m\u001b[0m\n",
+      "Action Input: 2**.12\u001b[0m\u001b[36;1m\u001b[1;3mAnswer: 1.086734862526058\u001b[0m\u001b[32;1m\u001b[1;3m\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -606,10 +608,10 @@
     {
      "data": {
       "text/plain": [
-       "'Answer: 1.2599210498948732'"
+       "'Answer: 1.086734862526058'"
       ]
      },
-     "execution_count": 17,
+     "execution_count": 23,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -618,10 +620,149 @@
     "agent.run(\"whats 2**.12\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8aa3c353-bd89-467c-9c27-b83a90cd4daa",
+   "metadata": {},
+   "source": [
+    "## Multi-argument tools\n",
+    "\n",
+    "Many functions expect structured inputs. These can also be supported using the Tool decorator or by directly subclassing `BaseTool`! We have to modify the LLM's OutputParser to map its string output to a dictionary to pass to the action, however."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "537bc628",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from typing import Optional, Union\n",
+    "\n",
+    "@tool\n",
+    "def custom_search(k: int, query: str, other_arg: Optional[str] = None):\n",
+    "    \"\"\"The custom search function.\"\"\"\n",
+    "    return f\"Here are the results for the custom search: k={k}, query={query}, other_arg={other_arg}\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "d5c992cf-776a-40cd-a6c4-e7cf65ea709e",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "from langchain.schema import (\n",
+    "    AgentAction,\n",
+    "    AgentFinish,\n",
+    ")\n",
+    "from langchain.agents import AgentOutputParser\n",
+    "\n",
+    "# We will add a custom parser to map the arguments to a dictionary\n",
+    "class CustomOutputParser(AgentOutputParser):\n",
+    "    \n",
+    "    def parse_tool_input(self, action_input: str) -> dict:\n",
+    "        # Regex pattern to match arguments and their values\n",
+    "        pattern = r\"(\\w+)\\s*=\\s*(None|\\\"[^\\\"]*\\\"|\\d+)\"\n",
+    "        matches = re.findall(pattern, action_input)\n",
+    "        \n",
+    "        if not matches:\n",
+    "            raise ValueError(f\"Could not parse action input: `{action_input}`\")\n",
+    "\n",
+    "        # Create a dictionary with the parsed arguments and their values\n",
+    "        parsed_input = {}\n",
+    "        for arg, value in matches:\n",
+    "            if value == \"None\":\n",
+    "                parsed_value = None\n",
+    "            elif value.isdigit():\n",
+    "                parsed_value = int(value)\n",
+    "            else:\n",
+    "                parsed_value = value.strip('\"')\n",
+    "            parsed_input[arg] = parsed_value\n",
+    "\n",
+    "        return parsed_input\n",
+    "    \n",
+    "    def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:\n",
+    "        # Check if agent should finish\n",
+    "        if \"Final Answer:\" in llm_output:\n",
+    "            return AgentFinish(\n",
+    "                # Return values is generally always a dictionary with a single `output` key\n",
+    "                # It is not recommended to try anything else at the moment :)\n",
+    "                return_values={\"output\": llm_output.split(\"Final Answer:\")[-1].strip()},\n",
+    "                log=llm_output,\n",
+    "            )\n",
+    "        # Parse out the action and action input\n",
+    "        regex = r\"Action\\s*\\d*\\s*:(.*?)\\nAction\\s*\\d*\\s*Input\\s*\\d*\\s*:[\\s]*(.*)\"\n",
+    "        match = re.search(regex, llm_output, re.DOTALL)\n",
+    "        if not match:\n",
+    "            raise ValueError(f\"Could not parse LLM output: `{llm_output}`\")\n",
+    "        action = match.group(1).strip()\n",
+    "        action_input = match.group(2)\n",
+    "        tool_input = self.parse_tool_input(action_input)\n",
+    "        # Return the action and action \n",
+    "        return AgentAction(tool=action, tool_input=tool_input, log=llm_output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "68269547-1482-4138-a6ea-58f00b4a9548",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "agent = initialize_agent([custom_search], llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True, agent_kwargs={\"output_parser\": CustomOutputParser()})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "id": "0947835a-691c-4f51-b8f4-6744e0e48ab1",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to use a search function to find the answer\n",
+      "Action: custom_search\n",
+      "Action Input: k=1, query=\"me\"\u001b[0m\u001b[36;1m\u001b[1;3mHere are the results for the custom search: k=1, query=me, other_arg=None\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: The results of the custom search for k=1, query=me, other_arg=None.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'The results of the custom search for k=1, query=me, other_arg=None.'"
+      ]
+     },
+     "execution_count": 27,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"Search for me and tell me whatever it says\")"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "537bc628",
+   "id": "caf39c66-102b-42c1-baf2-777a49886ce4",
    "metadata": {},
    "outputs": [],
    "source": []
@@ -643,7 +784,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.2"
   },
   "vscode": {
    "interpreter": {

docs/modules/agents/tools/tool_input_validation.ipynb

@@ -0,0 +1,190 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "# Tool Input Schema\n",
+    "\n",
+    "By default, tools infer the argument schema by inspecting the function signature. For more strict requirements, custom input schema can be specified, along with custom validation logic."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.agents import initialize_agent, Tool\n",
+    "from langchain.agents import AgentType\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.tools.python.tool import PythonREPLTool\n",
+    "from pydantic import BaseModel\n",
+    "from pydantic import Field, root_validator\n",
+    "from typing import Dict, Any"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "class ToolInputModel(BaseModel):\n",
+    "    query: str = Field(...)\n",
+    "    \n",
+    "    @root_validator\n",
+    "    def validate_query(cls, values: Dict[str, Any]) -> Dict:\n",
+    "        # Note: this is NOT a safe REPL! This is used for instructive purposes only\n",
+    "        if \"import os\" in values[\"query\"]:\n",
+    "            raise ValueError(\"'import os' not permitted in this python REPL.\")\n",
+    "        return values\n",
+    "    \n",
+    "tool = PythonREPLTool(args_schema=ToolInputModel)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "agent = initialize_agent([tool], llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to define a function that adds two numbers\n",
+      "Action: Python REPL\n",
+      "Action Input: def add_two_numbers(a, b):\n",
+      "    return a + b\u001b[0m\u001b[36;1m\u001b[1;3m\u001b[0m\u001b[32;1m\u001b[1;3m I need to call the function\n",
+      "Action: Python REPL\n",
+      "Action Input: print(add_two_numbers(2, 2))\u001b[0m\u001b[36;1m\u001b[1;3m4\n",
+      "\u001b[0m\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: 4\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'4'"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# This will succeed, since there aren't any arguments that will be triggered during validation\n",
+    "agent.run(\"Run a python function that adds 2 and 2\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m I need to import os and then list the dir\n",
+      "Action: Python REPL\n",
+      "Action Input: import os; print(os.listdir())\u001b[0m"
+     ]
+    },
+    {
+     "ename": "ValidationError",
+     "evalue": "1 validation error for ToolInputModel\n__root__\n  'import os' not  (type=value_error)",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[6], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43magent\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mRun a python function that imports os and lists the dir\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/chains/base.py:213\u001b[0m, in \u001b[0;36mChain.run\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    211\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(args) \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m    212\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m`run` supports only one positional argument.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m--> 213\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m[\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39moutput_keys[\u001b[38;5;241m0\u001b[39m]]\n\u001b[1;32m    215\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m kwargs \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m args:\n\u001b[1;32m    216\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m(kwargs)[\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39moutput_keys[\u001b[38;5;241m0\u001b[39m]]\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/chains/base.py:116\u001b[0m, in \u001b[0;36mChain.__call__\u001b[0;34m(self, inputs, return_only_outputs)\u001b[0m\n\u001b[1;32m    114\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\u001b[38;5;167;01mKeyboardInterrupt\u001b[39;00m, \u001b[38;5;167;01mException\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    115\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_manager\u001b[38;5;241m.\u001b[39mon_chain_error(e, verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mverbose)\n\u001b[0;32m--> 116\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m    117\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_manager\u001b[38;5;241m.\u001b[39mon_chain_end(outputs, verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mverbose)\n\u001b[1;32m    118\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprep_outputs(inputs, outputs, return_only_outputs)\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/chains/base.py:113\u001b[0m, in \u001b[0;36mChain.__call__\u001b[0;34m(self, inputs, return_only_outputs)\u001b[0m\n\u001b[1;32m    107\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_manager\u001b[38;5;241m.\u001b[39mon_chain_start(\n\u001b[1;32m    108\u001b[0m     {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mname\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__class__\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m},\n\u001b[1;32m    109\u001b[0m     inputs,\n\u001b[1;32m    110\u001b[0m     verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mverbose,\n\u001b[1;32m    111\u001b[0m )\n\u001b[1;32m    112\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 113\u001b[0m     outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43minputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    114\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\u001b[38;5;167;01mKeyboardInterrupt\u001b[39;00m, \u001b[38;5;167;01mException\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    115\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_manager\u001b[38;5;241m.\u001b[39mon_chain_error(e, verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mverbose)\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/agents/agent.py:792\u001b[0m, in \u001b[0;36mAgentExecutor._call\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m    790\u001b[0m \u001b[38;5;66;03m# We now enter the agent loop (until it returns something).\u001b[39;00m\n\u001b[1;32m    791\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_should_continue(iterations, time_elapsed):\n\u001b[0;32m--> 792\u001b[0m     next_step_output \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_take_next_step\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    793\u001b[0m \u001b[43m        \u001b[49m\u001b[43mname_to_tool_map\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcolor_mapping\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mintermediate_steps\u001b[49m\n\u001b[1;32m    794\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    795\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(next_step_output, AgentFinish):\n\u001b[1;32m    796\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_return(next_step_output, intermediate_steps)\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/agents/agent.py:695\u001b[0m, in \u001b[0;36mAgentExecutor._take_next_step\u001b[0;34m(self, name_to_tool_map, color_mapping, inputs, intermediate_steps)\u001b[0m\n\u001b[1;32m    693\u001b[0m         tool_run_kwargs[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mllm_prefix\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    694\u001b[0m     \u001b[38;5;66;03m# We then call the tool on the tool input to get an observation\u001b[39;00m\n\u001b[0;32m--> 695\u001b[0m     observation \u001b[38;5;241m=\u001b[39m \u001b[43mtool\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    696\u001b[0m \u001b[43m        \u001b[49m\u001b[43magent_action\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtool_input\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    697\u001b[0m \u001b[43m        \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mverbose\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    698\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcolor\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcolor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    699\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mtool_run_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    700\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    701\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    702\u001b[0m     tool_run_kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39magent\u001b[38;5;241m.\u001b[39mtool_run_logging_kwargs()\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/tools/base.py:146\u001b[0m, in \u001b[0;36mBaseTool.run\u001b[0;34m(self, tool_input, verbose, start_color, color, **kwargs)\u001b[0m\n\u001b[1;32m    137\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrun\u001b[39m(\n\u001b[1;32m    138\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m    139\u001b[0m     tool_input: Union[\u001b[38;5;28mstr\u001b[39m, Dict],\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    143\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any,\n\u001b[1;32m    144\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28mstr\u001b[39m:\n\u001b[1;32m    145\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Run the tool.\"\"\"\u001b[39;00m\n\u001b[0;32m--> 146\u001b[0m     run_input \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_parse_input\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtool_input\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    147\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mverbose \u001b[38;5;129;01mand\u001b[39;00m verbose \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    148\u001b[0m         verbose_ \u001b[38;5;241m=\u001b[39m verbose\n",
+      "File \u001b[0;32m~/code/lc/lckg/langchain/tools/base.py:112\u001b[0m, in \u001b[0;36mBaseTool._parse_input\u001b[0;34m(self, tool_input)\u001b[0m\n\u001b[1;32m    110\u001b[0m     tool_input \u001b[38;5;241m=\u001b[39m {field_name: tool_input}\n\u001b[1;32m    111\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m pydantic_input_type \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 112\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mpydantic_input_type\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_obj\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtool_input\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    113\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    114\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    115\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124margs_schema required for tool \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m in order to\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    116\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m accept input of type \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mtype\u001b[39m(tool_input)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    117\u001b[0m     )\n",
+      "File \u001b[0;32m~/code/lc/lckg/.venv/lib/python3.11/site-packages/pydantic/main.py:526\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.parse_obj\u001b[0;34m()\u001b[0m\n",
+      "File \u001b[0;32m~/code/lc/lckg/.venv/lib/python3.11/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
+      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for ToolInputModel\n__root__\n  'import os' not  (type=value_error)"
+     ]
+    }
+   ],
+   "source": [
+    "# This will fail, because the attempt to import os will trigger a validation error\n",
+    "agent.run(\"Run a python function that imports os and lists the dir\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/modules/indexes/document_loaders/examples/confluence.ipynb

@@ -0,0 +1,66 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Confluence\n",
+    "\n",
+    "A loader for Confluence pages.\n",
+    "\n",
+    "\n",
+    "This currently supports both username/api_key and Oauth2 login.\n",
+    "\n",
+    "\n",
+    "Specify a list page_ids and/or space_key to load in the corresponding pages into Document objects, if both are specified the union of both sets will be returned.\n",
+    "\n",
+    "\n",
+    "You can also specify a boolean `include_attachments` to include attachments, this is set to False by default, if set to True all attachments will be downloaded and ConfluenceReader will extract the text from the attachments and add it to the Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG, SVG, Word and Excel.\n",
+    "\n",
+    "Hint: space_key and page_id can both be found in the URL of a page in Confluence - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import ConfluenceLoader\n",
+    "\n",
+    "loader = ConfluenceLoader(\n",
+    "    url=\"https://yoursite.atlassian.com/wiki\",\n",
+    "    username=\"me\",\n",
+    "    api_key=\"12345\"\n",
+    ")\n",
+    "documents = loader.load(space_key=\"SPACE\", include_attachments=True, limit=50)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "cc99336516f23363341912c6723b01ace86f02e26b4290be1efc0677e2e2ec24"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/modules/indexes/document_loaders/examples/discord_loader.ipynb

@@ -0,0 +1,67 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6ad0a850",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3d4f1bdd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.read_csv(\"example_data/fake_discord_chat.csv\")\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "82880d80",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders.discord import DiscordChatLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b825a8d3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = DiscordChatLoader(df, user_id_col=\"user_id\")\n",
+    "print(loader.load())"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/indexes/document_loaders/examples/example_data/fake_discord_chat.csv

@@ -0,0 +1,170 @@
+user_id,message,timestamp
+6b118e52-bc2b-487e-9322-a84f217902f1,"Street of born. Order around toward serve box. More keep number some.
+Physical bring doctor wall finally share make. About nothing suddenly somebody gun receive job.",2022-01-01 00:00:00
+be5ea045-4478-45a8-9b8c-8e0a43222888,"Service describe size two ever. White marriage each sell line nature far.
+Religious organization type music. Often still more five. Religious present test prove fine.",2022-01-01 00:10:00
+3de7ddf9-e58b-4eb6-a7f5-083d1d022499,"Major with ball process now make. Almost hospital possible good media whom.
+Clearly entire while realize wind. Base game skin together probably sport like memory.",2022-01-01 00:20:00
+87e99005-780f-4f3b-b48d-91b17cce55fe,Page indeed clear different. Skin loss develop animal customer. Cultural number effect tonight would paper.,2022-01-01 00:30:00
+251099c2-ca02-4fb4-bd52-dca41f3d8b25,"World tree seat grow. Region believe matter himself message despite.
+Together type process kid enjoy watch story. Visit happen movie her. Practice few ok chance.",2022-01-01 00:40:00
+974ed6b3-c08c-4771-a8d1-945f20526850,Result visit amount production prevent attack agreement. Development month realize. Thus not whatever live decide young specific.,2022-01-01 00:50:00
+e0f88629-3df1-4f54-a40e-c75cf5909c26,"Evening past choice visit approach must. Avoid unit get stop. Image talk design only role write wrong.
+I model response character son smile degree job.
+Lead bill sound likely chair.",2022-01-01 01:00:00
+5f260d25-e1ff-491b-8720-29b7e7c40a1f,"Painting develop environmental message recently them area. Him for condition blood. Should brother little majority.
+Couple store because deep. Radio turn clear remember collection.",2022-01-01 01:10:00
+b217eaba-53ff-430c-8d4c-f1ebd1a77a09,Leg measure on age positive. Discussion party and get huge force building. So yes assume brother medical population. Fly will enter involve cup woman spend.,2022-01-01 01:20:00
+7b42ae71-12af-4222-a352-84ed9a3414ea,Knowledge goal single believe inside. Police human and fall. Trial participant indeed against since. Few miss radio successful.,2022-01-01 01:30:00
+2a4adb92-8caa-4b9d-96f4-c89905e48a32,"Else big which energy.
+Maybe Republican character conference social. Threat upon just receive.
+Easy national speak. Listen assume like ago practice bit animal.",2022-01-01 01:40:00
+66d0a700-7781-4702-802c-83632e595350,"Democratic night foot. Month force civil mind true. Institution rest peace none growth own.
+Start billion popular. A baby attention assume certainly result. Strong true teacher executive purpose.",2022-01-01 01:50:00
+2fe4b7c5-936d-4576-bf25-02d41acd9bb6,Us subject same war other. International how trial free. Parent certainly relationship summer four.,2022-01-01 02:00:00
+583739ec-0f39-488f-9728-d336bf6c9983,"Mention meeting similar letter easy reality. Upon morning little put.
+Ago rate huge me see edge.",2022-01-01 02:10:00
+d0f75d9a-0de9-4341-bba3-0abd1f89a961,"Black second bill study if reveal so your. Economic somebody walk attack meet. Artist threat natural world.
+Forget although range. My seven wait lot add production.",2022-01-01 02:20:00
+f0f494fa-e16d-4ee0-bbc8-42a92b7a6cc1,"When court officer exist yeah hold between light. Should can message character.
+Sea probably parent media song second weight trip. How process effect media reduce.",2022-01-01 02:30:00
+a08cd4e4-0962-42cb-89e7-2e2812af4f3a,Anything some charge authority something. I letter laugh investment including turn. Son receive matter care many now discuss.,2022-01-01 02:40:00
+11486a17-d5c5-42e3-bcb8-b75e08b8937b,Tax image relationship under increase keep. Really society get ago himself help popular society. If indicate alone until plan economic maintain stand.,2022-01-01 02:50:00
+9ee84827-7f9b-40d0-8c45-34d8cd42d3ba,Nation dream make both. Continue often pass truth national grow. Language sometimes process way million individual friend.,2022-01-01 03:00:00
+d7a679cc-dbe9-41ee-9626-9289cfc8ae37,"Determine them fine cut. East month memory. Treat like worker federal tonight.
+Gun option tree. Data start brother help citizen easy want reveal. Conference movement sea show open change.",2022-01-01 03:10:00
+470b83fb-dd86-44c9-908c-13a17c771452,"Option use bill later away.
+Trade would player fine evidence learn. Increase game them necessary. Sit range must fire hard serious. Available recognize contain act.",2022-01-01 03:20:00
+67dc28d3-77d7-4de2-a176-69a67a246704,Because conference lead election time. Purpose theory old talk their. Account often center consider see loss theory because. Minute however wonder agency.,2022-01-01 03:30:00
+e07a8306-ecb9-43da-bff5-72ef4af176cd,"Woman whose face short also data. Hand just each culture no.
+Despite new hold bed. Feel fund deep agency hundred drug. Pass surface continue positive mind same check hand. Across make fund similar.",2022-01-01 03:40:00
+d70c6f1f-f29c-46d7-9e2a-13fe4f4dd06f,"Quality arrive hair if over. Hotel day reach west spend toward. Any opportunity much participant property mission.
+Themselves throw mind indeed. Late half recently. Edge enjoy great ahead.",2022-01-01 03:50:00
+47d84bcb-0f77-42dc-a762-255733a37ed9,"Public front article sea. Very bit center certain experience whom.
+Probably reality final goal radio financial age.",2022-01-01 04:00:00
+af4890da-8ec8-4c41-ba07-7038fbe90b88,"Choose hope history none current.
+Usually loss floor garden value article. No arm decision create government agree often.",2022-01-01 04:10:00
+0fefc22a-3d8f-4de2-ad42-0cce61f497e2,Explain baby might among their exactly risk. Body big set across put class. Hit should appear situation.,2022-01-01 04:20:00
+5b3809e6-e86c-4e1e-b687-5cdb4b9f76fa,"Nation summer exactly. Pick car reality would.
+Quite Mr number question war throw. Certain career idea heavy. Hope process save common attack to.",2022-01-01 04:30:00
+ef0cd3a7-3368-4ccc-8232-940c041cc888,"Effort stand bag never future training. Defense attorney mouth play coach peace. Herself sing size compare.
+Speak surface old half available.",2022-01-01 04:40:00
+d5b1cf3f-9b31-4d35-b89a-c29caf09fdad,"Magazine finish capital guy really too ask. Item agency seat hospital away. Near four despite without.
+Congress property fund toward character. Watch detail food rock identify.",2022-01-01 04:50:00
+50a39171-4725-44dc-b88d-712a5202b949,Six this prevent. Usually sound history field little. Theory suggest else. President less team garden.,2022-01-01 05:00:00
+6aa242fb-606f-44ff-86e2-4f3c19162545,"Machine left yeah think clear. Serve role expert animal first up.
+Well contain nature million. Issue firm follow far growth spring everyone boy.",2022-01-01 05:10:00
+195a938a-9e78-43af-91d1-b3a05b2737d8,"Method high practice measure style always adult along. Them stay wait manager sometimes person likely do.
+Or fine conference job feeling number. Fact front compare east mind.",2022-01-01 05:20:00
+93fd62ea-8147-457b-96f2-8d9db53c53cd,"Mind well stock be morning federal then yeah. Save material stand cell poor.
+Notice occur success organization chance list. Soon deal receive nearly. Future allow night take wonder rate feeling.",2022-01-01 05:30:00
+f01fc201-2491-4757-8d13-c0f0411fb2c3,"Rock or fund. Middle man course task west open. Scene movie hope development travel.
+Public fine artist theory. Well across each new. Process bad eat win article want election.",2022-01-01 05:40:00
+e0c8eb7f-a4d7-411c-b337-e7eb6b28eb4f,"Moment arm may beautiful he firm. Single western dog themselves able media high material.
+Century today itself fear remember message than. Fund kid rest prevent. Director officer teacher.",2022-01-01 05:50:00
+28b8a3af-b5ce-4802-a48f-22058d3931b3,Drive write various raise environment. Time official country perhaps could though good. Style fast kitchen piece sense nothing agreement.,2022-01-01 06:00:00
+f5cc758a-4b9d-479d-861c-d01432a99a5e,Assume design worker. Skill although wear maintain rest member everyone marriage. Age along feeling beat method magazine.,2022-01-01 06:10:00
+8579650a-74b7-4dac-bfce-2f5eee7fbce4,"Candidate against reduce wrong size network exactly. It present able during manager whom hard. Near tend already.
+Serious major democratic remain main cost marriage. Team deal meeting group store.",2022-01-01 06:20:00
+9036fdeb-d814-4130-bc54-13e3bf8676e9,"How through support lead over there. Security over crime police attorney. Management popular item son remain if increase.
+White piece glass author hit. Mean theory citizen reason simply day coach.",2022-01-01 06:30:00
+8e7a773f-e4d8-402a-a7ce-5bfcb9235503,Child inside boy conference reveal catch economy. Develop window like tax. Money truth heart learn this anyone.,2022-01-01 06:40:00
+a95756f6-5796-4e01-9555-b9d6726bfbfe,Majority real so newspaper grow yourself. Station community half line. Ready little wind energy.,2022-01-01 06:50:00
+44866560-112b-42d2-90d3-e6d13c8eca64,National region third understand mind. Water hot attorney information. Magazine customer Mrs song. Authority investment become organization.,2022-01-01 07:00:00
+0561ef77-94bb-4325-874d-11c4bd797eb7,"Everyone however skin human room dream although. Where remain director.
+Eye heavy safe. Model score use exactly family structure. Two true family candidate.",2022-01-01 07:10:00
+fa71bdc4-b85b-48f5-82b8-fa9d22e9f6a3,"Model bring kitchen wish anything simple power.
+Able write six official individual matter forward. Room personal economic north. Investment war several politics serve expect.",2022-01-01 07:20:00
+1b91d11a-09c8-48f0-b042-911bea52a3b8,"Record suffer start. Most feeling find down door. Letter actually I dream dream. Law cost our personal Republican.
+Likely store situation protect provide half.
+School place it level nor fast order.",2022-01-01 07:30:00
+10d7a616-55a6-49d0-bf6d-8f3ffc24f35e,"Pull shoulder life visit best. Itself whose even hit practice exist campaign.
+Contain happy moment wide. Spring meeting part position happy population thousand.",2022-01-01 07:40:00
+6bd1ad19-a1b8-4ba0-83ee-7969b0547ee1,"Mother couple section audience we world. Huge husband window senior realize reality must message.
+Mention heart action other their. Drop claim check whom trouble. Go end discuss school senior drive.",2022-01-01 07:50:00
+68faa2b0-02f9-484a-900c-95f8cfd22554,Teach free west discuss note stage. Add war than build. Politics write business. Behavior success better determine who power top billion.,2022-01-01 08:00:00
+a01b6797-d94b-4eb0-956b-efe1a3d8269d,Any end worry event fear. Memory feeling ball team.,2022-01-01 08:10:00
+7ef25987-b910-4146-89b4-40bca6da73db,"Suffer worry west big next present travel. Standard thing consumer road your.
+Start decade off speech. Wall we recent civil life despite.",2022-01-01 08:20:00
+c046101f-65ed-4d22-a3a1-76efbccf6b07,Resource north environmental simply nice. Past together study part more.,2022-01-01 08:30:00
+1a55ca1e-dac2-4ee3-8739-e38494e2a603,Fund leader office animal physical matter. Attorney whole with knowledge practice create meeting question. Wind west always score art.,2022-01-01 08:40:00
+8cc516b3-d007-48be-a109-eb1a0ac1fa16,"Feeling notice five family believe goal increase.
+State page structure record use her. Student defense year myself out.
+Congress style perform cell mission sense. Yes when leg responsibility.",2022-01-01 08:50:00
+76411191-5c7a-4a9c-997f-ea61f3c138ff,Hope discover glass up own pressure. Two include generation TV. Fund drop government true west sit everything.,2022-01-01 09:00:00
+28ae91e1-26f1-4f40-8aaa-692a905b8b26,"Represent who smile blue model. Hour exactly lay white avoid let. Lose film technology affect agree.
+Old respond old stay general natural ever. Field phone safe. Mrs whole space answer color.",2022-01-01 09:10:00
+523b4e51-2261-4d61-8fe1-a379c8df5824,Weight sport allow before resource avoid. Wait leg space writer maybe actually third. Tax need by benefit.,2022-01-01 09:20:00
+32e8c5a2-bfc5-4e5a-ab97-7e36e4a221b9,"Develop write street after well. Offer along eat buy adult but.
+Never there everybody movie represent recognize ground. Result pretty pay once. Will newspaper away even dream yeah grow.",2022-01-01 09:30:00
+0c38e2ee-3ca2-44e8-be2b-7e4fe6fe4b68,Meeting mother easy newspaper check. Oil indicate available year different everyone leader. Present thus explain meeting. Head respond pretty big too natural born beautiful.,2022-01-01 09:40:00
+c09ef9ca-78a2-4796-9d4c-b791a9abcaa7,"Skill president around.
+Reduce just necessary choose surface card argue. Foot base cup player. Believe effect consider operation theory.",2022-01-01 09:50:00
+d2a1de02-9514-4a6c-ae37-9cd1e2beefa0,"Medical put instead between these.
+Network environment of social feel blue decide after. Glass notice simply Democrat and spring. Might need computer middle. Town turn skin skill your.",2022-01-01 10:00:00
+a9667646-d759-41b2-b22b-6c38fdba84b2,"Into daughter door especially plan expert set. Return although control join. Though blue agree maybe write.
+Send party ready spring house.",2022-01-01 10:10:00
+240ea2f1-3eda-47e3-96ef-cd4f7299e4f2,"Trip face stage game doctor name cover.
+Box may evening production none treatment ago. Behavior television show area news. Board leave collection director carry type whatever.",2022-01-01 10:20:00
+4f73aa5f-02d1-4cf2-bc73-45f050606c5d,"Almost country bring better member sister. Agency if around assume light development example.
+Tend way deep another possible quality white. Term from new good. Worry woman recently glass.",2022-01-01 10:30:00
+b076bf4d-7964-4c30-b3be-e68dcbffa7c7,"Audience act group small. Individual agreement realize special upon work wall sometimes.
+Main million and act various song where. Opportunity along generation trial consumer. Center great should Mrs.",2022-01-01 10:40:00
+10c6e307-9599-42c0-b418-35e340346eb6,So ahead bill term. Million represent election cut summer guess. Everybody though my other or stay it.,2022-01-01 10:50:00
+eb496b9d-4fbd-4355-bced-7d1065e63991,"Girl how gas. Sister popular matter.
+Before indeed week else whose rest church. Trade team them.
+Describe compare improve American act. Nation close example. Recent around through.",2022-01-01 11:00:00
+ddfab160-0481-4639-8407-35ec0745b657,After practice seven Republican. Task democratic general television cost matter. Arrive east crime role. Evening Mr ever national wide lose hour address.,2022-01-01 11:10:00
+cf8ceaea-9106-47ba-a4be-684e56ca9996,Process whatever public nation bar learn. Trip trouble best participant. Their among just agent check answer theory.,2022-01-01 11:20:00
+79e939eb-e6d9-4b87-a1c7-accef7802175,Standard story community analysis wife. Information affect least begin phone media establish. Challenge key film your stay entire court. City trip body kitchen increase.,2022-01-01 11:30:00
+915f81e6-870b-4e10-b43a-8d4d3b9680c4,Buy authority quality. Yet out general century record. Finally listen care specific citizen outside money.,2022-01-01 11:40:00
+3f3d1691-409e-485d-82c6-9de14d298e9b,Fear its head tough together concern matter. Measure catch visit city rise find modern. Teach least major resource first itself.,2022-01-01 11:50:00
+3cd4ad69-7347-41de-86fd-f2fe6bcf5696,Here huge tonight how audience team. Month land toward role. Someone ready mention building. Language democratic traditional but center figure.,2022-01-01 12:00:00
+5201d830-5cf2-4ac4-8953-5cbb47e7f030,Though run open police training see my tree. Never soldier hope take want. Training huge carry direction should write.,2022-01-01 12:10:00
+abcc0af7-0ba9-4896-8e3d-80f9be1dae23,"Agree when defense. Use might need behind any.
+Data walk let me not information. Against eight design.",2022-01-01 12:20:00
+92ed4204-74e7-4930-992f-5bfd135d91b6,"Customer writer body. Half run suggest travel. Nature various just.
+Recently wish short apply ask. Image bit share.",2022-01-01 12:30:00
+34493408-4367-4f2d-ab3b-27c9c0944928,"Wife result likely system person wind. Pm near contain reach group.
+At recent a responsibility. Evidence receive space himself no mind. Public eat now situation.",2022-01-01 12:40:00
+835653c0-14ca-4cc6-9849-955526a461ac,Or draw age name whose. Break individual region always. Pattern site three many painting.,2022-01-01 12:50:00
+9388ff42-cbdd-45f4-aa2c-c645df008470,State through pass car represent whatever conference. Structure law painting peace. Hold activity between Mr. Morning international end break summer natural.,2022-01-01 13:00:00
+e6739d96-31af-479b-b851-c4233136f5d4,"Guy play easy indicate even senior. Tree Mrs thought. Threat skill billion how agree.
+Truth future board themselves think when. Decision must newspaper general instead.",2022-01-01 13:10:00
+7a1ddff1-b8af-4e6e-ae84-3f3a96fc5a47,"Mind general shoulder accept remember.
+I cover choice possible whatever example. Dark well sing some.
+Computer interest each position. Agency hear fire discussion moment.",2022-01-01 13:20:00
+6c98dfb2-faa1-4eea-a972-b7112843e19b,"Religious agreement soldier. Price beautiful strategy heavy.
+Meet before action note. Sort someone amount foot. Pm full in either question it under. Model long any catch board election.",2022-01-01 13:30:00
+ede05137-7878-49f1-9cbe-119238b77c08,"Fund close thank technology individual member. Clearly at avoid throw. Culture environment young try ask line center.
+Attack continue week federal budget. Fund almost lose fast property.",2022-01-01 13:40:00
+f4aff2ea-899e-492f-8927-42858f569c2f,"Side factor wait wide simply last itself. Edge move ten.
+Ground boy theory per hit.
+Upon me public water simple east party candidate. Fact eat them difficult. Oil together stay ask rock idea.",2022-01-01 13:50:00
+0e788389-d481-4de3-9111-8432517c4a2a,Doctor interview just computer management live. Theory idea option board low box treatment. Hold couple difficult product.,2022-01-01 14:00:00
+c4060e04-fbfb-4b82-8d8b-4bd663790acf,"Set benefit they happen number especially. Finish Mrs position option.
+Business center simple question easy understand impact like. Enjoy important language.",2022-01-01 14:10:00
+8eca8bce-792a-475c-8fe8-01ef77e9316b,Plant consumer guy religious society animal sort worker. Mind use scientist us mean worker. Represent maybe general who poor reveal.,2022-01-01 14:20:00
+de89d520-24cc-4f0b-b97c-838b03949b1c,"Maintain through realize along. Make section writer many.
+Her out year upon simple easy picture. Help political imagine various nothing finish. Charge from trip yes blood.",2022-01-01 14:30:00
+4d96429c-63f5-48c5-9512-1aa8e1c1a60f,"Side behind power all thing current body. Send chair white take could wear. Three pattern might quickly spend cup life.
+Sister middle agency beyond. Strategy practice base continue.",2022-01-01 14:40:00
+bb9850ea-759c-4de7-98fc-8e21c040cf79,While rate data. Food land though street operation beyond study. Side foreign begin herself however movement.,2022-01-01 14:50:00
+b1ba7f50-b7a8-4f6a-b11e-a1d71bba8733,"Watch fine choice article. Item customer real force option treat.
+Travel thought health. Look test already former risk chair arrive task. Way red kid evidence eye.",2022-01-01 15:00:00
+ac0a4df6-9c7a-4878-9424-56c8ba02d513,"With anything create either child. Tonight later view cover arm measure.
+And industry speech several huge charge. Save phone natural present. Always conference plant build.",2022-01-01 15:10:00
+4a5de80b-4281-44cf-a151-ec6f0dc9d541,"Thing maybe order church near. Night price factor little total fact parent. Begin let air.
+Guy just result art how figure.
+Strong spring organization. Theory along issue attorney issue suffer pick.",2022-01-01 15:20:00
+97e4a76a-c8f6-4dc4-af20-857201d2bbbb,"Campaign account on glass of quality professional.
+Drop report them around kind community. Author off develop recent phone book.",2022-01-01 15:30:00
+7fa29328-27e0-4bbf-96bb-27b535a875ae,High result drop matter one since sure. Prove century little prepare very safe similar personal. Chance according enough.,2022-01-01 15:40:00
+773faa8b-713e-4541-a251-84150eb2df19,"Economy teacher bag item throw. Past significant scientist each yet book beat.
+That consider into. Perhaps agree approach bar political drop either.",2022-01-01 15:50:00
+f1e489ef-c16e-41ba-8590-0a118a5bbac6,Require phone low free. Meeting local use talk throughout general. Ready right work social wind near interesting.,2022-01-01 16:00:00
+7f247668-4e61-4504-a700-87f3056c57c5,"Democratic create put prepare maybe. Now development all focus.
+Already represent again mission. Enough summer watch someone ready reduce your activity.",2022-01-01 16:10:00
+cd6fab3b-6c60-4104-bae9-14657337a06c,Difficult enjoy government hair concern question charge. Chair best shoulder quality unit successful I. Our without to sometimes authority.,2022-01-01 16:20:00
+3b85c1b2-deaf-4550-8f50-6cce85d497f9,"Effect ball college particular. Best network entire there. Analysis picture interest anyone.
+Letter someone special avoid allow resource those.",2022-01-01 16:30:00

docs/modules/indexes/document_loaders/examples/googledrive.ipynb

@@ -44,7 +44,11 @@
    },
    "outputs": [],
    "source": [
-    "loader = GoogleDriveLoader(folder_id=\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\")"
+    "loader = GoogleDriveLoader(\n",
+    "    folder_id=\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\",\n",
+    "    # Optional: configure whether to recursively fetch files from subfolders. Defaults to False.\n",
+    "    recursive=False\n",
+    ")"
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/obsidian.ipynb

@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "1dc7df1d",
    "metadata": {},
@@ -8,7 +9,9 @@
     "# Obsidian\n",
     "This notebook covers how to load documents from an Obsidian database.\n",
     "\n",
-    "Since Obsidian is just stored on disk as a folder of Markdown files, the loader just takes a path to this directory."
+    "Since Obsidian is just stored on disk as a folder of Markdown files, the loader just takes a path to this directory.\n",
+    "\n",
+    "Obsidian files also sometimes contain [metadata](https://help.obsidian.md/Editing+and+formatting/Metadata) which is a YAML block at the top of the file. These values will be added to the document's metadata. (`ObsidianLoader` can also be passed a `collect_metadata=False` argument to disable this behavior.)"
    ]
   },
   {

docs/modules/indexes/document_loaders/examples/twitter.ipynb

@@ -0,0 +1,114 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "66a7777e",
+   "metadata": {},
+   "source": [
+    "# Twitter\n",
+    "\n",
+    "This loader fetches the text from the Tweets of a list of Twitter users, using the `tweepy` Python package.\n",
+    "You must initialize the loader with your Twitter API token, and you need to pass in the Twitter username you want to extract."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9ec8a3b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TwitterTweetLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "43128d8d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install tweepy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "35d6809a",
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "loader = TwitterTweetLoader.from_bearer_token(\n",
+    "    oauth2_bearer_token=\"YOUR BEARER TOKEN\",\n",
+    "    twitter_users=['elonmusk'],\n",
+    "    number_tweets=50,  # Default value is 100\n",
+    ")\n",
+    "\n",
+    "# Or load from access token and consumer keys\n",
+    "# loader = TwitterTweetLoader.from_secrets(\n",
+    "#     access_token='YOUR ACCESS TOKEN',\n",
+    "#     access_token_secret='YOUR ACCESS TOKEN SECRET',\n",
+    "#     consumer_key='YOUR CONSUMER KEY',\n",
+    "#     consumer_secret='YOUR CONSUMER SECRET',\n",
+    "#     twitter_users=['elonmusk'],\n",
+    "#     number_tweets=50,\n",
+    "# )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "05fe33b9",
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='@MrAndyNgo @REI One store after another shutting down', metadata={'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
+       " Document(page_content='@KanekoaTheGreat @joshrogin @glennbeck Large ships are fundamentally vulnerable to ballistic (hypersonic) missiles', metadata={'created_at': 'Tue Apr 18 03:43:25 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
+       " Document(page_content='@KanekoaTheGreat The Golden Rule', metadata={'created_at': 'Tue Apr 18 03:37:17 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
+       " Document(page_content='@KanekoaTheGreat 🧐', metadata={'created_at': 'Tue Apr 18 03:35:48 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),\n",
+       " Document(page_content='@TRHLofficial What’s he talking about and why is it sponsored by Erik’s son?', metadata={'created_at': 'Tue Apr 18 03:32:17 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳️\\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}})]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "documents = loader.load()\n",
+    "documents[:5]"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/indexes/retrievers/examples/metal.ipynb

@@ -32,9 +32,9 @@
     "from metal_sdk.metal import Metal\n",
     "API_KEY = \"\"\n",
     "CLIENT_ID = \"\"\n",
-    "APP_ID = \"\"\n",
+    "INDEX_ID = \"\"\n",
     "\n",
-    "metal = Metal(API_KEY, CLIENT_ID, APP_ID);\n"
+    "metal = Metal(API_KEY, CLIENT_ID, INDEX_ID);\n"
    ]
   },
   {

docs/modules/indexes/retrievers/examples/time_weighted_vectorstore.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "a90b7557",
    "metadata": {},
@@ -37,7 +36,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "6af7ea6b",
    "metadata": {},
@@ -72,7 +70,7 @@
     {
      "data": {
       "text/plain": [
-       "['30fac87b-0357-450c-9a92-63e0147d005c']"
+       "['5c9f7c06-c9eb-45f2-aea5-efce5fb9f2bd']"
       ]
      },
      "execution_count": 3,
@@ -95,7 +93,7 @@
     {
      "data": {
       "text/plain": [
-       "[Document(page_content='hello world', metadata={'last_accessed_at': datetime.datetime(2023, 4, 17, 6, 52, 55, 670204), 'created_at': datetime.datetime(2023, 4, 17, 6, 52, 54, 819346), 'buffer_idx': 0})]"
+       "[Document(page_content='hello world', metadata={'last_accessed_at': datetime.datetime(2023, 4, 16, 22, 9, 1, 966261), 'created_at': datetime.datetime(2023, 4, 16, 22, 9, 0, 374683), 'buffer_idx': 0})]"
       ]
      },
      "execution_count": 4,
@@ -109,7 +107,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "ca056896",
    "metadata": {},
@@ -144,7 +141,7 @@
     {
      "data": {
       "text/plain": [
-       "['a8d823e6-baee-46c8-aaa2-519cfb9b59ba']"
+       "['40011466-5bbe-4101-bfd1-e22e7f505de2']"
       ]
      },
      "execution_count": 6,
@@ -167,7 +164,7 @@
     {
      "data": {
       "text/plain": [
-       "[Document(page_content='hello foo', metadata={'last_accessed_at': datetime.datetime(2023, 4, 17, 6, 52, 57, 11210), 'created_at': datetime.datetime(2023, 4, 17, 6, 52, 56, 17459), 'buffer_idx': 1})]"
+       "[Document(page_content='hello foo', metadata={'last_accessed_at': datetime.datetime(2023, 4, 16, 22, 9, 2, 494798), 'created_at': datetime.datetime(2023, 4, 16, 22, 9, 2, 178722), 'buffer_idx': 1})]"
       ]
      },
      "execution_count": 7,
@@ -205,7 +202,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.2"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/modules/indexes/vectorstores/examples/pinecone.ipynb

@@ -58,6 +58,9 @@
     "\n",
     "docsearch = Pinecone.from_documents(docs, embeddings, index_name=index_name)\n",
     "\n",
+    "# if you already have an index, you can load it like this\n",
+    "# docsearch = Pinecone.from_existing_index(index_name, embeddings)\n",
+    "\n",
     "query = \"What did the president say about Ketanji Brown Jackson\"\n",
     "docs = docsearch.similarity_search(query)"
    ]

docs/modules/memory/types/entity_summary_memory.ipynb

@@ -161,7 +161,7 @@
       "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
       "\n",
       "Context:\n",
-      "{'Deven': '', 'Sam': ''}\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam.', 'Sam': 'Sam is working on a hackathon project with Deven.'}\n",
       "\n",
       "Current conversation:\n",
       "\n",
@@ -189,29 +189,29 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 14,
    "id": "0269f513",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "{'Deven': 'Deven is working on a hackathon project with Sam.',\n",
+       "{'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon.',\n",
        " 'Sam': 'Sam is working on a hackathon project with Deven.'}"
       ]
      },
-     "execution_count": 9,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "conversation.memory.store"
+    "conversation.memory.entity_store.store"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 15,
    "id": "46324ca8",
    "metadata": {},
    "outputs": [
@@ -232,7 +232,7 @@
       "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
       "\n",
       "Context:\n",
-      "{'Deven': 'Deven is working on a hackathon project with Sam.', 'Sam': 'Sam is working on a hackathon project with Deven.', 'Langchain': ''}\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon.', 'Sam': 'Sam is working on a hackathon project with Deven.', 'Langchain': ''}\n",
       "\n",
       "Current conversation:\n",
       "Human: Deven & Sam are working on a hackathon project\n",
@@ -250,7 +250,7 @@
        "' That sounds like an interesting project! What kind of memory structures are they trying to add?'"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -261,7 +261,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 16,
    "id": "ff2ebf6b",
    "metadata": {},
    "outputs": [
@@ -282,7 +282,7 @@
       "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
       "\n",
       "Context:\n",
-      "{'Deven': 'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.', 'Langchain': 'Langchain is a project that is trying to add more complex memory structures.', 'Key-Value Store': ''}\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon. They are trying to add more complex memory structures to Langchain.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.', 'Langchain': 'Langchain is a project that is trying to add more complex memory structures.', 'Key-Value Store': ''}\n",
       "\n",
       "Current conversation:\n",
       "Human: Deven & Sam are working on a hackathon project\n",
@@ -299,10 +299,10 @@
     {
      "data": {
       "text/plain": [
-       "' That sounds like a great idea! How will the key-value store work?'"
+       "' That sounds like a great idea! How will the key-value store help with the project?'"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 16,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -313,7 +313,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 17,
    "id": "56cfd4ba",
    "metadata": {},
    "outputs": [
@@ -334,7 +334,7 @@
       "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
       "\n",
       "Context:\n",
-      "{'Deven': 'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'}\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon. They are trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'}\n",
       "\n",
       "Current conversation:\n",
       "Human: Deven & Sam are working on a hackathon project\n",
@@ -342,7 +342,7 @@
       "Human: They are trying to add more complex memory structures to Langchain\n",
       "AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?\n",
       "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
-      "AI:  That sounds like a great idea! How will the key-value store work?\n",
+      "AI:  That sounds like a great idea! How will the key-value store help with the project?\n",
       "Last line:\n",
       "Human: What do you know about Deven & Sam?\n",
       "You:\u001b[0m\n",
@@ -353,10 +353,10 @@
     {
      "data": {
       "text/plain": [
-       "' Deven and Sam are working on a hackathon project together, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'"
+       "' Deven and Sam are working on a hackathon project together, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be working hard on this project and have a great idea for how the key-value store can help.'"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -376,7 +376,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 21,
    "id": "038b4d3f",
    "metadata": {},
    "outputs": [
@@ -384,28 +384,34 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "{'Deven': 'Deven is working on a hackathon project with Sam, attempting to add '\n",
-      "          'more complex memory structures to Langchain, including a key-value '\n",
-      "          'store for entities mentioned so far in the conversation.',\n",
-      " 'Key-Value Store': 'A key-value store that stores entities mentioned in the '\n",
-      "                    'conversation.',\n",
+      "{'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur.',\n",
+      " 'Deven': 'Deven is working on a hackathon project with Sam, which they are '\n",
+      "          'entering into a hackathon. They are trying to add more complex '\n",
+      "          'memory structures to Langchain, including a key-value store for '\n",
+      "          'entities mentioned so far in the conversation, and seem to be '\n",
+      "          'working hard on this project with a great idea for how the '\n",
+      "          'key-value store can help.',\n",
+      " 'Key-Value Store': 'A key-value store is being added to the project to store '\n",
+      "                    'entities mentioned in the conversation.',\n",
       " 'Langchain': 'Langchain is a project that is trying to add more complex '\n",
       "              'memory structures, including a key-value store for entities '\n",
       "              'mentioned so far in the conversation.',\n",
-      " 'Sam': 'Sam is working on a hackathon project with Deven, attempting to add '\n",
-      "        'more complex memory structures to Langchain, including a key-value '\n",
-      "        'store for entities mentioned so far in the conversation.'}\n"
+      " 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more '\n",
+      "        'complex memory structures to Langchain, including a key-value store '\n",
+      "        'for entities mentioned so far in the conversation. They seem to have '\n",
+      "        'a great idea for how the key-value store can help, and Sam is also '\n",
+      "        'the founder of a company called Daimon.'}\n"
      ]
     }
    ],
    "source": [
     "from pprint import pprint\n",
-    "pprint(conversation.memory.store)"
+    "pprint(conversation.memory.entity_store.store)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 22,
    "id": "2df4800e",
    "metadata": {},
    "outputs": [
@@ -426,15 +432,16 @@
       "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
       "\n",
       "Context:\n",
-      "{'Daimon': '', 'Sam': 'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.'}\n",
+      "{'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to have a great idea for how the key-value store can help, and Sam is also the founder of a company called Daimon.'}\n",
       "\n",
       "Current conversation:\n",
-      "Human: They are trying to add more complex memory structures to Langchain\n",
-      "AI:  That sounds like an interesting project! What kind of memory structures are they trying to add?\n",
       "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
-      "AI:  That sounds like a great idea! How will the key-value store work?\n",
+      "AI:  That sounds like a great idea! How will the key-value store help with the project?\n",
       "Human: What do you know about Deven & Sam?\n",
-      "AI:  Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success.\n",
+      "AI:  Deven and Sam are working on a hackathon project together, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be working hard on this project and have a great idea for how the key-value store can help.\n",
+      "Human: Sam is the founder of a company called Daimon.\n",
+      "AI: \n",
+      "That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\n",
       "Last line:\n",
       "Human: Sam is the founder of a company called Daimon.\n",
       "You:\u001b[0m\n",
@@ -445,10 +452,10 @@
     {
      "data": {
       "text/plain": [
-       "\"\\nThat's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\""
+       "\" That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\""
       ]
      },
-     "execution_count": 8,
+     "execution_count": 22,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -459,7 +466,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 24,
    "id": "ebe9e36f",
    "metadata": {},
    "outputs": [
@@ -467,32 +474,36 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "{'Daimon': 'Daimon is a company founded by Sam.',\n",
-      " 'Deven': 'Deven is working on a hackathon project with Sam to add more '\n",
-      "          'complex memory structures to Langchain, including a key-value store '\n",
-      "          'for entities mentioned so far in the conversation.',\n",
-      " 'Key-Value Store': 'Key-Value Store: A data structure that stores values '\n",
-      "                    'associated with a unique key, allowing for efficient '\n",
-      "                    'retrieval of values. Deven and Sam are adding a key-value '\n",
-      "                    'store for entities mentioned so far in the conversation.',\n",
-      " 'Langchain': 'Langchain is a project that seeks to add more complex memory '\n",
-      "              'structures, including a key-value store for entities mentioned '\n",
-      "              'so far in the conversation.',\n",
-      " 'Sam': 'Sam is working on a hackathon project with Deven to add more complex '\n",
-      "        'memory structures to Langchain, including a key-value store for '\n",
-      "        'entities mentioned so far in the conversation. He is also the founder '\n",
-      "        'of a company called Daimon.'}\n"
+      "{'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur, who '\n",
+      "           'is working on a hackathon project with Deven to add more complex '\n",
+      "           'memory structures to Langchain.',\n",
+      " 'Deven': 'Deven is working on a hackathon project with Sam, which they are '\n",
+      "          'entering into a hackathon. They are trying to add more complex '\n",
+      "          'memory structures to Langchain, including a key-value store for '\n",
+      "          'entities mentioned so far in the conversation, and seem to be '\n",
+      "          'working hard on this project with a great idea for how the '\n",
+      "          'key-value store can help.',\n",
+      " 'Key-Value Store': 'A key-value store is being added to the project to store '\n",
+      "                    'entities mentioned in the conversation.',\n",
+      " 'Langchain': 'Langchain is a project that is trying to add more complex '\n",
+      "              'memory structures, including a key-value store for entities '\n",
+      "              'mentioned so far in the conversation.',\n",
+      " 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more '\n",
+      "        'complex memory structures to Langchain, including a key-value store '\n",
+      "        'for entities mentioned so far in the conversation. They seem to have '\n",
+      "        'a great idea for how the key-value store can help, and Sam is also '\n",
+      "        'the founder of a successful company called Daimon.'}\n"
      ]
     }
    ],
    "source": [
     "from pprint import pprint\n",
-    "pprint(conversation.memory.store)"
+    "pprint(conversation.memory.entity_store.store)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 25,
    "id": "dd547144",
    "metadata": {},
    "outputs": [
@@ -513,16 +524,16 @@
       "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
       "\n",
       "Context:\n",
-      "{'Sam': 'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He is also the founder of a company called Daimon.', 'Daimon': 'Daimon is a company founded by Sam.'}\n",
+      "{'Deven': 'Deven is working on a hackathon project with Sam, which they are entering into a hackathon. They are trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation, and seem to be working hard on this project with a great idea for how the key-value store can help.', 'Sam': 'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to have a great idea for how the key-value store can help, and Sam is also the founder of a successful company called Daimon.', 'Langchain': 'Langchain is a project that is trying to add more complex memory structures, including a key-value store for entities mentioned so far in the conversation.', 'Daimon': 'Daimon is a company founded by Sam, a successful entrepreneur, who is working on a hackathon project with Deven to add more complex memory structures to Langchain.'}\n",
       "\n",
       "Current conversation:\n",
-      "Human: They are adding in a key-value store for entities mentioned so far in the conversation.\n",
-      "AI:  That sounds like a great idea! How will the key-value store work?\n",
       "Human: What do you know about Deven & Sam?\n",
-      "AI:  Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success.\n",
+      "AI:  Deven and Sam are working on a hackathon project together, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be working hard on this project and have a great idea for how the key-value store can help.\n",
       "Human: Sam is the founder of a company called Daimon.\n",
       "AI: \n",
       "That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\n",
+      "Human: Sam is the founder of a company called Daimon.\n",
+      "AI:  That's impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\n",
       "Last line:\n",
       "Human: What do you know about Sam?\n",
       "You:\u001b[0m\n",
@@ -533,10 +544,10 @@
     {
      "data": {
       "text/plain": [
-       "' Sam is the founder of a company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He seems to be very motivated and passionate about his project, and is working hard to make it a success.'"
+       "' Sam is the founder of a successful company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain. They seem to have a great idea for how the key-value store can help.'"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 25,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -570,7 +581,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.10"
   }
  },
  "nbformat": 4,

docs/modules/models/llms/integrations/ai21.ipynb

@@ -6,14 +6,55 @@
    "metadata": {},
    "source": [
     "# AI21\n",
-    "This example goes over how to use LangChain to interact with AI21 models"
+    "\n",
+    "[AI21 Studio](https://docs.ai21.com/) provides API access to `Jurassic-2` large language models.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with [AI21 models](https://docs.ai21.com/docs/jurassic-2-models)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
+   "id": "02be122d-04e8-4ec6-84d1-f1d8961d6828",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# install the package:\n",
+    "!pip install ai21"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "4229227e-6ca2-41ad-a3c3-5f29e3559091",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get AI21_API_KEY. Use https://studio.ai21.com/account/account\n",
+    "\n",
+    "from getpass import getpass\n",
+    "AI21_API_KEY  = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
    "id": "6fb585dd",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import AI21\n",
@@ -22,9 +63,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 9,
    "id": "035dea0f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -36,19 +79,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 10,
    "id": "3f3458d9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "llm = AI21()"
+    "llm = AI21(ai21_api_key=AI21_API_KEY)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 11,
    "id": "a641dbd9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
@@ -56,10 +103,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 12,
    "id": "9f0b1960",
-   "metadata": {},
-   "outputs": [],
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'\\n1. What year was Justin Bieber born?\\nJustin Bieber was born in 1994.\\n2. What team won the Super Bowl in 1994?\\nThe Dallas Cowboys won the Super Bowl in 1994.'"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
     "\n",
@@ -91,7 +151,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/modules/models/llms/integrations/aleph_alpha.ipynb

@@ -1,20 +1,61 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "9597802c",
    "metadata": {},
    "source": [
     "# Aleph Alpha\n",
+    "\n",
+    "[The Luminous series](https://docs.aleph-alpha.com/docs/introduction/luminous/) is a family of large language models.\n",
+    "\n",
     "This example goes over how to use LangChain to interact with Aleph Alpha models"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
+   "id": "fe1bf9fb-e9fa-49f3-a768-8f603225ccce",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Install the package\n",
+    "!pip install aleph-alpha-client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "0cb0f937-b610-42a2-b765-336eed037031",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# create a new token: https://docs.aleph-alpha.com/docs/account/#create-a-new-token\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "ALEPH_ALPHA_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
    "id": "6fb585dd",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import AlephAlpha\n",
@@ -23,9 +64,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 7,
    "id": "f81a230d",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Q: {question}\n",
@@ -37,19 +80,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 8,
    "id": "f0d26e48",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "llm = AlephAlpha(model=\"luminous-extended\", maximum_tokens=20, stop_sequences=[\"Q:\"])"
+    "llm = AlephAlpha(model=\"luminous-extended\", maximum_tokens=20, stop_sequences=[\"Q:\"], aleph_alpha_api_key=ALEPH_ALPHA_API_KEY)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 9,
    "id": "6811d621",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
@@ -57,9 +104,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 10,
    "id": "3058e63f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "data": {
@@ -67,7 +116,7 @@
        "' Artificial Intelligence (AI) is the simulation of human intelligence processes by machines, especially computer systems.\\n'"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -81,7 +130,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -95,7 +144,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/modules/models/llms/integrations/anthropic_example.ipynb

@@ -6,14 +6,46 @@
    "metadata": {},
    "source": [
     "# Anthropic\n",
-    "This example goes over how to use LangChain to interact with Anthropic models"
+    "\n",
+    "[Anthropic](https://console.anthropic.com/docs) is creator of the `Claude` LLM.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with Anthropic models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e55c0f2e-63e1-4e83-ac44-ffcc1dfeacc8",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Install the package\n",
+    "!pip install anthropic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cec62d45-afa2-422a-95ef-57f8ab41a6f9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# get a new token: https://www.anthropic.com/earlyaccess\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "ANTHROPIC_API_KEY = getpass()"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 1,
    "id": "6fb585dd",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import Anthropic\n",
@@ -24,7 +56,9 @@
    "cell_type": "code",
    "execution_count": 2,
    "id": "035dea0f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -36,12 +70,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "id": "3f3458d9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "llm = Anthropic()"
+    "llm = Anthropic(, anthropic_api_key=ANTHROPIC_API_KEY)"
    ]
   },
   {
@@ -102,7 +138,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/modules/models/llms/integrations/azure_openai_example.ipynb

@@ -5,7 +5,7 @@
    "id": "9e9b7651",
    "metadata": {},
    "source": [
-    "# Azure OpenAI LLM Example\n",
+    "# Azure OpenAI\n",
     "\n",
     "This notebook goes over how to use Langchain with [Azure OpenAI](https://aka.ms/azure-openai).\n",
     "\n",
@@ -49,6 +49,18 @@
     "```\n"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "89fdb593-5a42-4098-87b7-1496fa511b1c",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install openai"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 1,
@@ -146,7 +158,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/modules/models/llms/integrations/banana.ipynb

@@ -5,19 +5,50 @@
    "metadata": {},
    "source": [
     "# Banana\n",
+    "\n",
+    "\n",
+    "[Banana](https://www.banana.dev/about-us) is focused on building the machine learning infrastructure.\n",
+    "\n",
     "This example goes over how to use LangChain to interact with Banana models"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Install the package  https://docs.banana.dev/banana-docs/core-concepts/sdks/python\n",
+    "!pip install banana-dev"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
+    "# get new tokens: https://app.banana.dev/\n",
+    "# We need two tokens, not just an `api_key`: `BANANA_API_KEY` and `YOUR_MODEL_KEY`\n",
+    "\n",
     "import os\n",
+    "from getpass import getpass\n",
+    "\n",
+    "os.environ[\"BANANA_API_KEY\"] = \"YOUR_API_KEY\"\n",
+    "# OR\n",
+    "# BANANA_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "from langchain.llms import Banana\n",
-    "from langchain import PromptTemplate, LLMChain\n",
-    "os.environ[\"BANANA_API_KEY\"] = \"YOUR_API_KEY\""
+    "from langchain import PromptTemplate, LLMChain"
    ]
   },
   {
@@ -65,15 +96,22 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -81,5 +119,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/cerebriumai_example.ipynb

@@ -4,7 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# CerebriumAI LLM Example\n",
+    "# CerebriumAI\n",
+    "\n",
+    "`Cerebrium` is an AWS Sagemaker alternative. It also provides API access to [several LLM models](https://docs.cerebrium.ai/cerebrium/prebuilt-models/deploymen).\n",
+    "\n",
     "This notebook goes over how to use Langchain with [CerebriumAI](https://docs.cerebrium.ai/introduction)."
    ]
   },
@@ -13,7 +16,7 @@
    "metadata": {},
    "source": [
     "## Install cerebrium\n",
-    "The `cerebrium` package is required to use the CerebriumAI API. Install `cerebrium` using `pip3 install cerebrium`."
+    "The `cerebrium` package is required to use the `CerebriumAI` API. Install `cerebrium` using `pip3 install cerebrium`."
    ]
   },
   {
@@ -22,7 +25,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "$ pip3 install cerebrium"
+    "# Install the package\n",
+    "!pip3 install cerebrium"
    ]
   },
   {
@@ -48,7 +52,7 @@
    "metadata": {},
    "source": [
     "## Set the Environment API Key\n",
-    "Make sure to get your API key from CerebriumAI. You are given a 1 hour free of serverless GPU compute to test different models."
+    "Make sure to get your API key from CerebriumAI. See [here](https://dashboard.cerebrium.ai/login). You are given a 1 hour free of serverless GPU compute to test different models."
    ]
   },
   {
@@ -136,15 +140,22 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -152,5 +163,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/cohere.ipynb

@@ -6,14 +6,56 @@
    "metadata": {},
    "source": [
     "# Cohere\n",
-    "This example goes over how to use LangChain to interact with Cohere models"
+    "\n",
+    "[Cohere](https://cohere.ai/about) is a Canadian startup that provides natural language processing models that help companies improve human-machine interactions.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `Cohere` [models](https://docs.cohere.ai/docs/generation-card)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
+   "id": "91ea14ce-831d-409a-a88f-30353acdabd1",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Install the package\n",
+    "!pip install cohere"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "3f5dc9d7-65e3-4b5b-9086-3327d016cfe0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get a new token: https://dashboard.cohere.ai/\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "COHERE_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
    "id": "6fb585dd",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import Cohere\n",
@@ -22,9 +64,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 4,
    "id": "035dea0f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -36,19 +80,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 6,
    "id": "3f3458d9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "llm = Cohere()"
+    "llm = Cohere(cohere_api_key=COHERE_API_KEY)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 7,
    "id": "a641dbd9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
@@ -102,7 +150,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/modules/models/llms/integrations/deepinfra_example.ipynb

@@ -1,11 +1,13 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# DeepInfra LLM Example\n",
+    "# DeepInfra\n",
+    "\n",
+    "`DeepInfra` provides [several LLMs](https://deepinfra.com/models).\n",
+    "\n",
     "This notebook goes over how to use Langchain with [DeepInfra](https://deepinfra.com)."
    ]
   },
@@ -18,8 +20,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "import os\n",
@@ -32,17 +36,44 @@
    "metadata": {},
    "source": [
     "## Set the Environment API Key\n",
-    "Make sure to get your API key from DeepInfra. You are given a 1 hour free of serverless GPU compute to test different models.\n",
+    "Make sure to get your API key from DeepInfra. You have to [Login](https://deepinfra.com/login?from=%2Fdash) and get a new token.\n",
+    "\n",
+    "You are given a 1 hour free of serverless GPU compute to test different models. (see [here](https://github.com/deepinfra/deepctl#deepctl))\n",
     "You can print your token with `deepctl auth token`"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get a new token: https://deepinfra.com/login?from=%2Fdash\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "DEEPINFRA_API_TOKEN = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "os.environ[\"DEEPINFRA_API_TOKEN\"] = \"YOUR_KEY_HERE\""
+    "os.environ[\"DEEPINFRA_API_TOKEN\"] = DEEPINFRA_API_TOKEN"
    ]
   },
   {
@@ -50,7 +81,7 @@
    "metadata": {},
    "source": [
     "## Create the DeepInfra instance\n",
-    "Make sure to deploy your model first via `deepctl deploy create -m google/flat-t5-xl` (for example)"
+    "Make sure to deploy your model first via `deepctl deploy create -m google/flat-t5-xl` (see [here](https://github.com/deepinfra/deepctl#deepctl))"
    ]
   },
   {
@@ -121,15 +152,22 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -137,5 +175,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/forefrontai_example.ipynb

@@ -4,8 +4,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# ForefrontAI LLM Example\n",
-    "This notebook goes over how to use Langchain with [ForefrontAI](https://www.forefront.ai/)."
+    "# ForefrontAI\n",
+    "\n",
+    "\n",
+    "The `Forefront` platform gives you the ability to fine-tune and use [open source large language models](https://docs.forefront.ai/forefront/master/models).\n",
+    "\n",
+    "This notebook goes over how to use Langchain with [ForefrontAI](https://www.forefront.ai/).\n"
    ]
   },
   {
@@ -40,7 +44,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "os.environ[\"FOREFRONTAI_API_KEY\"] = \"YOUR_KEY_HERE\""
+    "# get a new token: https://docs.forefront.ai/forefront/api-reference/authentication\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "FOREFRONTAI_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"FOREFRONTAI_API_KEY\"] = FOREFRONTAI_API_KEY"
    ]
   },
   {
@@ -119,15 +136,22 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -135,5 +159,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/gooseai_example.ipynb

@@ -4,8 +4,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# GooseAI LLM Example\n",
-    "This notebook goes over how to use Langchain with [GooseAI](https://goose.ai/)."
+    "# GooseAI\n",
+    "\n",
+    "`GooseAI` is a fully managed NLP-as-a-Service, delivered via API. GooseAI provides access to [these models](https://goose.ai/docs/models).\n",
+    "\n",
+    "This notebook goes over how to use Langchain with [GooseAI](https://goose.ai/).\n"
    ]
   },
   {
@@ -57,7 +60,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "os.environ[\"GOOSEAI_API_KEY\"] = \"YOUR_KEY_HERE\""
+    "from getpass import getpass\n",
+    "\n",
+    "GOOSEAI_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"GOOSEAI_API_KEY\"] = GOOSEAI_API_KEY"
    ]
   },
   {
@@ -136,15 +150,22 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -152,5 +173,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/gpt4all.ipynb

@@ -6,22 +6,36 @@
    "source": [
     "# GPT4All\n",
     "\n",
-    "This example goes over how to use LangChain to interact with GPT4All models"
+    "[GitHub:nomic-ai/gpt4all](https://github.com/nomic-ai/gpt4all) an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `GPT4All` models."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
    "source": [
     "%pip install pyllamacpp > /dev/null"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain import PromptTemplate, LLMChain\n",
@@ -32,8 +46,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -51,6 +67,10 @@
     "\n",
     "To run locally, download a compatible ggml-formatted model. For more info, visit https://github.com/nomic-ai/pyllamacpp\n",
     "\n",
+    "For full installation instructions go [here](https://gpt4all.io/index.html).\n",
+    "\n",
+    "The GPT4All Chat installer needs to decompress a 3GB LLM model during the installation process!\n",
+    "\n",
     "Note that new models are uploaded regularly - check the link above for the most recent `.bin` URL"
    ]
   },
@@ -146,9 +166,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.2"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/huggingface_hub.ipynb

@@ -7,9 +7,57 @@
    "source": [
     "# Hugging Face Hub\n",
     "\n",
+    "The [Hugging Face Hub](https://huggingface.co/docs/hub/index) is a platform with over 120k models, 20k datasets, and 50k demo apps (Spaces), all open source and publicly available, in an online platform where people can easily collaborate and build ML together.\n",
+    "\n",
     "This example showcases how to connect to the Hugging Face Hub."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "4c1b8450-5eaf-4d34-8341-2d785448a1ff",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "To use, you should have the ``huggingface_hub`` python [package installed](https://huggingface.co/docs/huggingface_hub/installation)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d772b637-de00-4663-bd77-9bc96d798db2",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install huggingface_hub"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d597a792-354c-4ca5-b483-5965eec5d63d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# get a token: https://huggingface.co/docs/api-inference/quicktour#get-your-api-token\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "HUGGINGFACEHUB_API_TOKEN = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b8c5b88c-e4b8-4d0d-9a35-6e8f106452c2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"HUGGINGFACEHUB_API_TOKEN\"] = HUGGINGFACEHUB_API_TOKEN"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 41,
@@ -63,7 +111,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.12"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/modules/models/llms/integrations/llamacpp.ipynb

@@ -6,22 +6,38 @@
    "source": [
     "# Llama-cpp\n",
     "\n",
-    "This notebook goes over how to run llama-cpp within LangChain"
+    "[llama-cpp](https://github.com/abetlen/llama-cpp-python) is a Python binding for [llama.cpp](https://github.com/ggerganov/llama.cpp). \n",
+    "It supports [several LLMs](https://github.com/ggerganov/llama.cpp).\n",
+    "\n",
+    "This notebook goes over how to run `llama-cpp` within LangChain."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!pip install llama-cpp-python"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": 2,
+   "cell_type": "markdown",
    "metadata": {},
+   "source": [
+    "Make sure you are following all instructions to [install all necessary model files](https://github.com/ggerganov/llama.cpp).\n",
+    "\n",
+    "You don't need an `API_TOKEN`!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import LlamaCpp\n",
@@ -30,8 +46,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -44,7 +62,9 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm = LlamaCpp(model_path=\"./ggml-model-q4_0.bin\")"
@@ -98,9 +118,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/manifest.ipynb

@@ -15,14 +15,30 @@
    "id": "59fcaebc",
    "metadata": {},
    "source": [
-    "For more detailed information on `manifest`, and how to use it with local hugginface models like in this example, see https://github.com/HazyResearch/manifest"
+    "For more detailed information on `manifest`, and how to use it with local hugginface models like in this example, see https://github.com/HazyResearch/manifest\n",
+    "\n",
+    "Another example of [using Manifest with Langchain](https://github.com/HazyResearch/manifest/blob/main/examples/langchain_chatgpt.ipynb)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
+   "id": "1205d1e4-e6da-4d67-a0c7-b7e8fd1e98d5",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install manifest-ml"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
    "id": "04a0170a",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from manifest import Manifest\n",
@@ -31,18 +47,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "de250a6a",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'model_name': 'bigscience/T0_3B', 'model_path': 'bigscience/T0_3B'}\n"
-     ]
-    }
-   ],
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
    "source": [
     "manifest = Manifest(\n",
     "    client_name = \"huggingface\",\n",
@@ -202,7 +212,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/modules/models/llms/integrations/modal.ipynb

@@ -5,7 +5,60 @@
    "metadata": {},
    "source": [
     "# Modal\n",
-    "This example goes over how to use LangChain to interact with Modal models"
+    "\n",
+    "The [Modal Python Library](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. \n",
+    "The `Modal` itself does not provide any LLMs but only the infrastructure.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `Modal`.\n",
+    "\n",
+    "[Here](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example how to use LangChain to interact with `Modal`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install modal-client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[?25lLaunching login page in your browser window\u001b[33m...\u001b[0m\n",
+      "\u001b[2KIf this is not showing up, please copy this URL into your web browser manually:\n",
+      "\u001b[2Km⠙\u001b[0m Waiting for authentication in the web browser...\n",
+      "\u001b]8;id=417802;https://modal.com/token-flow/tf-ptEuGecm7T1T5YQe42kwM1\u001b\\\u001b[4;94mhttps://modal.com/token-flow/tf-ptEuGecm7T1T5YQe42kwM1\u001b[0m\u001b]8;;\u001b\\\n",
+      "\n",
+      "\u001b[2K\u001b[32m⠙\u001b[0m Waiting for authentication in the web browser...\n",
+      "\u001b[1A\u001b[2K^C\n",
+      "\n",
+      "\u001b[31mAborted.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# register and get a new token\n",
+    "\n",
+    "!modal token new"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Follow [these instructions](https://modal.com/docs/guide/secrets) to deal with secrets."
    ]
   },
   {
@@ -63,15 +116,22 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -79,5 +139,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/nlpcloud.ipynb

@@ -0,0 +1,171 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9597802c",
+   "metadata": {},
+   "source": [
+    "# NLP Cloud\n",
+    "\n",
+    "The [NLP Cloud](https://nlpcloud.io) serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, grammar and spelling correction, keywords and keyphrases extraction, chatbot, product description and ad generation, intent classification, text generation, image generation, blog post generation, code generation, question answering, automatic speech recognition, machine translation, language detection, semantic search, semantic similarity, tokenization, POS tagging, embeddings, and dependency parsing. It is ready for production, served through a REST API.\n",
+    "\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `NLP Cloud` [models](https://docs.nlpcloud.com/#models)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8e94b1ca-6e84-44c4-91ca-df7364c007f0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install nlpcloud"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "ea7adb58-cabe-4a2c-b0a2-988fc3aac012",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get a token: https://docs.nlpcloud.com/#authentication\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "NLPCLOUD_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "9cc2d68f-52a8-4a11-ba34-bb6c068e0b6a",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"NLPCLOUD_API_KEY\"] = NLPCLOUD_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "6fb585dd",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.llms import NLPCloud\n",
+    "from langchain import PromptTemplate, LLMChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "035dea0f",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "template = \"\"\"Question: {question}\n",
+    "\n",
+    "Answer: Let's think step by step.\"\"\"\n",
+    "\n",
+    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "3f3458d9",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "llm = NLPCloud()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "a641dbd9",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "9f844993",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' Justin Bieber was born in 1994, so the team that won the Super Bowl that year was the San Francisco 49ers.'"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
+    "\n",
+    "llm_chain.run(question)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/modules/models/llms/integrations/openai.ipynb

@@ -6,14 +6,57 @@
    "metadata": {},
    "source": [
     "# OpenAI\n",
-    "This example goes over how to use LangChain to interact with OpenAI models"
+    "\n",
+    "[OpenAI](https://platform.openai.com/docs/introduction) offers a spectrum of models with different levels of power suitable for different tasks.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `OpenAI` [models](https://platform.openai.com/docs/models)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 4,
+   "id": "5d71df86-8a17-4283-83d7-4e46e7c06c44",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get a token: https://platform.openai.com/account/api-keys\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "OPENAI_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "5472a7cd-af26-48ca-ae9b-5f6ae73c74d2",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
    "id": "6fb585dd",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import OpenAI\n",
@@ -22,9 +65,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 7,
    "id": "035dea0f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -36,9 +81,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 8,
    "id": "3f3458d9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm = OpenAI()"
@@ -46,9 +93,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 9,
    "id": "a641dbd9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
@@ -56,17 +105,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 10,
    "id": "9f844993",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "' Justin Bieber was born in 1994, so the NFL team that won the Super Bowl in that year was the Dallas Cowboys.'"
+       "' Justin Bieber was born in 1994, so we are looking for the Super Bowl winner from that year. The Super Bowl in 1994 was Super Bowl XXVIII, and the winner was the Dallas Cowboys.'"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -94,7 +145,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/modules/models/llms/integrations/petals_example.ipynb

@@ -4,7 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Petals LLM Example\n",
+    "# Petals\n",
+    "\n",
+    "`Petals` runs 100B+ language models at home, BitTorrent-style.\n",
+    "\n",
     "This notebook goes over how to use Langchain with [Petals](https://github.com/bigscience-workshop/petals)."
    ]
   },
@@ -22,7 +25,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "$ pip3 install petals"
+    "!pip3 install petals"
    ]
   },
   {
@@ -34,7 +37,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -48,16 +51,37 @@
    "metadata": {},
    "source": [
     "## Set the Environment API Key\n",
-    "Make sure to get your API key from Huggingface."
+    "Make sure to get [your API key](https://huggingface.co/docs/api-inference/quicktour#get-your-api-token) from Huggingface."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "from getpass import getpass\n",
+    "\n",
+    "HUGGINGFACE_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
    "metadata": {},
    "outputs": [],
    "source": [
-    "os.environ[\"HUGGINGFACE_API_KEY\"] = \"YOUR_KEY_HERE\""
+    "os.environ[\"HUGGINGFACE_API_KEY\"] = HUGGINGFACE_API_KEY"
    ]
   },
   {
@@ -72,8 +96,18 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Downloading:   1%|▏                        | 40.8M/7.19G [00:24<15:44, 7.57MB/s]"
+     ]
+    }
+   ],
    "source": [
+    "# this can take several minutes to download big files!\n",
+    "\n",
     "llm = Petals(model_name=\"bigscience/bloom-petals\")"
    ]
   },
@@ -150,7 +184,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {
@@ -159,5 +193,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/promptlayer_openai.ipynb

@@ -1,18 +1,23 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "959300d4",
    "metadata": {},
    "source": [
     "# PromptLayer OpenAI\n",
     "\n",
-    "This example showcases how to connect to [PromptLayer](https://www.promptlayer.com) to start recording your OpenAI requests."
+    "`PromptLayer` is the first platform that allows you to track, manage, and share your GPT prompt engineering. `PromptLayer` acts a middleware between your code and `OpenAI’s` python library.\n",
+    "\n",
+    "`PromptLayer` records all your `OpenAI API` requests, allowing you to search and explore request history in the `PromptLayer` dashboard.\n",
+    "\n",
+    "\n",
+    "This example showcases how to connect to [PromptLayer](https://www.promptlayer.com) to start recording your OpenAI requests.\n",
+    "\n",
+    "Another example is [here](https://python.langchain.com/en/latest/ecosystem/promptlayer.html)."
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "6a45943e",
    "metadata": {},
@@ -26,13 +31,14 @@
    "execution_count": null,
    "id": "dbe09bd8",
    "metadata": {
+    "tags": [],
     "vscode": {
      "languageId": "powershell"
     }
    },
    "outputs": [],
    "source": [
-    "pip install promptlayer"
+    "!pip install promptlayer"
    ]
   },
   {
@@ -45,9 +51,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "id": "c16da3b5",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "import os\n",
@@ -56,7 +64,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "8564ce7d",
    "metadata": {},
@@ -64,21 +71,80 @@
     "## Set the Environment API Key\n",
     "You can create a PromptLayer API Key at [www.promptlayer.com](https://www.promptlayer.com) by clicking the settings cog in the navbar.\n",
     "\n",
-    "Set it as an environment variable called `PROMPTLAYER_API_KEY`."
+    "Set it as an environment variable called `PROMPTLAYER_API_KEY`.\n",
+    "\n",
+    "You also need an OpenAI Key, called `OPENAI_API_KEY`."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "46ba25dc",
-   "metadata": {},
-   "outputs": [],
+   "execution_count": 2,
+   "id": "1df96674-a9fb-4126-bb87-541082782240",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
    "source": [
-    "os.environ[\"PROMPTLAYER_API_KEY\"] = \"********\""
+    "from getpass import getpass\n",
+    "\n",
+    "PROMPTLAYER_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "46ba25dc",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "os.environ[\"PROMPTLAYER_API_KEY\"] = PROMPTLAYER_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "9aa68c46-4d88-45ba-8a83-18fa41b4daed",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "from getpass import getpass\n",
+    "\n",
+    "OPENAI_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "6023b6fa-d9db-49d6-b713-0e19686119b0",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "bf0294de",
    "metadata": {},
@@ -89,28 +155,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "3acf0069",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' to go outside\\n\\nUnfortunately, cats cannot go outside without being supervised by a human. Going outside can be dangerous for cats, as they may come into contact with cars, other animals, or other dangers. If you want to go outside, ask your human to take you on a supervised walk or to a safe, enclosed outdoor space.'"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
    "source": [
     "llm = PromptLayerOpenAI(pl_tags=[\"langchain\"])\n",
     "llm(\"I am a cat and I want\")"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "a2d76826",
    "metadata": {},
@@ -119,7 +175,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "05e9e2fe",
    "metadata": {},
@@ -144,7 +199,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "7eb19139",
    "metadata": {},
@@ -156,7 +210,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "base",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -170,7 +224,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.8 (default, Apr 13 2021, 12:59:45) \n[Clang 10.0.0 ]"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/modules/models/llms/integrations/replicate.ipynb

@@ -5,20 +5,10 @@
    "metadata": {},
    "source": [
     "# Replicate\n",
-    "This example goes over how to use LangChain to interact with Replicate models"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from langchain.llms import Replicate\n",
-    "from langchain import PromptTemplate, LLMChain\n",
     "\n",
-    "os.environ[\"REPLICATE_API_TOKEN\"] = \"YOUR REPLICATE API TOKEN\""
+    ">[Replicate](https://replicate.com/blog/machine-learning-needs-better-tools) runs machine learning models in the cloud. We have a library of open-source models that you can run with a few lines of code. If you're building your own machine learning models, Replicate makes it easy to deploy them at scale.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `Replicate` [models](https://replicate.com/explore)"
    ]
   },
   {
@@ -35,6 +25,65 @@
     "To run this notebook, you'll need to create a [replicate](https://replicate.com) account and install the [replicate python client](https://github.com/replicate/replicate-python)."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!pip install replicate"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get a token: https://replicate.com/account\n",
+    "\n",
+    "from getpass import getpass\n",
+    "\n",
+    "REPLICATE_API_TOKEN = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.llms import Replicate\n",
+    "from langchain import PromptTemplate, LLMChain"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -58,8 +107,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm = Replicate(model=\"daanelson/flan-t5:04e422a9b85baed86a4f24981d7f9953e20c5fd82f6103b74ebc431588e1cec8\")"
@@ -339,7 +390,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {
@@ -348,5 +399,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/runhouse.ipynb

@@ -5,18 +5,43 @@
    "id": "9597802c",
    "metadata": {},
    "source": [
-    "# Self-Hosted Models via Runhouse\n",
+    "# Runhouse\n",
+    "\n",
+    "The [Runhouse](https://github.com/run-house/runhouse) allows remote compute and data across environments and users. See the [Runhouse docs](https://runhouse-docs.readthedocs-hosted.com/en/latest/).\n",
+    "\n",
     "This example goes over how to use LangChain and [Runhouse](https://github.com/run-house/runhouse) to interact with models hosted on your own GPU, or on-demand GPUs on AWS, GCP, AWS, or Lambda.\n",
     "\n",
-    "For more information, see [Runhouse](https://github.com/run-house/runhouse) or the [Runhouse docs](https://runhouse-docs.readthedocs-hosted.com/en/latest/)."
+    "**Note**: Code uses `SelfHosted` name instead of the `Runhouse`."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6fb585dd",
-   "metadata": {},
+   "id": "6066fede-2300-4173-9722-6f01f4fa34b4",
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
+   "source": [
+    "!pip install runhouse"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "6fb585dd",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "INFO | 2023-04-17 16:47:36,173 | No auth token provided, so not using RNS API to save and load configs\n"
+     ]
+    }
+   ],
    "source": [
     "from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM\n",
     "from langchain import PromptTemplate, LLMChain\n",
@@ -25,9 +50,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "id": "06d6866e",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# For an on-demand A100 with GCP, Azure, or Lambda\n",
@@ -44,9 +71,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "id": "035dea0f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -60,7 +89,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "3f3458d9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm = SelfHostedHuggingFaceLLM(model_id=\"gpt2\", hardware=gpu, model_reqs=[\"pip:./\", \"transformers\", \"torch\"])"
@@ -288,7 +319,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.15"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/modules/models/llms/integrations/sagemaker.ipynb

@@ -6,22 +6,56 @@
    "source": [
     "# SageMakerEndpoint\n",
     "\n",
-    "This notebooks goes over how to use an LLM hosted on a SageMaker endpoint."
+    "[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a system that can build, train, and deploy machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows.\n",
+    "\n",
+    "This notebooks goes over how to use an LLM hosted on a `SageMaker endpoint`."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!pip3 install langchain boto3"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
+   "source": [
+    "## Set up"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You have to set up following required parameters of the `SagemakerEndpoint` call:\n",
+    "- `endpoint_name`: The name of the endpoint from the deployed Sagemaker model.\n",
+    "    Must be unique within an AWS Region.\n",
+    "- `credentials_profile_name`: The name of the profile in the ~/.aws/credentials or ~/.aws/config files, which\n",
+    "    has either access keys or role information specified.\n",
+    "    If not specified, the default credential profile or, if on an EC2 instance,\n",
+    "    credentials from IMDS will be used.\n",
+    "    See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.docstore.document import Document"
@@ -29,8 +63,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "example_doc_1 = \"\"\"\n",
@@ -49,7 +85,9 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from typing import Dict\n",
@@ -118,7 +156,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {
@@ -127,5 +165,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/stochasticai.ipynb

@@ -5,13 +5,78 @@
    "metadata": {},
    "source": [
     "# StochasticAI\n",
-    "This example goes over how to use LangChain to interact with StochasticAI models"
+    "\n",
+    ">[Stochastic Acceleration Platform](https://docs.stochastic.ai/docs/introduction/) aims to simplify the life cycle of a Deep Learning model. From uploading and versioning the model, through training, compression and acceleration to putting it into production.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `StochasticAI` models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You have to get the API_KEY and the API_URL [here](https://app.stochastic.ai/workspace/profile/settings?tab=profile)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "from getpass import getpass\n",
+    "\n",
+    "STOCHASTICAI_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"STOCHASTICAI_API_KEY\"] = STOCHASTICAI_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "YOUR_API_URL = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import StochasticAI\n",
@@ -20,8 +85,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -33,17 +100,21 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 11,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "llm = StochasticAI(api_url=\"YOUR_API_URL\")"
+    "llm = StochasticAI(api_url=YOUR_API_URL)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 12,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
@@ -51,27 +122,54 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
+   "execution_count": 13,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"\\n\\nStep 1: In 1999, the St. Louis Rams won the Super Bowl.\\n\\nStep 2: In 1999, Beiber was born.\\n\\nStep 3: The Rams were in Los Angeles at the time.\\n\\nStep 4: So they didn't play in the Super Bowl that year.\\n\""
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
     "\n",
     "llm_chain.run(question)"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -79,5 +177,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/modules/models/llms/integrations/writer.ipynb

@@ -5,13 +5,54 @@
    "metadata": {},
    "source": [
     "# Writer\n",
-    "This example goes over how to use LangChain to interact with Writer models"
+    "\n",
+    "[Writer](https://writer.com/) is a platform to generate different language content.\n",
+    "\n",
+    "This example goes over how to use LangChain to interact with `Writer` [models](https://dev.writer.com/docs/models).\n",
+    "\n",
+    "You have to get the WRITER_API_KEY [here](https://dev.writer.com/docs)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      " ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "from getpass import getpass\n",
+    "\n",
+    "WRITER_API_KEY = getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"WRITER_API_KEY\"] = WRITER_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.llms import Writer\n",
@@ -20,8 +61,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 7,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "template = \"\"\"Question: {question}\n",
@@ -33,17 +76,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 14,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
+    "# If you get an error, probably, you need to set up the \"base_url\" parameter that can be taken from the error log.\n",
+    "\n",
     "llm = Writer()"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
+   "execution_count": 15,
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
@@ -52,26 +101,42 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
     "\n",
     "llm_chain.run(question)"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.9.12 ('palm')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
    "name": "python",
-   "version": "3.9.12"
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
   },
-  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
@@ -79,5 +144,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/reference.rst

@@ -8,7 +8,7 @@ Full documentation on all methods, classes, and APIs in LangChain.
    :maxdepth: 1
 
    ./reference/prompts.rst
-   LLMs<./refeence/modules/llms>
+   LLMs<./reference/modules/llms>
    ./reference/utils.rst
    Chains<./reference/modules/chains>
    Agents<./reference/modules/agents>

docs/use_cases/agents/characters.ipynb

@@ -25,7 +25,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 41,
+   "execution_count": 2,
    "id": "8851c370-b395-4b80-a79d-486a38ffc244",
    "metadata": {
     "tags": []
@@ -51,7 +51,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 42,
+   "execution_count": 3,
    "id": "81824e76",
    "metadata": {
     "tags": []
@@ -86,7 +86,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 43,
+   "execution_count": 4,
    "id": "043e5203-6a41-431c-9efa-3e1743d7d25a",
    "metadata": {
     "tags": []
@@ -421,7 +421,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 44,
+   "execution_count": 5,
    "id": "ee9c1a1d-c311-4f1c-8131-75fccd9025b1",
    "metadata": {
     "tags": []
@@ -431,7 +431,7 @@
     "import math\n",
     "import faiss\n",
     "\n",
-    "def normalize_score_fn(score: float) -> float:\n",
+    "def relevance_score_fn(score: float) -> float:\n",
     "    \"\"\"Return a similarity score on a scale [0, 1].\"\"\"\n",
     "    # This will differ depending on a few things:\n",
     "    # - the distance / similarity metric used by the VectorStore\n",
@@ -448,13 +448,13 @@
     "    # Initialize the vectorstore as empty\n",
     "    embedding_size = 1536\n",
     "    index = faiss.IndexFlatL2(embedding_size)\n",
-    "    vectorstore = FAISS(embeddings_model.embed_query, index, InMemoryDocstore({}), {}, normalize_score_fn=normalize_score_fn)\n",
+    "    vectorstore = FAISS(embeddings_model.embed_query, index, InMemoryDocstore({}), {}, relevance_score_fn=relevance_score_fn)\n",
     "    return TimeWeightedVectorStoreRetriever(vectorstore=vectorstore, other_score_keys=[\"importance\"], k=15)    "
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 45,
+   "execution_count": 6,
    "id": "7884f9dd-c597-4c27-8c77-1402c71bc2f8",
    "metadata": {
     "tags": []
@@ -476,7 +476,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 46,
+   "execution_count": 7,
    "id": "c524d529",
    "metadata": {
     "tags": []
@@ -488,19 +488,19 @@
      "text": [
       "Name: Tommie (age: 25)\n",
       "Innate traits: anxious, likes design\n",
-      "It is not possible to provide a summary without any statements or information about Tommie's characteristics.\n"
+      "Unfortunately, there are no statements provided to summarize Tommie's core characteristics.\n"
      ]
     }
    ],
    "source": [
-    "# We can see a current \"Summary\" of a character based on their own perception of self.\n",
-    "# It isn't very complete right now since the character doesn't have any memories.\n",
+    "# The current \"Summary\" of a character can't be made because the agent hasn't made\n",
+    "# any observations yet.\n",
     "print(tommie.get_summary())"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 47,
+   "execution_count": 8,
    "id": "4be60979-d56e-4abf-a636-b34ffa8b7fba",
    "metadata": {
     "tags": []
@@ -523,7 +523,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 48,
+   "execution_count": 9,
    "id": "6992b48b-697f-4973-9560-142ef85357d7",
    "metadata": {
     "tags": []
@@ -535,12 +535,13 @@
      "text": [
       "Name: Tommie (age: 25)\n",
       "Innate traits: anxious, likes design\n",
-      "Tommie is observant, has a sentimental side, gets hungry, tries to rest when possible, gets tired easily, notices details about his surroundings, and is affected by noise.\n"
+      "Tommie is observant, nostalgic, tired, and hungry.\n"
      ]
     }
    ],
    "source": [
-    "# Now that Tommie has 'memories', their self-summary is more descriptive\n",
+    "# Now that Tommie has 'memories', their self-summary is more descriptive, though still rudimentary.\n",
+    "# We will see how this summary updates after more observations to create a more rich description.\n",
     "print(tommie.get_summary(force_refresh=True))"
    ]
   },
@@ -558,7 +559,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 49,
+   "execution_count": 10,
    "id": "eaf125d8-f54c-4c5f-b6af-32789b1f7d3a",
    "metadata": {
     "tags": []
@@ -573,7 +574,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 50,
+   "execution_count": 11,
    "id": "54024d41-6e83-4914-91e5-73140e2dd9c8",
    "metadata": {
     "tags": []
@@ -582,10 +583,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"I really enjoy design, especially graphic design. It\\'s something I\\'ve been interested in for a while now.\"'"
+       "'Tommie said \"I really enjoy design, especially interior design. I find it calming and rewarding to create a space that is both functional and aesthetically pleasing. Unfortunately, I haven\\'t been able to find a job in that field yet.\"'"
       ]
      },
-     "execution_count": 50,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -596,7 +597,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 51,
+   "execution_count": 12,
    "id": "71e2e8cc-921e-4816-82f1-66962b2c1055",
    "metadata": {
     "tags": []
@@ -605,10 +606,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"Well, I\\'m actually on the hunt for a job right now. So, I\\'m hoping to make some progress in that area today. How about you? Any exciting plans?\"'"
+       "'Tommie said \"Well, I\\'m actually on the hunt for a job right now. I\\'m hoping to find something in the design field, but I\\'m open to exploring other options as well. How about you, what are your plans for the day?\"'"
       ]
      },
-     "execution_count": 51,
+     "execution_count": 12,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -619,7 +620,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 52,
+   "execution_count": 13,
    "id": "a2521ffc-7050-4ac3-9a18-4cccfc798c31",
    "metadata": {
     "tags": []
@@ -628,10 +629,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"Honestly, I\\'m feeling pretty anxious about finding a job. It\\'s been a tough process so far. But I\\'m trying to stay positive and keep pushing forward. How about you, what are you worried about today?\"'"
+       "'Tommie said \"Honestly, I\\'m feeling pretty anxious about finding a job. It\\'s been a bit of a struggle and I\\'m not sure what my next step should be. But I\\'m trying to stay positive and keep pushing forward.\"'"
       ]
      },
-     "execution_count": 52,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -650,7 +651,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 53,
+   "execution_count": 14,
    "id": "154dee3d-bfe0-4828-b963-ed7e885799b3",
    "metadata": {
     "tags": []
@@ -692,7 +693,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 54,
+   "execution_count": 15,
    "id": "238be49c-edb3-4e26-a2b6-98777ba8de86",
    "metadata": {
     "tags": []
@@ -702,40 +703,40 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "\u001b[32mTommie wakes up to the sound of a noisy construction site outside his window.\u001b[0m Tommie Tommie may feel agitated or annoyed by the noise, but it depends on his tolerance level and mood at the moment.\n",
-      "\u001b[32mTommie gets out of bed and heads to the kitchen to make himself some coffee.\u001b[0m Tommie Tommie feels a sense of relief and comfort as he prepares his coffee.\n",
-      "\u001b[32mTommie realizes he forgot to buy coffee filters and starts rummaging through his moving boxes to find some.\u001b[0m Tommie Tommie may feel frustrated or annoyed with himself for forgetting to buy coffee filters.\n",
-      "\u001b[32mTommie finally finds the filters and makes himself a cup of coffee.\u001b[0m Tommie Tommie may feel relieved and satisfied as he sips his coffee.\n",
-      "\u001b[32mThe coffee tastes bitter, and Tommie regrets not buying a better brand.\u001b[0m Tommie Tommie may feel disappointed or regretful about the bitter coffee, but he may still finish it or consider buying a better brand next time.\n",
-      "\u001b[32mTommie checks his email and sees that he has no job offers yet.\u001b[0m Tommie Tommie may feel disappointed or discouraged, but he may also try to stay positive and continue his job search.\n",
-      "\u001b[32mTommie spends some time updating his resume and cover letter.\u001b[0m Tommie Tommie may feel productive or accomplished for updating his resume and cover letter.\n",
-      "\u001b[32mTommie heads out to explore the city and look for job openings.\u001b[0m Tommie Tommie may feel excited or optimistic about exploring the city and finding job openings.\n",
-      "\u001b[32mTommie sees a sign for a job fair and decides to attend.\u001b[0m Tommie Tommie may feel hopeful or motivated about attending the job fair.\n",
-      "\u001b[32mThe line to get in is long, and Tommie has to wait for an hour.\u001b[0m Tommie Tommie may feel impatient or frustrated with having to wait in line for an hour.\n",
-      "\u001b[32mTommie meets several potential employers at the job fair but doesn't receive any offers.\u001b[0m Tommie Tommie may feel disappointed or discouraged that he did not receive any job offers at the fair.\n",
-      "\u001b[32mTommie leaves the job fair feeling disappointed.\u001b[0m Tommie Tommie may feel discouraged or frustrated about not receiving any job offers at the fair.\n",
-      "\u001b[32mTommie stops by a local diner to grab some lunch.\u001b[0m Tommie Tommie may feel hungry and relieved to have found a place to eat lunch.\n",
-      "\u001b[32mThe service is slow, and Tommie has to wait for 30 minutes to get his food.\u001b[0m Tommie Tommie may feel frustrated or annoyed with the slow service, but he may try to be patient or find something else to do while waiting.\n",
-      "\u001b[32mTommie overhears a conversation at the next table about a job opening.\u001b[0m Tommie said \"Excuse me, I couldn't help but overhear about the job opening. Can you tell me more about it?\"\n",
-      "\u001b[32mTommie asks the diners about the job opening and gets some information about the company.\u001b[0m Tommie said \"Thank you for the information. Do you happen to know how I can apply for the job?\"\n",
-      "\u001b[32mTommie decides to apply for the job and sends his resume and cover letter.\u001b[0m Tommie said \"Thank you for the information. Do you happen to know how I can apply for the job?\"\n",
-      "\u001b[32mTommie continues his search for job openings and drops off his resume at several local businesses.\u001b[0m Tommie Tommie may feel hopeful or determined to continue his search for job openings.\n",
-      "\u001b[32mTommie takes a break from his job search to go for a walk in a nearby park.\u001b[0m Tommie Tommie may feel relaxed or refreshed by taking a break and going for a walk in the park.\n",
-      "\u001b[32mA dog approaches and licks Tommie's feet, and he pets it for a few minutes.\u001b[0m Tommie Tommie may feel surprised or amused by the dog's behavior, but he may also enjoy the brief interaction and continue on his walk.\n",
+      "\u001b[32mTommie wakes up to the sound of a noisy construction site outside his window.\u001b[0m Tommie Tommie groans and covers their head with a pillow, trying to block out the noise.\n",
+      "\u001b[32mTommie gets out of bed and heads to the kitchen to make himself some coffee.\u001b[0m Tommie Tommie starts making coffee, feeling grateful for the little bit of energy it will give him.\n",
+      "\u001b[32mTommie realizes he forgot to buy coffee filters and starts rummaging through his moving boxes to find some.\u001b[0m Tommie Tommie sighs in frustration and continues to search for the coffee filters.\n",
+      "\u001b[32mTommie finally finds the filters and makes himself a cup of coffee.\u001b[0m Tommie Tommie takes a sip of the coffee and feels a little more awake.\n",
+      "\u001b[32mThe coffee tastes bitter, and Tommie regrets not buying a better brand.\u001b[0m Tommie Tommie grimaces at the taste of the coffee and decides to make a mental note to buy a better brand next time.\n",
+      "\u001b[32mTommie checks his email and sees that he has no job offers yet.\u001b[0m Tommie Tommie feels disappointed and discouraged, but tries to stay positive and continue the job search.\n",
+      "\u001b[32mTommie spends some time updating his resume and cover letter.\u001b[0m Tommie Tommie feels determined to keep working on his job search.\n",
+      "\u001b[32mTommie heads out to explore the city and look for job openings.\u001b[0m Tommie Tommie feels hopeful but also anxious as he heads out to explore the city and look for job openings.\n",
+      "\u001b[32mTommie sees a sign for a job fair and decides to attend.\u001b[0m Tommie said \"That job fair could be a great opportunity to meet potential employers.\"\n",
+      "\u001b[32mThe line to get in is long, and Tommie has to wait for an hour.\u001b[0m Tommie Tommie feels frustrated and restless while waiting in line.\n",
+      "\u001b[32mTommie meets several potential employers at the job fair but doesn't receive any offers.\u001b[0m Tommie Tommie feels disappointed but remains determined to keep searching for job openings.\n",
+      "\u001b[32mTommie leaves the job fair feeling disappointed.\u001b[0m Tommie Tommie feels discouraged but remains determined to keep searching for job openings.\n",
+      "\u001b[32mTommie stops by a local diner to grab some lunch.\u001b[0m Tommie Tommie feels relieved to take a break from job searching and enjoy a meal.\n",
+      "\u001b[32mThe service is slow, and Tommie has to wait for 30 minutes to get his food.\u001b[0m Tommie Tommie feels impatient and frustrated while waiting for his food.\n",
+      "\u001b[32mTommie overhears a conversation at the next table about a job opening.\u001b[0m Tommie said \"Excuse me, I couldn't help but overhear about the job opening. Could you tell me more about it?\"\n",
+      "\u001b[32mTommie asks the diners about the job opening and gets some information about the company.\u001b[0m Tommie said \"Could you tell me more about it?\"\n",
+      "\u001b[32mTommie decides to apply for the job and sends his resume and cover letter.\u001b[0m Tommie said \"Thank you for the information, I'll definitely apply for the job and keep my fingers crossed.\"\n",
+      "\u001b[32mTommie continues his search for job openings and drops off his resume at several local businesses.\u001b[0m Tommie Tommie feels hopeful but also anxious as he continues his search for job openings and drops off his resume at several local businesses.\n",
+      "\u001b[32mTommie takes a break from his job search to go for a walk in a nearby park.\u001b[0m Tommie Tommie takes a deep breath and enjoys the fresh air in the park.\n",
+      "\u001b[32mA dog approaches and licks Tommie's feet, and he pets it for a few minutes.\u001b[0m Tommie Tommie smiles and enjoys the momentary distraction from his job search.\n",
       "****************************************\n",
       "\u001b[34mAfter 20 observations, Tommie's summary is:\n",
       "Name: Tommie (age: 25)\n",
       "Innate traits: anxious, likes design\n",
-      "Tommie is optimistic and determined in his job search, but also experiences disappointment and discouragement. He enjoys design, has a sense of nostalgia for his past, and finds comfort in simple pleasures like making coffee and going for walks.\u001b[0m\n",
+      "Tommie is a determined individual who is actively searching for job opportunities. He feels both hopeful and anxious about his search and remains positive despite facing disappointments. He takes breaks to rest and enjoy the little things in life, like going for a walk or grabbing a meal. Tommie is also open to asking for help and seeking information about potential job openings. He is grateful for the little things that give him energy and tries to stay positive even when faced with discouragement. Overall, Tommie's core characteristics include determination, positivity, and a willingness to seek help and take breaks when needed.\u001b[0m\n",
       "****************************************\n",
-      "\u001b[32mTommie sees a group of people playing frisbee and decides to join in.\u001b[0m Tommie said \"Mind if I join in?\"\n",
-      "\u001b[32mTommie has fun playing frisbee but gets hit in the face with the frisbee and hurts his nose.\u001b[0m Tommie Tommie may feel embarrassed or in pain after getting hit in the face with the frisbee.\n",
-      "\u001b[32mTommie goes back to his apartment to rest for a bit.\u001b[0m Tommie Tommie may feel tired or in need of rest after a long day of job searching and activities.\n",
-      "\u001b[32mA raccoon tore open the trash bag outside his apartment, and the garbage is all over the floor.\u001b[0m Tommie Tommie may feel frustrated or annoyed with the mess and may need to clean it up.\n",
-      "\u001b[32mTommie starts to feel frustrated with his job search.\u001b[0m Tommie Tommie may feel discouraged or overwhelmed with his job search and may need to take a break or try a different approach.\n",
-      "\u001b[32mTommie calls his best friend to vent about his struggles.\u001b[0m Tommie said \"Hey, I just needed to vent about my job search. It's been a tough day.\"\n",
-      "\u001b[32mTommie's friend offers some words of encouragement and tells him to keep trying.\u001b[0m Tommie said \"Thanks for the encouragement, it means a lot to me.\"\n",
-      "\u001b[32mTommie feels slightly better after talking to his friend.\u001b[0m Tommie said \"Thanks for listening, I really needed to hear that.\"\n"
+      "\u001b[32mTommie sees a group of people playing frisbee and decides to join in.\u001b[0m Tommie said \"Mind if I join in on the game?\"\n",
+      "\u001b[32mTommie has fun playing frisbee but gets hit in the face with the frisbee and hurts his nose.\u001b[0m Tommie Tommie winces in pain and puts his hand to his nose to check for any bleeding.\n",
+      "\u001b[32mTommie goes back to his apartment to rest for a bit.\u001b[0m Tommie Tommie takes a deep breath and sits down to rest for a bit.\n",
+      "\u001b[32mA raccoon tore open the trash bag outside his apartment, and the garbage is all over the floor.\u001b[0m Tommie Tommie sighs and grabs a broom to clean up the mess.\n",
+      "\u001b[32mTommie starts to feel frustrated with his job search.\u001b[0m Tommie Tommie takes a deep breath and reminds himself to stay positive and keep searching for job opportunities.\n",
+      "\u001b[32mTommie calls his best friend to vent about his struggles.\u001b[0m Tommie said \"Hey, can I vent to you for a bit about my job search struggles?\"\n",
+      "\u001b[32mTommie's friend offers some words of encouragement and tells him to keep trying.\u001b[0m Tommie said \"Thank you for the encouragement, it means a lot. I'll keep trying.\"\n",
+      "\u001b[32mTommie feels slightly better after talking to his friend.\u001b[0m Tommie said \"Thank you for your support, it really means a lot to me.\"\n"
      ]
     }
    ],
@@ -760,7 +761,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 55,
+   "execution_count": 16,
    "id": "6336ab5d-3074-4831-951f-c9e2cba5dfb5",
    "metadata": {
     "tags": []
@@ -769,10 +770,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"It\\'s been a bit of a rollercoaster, to be honest. I went to a job fair but didn\\'t get any offers, but then I heard about a job opening at a diner and applied for it. I also went for a walk in the park and played frisbee, but got hit in the face with it. Overall, it\\'s been a mix of good and bad experiences.\"'"
+       "'Tommie said \"It\\'s been a bit of a rollercoaster, to be honest. I went to a job fair and met some potential employers, but didn\\'t get any offers. But then I overheard about a job opening at a diner and applied for it. I also took a break to go for a walk in the park and played frisbee with some people, which was a nice distraction. Overall, it\\'s been a bit frustrating, but I\\'m trying to stay positive and keep searching for job opportunities.\"'"
       ]
      },
-     "execution_count": 55,
+     "execution_count": 16,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -783,7 +784,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 56,
+   "execution_count": 17,
    "id": "809ac906-69b7-4326-99ec-af638d32bb20",
    "metadata": {
     "tags": []
@@ -792,10 +793,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"I actually really enjoy coffee, although I recently bought a brand that turned out to be too bitter for my taste. And I always seem to forget to buy coffee filters when I need them. But overall, I find making and drinking coffee to be a comforting ritual.\"'"
+       "'Tommie would say: \"I rely on coffee to give me a little boost, but I regret not buying a better brand lately. The taste has been pretty bitter. But overall, it\\'s not a huge factor in my life.\" '"
       ]
      },
-     "execution_count": 56,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -806,7 +807,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 57,
+   "execution_count": 18,
    "id": "f733a431-19ea-421a-9101-ae2593a8c626",
    "metadata": {
     "tags": []
@@ -815,10 +816,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"Oh, I had a dog when I was a kid named Bruno. He was a golden retriever and we were really close. I have a lot of happy memories with him.\"'"
+       "'Tommie said \"Oh, I actually don\\'t have a childhood dog, but I do love animals. Have you had any pets?\"'"
       ]
      },
-     "execution_count": 57,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -839,7 +840,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 58,
+   "execution_count": 19,
    "id": "ec8bbe18-a021-419c-bf1f-23d34732cd99",
    "metadata": {
     "tags": []
@@ -861,7 +862,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 59,
+   "execution_count": 20,
    "id": "1e2745f5-e0da-4abd-98b4-830802ce6698",
    "metadata": {
     "tags": []
@@ -884,7 +885,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 60,
+   "execution_count": 21,
    "id": "de4726e3-4bb1-47da-8fd9-f317a036fe0f",
    "metadata": {
     "tags": []
@@ -896,7 +897,7 @@
      "text": [
       "Name: Eve (age: 34)\n",
       "Innate traits: curious, helpful\n",
-      "Eve is helpful, active, attentive, routine-oriented, and aware of her colleagues.\n"
+      "Eve is helpful, active, eats breakfast, is attentive to her surroundings, and works with colleagues.\n"
      ]
     }
    ],
@@ -917,7 +918,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 61,
+   "execution_count": 22,
    "id": "6cda916d-800c-47bc-a7f9-6a2f19187472",
    "metadata": {
     "tags": []
@@ -926,10 +927,10 @@
     {
      "data": {
       "text/plain": [
-       "'Eve said \"I\\'m feeling curious about what the day has in store. How about you?\"'"
+       "'Eve said \"I\\'m feeling curious about what\\'s on the agenda for today. Anything special we should be aware of?\"'"
       ]
      },
-     "execution_count": 61,
+     "execution_count": 22,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -940,7 +941,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 62,
+   "execution_count": 23,
    "id": "448ae644-0a66-4eb2-a03a-319f36948b37",
    "metadata": {
     "tags": []
@@ -949,10 +950,10 @@
     {
      "data": {
       "text/plain": [
-       "'Eve said \"I overheard someone say that Tommie is hard to work with, but I don\\'t know much else. Have you worked with them before?\"'"
+       "'Eve said \"I overheard someone say Tommie is hard to work with. Is there something I can help with?\"'"
       ]
      },
-     "execution_count": 62,
+     "execution_count": 23,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -963,7 +964,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 63,
+   "execution_count": 24,
    "id": "493fc5b8-8730-4ef8-9820-0f1769ce1691",
    "metadata": {
     "tags": []
@@ -972,10 +973,10 @@
     {
      "data": {
       "text/plain": [
-       "'Eve said \"That\\'s interesting. What kind of job is Tommie looking for?\"'"
+       "'Eve said \"Oh, I didn\\'t realize Tommie was looking for a new job. Is there anything I can do to help? Maybe I could introduce him to some people in my network or help him with his resume.\"'"
       ]
      },
-     "execution_count": 63,
+     "execution_count": 24,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -986,7 +987,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 64,
+   "execution_count": 25,
    "id": "4b46452a-6c54-4db2-9d87-18597f70fec8",
    "metadata": {
     "tags": []
@@ -995,16 +996,16 @@
     {
      "data": {
       "text/plain": [
-       "'Eve said \"Sure, I\\'ll do my best to make him comfortable and ask about his experiences and goals. Thanks for the heads up.\"'"
+       "'Eve said \"Sure, I can definitely help keep the conversation going and ask him plenty of questions. Is there anything specific you would like me to ask him about his skills or experience? I want to make sure the conversation is productive.\"'"
       ]
      },
-     "execution_count": 64,
+     "execution_count": 25,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "interview_agent(eve, \"You'll have to ask him. He may be a bit shy, so I'd appreciate it if you keep the conversation go and ask as many questions as possible.\")"
+    "interview_agent(eve, \"You'll have to ask him. He may be a bit anxious, so I'd appreciate it if you keep the conversation going and ask as many questions as possible.\")"
    ]
   },
   {
@@ -1019,14 +1020,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 65,
+   "execution_count": 26,
    "id": "042ea271-4bf1-4247-9082-239a6fea43b8",
    "metadata": {
     "tags": []
    },
    "outputs": [],
    "source": [
-    "def run_conversation(agents: List[GenerativeAgent], initial_observation: str, max_turns:int = 25) -> None:\n",
+    "def run_conversation(agents: List[GenerativeAgent], initial_observation: str) -> None:\n",
     "    \"\"\"Runs a conversation between agents.\"\"\"\n",
     "    _, observation = agents[1].generate_reaction(initial_observation)\n",
     "    print(observation)\n",
@@ -1046,7 +1047,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 66,
+   "execution_count": 27,
    "id": "d5462b14-218e-4d85-b035-df57ea8e0f80",
    "metadata": {
     "tags": []
@@ -1056,15 +1057,25 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Eve said \"Of course, Tommie. I'd be happy to share my story with you and offer any advice I can.\"\n",
-      "There is no mention of Eve in the given context, so it is unclear what the conversation is about.\n",
-      "Eve said \"That's a great place to start, Tommie. I actually majored in marketing in college and landed my first job in the industry as a marketing assistant. From there, I worked my way up through various roles and eventually ended up in my current position. What about you? What led you to pursue a career in this field?\"\n"
+      "Eve said \"Of course, Tommie! I'm happy to help in any way I can. What specifically would you like advice on?\"\n",
+      "Tommie said \"Thank you, Eve. I was actually wondering if you knew of any job openings in the design field? That's my area of interest and expertise.\"\n",
+      "Eve said \"Sure, Tommie! I can definitely keep an eye out for any job openings in the design field and let you know if I come across anything. Have you updated your resume and portfolio recently? I could take a look and provide some feedback if you'd like.\"\n",
+      "Tommie said \"Thank you so much, Eve! That would be incredibly helpful. And yes, I have updated my resume and portfolio recently, but I would love any feedback you can provide. I really appreciate your support and willingness to help.\"\n",
+      "Eve said \"Great, Tommie! I'll take a look at your resume and portfolio and provide you with some feedback. Do you have any specific companies or types of design jobs that you're interested in? I might be able to provide more targeted advice and resources based on that information.\"\n",
+      "Tommie said \"Thank you, Eve! I'm primarily interested in graphic design and UI/UX design positions, but I'm open to other opportunities as well. I appreciate any advice or resources you can provide. Thanks again for your help.\"\n",
+      "Eve said \"Great, Tommie! Those are both really interesting fields. I'll definitely keep an eye out for any job openings or resources related to graphic design and UI/UX design. In the meantime, I can take a look at your resume and portfolio and provide you with some feedback. Would you like me to email you my feedback or would you prefer to discuss it in person?\"\n",
+      "Tommie said \"Thank you, Eve! I would really appreciate discussing your feedback in person. When would be a good time for us to meet?\"\n",
+      "Eve said \"Sure, Tommie! How about we schedule a time to meet next week? I have some availability on Tuesday or Thursday afternoon. What works best for you?\"\n",
+      "Tommie said \"Thank you, Eve! Tuesday afternoon works better for me. How about 2 PM?\"\n",
+      "Eve said \"Great, Tommie! Tuesday at 2 PM works for me as well. Let's plan to meet then. In the meantime, if you have any questions or if there's anything else I can help with, please don't hesitate to reach out. Good luck with your job search!\"\n",
+      "Tommie said \"Thank you so much, Eve! I really appreciate your support and willingness to help. I'll see you next Tuesday at 2 PM. Have a great day!\"\n",
+      "Eve said \"You're welcome, Tommie! Looking forward to meeting with you on Tuesday. Have a great day and good luck with your job search!\"\n"
      ]
     }
    ],
    "source": [
     "agents = [tommie, eve]\n",
-    "run_conversation(agents, \"Tommie said: Hi, Eve. Thanks for agreeing to share your story with me and give me advice. Maybe we can start with how you got into the industry.\")"
+    "run_conversation(agents, \"Tommie said: Hi, Eve. Thanks for agreeing to share your story with me and give me advice. I have a bunch of questions.\")"
    ]
   },
   {
@@ -1081,7 +1092,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 67,
+   "execution_count": 28,
    "id": "c4d252f3-fcc1-474c-846e-a7605a6b4ce7",
    "metadata": {
     "tags": []
@@ -1093,18 +1104,19 @@
      "text": [
       "Name: Tommie (age: 25)\n",
       "Innate traits: anxious, likes design\n",
-      "Tommie is a hopeful and determined individual who is actively searching for job openings. He is also sentimental and remembers his childhood dog, Bruno. Tommie can become frustrated and discouraged at times during his job search but is productive and accomplished when updating his resume and cover letter. He takes breaks when needed and tries to stay positive, even when he receives no job offers. Tommie enjoys design, especially graphic design, and finds comfort in making coffee.\n"
+      "Tommie is a determined person who is actively searching for job opportunities. He feels both hopeful and anxious about his job search, and remains persistent despite facing disappointment and discouragement. He seeks support from friends and takes breaks to recharge. He tries to stay positive and continues to work on improving his resume and cover letter. He also values the importance of self-care and takes breaks to rest and enjoy nature.\n"
      ]
     }
    ],
    "source": [
-    "# We can see how the agents have \n",
+    "# We can see a current \"Summary\" of a character based on their own perception of self\n",
+    "# has changed\n",
     "print(tommie.get_summary(force_refresh=True))"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 68,
+   "execution_count": 29,
    "id": "c04db9a4",
    "metadata": {
     "tags": []
@@ -1116,18 +1128,17 @@
      "text": [
       "Name: Eve (age: 34)\n",
       "Innate traits: curious, helpful\n",
-      "Eve is a helpful and active individual who enjoys playing tennis and eating porridge. She is attentive and curious, often observing and engaging in conversations with colleagues. She is willing to share her experiences and offer advice, and is considerate of others' comfort levels in social situations.\n"
+      "Eve is a helpful and proactive coworker who values relationships and communication. She is attentive to her colleagues' needs and willing to offer support and assistance. She is also curious and interested in learning more about her work and the people around her. Overall, Eve demonstrates a strong sense of empathy and collaboration in her interactions with others.\n"
      ]
     }
    ],
    "source": [
-    "# We can see a current \"Summary\" of a character based on their own perception of self\n",
     "print(eve.get_summary(force_refresh=True))"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 69,
+   "execution_count": 30,
    "id": "71762558-8fb6-44d7-8483-f5b47fb2a862",
    "metadata": {
     "tags": []
@@ -1136,10 +1147,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"I\\'m sorry, I don\\'t know who Eve is. I haven\\'t had a conversation with her.\"'"
+       "'Tommie said \"It was really helpful! Eve offered to provide feedback on my resume and portfolio, and she\\'s going to keep an eye out for job openings in the design field. We\\'re planning to meet next Tuesday to discuss her feedback. Thanks for asking!\"'"
       ]
      },
-     "execution_count": 69,
+     "execution_count": 30,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1150,7 +1161,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 70,
+   "execution_count": 31,
    "id": "085af3d8-ac21-41ea-8f8b-055c56976a67",
    "metadata": {
     "tags": []
@@ -1159,10 +1170,10 @@
     {
      "data": {
       "text/plain": [
-       "'Eve said \"It was great! Tommie seems really interested in learning more about the industry and he\\'s asking all the right questions. I\\'m happy to help him out.\"'"
+       "'Eve said \"It was really productive! Tommie is interested in graphic design and UI/UX design positions, so I\\'m going to keep an eye out for any job openings or resources related to those fields. I\\'m also going to provide him with some feedback on his resume and portfolio. We\\'re scheduled to meet next Tuesday at 2 PM to discuss everything in person. Is there anything else you would like me to ask him or anything else I can do to help?\".'"
       ]
      },
-     "execution_count": 70,
+     "execution_count": 31,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1173,7 +1184,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 71,
+   "execution_count": 32,
    "id": "5b439f3c-7849-4432-a697-2bcc85b89dae",
    "metadata": {
     "tags": []
@@ -1182,10 +1193,10 @@
     {
      "data": {
       "text/plain": [
-       "'Eve said \"I think I covered most of what I wanted to share with Tommie, but if there\\'s anything else he needs advice on, I\\'m happy to help. Did you have any specific questions in mind, Person A?\"'"
+       "'Eve said \"I feel like I covered everything I wanted to with Tommie, but thank you for asking! If there\\'s anything else that comes up or if you have any further questions, please let me know.\"'"
       ]
      },
-     "execution_count": 71,
+     "execution_count": 32,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1196,7 +1207,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 72,
+   "execution_count": 33,
    "id": "526e8863-8b32-4216-8e61-2dfe82e3fb47",
    "metadata": {
     "tags": []
@@ -1205,10 +1216,10 @@
     {
      "data": {
       "text/plain": [
-       "'Tommie said \"I actually forgot to buy coffee filters, so I had to improvise with a paper towel. It wasn\\'t the best cup of coffee, but it did the job.\"'"
+       "'Tommie said \"Oh, I actually forgot to buy coffee filters yesterday, so I couldn\\'t make coffee this morning. But I\\'m planning to grab some later today. Thanks for asking!\"'"
       ]
      },
-     "execution_count": 72,
+     "execution_count": 33,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1242,7 +1253,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,

docs/use_cases/personal_assistants.md

@@ -23,3 +23,4 @@ Specific examples of this include:
 - [Baby AGI with Tools](agents/baby_agi_with_agent.ipynb): building off the above notebook, this example substitutes in an agent with tools as the execution tools, allowing it to actually take actions.
 - [CAMEL](agents/camel_role_playing.ipynb): an implementation of the CAMEL (Communicative Agents for “Mind” Exploration of Large Scale Language Model Society) paper, where two agents communicate with eachother.
 - [AI Plugins](agents/custom_agent_with_plugin_retrieval.ipynb): an implementation of an agent that is designed to be able to use all AI Plugins.
+- [Generative Agents](agents/characters.ipynb): This notebook implements a generative agent based on the paper [Generative Agents: Interactive Simulacra of Human Behavior](https://arxiv.org/abs/2304.03442) by Park, et. al.

docs/use_cases/question_answering.md

@@ -2,8 +2,7 @@
 
 > [Conceptual Guide](https://docs.langchain.com/docs/use-cases/qa-docs)
 
-
-Question answering in this context refers to question answering over your document data. 
+Question answering in this context refers to question answering over your document data.
 For question answering over other types of data, please see other sources documentation like [SQL database Question Answering](./tabular.md) or [Interacting with APIs](./apis.md).
 
 For question answering over many documents, you almost always want to create an index over the data.
@@ -12,29 +11,37 @@ This can be used to smartly access the most relevant documents for a given quest
 See [this notebook](../modules/indexes/getting_started.ipynb) for a more detailed introduction to this, but for a super quick start the steps involved are:
 
 **Load Your Documents**
+
 ```python
 from langchain.document_loaders import TextLoader
 loader = TextLoader('../state_of_the_union.txt')
 ```
+
 See [here](../modules/indexes/document_loaders.rst) for more information on how to get started with document loading.
 
 **Create Your Index**
+
 ```python
 from langchain.indexes import VectorstoreIndexCreator
 index = VectorstoreIndexCreator().from_loaders([loader])
 ```
+
 The best and most popular index by far at the moment is the VectorStore index.
 
 **Query Your Index**
+
 ```python
 query = "What did the president say about Ketanji Brown Jackson"
 index.query(query)
 ```
+
 Alternatively, use `query_with_sources` to also get back the sources involved
+
 ```python
 query = "What did the president say about Ketanji Brown Jackson"
 index.query_with_sources(query)
 ```
+
 Again, these high level interfaces obfuscate a lot of what is going on under the hood, so please see [this notebook](../modules/indexes/getting_started.ipynb) for a lower level walkthrough.
 
 ## Document Question Answering
@@ -51,6 +58,7 @@ chain.run(input_documents=docs, question=query)
 ```
 
 The following resources exist:
+
 - [Question Answering Notebook](../modules/chains/index_examples/question_answering.ipynb): A notebook walking through how to accomplish this task.
 - [VectorDB Question Answering Notebook](../modules/chains/index_examples/vector_db_qa.ipynb): A notebook walking through how to do question answering over a vector database. This can often be useful for when you have a LOT of documents, and you don't want to pass them all to the LLM, but rather first want to do some semantic search over embeddings.
 
@@ -67,11 +75,19 @@ chain({"input_documents": docs, "question": query}, return_only_outputs=True)
 ```
 
 The following resources exist:
+
 - [QA With Sources Notebook](../modules/chains/index_examples/qa_with_sources.ipynb): A notebook walking through how to accomplish this task.
 - [VectorDB QA With Sources Notebook](../modules/chains/index_examples/vector_db_qa_with_sources.ipynb): A notebook walking through how to do question answering with sources over a vector database. This can often be useful for when you have a LOT of documents, and you don't want to pass them all to the LLM, but rather first want to do some semantic search over embeddings.
 
 ## Additional Related Resources
 
 Additional related resources include:
+
 - [Utilities for working with Documents](/modules/utils/how_to_guides.rst): Guides on how to use several of the utilities which will prove helpful for this task, including Text Splitters (for splitting up long documents) and Embeddings & Vectorstores (useful for the above Vector DB example).
 - [CombineDocuments Chains](/modules/indexes/combine_docs.md): A conceptual overview of specific types of chains by which you can accomplish this task.
+
+## End-to-end examples
+
+For examples to this done in an end-to-end manner, please see the following resources:
+
+- [Semantic search over a group chat with Sources Notebook](question_answering/semantic-search-over-chat.ipynb): A notebook that semantically searches over a group chat conversation.

docs/use_cases/question_answering/messages.txt

@@ -0,0 +1,246 @@
+Sure, here's a sample chat with Joey, Rachel, and Monica!
+
+Rachel: Hey guys, how was your day?
+
+Joey: It was pretty good, Rach. I went to the Museum of Natural History today.
+
+Monica: Oh, I love that place! Did you see the dinosaur exhibit?
+
+Joey: Yeah, it was amazing! They had this huge T-Rex skeleton and I was like, "Whoa, that guy's got some serious bite!"
+
+Rachel: (laughs) Classic Joey.
+
+Monica: So, Rachel, what did you do today?
+
+Rachel: Well, I had to work at Central Perk all day. But it wasn't too bad because Gunther let me have an extra espresso shot in my latte.
+
+Joey: (smirks) That's my girl, always pushing the limits.
+
+Monica: Speaking of limits, I went to Barry's Bootcamp this morning and it was insane. I feel like I worked out every muscle in my body.
+
+Rachel: Ugh, I hate working out. Can we please talk about something more fun?
+
+Joey: I know what's fun! I saw my old buddy Chandler today.
+
+Monica: Chandler?! I haven't seen him in ages! How's he doing?
+
+Joey: He's good. He's still living in Tulsa and working at that data processing company.
+
+Rachel: (sarcastically) Wow, sounds thrilling.
+
+Joey: Hey, don't knock it 'til you try it. Chandler's got a great life there. He's got a house, a car, and a pet duck named Yasmine.
+
+Monica: (laughs) A pet duck? Only Chandler would do something like that.
+
+Rachel: You know what's even crazier? I heard Ross went to a furry convention last weekend.
+
+Joey: (spits out his drink) What?! That's insane! Why would he do that?
+
+Monica: I don't know, but I heard he dressed up like a giant squirrel.
+
+Rachel: (laughs) I can't even imagine what that would look like.
+
+Joey: (shakes his head) Ross, man. He's always been a little...quirky.
+
+Monica: (smiling) That's why we love him.
+
+Rachel: (smiling back) Yeah, he's our weird little friend.
+
+Joey: (raising his glass) To weird friends and fun days!
+
+Monica and Rachel: (raising their glasses) Cheers!
+
+Joey: Hey, did you guys hear about that new restaurant that just opened up in the city?
+
+Monica: No, what restaurant?
+
+Joey: It's called "The Hungry Lobster". They serve the biggest lobster you've ever seen!
+
+Rachel: Oh, that sounds amazing! We have to go there.
+
+Monica: Definitely. But first, let's plan a night out with everyone. Maybe we can invite Phoebe and Chandler too?
+
+Joey: (excitedly) Yes! The whole gang together again.
+
+Rachel: (smiling) It'll be just like old times.
+
+Monica: (nodding) We can go to a comedy club or a karaoke bar. What do you guys think?
+
+Joey: Karaoke, definitely. I'll show you guys my killer rendition of "Livin' on a Prayer".
+
+Rachel: (laughs) Oh boy, I can't wait for that.
+
+Monica: (smiling) Okay, so we'll plan for next weekend. Let's make sure everyone can come.
+
+Joey: (raising his glass) To old friends and new memories!
+
+Rachel and Monica: (raising their glasses) Cheers!
+
+As they continued chatting and laughing, they couldn't help but feel grateful for their close friendship and the memories they've shared together. It didn't matter where they were or what they were doing, as long as they were together, they knew they would always have a good time.
+
+As the night went on, the trio found themselves reminiscing about some of their funniest moments together.
+
+Joey: Hey, remember when we all went to that escape room and Ross got stuck in the trapdoor?
+
+Monica: (laughing) Yes! And he was yelling for help like a little kid.
+
+Rachel: (chuckles) Or what about the time when Phoebe got her head stuck in the turkey?
+
+Joey: (cracking up) Oh man, that was hilarious. She looked like she had a turkey on her head!
+
+Monica: (smiling) And let's not forget about the time when Joey got his head stuck in the doorframe.
+
+Joey: (rolling his eyes) Yeah, thanks for bringing that up, Mon.
+
+Rachel: (grinning) Hey, it was a classic moment. We all got a good laugh out of it.
+
+As the night came to an end, the trio hugged each other goodbye, promising to make more fun memories in the future.
+
+Monica: (smiling) I'm so glad we have each other.
+
+Rachel: (nodding) Me too. You guys are like family to me.
+
+Joey: (grinning) Yeah, and the best part is, we're not related by blood so we don't have to invite our weird cousins to our parties.
+
+The girls laughed and waved goodbye to Joey as he walked away, still smiling.
+
+As they walked home, Rachel turned to Monica and said, "You know, we really are lucky to have each other."
+
+Monica nodded in agreement. "Yeah, we are. And I wouldn't trade our crazy, hilarious, and unforgettable moments for anything in the world."
+
+Rachel smiled. "Me neither."
+
+And with that, they continued walking, looking forward to many more fun-filled days and nights with their beloved friends.
+
+The next day, Rachel decided to surprise the gang by booking a private cooking class with a famous chef from Paris.
+
+Rachel: Hey guys, I have a surprise for you!
+
+Joey: (excitedly) What is it? Did you win the lottery?
+
+Monica: (rolling her eyes) Oh, stop it Joey.
+
+Rachel: (smiling) No, even better. I booked us a private cooking class with Chef Jean-Pierre!
+
+Monica: (gasping) What?! Rachel, that's amazing!
+
+Joey: (impressed) Chef Jean-Pierre? He's like, the king of cuisine!
+
+Rachel: (nodding) I know, right? We're going to learn how to make his famous coq au vin.
+
+Monica: (grinning) I can't wait! When is it?
+
+Rachel: (checking her phone) It's this Saturday at 2 pm.
+
+Joey: (smiling) This is going to be awesome.
+
+As Saturday approached, the gang found themselves eagerly anticipating the cooking class. They arrived at the kitchen and were greeted by Chef Jean-Pierre himself.
+
+Chef Jean-Pierre: Bonjour mes amis! Welcome to my kitchen.
+
+Monica: (smiling) Bonjour Chef! We're so excited to learn from you.
+
+Rachel: (nodding) Yes, we're huge fans of your cuisine.
+
+Chef Jean-Pierre: (smiling) Merci beaucoup. Now, let's get started.
+
+For the next few hours, the gang chopped, sautéed, and simmered their way to making the perfect coq au vin. Chef Jean-Pierre showed them his secret ingredients and techniques, and they all enjoyed the delicious meal together.
+
+Joey: (smiling) This is amazing. I feel like I'm in a fancy French restaurant.
+
+Monica: (nodding) And to think, we made this ourselves!
+
+Rachel: (grinning) We're like professional chefs now.
+
+As they finished their meal and said their goodbyes to Chef Jean-Pierre, the gang couldn't help but feel grateful for their friendship and the amazing experiences they shared together.
+
+Rachel: (smiling) This was definitely one of the best days ever.
+
+Monica: (nodding) I agree. And it's all thanks to you, Rach.
+
+Joey: (grinning) Yeah, you're like our own personal Julia Child.
+
+Rachel: (laughing) Hey, I'll take that as a compliment.
+
+And with that, they walked out of the kitchen, already planning their next adventure together.
+
+As they walked down the street, they stumbled upon a street fair happening just a few blocks away.
+
+Monica: (excitedly) Oh my gosh, look at all the food stalls!
+
+Joey: (smelling the air) I can smell the funnel cakes from here.
+
+Rachel: (grinning) Let's check it out!
+
+They weaved their way through the crowd, trying all sorts of delicious street food and drinks. They even came across a dunk tank with a familiar face sitting inside.
+
+Rachel: (gasping) Oh my gosh, it's Gunther!
+
+Joey: (laughing) This I gotta see.
+
+Monica: (smiling) Let's do it!
+
+They each took a turn trying to dunk Gunther, and eventually Rachel got the bullseye and sent him tumbling into the water.
+
+Rachel: (laughing) Sorry Gunther, but that was too fun.
+
+Gunther: (climbing out of the tank) No worries, Rachel. You got me good.
+
+As the night wore on, the gang found themselves exhausted but happy from their fun-filled day.
+
+Joey: (smiling) I don't know about you guys, but I think I'm ready for bed.
+
+Monica: (nodding) Yeah, me too.
+
+Rachel: (grinning) But before we go, we have to take a picture together!
+
+They huddled together and snapped a selfie, capturing the moment forever.
+
+Rachel: (smiling) This was such an amazing day.
+
+Monica: (nodding) Agreed. And we have to do it again soon.
+
+Joey: (grinning) Yeah, and maybe next time we can try something even crazier.
+
+As they said their goodbyes and went their separate ways, the gang couldn't help but feel grateful for their friendship and the many adventures they shared together. They knew that no matter where life took them, they would always have each other to rely on and to make the most of every moment.
+
+
+The next day, Monica decided to invite the gang over to her apartment for a game night. She prepared some snacks and drinks and set up a board game on the coffee table.
+
+Monica: (smiling) Okay guys, who's ready to play?
+
+Rachel: (grinning) I am! What game are we playing?
+
+Monica: (holding up the box) We're playing Monopoly!
+
+Joey: (groaning) Oh no, not Monopoly. That game goes on forever.
+
+Monica: (laughing) Don't worry Joey, we'll make sure it doesn't go on too long.
+
+As they played the game, the gang found themselves getting more and more competitive. Rachel ended up buying all the properties on one side of the board, while Joey and Monica fought over who would control the other side.
+
+Monica: (smirking) You landed on my property, Joey. That's gonna cost you $200.
+
+Joey: (grumbling) Come on Mon, be a pal. I'm already broke!
+
+Rachel: (grinning) Looks like someone's not cut out for the real estate business.
+
+As the game went on, the gang found themselves laughing and joking with each other, forgetting about the stresses of their daily lives.
+
+Monica: (smiling) This is so much fun. We should do this more often.
+
+Rachel: (nodding) Yeah, it's like we're kids again.
+
+Joey: (grinning) And I love beating you guys at this game.
+
+Monica: (laughing) Oh yeah, keep dreaming Joey.
+
+As the night wore on, the gang decided to take a break from the game and watch a movie together. They snuggled up on the couch and watched a comedy, laughing and enjoying each other's company.
+
+Rachel: (yawning) Okay guys, I think it's time for me to head to bed.
+
+Monica: (nodding) Yeah, it's getting late. Thanks for coming over, everyone.
+
+Joey: (smiling) Yeah, this was great. Let's do it again soon.
+
+As they hugged and said their goodbyes, the gang couldn't help but feel grateful for their friendship and the simple pleasures of spending time together. They knew that even though life could be tough sometimes, they had each other to rely on and to make even the most mundane days into something special.

docs/use_cases/question_answering/semantic-search-over-chat.ipynb

@@ -0,0 +1,175 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Question answering over a group chat messages\n",
+    "In this tutorial, we are going to use Langchain + Deep Lake with GPT4 to semantically search and ask questions over a group chat.\n",
+    "\n",
+    "View a working demo [here](https://twitter.com/thisissukh_/status/1647223328363679745)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Install required packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!python3 -m pip install --upgrade langchain deeplake openai tiktoken"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Add API keys"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import getpass\n",
+    "from langchain.document_loaders import PyPDFLoader, TextLoader\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter, CharacterTextSplitter\n",
+    "from langchain.vectorstores import DeepLake\n",
+    "from langchain.chains import ConversationalRetrievalChain, RetrievalQA\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.llms import OpenAI\n",
+    "\n",
+    "os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')\n",
+    "os.environ['ACTIVELOOP_TOKEN'] = getpass.getpass('Activeloop Token:')\n",
+    "os.environ['ACTIVELOOP_ORG'] = getpass.getpass('Activeloop Org:')\n",
+    "\n",
+    "org = os.environ['ACTIVELOOP_ORG']\n",
+    "embeddings = OpenAIEmbeddings()\n",
+    "\n",
+    "dataset_path = 'hub://' + org + '/data'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "## 2. Create sample data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can generate a sample group chat conversation using ChatGPT with this prompt:\n",
+    "\n",
+    "```\n",
+    "Generate a group chat conversation with three friends talking about their day, referencing real places and fictional names. Make it funny and as detailed as possible.\n",
+    "```\n",
+    "\n",
+    "I've already generated such a chat in `messages.txt`. We can keep it simple and use this for our example.\n",
+    "\n",
+    "## 3. Ingest chat embeddings\n",
+    "\n",
+    "We load the messages in the text file, chunk and upload to ActiveLoop Vector store."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"messages.txt\") as f:\n",
+    "    state_of_the_union = f.read()\n",
+    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+    "pages = text_splitter.split_text(state_of_the_union)\n",
+    "\n",
+    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)\n",
+    "texts = text_splitter.split_documents(pages)\n",
+    "\n",
+    "print (texts)\n",
+    "\n",
+    "dataset_path = 'hub://'+org+'/data'\n",
+    "embeddings = OpenAIEmbeddings()\n",
+    "db = DeepLake.from_documents(texts, embeddings, dataset_path=dataset_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Ask questions\n",
+    "\n",
+    "Now we can ask a question and get an answer back with a semantic search:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db = DeepLake(dataset_path=dataset_path, read_only=True, embedding_function=embeddings)\n",
+    "\n",
+    "retriever = db.as_retriever()\n",
+    "retriever.search_kwargs['distance_metric'] = 'cos'\n",
+    "retriever.search_kwargs['k'] = 4\n",
+    "\n",
+    "qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type=\"stuff\", retriever=retriever, return_source_documents=False)\n",
+    "\n",
+    "# What was the restaurant the group was talking about called?\n",
+    "query = input(\"Enter query:\")\n",
+    "\n",
+    "# The Hungry Lobster\n",
+    "ans = qa({\"query\": query})\n",
+    "\n",
+    "print(ans)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

langchain/agents/agent.py

@@ -117,7 +117,7 @@ class BaseSingleActionAgent(BaseModel):
     def dict(self, **kwargs: Any) -> Dict:
         """Return dictionary representation of agent."""
         _dict = super().dict()
-        _dict["_type"] = self._agent_type
+        _dict["_type"] = str(self._agent_type)
         return _dict
 
     def save(self, file_path: Union[Path, str]) -> None:
@@ -229,7 +229,7 @@ class BaseMultiActionAgent(BaseModel):
     def dict(self, **kwargs: Any) -> Dict:
         """Return dictionary representation of agent."""
         _dict = super().dict()
-        _dict["_type"] = self._agent_type
+        _dict["_type"] = str(self._agent_type)
         return _dict
 
     def save(self, file_path: Union[Path, str]) -> None:

langchain/agents/agent_toolkits/__init__.py

@@ -1,6 +1,7 @@
 """Agent toolkits."""
 
 from langchain.agents.agent_toolkits.csv.base import create_csv_agent
+from langchain.agents.agent_toolkits.jira.toolkit import JiraToolkit
 from langchain.agents.agent_toolkits.json.base import create_json_agent
 from langchain.agents.agent_toolkits.json.toolkit import JsonToolkit
 from langchain.agents.agent_toolkits.nla.toolkit import NLAToolkit
@@ -38,4 +39,5 @@ __all__ = [
     "create_pandas_dataframe_agent",
     "create_csv_agent",
     "ZapierToolkit",
+    "JiraToolkit",
 ]

langchain/agents/agent_toolkits/jira/__init__.py

@@ -0,0 +1 @@
+"""Jira Toolkit."""

langchain/agents/agent_toolkits/jira/toolkit.py

@@ -0,0 +1,31 @@
+"""Jira Toolkit."""
+from typing import List
+
+from langchain.agents.agent_toolkits.base import BaseToolkit
+from langchain.tools import BaseTool
+from langchain.tools.jira.tool import JiraAction
+from langchain.utilities.jira import JiraAPIWrapper
+
+
+class JiraToolkit(BaseToolkit):
+    """Jira Toolkit."""
+
+    tools: List[BaseTool] = []
+
+    @classmethod
+    def from_jira_api_wrapper(cls, jira_api_wrapper: JiraAPIWrapper) -> "JiraToolkit":
+        actions = jira_api_wrapper.list()
+        tools = [
+            JiraAction(
+                name=action["name"],
+                description=action["description"],
+                mode=action["mode"],
+                api_wrapper=jira_api_wrapper,
+            )
+            for action in actions
+        ]
+        return cls(tools=tools)
+
+    def get_tools(self) -> List[BaseTool]:
+        """Get the tools in the toolkit."""
+        return self.tools

langchain/agents/chat/output_parser.py

@@ -2,21 +2,25 @@ import json
 from typing import Union
 
 from langchain.agents.agent import AgentOutputParser
-from langchain.schema import AgentAction, AgentFinish
+from langchain.agents.chat.prompt import FORMAT_INSTRUCTIONS
+from langchain.schema import AgentAction, AgentFinish, OutputParserException
 
 FINAL_ANSWER_ACTION = "Final Answer:"
 
 
 class ChatOutputParser(AgentOutputParser):
+    def get_format_instructions(self) -> str:
+        return FORMAT_INSTRUCTIONS
+
     def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
         if FINAL_ANSWER_ACTION in text:
             return AgentFinish(
                 {"output": text.split(FINAL_ANSWER_ACTION)[-1].strip()}, text
             )
         try:
-            _, action, _ = text.split("```")
+            action = text.split("```")[1]
             response = json.loads(action.strip())
             return AgentAction(response["action"], response["action_input"], text)
 
         except Exception:
-            raise ValueError(f"Could not parse LLM output: {text}")
+            raise OutputParserException(f"Could not parse LLM output: {text}")

langchain/agents/conversational/output_parser.py

@@ -2,12 +2,16 @@ import re
 from typing import Union
 
 from langchain.agents.agent import AgentOutputParser
-from langchain.schema import AgentAction, AgentFinish
+from langchain.agents.conversational.prompt import FORMAT_INSTRUCTIONS
+from langchain.schema import AgentAction, AgentFinish, OutputParserException
 
 
 class ConvoOutputParser(AgentOutputParser):
     ai_prefix: str = "AI"
 
+    def get_format_instructions(self) -> str:
+        return FORMAT_INSTRUCTIONS
+
     def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
         if f"{self.ai_prefix}:" in text:
             return AgentFinish(
@@ -16,7 +20,7 @@ class ConvoOutputParser(AgentOutputParser):
         regex = r"Action: (.*?)[\n]*Action Input: (.*)"
         match = re.search(regex, text)
         if not match:
-            raise ValueError(f"Could not parse LLM output: `{text}`")
+            raise OutputParserException(f"Could not parse LLM output: `{text}`")
         action = match.group(1)
         action_input = match.group(2)
         return AgentAction(action.strip(), action_input.strip(" ").strip('"'), text)

langchain/agents/conversational_chat/prompt.py

@@ -10,7 +10,7 @@ Overall, Assistant is a powerful system that can help with a wide range of tasks
 FORMAT_INSTRUCTIONS = """RESPONSE FORMAT INSTRUCTIONS
 ----------------------------
 
-When responding to me please, please output a response in one of two formats:
+When responding to me, please output a response in one of two formats:
 
 **Option 1:**
 Use this if you want the human to use a tool.

langchain/agents/loading.py

@@ -37,8 +37,6 @@ def _load_agent_from_tools(
     if config_type not in AGENT_TO_CLASS:
         raise ValueError(f"Loading {config_type} agent not supported")
 
-    if config_type not in AGENT_TO_CLASS:
-        raise ValueError(f"Loading {config_type} agent not supported")
     agent_cls = AGENT_TO_CLASS[config_type]
     combined_config = {**config, **kwargs}
     return agent_cls.from_llm_and_tools(llm, tools, **combined_config)

langchain/agents/mrkl/output_parser.py

@@ -2,12 +2,16 @@ import re
 from typing import Union
 
 from langchain.agents.agent import AgentOutputParser
-from langchain.schema import AgentAction, AgentFinish
+from langchain.agents.mrkl.prompt import FORMAT_INSTRUCTIONS
+from langchain.schema import AgentAction, AgentFinish, OutputParserException
 
 FINAL_ANSWER_ACTION = "Final Answer:"
 
 
 class MRKLOutputParser(AgentOutputParser):
+    def get_format_instructions(self) -> str:
+        return FORMAT_INSTRUCTIONS
+
     def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
         if FINAL_ANSWER_ACTION in text:
             return AgentFinish(
@@ -17,7 +21,7 @@ class MRKLOutputParser(AgentOutputParser):
         regex = r"Action\s*\d*\s*:(.*?)\nAction\s*\d*\s*Input\s*\d*\s*:[\s]*(.*)"
         match = re.search(regex, text, re.DOTALL)
         if not match:
-            raise ValueError(f"Could not parse LLM output: `{text}`")
+            raise OutputParserException(f"Could not parse LLM output: `{text}`")
         action = match.group(1).strip()
         action_input = match.group(2)
         return AgentAction(action, action_input.strip(" ").strip('"'), text)

langchain/agents/react/output_parser.py

@@ -2,21 +2,23 @@ import re
 from typing import Union
 
 from langchain.agents.agent import AgentOutputParser
-from langchain.schema import AgentAction, AgentFinish
+from langchain.schema import AgentAction, AgentFinish, OutputParserException
 
 
 class ReActOutputParser(AgentOutputParser):
     def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
         action_prefix = "Action: "
         if not text.strip().split("\n")[-1].startswith(action_prefix):
-            raise ValueError(f"Could not parse LLM Output: {text}")
+            raise OutputParserException(f"Could not parse LLM Output: {text}")
         action_block = text.strip().split("\n")[-1]
 
         action_str = action_block[len(action_prefix) :]
         # Parse out the action and the directive.
         re_matches = re.search(r"(.*?)\[(.*?)\]", action_str)
         if re_matches is None:
-            raise ValueError(f"Could not parse action directive: {action_str}")
+            raise OutputParserException(
+                f"Could not parse action directive: {action_str}"
+            )
         action, action_input = re_matches.group(1), re_matches.group(2)
         if action == "Finish":
             return AgentFinish({"output": action_input}, text)

langchain/agents/self_ask_with_search/output_parser.py

@@ -1,7 +1,7 @@
 from typing import Union
 
 from langchain.agents.agent import AgentOutputParser
-from langchain.schema import AgentAction, AgentFinish
+from langchain.schema import AgentAction, AgentFinish, OutputParserException
 
 
 class SelfAskOutputParser(AgentOutputParser):
@@ -12,7 +12,7 @@ class SelfAskOutputParser(AgentOutputParser):
         if followup not in last_line:
             finish_string = "So the final answer is: "
             if finish_string not in last_line:
-                raise ValueError(f"Could not parse output: {text}")
+                raise OutputParserException(f"Could not parse output: {text}")
             return AgentFinish({"output": last_line[len(finish_string) :]}, text)
 
         after_colon = text.split(":")[-1]

langchain/agents/tools.py

@@ -1,25 +1,38 @@
 """Interface for tools."""
 from inspect import signature
-from typing import Any, Awaitable, Callable, Optional, Union
+from typing import Any, Awaitable, Callable, Optional, Type, Union
 
-from langchain.tools.base import BaseTool
+from pydantic import BaseModel
+
+from langchain.tools.base import BaseTool, create_args_schema_model_from_signature
 
 
 class Tool(BaseTool):
     """Tool that takes in function or coroutine directly."""
 
     description: str = ""
-    func: Callable[[str], str]
-    coroutine: Optional[Callable[[str], Awaitable[str]]] = None
+    func: Callable[..., str]
+    """The function to run when the tool is called."""
+    coroutine: Optional[Callable[..., Awaitable[str]]] = None
+    """The asynchronous version of the function."""
 
-    def _run(self, tool_input: str) -> str:
+    @property
+    def args(self) -> Type[BaseModel]:
+        """Generate an input pydantic model."""
+        if self.args_schema is not None:
+            return self.args_schema
+        # Infer the schema directly from the function to add more structured
+        # arguments.
+        return create_args_schema_model_from_signature(self.func)
+
+    def _run(self, *args: Any, **kwargs: Any) -> str:
         """Use the tool."""
-        return self.func(tool_input)
+        return self.func(*args, **kwargs)
 
-    async def _arun(self, tool_input: str) -> str:
+    async def _arun(self, *args: Any, **kwargs: Any) -> str:
         """Use the tool asynchronously."""
         if self.coroutine:
-            return await self.coroutine(tool_input)
+            return await self.coroutine(*args, **kwargs)
         raise NotImplementedError("Tool does not support async")
 
     # TODO: this is for backwards compatibility, remove in future
@@ -69,14 +82,16 @@ def tool(*args: Union[str, Callable], return_direct: bool = False) -> Callable:
     """
 
     def _make_with_name(tool_name: str) -> Callable:
-        def _make_tool(func: Callable[[str], str]) -> Tool:
+        def _make_tool(func: Callable) -> Tool:
             assert func.__doc__, "Function must have a docstring"
             # Description example:
-            #   search_api(query: str) - Searches the API for the query.
+            # search_api(query: str) - Searches the API for the query.
             description = f"{tool_name}{signature(func)} - {func.__doc__.strip()}"
+            args_schema = create_args_schema_model_from_signature(func)
             tool_ = Tool(
                 name=tool_name,
                 func=func,
+                args_schema=args_schema,
                 description=description,
                 return_direct=return_direct,
             )

langchain/callbacks/comet_ml_callback.py

@@ -96,7 +96,7 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
         self,
         task_type: Optional[str] = "inference",
         workspace: Optional[str] = None,
-        project_name: Optional[str] = "comet-langchain-demo",
+        project_name: Optional[str] = None,
         tags: Optional[Sequence] = None,
         name: Optional[str] = None,
         visualizations: Optional[List[str]] = None,
@@ -106,7 +106,7 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
     ) -> None:
         """Initialize callback handler."""
 
-        comet_ml = import_comet_ml()
+        self.comet_ml = import_comet_ml()
         super().__init__()
 
         self.task_type = task_type
@@ -133,7 +133,7 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
             "https://github.com/comet-ml/issue_tracking/issues with the tag "
             "`langchain`."
         )
-        comet_ml.LOGGER.warning(warning)
+        self.comet_ml.LOGGER.warning(warning)
 
         self.callback_columns: list = []
         self.action_records: list = []
@@ -242,8 +242,6 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
         resp.update(flatten_dict(serialized))
         resp.update(self.get_custom_callback_meta())
 
-        comet_ml = import_comet_ml()
-
         for chain_input_key, chain_input_val in inputs.items():
             if isinstance(chain_input_val, str):
                 input_resp = deepcopy(resp)
@@ -253,7 +251,7 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
                 self.action_records.append(input_resp)
 
             else:
-                comet_ml.LOGGER.warning(
+                self.comet_ml.LOGGER.warning(
                     f"Unexpected data format provided! "
                     f"Input Value for {chain_input_key} will not be logged"
                 )
@@ -268,8 +266,6 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
         resp.update({"action": "on_chain_end"})
         resp.update(self.get_custom_callback_meta())
 
-        comet_ml = import_comet_ml()
-
         for chain_output_key, chain_output_val in outputs.items():
             if isinstance(chain_output_val, str):
                 output_resp = deepcopy(resp)
@@ -278,7 +274,7 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
                 output_resp.update({chain_output_key: chain_output_val})
                 self.action_records.append(output_resp)
             else:
-                comet_ml.LOGGER.warning(
+                self.comet_ml.LOGGER.warning(
                     f"Unexpected data format provided! "
                     f"Output Value for {chain_output_key} will not be logged"
                 )
@@ -371,7 +367,7 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
         self.starts += 1
 
         tool = action.tool
-        tool_input = action.tool_input
+        tool_input = str(action.tool_input)
         log = action.log
 
         resp = self._init_resp()
@@ -449,7 +445,14 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
         self._log_session(langchain_asset)
 
         if langchain_asset:
-            self._log_model(langchain_asset)
+            try:
+                self._log_model(langchain_asset)
+            except Exception:
+                self.comet_ml.LOGGER.error(
+                    "Failed to export agent or LLM to Comet",
+                    exc_info=True,
+                    extra={"show_traceback": True},
+                )
 
         if finish:
             self.experiment.end()
@@ -470,8 +473,6 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
         self.experiment.log_text(prompt, metadata=metadata, step=step)
 
     def _log_model(self, langchain_asset: Any) -> None:
-        comet_ml = import_comet_ml()
-
         model_parameters = self._get_llm_parameters(langchain_asset)
         self.experiment.log_parameters(model_parameters, prefix="model")
 
@@ -487,24 +488,45 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
                 langchain_asset.save_agent(langchain_asset_path)
                 self.experiment.log_model(model_name, str(langchain_asset_path))
             else:
-                comet_ml.LOGGER.warning(
+                self.comet_ml.LOGGER.error(
                     f"{e}"
                     " Could not save Langchain Asset "
                     f"for {langchain_asset.__class__.__name__}"
                 )
 
     def _log_session(self, langchain_asset: Optional[Any] = None) -> None:
-        llm_session_df = self._create_session_analysis_dataframe(langchain_asset)
-        # Log the cleaned dataframe as a table
-        self.experiment.log_table("langchain-llm-session.csv", llm_session_df)
+        try:
+            llm_session_df = self._create_session_analysis_dataframe(langchain_asset)
+            # Log the cleaned dataframe as a table
+            self.experiment.log_table("langchain-llm-session.csv", llm_session_df)
+        except Exception:
+            self.comet_ml.LOGGER.warning(
+                "Failed to log session data to Comet",
+                exc_info=True,
+                extra={"show_traceback": True},
+            )
 
-        metadata = {"langchain_version": str(langchain.__version__)}
-        # Log the langchain low-level records as a JSON file directly
-        self.experiment.log_asset_data(
-            self.action_records, "langchain-action_records.json", metadata=metadata
-        )
+        try:
+            metadata = {"langchain_version": str(langchain.__version__)}
+            # Log the langchain low-level records as a JSON file directly
+            self.experiment.log_asset_data(
+                self.action_records, "langchain-action_records.json", metadata=metadata
+            )
+        except Exception:
+            self.comet_ml.LOGGER.warning(
+                "Failed to log session data to Comet",
+                exc_info=True,
+                extra={"show_traceback": True},
+            )
 
-        self._log_visualizations(llm_session_df)
+        try:
+            self._log_visualizations(llm_session_df)
+        except Exception:
+            self.comet_ml.LOGGER.warning(
+                "Failed to log visualizations to Comet",
+                exc_info=True,
+                extra={"show_traceback": True},
+            )
 
     def _log_text_metrics(self, metrics: Sequence[dict], step: int) -> None:
         if not metrics:
@@ -519,7 +541,6 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
             return
 
         spacy = import_spacy()
-        comet_ml = import_comet_ml()
 
         prompts = session_df["prompts"].tolist()
         outputs = session_df["text"].tolist()
@@ -544,7 +565,9 @@ class CometCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
                         step=idx,
                     )
                 except Exception as e:
-                    comet_ml.LOGGER.warning(e)
+                    self.comet_ml.LOGGER.warning(
+                        e, exc_info=True, extra={"show_traceback": True}
+                    )
 
         return
 

langchain/chains/combine_documents/base.py

@@ -7,9 +7,28 @@ from pydantic import Field
 
 from langchain.chains.base import Chain
 from langchain.docstore.document import Document
+from langchain.prompts.base import BasePromptTemplate
 from langchain.text_splitter import RecursiveCharacterTextSplitter, TextSplitter
 
 
+def format_document(doc: Document, prompt: BasePromptTemplate) -> str:
+    """Format a document into a string based on a prompt template."""
+    base_info = {"page_content": doc.page_content}
+    base_info.update(doc.metadata)
+    missing_metadata = set(prompt.input_variables).difference(base_info)
+    if len(missing_metadata) > 0:
+        required_metadata = [
+            iv for iv in prompt.input_variables if iv != "page_content"
+        ]
+        raise ValueError(
+            f"Document prompt requires documents to have metadata variables: "
+            f"{required_metadata}. Received document with missing metadata: "
+            f"{list(missing_metadata)}."
+        )
+    document_info = {k: base_info[k] for k in prompt.input_variables}
+    return prompt.format(**document_info)
+
+
 class BaseCombineDocumentsChain(Chain, ABC):
     """Base interface for chains combining documents."""
 
@@ -70,7 +89,6 @@ class AnalyzeDocumentChain(Chain):
     """Chain that splits documents, then analyzes it in pieces."""
 
     input_key: str = "input_document"  #: :meta private:
-    output_key: str = "output_text"  #: :meta private:
     text_splitter: TextSplitter = Field(default_factory=RecursiveCharacterTextSplitter)
     combine_docs_chain: BaseCombineDocumentsChain
 
@@ -88,7 +106,7 @@ class AnalyzeDocumentChain(Chain):
 
         :meta private:
         """
-        return [self.output_key]
+        return self.combine_docs_chain.output_keys
 
     def _call(self, inputs: Dict[str, Any]) -> Dict[str, str]:
         document = inputs[self.input_key]

langchain/chains/combine_documents/refine.py

@@ -6,7 +6,10 @@ from typing import Any, Dict, List, Tuple
 
 from pydantic import Extra, Field, root_validator
 
-from langchain.chains.combine_documents.base import BaseCombineDocumentsChain
+from langchain.chains.combine_documents.base import (
+    BaseCombineDocumentsChain,
+    format_document,
+)
 from langchain.chains.llm import LLMChain
 from langchain.docstore.document import Document
 from langchain.prompts.base import BasePromptTemplate
@@ -116,14 +119,10 @@ class RefineDocumentsChain(BaseCombineDocumentsChain):
         return res, extra_return_dict
 
     def _construct_refine_inputs(self, doc: Document, res: str) -> Dict[str, Any]:
-        base_info = {"page_content": doc.page_content}
-        base_info.update(doc.metadata)
-        document_info = {k: base_info[k] for k in self.document_prompt.input_variables}
-        base_inputs = {
-            self.document_variable_name: self.document_prompt.format(**document_info),
+        return {
+            self.document_variable_name: format_document(doc, self.document_prompt),
             self.initial_response_name: res,
         }
-        return base_inputs
 
     def _construct_initial_inputs(
         self, docs: List[Document], **kwargs: Any

langchain/chains/combine_documents/stuff.py

@@ -4,7 +4,10 @@ from typing import Any, Dict, List, Optional, Tuple
 
 from pydantic import Extra, Field, root_validator
 
-from langchain.chains.combine_documents.base import BaseCombineDocumentsChain
+from langchain.chains.combine_documents.base import (
+    BaseCombineDocumentsChain,
+    format_document,
+)
 from langchain.chains.llm import LLMChain
 from langchain.docstore.document import Document
 from langchain.prompts.base import BasePromptTemplate
@@ -56,17 +59,8 @@ class StuffDocumentsChain(BaseCombineDocumentsChain):
         return values
 
     def _get_inputs(self, docs: List[Document], **kwargs: Any) -> dict:
-        # Get relevant information from each document.
-        doc_dicts = []
-        for doc in docs:
-            base_info = {"page_content": doc.page_content}
-            base_info.update(doc.metadata)
-            document_info = {
-                k: base_info[k] for k in self.document_prompt.input_variables
-            }
-            doc_dicts.append(document_info)
         # Format each document according to the prompt
-        doc_strings = [self.document_prompt.format(**doc) for doc in doc_dicts]
+        doc_strings = [format_document(doc, self.document_prompt) for doc in docs]
         # Join the documents together to put them in the prompt.
         inputs = {
             k: v

langchain/chains/constitutional_ai/prompts.py

@@ -55,7 +55,7 @@ examples = [
 CRITIQUE_PROMPT = FewShotPromptTemplate(
     example_prompt=critique_example,
     examples=examples,
-    prefix="Below is conservation between a human and an AI model.",
+    prefix="Below is conversation between a human and an AI model.",
     suffix="""Human: {input_prompt}
 Model: {output_from_model}
 
@@ -69,7 +69,7 @@ Critique:""",
 REVISION_PROMPT = FewShotPromptTemplate(
     example_prompt=critique_example,
     examples=examples,
-    prefix="Below is conservation between a human and an AI model.",
+    prefix="Below is conversation between a human and an AI model.",
     suffix="""Human: {input_prompt}
 Model: {output_from_model}
 

langchain/chains/sql_database/prompt.py

@@ -41,6 +41,29 @@ DECIDER_PROMPT = PromptTemplate(
 )
 
 
+_googlesql_prompt = """You are a GoogleSQL expert. Given an input question, first create a syntactically correct GoogleSQL query to run, then look at the results of the query and return the answer to the input question.
+Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the LIMIT clause as per GoogleSQL. You can order the results to return the most informative data in the database.
+Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in backticks (`) to denote them as delimited identifiers.
+Pay attention to use only the column names you can see in the tables below. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.
+
+Use the following format:
+
+Question: "Question here"
+SQLQuery: "SQL Query to run"
+SQLResult: "Result of the SQLQuery"
+Answer: "Final answer here"
+
+Only use the following tables:
+{table_info}
+
+Question: {input}"""
+
+GOOGLESQL_PROMPT = PromptTemplate(
+    input_variables=["input", "table_info", "top_k"],
+    template=_googlesql_prompt,
+)
+
+
 _mssql_prompt = """You are an MS SQL expert. Given an input question, first create a syntactically correct MS SQL query to run, then look at the results of the query and return the answer to the input question.
 Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the TOP clause as per MS SQL. You can order the results to return the most informative data in the database.
 Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in square brackets ([]) to denote them as delimited identifiers.
@@ -178,6 +201,7 @@ SQLITE_PROMPT = PromptTemplate(
 
 
 SQL_PROMPTS = {
+    "googlesql": GOOGLESQL_PROMPT,
     "mssql": MSSQL_PROMPT,
     "mysql": MYSQL_PROMPT,
     "mariadb": MARIADB_PROMPT,

langchain/document_loaders/__init__.py

@@ -18,6 +18,7 @@ from langchain.document_loaders.csv_loader import CSVLoader
 from langchain.document_loaders.dataframe import DataFrameLoader
 from langchain.document_loaders.diffbot import DiffbotLoader
 from langchain.document_loaders.directory import DirectoryLoader
+from langchain.document_loaders.discord import DiscordChatLoader
 from langchain.document_loaders.duckdb_loader import DuckDBLoader
 from langchain.document_loaders.email import (
     OutlookMessageLoader,
@@ -37,6 +38,7 @@ from langchain.document_loaders.html import UnstructuredHTMLLoader
 from langchain.document_loaders.html_bs import BSHTMLLoader
 from langchain.document_loaders.ifixit import IFixitLoader
 from langchain.document_loaders.image import UnstructuredImageLoader
+from langchain.document_loaders.image_captions import ImageCaptionLoader
 from langchain.document_loaders.imsdb import IMSDbLoader
 from langchain.document_loaders.markdown import UnstructuredMarkdownLoader
 from langchain.document_loaders.notebook import NotebookLoader
@@ -61,6 +63,7 @@ from langchain.document_loaders.slack_directory import SlackDirectoryLoader
 from langchain.document_loaders.srt import SRTLoader
 from langchain.document_loaders.telegram import TelegramChatLoader
 from langchain.document_loaders.text import TextLoader
+from langchain.document_loaders.twitter import TwitterTweetLoader
 from langchain.document_loaders.unstructured import (
     UnstructuredFileIOLoader,
     UnstructuredFileLoader,
@@ -146,4 +149,7 @@ __all__ = [
     "BiliBiliLoader",
     "SlackDirectoryLoader",
     "GitLoader",
+    "TwitterTweetLoader",
+    "ImageCaptionLoader",
+    "DiscordChatLoader",
 ]

langchain/document_loaders/confluence.py

@@ -0,0 +1,436 @@
+"""Load Data from a Confluence Space"""
+from typing import Any, Callable, List, Optional, Union
+
+from langchain.docstore.document import Document
+from langchain.document_loaders.base import BaseLoader
+
+
+class ConfluenceLoader(BaseLoader):
+    """
+    Load Confluence pages. Port of https://llamahub.ai/l/confluence
+    This currently supports both username/api_key and Oauth2 login.
+
+    Specify a list page_ids and/or space_key to load in the corresponding pages into
+    Document objects, if both are specified the union of both sets will be returned.
+
+    You can also specify a boolean `include_attachments` to include attachments, this
+    is set to False by default, if set to True all attachments will be downloaded and
+    ConfluenceReader will extract the text from the attachments and add it to the
+    Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG,
+    SVG, Word and Excel.
+
+    Hint: space_key and page_id can both be found in the URL of a page in Confluence
+    - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>
+
+    Example:
+        .. code-block:: python
+
+            from langchain.document_loaders import ConfluenceLoader
+
+            loader = ConfluenceLoader(
+                url="https://yoursite.atlassian.com/wiki",
+                username="me",
+                api_key="12345"
+            )
+            documents = loader.load(space_key="SPACE",limit=50)
+
+    :param url: _description_
+    :type url: str
+    :param api_key: _description_, defaults to None
+    :type api_key: str, optional
+    :param username: _description_, defaults to None
+    :type username: str, optional
+    :param oauth2: _description_, defaults to {}
+    :type oauth2: dict, optional
+    :param cloud: _description_, defaults to True
+    :type cloud: bool, optional
+    :raises ValueError: _description_
+    :raises ImportError: _description_
+    """
+
+    def __init__(
+        self,
+        url: str,
+        api_key: Optional[str] = None,
+        username: Optional[str] = None,
+        oauth2: Optional[dict] = None,
+        cloud: bool = True,
+    ):
+        errors = ConfluenceLoader.validate_init_args(url, api_key, username, oauth2)
+        if errors:
+            raise ValueError(f"Error(s) while validating input: {errors}")
+
+        self.base_url = url
+
+        try:
+            from atlassian import Confluence  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "`atlassian` package not found, please run"
+                "`pip install atlassian-python-api`"
+            )
+
+        if oauth2:
+            self.confluence = Confluence(url=url, oauth2=oauth2, cloud=cloud)
+        else:
+            self.confluence = Confluence(
+                url=url, username=username, password=api_key, cloud=cloud
+            )
+
+    @staticmethod
+    def validate_init_args(
+        url: Optional[str] = None,
+        api_key: Optional[str] = None,
+        username: Optional[str] = None,
+        oauth2: Optional[dict] = None,
+    ) -> Union[List, None]:
+        """Validates proper combinations of init arguments"""
+
+        errors = []
+        if url is None:
+            errors.append("Must provide `base_url`")
+
+        if (api_key and not username) or (username and not api_key):
+            errors.append(
+                "If one of `api_key` or `username` is provided,"
+                "the other must be as well."
+            )
+
+        if (api_key or username) and oauth2:
+            errors.append(
+                "Cannot provide a value for `api_key` and/or"
+                "`username` and provide a value for `oauth2`"
+            )
+
+        if oauth2 and oauth2.keys() != [
+            "access_token",
+            "access_token_secret",
+            "consumer_key",
+            "key_cert",
+        ]:
+            errors.append(
+                "You have either ommited require keys or added extra"
+                "keys to the oauth2 dictionary. key values should be"
+                "`['access_token', 'access_token_secret', 'consumer_key', 'key_cert']`"
+            )
+
+        if errors:
+            return errors
+        return None
+
+    def load(
+        self,
+        space_key: Optional[str] = None,
+        page_ids: Optional[List[str]] = None,
+        label: Optional[str] = None,
+        cql: Optional[str] = None,
+        include_attachments: bool = False,
+        limit: Optional[int] = 50,
+    ) -> List[Document]:
+        """
+        :param space_key: Space key retrieved from a confluence URL, defaults to None
+        :type space_key: Optional[str], optional
+        :param page_ids: List of specific page IDs to load, defaults to None
+        :type page_ids: Optional[List[str]], optional
+        :param label: Get all pages with this label, defaults to None
+        :type label: Optional[str], optional
+        :param cql: CQL Expression, defaults to None
+        :type cql: Optional[str], optional
+        :param include_attachments: defaults to False
+        :type include_attachments: bool, optional
+        :param limit: Maximum number of pages to retrieve, defaults to 50
+        :type limit: int, optional
+        :raises ValueError: _description_
+        :raises ImportError: _description_
+        :return: _description_
+        :rtype: List[Document]
+        """
+        if not space_key and not page_ids and not label and not cql:
+            raise ValueError(
+                "Must specify at least one among `space_key`, `page_ids`,"
+                "`label`, `cql` parameters."
+            )
+
+        try:
+            import html2text  # type: ignore
+        except ImportError:
+            raise ImportError(
+                "`html2text` package not found, please run `pip install html2text`"
+            )
+
+        docs = []
+
+        text_maker = html2text.HTML2Text()
+        text_maker.ignore_links = True
+        text_maker.ignore_images = True
+
+        if space_key:
+            pages = self.paginate_request(
+                self.confluence.get_all_pages_from_space,
+                space=space_key,
+                limit=limit,
+                expand="body.storage.value",
+            )
+            for page in pages:
+                doc = self.process_page(page, include_attachments, text_maker)
+                docs.append(doc)
+
+        if label:
+            pages = self.paginate_request(
+                self.confluence.get_all_pages_by_label,
+                label=label,
+                limit=limit,
+                expand="body.storage.value",
+            )
+            for page in pages:
+                doc = self.process_page(page, include_attachments, text_maker)
+                docs.append(doc)
+
+        if cql:
+            pages = self.paginate_request(
+                self.confluence.cql, cql=cql, limit=limit, expand="body.storage.value"
+            )
+            for page in pages:
+                doc = self.process_page(page, include_attachments, text_maker)
+                docs.append(doc)
+
+        if page_ids:
+            for page_id in page_ids:
+                page = self.confluence.get_page_by_id(
+                    page_id=page_id, expand="body.storage.value"
+                )
+                doc = self.process_page(page, include_attachments, text_maker)
+                docs.append(doc)
+
+        return docs
+
+    def paginate_request(self, retrieval_method: Callable, **kwargs: Any) -> List:
+        """Paginate the various methods to retrieve groups of pages.
+
+        Unforunately, due to page size, sometimes the Confluence API
+        doesn't match the limit value. Also, due to the Atlassian Python
+        package, we don't get the "next" values from the "_links" key because
+        they only return the value from the results key. So here, the pagination
+        starts from 0 and goes until the limit. We have to manually check if there
+        are more docs based on the length of the returned list of pages, rather than
+        just checking for the presence of a `next` key in the response like this page
+        would have you do:
+        https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/
+
+        :param retrieval_method: Function used to retrieve docs
+        :type retrieval_method: callable
+        :return: List of documents
+        :rtype: List
+        """
+
+        limit = kwargs["limit"]
+        page = 0
+        docs = []
+        while page < limit:
+            batch = retrieval_method(**kwargs, start=page)
+            if len(batch) < limit:
+                page = limit
+            else:
+                page += len(batch)
+            docs.extend(batch)
+        return docs
+
+    def process_page(
+        self, page: dict, include_attachments: bool, text_maker: Any
+    ) -> Document:
+        if include_attachments:
+            attachment_texts = self.process_attachment(page["id"])
+        else:
+            attachment_texts = []
+        text = text_maker.handle(page["body"]["storage"]["value"]) + "".join(
+            attachment_texts
+        )
+        return Document(
+            page_content=text, metadata={"title": page["title"], "id": page["id"]}
+        )
+
+    def process_attachment(self, page_id: str) -> List[str]:
+        try:
+            import requests  # noqa: F401
+            from PIL import Image  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "`pytesseract` or `pdf2image` or `Pillow` package not found,"
+                "please run `pip install pytesseract pdf2image Pillow`"
+            )
+
+        # depending on setup you may also need to set the correct path for
+        # poppler and tesseract
+        attachments = self.confluence.get_attachments_from_content(page_id)["results"]
+        texts = []
+        for attachment in attachments:
+            media_type = attachment["metadata"]["mediaType"]
+            absolute_url = self.base_url + attachment["_links"]["download"]
+            title = attachment["title"]
+            if media_type == "application/pdf":
+                text = title + self.process_pdf(absolute_url)
+            elif (
+                media_type == "image/png"
+                or media_type == "image/jpg"
+                or media_type == "image/jpeg"
+            ):
+                text = title + self.process_image(absolute_url)
+            elif (
+                media_type == "application/vnd.openxmlformats-officedocument"
+                ".wordprocessingml.document"
+            ):
+                text = title + self.process_doc(absolute_url)
+            elif media_type == "application/vnd.ms-excel":
+                text = title + self.process_xls(absolute_url)
+            elif media_type == "image/svg+xml":
+                text = title + self.process_svg(absolute_url)
+            else:
+                continue
+            texts.append(text)
+
+        return texts
+
+    def process_pdf(self, link: str) -> str:
+        try:
+            import pytesseract  # noqa: F401
+            from pdf2image import convert_from_bytes  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "`pytesseract` or `pdf2image` package not found,"
+                "please run `pip install pytesseract pdf2image`"
+            )
+
+        import pytesseract  # noqa: F811
+        from pdf2image import convert_from_bytes  # noqa: F811
+
+        response = self.confluence.request(path=link, absolute=True)
+        text = ""
+
+        if (
+            response.status_code != 200
+            or response.content == b""
+            or response.content is None
+        ):
+            return text
+        try:
+            images = convert_from_bytes(response.content)
+        except ValueError:
+            return text
+
+        for i, image in enumerate(images):
+            image_text = pytesseract.image_to_string(image)
+            text += f"Page {i + 1}:\n{image_text}\n\n"
+
+        return text
+
+    def process_image(self, link: str) -> str:
+        try:
+            from io import BytesIO  # noqa: F401
+
+            import pytesseract  # noqa: F401
+            from PIL import Image  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "`pytesseract` or `Pillow` package not found,"
+                "please run `pip install pytesseract Pillow`"
+            )
+
+        response = self.confluence.request(path=link, absolute=True)
+        text = ""
+
+        if (
+            response.status_code != 200
+            or response.content == b""
+            or response.content is None
+        ):
+            return text
+        try:
+            image = Image.open(BytesIO(response.content))
+        except OSError:
+            return text
+
+        return pytesseract.image_to_string(image)
+
+    def process_doc(self, link: str) -> str:
+        try:
+            from io import BytesIO  # noqa: F401
+
+            import docx2txt  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "`docx2txt` package not found, please run `pip install docx2txt`"
+            )
+
+        response = self.confluence.request(path=link, absolute=True)
+        text = ""
+
+        if (
+            response.status_code != 200
+            or response.content == b""
+            or response.content is None
+        ):
+            return text
+        file_data = BytesIO(response.content)
+
+        return docx2txt.process(file_data)
+
+    def process_xls(self, link: str) -> str:
+        try:
+            import xlrd  # noqa: F401
+        except ImportError:
+            raise ImportError("`xlrd` package not found, please run `pip install xlrd`")
+
+        response = self.confluence.request(path=link, absolute=True)
+        text = ""
+
+        if (
+            response.status_code != 200
+            or response.content == b""
+            or response.content is None
+        ):
+            return text
+
+        workbook = xlrd.open_workbook(file_contents=response.content)
+        for sheet in workbook.sheets():
+            text += f"{sheet.name}:\n"
+            for row in range(sheet.nrows):
+                for col in range(sheet.ncols):
+                    text += f"{sheet.cell_value(row, col)}\t"
+                text += "\n"
+            text += "\n"
+
+        return text
+
+    def process_svg(self, link: str) -> str:
+        try:
+            from io import BytesIO  # noqa: F401
+
+            import pytesseract  # noqa: F401
+            from PIL import Image  # noqa: F401
+            from reportlab.graphics import renderPM  # noqa: F401
+            from reportlab.graphics.shapes import Drawing  # noqa: F401
+            from svglib.svglib import svg2rlg  # noqa: F401
+        except ImportError:
+            raise ImportError(
+                "`pytesseract`, `Pillow`, or `svglib` package not found,"
+                "please run `pip install pytesseract Pillow svglib`"
+            )
+
+        response = self.confluence.request(path=link, absolute=True)
+        text = ""
+
+        if (
+            response.status_code != 200
+            or response.content == b""
+            or response.content is None
+        ):
+            return text
+
+        drawing = svg2rlg(BytesIO(response.content))
+
+        img_data = BytesIO()
+        renderPM.drawToFile(drawing, img_data, fmt="PNG")
+        img_data.seek(0)
+        image = Image.open(img_data)
+
+        return pytesseract.image_to_string(image)

langchain/document_loaders/discord.py

@@ -0,0 +1,33 @@
+"""Load from Discord chat dump"""
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List
+
+from langchain.docstore.document import Document
+from langchain.document_loaders.base import BaseLoader
+
+if TYPE_CHECKING:
+    import pandas as pd
+
+
+class DiscordChatLoader(BaseLoader):
+    """Load Discord chat logs."""
+
+    def __init__(self, chat_log: pd.DataFrame, user_id_col: str = "user_id"):
+        """Initialize with a Pandas DataFrame containing chat logs."""
+        if not isinstance(chat_log, pd.DataFrame):
+            raise ValueError(
+                f"Expected chat_log to be a pd.DataFrame, got {type(chat_log)}"
+            )
+        self.chat_log = chat_log
+        self.user_id_col = user_id_col
+
+    def load(self) -> List[Document]:
+        """Load all chat messages."""
+        result = []
+        for _, row in self.chat_log.iterrows():
+            user_id = row[self.user_id_col]
+            metadata = row.to_dict()
+            metadata.pop(self.user_id_col)
+            result.append(Document(page_content=user_id, metadata=metadata))
+        return result

langchain/document_loaders/googledrive.py

@@ -10,7 +10,7 @@
 #   https://cloud.google.com/iam/docs/service-accounts-create
 
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Union
 
 from pydantic import BaseModel, root_validator, validator
 
@@ -29,6 +29,7 @@ class GoogleDriveLoader(BaseLoader, BaseModel):
     folder_id: Optional[str] = None
     document_ids: Optional[List[str]] = None
     file_ids: Optional[List[str]] = None
+    recursive: bool = False
 
     @root_validator
     def validate_folder_id_or_document_ids(
@@ -170,35 +171,49 @@ class GoogleDriveLoader(BaseLoader, BaseModel):
         }
         return Document(page_content=text, metadata=metadata)
 
-    def _load_documents_from_folder(self) -> List[Document]:
+    def _load_documents_from_folder(self, folder_id: str) -> List[Document]:
         """Load documents from a folder."""
         from googleapiclient.discovery import build
 
         creds = self._load_credentials()
         service = build("drive", "v3", credentials=creds)
+        files = self._fetch_files_recursive(service, folder_id)
+        returns = []
+        for file in files:
+            if file["mimeType"] == "application/vnd.google-apps.document":
+                returns.append(self._load_document_from_id(file["id"]))  # type: ignore
+            elif file["mimeType"] == "application/vnd.google-apps.spreadsheet":
+                returns.extend(self._load_sheet_from_id(file["id"]))  # type: ignore
+            elif file["mimeType"] == "application/pdf":
+                returns.extend(self._load_file_from_id(file["id"]))  # type: ignore
+            else:
+                pass
 
+        return returns
+
+    def _fetch_files_recursive(
+        self, service: Any, folder_id: str
+    ) -> List[Dict[str, Union[str, List[str]]]]:
+        """Fetch all files and subfolders recursively."""
         results = (
             service.files()
             .list(
-                q=f"'{self.folder_id}' in parents",
+                q=f"'{folder_id}' in parents",
                 pageSize=1000,
                 includeItemsFromAllDrives=True,
                 supportsAllDrives=True,
-                fields="nextPageToken, files(id, name, mimeType)",
+                fields="nextPageToken, files(id, name, mimeType, parents)",
             )
             .execute()
         )
-        items = results.get("files", [])
+        files = results.get("files", [])
         returns = []
-        for item in items:
-            if item["mimeType"] == "application/vnd.google-apps.document":
-                returns.append(self._load_document_from_id(item["id"]))
-            elif item["mimeType"] == "application/vnd.google-apps.spreadsheet":
-                returns.extend(self._load_sheet_from_id(item["id"]))
-            elif item["mimeType"] == "application/pdf":
-                returns.extend(self._load_file_from_id(item["id"]))
+        for file in files:
+            if file["mimeType"] == "application/vnd.google-apps.folder":
+                if self.recursive:
+                    returns.extend(self._fetch_files_recursive(service, file["id"]))
             else:
-                pass
+                returns.append(file)
 
         return returns
 
@@ -256,7 +271,7 @@ class GoogleDriveLoader(BaseLoader, BaseModel):
     def load(self) -> List[Document]:
         """Load documents."""
         if self.folder_id:
-            return self._load_documents_from_folder()
+            return self._load_documents_from_folder(self.folder_id)
         elif self.document_ids:
             return self._load_documents_from_ids()
         else:

langchain/document_loaders/image_captions.py

@@ -0,0 +1,89 @@
+"""
+Loader that loads image captions
+By default, the loader utilizes the pre-trained BLIP image captioning model.
+https://huggingface.co/Salesforce/blip-image-captioning-base
+
+"""
+from typing import Any, List, Tuple, Union
+
+import requests
+
+from langchain.docstore.document import Document
+from langchain.document_loaders.base import BaseLoader
+
+
+class ImageCaptionLoader(BaseLoader):
+    """Loader that loads the captions of an image"""
+
+    def __init__(
+        self,
+        path_images: Union[str, List[str]],
+        blip_processor: str = "Salesforce/blip-image-captioning-base",
+        blip_model: str = "Salesforce/blip-image-captioning-base",
+    ):
+        """
+        Initialize with a list of image paths
+        """
+        if isinstance(path_images, str):
+            self.image_paths = [path_images]
+        else:
+            self.image_paths = path_images
+
+        self.blip_processor = blip_processor
+        self.blip_model = blip_model
+
+    def load(self) -> List[Document]:
+        """
+        Load from a list of image files
+        """
+        try:
+            from transformers import BlipForConditionalGeneration, BlipProcessor
+        except ImportError:
+            raise ValueError(
+                "transformers package not found, please install with"
+                "`pip install transformers`"
+            )
+
+        processor = BlipProcessor.from_pretrained(self.blip_processor)
+        model = BlipForConditionalGeneration.from_pretrained(self.blip_model)
+
+        results = []
+        for path_image in self.image_paths:
+            caption, metadata = self._get_captions_and_metadata(
+                model=model, processor=processor, path_image=path_image
+            )
+            doc = Document(page_content=caption, metadata=metadata)
+            results.append(doc)
+
+        return results
+
+    def _get_captions_and_metadata(
+        self, model: Any, processor: Any, path_image: str
+    ) -> Tuple[str, dict]:
+        """
+        Helper function for getting the captions and metadata of an image
+        """
+        try:
+            from PIL import Image
+        except ImportError:
+            raise ValueError(
+                "PIL package not found, please install with `pip install pillow`"
+            )
+
+        try:
+            if path_image.startswith("http://") or path_image.startswith("https://"):
+                image = Image.open(requests.get(path_image, stream=True).raw).convert(
+                    "RGB"
+                )
+            else:
+                image = Image.open(path_image).convert("RGB")
+        except Exception:
+            raise ValueError(f"Could not get image data for {path_image}")
+
+        inputs = processor(image, "an image of", return_tensors="pt")
+        output = model.generate(**inputs)
+
+        caption: str = processor.decode(output[0])
+        metadata: dict = {"image_path": path_image}
+
+        return caption, metadata

langchain/document_loaders/obsidian.py

@@ -1,4 +1,5 @@
 """Loader that loads Obsidian directory dump."""
+import re
 from pathlib import Path
 from typing import List
 
@@ -9,10 +10,38 @@ from langchain.document_loaders.base import BaseLoader
 class ObsidianLoader(BaseLoader):
     """Loader that loads Obsidian files from disk."""
 
-    def __init__(self, path: str, encoding: str = "UTF-8"):
+    FRONT_MATTER_REGEX = re.compile(r"^---\n(.*?)\n---\n", re.MULTILINE | re.DOTALL)
+
+    def __init__(
+        self, path: str, encoding: str = "UTF-8", collect_metadata: bool = True
+    ):
         """Initialize with path."""
         self.file_path = path
         self.encoding = encoding
+        self.collect_metadata = collect_metadata
+
+    def _parse_front_matter(self, content: str) -> dict:
+        """Parse front matter metadata from the content and return it as a dict."""
+        if not self.collect_metadata:
+            return {}
+        match = self.FRONT_MATTER_REGEX.search(content)
+        front_matter = {}
+        if match:
+            lines = match.group(1).split("\n")
+            for line in lines:
+                if ":" in line:
+                    key, value = line.split(":", 1)
+                    front_matter[key.strip()] = value.strip()
+                else:
+                    # Skip lines without a colon
+                    continue
+        return front_matter
+
+    def _remove_front_matter(self, content: str) -> str:
+        """Remove front matter metadata from the given content."""
+        if not self.collect_metadata:
+            return content
+        return self.FRONT_MATTER_REGEX.sub("", content)
 
     def load(self) -> List[Document]:
         """Load documents."""
@@ -21,6 +50,10 @@ class ObsidianLoader(BaseLoader):
         for p in ps:
             with open(p, encoding=self.encoding) as f:
                 text = f.read()
-            metadata = {"source": str(p)}
+
+            front_matter = self._parse_front_matter(text)
+            text = self._remove_front_matter(text)
+            metadata = {"source": str(p.name), "path": str(p), **front_matter}
             docs.append(Document(page_content=text, metadata=metadata))
+
         return docs

langchain/document_loaders/twitter.py

@@ -0,0 +1,109 @@
+"""Twitter document loader."""
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Sequence, Union
+
+from langchain.docstore.document import Document
+from langchain.document_loaders.base import BaseLoader
+
+if TYPE_CHECKING:
+    import tweepy
+    from tweepy import OAuth2BearerHandler, OAuthHandler
+
+
+def _dependable_tweepy_import() -> tweepy:
+    try:
+        import tweepy
+    except ImportError:
+        raise ValueError(
+            "tweepy package not found, please install it with `pip install tweepy`"
+        )
+    return tweepy
+
+
+class TwitterTweetLoader(BaseLoader):
+    """Twitter tweets loader.
+    Read tweets of user twitter handle.
+
+    First you need to go to
+    `https://developer.twitter.com/en/docs/twitter-api
+    /getting-started/getting-access-to-the-twitter-api`
+    to get your token. And create a v2 version of the app.
+    """
+
+    def __init__(
+        self,
+        auth_handler: Union[OAuthHandler, OAuth2BearerHandler],
+        twitter_users: Sequence[str],
+        number_tweets: Optional[int] = 100,
+    ):
+        self.auth = auth_handler
+        self.twitter_users = twitter_users
+        self.number_tweets = number_tweets
+
+    def load(self) -> List[Document]:
+        """Load tweets."""
+        tweepy = _dependable_tweepy_import()
+        api = tweepy.API(self.auth, parser=tweepy.parsers.JSONParser())
+
+        results: List[Document] = []
+        for username in self.twitter_users:
+            tweets = api.user_timeline(screen_name=username, count=self.number_tweets)
+            user = api.get_user(screen_name=username)
+            docs = self._format_tweets(tweets, user)
+            results.extend(docs)
+        return results
+
+    def _format_tweets(
+        self, tweets: List[Dict[str, Any]], user_info: dict
+    ) -> Iterable[Document]:
+        """Format tweets into a string."""
+        for tweet in tweets:
+            metadata = {
+                "created_at": tweet["created_at"],
+                "user_info": user_info,
+            }
+            yield Document(
+                page_content=tweet["text"],
+                metadata=metadata,
+            )
+
+    @classmethod
+    def from_bearer_token(
+        cls,
+        oauth2_bearer_token: str,
+        twitter_users: Sequence[str],
+        number_tweets: Optional[int] = 100,
+    ) -> TwitterTweetLoader:
+        """Create a TwitterTweetLoader from OAuth2 bearer token."""
+        tweepy = _dependable_tweepy_import()
+        auth = tweepy.OAuth2BearerHandler(oauth2_bearer_token)
+        return cls(
+            auth_handler=auth,
+            twitter_users=twitter_users,
+            number_tweets=number_tweets,
+        )
+
+    @classmethod
+    def from_secrets(
+        cls,
+        access_token: str,
+        access_token_secret: str,
+        consumer_key: str,
+        consumer_secret: str,
+        twitter_users: Sequence[str],
+        number_tweets: Optional[int] = 100,
+    ) -> TwitterTweetLoader:
+        """Create a TwitterTweetLoader from access tokens and secrets."""
+        tweepy = _dependable_tweepy_import()
+        auth = tweepy.OAuthHandler(
+            access_token=access_token,
+            access_token_secret=access_token_secret,
+            consumer_key=consumer_key,
+            consumer_secret=consumer_secret,
+        )
+        return cls(
+            auth_handler=auth,
+            twitter_users=twitter_users,
+            number_tweets=number_tweets,
+        )

langchain/embeddings/huggingface.py

@@ -1,5 +1,5 @@
 """Wrapper around HuggingFace embedding models."""
-from typing import Any, List
+from typing import Any, List, Optional
 
 from pydantic import BaseModel, Extra
 
@@ -29,6 +29,9 @@ class HuggingFaceEmbeddings(BaseModel, Embeddings):
     client: Any  #: :meta private:
     model_name: str = DEFAULT_MODEL_NAME
     """Model name to use."""
+    cache_folder: Optional[str] = None
+    """Path to store models. 
+    Can be also set by SENTENCE_TRANSFORMERS_HOME enviroment variable."""
 
     def __init__(self, **kwargs: Any):
         """Initialize the sentence_transformer."""
@@ -36,7 +39,9 @@ class HuggingFaceEmbeddings(BaseModel, Embeddings):
         try:
             import sentence_transformers
 
-            self.client = sentence_transformers.SentenceTransformer(self.model_name)
+            self.client = sentence_transformers.SentenceTransformer(
+                self.model_name, self.cache_folder
+            )
         except ImportError:
             raise ValueError(
                 "Could not import sentence_transformers python package. "

langchain/output_parsers/combining.py

@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+from typing import Any, Dict, List
+
+from pydantic import root_validator
+
+from langchain.schema import BaseOutputParser
+
+
+class CombiningOutputParser(BaseOutputParser):
+    """Class to combine multiple output parsers into one."""
+
+    parsers: List[BaseOutputParser]
+
+    @root_validator()
+    def validate_parsers(cls, values: Dict[str, Any]) -> Dict[str, Any]:
+        """Validate the parsers."""
+        parsers = values["parsers"]
+        if len(parsers) < 2:
+            raise ValueError("Must have at least two parsers")
+        for parser in parsers:
+            if parser._type == "combining":
+                raise ValueError("Cannot nest combining parsers")
+            if parser._type == "list":
+                raise ValueError("Cannot comine list parsers")
+        return values
+
+    @property
+    def _type(self) -> str:
+        """Return the type key."""
+        return "combining"
+
+    def get_format_instructions(self) -> str:
+        """Instructions on how the LLM output should be formatted."""
+
+        initial = f"For your first output: {self.parsers[0].get_format_instructions()}"
+        subsequent = "\n".join(
+            [
+                f"Complete that output fully. Then produce another output, separated by two newline characters: {p.get_format_instructions()}"  # noqa: E501
+                for p in self.parsers[1:]
+            ]
+        )
+        return f"{initial}\n{subsequent}"
+
+    def parse(self, text: str) -> Dict[str, Any]:
+        """Parse the output of an LLM call."""
+        texts = text.split("\n\n")
+        output = dict()
+        for i, parser in enumerate(self.parsers):
+            output.update(parser.parse(texts[i].strip()))
+        return output

langchain/output_parsers/fix.py

@@ -41,3 +41,7 @@ class OutputFixingParser(BaseOutputParser[T]):
 
     def get_format_instructions(self) -> str:
         return self.parser.get_format_instructions()
+
+    @property
+    def _type(self) -> str:
+        return self.parser._type

langchain/output_parsers/format_instructions.py

@@ -1,6 +1,6 @@
 # flake8: noqa
 
-STRUCTURED_FORMAT_INSTRUCTIONS = """The output should be a markdown code snippet formatted in the following schema:
+STRUCTURED_FORMAT_INSTRUCTIONS = """The output should be a markdown code snippet formatted in the following schema, including the leading and trailing "\`\`\`json" and "\`\`\`":
 
 ```json
 {{

langchain/output_parsers/list.py

@@ -9,6 +9,10 @@ from langchain.schema import BaseOutputParser
 class ListOutputParser(BaseOutputParser):
     """Class to parse the output of an LLM call to a list."""
 
+    @property
+    def _type(self) -> str:
+        return "list"
+
     @abstractmethod
     def parse(self, text: str) -> List[str]:
         """Parse the output of an LLM call."""

langchain/output_parsers/pydantic.py

@@ -43,3 +43,7 @@ class PydanticOutputParser(BaseOutputParser[T]):
         schema_str = json.dumps(reduced_schema)
 
         return PYDANTIC_FORMAT_INSTRUCTIONS.format(schema=schema_str)
+
+    @property
+    def _type(self) -> str:
+        return "pydantic"

langchain/output_parsers/retry.py

@@ -76,6 +76,10 @@ class RetryOutputParser(BaseOutputParser[T]):
     def get_format_instructions(self) -> str:
         return self.parser.get_format_instructions()
 
+    @property
+    def _type(self) -> str:
+        return self.parser._type
+
 
 class RetryWithErrorOutputParser(BaseOutputParser[T]):
     """Wraps a parser and tries to fix parsing errors.

langchain/output_parsers/structured.py

@@ -38,6 +38,12 @@ class StructuredOutputParser(BaseOutputParser):
         return STRUCTURED_FORMAT_INSTRUCTIONS.format(format=schema_str)
 
     def parse(self, text: str) -> Any:
+        if "```json" not in text:
+            raise OutputParserException(
+                f"Got invalid return object. Expected markdown code snippet with JSON "
+                f"object, but got:\n{text}"
+            )
+
         json_string = text.split("```json")[1].strip().strip("```").strip()
         try:
             json_obj = json.loads(json_string)
@@ -50,3 +56,7 @@ class StructuredOutputParser(BaseOutputParser):
                     f"to be present, but got {json_obj}"
                 )
         return json_obj
+
+    @property
+    def _type(self) -> str:
+        return "structured"

langchain/prompts/prompt.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 
 from pathlib import Path
 from string import Formatter
-from typing import Any, Dict, List, Union
+from typing import Any, Dict, List, Set, Union
 
+from jinja2 import Environment, meta
 from pydantic import Extra, root_validator
 
 from langchain.prompts.base import (
@@ -14,6 +15,13 @@ from langchain.prompts.base import (
 )
 
 
+def _get_jinja2_variables_from_template(template: str) -> Set[str]:
+    env = Environment()
+    ast = env.parse(template)
+    variables = meta.find_undeclared_variables(ast)
+    return variables
+
+
 class PromptTemplate(StringPromptTemplate):
     """Schema to represent a prompt for an LLM.
 
@@ -125,9 +133,15 @@ class PromptTemplate(StringPromptTemplate):
     @classmethod
     def from_template(cls, template: str, **kwargs: Any) -> PromptTemplate:
         """Load a prompt template from a template."""
-        input_variables = {
-            v for _, v, _, _ in Formatter().parse(template) if v is not None
-        }
+        if "template_format" in kwargs and kwargs["template_format"] == "jinja2":
+            # Get the variables for the template
+            input_variables = _get_jinja2_variables_from_template(template)
+
+        else:
+            input_variables = {
+                v for _, v, _, _ in Formatter().parse(template) if v is not None
+            }
+
         return cls(
             input_variables=list(sorted(input_variables)), template=template, **kwargs
         )

langchain/retrievers/chatgpt_plugin_retriever.py

@@ -13,7 +13,7 @@ class ChatGPTPluginRetriever(BaseRetriever, BaseModel):
     url: str
     bearer_token: str
     top_k: int = 3
-    filter: Optional[None] = None
+    filter: Optional[dict] = None
     aiosession: Optional[aiohttp.ClientSession] = None
 
     class Config:

langchain/schema.py

@@ -2,7 +2,7 @@
 from __future__ import annotations
 
 from abc import ABC, abstractmethod
-from typing import Any, Dict, Generic, List, NamedTuple, Optional, TypeVar
+from typing import Any, Dict, Generic, List, NamedTuple, Optional, TypeVar, Union
 
 from pydantic import BaseModel, Extra, Field, root_validator
 
@@ -31,7 +31,7 @@ class AgentAction(NamedTuple):
     """Agent's action to take."""
 
     tool: str
-    tool_input: str
+    tool_input: Union[str, dict]
     log: str
 
 

langchain/tools/base.py

@@ -1,19 +1,84 @@
 """Base implementation for tools or skills."""
 
-from abc import abstractmethod
-from typing import Any, Optional
+import inspect
+from abc import ABC, abstractmethod
+from typing import Any, Callable, Dict, Optional, Sequence, Tuple, Type, Union
 
-from pydantic import BaseModel, Extra, Field, validator
+from pydantic import BaseModel, Extra, Field, create_model, validator
 
 from langchain.callbacks import get_callback_manager
 from langchain.callbacks.base import BaseCallbackManager
 
 
-class BaseTool(BaseModel):
-    """Class responsible for defining a tool or skill for an LLM."""
+def create_args_schema_model_from_signature(run_func: Callable) -> Type[BaseModel]:
+    """Create a pydantic model type from a function's signature."""
+    signature_ = inspect.signature(run_func)
+    field_definitions: Dict[str, Any] = {}
+
+    for name, param in signature_.parameters.items():
+        if name == "self":
+            continue
+        default_value = (
+            param.default if param.default != inspect.Parameter.empty else None
+        )
+        annotation = (
+            param.annotation if param.annotation != inspect.Parameter.empty else Any
+        )
+        # Handle functions with *args in the signature
+        if param.kind == inspect.Parameter.VAR_POSITIONAL:
+            field_definitions[name] = (
+                Any,
+                Field(default=None, extra={"is_var_positional": True}),
+            )
+        # handle functions with **kwargs in the signature
+        elif param.kind == inspect.Parameter.VAR_KEYWORD:
+            field_definitions[name] = (
+                Any,
+                Field(default=None, extra={"is_var_keyword": True}),
+            )
+        # Handle all other named parameters
+        else:
+            is_keyword_only = param.kind == inspect.Parameter.KEYWORD_ONLY
+            field_definitions[name] = (
+                annotation,
+                Field(
+                    default=default_value, extra={"is_keyword_only": is_keyword_only}
+                ),
+            )
+    return create_model("ArgsModel", **field_definitions)  # type: ignore
+
+
+def _to_args_and_kwargs(model: BaseModel) -> Tuple[Sequence, dict]:
+    args = []
+    kwargs = {}
+    for name, field in model.__fields__.items():
+        value = getattr(model, name)
+        # Handle *args in the function signature
+        if field.field_info.extra.get("extra", {}).get("is_var_positional"):
+            if isinstance(value, str):
+                # Base case for backwards compatability
+                args.append(value)
+            elif value is not None:
+                args.extend(value)
+        # Handle **kwargs in the function signature
+        elif field.field_info.extra.get("extra", {}).get("is_var_keyword"):
+            if value is not None:
+                kwargs.update(value)
+        elif field.field_info.extra.get("extra", {}).get("is_keyword_only"):
+            kwargs[name] = value
+        else:
+            args.append(value)
+
+    return tuple(args), kwargs
+
+
+class BaseTool(ABC, BaseModel):
+    """Interface LangChain tools must implement."""
 
     name: str
     description: str
+    args_schema: Optional[Type[BaseModel]] = None
+    """Pydantic model class to validate and parse the tool's input arguments."""
     return_direct: bool = False
     verbose: bool = False
     callback_manager: BaseCallbackManager = Field(default_factory=get_callback_manager)
@@ -24,6 +89,33 @@ class BaseTool(BaseModel):
         extra = Extra.forbid
         arbitrary_types_allowed = True
 
+    @property
+    def args(self) -> Type[BaseModel]:
+        """Generate an input pydantic model."""
+        if self.args_schema is not None:
+            return self.args_schema
+        return create_args_schema_model_from_signature(self._run)
+
+    def _parse_input(
+        self,
+        tool_input: Union[str, Dict],
+    ) -> BaseModel:
+        """Convert tool input to pydantic model."""
+        pydantic_input_type = self.args
+        if isinstance(tool_input, str):
+            # For backwards compatibility, a tool that only takes
+            # a single string input will be converted to a dict.
+            # to be validated.
+            field_name = next(iter(pydantic_input_type.__fields__))
+            tool_input = {field_name: tool_input}
+        if pydantic_input_type is not None:
+            return pydantic_input_type.parse_obj(tool_input)
+        else:
+            raise ValueError(
+                f"args_schema required for tool {self.name} in order to"
+                f" accept input of type {type(tool_input)}"
+            )
+
     @validator("callback_manager", pre=True, always=True)
     def set_callback_manager(
         cls, callback_manager: Optional[BaseCallbackManager]
@@ -35,39 +127,37 @@ class BaseTool(BaseModel):
         return callback_manager or get_callback_manager()
 
     @abstractmethod
-    def _run(self, tool_input: str) -> str:
+    def _run(self, *args: Any, **kwargs: Any) -> str:
         """Use the tool."""
 
     @abstractmethod
-    async def _arun(self, tool_input: str) -> str:
+    async def _arun(self, *args: Any, **kwargs: Any) -> str:
         """Use the tool asynchronously."""
 
-    def __call__(self, tool_input: str) -> str:
-        """Make tools callable with str input."""
-        return self.run(tool_input)
-
     def run(
         self,
-        tool_input: str,
+        tool_input: Union[str, Dict],
         verbose: Optional[bool] = None,
         start_color: Optional[str] = "green",
         color: Optional[str] = "green",
-        **kwargs: Any
+        **kwargs: Any,
     ) -> str:
         """Run the tool."""
+        run_input = self._parse_input(tool_input)
         if not self.verbose and verbose is not None:
             verbose_ = verbose
         else:
             verbose_ = self.verbose
         self.callback_manager.on_tool_start(
             {"name": self.name, "description": self.description},
-            tool_input,
+            str(run_input),
             verbose=verbose_,
             color=start_color,
             **kwargs,
         )
         try:
-            observation = self._run(tool_input)
+            args, kwargs = _to_args_and_kwargs(run_input)
+            observation = self._run(*args, **kwargs)
         except (Exception, KeyboardInterrupt) as e:
             self.callback_manager.on_tool_error(e, verbose=verbose_)
             raise e
@@ -78,13 +168,14 @@ class BaseTool(BaseModel):
 
     async def arun(
         self,
-        tool_input: str,
+        tool_input: Union[str, Dict],
         verbose: Optional[bool] = None,
         start_color: Optional[str] = "green",
         color: Optional[str] = "green",
-        **kwargs: Any
+        **kwargs: Any,
     ) -> str:
         """Run the tool asynchronously."""
+        run_input = self._parse_input(tool_input)
         if not self.verbose and verbose is not None:
             verbose_ = verbose
         else:
@@ -92,7 +183,7 @@ class BaseTool(BaseModel):
         if self.callback_manager.is_async:
             await self.callback_manager.on_tool_start(
                 {"name": self.name, "description": self.description},
-                tool_input,
+                str(run_input.dict()),
                 verbose=verbose_,
                 color=start_color,
                 **kwargs,
@@ -100,14 +191,15 @@ class BaseTool(BaseModel):
         else:
             self.callback_manager.on_tool_start(
                 {"name": self.name, "description": self.description},
-                tool_input,
+                str(run_input.dict()),
                 verbose=verbose_,
                 color=start_color,
                 **kwargs,
             )
         try:
             # We then call the tool on the tool input to get an observation
-            observation = await self._arun(tool_input)
+            args, kwargs = _to_args_and_kwargs(run_input)
+            observation = await self._arun(*args, **kwargs)
         except (Exception, KeyboardInterrupt) as e:
             if self.callback_manager.is_async:
                 await self.callback_manager.on_tool_error(e, verbose=verbose_)
@@ -123,3 +215,7 @@ class BaseTool(BaseModel):
                 observation, verbose=verbose_, color=color, name=self.name, **kwargs
             )
         return observation
+
+    def __call__(self, tool_input: str) -> str:
+        """Make tool callable."""
+        return self.run(tool_input)

langchain/tools/file_management/read.py

@@ -0,0 +1,29 @@
+from typing import Type
+
+from pydantic import BaseModel, Field
+
+from langchain.tools.base import BaseTool
+
+
+class ReadFileInput(BaseModel):
+    """Input for ReadFileTool."""
+
+    file_path: str = Field(..., description="name of file")
+
+
+class ReadFileTool(BaseTool):
+    name: str = "read_file"
+    tool_args: Type[BaseModel] = ReadFileInput
+    description: str = "Read file from disk"
+
+    def _run(self, file_path: str) -> str:
+        try:
+            with open(file_path, "r", encoding="utf-8") as f:
+                content = f.read()
+            return content
+        except Exception as e:
+            return "Error: " + str(e)
+
+    async def _arun(self, tool_input: str) -> str:
+        # TODO: Add aiofiles method
+        raise NotImplementedError

langchain/tools/file_management/write.py

@@ -0,0 +1,34 @@
+import os
+from typing import Type
+
+from pydantic import BaseModel, Field
+
+from langchain.tools.base import BaseTool
+
+
+class WriteFileInput(BaseModel):
+    """Input for WriteFileTool."""
+
+    file_path: str = Field(..., description="name of file")
+    text: str = Field(..., description="text to write to file")
+
+
+class WriteFileTool(BaseTool):
+    name: str = "write_file"
+    tool_args: Type[BaseModel] = WriteFileInput
+    description: str = "Write file to disk"
+
+    def _run(self, file_path: str, text: str) -> str:
+        try:
+            directory = os.path.dirname(file_path)
+            if not os.path.exists(directory) and directory:
+                os.makedirs(directory)
+            with open(file_path, "w", encoding="utf-8") as f:
+                f.write(text)
+            return "File written to successfully."
+        except Exception as e:
+            return "Error: " + str(e)
+
+    async def _arun(self, tool_input: str) -> str:
+        # TODO: Add aiofiles method
+        raise NotImplementedError

langchain/tools/jira/__init__.py

@@ -0,0 +1 @@
+"""Zapier Tool."""

langchain/tools/jira/prompt.py

@@ -0,0 +1,34 @@
+# flake8: noqa
+JIRA_ISSUE_CREATE_PROMPT = """
+    This tool is a wrapper around atlassian-python-api's Jira issue_create API, useful when you need to create a Jira issue. 
+    The input to this tool is a dictionary specifying the fields of the Jira issue, and will be passed into atlassian-python-api's Jira `issue_create` function.
+    For example, to create a low priority task called "test issue" with description "test description", you would pass in the following dictionary: 
+    {{"summary": "test issue", "description": "test description", "issuetype": {{"name": "Task"}}, "priority": {{"name": "Low"}}}}
+    """
+
+JIRA_GET_ALL_PROJECTS_PROMPT = """
+    This tool is a wrapper around atlassian-python-api's Jira project API, 
+    useful when you need to fetch all the projects the user has access to, find out how many projects there are, or as an intermediary step that involv searching by projects. 
+    there is no input to this tool.
+    """
+
+JIRA_JQL_PROMPT = """
+    This tool is a wrapper around atlassian-python-api's Jira jql API, useful when you need to search for Jira issues.
+    The input to this tool is a JQL query string, and will be passed into atlassian-python-api's Jira `jql` function,
+    For example, to find all the issues in project "Test" assigned to the me, you would pass in the following string:
+    project = Test AND assignee = currentUser()
+    or to find issues with summaries that contain the word "test", you would pass in the following string:
+    summary ~ 'test'
+    """
+
+JIRA_CATCH_ALL_PROMPT = """
+    This tool is a wrapper around atlassian-python-api's Jira API.
+    There are other dedicated tools for fetching all projects, and creating and searching for issues, 
+    use this tool if you need to perform any other actions allowed by the atlassian-python-api Jira API.
+    The input to this tool is line of python code that calls a function from atlassian-python-api's Jira API
+    For example, to update the summary field of an issue, you would pass in the following string:
+    self.jira.update_issue_field(key, {{"summary": "New summary"}})
+    or to find out how many projects are in the Jira instance, you would pass in the following string:
+    self.jira.projects()
+    For more information on the Jira API, refer to https://atlassian-python-api.readthedocs.io/jira.html
+    """

langchain/tools/jira/tool.py

@@ -0,0 +1,49 @@
+"""
+This tool allows agents to interact with the atlassian-python-api library
+and operate on a Jira instance. For more information on the
+atlassian-python-api library, see https://atlassian-python-api.readthedocs.io/jira.html
+
+To use this tool, you must first set as environment variables:
+    JIRA_API_TOKEN
+    JIRA_USERNAME
+    JIRA_INSTANCE_URL
+
+Below is a sample script that uses the Jira tool:
+
+```python
+from langchain.agents import AgentType
+from langchain.agents import initialize_agent
+from langchain.agents.agent_toolkits.jira.toolkit import JiraToolkit
+from langchain.llms import OpenAI
+from langchain.utilities.jira import JiraAPIWrapper
+
+llm = OpenAI(temperature=0)
+jira = JiraAPIWrapper()
+toolkit = JiraToolkit.from_jira_api_wrapper(jira)
+agent = initialize_agent(
+    toolkit.get_tools(),
+    llm,
+    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
+    verbose=True
+)
+```
+"""
+from pydantic import Field
+
+from langchain.tools.base import BaseTool
+from langchain.utilities.jira import JiraAPIWrapper
+
+
+class JiraAction(BaseTool):
+    api_wrapper: JiraAPIWrapper = Field(default_factory=JiraAPIWrapper)
+    mode: str
+    name = ""
+    description = ""
+
+    def _run(self, instructions: str) -> str:
+        """Use the Atlassian Jira API to run an operation."""
+        return self.api_wrapper.run(self.mode, instructions)
+
+    async def _arun(self, _: str) -> str:
+        """Use the Atlassian Jira API to run an operation."""
+        raise NotImplementedError("JiraAction does not support async")

langchain/utilities/jira.py

@@ -0,0 +1,180 @@
+"""Util that calls Jira."""
+from typing import Any, Dict, List, Optional
+
+from pydantic import BaseModel, Extra, root_validator
+
+from langchain.tools.jira.prompt import (
+    JIRA_CATCH_ALL_PROMPT,
+    JIRA_GET_ALL_PROJECTS_PROMPT,
+    JIRA_ISSUE_CREATE_PROMPT,
+    JIRA_JQL_PROMPT,
+)
+from langchain.utils import get_from_dict_or_env
+
+
+# TODO: think about error handling, more specific api specs, and jql/project limits
+class JiraAPIWrapper(BaseModel):
+    """Wrapper for Jira API."""
+
+    jira: Any  #: :meta private:
+    jira_username: Optional[str] = None
+    jira_api_token: Optional[str] = None
+    jira_instance_url: Optional[str] = None
+
+    operations: List[Dict] = [
+        {
+            "mode": "jql",
+            "name": "JQL Query",
+            "description": JIRA_JQL_PROMPT,
+        },
+        {
+            "mode": "get_projects",
+            "name": "Get Projects",
+            "description": JIRA_GET_ALL_PROJECTS_PROMPT,
+        },
+        {
+            "mode": "create_issue",
+            "name": "Create Issue",
+            "description": JIRA_ISSUE_CREATE_PROMPT,
+        },
+        {
+            "mode": "other",
+            "name": "Catch all Jira API call",
+            "description": JIRA_CATCH_ALL_PROMPT,
+        },
+    ]
+
+    class Config:
+        """Configuration for this pydantic object."""
+
+        extra = Extra.forbid
+
+    def list(self) -> List[Dict]:
+        return self.operations
+
+    @root_validator()
+    def validate_environment(cls, values: Dict) -> Dict:
+        """Validate that api key and python package exists in environment."""
+        jira_username = get_from_dict_or_env(values, "jira_username", "JIRA_USERNAME")
+        values["jira_username"] = jira_username
+
+        jira_api_token = get_from_dict_or_env(
+            values, "jira_api_token", "JIRA_API_TOKEN"
+        )
+        values["jira_api_token"] = jira_api_token
+
+        jira_instance_url = get_from_dict_or_env(
+            values, "jira_instance_url", "JIRA_INSTANCE_URL"
+        )
+        values["jira_instance_url"] = jira_instance_url
+
+        try:
+            from atlassian import Jira
+        except ImportError:
+            raise ImportError(
+                "atlassian-python-api is not installed. "
+                "Please install it with `pip install atlassian-python-api`"
+            )
+
+        jira = Jira(
+            url=jira_instance_url,
+            username=jira_username,
+            password=jira_api_token,
+            cloud=True,
+        )
+        values["jira"] = jira
+
+        return values
+
+    def parse_issues(self, issues: Dict) -> List[dict]:
+        parsed = []
+        for issue in issues["issues"]:
+            key = issue["key"]
+            summary = issue["fields"]["summary"]
+            created = issue["fields"]["created"][0:10]
+            priority = issue["fields"]["priority"]["name"]
+            status = issue["fields"]["status"]["name"]
+            try:
+                assignee = issue["fields"]["assignee"]["displayName"]
+            except Exception:
+                assignee = "None"
+            rel_issues = {}
+            for related_issue in issue["fields"]["issuelinks"]:
+                if "inwardIssue" in related_issue.keys():
+                    rel_type = related_issue["type"]["inward"]
+                    rel_key = related_issue["inwardIssue"]["key"]
+                    rel_summary = related_issue["inwardIssue"]["fields"]["summary"]
+                if "outwardIssue" in related_issue.keys():
+                    rel_type = related_issue["type"]["outward"]
+                    rel_key = related_issue["outwardIssue"]["key"]
+                    rel_summary = related_issue["outwardIssue"]["fields"]["summary"]
+                rel_issues = {"type": rel_type, "key": rel_key, "summary": rel_summary}
+            parsed.append(
+                {
+                    "key": key,
+                    "summary": summary,
+                    "created": created,
+                    "assignee": assignee,
+                    "priority": priority,
+                    "status": status,
+                    "related_issues": rel_issues,
+                }
+            )
+        return parsed
+
+    def parse_projects(self, projects: List[dict]) -> List[dict]:
+        parsed = []
+        for project in projects:
+            id = project["id"]
+            key = project["key"]
+            name = project["name"]
+            type = project["projectTypeKey"]
+            style = project["style"]
+            parsed.append(
+                {"id": id, "key": key, "name": name, "type": type, "style": style}
+            )
+        return parsed
+
+    def search(self, query: str) -> str:
+        issues = self.jira.jql(query)
+        parsed_issues = self.parse_issues(issues)
+        parsed_issues_str = (
+            "Found " + str(len(parsed_issues)) + " issues:\n" + str(parsed_issues)
+        )
+        return parsed_issues_str
+
+    def project(self) -> str:
+        projects = self.jira.projects()
+        parsed_projects = self.parse_projects(projects)
+        parsed_projects_str = (
+            "Found " + str(len(parsed_projects)) + " projects:\n" + str(parsed_projects)
+        )
+        return parsed_projects_str
+
+    def create(self, query: str) -> str:
+        try:
+            import json
+        except ImportError:
+            raise ImportError(
+                "json is not installed. " "Please install it with `pip install json`"
+            )
+        params = json.loads(query)
+        return self.jira.create_issue(fields=dict(params))
+
+    def other(self, query: str) -> str:
+        context = {"self": self}
+        exec(f"result = {query}", context)
+        result = context["result"]
+        return str(result)
+
+    def run(self, mode: str, query: str) -> str:
+        if mode == "jql":
+            return self.search(query)
+        elif mode == "get_projects":
+            return self.project()
+        elif mode == "create_issue":
+            return self.create(query)
+        elif mode == "other":
+            return self.other(query)
+        else:
+            raise ValueError(f"Got unexpected mode {mode}")

langchain/vectorstores/chroma.py

@@ -60,6 +60,7 @@ class Chroma(VectorStore):
         persist_directory: Optional[str] = None,
         client_settings: Optional[chromadb.config.Settings] = None,
         collection_metadata: Optional[Dict] = None,
+        client: Optional[chromadb.Client] = None,
     ) -> None:
         """Initialize with Chroma client."""
         try:
@@ -71,15 +72,20 @@ class Chroma(VectorStore):
                 "Please install it with `pip install chromadb`."
             )
 
-        if client_settings:
-            self._client_settings = client_settings
+        if client is not None:
+            self._client = client
         else:
-            self._client_settings = chromadb.config.Settings()
-            if persist_directory is not None:
-                self._client_settings = chromadb.config.Settings(
-                    chroma_db_impl="duckdb+parquet", persist_directory=persist_directory
-                )
-        self._client = chromadb.Client(self._client_settings)
+            if client_settings:
+                self._client_settings = client_settings
+            else:
+                self._client_settings = chromadb.config.Settings()
+                if persist_directory is not None:
+                    self._client_settings = chromadb.config.Settings(
+                        chroma_db_impl="duckdb+parquet",
+                        persist_directory=persist_directory,
+                    )
+            self._client = chromadb.Client(self._client_settings)
+
         self._embedding_function = embedding_function
         self._persist_directory = persist_directory
         self._collection = self._client.get_or_create_collection(
@@ -279,6 +285,7 @@ class Chroma(VectorStore):
         collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,
         persist_directory: Optional[str] = None,
         client_settings: Optional[chromadb.config.Settings] = None,
+        client: Optional[chromadb.Client] = None,
         **kwargs: Any,
     ) -> Chroma:
         """Create a Chroma vectorstore from a raw documents.
@@ -303,6 +310,7 @@ class Chroma(VectorStore):
             embedding_function=embedding,
             persist_directory=persist_directory,
             client_settings=client_settings,
+            client=client,
         )
         chroma_collection.add_texts(texts=texts, metadatas=metadatas, ids=ids)
         return chroma_collection
@@ -316,6 +324,7 @@ class Chroma(VectorStore):
         collection_name: str = _LANGCHAIN_DEFAULT_COLLECTION_NAME,
         persist_directory: Optional[str] = None,
         client_settings: Optional[chromadb.config.Settings] = None,
+        client: Optional[chromadb.Client] = None,  # Add this line
         **kwargs: Any,
     ) -> Chroma:
         """Create a Chroma vectorstore from a list of documents.
@@ -343,4 +352,5 @@ class Chroma(VectorStore):
             collection_name=collection_name,
             persist_directory=persist_directory,
             client_settings=client_settings,
+            client=client,
         )

langchain/vectorstores/opensearch_vector_search.py

@@ -146,6 +146,28 @@ def _default_approximate_search_query(
     }
 
 
+def _approximate_search_query_with_boolean_filter(
+    query_vector: List[float],
+    boolean_filter: Dict,
+    size: int = 4,
+    k: int = 4,
+    vector_field: str = "vector_field",
+    subquery_clause: str = "must",
+) -> Dict:
+    """For Approximate k-NN Search, with Boolean Filter."""
+    return {
+        "size": size,
+        "query": {
+            "bool": {
+                "filter": boolean_filter,
+                subquery_clause: [
+                    {"knn": {vector_field: {"vector": query_vector, "k": k}}}
+                ],
+            }
+        },
+    }
+
+
 def _default_script_query(
     query_vector: List[float],
     space_type: str = "l2",
@@ -317,6 +339,11 @@ class OpenSearchVectorSearch(VectorStore):
 
             size: number of results the query actually returns; default: 4
 
+            boolean_filter: A Boolean filter consists of a Boolean query that
+            contains a k-NN query and a filter
+
+            subquery_clause: Query clause on the knn vector field; default: "must"
+
         Optional Args for Script Scoring Search:
             search_type: "script_scoring"; default: "approximate_search"
 
@@ -339,11 +366,19 @@ class OpenSearchVectorSearch(VectorStore):
         text_field = _get_kwargs_value(kwargs, "text_field", "text")
         metadata_field = _get_kwargs_value(kwargs, "metadata_field", "metadata")
         vector_field = _get_kwargs_value(kwargs, "vector_field", "vector_field")
+
         if search_type == "approximate_search":
             size = _get_kwargs_value(kwargs, "size", 4)
-            search_query = _default_approximate_search_query(
-                embedding, size, k, vector_field
-            )
+            boolean_filter = _get_kwargs_value(kwargs, "boolean_filter", {})
+            subquery_clause = _get_kwargs_value(kwargs, "subquery_clause", "must")
+            if boolean_filter != {}:
+                search_query = _approximate_search_query_with_boolean_filter(
+                    embedding, boolean_filter, size, k, vector_field, subquery_clause
+                )
+            else:
+                search_query = _default_approximate_search_query(
+                    embedding, size, k, vector_field
+                )
         elif search_type == SCRIPT_SCORING_SEARCH:
             space_type = _get_kwargs_value(kwargs, "space_type", "l2")
             pre_filter = _get_kwargs_value(kwargs, "pre_filter", MATCH_ALL_QUERY)
@@ -428,7 +463,20 @@ class OpenSearchVectorSearch(VectorStore):
         opensearch_url = get_from_dict_or_env(
             kwargs, "opensearch_url", "OPENSEARCH_URL"
         )
-        client = _get_opensearch_client(opensearch_url)
+        # List of arguments that needs to be removed from kwargs
+        # before passing kwargs to get opensearch client
+        keys_list = [
+            "opensearch_url",
+            "index_name",
+            "is_appx_search",
+            "vector_field",
+            "text_field",
+            "engine",
+            "space_type",
+            "ef_search",
+            "ef_construction",
+            "m",
+        ]
         embeddings = embedding.embed_documents(texts)
         _validate_embeddings_and_bulk_size(len(embeddings), bulk_size)
         dim = len(embeddings[0])
@@ -453,6 +501,8 @@ class OpenSearchVectorSearch(VectorStore):
         else:
             mapping = _default_scripting_text_mapping(dim)
 
+        [kwargs.pop(key, None) for key in keys_list]
+        client = _get_opensearch_client(opensearch_url, **kwargs)
         client.indices.create(index=index_name, body=mapping)
         _bulk_ingest_embeddings(
             client, index_name, embeddings, texts, metadatas, vector_field, text_field

langchain/vectorstores/redis.py

@@ -328,7 +328,15 @@ class Redis(VectorStore):
                 },
             )
         pipeline.execute()
-        return cls(redis_url, index_name, embedding.embed_query)
+        return cls(
+            redis_url,
+            index_name,
+            embedding.embed_query,
+            content_key=content_key,
+            metadata_key=metadata_key,
+            vector_key=vector_key,
+            **kwargs,
+        )
 
     @staticmethod
     def drop_index(
@@ -375,6 +383,9 @@ class Redis(VectorStore):
         cls,
         embedding: Embeddings,
         index_name: str,
+        content_key: str = "content",
+        metadata_key: str = "metadata",
+        vector_key: str = "content_vector",
         **kwargs: Any,
     ) -> Redis:
         """Connect to an existing Redis index."""
@@ -400,7 +411,15 @@ class Redis(VectorStore):
         except Exception as e:
             raise ValueError(f"Redis failed to connect: {e}")
 
-        return cls(redis_url, index_name, embedding.embed_query)
+        return cls(
+            redis_url,
+            index_name,
+            embedding.embed_query,
+            content_key=content_key,
+            metadata_key=metadata_key,
+            vector_key=vector_key,
+            **kwargs,
+        )
 
     def as_retriever(self, **kwargs: Any) -> BaseRetriever:
         return RedisVectorStoreRetriever(vectorstore=self, **kwargs)

poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.4.2 and should not be changed by hand.
+# This file is automatically @generated by Poetry and should not be changed by hand.
 
 [[package]]
 name = "absl-py"
@@ -475,6 +475,27 @@ files = [
     {file = "async_timeout-4.0.2-py3-none-any.whl", hash = "sha256:8ca1e4fcf50d07413d66d1a5e416e42cfdf5851c981d679a09851a6853383b3c"},
 ]
 
+[[package]]
+name = "atlassian-python-api"
+version = "3.36.0"
+description = "Python Atlassian REST API Wrapper"
+category = "main"
+optional = true
+python-versions = "*"
+files = [
+    {file = "atlassian-python-api-3.36.0.tar.gz", hash = "sha256:b1c1f10232818ee3f7e5f59417589971d6f538d12aa79a9784dea09263cf7322"},
+]
+
+[package.dependencies]
+deprecated = "*"
+oauthlib = "*"
+requests = "*"
+requests_oauthlib = "*"
+six = "*"
+
+[package.extras]
+kerberos = ["requests-kerberos"]
+
 [[package]]
 name = "attrs"
 version = "22.2.0"
@@ -2400,6 +2421,18 @@ files = [
     {file = "hpack-4.0.0.tar.gz", hash = "sha256:fc41de0c63e687ebffde81187a948221294896f6bdc0ae2312708df339430095"},
 ]
 
+[[package]]
+name = "html2text"
+version = "2020.1.16"
+description = "Turn HTML into equivalent Markdown-structured text."
+category = "main"
+optional = true
+python-versions = ">=3.5"
+files = [
+    {file = "html2text-2020.1.16-py3-none-any.whl", hash = "sha256:c7c629882da0cf377d66f073329ccf34a12ed2adf0169b9285ae4e63ef54c82b"},
+    {file = "html2text-2020.1.16.tar.gz", hash = "sha256:e296318e16b059ddb97f7a8a1d6a5c1d7af4544049a01e261731d2d5cc277bbb"},
+]
+
 [[package]]
 name = "httpcore"
 version = "0.17.0"
@@ -5906,6 +5939,22 @@ files = [
     {file = "PySocks-1.7.1.tar.gz", hash = "sha256:3f8804571ebe159c380ac6de37643bb4685970655d3bba243530d6558b799aa0"},
 ]
 
+[[package]]
+name = "pytesseract"
+version = "0.3.10"
+description = "Python-tesseract is a python wrapper for Google's Tesseract-OCR"
+category = "main"
+optional = true
+python-versions = ">=3.7"
+files = [
+    {file = "pytesseract-0.3.10-py3-none-any.whl", hash = "sha256:8f22cc98f765bf13517ead0c70effedb46c153540d25783e04014f28b55a5fc6"},
+    {file = "pytesseract-0.3.10.tar.gz", hash = "sha256:f1c3a8b0f07fd01a1085d451f5b8315be6eec1d5577a6796d46dc7a62bd4120f"},
+]
+
+[package.dependencies]
+packaging = ">=21.3"
+Pillow = ">=8.0.0"
+
 [[package]]
 name = "pytest"
 version = "7.3.1"
@@ -7301,7 +7350,7 @@ files = [
 ]
 
 [package.dependencies]
-greenlet = {version = "!=0.4.17", markers = "python_version >= \"3\" and platform_machine == \"aarch64\" or python_version >= \"3\" and platform_machine == \"ppc64le\" or python_version >= \"3\" and platform_machine == \"x86_64\" or python_version >= \"3\" and platform_machine == \"amd64\" or python_version >= \"3\" and platform_machine == \"AMD64\" or python_version >= \"3\" and platform_machine == \"win32\" or python_version >= \"3\" and platform_machine == \"WIN32\""}
+greenlet = {version = "!=0.4.17", markers = "python_version >= \"3\" and (platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\")"}
 
 [package.extras]
 aiomysql = ["aiomysql", "greenlet (!=0.4.17)"]
@@ -9069,13 +9118,13 @@ cffi = {version = ">=1.11", markers = "platform_python_implementation == \"PyPy\
 cffi = ["cffi (>=1.11)"]
 
 [extras]
-all = ["aleph-alpha-client", "anthropic", "beautifulsoup4", "cohere", "deeplake", "elasticsearch", "faiss-cpu", "google-api-python-client", "google-search-results", "huggingface_hub", "jina", "jinja2", "manifest-ml", "networkx", "nlpcloud", "nltk", "nomic", "openai", "opensearch-py", "pgvector", "pinecone-client", "pinecone-text", "psycopg2-binary", "pyowm", "pypdf", "qdrant-client", "redis", "sentence-transformers", "spacy", "tensorflow-text", "tiktoken", "torch", "transformers", "weaviate-client", "wikipedia", "wolframalpha"]
+all = ["anthropic", "cohere", "openai", "nlpcloud", "huggingface_hub", "jina", "manifest-ml", "elasticsearch", "opensearch-py", "google-search-results", "faiss-cpu", "sentence-transformers", "transformers", "spacy", "nltk", "wikipedia", "beautifulsoup4", "tiktoken", "torch", "jinja2", "pinecone-client", "pinecone-text", "weaviate-client", "redis", "google-api-python-client", "wolframalpha", "qdrant-client", "tensorflow-text", "pypdf", "networkx", "nomic", "aleph-alpha-client", "deeplake", "pgvector", "psycopg2-binary", "pyowm"]
 cohere = ["cohere"]
-llms = ["anthropic", "cohere", "huggingface_hub", "manifest-ml", "nlpcloud", "openai", "torch", "transformers"]
+llms = ["anthropic", "cohere", "openai", "nlpcloud", "huggingface_hub", "manifest-ml", "torch", "transformers"]
 openai = ["openai"]
 qdrant = ["qdrant-client"]
 
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.8.1,<4.0"
-content-hash = "47ad0cfafaf5ec6f27bd1713ac237077cd54083960e937ebd005a7c4b25bbe5e"
+content-hash = "f563ddd77272f04b687d9f7e3b97bd39513b38584e271ba5315b85191360cf7a"

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "langchain"
-version = "0.0.141"
+version = "0.0.143"
 description = "Building applications with LLMs through composability"
 authors = []
 license = "MIT"
@@ -60,6 +60,9 @@ psycopg2-binary = {version = "^2.9.5", optional = true}
 pyowm = {version = "^3.3.0", optional = true}
 async-timeout = {version = "^4.0.0", python = "<3.11"}
 gptcache = {version = ">=0.1.7", optional = true}
+atlassian-python-api = {version = "^3.36.0", optional=true}
+pytesseract = {version = "^0.3.10", optional=true}
+html2text = {version="^2020.1.16", optional=true}
 numexpr = "^2.8.4"
 
 [tool.poetry.group.docs.dependencies]
@@ -134,7 +137,7 @@ llms = ["anthropic", "cohere", "openai", "nlpcloud", "huggingface_hub", "manifes
 qdrant = ["qdrant-client"]
 openai = ["openai"]
 cohere = ["cohere"]
-all = ["anthropic", "cohere", "openai", "nlpcloud", "huggingface_hub", "jina", "manifest-ml", "elasticsearch", "opensearch-py", "google-search-results", "faiss-cpu", "sentence_transformers", "transformers", "spacy", "nltk", "wikipedia", "beautifulsoup4", "tiktoken", "torch", "jinja2", "pinecone-client", "pinecone-text", "weaviate-client", "redis", "google-api-python-client", "wolframalpha", "qdrant-client", "tensorflow-text", "pypdf", "networkx", "nomic", "aleph-alpha-client", "deeplake", "pgvector", "psycopg2-binary", "boto3", "pyowm"]
+all = ["anthropic", "cohere", "openai", "nlpcloud", "huggingface_hub", "jina", "manifest-ml", "elasticsearch", "opensearch-py", "google-search-results", "faiss-cpu", "sentence_transformers", "transformers", "spacy", "nltk", "wikipedia", "beautifulsoup4", "tiktoken", "torch", "jinja2", "pinecone-client", "pinecone-text", "weaviate-client", "redis", "google-api-python-client", "wolframalpha", "qdrant-client", "tensorflow-text", "pypdf", "networkx", "nomic", "aleph-alpha-client", "deeplake", "pgvector", "psycopg2-binary", "boto3", "pyowm", "pytesseract", "html2text", "atlassian-python-api", "gptcache"]
 
 [tool.ruff]
 select = [

tests/integration_tests/document_loaders/test_confluence.py

@@ -0,0 +1,39 @@
+import pytest
+
+from langchain.document_loaders.confluence import ConfluenceLoader
+
+try:
+    from atlassian import Confluence  # noqa: F401
+
+    confluence_installed = True
+except ImportError:
+    confluence_installed = False
+
+
+@pytest.mark.skipif(not confluence_installed, reason="Atlassian package not installed")
+def test_load_single_confluence_page() -> None:
+    loader = ConfluenceLoader(url="https://templates.atlassian.net/wiki/")
+    docs = loader.load(page_ids=["33189"])
+
+    assert len(docs) == 1
+    assert docs[0].page_content is not None
+    assert docs[0].metadata["id"] == "33189"
+    assert docs[0].metadata["title"] == "An easy intro to using Confluence"
+
+
+@pytest.mark.skipif(not confluence_installed, reason="Atlassian package not installed")
+def test_load_full_confluence_space() -> None:
+    loader = ConfluenceLoader(url="https://templates.atlassian.net/wiki/")
+    docs = loader.load(space_key="RD")
+
+    assert len(docs) == 14
+    assert docs[0].page_content is not None
+
+
+@pytest.mark.skipif(not confluence_installed, reason="Atlassian package not installed")
+def test_confluence_pagination() -> None:
+    loader = ConfluenceLoader(url="https://templates.atlassian.net/wiki/")
+    docs = loader.load(space_key="RD", limit=5)
+
+    assert len(docs) == 5
+    assert docs[0].page_content is not None

tests/integration_tests/vectorstores/test_opensearch.py

@@ -150,3 +150,17 @@ def test_opensearch_embedding_size_zero() -> None:
         OpenSearchVectorSearch.from_texts(
             [], FakeEmbeddings(), opensearch_url=DEFAULT_OPENSEARCH_URL
         )
+
+
+def test_appx_search_with_boolean_filter() -> None:
+    """Test Approximate Search with Boolean Filter."""
+    boolean_filter_val = {"bool": {"must": [{"term": {"text": "bar"}}]}}
+    docsearch = OpenSearchVectorSearch.from_texts(
+        texts,
+        FakeEmbeddings(),
+        opensearch_url=DEFAULT_OPENSEARCH_URL,
+    )
+    output = docsearch.similarity_search(
+        "foo", k=3, boolean_filter=boolean_filter_val, subquery_clause="should"
+    )
+    assert output == [Document(page_content="bar")]