rm test release condition

- **Description:** This commit fixed the problem that Redis vector store
will change the value of a metadata from 0 to empty when saving the
document, which should be an un-intended behavior.
  - **Issue:** N/A
  - **Dependencies:** N/A

.github/workflows/_release.yml

@@ -14,7 +14,6 @@ env:
 
 jobs:
   build:
-    if: github.ref == 'refs/heads/master'
     runs-on: ubuntu-latest
 
     outputs:
@@ -97,19 +96,18 @@ jobs:
         env:
           PKG_NAME: ${{ needs.build.outputs.pkg-name }}
           VERSION: ${{ needs.build.outputs.version }}
-        # Here we specify:
-        # - The test PyPI index as the *primary* index, meaning that it takes priority.
-        # - The regular PyPI index as an extra index, so that any dependencies that
+        # Here we use:
+        # - The default regular PyPI index as the *primary* index, meaning 
+        #   that it takes priority (https://pypi.org/simple)
+        # - The test PyPI index as an extra index, so that any dependencies that
         #   are not found on test PyPI can be resolved and installed anyway.
-        #
-        # Without the former, we might install the wrong langchain release.
-        # Without the latter, we might not be able to install langchain's dependencies.
+        #   (https://test.pypi.org/simple). This will include the PKG_NAME==VERSION
+        #   package because VERSION will not have been uploaded to regular PyPI yet.
         #
         # TODO: add more in-depth pre-publish tests after testing that importing works
         run: |
           pip install \
-            --index-url https://test.pypi.org/simple/ \
-            --extra-index-url https://pypi.org/simple/ \
+            --extra-index-url https://test.pypi.org/simple/ \
             "$PKG_NAME==$VERSION"
 
           # Replace all dashes in the package name with underscores,

.github/workflows/_test_release.yml

@@ -14,7 +14,6 @@ env:
 
 jobs:
   build:
-    if: github.ref == 'refs/heads/master'
     runs-on: ubuntu-latest
 
     outputs:

.github/workflows/extract_ignored_words_list.py

@@ -3,6 +3,8 @@ import toml
 pyproject_toml = toml.load("pyproject.toml")
 
 # Extract the ignore words list (adjust the key as per your TOML structure)
-ignore_words_list = pyproject_toml.get("tool", {}).get("codespell", {}).get("ignore-words-list")
+ignore_words_list = (
+    pyproject_toml.get("tool", {}).get("codespell", {}).get("ignore-words-list")
+)
 
-print(f"::set-output name=ignore_words_list::{ignore_words_list}")
+print(f"::set-output name=ignore_words_list::{ignore_words_list}")

Makefile

@@ -43,10 +43,10 @@ spell_fix:
 
 lint:
 	poetry run ruff docs templates cookbook
-	poetry run black docs templates cookbook --diff
+	poetry run ruff format docs templates cookbook --diff
 
 format format_diff:
-	poetry run black docs templates cookbook
+	poetry run ruff format docs templates cookbook
 	poetry run ruff --select I --fix docs templates cookbook
 
 ######################

cookbook/LLaMA2_sql_chat.ipynb

@@ -67,7 +67,6 @@
     "llama2_code = ChatOllama(model=\"codellama:7b-instruct\")\n",
     "\n",
     "# API\n",
-    "from getpass import getpass\n",
     "from langchain.llms import Replicate\n",
     "\n",
     "# REPLICATE_API_TOKEN = getpass()\n",

cookbook/README.md

@@ -8,6 +8,7 @@ Notebook | Description
 [Semi_Structured_RAG.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_Structured_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data, including text and tables, using unstructured for parsing, multi-vector retriever for storing, and lcel for implementing chains.
 [Semi_structured_and_multi_moda...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using unstructured for parsing, multi-vector retriever for storage and retrieval, and lcel for implementing chains.
 [Semi_structured_multi_modal_RA...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using various tools and methods such as unstructured for parsing, multi-vector retriever for storing, lcel for implementing chains, and open source language models like llama2, llava, and gpt4all.
+[analyze_document.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/analyze_document.ipynb) | Analyze a single long document.
 [autogpt/autogpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/autogpt.ipynb) | Implement autogpt, a language model, with langchain primitives such as llms, prompttemplates, vectorstores, embeddings, and tools.
 [autogpt/marathon_times.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/marathon_times.ipynb) | Implement autogpt for finding winning marathon times.
 [baby_agi.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi.ipynb) | Implement babyagi, an ai agent that can generate and execute tasks based on a given objective, with the flexibility to swap out specific vectorstores/model providers.
@@ -44,6 +45,7 @@ Notebook | Description
 [plan_and_execute_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/plan_and_execute_agent.ipynb) | Create plan-and-execute agents that accomplish objectives by planning tasks with a language model (llm) and executing them with a separate agent.
 [press_releases.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/press_releases.ipynb) | Retrieve and query company press release data powered by [Kay.ai](https://kay.ai).
 [program_aided_language_model.i...](https://github.com/langchain-ai/langchain/tree/master/cookbook/program_aided_language_model.ipynb) | Implement program-aided language models as described in the provided research paper.
+[qa_citations.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/qa_citations.ipynb) | Different ways to get a model to cite its sources.
 [retrieval_in_sql.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/retrieval_in_sql.ipynb) | Perform retrieval-augmented-generation (rag) on a PostgreSQL database using pgvector.
 [sales_agent_with_context.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/sales_agent_with_context.ipynb) | Implement a context-aware ai sales agent, salesgpt, that can have natural sales conversations, interact with other systems, and use a product knowledge base to discuss a company's offerings.
 [self_query_hotel_search.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/self_query_hotel_search.ipynb) | Build a hotel room search feature with self-querying retrieval, using a specific hotel recommendation dataset.

cookbook/Semi_Structured_RAG.ipynb

@@ -102,9 +102,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from lxml import html\n",
+    "from typing import Any\n",
+    "\n",
     "from pydantic import BaseModel\n",
-    "from typing import Any, Optional\n",
     "from unstructured.partition.pdf import partition_pdf\n",
     "\n",
     "# Get elements\n",
@@ -317,11 +317,12 @@
    "outputs": [],
    "source": [
     "import uuid\n",
-    "from langchain.vectorstores import Chroma\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.schema.document import Document\n",
+    "\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
+    "from langchain.storage import InMemoryStore\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(collection_name=\"summaries\", embedding_function=OpenAIEmbeddings())\n",
@@ -373,7 +374,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from operator import itemgetter\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",

cookbook/Semi_structured_and_multi_modal_RAG.ipynb

@@ -92,9 +92,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from lxml import html\n",
+    "from typing import Any\n",
+    "\n",
     "from pydantic import BaseModel\n",
-    "from typing import Any, Optional\n",
     "from unstructured.partition.pdf import partition_pdf\n",
     "\n",
     "# Get elements\n",
@@ -224,7 +224,7 @@
    "outputs": [],
    "source": [
     "# Prompt\n",
-    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\ \n",
+    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\\n",
     "Give a concise summary of the table or text. Table or text chunk: {element} \"\"\"\n",
     "prompt = ChatPromptTemplate.from_template(prompt_text)\n",
     "\n",
@@ -313,7 +313,7 @@
     "    # Execute the command and save the output to the defined output file\n",
     "    /Users/rlm/Desktop/Code/llama.cpp/bin/llava -m ../models/llava-7b/ggml-model-q5_k.gguf --mmproj ../models/llava-7b/mmproj-model-f16.gguf --temp 0.1 -p \"Describe the image in detail. Be specific about graphs, such as bar plots.\" --image \"$img\" > \"$output_file\"\n",
     "\n",
-    "done"
+    "done\n"
    ]
   },
   {
@@ -337,7 +337,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os, glob\n",
+    "import glob\n",
+    "import os\n",
     "\n",
     "# Get all .txt file summaries\n",
     "file_paths = glob.glob(os.path.expanduser(os.path.join(path, \"*.txt\")))\n",
@@ -371,11 +372,12 @@
    "outputs": [],
    "source": [
     "import uuid\n",
-    "from langchain.vectorstores import Chroma\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.schema.document import Document\n",
+    "\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
+    "from langchain.storage import InMemoryStore\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(collection_name=\"summaries\", embedding_function=OpenAIEmbeddings())\n",
@@ -644,7 +646,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from operator import itemgetter\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",

cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb

@@ -82,10 +82,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import pandas as pd\n",
-    "from lxml import html\n",
+    "from typing import Any\n",
+    "\n",
     "from pydantic import BaseModel\n",
-    "from typing import Any, Optional\n",
     "from unstructured.partition.pdf import partition_pdf\n",
     "\n",
     "# Path to save images\n",
@@ -223,7 +222,7 @@
    "outputs": [],
    "source": [
     "# Prompt\n",
-    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\ \n",
+    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\\n",
     "Give a concise summary of the table or text. Table or text chunk: {element} \"\"\"\n",
     "prompt = ChatPromptTemplate.from_template(prompt_text)\n",
     "\n",
@@ -312,7 +311,7 @@
     "    # Execute the command and save the output to the defined output file\n",
     "    /Users/rlm/Desktop/Code/llama.cpp/bin/llava -m ../models/llava-7b/ggml-model-q5_k.gguf --mmproj ../models/llava-7b/mmproj-model-f16.gguf --temp 0.1 -p \"Describe the image in detail. Be specific about graphs, such as bar plots.\" --image \"$img\" > \"$output_file\"\n",
     "\n",
-    "done"
+    "done\n"
    ]
   },
   {
@@ -322,7 +321,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os, glob\n",
+    "import glob\n",
+    "import os\n",
     "\n",
     "# Get all .txt files in the directory\n",
     "file_paths = glob.glob(os.path.expanduser(os.path.join(path, \"*.txt\")))\n",
@@ -375,11 +375,12 @@
    ],
    "source": [
     "import uuid\n",
-    "from langchain.vectorstores import Chroma\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.schema.document import Document\n",
+    "\n",
     "from langchain.embeddings import GPT4AllEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
+    "from langchain.storage import InMemoryStore\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(\n",
@@ -531,7 +532,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from operator import itemgetter\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",

cookbook/advanced_rag_eval.ipynb

@@ -63,11 +63,13 @@
     "\n",
     "# Load\n",
     "from langchain.document_loaders import PyPDFLoader\n",
+    "\n",
     "loader = PyPDFLoader(path + \"cpi.pdf\")\n",
     "pdf_pages = loader.load()\n",
     "\n",
     "# Split\n",
     "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "\n",
     "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
     "all_splits_pypdf = text_splitter.split_documents(pdf_pages)\n",
     "all_splits_pypdf_texts = [d.page_content for d in all_splits_pypdf]"
@@ -130,12 +132,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.vectorstores import Chroma\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "baseline = Chroma.from_texts(texts=all_splits_pypdf_texts,\n",
-    "                             collection_name=\"baseline\",\n",
-    "                             embedding=OpenAIEmbeddings())\n",
-    "retriever_baseline=baseline.as_retriever()"
+    "from langchain.vectorstores import Chroma\n",
+    "\n",
+    "baseline = Chroma.from_texts(\n",
+    "    texts=all_splits_pypdf_texts,\n",
+    "    collection_name=\"baseline\",\n",
+    "    embedding=OpenAIEmbeddings(),\n",
+    ")\n",
+    "retriever_baseline = baseline.as_retriever()"
    ]
   },
   {
@@ -160,7 +165,7 @@
     "from langchain.schema.output_parser import StrOutputParser\n",
     "\n",
     "# Prompt\n",
-    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\ \n",
+    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\\n",
     "These summaries will be embedded and used to retrieve the raw text or table elements. \\\n",
     "Give a concise summary of the table or text that is well optimized for retrieval. Table or text: {element} \"\"\"\n",
     "prompt = ChatPromptTemplate.from_template(prompt_text)\n",
@@ -169,7 +174,7 @@
     "model = ChatOpenAI(temperature=0, model=\"gpt-4\")\n",
     "summarize_chain = {\"element\": lambda x: x} | prompt | model | StrOutputParser()\n",
     "\n",
-    "# Apply to text \n",
+    "# Apply to text\n",
     "text_summaries = summarize_chain.batch(texts, {\"max_concurrency\": 5})\n",
     "\n",
     "# Apply to tables\n",
@@ -192,31 +197,33 @@
    "outputs": [],
    "source": [
     "# Image summary chain\n",
-    "import os, base64, io\n",
+    "import base64\n",
+    "import io\n",
+    "import os\n",
     "from io import BytesIO\n",
-    "from PIL import Image\n",
+    "\n",
     "from langchain.schema.messages import HumanMessage\n",
+    "from PIL import Image\n",
+    "\n",
     "\n",
     "def encode_image(image_path):\n",
-    "    ''' Getting the base64 string '''\n",
+    "    \"\"\"Getting the base64 string\"\"\"\n",
     "    with open(image_path, \"rb\") as image_file:\n",
-    "        return base64.b64encode(image_file.read()).decode('utf-8') \n",
+    "        return base64.b64encode(image_file.read()).decode(\"utf-8\")\n",
+    "\n",
+    "\n",
+    "def image_summarize(img_base64, prompt):\n",
+    "    \"\"\"Image summary\"\"\"\n",
+    "    chat = ChatOpenAI(model=\"gpt-4-vision-preview\", max_tokens=1024)\n",
     "\n",
-    "def image_summarize(img_base64,prompt):\n",
-    "    ''' Image summary '''\n",
-    "    chat = ChatOpenAI(model=\"gpt-4-vision-preview\",\n",
-    "                      max_tokens=1024)\n",
-    "    \n",
     "    msg = chat.invoke(\n",
     "        [\n",
     "            HumanMessage(\n",
     "                content=[\n",
-    "                    {\"type\": \"text\", \"text\":prompt},\n",
+    "                    {\"type\": \"text\", \"text\": prompt},\n",
     "                    {\n",
     "                        \"type\": \"image_url\",\n",
-    "                        \"image_url\": {\n",
-    "                            \"url\": f\"data:image/jpeg;base64,{img_base64}\"\n",
-    "                        },\n",
+    "                        \"image_url\": {\"url\": f\"data:image/jpeg;base64,{img_base64}\"},\n",
     "                    },\n",
     "                ]\n",
     "            )\n",
@@ -224,6 +231,7 @@
     "    )\n",
     "    return msg.content\n",
     "\n",
+    "\n",
     "# Store base64 encoded images\n",
     "img_base64_list = []\n",
     "\n",
@@ -231,17 +239,17 @@
     "image_summaries = []\n",
     "\n",
     "# Prompt\n",
-    "prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\ \n",
+    "prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\\n",
     "These summaries will be embedded and used to retrieve the raw image. \\\n",
     "Give a concise summary of the image that is well optimized for retrieval.\"\"\"\n",
     "\n",
     "# Apply to images\n",
     "for img_file in sorted(os.listdir(path)):\n",
-    "    if img_file.endswith('.jpg'):\n",
+    "    if img_file.endswith(\".jpg\"):\n",
     "        img_path = os.path.join(path, img_file)\n",
     "        base64_image = encode_image(img_path)\n",
     "        img_base64_list.append(base64_image)\n",
-    "        image_summaries.append(image_summarize(base64_image,prompt))"
+    "        image_summaries.append(image_summarize(base64_image, prompt))"
    ]
   },
   {
@@ -263,18 +271,15 @@
    "source": [
     "import uuid\n",
     "from base64 import b64decode\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.schema.document import Document\n",
-    "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
     "\n",
-    "def create_multi_vector_retriever(vectorstore, \n",
-    "                                  text_summaries, \n",
-    "                                  texts, \n",
-    "                                  table_summaries, \n",
-    "                                  tables, \n",
-    "                                  image_summaries, \n",
-    "                                  images):\n",
-    "    \n",
+    "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
+    "from langchain.storage import InMemoryStore\n",
+    "\n",
+    "\n",
+    "def create_multi_vector_retriever(\n",
+    "    vectorstore, text_summaries, texts, table_summaries, tables, image_summaries, images\n",
+    "):\n",
     "    # Initialize the storage layer\n",
     "    store = InMemoryStore()\n",
     "    id_key = \"doc_id\"\n",
@@ -309,18 +314,22 @@
     "\n",
     "    return retriever\n",
     "\n",
+    "\n",
     "# The vectorstore to use to index the summaries\n",
-    "multi_vector_img = Chroma(collection_name=\"multi_vector_img\", \n",
-    "                     embedding_function=OpenAIEmbeddings())\n",
+    "multi_vector_img = Chroma(\n",
+    "    collection_name=\"multi_vector_img\", embedding_function=OpenAIEmbeddings()\n",
+    ")\n",
     "\n",
     "# Create retriever\n",
-    "retriever_multi_vector_img = create_multi_vector_retriever(multi_vector_img,\n",
-    "                              text_summaries,\n",
-    "                              texts,\n",
-    "                              table_summaries, \n",
-    "                              tables, \n",
-    "                              image_summaries, \n",
-    "                              img_base64_list)"
+    "retriever_multi_vector_img = create_multi_vector_retriever(\n",
+    "    multi_vector_img,\n",
+    "    text_summaries,\n",
+    "    texts,\n",
+    "    table_summaries,\n",
+    "    tables,\n",
+    "    image_summaries,\n",
+    "    img_base64_list,\n",
+    ")"
    ]
   },
   {
@@ -330,10 +339,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Testing on retrieval \n",
-    "query=\"What percentage of CPI is dedicated to Housing, and how does it compare to the combined percentage of Medical Care, Apparel, and Other Goods and Services?\"\n",
-    "suffix_for_images=\" Include any pie charts, graphs, or tables.\"\n",
-    "docs = retriever_multi_vector_img.get_relevant_documents(query+suffix_for_images)"
+    "# Testing on retrieval\n",
+    "query = \"What percentage of CPI is dedicated to Housing, and how does it compare to the combined percentage of Medical Care, Apparel, and Other Goods and Services?\"\n",
+    "suffix_for_images = \" Include any pie charts, graphs, or tables.\"\n",
+    "docs = retriever_multi_vector_img.get_relevant_documents(query + suffix_for_images)"
    ]
   },
   {
@@ -356,15 +365,17 @@
     }
    ],
    "source": [
-    "from IPython.display import display, HTML\n",
-    "def plt_img_base64(img_base64):\n",
+    "from IPython.display import HTML, display\n",
     "\n",
+    "\n",
+    "def plt_img_base64(img_base64):\n",
     "    # Create an HTML img tag with the base64 string as the source\n",
     "    image_html = f'<img src=\"data:image/jpeg;base64,{img_base64}\" />'\n",
-    "    \n",
+    "\n",
     "    # Display the image by rendering the HTML\n",
     "    display(HTML(image_html))\n",
     "\n",
+    "\n",
     "plt_img_base64(docs[1])"
    ]
   },
@@ -386,17 +397,20 @@
    "outputs": [],
    "source": [
     "# The vectorstore to use to index the summaries\n",
-    "multi_vector_text = Chroma(collection_name=\"multi_vector_text\", \n",
-    "                           embedding_function=OpenAIEmbeddings())\n",
+    "multi_vector_text = Chroma(\n",
+    "    collection_name=\"multi_vector_text\", embedding_function=OpenAIEmbeddings()\n",
+    ")\n",
     "\n",
     "# Create retriever\n",
-    "retriever_multi_vector_img_summary = create_multi_vector_retriever(multi_vector_text,\n",
-    "                              text_summaries,\n",
-    "                              texts,\n",
-    "                              table_summaries, \n",
-    "                              tables, \n",
-    "                              image_summaries, \n",
-    "                              image_summaries)"
+    "retriever_multi_vector_img_summary = create_multi_vector_retriever(\n",
+    "    multi_vector_text,\n",
+    "    text_summaries,\n",
+    "    texts,\n",
+    "    table_summaries,\n",
+    "    tables,\n",
+    "    image_summaries,\n",
+    "    image_summaries,\n",
+    ")"
    ]
   },
   {
@@ -418,14 +432,17 @@
     "\n",
     "# Create chroma w/ multi-modal embeddings\n",
     "multimodal_embd = Chroma(\n",
-    "    collection_name=\"multimodal_embd\",\n",
-    "    embedding_function=OpenCLIPEmbeddings()\n",
+    "    collection_name=\"multimodal_embd\", embedding_function=OpenCLIPEmbeddings()\n",
     ")\n",
     "\n",
     "# Get image URIs\n",
-    "image_uris = sorted([os.path.join(path, image_name) \n",
-    "                     for image_name in os.listdir(path) \n",
-    "                     if image_name.endswith('.jpg')])\n",
+    "image_uris = sorted(\n",
+    "    [\n",
+    "        os.path.join(path, image_name)\n",
+    "        for image_name in os.listdir(path)\n",
+    "        if image_name.endswith(\".jpg\")\n",
+    "    ]\n",
+    ")\n",
     "\n",
     "# Add images and documents\n",
     "if image_uris:\n",
@@ -435,7 +452,7 @@
     "if tables:\n",
     "    multimodal_embd.add_texts(texts=tables)\n",
     "\n",
-    "# Make retriever \n",
+    "# Make retriever\n",
     "retriever_multimodal_embd = multimodal_embd.as_retriever()"
    ]
   },
@@ -457,6 +474,7 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
+    "\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt\n",
@@ -466,14 +484,14 @@
     "\"\"\"\n",
     "rag_prompt_text = ChatPromptTemplate.from_template(template)\n",
     "\n",
-    "# Build \n",
+    "\n",
+    "# Build\n",
     "def text_rag_chain(retriever):\n",
-    "    \n",
-    "    ''' RAG chain '''\n",
+    "    \"\"\"RAG chain\"\"\"\n",
     "\n",
     "    # LLM\n",
     "    model = ChatOpenAI(temperature=0, model=\"gpt-4\")\n",
-    "    \n",
+    "\n",
     "    # RAG pipeline\n",
     "    chain = (\n",
     "        {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
@@ -500,13 +518,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re \n",
+    "import re\n",
+    "\n",
     "from langchain.schema import Document\n",
     "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
+    "\n",
     "def looks_like_base64(sb):\n",
     "    \"\"\"Check if the string looks like base64.\"\"\"\n",
-    "    return re.match('^[A-Za-z0-9+/]+[=]{0,2}$', sb) is not None\n",
+    "    return re.match(\"^[A-Za-z0-9+/]+[=]{0,2}$\", sb) is not None\n",
+    "\n",
     "\n",
     "def is_image_data(b64data):\n",
     "    \"\"\"Check if the base64 data is an image by looking at the start of the data.\"\"\"\n",
@@ -514,7 +535,7 @@
     "        b\"\\xFF\\xD8\\xFF\": \"jpg\",\n",
     "        b\"\\x89\\x50\\x4E\\x47\\x0D\\x0A\\x1A\\x0A\": \"png\",\n",
     "        b\"\\x47\\x49\\x46\\x38\": \"gif\",\n",
-    "        b\"\\x52\\x49\\x46\\x46\": \"webp\"\n",
+    "        b\"\\x52\\x49\\x46\\x46\": \"webp\",\n",
     "    }\n",
     "    try:\n",
     "        header = base64.b64decode(b64data)[:8]  # Decode and get the first 8 bytes\n",
@@ -525,6 +546,7 @@
     "    except Exception:\n",
     "        return False\n",
     "\n",
+    "\n",
     "def split_image_text_types(docs):\n",
     "    \"\"\"Split base64-encoded images and texts.\"\"\"\n",
     "    b64_images = []\n",
@@ -539,6 +561,7 @@
     "            texts.append(doc)\n",
     "    return {\"images\": b64_images, \"texts\": texts}\n",
     "\n",
+    "\n",
     "def img_prompt_func(data_dict):\n",
     "    # Joining the context texts into a single string\n",
     "    formatted_texts = \"\\n\".join(data_dict[\"context\"][\"texts\"])\n",
@@ -550,7 +573,7 @@
     "            \"type\": \"image_url\",\n",
     "            \"image_url\": {\n",
     "                \"url\": f\"data:image/jpeg;base64,{data_dict['context']['images'][0]}\"\n",
-    "            }\n",
+    "            },\n",
     "        }\n",
     "        messages.append(image_message)\n",
     "\n",
@@ -563,22 +586,24 @@
     "            f\"User-provided question / keywords: {data_dict['question']}\\n\\n\"\n",
     "            \"Text and / or tables:\\n\"\n",
     "            f\"{formatted_texts}\"\n",
-    "        )\n",
+    "        ),\n",
     "    }\n",
     "    messages.append(text_message)\n",
     "    return [HumanMessage(content=messages)]\n",
     "\n",
+    "\n",
     "def multi_modal_rag_chain(retriever):\n",
-    "    ''' Multi-modal RAG chain '''\n",
+    "    \"\"\"Multi-modal RAG chain\"\"\"\n",
     "\n",
     "    # Multi-modal LLM\n",
-    "    model = ChatOpenAI(temperature=0, \n",
-    "                       model=\"gpt-4-vision-preview\", \n",
-    "                       max_tokens=1024)\n",
-    "    \n",
+    "    model = ChatOpenAI(temperature=0, model=\"gpt-4-vision-preview\", max_tokens=1024)\n",
+    "\n",
     "    # RAG pipeline\n",
     "    chain = (\n",
-    "        {\"context\": retriever | RunnableLambda(split_image_text_types), \"question\": RunnablePassthrough()}\n",
+    "        {\n",
+    "            \"context\": retriever | RunnableLambda(split_image_text_types),\n",
+    "            \"question\": RunnablePassthrough(),\n",
+    "        }\n",
     "        | RunnableLambda(img_prompt_func)\n",
     "        | model\n",
     "        | StrOutputParser()\n",
@@ -603,12 +628,12 @@
    "outputs": [],
    "source": [
     "# RAG chains\n",
-    "chain_baseline=text_rag_chain(retriever_baseline)\n",
-    "chain_mv_text=text_rag_chain(retriever_multi_vector_img_summary)\n",
+    "chain_baseline = text_rag_chain(retriever_baseline)\n",
+    "chain_mv_text = text_rag_chain(retriever_multi_vector_img_summary)\n",
     "\n",
     "# Multi-modal RAG chains\n",
-    "chain_multimodal_mv_img=multi_modal_rag_chain(retriever_multi_vector_img)\n",
-    "chain_multimodal_embd=multi_modal_rag_chain(retriever_multimodal_embd)"
+    "chain_multimodal_mv_img = multi_modal_rag_chain(retriever_multi_vector_img)\n",
+    "chain_multimodal_embd = multi_modal_rag_chain(retriever_multimodal_embd)"
    ]
   },
   {
@@ -694,7 +719,8 @@
    "source": [
     "# Read\n",
     "import pandas as pd\n",
-    "eval_set = pd.read_csv(path+'cpi_eval.csv')\n",
+    "\n",
+    "eval_set = pd.read_csv(path + \"cpi_eval.csv\")\n",
     "eval_set.head(3)"
    ]
   },
@@ -715,12 +741,12 @@
     "# Populate dataset\n",
     "for _, row in eval_set.iterrows():\n",
     "    # Get Q, A\n",
-    "    q = row['Question']\n",
-    "    a = row['Answer']\n",
+    "    q = row[\"Question\"]\n",
+    "    a = row[\"Answer\"]\n",
     "    # Use the values in your function\n",
-    "    client.create_example(inputs={\"question\": q}, \n",
-    "                          outputs={\"answer\": a}, \n",
-    "                          dataset_id=dataset.id)"
+    "    client.create_example(\n",
+    "        inputs={\"question\": q}, outputs={\"answer\": a}, dataset_id=dataset.id\n",
+    "    )"
    ]
   },
   {
@@ -764,17 +790,22 @@
     "    evaluators=[\"qa\"],\n",
     ")\n",
     "\n",
-    "def run_eval(chain,run_name,dataset_name):\n",
+    "\n",
+    "def run_eval(chain, run_name, dataset_name):\n",
     "    _ = client.run_on_dataset(\n",
     "        dataset_name=dataset_name,\n",
-    "        llm_or_chain_factory=lambda: (lambda x: x[\"question\"]+suffix_for_images) | chain,\n",
+    "        llm_or_chain_factory=lambda: (lambda x: x[\"question\"] + suffix_for_images)\n",
+    "        | chain,\n",
     "        evaluation=eval_config,\n",
     "        project_name=run_name,\n",
     "    )\n",
     "\n",
-    "for chain, run in zip([chain_baseline, chain_mv_text, chain_multimodal_mv_img, chain_multimodal_embd], \n",
-    "                      [\"baseline\", \"mv_text\", \"mv_img\", \"mm_embd\"]):\n",
-    "    run_eval(chain, dataset_name+\"-\"+run, dataset_name)"
+    "\n",
+    "for chain, run in zip(\n",
+    "    [chain_baseline, chain_mv_text, chain_multimodal_mv_img, chain_multimodal_embd],\n",
+    "    [\"baseline\", \"mv_text\", \"mv_img\", \"mm_embd\"],\n",
+    "):\n",
+    "    run_eval(chain, dataset_name + \"-\" + run, dataset_name)"
    ]
   }
  ],

cookbook/analyze_document.ipynb

@@ -0,0 +1,105 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f69d4a4c-137d-47e9-bea1-786afce9c1c0",
+   "metadata": {},
+   "source": [
+    "# Analyze a single long document\n",
+    "\n",
+    "The AnalyzeDocumentChain takes in a single document, splits it up, and then runs it through a CombineDocumentsChain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "2a0707ce-6d2d-471b-bc33-64da32a7b3f0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"../docs/docs/modules/state_of_the_union.txt\") as f:\n",
+    "    state_of_the_union = f.read()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "ca14d161-2d5b-4a6c-a296-77d8ce4b28cd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import AnalyzeDocumentChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "\n",
+    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "9f97406c-85a9-45fb-99ce-9138c0ba3731",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains.question_answering import load_qa_chain\n",
+    "\n",
+    "qa_chain = load_qa_chain(llm, chain_type=\"map_reduce\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "0871a753-f5bb-4b4f-a394-f87f2691f659",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "qa_document_chain = AnalyzeDocumentChain(combine_docs_chain=qa_chain)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "e6f86428-3c2c-46a0-a57c-e22826fdbf91",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The President said, \"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.\"'"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "qa_document_chain.run(\n",
+    "    input_document=state_of_the_union,\n",
+    "    question=\"what did the president say about justice breyer?\",\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

cookbook/autogpt/autogpt.ipynb

@@ -27,10 +27,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.utilities import SerpAPIWrapper\n",
     "from langchain.agents import Tool\n",
-    "from langchain.tools.file_management.write import WriteFileTool\n",
     "from langchain.tools.file_management.read import ReadFileTool\n",
+    "from langchain.tools.file_management.write import WriteFileTool\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
     "\n",
     "search = SerpAPIWrapper()\n",
     "tools = [\n",
@@ -61,9 +61,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.vectorstores import FAISS\n",
     "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain.embeddings import OpenAIEmbeddings"
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {
@@ -100,8 +100,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_experimental.autonomous_agents import AutoGPT\n",
-    "from langchain.chat_models import ChatOpenAI"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain_experimental.autonomous_agents import AutoGPT"
    ]
   },
   {

cookbook/autogpt/marathon_times.ipynb

@@ -34,16 +34,15 @@
    "outputs": [],
    "source": [
     "# General\n",
-    "import os\n",
-    "import pandas as pd\n",
-    "from langchain_experimental.autonomous_agents import AutoGPT\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "from langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent\n",
-    "from langchain.docstore.document import Document\n",
     "import asyncio\n",
-    "import nest_asyncio\n",
+    "import os\n",
     "\n",
+    "import nest_asyncio\n",
+    "import pandas as pd\n",
+    "from langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.docstore.document import Document\n",
+    "from langchain_experimental.autonomous_agents import AutoGPT\n",
     "\n",
     "# Needed synce jupyter runs an async eventloop\n",
     "nest_asyncio.apply()"
@@ -92,6 +91,7 @@
     "import os\n",
     "from contextlib import contextmanager\n",
     "from typing import Optional\n",
+    "\n",
     "from langchain.agents import tool\n",
     "from langchain.tools.file_management.read import ReadFileTool\n",
     "from langchain.tools.file_management.write import WriteFileTool\n",
@@ -223,14 +223,13 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.tools import BaseTool, DuckDuckGoSearchRun\n",
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "\n",
-    "from pydantic import Field\n",
     "from langchain.chains.qa_with_sources.loading import (\n",
-    "    load_qa_with_sources_chain,\n",
     "    BaseCombineDocumentsChain,\n",
+    "    load_qa_with_sources_chain,\n",
     ")\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "from langchain.tools import BaseTool, DuckDuckGoSearchRun\n",
+    "from pydantic import Field\n",
     "\n",
     "\n",
     "def _get_text_splitter():\n",
@@ -311,10 +310,9 @@
    "source": [
     "# Memory\n",
     "import faiss\n",
-    "from langchain.vectorstores import FAISS\n",
     "from langchain.docstore import InMemoryDocstore\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.tools.human.tool import HumanInputRun\n",
+    "from langchain.vectorstores import FAISS\n",
     "\n",
     "embeddings_model = OpenAIEmbeddings()\n",
     "embedding_size = 1536\n",

cookbook/baby_agi.ipynb

@@ -29,16 +29,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
-    "from collections import deque\n",
-    "from typing import Dict, List, Optional, Any\n",
+    "from typing import Optional\n",
     "\n",
-    "from langchain.chains import LLMChain\nfrom langchain.llms import OpenAI\nfrom langchain.prompts import PromptTemplate\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.llms import BaseLLM\n",
-    "from langchain.schema.vectorstore import VectorStore\n",
-    "from pydantic import BaseModel, Field\n",
-    "from langchain.chains.base import Chain\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.autonomous_agents import BabyAGI"
    ]
   },
@@ -59,8 +53,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.vectorstores import FAISS\n",
-    "from langchain.docstore import InMemoryDocstore"
+    "from langchain.docstore import InMemoryDocstore\n",
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {

cookbook/baby_agi_with_agent.ipynb

@@ -25,16 +25,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
-    "from collections import deque\n",
-    "from typing import Dict, List, Optional, Any\n",
+    "from typing import Optional\n",
     "\n",
-    "from langchain.chains import LLMChain\nfrom langchain.llms import OpenAI\nfrom langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.llms import BaseLLM\n",
-    "from langchain.schema.vectorstore import VectorStore\n",
-    "from pydantic import BaseModel, Field\n",
-    "from langchain.chains.base import Chain\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain_experimental.autonomous_agents import BabyAGI"
    ]
   },
@@ -66,8 +62,8 @@
    "source": [
     "%pip install faiss-cpu > /dev/null\n",
     "%pip install google-search-results > /dev/null\n",
-    "from langchain.vectorstores import FAISS\n",
-    "from langchain.docstore import InMemoryDocstore"
+    "from langchain.docstore import InMemoryDocstore\n",
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {
@@ -110,8 +106,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import ZeroShotAgent, Tool, AgentExecutor\n",
-    "from langchain.llms import OpenAI\nfrom langchain.utilities import SerpAPIWrapper\nfrom langchain.chains import LLMChain\n",
+    "from langchain.agents import AgentExecutor, Tool, ZeroShotAgent\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
     "\n",
     "todo_prompt = PromptTemplate.from_template(\n",
     "    \"You are a planner who is an expert at coming up with a todo list for a given objective. Come up with a todo list for this objective: {objective}\"\n",

cookbook/camel_role_playing.ipynb

@@ -35,16 +35,17 @@
    "outputs": [],
    "source": [
     "from typing import List\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts.chat import (\n",
-    "    SystemMessagePromptTemplate,\n",
     "    HumanMessagePromptTemplate,\n",
+    "    SystemMessagePromptTemplate,\n",
     ")\n",
     "from langchain.schema import (\n",
     "    AIMessage,\n",
+    "    BaseMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/causal_program_aided_language_model.ipynb

@@ -47,10 +47,9 @@
    "outputs": [],
    "source": [
     "from IPython.display import SVG\n",
-    "\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.cpal.base import CPALChain\n",
     "from langchain_experimental.pal_chain import PALChain\n",
-    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0, max_tokens=512)\n",
     "cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)\n",

cookbook/code-analysis-deeplake.ipynb

@@ -177,7 +177,7 @@
     "            try:\n",
     "                loader = TextLoader(os.path.join(dirpath, file), encoding=\"utf-8\")\n",
     "                docs.extend(loader.load_and_split())\n",
-    "            except Exception as e:\n",
+    "            except Exception:\n",
     "                pass\n",
     "print(f\"{len(docs)}\")"
    ]
@@ -717,7 +717,6 @@
    "source": [
     "from langchain.vectorstores import DeepLake\n",
     "\n",
-    "\n",
     "username = \"<USERNAME_OR_ORG>\"\n",
     "\n",
     "\n",
@@ -834,8 +833,8 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.chains import ConversationalRetrievalChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "model = ChatOpenAI(\n",
     "    model_name=\"gpt-3.5-turbo-0613\"\n",

cookbook/custom_agent_with_plugin_retrieval.ipynb

@@ -32,19 +32,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import re\n",
+    "from typing import Union\n",
+    "\n",
     "from langchain.agents import (\n",
-    "    Tool,\n",
     "    AgentExecutor,\n",
-    "    LLMSingleActionAgent,\n",
     "    AgentOutputParser,\n",
+    "    LLMSingleActionAgent,\n",
     ")\n",
-    "from langchain.prompts import StringPromptTemplate\n",
-    "from langchain.llms import OpenAI\nfrom langchain.utilities import SerpAPIWrapper\nfrom langchain.chains import LLMChain\n",
-    "from typing import List, Union\n",
-    "from langchain.schema import AgentAction, AgentFinish\n",
     "from langchain.agents.agent_toolkits import NLAToolkit\n",
-    "from langchain.tools.plugin import AIPlugin\n",
-    "import re"
+    "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.prompts import StringPromptTemplate\n",
+    "from langchain.schema import AgentAction, AgentFinish\n",
+    "from langchain.tools.plugin import AIPlugin"
    ]
   },
   {
@@ -113,9 +114,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.vectorstores import FAISS\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema import Document"
+    "from langchain.schema import Document\n",
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {

cookbook/custom_agent_with_plugin_retrieval_using_plugnplai.ipynb

@@ -56,20 +56,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import (\n",
-    "    Tool,\n",
-    "    AgentExecutor,\n",
-    "    LLMSingleActionAgent,\n",
-    "    AgentOutputParser,\n",
-    ")\n",
-    "from langchain.prompts import StringPromptTemplate\n",
-    "from langchain.llms import OpenAI\nfrom langchain.utilities import SerpAPIWrapper\nfrom langchain.chains import LLMChain\n",
-    "from typing import List, Union\n",
-    "from langchain.schema import AgentAction, AgentFinish\n",
-    "from langchain.agents.agent_toolkits import NLAToolkit\n",
-    "from langchain.tools.plugin import AIPlugin\n",
     "import re\n",
-    "import plugnplai"
+    "from typing import Union\n",
+    "\n",
+    "import plugnplai\n",
+    "from langchain.agents import (\n",
+    "    AgentExecutor,\n",
+    "    AgentOutputParser,\n",
+    "    LLMSingleActionAgent,\n",
+    ")\n",
+    "from langchain.agents.agent_toolkits import NLAToolkit\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.prompts import StringPromptTemplate\n",
+    "from langchain.schema import AgentAction, AgentFinish\n",
+    "from langchain.tools.plugin import AIPlugin"
    ]
   },
   {
@@ -137,9 +138,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.vectorstores import FAISS\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema import Document"
+    "from langchain.schema import Document\n",
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {

cookbook/deeplake_semantic_search_over_chat.ipynb

@@ -48,18 +48,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
     "import getpass\n",
-    "from langchain.document_loaders import PyPDFLoader, TextLoader\n",
+    "import os\n",
+    "\n",
+    "from langchain.chains import RetrievalQA\n",
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.text_splitter import (\n",
-    "    RecursiveCharacterTextSplitter,\n",
     "    CharacterTextSplitter,\n",
+    "    RecursiveCharacterTextSplitter,\n",
     ")\n",
     "from langchain.vectorstores import DeepLake\n",
-    "from langchain.chains import ConversationalRetrievalChain, RetrievalQA\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.llms import OpenAI\n",
     "\n",
     "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
     "activeloop_token = getpass.getpass(\"Activeloop Token:\")\n",

cookbook/elasticsearch_db_qa.ipynb

@@ -38,9 +38,8 @@
    "outputs": [],
    "source": [
     "from elasticsearch import Elasticsearch\n",
-    "\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.chains.elasticsearch_database import ElasticsearchDatabaseChain"
+    "from langchain.chains.elasticsearch_database import ElasticsearchDatabaseChain\n",
+    "from langchain.chat_models import ChatOpenAI"
    ]
   },
   {
@@ -112,7 +111,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.elasticsearch_database.prompts import DEFAULT_DSL_TEMPLATE\n",
     "from langchain.prompts.prompt import PromptTemplate\n",
     "\n",
     "PROMPT_TEMPLATE = \"\"\"Given an input question, create a syntactically correct Elasticsearch query to run. Unless the user specifies in their question a specific number of examples they wish to obtain, always limit your query to at most {top_k} results. You can order the results by a relevant column to return the most interesting examples in the database.\n",

cookbook/extraction_openai_tools.ipynb

@@ -19,10 +19,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from typing import List, Optional\n",
+    "\n",
+    "from langchain.chains.openai_tools import create_extraction_chain_pydantic\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.pydantic_v1 import BaseModel\n",
-    "from typing import Optional, List\n",
-    "from langchain.chains.openai_tools import create_extraction_chain_pydantic"
+    "from langchain.pydantic_v1 import BaseModel"
    ]
   },
   {

cookbook/fake_llm.ipynb

@@ -30,9 +30,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import load_tools\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents import AgentType"
+    "from langchain.agents import AgentType, initialize_agent, load_tools"
    ]
   },
   {

cookbook/forward_looking_retrieval_augmented_generation.ipynb

@@ -56,7 +56,8 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"SERPER_API_KEY\"] = \"\"os.environ[\"OPENAI_API_KEY\"] = \"\""
+    "os.environ[\"SERPER_API_KEY\"] = \"\"\n",
+    "os.environ[\"OPENAI_API_KEY\"] = \"\""
    ]
   },
   {
@@ -66,21 +67,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re\n",
+    "from typing import Any, List\n",
     "\n",
-    "import numpy as np\n",
-    "\n",
-    "from langchain.schema import BaseRetriever\n",
     "from langchain.callbacks.manager import (\n",
     "    AsyncCallbackManagerForRetrieverRun,\n",
     "    CallbackManagerForRetrieverRun,\n",
     ")\n",
-    "from langchain.utilities import GoogleSerperAPIWrapper\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.llms import OpenAI\n",
-    "from langchain.schema import Document\n",
-    "from typing import Any, List"
+    "from langchain.schema import BaseRetriever, Document\n",
+    "from langchain.utilities import GoogleSerperAPIWrapper"
    ]
   },
   {

cookbook/generative_agents_interactive_simulacra_of_human_behavior.ipynb

@@ -46,14 +46,13 @@
    "source": [
     "from datetime import datetime, timedelta\n",
     "from typing import List\n",
-    "from termcolor import colored\n",
-    "\n",
     "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.docstore import InMemoryDocstore\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers import TimeWeightedVectorStoreRetriever\n",
-    "from langchain.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS\n",
+    "from termcolor import colored"
    ]
   },
   {
@@ -153,6 +152,7 @@
    "outputs": [],
    "source": [
     "import math\n",
+    "\n",
     "import faiss\n",
     "\n",
     "\n",

cookbook/gymnasium_agent_simulation.ipynb

@@ -27,18 +27,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import gymnasium as gym\n",
-    "import inspect\n",
     "import tenacity\n",
-    "\n",
-    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.output_parsers import RegexParser\n",
     "from langchain.schema import (\n",
-    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    "    BaseMessage,\n",
-    ")\n",
-    "from langchain.output_parsers import RegexParser"
+    ")"
    ]
   },
   {
@@ -131,7 +125,7 @@
     "            ):\n",
     "                with attempt:\n",
     "                    action = self._act()\n",
-    "        except tenacity.RetryError as e:\n",
+    "        except tenacity.RetryError:\n",
     "            action = self.random_action()\n",
     "        return action"
    ]

cookbook/human_input_chat_model.ipynb

@@ -55,9 +55,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import load_tools\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents import AgentType"
+    "from langchain.agents import AgentType, initialize_agent, load_tools"
    ]
   },
   {

cookbook/human_input_llm.ipynb

@@ -28,9 +28,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import load_tools\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents import AgentType"
+    "from langchain.agents import AgentType, initialize_agent, load_tools"
    ]
   },
   {

cookbook/hypothetical_document_embeddings.ipynb

@@ -20,9 +20,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import OpenAI\n",
+    "from langchain.chains import HypotheticalDocumentEmbedder, LLMChain\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.chains import LLMChain, HypotheticalDocumentEmbedder\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import PromptTemplate"
    ]
   },

cookbook/learned_prompt_optimization.ipynb

@@ -790,8 +790,8 @@
     }
    ],
    "source": [
-    "from langchain.prompts.prompt import PromptTemplate\n",
     "from langchain.globals import set_debug\n",
+    "from langchain.prompts.prompt import PromptTemplate\n",
     "\n",
     "set_debug(True)\n",
     "\n",

cookbook/llm_bash.ipynb

@@ -43,8 +43,8 @@
     }
    ],
    "source": [
-    "from langchain_experimental.llm_bash.base import LLMBashChain\n",
     "from langchain.llms import OpenAI\n",
+    "from langchain_experimental.llm_bash.base import LLMBashChain\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "\n",
@@ -69,8 +69,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts.prompt import PromptTemplate\n",
     "from langchain.chains.llm_bash.prompt import BashOutputParser\n",
+    "from langchain.prompts.prompt import PromptTemplate\n",
     "\n",
     "_PROMPT_TEMPLATE = \"\"\"If someone asks you to perform a task, your job is to come up with a series of bash commands that will perform the task. There is no need to put \"#!/bin/bash\" in your answer. Make sure to reason step by step, using this format:\n",
     "Question: \"copy the files in the directory named 'target' into a new directory at the same level as target called 'myNewDirectory'\"\n",
@@ -185,7 +185,6 @@
    "source": [
     "from langchain_experimental.llm_bash.bash import BashProcess\n",
     "\n",
-    "\n",
     "persistent_process = BashProcess(persistent=True)\n",
     "bash_chain = LLMBashChain.from_llm(llm, bash_process=persistent_process, verbose=True)\n",
     "\n",

cookbook/llm_math.ipynb

@@ -45,7 +45,8 @@
     }
    ],
    "source": [
-    "from langchain.llms import OpenAI\nfrom langchain.chains import LLMMathChain\n",
+    "from langchain.chains import LLMMathChain\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "llm_math = LLMMathChain.from_llm(llm, verbose=True)\n",

cookbook/meta_prompt.ipynb

@@ -56,8 +56,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\n",
-    "from langchain.memory import ConversationBufferWindowMemory"
+    "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.memory import ConversationBufferWindowMemory\n",
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {
@@ -152,13 +154,13 @@
     "        for j in range(max_iters):\n",
     "            print(f\"(Step {j+1}/{max_iters})\")\n",
     "            print(f\"Assistant: {output}\")\n",
-    "            print(f\"Human: \")\n",
+    "            print(\"Human: \")\n",
     "            human_input = input()\n",
     "            if any(phrase in human_input.lower() for phrase in key_phrases):\n",
     "                break\n",
     "            output = chain.predict(human_input=human_input)\n",
     "        if success_phrase in human_input.lower():\n",
-    "            print(f\"You succeeded! Thanks for playing!\")\n",
+    "            print(\"You succeeded! Thanks for playing!\")\n",
     "            return\n",
     "        meta_chain = initialize_meta_chain()\n",
     "        meta_output = meta_chain.predict(chat_history=get_chat_history(chain.memory))\n",
@@ -166,7 +168,7 @@
     "        instructions = get_new_instructions(meta_output)\n",
     "        print(f\"New Instructions: {instructions}\")\n",
     "        print(\"\\n\" + \"#\" * 80 + \"\\n\")\n",
-    "    print(f\"You failed! Thanks for playing!\")"
+    "    print(\"You failed! Thanks for playing!\")"
    ]
   },
   {

cookbook/multi_modal_QA.ipynb

@@ -40,12 +40,13 @@
     }
    ],
    "source": [
-    "import os\n",
-    "import io\n",
     "import base64\n",
+    "import io\n",
+    "import os\n",
+    "\n",
     "import numpy as np\n",
+    "from IPython.display import HTML, display\n",
     "from PIL import Image\n",
-    "from IPython.display import display, HTML\n",
     "\n",
     "\n",
     "def encode_image(image_path):\n",

cookbook/multi_modal_RAG_chroma.ipynb

@@ -115,7 +115,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Folder with pdf and extracted images \n",
+    "# Folder with pdf and extracted images\n",
     "path = \"/Users/rlm/Desktop/photos/\""
    ]
   },
@@ -128,9 +128,10 @@
    "source": [
     "# Extract images, tables, and chunk text\n",
     "from unstructured.partition.pdf import partition_pdf\n",
+    "\n",
     "raw_pdf_elements = partition_pdf(\n",
     "    filename=path + \"photos.pdf\",\n",
-    "    extract_images_in_pdf=True, \n",
+    "    extract_images_in_pdf=True,\n",
     "    infer_table_structure=True,\n",
     "    chunking_strategy=\"by_title\",\n",
     "    max_characters=4000,\n",
@@ -183,22 +184,26 @@
    "source": [
     "import os\n",
     "import uuid\n",
+    "\n",
     "import chromadb\n",
     "import numpy as np\n",
-    "from PIL import Image as _PILImage\n",
     "from langchain.vectorstores import Chroma\n",
     "from langchain_experimental.open_clip import OpenCLIPEmbeddings\n",
+    "from PIL import Image as _PILImage\n",
     "\n",
     "# Create chroma\n",
     "vectorstore = Chroma(\n",
-    "    collection_name=\"mm_rag_clip_photos\",\n",
-    "    embedding_function=OpenCLIPEmbeddings()\n",
+    "    collection_name=\"mm_rag_clip_photos\", embedding_function=OpenCLIPEmbeddings()\n",
     ")\n",
     "\n",
     "# Get image URIs with .jpg extension only\n",
-    "image_uris = sorted([os.path.join(path, image_name) \n",
-    "                     for image_name in os.listdir(path) \n",
-    "                     if image_name.endswith('.jpg')])\n",
+    "image_uris = sorted(\n",
+    "    [\n",
+    "        os.path.join(path, image_name)\n",
+    "        for image_name in os.listdir(path)\n",
+    "        if image_name.endswith(\".jpg\")\n",
+    "    ]\n",
+    ")\n",
     "\n",
     "# Add images\n",
     "vectorstore.add_images(uris=image_uris)\n",
@@ -206,7 +211,7 @@
     "# Add documents\n",
     "vectorstore.add_texts(texts=texts)\n",
     "\n",
-    "# Make retriever \n",
+    "# Make retriever\n",
     "retriever = vectorstore.as_retriever()"
    ]
   },
@@ -229,12 +234,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import io\n",
-    "import numpy as np\n",
     "import base64\n",
+    "import io\n",
     "from io import BytesIO\n",
+    "\n",
+    "import numpy as np\n",
     "from PIL import Image\n",
     "\n",
+    "\n",
     "def resize_base64_image(base64_string, size=(128, 128)):\n",
     "    \"\"\"\n",
     "    Resize an image encoded as a Base64 string.\n",
@@ -258,30 +265,31 @@
     "    resized_img.save(buffered, format=img.format)\n",
     "\n",
     "    # Encode the resized image to Base64\n",
-    "    return base64.b64encode(buffered.getvalue()).decode('utf-8')\n",
+    "    return base64.b64encode(buffered.getvalue()).decode(\"utf-8\")\n",
+    "\n",
     "\n",
     "def is_base64(s):\n",
-    "    ''' Check if a string is Base64 encoded '''\n",
+    "    \"\"\"Check if a string is Base64 encoded\"\"\"\n",
     "    try:\n",
     "        return base64.b64encode(base64.b64decode(s)) == s.encode()\n",
     "    except Exception:\n",
     "        return False\n",
-    "        \n",
+    "\n",
+    "\n",
     "def split_image_text_types(docs):\n",
-    "    ''' Split numpy array images and texts '''\n",
+    "    \"\"\"Split numpy array images and texts\"\"\"\n",
     "    images = []\n",
     "    text = []\n",
     "    for doc in docs:\n",
-    "        doc = doc.page_content # Extract Document contents \n",
+    "        doc = doc.page_content  # Extract Document contents\n",
     "        if is_base64(doc):\n",
     "            # Resize image to avoid OAI server error\n",
-    "            images.append(resize_base64_image(doc, size=(250, 250)))  # base64 encoded str \n",
+    "            images.append(\n",
+    "                resize_base64_image(doc, size=(250, 250))\n",
+    "            )  # base64 encoded str\n",
     "        else:\n",
-    "            text.append(doc) \n",
-    "    return {\n",
-    "        \"images\": images,\n",
-    "        \"texts\": text\n",
-    "    }"
+    "            text.append(doc)\n",
+    "    return {\"images\": images, \"texts\": text}"
    ]
   },
   {
@@ -306,10 +314,12 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
     "from langchain.schema.messages import HumanMessage, SystemMessage\n",
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
+    "\n",
     "\n",
     "def prompt_func(data_dict):\n",
     "    # Joining the context texts into a single string\n",
@@ -322,7 +332,7 @@
     "            \"type\": \"image_url\",\n",
     "            \"image_url\": {\n",
     "                \"url\": f\"data:image/jpeg;base64,{data_dict['context']['images'][0]}\"\n",
-    "            }\n",
+    "            },\n",
     "        }\n",
     "        messages.append(image_message)\n",
     "\n",
@@ -342,17 +352,21 @@
     "            f\"User-provided keywords: {data_dict['question']}\\n\\n\"\n",
     "            \"Text and / or tables:\\n\"\n",
     "            f\"{formatted_texts}\"\n",
-    "        )\n",
+    "        ),\n",
     "    }\n",
     "    messages.append(text_message)\n",
     "\n",
     "    return [HumanMessage(content=messages)]\n",
-    "    \n",
+    "\n",
+    "\n",
     "model = ChatOpenAI(temperature=0, model=\"gpt-4-vision-preview\", max_tokens=1024)\n",
     "\n",
     "# RAG pipeline\n",
     "chain = (\n",
-    "    {\"context\": retriever | RunnableLambda(split_image_text_types), \"question\": RunnablePassthrough()}\n",
+    "    {\n",
+    "        \"context\": retriever | RunnableLambda(split_image_text_types),\n",
+    "        \"question\": RunnablePassthrough(),\n",
+    "    }\n",
     "    | RunnableLambda(prompt_func)\n",
     "    | model\n",
     "    | StrOutputParser()\n",
@@ -410,17 +424,18 @@
     }
    ],
    "source": [
-    "from IPython.display import display, HTML\n",
+    "from IPython.display import HTML, display\n",
+    "\n",
     "\n",
     "def plt_img_base64(img_base64):\n",
-    "\n",
     "    # Create an HTML img tag with the base64 string as the source\n",
     "    image_html = f'<img src=\"data:image/jpeg;base64,{img_base64}\" />'\n",
-    "    \n",
+    "\n",
     "    # Display the image by rendering the HTML\n",
     "    display(HTML(image_html))\n",
     "\n",
-    "docs = retriever.get_relevant_documents(\"Woman with children\",k=10)\n",
+    "\n",
+    "docs = retriever.get_relevant_documents(\"Woman with children\", k=10)\n",
     "for doc in docs:\n",
     "    if is_base64(doc.page_content):\n",
     "        plt_img_base64(doc.page_content)\n",
@@ -446,9 +461,7 @@
     }
    ],
    "source": [
-    "chain.invoke(\n",
-    "    \"Woman with children\"\n",
-    ")"
+    "chain.invoke(\"Woman with children\")"
    ]
   },
   {

cookbook/multi_modal_output_agent.ipynb

@@ -29,9 +29,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from steamship import Block, Steamship\n",
     "import re\n",
-    "from IPython.display import Image"
+    "\n",
+    "from IPython.display import Image\n",
+    "from steamship import Block, Steamship"
    ]
   },
   {
@@ -41,9 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.agents import AgentType, initialize_agent\n",
     "from langchain.llms import OpenAI\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents import AgentType\n",
     "from langchain.tools import SteamshipImageGenerationTool"
    ]
   },

cookbook/multi_player_dnd.ipynb

@@ -26,13 +26,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import List, Dict, Callable\n",
+    "from typing import Callable, List\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
-    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/multiagent_authoritarian.ipynb

@@ -27,26 +27,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from collections import OrderedDict\n",
     "import functools\n",
     "import random\n",
-    "import re\n",
-    "import tenacity\n",
-    "from typing import List, Dict, Callable\n",
+    "from collections import OrderedDict\n",
+    "from typing import Callable, List\n",
     "\n",
-    "from langchain.prompts import (\n",
-    "    ChatPromptTemplate,\n",
-    "    HumanMessagePromptTemplate,\n",
-    "    PromptTemplate,\n",
-    ")\n",
-    "from langchain.chains import LLMChain\n",
+    "import tenacity\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.output_parsers import RegexParser\n",
+    "from langchain.prompts import (\n",
+    "    PromptTemplate,\n",
+    ")\n",
     "from langchain.schema import (\n",
-    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/multiagent_bidding.ipynb

@@ -24,17 +24,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "import re\n",
+    "from typing import Callable, List\n",
+    "\n",
     "import tenacity\n",
-    "from typing import List, Dict, Callable\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.output_parsers import RegexParser\n",
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain.schema import (\n",
-    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/myscale_vector_sql.ipynb

@@ -27,18 +27,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from os import environ\n",
     "import getpass\n",
-    "from typing import Dict, Any\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.utilities import SQLDatabase\n",
+    "from os import environ\n",
+    "\n",
     "from langchain.chains import LLMChain\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
-    "from sqlalchemy import create_engine, Column, MetaData\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "\n",
-    "from sqlalchemy import create_engine\n",
+    "from langchain.utilities import SQLDatabase\n",
+    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
+    "from sqlalchemy import MetaData, create_engine\n",
     "\n",
     "MYSCALE_HOST = \"msc-4a9e710a.us-east-1.aws.staging.myscale.cloud\"\n",
     "MYSCALE_PORT = 443\n",
@@ -77,9 +74,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import OpenAI\n",
     "from langchain.callbacks import StdOutCallbackHandler\n",
-    "\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.utilities.sql_database import SQLDatabase\n",
     "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
     "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
@@ -120,15 +116,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.chains.qa_with_sources.retrieval import RetrievalQAWithSourcesChain\n",
-    "\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain_experimental.retrievers.vector_sql_database import (\n",
     "    VectorSQLDatabaseChainRetriever,\n",
     ")\n",
     "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLRetrieveAllOutputParser\n",
+    "from langchain_experimental.sql.vector_sql import (\n",
+    "    VectorSQLDatabaseChain,\n",
+    "    VectorSQLRetrieveAllOutputParser,\n",
+    ")\n",
     "\n",
     "output_parser_retrieve_all = VectorSQLRetrieveAllOutputParser.from_embeddings(\n",
     "    output_parser.model\n",

cookbook/openai_functions_retrieval_qa.ipynb

@@ -50,10 +50,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains import create_qa_with_sources_chain\n",
     "from langchain.chains.combine_documents.stuff import StuffDocumentsChain\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chains import create_qa_with_sources_chain"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {
@@ -230,9 +230,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import ConversationalRetrievalChain\n",
+    "from langchain.chains import ConversationalRetrievalChain, LLMChain\n",
     "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.chains import LLMChain\n",
     "\n",
     "memory = ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True)\n",
     "_template = \"\"\"Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.\\\n",
@@ -357,12 +356,10 @@
    "source": [
     "from typing import List\n",
     "\n",
-    "from pydantic import BaseModel, Field\n",
-    "\n",
     "from langchain.chains.openai_functions import create_qa_with_structure_chain\n",
-    "\n",
     "from langchain.prompts.chat import ChatPromptTemplate, HumanMessagePromptTemplate\n",
-    "from langchain.schema import SystemMessage, HumanMessage"
+    "from langchain.schema import HumanMessage, SystemMessage\n",
+    "from pydantic import BaseModel, Field"
    ]
   },
   {

cookbook/openai_v1_cookbook.ipynb

@@ -167,7 +167,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.tools import E2BDataAnalysisTool, DuckDuckGoSearchRun\n",
+    "from langchain.tools import DuckDuckGoSearchRun, E2BDataAnalysisTool\n",
     "\n",
     "tools = [E2BDataAnalysisTool(api_key=\"...\"), DuckDuckGoSearchRun()]"
    ]
@@ -419,7 +419,7 @@
     "\n",
     "\n",
     "RECOMMENDED CHANGES:\n",
-    "- When using AzureChatOpenAI, if passing in an Azure endpoint (eg https://example-resource.azure.openai.com/) this should be specified via the `azure_endpoint` parameter or the `AZURE_OPENAI_ENDPOINT`. We're maintaining backwards compatibility for now with specifying this via `openai_api_base`/`base_url` or env var `OPENAI_API_BASE` but this shouldn't be relied upon.\n",
+    "- When using `AzureChatOpenAI` or `AzureOpenAI`, if passing in an Azure endpoint (eg https://example-resource.azure.openai.com/) this should be specified via the `azure_endpoint` parameter or the `AZURE_OPENAI_ENDPOINT`. We're maintaining backwards compatibility for now with specifying this via `openai_api_base`/`base_url` or env var `OPENAI_API_BASE` but this shouldn't be relied upon.\n",
     "- When using Azure chat or embedding models, pass in API keys either via `openai_api_key` parameter or `AZURE_OPENAI_API_KEY` parameter. We're maintaining backwards compatibility for now with specifying this via `OPENAI_API_KEY` but this shouldn't be relied upon."
    ]
   },
@@ -456,9 +456,9 @@
     "from typing import Literal\n",
     "\n",
     "from langchain.output_parsers.openai_tools import PydanticToolsParser\n",
-    "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
     "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.pydantic_v1 import BaseModel, Field\n",
+    "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
     "\n",
     "\n",
     "class GetCurrentWeather(BaseModel):\n",

cookbook/petting_zoo.ipynb

@@ -45,14 +45,14 @@
    "source": [
     "import collections\n",
     "import inspect\n",
-    "import tenacity\n",
     "\n",
+    "import tenacity\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.output_parsers import RegexParser\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain.output_parsers import RegexParser"
+    ")"
    ]
   },
   {
@@ -146,7 +146,7 @@
     "            ):\n",
     "                with attempt:\n",
     "                    action = self._act()\n",
-    "        except tenacity.RetryError as e:\n",
+    "        except tenacity.RetryError:\n",
     "            action = self.random_action()\n",
     "        return action"
    ]

cookbook/program_aided_language_model.ipynb

@@ -17,8 +17,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_experimental.pal_chain import PALChain\n",
-    "from langchain.llms import OpenAI"
+    "from langchain.llms import OpenAI\n",
+    "from langchain_experimental.pal_chain import PALChain"
    ]
   },
   {

cookbook/qianfan_baidu_elasticesearch_RAG.ipynb

@@ -54,13 +54,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from baidubce.bce_client_configuration import BceClientConfiguration\n",
     "from baidubce.auth.bce_credentials import BceCredentials\n",
+    "from baidubce.bce_client_configuration import BceClientConfiguration\n",
     "from langchain.document_loaders.baiducloud_bos_directory import BaiduBOSDirectoryLoader\n",
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
     "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
-    "from langchain.vectorstores import BESVectorStore\n",
-    "from langchain.llms.baidu_qianfan_endpoint import QianfanLLMEndpoint"
+    "from langchain.llms.baidu_qianfan_endpoint import QianfanLLMEndpoint\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "from langchain.vectorstores import BESVectorStore"
    ]
   },
   {
@@ -82,7 +82,9 @@
     "secret_access_key = \"your bos access sk\"\n",
     "\n",
     "# create BceClientConfiguration\n",
-    "config = BceClientConfiguration(credentials=BceCredentials(access_key_id, secret_access_key), endpoint = bos_host)\n",
+    "config = BceClientConfiguration(\n",
+    "    credentials=BceCredentials(access_key_id, secret_access_key), endpoint=bos_host\n",
+    ")\n",
     "\n",
     "loader = BaiduBOSDirectoryLoader(conf=config, bucket=\"llm-test\", prefix=\"llm/\")\n",
     "documents = loader.load()\n",
@@ -109,10 +111,14 @@
     "embeddings.client = sentence_transformers.SentenceTransformer(embeddings.model_name)\n",
     "\n",
     "db = BESVectorStore.from_documents(\n",
-    "  documents=split_docs, embedding=embeddings, bes_url=\"your bes url\", index_name='test-index', vector_query_field='vector'\n",
-    " )\n",
+    "    documents=split_docs,\n",
+    "    embedding=embeddings,\n",
+    "    bes_url=\"your bes url\",\n",
+    "    index_name=\"test-index\",\n",
+    "    vector_query_field=\"vector\",\n",
+    ")\n",
     "\n",
-    "db.client.indices.refresh(index='test-index')\n",
+    "db.client.indices.refresh(index=\"test-index\")\n",
     "retriever = db.as_retriever()"
    ]
   },
@@ -130,8 +136,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "llm = QianfanLLMEndpoint(model=\"ERNIE-Bot\", qianfan_ak='your qianfan ak', qianfan_sk='your qianfan sk', streaming=True)\n",
-    "qa = RetrievalQA.from_chain_type(llm=llm, chain_type=\"refine\", retriever=retriever, return_source_documents=True)\n",
+    "llm = QianfanLLMEndpoint(\n",
+    "    model=\"ERNIE-Bot\",\n",
+    "    qianfan_ak=\"your qianfan ak\",\n",
+    "    qianfan_sk=\"your qianfan sk\",\n",
+    "    streaming=True,\n",
+    ")\n",
+    "qa = RetrievalQA.from_chain_type(\n",
+    "    llm=llm, chain_type=\"refine\", retriever=retriever, return_source_documents=True\n",
+    ")\n",
     "\n",
     "query = \"什么是张量?\"\n",
     "print(qa.run(query))"

cookbook/rag_fusion.ipynb

@@ -30,8 +30,8 @@
    "outputs": [],
    "source": [
     "import pinecone\n",
-    "from langchain.vectorstores import Pinecone\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import Pinecone\n",
     "\n",
     "pinecone.init(api_key=\"...\", environment=\"...\")"
    ]
@@ -87,7 +87,6 @@
    "outputs": [],
    "source": [
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser"
    ]
   },

cookbook/retrieval_in_sql.ipynb

@@ -28,8 +28,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
     "import getpass\n",
+    "import os\n",
     "\n",
     "os.environ[\"OPENAI_API_KEY\"] = os.environ.get(\"OPENAI_API_KEY\") or getpass.getpass(\n",
     "    \"OpenAI API Key:\"\n",
@@ -42,8 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.sql_database import SQLDatabase\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.sql_database import SQLDatabase\n",
     "\n",
     "CONNECTION_STRING = \"postgresql+psycopg2://postgres:test@localhost:5432/vectordb\"  # Replace with your own\n",
     "db = SQLDatabase.from_uri(CONNECTION_STRING)"
@@ -323,6 +323,7 @@
    "outputs": [],
    "source": [
     "import re\n",
+    "\n",
     "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
     "\n",

cookbook/rewrite.ipynb

@@ -31,12 +31,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from operator import itemgetter\n",
-    "\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "from langchain.utilities import DuckDuckGoSearchAPIWrapper"
    ]
   },

cookbook/sales_agent_with_context.ipynb

@@ -42,22 +42,22 @@
     "OPENAI_API_KEY = \"sk-xx\"\n",
     "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY\n",
     "\n",
-    "from typing import Dict, List, Any, Union, Callable\n",
-    "from pydantic import BaseModel, Field\n",
-    "from langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\n",
-    "from langchain.llms import BaseLLM\n",
-    "from langchain.chains.base import Chain\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.agents import Tool, LLMSingleActionAgent, AgentExecutor\n",
-    "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain.chains import RetrievalQA\n",
-    "from langchain.vectorstores import Chroma\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts.base import StringPromptTemplate\n",
+    "from typing import Any, Callable, Dict, List, Union\n",
+    "\n",
+    "from langchain.agents import AgentExecutor, LLMSingleActionAgent, Tool\n",
     "from langchain.agents.agent import AgentOutputParser\n",
     "from langchain.agents.conversational.prompt import FORMAT_INSTRUCTIONS\n",
-    "from langchain.schema import AgentAction, AgentFinish"
+    "from langchain.chains import LLMChain, RetrievalQA\n",
+    "from langchain.chains.base import Chain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.llms import BaseLLM, OpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.prompts.base import StringPromptTemplate\n",
+    "from langchain.schema import AgentAction, AgentFinish\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.vectorstores import Chroma\n",
+    "from pydantic import BaseModel, Field"
    ]
   },
   {

cookbook/selecting_llms_based_on_context_length.ipynb

@@ -17,12 +17,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.schema.prompt import PromptValue\n",
-    "from langchain.schema.messages import BaseMessage\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from typing import Union, Sequence"
+    "from langchain.schema.prompt import PromptValue"
    ]
   },
   {

cookbook/self_query_hotel_search.ipynb

@@ -1084,7 +1084,6 @@
    "outputs": [],
    "source": [
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema import Document\n",
     "from langchain.vectorstores import ElasticsearchStore\n",
     "\n",
     "embeddings = OpenAIEmbeddings()"

cookbook/smart_llm.ipynb

@@ -51,8 +51,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain_experimental.smart_llm import SmartLLMChain"
    ]
   },

cookbook/stepback-qa.ipynb

@@ -131,7 +131,6 @@
    "source": [
     "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
     "\n",
-    "\n",
     "search = DuckDuckGoSearchAPIWrapper(max_results=4)\n",
     "\n",
     "\n",

cookbook/tree_of_thought.ipynb

@@ -84,10 +84,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import re\n",
     "from typing import Tuple\n",
+    "\n",
     "from langchain_experimental.tot.checker import ToTChecker\n",
     "from langchain_experimental.tot.thought import ThoughtValidity\n",
-    "import re\n",
     "\n",
     "\n",
     "class MyChecker(ToTChecker):\n",

cookbook/twitter-the-algorithm-analysis-deeplake.ipynb

@@ -34,8 +34,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
     "import getpass\n",
+    "import os\n",
     "\n",
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "from langchain.vectorstores import DeepLake\n",
@@ -109,6 +109,7 @@
    "outputs": [],
    "source": [
     "import os\n",
+    "\n",
     "from langchain.document_loaders import TextLoader\n",
     "\n",
     "root_dir = \"./the-algorithm\"\n",
@@ -118,7 +119,7 @@
     "        try:\n",
     "            loader = TextLoader(os.path.join(dirpath, file), encoding=\"utf-8\")\n",
     "            docs.extend(loader.load_and_split())\n",
-    "        except Exception as e:\n",
+    "        except Exception:\n",
     "            pass"
    ]
   },
@@ -3807,8 +3808,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.chains import ConversationalRetrievalChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "model = ChatOpenAI(model_name=\"gpt-3.5-turbo-0613\")  # switch to 'gpt-4'\n",
     "qa = ConversationalRetrievalChain.from_llm(model, retriever=retriever)"

cookbook/two_agent_debate_tools.ipynb

@@ -22,17 +22,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import List, Dict, Callable\n",
-    "from langchain.chains import ConversationChain\n",
+    "from typing import Callable, List\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.llms import OpenAI\n",
     "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.prompts.prompt import PromptTemplate\n",
     "from langchain.schema import (\n",
     "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    "    BaseMessage,\n",
     ")"
    ]
   },
@@ -49,10 +46,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import Tool\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents import AgentType\n",
-    "from langchain.agents import load_tools"
+    "from langchain.agents import AgentType, initialize_agent, load_tools"
    ]
   },
   {

cookbook/two_player_dnd.ipynb

@@ -22,7 +22,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import List, Dict, Callable\n",
+    "from typing import Callable, List\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",

cookbook/wikibase_agent.ipynb

@@ -192,10 +192,10 @@
     "    return current\n",
     "\n",
     "\n",
-    "import requests\n",
-    "\n",
     "from typing import Optional\n",
     "\n",
+    "import requests\n",
+    "\n",
     "\n",
     "def vocab_lookup(\n",
     "    search: str,\n",
@@ -319,9 +319,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import requests\n",
-    "from typing import List, Dict, Any\n",
     "import json\n",
+    "from typing import Any, Dict, List\n",
+    "\n",
+    "import requests\n",
     "\n",
     "\n",
     "def run_sparql(\n",
@@ -389,17 +390,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import (\n",
-    "    Tool,\n",
-    "    AgentExecutor,\n",
-    "    LLMSingleActionAgent,\n",
-    "    AgentOutputParser,\n",
-    ")\n",
-    "from langchain.prompts import StringPromptTemplate\n",
-    "from langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\n",
+    "import re\n",
     "from typing import List, Union\n",
-    "from langchain.schema import AgentAction, AgentFinish\n",
-    "import re"
+    "\n",
+    "from langchain.agents import (\n",
+    "    AgentExecutor,\n",
+    "    AgentOutputParser,\n",
+    "    LLMSingleActionAgent,\n",
+    "    Tool,\n",
+    ")\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.prompts import StringPromptTemplate\n",
+    "from langchain.schema import AgentAction, AgentFinish"
    ]
   },
   {

docs/api_reference/_static/css/custom.css

@@ -15,3 +15,11 @@ pre {
 #my-component-root *, #headlessui-portal-root * {
   z-index: 10000;
 }
+
+table.longtable code {
+  white-space: normal;
+}
+
+table.longtable td {
+  max-width: 600px;
+}

docs/docs/expression_language/cookbook/agent.ipynb

@@ -17,7 +17,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import XMLAgent, tool, AgentExecutor\n",
+    "from langchain.agents import AgentExecutor, XMLAgent, tool\n",
     "from langchain.chat_models import ChatAnthropic"
    ]
   },

docs/docs/expression_language/cookbook/code_writing.ipynb

@@ -20,8 +20,6 @@
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import (\n",
     "    ChatPromptTemplate,\n",
-    "    SystemMessagePromptTemplate,\n",
-    "    HumanMessagePromptTemplate,\n",
     ")\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
     "from langchain_experimental.utilities import PythonREPL"

docs/docs/expression_language/cookbook/embedding_router.ipynb

@@ -26,7 +26,6 @@
     "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
     "from langchain.utils.math import cosine_similarity\n",
     "\n",
-    "\n",
     "physics_template = \"\"\"You are a very smart physics professor. \\\n",
     "You are great at answering questions about physics in a concise and easy to understand manner. \\\n",
     "When you don't know the answer to a question you admit that you don't know.\n",

docs/docs/expression_language/cookbook/memory.ipynb

@@ -18,10 +18,11 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
     "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
+    "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
     "\n",
     "model = ChatOpenAI()\n",
     "prompt = ChatPromptTemplate.from_messages(\n",

docs/docs/expression_language/cookbook/multiple_chains.ipynb

@@ -69,7 +69,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.schema.runnable import RunnableMap, RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "prompt1 = ChatPromptTemplate.from_template(\n",
     "    \"generate a {attribute} color. Return the name of the color and nothing else:\"\n",

docs/docs/expression_language/cookbook/prompt_llm_parser.ipynb

@@ -42,8 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
     "\n",
     "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {foo}\")\n",
     "model = ChatOpenAI()\n",

docs/docs/expression_language/cookbook/retrieval.ipynb

@@ -38,11 +38,11 @@
    "source": [
     "from operator import itemgetter\n",
     "\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
+    "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
     "from langchain.vectorstores import FAISS"
    ]
   },
@@ -170,8 +170,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.schema.runnable import RunnableMap\n",
-    "from langchain.schema import format_document"
+    "from langchain.schema import format_document\n",
+    "from langchain.schema.runnable import RunnableMap"
    ]
   },
   {
@@ -231,7 +231,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Tuple, List\n",
+    "from typing import List, Tuple\n",
     "\n",
     "\n",
     "def _format_chat_history(chat_history: List[Tuple]) -> str:\n",
@@ -335,6 +335,7 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
+    "\n",
     "from langchain.memory import ConversationBufferMemory"
    ]
   },

docs/docs/expression_language/how_to/configure.ipynb

@@ -262,9 +262,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic\n",
-    "from langchain.schema.runnable import ConfigurableField\n",
-    "from langchain.prompts import PromptTemplate"
+    "from langchain.chat_models import ChatAnthropic, ChatOpenAI\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.schema.runnable import ConfigurableField"
    ]
   },
   {

docs/docs/expression_language/how_to/fallbacks.ipynb

@@ -31,7 +31,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic"
+    "from langchain.chat_models import ChatAnthropic, ChatOpenAI"
    ]
   },
   {
@@ -50,6 +50,7 @@
    "outputs": [],
    "source": [
     "from unittest.mock import patch\n",
+    "\n",
     "from openai.error import RateLimitError"
    ]
   },

docs/docs/expression_language/how_to/functions.ipynb

@@ -19,11 +19,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.schema.runnable import RunnableLambda\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
     "from operator import itemgetter\n",
     "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "from langchain.schema.runnable import RunnableLambda\n",
+    "\n",
     "\n",
     "def length_function(text):\n",
     "    return len(text)\n",
@@ -91,8 +92,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.schema.runnable import RunnableConfig\n",
-    "from langchain.schema.output_parser import StrOutputParser"
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnableConfig"
    ]
   },
   {

docs/docs/expression_language/how_to/generators.ipynb

@@ -29,7 +29,6 @@
     "from langchain.prompts.chat import ChatPromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
     "\n",
-    "\n",
     "prompt = ChatPromptTemplate.from_template(\n",
     "    \"Write a comma-separated list of 5 animals similar to: {animal}\"\n",
     ")\n",

docs/docs/expression_language/how_to/map.ipynb

@@ -33,7 +33,6 @@
     "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.schema.runnable import RunnableParallel\n",
     "\n",
-    "\n",
     "model = ChatOpenAI()\n",
     "joke_chain = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\") | model\n",
     "poem_chain = (\n",

docs/docs/expression_language/how_to/message_history.ipynb

@@ -0,0 +1,396 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "6a4becbd-238e-4c1d-a02d-08e61fbc3763",
+   "metadata": {},
+   "source": [
+    "# Add message history (memory)\n",
+    "\n",
+    "The `RunnableWithMessageHistory` let's us add message history to certain types of chains.\n",
+    "\n",
+    "Specifically, it can be used for any Runnable that takes as input one of\n",
+    "* a sequence of `BaseMessage`\n",
+    "* a dict with a key that takes a sequence of `BaseMessage`\n",
+    "* a dict with a key that takes the latest message(s) as a string or sequence of `BaseMessage`, and a separate key that takes historical messages\n",
+    "\n",
+    "And returns as output one of\n",
+    "* a string that can be treated as the contents of an `AIMessage`\n",
+    "* a sequence of `BaseMessage`\n",
+    "* a dict with a key that contains a sequence of `BaseMessage`\n",
+    "\n",
+    "Let's take a look at some examples to see how it works."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6bca45e5-35d9-4603-9ca9-6ac0ce0e35cd",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "We'll use Redis to store our chat message histories and Anthropic's claude-2 model so we'll need to install the following dependencies:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "477d04b3-c2b6-4ba5-962f-492c0d625cd5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -U langchain redis anthropic"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "93776323-d6b8-4912-bb6a-867c5e655f46",
+   "metadata": {},
+   "source": [
+    "Set your [Anthropic API  key](https://console.anthropic.com/):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c7f56f69-d2f1-4a21-990c-b5551eb012fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import getpass\n",
+    "import os\n",
+    "\n",
+    "os.environ[\"ANTHROPIC_API_KEY\"] = getpass.getpass()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a0ec9e0-7b1c-4c6f-b570-e61d520b47c6",
+   "metadata": {},
+   "source": [
+    "Start a local Redis Stack server if we don't have an existing Redis deployment to connect to:\n",
+    "```bash\n",
+    "docker run -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "cd6a250e-17fe-4368-a39d-1fe6b2cbde68",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "REDIS_URL = \"redis://localhost:6379/0\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "36f43b87-655c-4f64-aa7b-bd8c1955d8e5",
+   "metadata": {},
+   "source": [
+    "### [LangSmith](/docs/langsmith)\n",
+    "\n",
+    "LangSmith is especially useful for something like message history injection, where it can be hard to otherwise understand what the inputs are to various parts of the chain.\n",
+    "\n",
+    "Note that LangSmith is not needed, but it is helpful.\n",
+    "If you do want to use LangSmith, after you sign up at the link above, make sure to uncoment the below and set your environment variables to start logging traces:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2afc1556-8da1-4499-ba11-983b66c58b18",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
+    "# os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1a5a632e-ba9e-4488-b586-640ad5494f62",
+   "metadata": {},
+   "source": [
+    "## Example: Dict input, message output\n",
+    "\n",
+    "Let's create a simple chain that takes a dict as input and returns a BaseMessage.\n",
+    "\n",
+    "In this case the `\"question\"` key in the input represents our input message, and the `\"history\"` key is where our historical messages will be injected."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2a150d6f-8878-4950-8634-a608c5faad56",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from typing import Optional\n",
+    "\n",
+    "from langchain.chat_models import ChatAnthropic\n",
+    "from langchain.memory.chat_message_histories import RedisChatMessageHistory\n",
+    "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
+    "from langchain.schema.chat_history import BaseChatMessageHistory\n",
+    "from langchain.schema.runnable.history import RunnableWithMessageHistory"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "3185edba-4eb6-4b32-80c6-577c0d19af97",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt = ChatPromptTemplate.from_messages(\n",
+    "    [\n",
+    "        (\"system\", \"You're an assistant who's good at {ability}\"),\n",
+    "        MessagesPlaceholder(variable_name=\"history\"),\n",
+    "        (\"human\", \"{question}\"),\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "chain = prompt | ChatAnthropic(model=\"claude-2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f9d81796-ce61-484c-89e2-6c567d5e54ef",
+   "metadata": {},
+   "source": [
+    "### Adding message history\n",
+    "\n",
+    "To add message history to our original chain we wrap it in the `RunnableWithMessageHistory` class.\n",
+    "\n",
+    "Crucially, we also need to  define a method that takes a session_id string and based on it returns a `BaseChatMessageHistory`. Given the same input, this method should return an equivalent output.\n",
+    "\n",
+    "In this case we'll also want to specify `input_messages_key` (the key to be treated as the latest input message) and `history_messages_key` (the key to add historical messages to)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "ca7c64d8-e138-4ef8-9734-f82076c47d80",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_with_history = RunnableWithMessageHistory(\n",
+    "    chain,\n",
+    "    lambda session_id: RedisChatMessageHistory(session_id, url=REDIS_URL),\n",
+    "    input_messages_key=\"question\",\n",
+    "    history_messages_key=\"history\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "37eefdec-9901-4650-b64c-d3c097ed5f4d",
+   "metadata": {},
+   "source": [
+    "## Invoking with config\n",
+    "\n",
+    "Whenever we call our chain with message history, we need to include a config that contains the `session_id`\n",
+    "```python\n",
+    "config={\"configurable\": {\"session_id\": \"<SESSION_ID>\"}}\n",
+    "```\n",
+    "\n",
+    "Given the same configuration, our chain should be pulling from the same chat message history."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a85bcc22-ca4c-4ad5-9440-f94be7318f3e",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=' Cosine is one of the basic trigonometric functions in mathematics. It is defined as the ratio of the adjacent side to the hypotenuse in a right triangle.\\n\\nSome key properties and facts about cosine:\\n\\n- It is denoted by cos(θ), where θ is the angle in a right triangle. \\n\\n- The cosine of an acute angle is always positive. For angles greater than 90 degrees, cosine can be negative.\\n\\n- Cosine is one of the three main trig functions along with sine and tangent.\\n\\n- The cosine of 0 degrees is 1. As the angle increases towards 90 degrees, the cosine value decreases towards 0.\\n\\n- The range of values for cosine is -1 to 1.\\n\\n- The cosine function maps angles in a circle to the x-coordinate on the unit circle.\\n\\n- Cosine is used to find adjacent side lengths in right triangles, and has many other applications in mathematics, physics, engineering and more.\\n\\n- Key cosine identities include: cos(A+B) = cosAcosB − sinAsinB and cos(2A) = cos^2(A) − sin^2(A)\\n\\nSo in summary, cosine is a fundamental trig')"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain_with_history.invoke(\n",
+    "    {\"ability\": \"math\", \"question\": \"What does cosine mean?\"},\n",
+    "    config={\"configurable\": {\"session_id\": \"foobar\"}},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "ab29abd3-751f-41ce-a1b0-53f6b565e79d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=' The inverse of the cosine function is called the arccosine or inverse cosine, often denoted as cos-1(x) or arccos(x).\\n\\nThe key properties and facts about arccosine:\\n\\n- It is defined as the angle θ between 0 and π radians whose cosine is x. So arccos(x) = θ such that cos(θ) = x.\\n\\n- The range of arccosine is 0 to π radians (0 to 180 degrees).\\n\\n- The domain of arccosine is -1 to 1. \\n\\n- arccos(cos(θ)) = θ for values of θ from 0 to π radians.\\n\\n- arccos(x) is the angle in a right triangle whose adjacent side is x and hypotenuse is 1.\\n\\n- arccos(0) = 90 degrees. As x increases from 0 to 1, arccos(x) decreases from 90 to 0 degrees.\\n\\n- arccos(1) = 0 degrees. arccos(-1) = 180 degrees.\\n\\n- The graph of y = arccos(x) is part of the unit circle, restricted to x')"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain_with_history.invoke(\n",
+    "    {\"ability\": \"math\", \"question\": \"What's its inverse\"},\n",
+    "    config={\"configurable\": {\"session_id\": \"foobar\"}},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "da3d1feb-b4bb-4624-961c-7db2e1180df7",
+   "metadata": {},
+   "source": [
+    ":::tip [Langsmith trace](https://smith.langchain.com/public/863a003b-7ca8-4b24-be9e-d63ec13c106e/r)\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "61d5115e-64a1-4ad5-b676-8afd4ef6093e",
+   "metadata": {},
+   "source": [
+    "Looking at the Langsmith trace for the second call, we can see that when constructing the prompt, a \"history\" variable has been injected which is a list of two messages (our first input and first output)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "028cf151-6cd5-4533-b3cf-c8d735554647",
+   "metadata": {},
+   "source": [
+    "## Example: messages input, dict output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "0bb446b5-6251-45fe-a92a-4c6171473c53",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output_message': AIMessage(content=' Here is a summary of Simone de Beauvoir\\'s views on free will:\\n\\n- De Beauvoir was an existentialist philosopher and believed strongly in the concept of free will. She rejected the idea that human nature or instincts determine behavior.\\n\\n- Instead, de Beauvoir argued that human beings define their own essence or nature through their actions and choices. As she famously wrote, \"One is not born, but rather becomes, a woman.\"\\n\\n- De Beauvoir believed that while individuals are situated in certain cultural contexts and social conditions, they still have agency and the ability to transcend these situations. Freedom comes from choosing one\\'s attitude toward these constraints.\\n\\n- She emphasized the radical freedom and responsibility of the individual. We are \"condemned to be free\" because we cannot escape making choices and taking responsibility for our choices. \\n\\n- De Beauvoir felt that many people evade their freedom and responsibility by adopting rigid mindsets, ideologies, or conforming uncritically to social roles.\\n\\n- She advocated for the recognition of ambiguity in the human condition and warned against the quest for absolute rules that deny freedom and responsibility. Authentic living involves embracing ambiguity.\\n\\nIn summary, de Beauvoir promoted an existential ethics')}"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain.schema.messages import HumanMessage\n",
+    "from langchain.schema.runnable import RunnableMap\n",
+    "\n",
+    "chain = RunnableMap({\"output_message\": ChatAnthropic(model=\"claude-2\")})\n",
+    "chain_with_history = RunnableWithMessageHistory(\n",
+    "    chain,\n",
+    "    lambda session_id: RedisChatMessageHistory(session_id, url=REDIS_URL),\n",
+    "    output_messages_key=\"output_message\",\n",
+    ")\n",
+    "\n",
+    "chain_with_history.invoke(\n",
+    "    [HumanMessage(content=\"What did Simone de Beauvoir believe about free will\")],\n",
+    "    config={\"configurable\": {\"session_id\": \"baz\"}},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "601ce3ff-aea8-424d-8e54-fd614256af4f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output_message': AIMessage(content=\" There are many similarities between Simone de Beauvoir's views on free will and those of Jean-Paul Sartre, though some key differences emerge as well:\\n\\nSimilarities with Sartre:\\n\\n- Both were existentialist thinkers who rejected determinism and emphasized human freedom and responsibility.\\n\\n- They agreed that existence precedes essence - there is no predefined human nature that determines who we are.\\n\\n- Individuals must define themselves through their choices and actions. This leads to anxiety but also freedom.\\n\\n- The human condition is characterized by ambiguity and uncertainty, rather than fixed meanings/values.\\n\\n- Both felt that most people evade their freedom through self-deception, conformity, or adopting collective identities/values uncritically.\\n\\nDifferences from Sartre: \\n\\n- Sartre placed more emphasis on the burden and anguish of radical freedom. De Beauvoir focused more on its positive potential.\\n\\n- De Beauvoir critiqued Sartre's premise that human relations are necessarily conflictual. She saw more potential for mutual recognition.\\n\\n- Sartre saw the Other's gaze as a threat to freedom. De Beauvoir put more stress on how the Other's gaze can confirm\")}"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain_with_history.invoke(\n",
+    "    [HumanMessage(content=\"How did this compare to Sartre\")],\n",
+    "    config={\"configurable\": {\"session_id\": \"baz\"}},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b898d1b1-11e6-4d30-a8dd-cc5e45533611",
+   "metadata": {},
+   "source": [
+    ":::tip [LangSmith trace](https://smith.langchain.com/public/f6c3e1d1-a49d-4955-a9fa-c6519df74fa7/r)\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1724292c-01c6-44bb-83e8-9cdb6bf01483",
+   "metadata": {},
+   "source": [
+    "## More examples\n",
+    "\n",
+    "We could also do any of the below:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fd89240b-5a25-48f8-9568-5c1127f9ffad",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from operator import itemgetter\n",
+    "\n",
+    "# messages in, messages out\n",
+    "RunnableWithMessageHistory(\n",
+    "    ChatAnthropic(model=\"claude-2\"),\n",
+    "    lambda session_id: RedisChatMessageHistory(session_id, url=REDIS_URL),\n",
+    ")\n",
+    "\n",
+    "# dict with single key for all messages in, messages out\n",
+    "RunnableWithMessageHistory(\n",
+    "    itemgetter(\"input_messages\") | ChatAnthropic(model=\"claude-2\"),\n",
+    "    lambda session_id: RedisChatMessageHistory(session_id, url=REDIS_URL),\n",
+    "    input_messages_key=\"input_messages\",\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "poetry-venv",
+   "language": "python",
+   "name": "poetry-venv"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/docs/expression_language/how_to/routing.ipynb

@@ -40,8 +40,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.chat_models import ChatAnthropic\n",
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser"
    ]
   },

docs/docs/expression_language/index.mdx

@@ -30,4 +30,4 @@ As your chains get more and more complex, it becomes increasingly important to u
 With LCEL, **all** steps are automatically logged to [LangSmith](/docs/langsmith/) for maximum observability and debuggability.
 
 **Seamless LangServe deployment integration**
-Any chain created with LCEL can be easily deployed using LangServe.
+Any chain created with LCEL can be easily deployed using [LangServe](/docs/langserve).

docs/docs/expression_language/interface.ipynb

@@ -57,8 +57,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
     "\n",
     "model = ChatOpenAI()\n",
     "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\")\n",

docs/docs/get_started/installation.mdx

@@ -29,7 +29,7 @@ If you want to install from source, you can do so by cloning the repo and be sur
 pip install -e .
 ```
 
-## Langchain experimental
+## LangChain experimental
 The `langchain-experimental` package holds experimental LangChain code, intended for research and experimental uses.
 Install with:
 
@@ -37,14 +37,6 @@ Install with:
 pip install langchain-experimental
 ```
 
-## LangChain CLI
-The LangChain CLI is useful for working with LangChain templates and other LangServe projects.
-Install with:
-
-```bash
-pip install langchain-cli
-```
-
 ## LangServe
 LangServe helps developers deploy LangChain runnables and chains as a REST API.
 LangServe is automatically installed by LangChain CLI.
@@ -55,6 +47,14 @@ pip install "langserve[all]"
 ```
 for both client and server dependencies. Or `pip install "langserve[client]"` for client code, and `pip install "langserve[server]"` for server code.
 
+## LangChain CLI
+The LangChain CLI is useful for working with LangChain templates and other LangServe projects.
+Install with:
+
+```bash
+pip install langchain-cli
+```
+
 ## LangSmith SDK
 The LangSmith SDK is automatically installed by LangChain.
 If not using LangChain, install with:

docs/docs/get_started/introduction.mdx

@@ -14,7 +14,7 @@ This framework consists of several parts.
 - **[LangServe](/docs/langserve)**: A library for deploying LangChain chains as a REST API.
 - **[LangSmith](/docs/langsmith)**: A developer platform that lets you debug, test, evaluate, and monitor chains built on any LLM framework and seamlessly integrates with LangChain.
 
-![LangChain Diagram](/img/langchain_stack.png)
+![LangChain Diagram](/svg/langchain_stack.svg)
 
 Together, these products simplify the entire application lifecycle:
 - **Develop**: Write your applications in LangChain/LangChain.js. Hit the ground running using Templates for reference.
@@ -49,7 +49,7 @@ LCEL is a declarative way to compose chains. LCEL was designed from day 1 to sup
 
 - **[Overview](/docs/expression_language/)**: LCEL and its benefits
 - **[Interface](/docs/expression_language/interface)**: The standard interface for LCEL objects
-- **[How-to](/docs/expression_language/interface)**: Key features of LCEL
+- **[How-to](/docs/expression_language/how_to)**: Key features of LCEL
 - **[Cookbook](/docs/expression_language/cookbook)**: Example code for accomplishing common tasks
 
 

docs/docs/get_started/quickstart.mdx

@@ -4,7 +4,7 @@ In this quickstart we'll show you how to:
 - Get setup with LangChain, LangSmith and LangServe
 - Use the most basic and common components of LangChain: prompt templates, models, and output parsers
 - Use LangChain Expression Language, the protocol that LangChain is built on and which facilitates component chaining
-- Build simple application with LangChain
+- Build a simple application with LangChain
 - Trace your application with LangSmith
 - Serve your application with LangServe
 

docs/docs/guides/evaluation/comparison/custom.ipynb

@@ -34,7 +34,8 @@
    },
    "outputs": [],
    "source": [
-    "from typing import Optional, Any\n",
+    "from typing import Any, Optional\n",
+    "\n",
     "from langchain.evaluation import PairwiseStringEvaluator\n",
     "\n",
     "\n",
@@ -116,10 +117,11 @@
    },
    "outputs": [],
    "source": [
-    "from typing import Optional, Any\n",
-    "from langchain.evaluation import PairwiseStringEvaluator\n",
-    "from langchain.chat_models import ChatAnthropic\n",
+    "from typing import Any, Optional\n",
+    "\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.chat_models import ChatAnthropic\n",
+    "from langchain.evaluation import PairwiseStringEvaluator\n",
     "\n",
     "\n",
     "class CustomPreferenceEvaluator(PairwiseStringEvaluator):\n",

docs/docs/guides/evaluation/examples/comparisons.ipynb

@@ -1,448 +1,447 @@
 {
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "# Comparing Chain Outputs\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/examples/comparisons.ipynb)\n",
-                "\n",
-                "Suppose you have two different prompts (or LLMs). How do you know which will generate \"better\" results?\n",
-                "\n",
-                "One automated way to predict the preferred configuration is to use a `PairwiseStringEvaluator` like the `PairwiseStringEvalChain`<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1). This chain prompts an LLM to select which output is preferred, given a specific input.\n",
-                "\n",
-                "For this evaluation, we will need 3 things:\n",
-                "1. An evaluator\n",
-                "2. A dataset of inputs\n",
-                "3. 2 (or more) LLMs, Chains, or Agents to compare\n",
-                "\n",
-                "Then we will aggregate the results to determine the preferred model.\n",
-                "\n",
-                "### Step 1. Create the Evaluator\n",
-                "\n",
-                "In this example, you will use gpt-4 to select which output is preferred."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "eval_chain = load_evaluator(\"pairwise_string\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Step 2. Select Dataset\n",
-                "\n",
-                "If you already have real usage data for your LLM, you can use a representative sample. More examples\n",
-                "provide more reliable results. We will use some example queries someone might have about how to use langchain here."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7)\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "a2358d37246640ce95e0f9940194590a",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/1 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                }
-            ],
-            "source": [
-                "from langchain.evaluation.loading import load_dataset\n",
-                "\n",
-                "dataset = load_dataset(\"langchain-howto-queries\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Step 3. Define Models to Compare\n",
-                "\n",
-                "We will be comparing two agents in this case."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.utilities import SerpAPIWrapper\n",
-                "from langchain.agents import initialize_agent, Tool\n",
-                "from langchain.agents import AgentType\n",
-                "from langchain.chat_models import ChatOpenAI\n",
-                "\n",
-                "\n",
-                "# Initialize the language model\n",
-                "# You can add your own OpenAI API key by adding openai_api_key=\"<your_api_key>\"\n",
-                "llm = ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-0613\")\n",
-                "\n",
-                "# Initialize the SerpAPIWrapper for search functionality\n",
-                "# Replace <your_api_key> in openai_api_key=\"<your_api_key>\" with your actual SerpAPI key.\n",
-                "search = SerpAPIWrapper()\n",
-                "\n",
-                "# Define a list of tools offered by the agent\n",
-                "tools = [\n",
-                "    Tool(\n",
-                "        name=\"Search\",\n",
-                "        func=search.run,\n",
-                "        coroutine=search.arun,\n",
-                "        description=\"Useful when you need to answer questions about current events. You should ask targeted questions.\",\n",
-                "    ),\n",
-                "]"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "functions_agent = initialize_agent(\n",
-                "    tools, llm, agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=False\n",
-                ")\n",
-                "conversations_agent = initialize_agent(\n",
-                "    tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=False\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Step 4. Generate Responses\n",
-                "\n",
-                "We will generate outputs for each of the models before evaluating them."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "87277cb39a1a4726bb7cc533a24e2ea4",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/20 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                }
-            ],
-            "source": [
-                "from tqdm.notebook import tqdm\n",
-                "import asyncio\n",
-                "\n",
-                "results = []\n",
-                "agents = [functions_agent, conversations_agent]\n",
-                "concurrency_level = 6  # How many concurrent agents to run. May need to decrease if OpenAI is rate limiting.\n",
-                "\n",
-                "# We will only run the first 20 examples of this dataset to speed things up\n",
-                "# This will lead to larger confidence intervals downstream.\n",
-                "batch = []\n",
-                "for example in tqdm(dataset[:20]):\n",
-                "    batch.extend([agent.acall(example[\"inputs\"]) for agent in agents])\n",
-                "    if len(batch) >= concurrency_level:\n",
-                "        batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
-                "        results.extend(list(zip(*[iter(batch_results)] * 2)))\n",
-                "        batch = []\n",
-                "if batch:\n",
-                "    batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
-                "    results.extend(list(zip(*[iter(batch_results)] * 2)))"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "## Step 5. Evaluate Pairs\n",
-                "\n",
-                "Now it's time to evaluate the results. For each agent response, run the evaluation chain to select which output is preferred (or return a tie).\n",
-                "\n",
-                "Randomly select the input order to reduce the likelihood that one model will be preferred just because it is presented first."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "import random\n",
-                "\n",
-                "\n",
-                "def predict_preferences(dataset, results) -> list:\n",
-                "    preferences = []\n",
-                "\n",
-                "    for example, (res_a, res_b) in zip(dataset, results):\n",
-                "        input_ = example[\"inputs\"]\n",
-                "        # Flip a coin to reduce persistent position bias\n",
-                "        if random.random() < 0.5:\n",
-                "            pred_a, pred_b = res_a, res_b\n",
-                "            a, b = \"a\", \"b\"\n",
-                "        else:\n",
-                "            pred_a, pred_b = res_b, res_a\n",
-                "            a, b = \"b\", \"a\"\n",
-                "        eval_res = eval_chain.evaluate_string_pairs(\n",
-                "            prediction=pred_a[\"output\"] if isinstance(pred_a, dict) else str(pred_a),\n",
-                "            prediction_b=pred_b[\"output\"] if isinstance(pred_b, dict) else str(pred_b),\n",
-                "            input=input_,\n",
-                "        )\n",
-                "        if eval_res[\"value\"] == \"A\":\n",
-                "            preferences.append(a)\n",
-                "        elif eval_res[\"value\"] == \"B\":\n",
-                "            preferences.append(b)\n",
-                "        else:\n",
-                "            preferences.append(None)  # No preference\n",
-                "    return preferences"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "preferences = predict_preferences(dataset, results)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "**Print out the ratio of preferences.**"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "OpenAI Functions Agent: 95.00%\n",
-                        "None: 5.00%\n"
-                    ]
-                }
-            ],
-            "source": [
-                "from collections import Counter\n",
-                "\n",
-                "name_map = {\n",
-                "    \"a\": \"OpenAI Functions Agent\",\n",
-                "    \"b\": \"Structured Chat Agent\",\n",
-                "}\n",
-                "counts = Counter(preferences)\n",
-                "pref_ratios = {k: v / len(preferences) for k, v in counts.items()}\n",
-                "for k, v in pref_ratios.items():\n",
-                "    print(f\"{name_map.get(k)}: {v:.2%}\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Estimate Confidence Intervals\n",
-                "\n",
-                "The results seem pretty clear, but if you want to have a better sense of how confident we are, that model \"A\" (the OpenAI Functions Agent) is the preferred model, we can calculate confidence intervals. \n",
-                "\n",
-                "Below, use the Wilson score to estimate the confidence interval."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 9,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from math import sqrt\n",
-                "\n",
-                "\n",
-                "def wilson_score_interval(\n",
-                "    preferences: list, which: str = \"a\", z: float = 1.96\n",
-                ") -> tuple:\n",
-                "    \"\"\"Estimate the confidence interval using the Wilson score.\n",
-                "\n",
-                "    See: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval\n",
-                "    for more details, including when to use it and when it should not be used.\n",
-                "    \"\"\"\n",
-                "    total_preferences = preferences.count(\"a\") + preferences.count(\"b\")\n",
-                "    n_s = preferences.count(which)\n",
-                "\n",
-                "    if total_preferences == 0:\n",
-                "        return (0, 0)\n",
-                "\n",
-                "    p_hat = n_s / total_preferences\n",
-                "\n",
-                "    denominator = 1 + (z**2) / total_preferences\n",
-                "    adjustment = (z / denominator) * sqrt(\n",
-                "        p_hat * (1 - p_hat) / total_preferences\n",
-                "        + (z**2) / (4 * total_preferences * total_preferences)\n",
-                "    )\n",
-                "    center = (p_hat + (z**2) / (2 * total_preferences)) / denominator\n",
-                "    lower_bound = min(max(center - adjustment, 0.0), 1.0)\n",
-                "    upper_bound = min(max(center + adjustment, 0.0), 1.0)\n",
-                "\n",
-                "    return (lower_bound, upper_bound)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 10,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "The \"OpenAI Functions Agent\" would be preferred between 83.18% and 100.00% percent of the time (with 95% confidence).\n",
-                        "The \"Structured Chat Agent\" would be preferred between 0.00% and 16.82% percent of the time (with 95% confidence).\n"
-                    ]
-                }
-            ],
-            "source": [
-                "for which_, name in name_map.items():\n",
-                "    low, high = wilson_score_interval(preferences, which=which_)\n",
-                "    print(\n",
-                "        f'The \"{name}\" would be preferred between {low:.2%} and {high:.2%} percent of the time (with 95% confidence).'\n",
-                "    )"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "**Print out the p-value.**"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 11,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "The p-value is 0.00000. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
-                        "then there is a 0.00038% chance of observing the OpenAI Functions Agent be preferred at least 19\n",
-                        "times out of 19 trials.\n"
-                    ]
-                },
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_15978/384907688.py:6: DeprecationWarning: 'binom_test' is deprecated in favour of 'binomtest' from version 1.7.0 and will be removed in Scipy 1.12.0.\n",
-                        "  p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n"
-                    ]
-                }
-            ],
-            "source": [
-                "from scipy import stats\n",
-                "\n",
-                "preferred_model = max(pref_ratios, key=pref_ratios.get)\n",
-                "successes = preferences.count(preferred_model)\n",
-                "n = len(preferences) - preferences.count(None)\n",
-                "p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n",
-                "print(\n",
-                "    f\"\"\"The p-value is {p_value:.5f}. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
-                "then there is a {p_value:.5%} chance of observing the {name_map.get(preferred_model)} be preferred at least {successes}\n",
-                "times out of {n} trials.\"\"\"\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "<a name=\"cite_note-1\"></a>_1. Note: Automated evals are still an open research topic and are best used alongside other evaluation approaches. \n",
-                "LLM preferences exhibit biases, including banal ones like the order of outputs.\n",
-                "In choosing preferences, \"ground truth\" may not be taken into account, which may lead to scores that aren't grounded in utility._"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "metadata": {},
-            "outputs": [],
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Comparing Chain Outputs\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/examples/comparisons.ipynb)\n",
+    "\n",
+    "Suppose you have two different prompts (or LLMs). How do you know which will generate \"better\" results?\n",
+    "\n",
+    "One automated way to predict the preferred configuration is to use a `PairwiseStringEvaluator` like the `PairwiseStringEvalChain`<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1). This chain prompts an LLM to select which output is preferred, given a specific input.\n",
+    "\n",
+    "For this evaluation, we will need 3 things:\n",
+    "1. An evaluator\n",
+    "2. A dataset of inputs\n",
+    "3. 2 (or more) LLMs, Chains, or Agents to compare\n",
+    "\n",
+    "Then we will aggregate the results to determine the preferred model.\n",
+    "\n",
+    "### Step 1. Create the Evaluator\n",
+    "\n",
+    "In this example, you will use gpt-4 to select which output is preferred."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation import load_evaluator\n",
+    "\n",
+    "eval_chain = load_evaluator(\"pairwise_string\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2. Select Dataset\n",
+    "\n",
+    "If you already have real usage data for your LLM, you can use a representative sample. More examples\n",
+    "provide more reliable results. We will use some example queries someone might have about how to use langchain here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7)\n"
+     ]
     },
-    "nbformat": 4,
-    "nbformat_minor": 4
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a2358d37246640ce95e0f9940194590a",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"langchain-howto-queries\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3. Define Models to Compare\n",
+    "\n",
+    "We will be comparing two agents in this case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.agents import AgentType, Tool, initialize_agent\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
+    "\n",
+    "# Initialize the language model\n",
+    "# You can add your own OpenAI API key by adding openai_api_key=\"<your_api_key>\"\n",
+    "llm = ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-0613\")\n",
+    "\n",
+    "# Initialize the SerpAPIWrapper for search functionality\n",
+    "# Replace <your_api_key> in openai_api_key=\"<your_api_key>\" with your actual SerpAPI key.\n",
+    "search = SerpAPIWrapper()\n",
+    "\n",
+    "# Define a list of tools offered by the agent\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name=\"Search\",\n",
+    "        func=search.run,\n",
+    "        coroutine=search.arun,\n",
+    "        description=\"Useful when you need to answer questions about current events. You should ask targeted questions.\",\n",
+    "    ),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "functions_agent = initialize_agent(\n",
+    "    tools, llm, agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=False\n",
+    ")\n",
+    "conversations_agent = initialize_agent(\n",
+    "    tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=False\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 4. Generate Responses\n",
+    "\n",
+    "We will generate outputs for each of the models before evaluating them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "87277cb39a1a4726bb7cc533a24e2ea4",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/20 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "import asyncio\n",
+    "\n",
+    "from tqdm.notebook import tqdm\n",
+    "\n",
+    "results = []\n",
+    "agents = [functions_agent, conversations_agent]\n",
+    "concurrency_level = 6  # How many concurrent agents to run. May need to decrease if OpenAI is rate limiting.\n",
+    "\n",
+    "# We will only run the first 20 examples of this dataset to speed things up\n",
+    "# This will lead to larger confidence intervals downstream.\n",
+    "batch = []\n",
+    "for example in tqdm(dataset[:20]):\n",
+    "    batch.extend([agent.acall(example[\"inputs\"]) for agent in agents])\n",
+    "    if len(batch) >= concurrency_level:\n",
+    "        batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
+    "        results.extend(list(zip(*[iter(batch_results)] * 2)))\n",
+    "        batch = []\n",
+    "if batch:\n",
+    "    batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
+    "    results.extend(list(zip(*[iter(batch_results)] * 2)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 5. Evaluate Pairs\n",
+    "\n",
+    "Now it's time to evaluate the results. For each agent response, run the evaluation chain to select which output is preferred (or return a tie).\n",
+    "\n",
+    "Randomly select the input order to reduce the likelihood that one model will be preferred just because it is presented first."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "\n",
+    "\n",
+    "def predict_preferences(dataset, results) -> list:\n",
+    "    preferences = []\n",
+    "\n",
+    "    for example, (res_a, res_b) in zip(dataset, results):\n",
+    "        input_ = example[\"inputs\"]\n",
+    "        # Flip a coin to reduce persistent position bias\n",
+    "        if random.random() < 0.5:\n",
+    "            pred_a, pred_b = res_a, res_b\n",
+    "            a, b = \"a\", \"b\"\n",
+    "        else:\n",
+    "            pred_a, pred_b = res_b, res_a\n",
+    "            a, b = \"b\", \"a\"\n",
+    "        eval_res = eval_chain.evaluate_string_pairs(\n",
+    "            prediction=pred_a[\"output\"] if isinstance(pred_a, dict) else str(pred_a),\n",
+    "            prediction_b=pred_b[\"output\"] if isinstance(pred_b, dict) else str(pred_b),\n",
+    "            input=input_,\n",
+    "        )\n",
+    "        if eval_res[\"value\"] == \"A\":\n",
+    "            preferences.append(a)\n",
+    "        elif eval_res[\"value\"] == \"B\":\n",
+    "            preferences.append(b)\n",
+    "        else:\n",
+    "            preferences.append(None)  # No preference\n",
+    "    return preferences"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "preferences = predict_preferences(dataset, results)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "**Print out the ratio of preferences.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "OpenAI Functions Agent: 95.00%\n",
+      "None: 5.00%\n"
+     ]
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "name_map = {\n",
+    "    \"a\": \"OpenAI Functions Agent\",\n",
+    "    \"b\": \"Structured Chat Agent\",\n",
+    "}\n",
+    "counts = Counter(preferences)\n",
+    "pref_ratios = {k: v / len(preferences) for k, v in counts.items()}\n",
+    "for k, v in pref_ratios.items():\n",
+    "    print(f\"{name_map.get(k)}: {v:.2%}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Estimate Confidence Intervals\n",
+    "\n",
+    "The results seem pretty clear, but if you want to have a better sense of how confident we are, that model \"A\" (the OpenAI Functions Agent) is the preferred model, we can calculate confidence intervals. \n",
+    "\n",
+    "Below, use the Wilson score to estimate the confidence interval."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from math import sqrt\n",
+    "\n",
+    "\n",
+    "def wilson_score_interval(\n",
+    "    preferences: list, which: str = \"a\", z: float = 1.96\n",
+    ") -> tuple:\n",
+    "    \"\"\"Estimate the confidence interval using the Wilson score.\n",
+    "\n",
+    "    See: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval\n",
+    "    for more details, including when to use it and when it should not be used.\n",
+    "    \"\"\"\n",
+    "    total_preferences = preferences.count(\"a\") + preferences.count(\"b\")\n",
+    "    n_s = preferences.count(which)\n",
+    "\n",
+    "    if total_preferences == 0:\n",
+    "        return (0, 0)\n",
+    "\n",
+    "    p_hat = n_s / total_preferences\n",
+    "\n",
+    "    denominator = 1 + (z**2) / total_preferences\n",
+    "    adjustment = (z / denominator) * sqrt(\n",
+    "        p_hat * (1 - p_hat) / total_preferences\n",
+    "        + (z**2) / (4 * total_preferences * total_preferences)\n",
+    "    )\n",
+    "    center = (p_hat + (z**2) / (2 * total_preferences)) / denominator\n",
+    "    lower_bound = min(max(center - adjustment, 0.0), 1.0)\n",
+    "    upper_bound = min(max(center + adjustment, 0.0), 1.0)\n",
+    "\n",
+    "    return (lower_bound, upper_bound)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The \"OpenAI Functions Agent\" would be preferred between 83.18% and 100.00% percent of the time (with 95% confidence).\n",
+      "The \"Structured Chat Agent\" would be preferred between 0.00% and 16.82% percent of the time (with 95% confidence).\n"
+     ]
+    }
+   ],
+   "source": [
+    "for which_, name in name_map.items():\n",
+    "    low, high = wilson_score_interval(preferences, which=which_)\n",
+    "    print(\n",
+    "        f'The \"{name}\" would be preferred between {low:.2%} and {high:.2%} percent of the time (with 95% confidence).'\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Print out the p-value.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The p-value is 0.00000. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+      "then there is a 0.00038% chance of observing the OpenAI Functions Agent be preferred at least 19\n",
+      "times out of 19 trials.\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_15978/384907688.py:6: DeprecationWarning: 'binom_test' is deprecated in favour of 'binomtest' from version 1.7.0 and will be removed in Scipy 1.12.0.\n",
+      "  p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n"
+     ]
+    }
+   ],
+   "source": [
+    "from scipy import stats\n",
+    "\n",
+    "preferred_model = max(pref_ratios, key=pref_ratios.get)\n",
+    "successes = preferences.count(preferred_model)\n",
+    "n = len(preferences) - preferences.count(None)\n",
+    "p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n",
+    "print(\n",
+    "    f\"\"\"The p-value is {p_value:.5f}. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+    "then there is a {p_value:.5%} chance of observing the {name_map.get(preferred_model)} be preferred at least {successes}\n",
+    "times out of {n} trials.\"\"\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"cite_note-1\"></a>_1. Note: Automated evals are still an open research topic and are best used alongside other evaluation approaches. \n",
+    "LLM preferences exhibit biases, including banal ones like the order of outputs.\n",
+    "In choosing preferences, \"ground truth\" may not be taken into account, which may lead to scores that aren't grounded in utility._"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
 }

docs/docs/guides/evaluation/string/custom.ipynb

@@ -1,209 +1,209 @@
 {
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "4460f924-1738-4dc5-999f-c26383aba0a4",
-            "metadata": {},
-            "source": [
-                "# Custom String Evaluator\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/custom.ipynb)\n",
-                "\n",
-                "You can make your own custom string evaluators by inheriting from the `StringEvaluator` class and implementing the `_evaluate_strings` (and `_aevaluate_strings` for async support) methods.\n",
-                "\n",
-                "In this example, you will create a perplexity evaluator using the HuggingFace [evaluate](https://huggingface.co/docs/evaluate/index) library.\n",
-                "[Perplexity](https://en.wikipedia.org/wiki/Perplexity) is a measure of how well the generated text would be predicted by the model used to compute the metric."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "90ec5942-4b14-47b1-baff-9dd2a9f17a4e",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# %pip install evaluate > /dev/null"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "54fdba68-0ae7-4102-a45b-dabab86c97ac",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from typing import Any, Optional\n",
-                "\n",
-                "from langchain.evaluation import StringEvaluator\n",
-                "from evaluate import load\n",
-                "\n",
-                "\n",
-                "class PerplexityEvaluator(StringEvaluator):\n",
-                "    \"\"\"Evaluate the perplexity of a predicted string.\"\"\"\n",
-                "\n",
-                "    def __init__(self, model_id: str = \"gpt2\"):\n",
-                "        self.model_id = model_id\n",
-                "        self.metric_fn = load(\n",
-                "            \"perplexity\", module_type=\"metric\", model_id=self.model_id, pad_token=0\n",
-                "        )\n",
-                "\n",
-                "    def _evaluate_strings(\n",
-                "        self,\n",
-                "        *,\n",
-                "        prediction: str,\n",
-                "        reference: Optional[str] = None,\n",
-                "        input: Optional[str] = None,\n",
-                "        **kwargs: Any,\n",
-                "    ) -> dict:\n",
-                "        results = self.metric_fn.compute(\n",
-                "            predictions=[prediction], model_id=self.model_id\n",
-                "        )\n",
-                "        ppl = results[\"perplexities\"][0]\n",
-                "        return {\"score\": ppl}"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "52767568-8075-4f77-93c9-80e1a7e5cba3",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "evaluator = PerplexityEvaluator()"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "697ee0c0-d1ae-4a55-a542-a0f8e602c28a",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "Using pad_token, but it is not set yet.\n"
-                    ]
-                },
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
-                        "To disable this warning, you can either:\n",
-                        "\t- Avoid using `tokenizers` before the fork if possible\n",
-                        "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "467109d44654486e8b415288a319fc2c",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/1 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                },
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 190.3675537109375}"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(prediction=\"The rains in Spain fall mainly on the plain.\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "5089d9d1-eae6-4d47-b4f6-479e5d887d74",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "Using pad_token, but it is not set yet.\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "d3266f6f06d746e1bb03ce4aca07d9b9",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/1 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                },
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1982.0709228515625}"
-                        ]
-                    },
-                    "execution_count": 6,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# The perplexity is much higher since LangChain was introduced after 'gpt-2' was released and because it is never used in the following context.\n",
-                "evaluator.evaluate_strings(prediction=\"The rains in Spain fall mainly on LangChain.\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "id": "5eaa178f-6ba3-47ae-b3dc-1b196af6d213",
-            "metadata": {},
-            "outputs": [],
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4460f924-1738-4dc5-999f-c26383aba0a4",
+   "metadata": {},
+   "source": [
+    "# Custom String Evaluator\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/custom.ipynb)\n",
+    "\n",
+    "You can make your own custom string evaluators by inheriting from the `StringEvaluator` class and implementing the `_evaluate_strings` (and `_aevaluate_strings` for async support) methods.\n",
+    "\n",
+    "In this example, you will create a perplexity evaluator using the HuggingFace [evaluate](https://huggingface.co/docs/evaluate/index) library.\n",
+    "[Perplexity](https://en.wikipedia.org/wiki/Perplexity) is a measure of how well the generated text would be predicted by the model used to compute the metric."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "90ec5942-4b14-47b1-baff-9dd2a9f17a4e",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install evaluate > /dev/null"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "54fdba68-0ae7-4102-a45b-dabab86c97ac",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from typing import Any, Optional\n",
+    "\n",
+    "from evaluate import load\n",
+    "from langchain.evaluation import StringEvaluator\n",
+    "\n",
+    "\n",
+    "class PerplexityEvaluator(StringEvaluator):\n",
+    "    \"\"\"Evaluate the perplexity of a predicted string.\"\"\"\n",
+    "\n",
+    "    def __init__(self, model_id: str = \"gpt2\"):\n",
+    "        self.model_id = model_id\n",
+    "        self.metric_fn = load(\n",
+    "            \"perplexity\", module_type=\"metric\", model_id=self.model_id, pad_token=0\n",
+    "        )\n",
+    "\n",
+    "    def _evaluate_strings(\n",
+    "        self,\n",
+    "        *,\n",
+    "        prediction: str,\n",
+    "        reference: Optional[str] = None,\n",
+    "        input: Optional[str] = None,\n",
+    "        **kwargs: Any,\n",
+    "    ) -> dict:\n",
+    "        results = self.metric_fn.compute(\n",
+    "            predictions=[prediction], model_id=self.model_id\n",
+    "        )\n",
+    "        ppl = results[\"perplexities\"][0]\n",
+    "        return {\"score\": ppl}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "52767568-8075-4f77-93c9-80e1a7e5cba3",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "evaluator = PerplexityEvaluator()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "697ee0c0-d1ae-4a55-a542-a0f8e602c28a",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Using pad_token, but it is not set yet.\n"
+     ]
     },
-    "nbformat": 4,
-    "nbformat_minor": 5
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
+      "To disable this warning, you can either:\n",
+      "\t- Avoid using `tokenizers` before the fork if possible\n",
+      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "467109d44654486e8b415288a319fc2c",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 190.3675537109375}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "evaluator.evaluate_strings(prediction=\"The rains in Spain fall mainly on the plain.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "5089d9d1-eae6-4d47-b4f6-479e5d887d74",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Using pad_token, but it is not set yet.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d3266f6f06d746e1bb03ce4aca07d9b9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 1982.0709228515625}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# The perplexity is much higher since LangChain was introduced after 'gpt-2' was released and because it is never used in the following context.\n",
+    "evaluator.evaluate_strings(prediction=\"The rains in Spain fall mainly on LangChain.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5eaa178f-6ba3-47ae-b3dc-1b196af6d213",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
 }

docs/docs/guides/evaluation/string/json.ipynb

@@ -33,7 +33,7 @@
     }
    ],
    "source": [
-    "from langchain.evaluation import JsonValidityEvaluator, load_evaluator\n",
+    "from langchain.evaluation import JsonValidityEvaluator\n",
     "\n",
     "evaluator = JsonValidityEvaluator()\n",
     "# Equivalently\n",

docs/docs/guides/evaluation/string/scoring_eval_chain.ipynb

@@ -24,8 +24,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.evaluation import load_evaluator\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.evaluation import load_evaluator\n",
     "\n",
     "evaluator = load_evaluator(\"labeled_score_string\", llm=ChatOpenAI(model=\"gpt-4\"))"
    ]

docs/docs/guides/evaluation/trajectory/custom.ipynb

@@ -1,142 +1,143 @@
 {
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "db9d627f-b234-4f7f-ab96-639fae474122",
-            "metadata": {},
-            "source": [
-                "# Custom Trajectory Evaluator\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/trajectory/custom.ipynb)\n",
-                "\n",
-                "You can make your own custom trajectory evaluators by inheriting from the [AgentTrajectoryEvaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator) class and overwriting the `_evaluate_agent_trajectory` (and `_aevaluate_agent_action`) method.\n",
-                "\n",
-                "\n",
-                "In this example, you will make a simple trajectory evaluator that uses an LLM to determine if any actions were unnecessary."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "ca84ab0c-e7e2-4c03-bd74-9cc4e6338eec",
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "from typing import Any, Optional, Sequence, Tuple\n",
-                "from langchain.chat_models import ChatOpenAI\n",
-                "from langchain.chains import LLMChain\n",
-                "from langchain.schema import AgentAction\n",
-                "from langchain.evaluation import AgentTrajectoryEvaluator\n",
-                "\n",
-                "\n",
-                "class StepNecessityEvaluator(AgentTrajectoryEvaluator):\n",
-                "    \"\"\"Evaluate the perplexity of a predicted string.\"\"\"\n",
-                "\n",
-                "    def __init__(self) -> None:\n",
-                "        llm = ChatOpenAI(model=\"gpt-4\", temperature=0.0)\n",
-                "        template = \"\"\"Are any of the following steps unnecessary in answering {input}? Provide the verdict on a new line as a single \"Y\" for yes or \"N\" for no.\n",
-                "\n",
-                "        DATA\n",
-                "        ------\n",
-                "        Steps: {trajectory}\n",
-                "        ------\n",
-                "\n",
-                "        Verdict:\"\"\"\n",
-                "        self.chain = LLMChain.from_string(llm, template)\n",
-                "\n",
-                "    def _evaluate_agent_trajectory(\n",
-                "        self,\n",
-                "        *,\n",
-                "        prediction: str,\n",
-                "        input: str,\n",
-                "        agent_trajectory: Sequence[Tuple[AgentAction, str]],\n",
-                "        reference: Optional[str] = None,\n",
-                "        **kwargs: Any,\n",
-                "    ) -> dict:\n",
-                "        vals = [\n",
-                "            f\"{i}: Action=[{action.tool}] returned observation = [{observation}]\"\n",
-                "            for i, (action, observation) in enumerate(agent_trajectory)\n",
-                "        ]\n",
-                "        trajectory = \"\\n\".join(vals)\n",
-                "        response = self.chain.run(dict(trajectory=trajectory, input=input), **kwargs)\n",
-                "        decision = response.split(\"\\n\")[-1].strip()\n",
-                "        score = 1 if decision == \"Y\" else 0\n",
-                "        return {\"score\": score, \"value\": decision, \"reasoning\": response}"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "297dea4b-fb28-4292-b6e0-1c769cfb9cbd",
-            "metadata": {},
-            "source": [
-                "The example above will return a score of 1 if the language model predicts that any of the actions were unnecessary, and it returns a score of 0 if all of them were predicted to be necessary. It returns the string 'decision' as the 'value', and includes the rest of the generated text as 'reasoning' to let you audit the decision.\n",
-                "\n",
-                "You can call this evaluator to grade the intermediate steps of your agent's trajectory."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "a3fbcc1d-249f-4e00-8841-b6872c73c486",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1, 'value': 'Y', 'reasoning': 'Y'}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator = StepNecessityEvaluator()\n",
-                "\n",
-                "evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=\"The answer is pi\",\n",
-                "    input=\"What is today?\",\n",
-                "    agent_trajectory=[\n",
-                "        (\n",
-                "            AgentAction(tool=\"ask\", tool_input=\"What is today?\", log=\"\"),\n",
-                "            \"tomorrow's yesterday\",\n",
-                "        ),\n",
-                "        (\n",
-                "            AgentAction(tool=\"check_tv\", tool_input=\"Watch tv for half hour\", log=\"\"),\n",
-                "            \"bzzz\",\n",
-                "        ),\n",
-                "    ],\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "77353528-723e-4075-939e-aebdb17c1e4f",
-            "metadata": {},
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "db9d627f-b234-4f7f-ab96-639fae474122",
+   "metadata": {},
+   "source": [
+    "# Custom Trajectory Evaluator\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/trajectory/custom.ipynb)\n",
+    "\n",
+    "You can make your own custom trajectory evaluators by inheriting from the [AgentTrajectoryEvaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator) class and overwriting the `_evaluate_agent_trajectory` (and `_aevaluate_agent_action`) method.\n",
+    "\n",
+    "\n",
+    "In this example, you will make a simple trajectory evaluator that uses an LLM to determine if any actions were unnecessary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ca84ab0c-e7e2-4c03-bd74-9cc4e6338eec",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from typing import Any, Optional, Sequence, Tuple\n",
+    "\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.evaluation import AgentTrajectoryEvaluator\n",
+    "from langchain.schema import AgentAction\n",
+    "\n",
+    "\n",
+    "class StepNecessityEvaluator(AgentTrajectoryEvaluator):\n",
+    "    \"\"\"Evaluate the perplexity of a predicted string.\"\"\"\n",
+    "\n",
+    "    def __init__(self) -> None:\n",
+    "        llm = ChatOpenAI(model=\"gpt-4\", temperature=0.0)\n",
+    "        template = \"\"\"Are any of the following steps unnecessary in answering {input}? Provide the verdict on a new line as a single \"Y\" for yes or \"N\" for no.\n",
+    "\n",
+    "        DATA\n",
+    "        ------\n",
+    "        Steps: {trajectory}\n",
+    "        ------\n",
+    "\n",
+    "        Verdict:\"\"\"\n",
+    "        self.chain = LLMChain.from_string(llm, template)\n",
+    "\n",
+    "    def _evaluate_agent_trajectory(\n",
+    "        self,\n",
+    "        *,\n",
+    "        prediction: str,\n",
+    "        input: str,\n",
+    "        agent_trajectory: Sequence[Tuple[AgentAction, str]],\n",
+    "        reference: Optional[str] = None,\n",
+    "        **kwargs: Any,\n",
+    "    ) -> dict:\n",
+    "        vals = [\n",
+    "            f\"{i}: Action=[{action.tool}] returned observation = [{observation}]\"\n",
+    "            for i, (action, observation) in enumerate(agent_trajectory)\n",
+    "        ]\n",
+    "        trajectory = \"\\n\".join(vals)\n",
+    "        response = self.chain.run(dict(trajectory=trajectory, input=input), **kwargs)\n",
+    "        decision = response.split(\"\\n\")[-1].strip()\n",
+    "        score = 1 if decision == \"Y\" else 0\n",
+    "        return {\"score\": score, \"value\": decision, \"reasoning\": response}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "297dea4b-fb28-4292-b6e0-1c769cfb9cbd",
+   "metadata": {},
+   "source": [
+    "The example above will return a score of 1 if the language model predicts that any of the actions were unnecessary, and it returns a score of 0 if all of them were predicted to be necessary. It returns the string 'decision' as the 'value', and includes the rest of the generated text as 'reasoning' to let you audit the decision.\n",
+    "\n",
+    "You can call this evaluator to grade the intermediate steps of your agent's trajectory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "a3fbcc1d-249f-4e00-8841-b6872c73c486",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 1, 'value': 'Y', 'reasoning': 'Y'}"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "evaluator = StepNecessityEvaluator()\n",
+    "\n",
+    "evaluator.evaluate_agent_trajectory(\n",
+    "    prediction=\"The answer is pi\",\n",
+    "    input=\"What is today?\",\n",
+    "    agent_trajectory=[\n",
+    "        (\n",
+    "            AgentAction(tool=\"ask\", tool_input=\"What is today?\", log=\"\"),\n",
+    "            \"tomorrow's yesterday\",\n",
+    "        ),\n",
+    "        (\n",
+    "            AgentAction(tool=\"check_tv\", tool_input=\"Watch tv for half hour\", log=\"\"),\n",
+    "            \"bzzz\",\n",
+    "        ),\n",
+    "    ],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "77353528-723e-4075-939e-aebdb17c1e4f",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
 }

docs/docs/guides/evaluation/trajectory/trajectory_eval.ipynb

@@ -1,305 +1,303 @@
 {
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "6e5ea1a1-7e74-459b-bf14-688f87d09124",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "# Agent Trajectory\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/trajectory/trajectory_eval.ipynb)\n",
-                "\n",
-                "Agents can be difficult to holistically evaluate due to the breadth of actions and generation they can make. We recommend using multiple evaluation techniques appropriate to your use case. One way to evaluate an agent is to look at the whole trajectory of actions taken along with their responses.\n",
-                "\n",
-                "Evaluators that do this can implement the `AgentTrajectoryEvaluator` interface. This walkthrough will show how to use the `trajectory` evaluator to grade  an OpenAI functions agent.\n",
-                "\n",
-                "For more information, check out the reference docs for the [TrajectoryEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain) for more info."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "149402da-5212-43e2-b7c0-a701727f5293",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"trajectory\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "b1c64c1a",
-            "metadata": {},
-            "source": [
-                "## Methods\n",
-                "\n",
-                "\n",
-                "The Agent Trajectory Evaluators are used with the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.evaluate_agent_trajectory) (and async [aevaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.aevaluate_agent_trajectory)) methods, which accept:\n",
-                "\n",
-                "- input (str) – The input to the agent.\n",
-                "- prediction (str) – The final predicted response.\n",
-                "- agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory\n",
-                "\n",
-                "They return a dictionary with the following values:\n",
-                "- score: Float from 0 to 1, where 1 would mean \"most effective\" and 0 would mean \"least effective\"\n",
-                "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "e733562c-4c17-4942-9647-acfc5ebfaca2",
-            "metadata": {},
-            "source": [
-                "## Capturing Trajectory\n",
-                "\n",
-                "The easiest way to return an agent's trajectory (without using tracing callbacks like those in LangSmith) for evaluation is to initialize the agent with `return_intermediate_steps=True`.\n",
-                "\n",
-                "Below, create an example agent we will call to evaluate."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "451cb0cb-6f42-4abd-aa6d-fb871fce034d",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "import os\n",
-                "import subprocess\n",
-                "\n",
-                "from langchain.chat_models import ChatOpenAI\n",
-                "from langchain.tools import tool\n",
-                "from langchain.agents import AgentType, initialize_agent\n",
-                "\n",
-                "from pydantic import HttpUrl\n",
-                "from urllib.parse import urlparse\n",
-                "\n",
-                "\n",
-                "@tool\n",
-                "def ping(url: HttpUrl, return_error: bool) -> str:\n",
-                "    \"\"\"Ping the fully specified url. Must include https:// in the url.\"\"\"\n",
-                "    hostname = urlparse(str(url)).netloc\n",
-                "    completed_process = subprocess.run(\n",
-                "        [\"ping\", \"-c\", \"1\", hostname], capture_output=True, text=True\n",
-                "    )\n",
-                "    output = completed_process.stdout\n",
-                "    if return_error and completed_process.returncode != 0:\n",
-                "        return completed_process.stderr\n",
-                "    return output\n",
-                "\n",
-                "\n",
-                "@tool\n",
-                "def trace_route(url: HttpUrl, return_error: bool) -> str:\n",
-                "    \"\"\"Trace the route to the specified url. Must include https:// in the url.\"\"\"\n",
-                "    hostname = urlparse(str(url)).netloc\n",
-                "    completed_process = subprocess.run(\n",
-                "        [\"traceroute\", hostname], capture_output=True, text=True\n",
-                "    )\n",
-                "    output = completed_process.stdout\n",
-                "    if return_error and completed_process.returncode != 0:\n",
-                "        return completed_process.stderr\n",
-                "    return output\n",
-                "\n",
-                "\n",
-                "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0613\", temperature=0)\n",
-                "agent = initialize_agent(\n",
-                "    llm=llm,\n",
-                "    tools=[ping, trace_route],\n",
-                "    agent=AgentType.OPENAI_MULTI_FUNCTIONS,\n",
-                "    return_intermediate_steps=True,  # IMPORTANT!\n",
-                ")\n",
-                "\n",
-                "result = agent(\"What's the latency like for https://langchain.com?\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "2df34eed-45a5-4f91-88d3-9aa55f28391a",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "## Evaluate Trajectory\n",
-                "\n",
-                "Pass the input, trajectory, and pass to the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator.evaluate_agent_trajectory) method."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "8d2c8703-98ed-4068-8a8b-393f0f1f64ea",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1.0,\n",
-                            " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the website https://langchain.com.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. It uses the 'ping' tool to measure the latency of the website, which is the correct tool for this task.\\n\\niii. The AI language model uses the tool in a helpful way. It inputs the URL into the 'ping' tool and correctly interprets the output to provide the latency in milliseconds.\\n\\niv. The AI language model does not use too many steps to answer the question. It only uses one step, which is appropriate for this type of question.\\n\\nv. The appropriate tool is used to answer the question. The 'ping' tool is the correct tool to measure website latency.\\n\\nGiven these considerations, the AI language model's performance is excellent. It uses the correct tool, interprets the output correctly, and provides a helpful and direct answer to the user's question.\"}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=result[\"output\"],\n",
-                "    input=result[\"input\"],\n",
-                "    agent_trajectory=result[\"intermediate_steps\"],\n",
-                ")\n",
-                "evaluation_result"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "fc5467c1-ea92-405f-949a-3011388fa9ee",
-            "metadata": {},
-            "source": [
-                "## Configuring the Evaluation LLM\n",
-                "\n",
-                "If you don't select an LLM to use for evaluation, the [load_evaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.loading.load_evaluator.html#langchain.evaluation.loading.load_evaluator) function will use `gpt-4` to power the evaluation chain. You can select any chat model for the agent trajectory evaluator as below."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "1f6318f3-642a-4766-bc7a-f91239795ee7",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# %pip install anthropic\n",
-                "# ANTHROPIC_API_KEY=<YOUR ANTHROPIC API KEY>"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "id": "b2852289-5df9-402e-95b5-7efebf0fc943",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.chat_models import ChatAnthropic\n",
-                "\n",
-                "eval_llm = ChatAnthropic(temperature=0)\n",
-                "evaluator = load_evaluator(\"trajectory\", llm=eval_llm)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "ff72d21a-93b9-4c2f-8613-733d9c9330d7",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1.0,\n",
-                            " 'reasoning': \"Here is my detailed evaluation of the AI's response:\\n\\ni. The final answer is helpful, as it directly provides the latency measurement for the requested website.\\n\\nii. The sequence of using the ping tool to measure latency is logical for this question.\\n\\niii. The ping tool is used in a helpful way, with the website URL provided as input and the output latency measurement extracted.\\n\\niv. Only one step is used, which is appropriate for simply measuring latency. More steps are not needed.\\n\\nv. The ping tool is an appropriate choice to measure latency. \\n\\nIn summary, the AI uses an optimal single step approach with the right tool and extracts the needed output. The final answer directly answers the question in a helpful way.\\n\\nOverall\"}"
-                        ]
-                    },
-                    "execution_count": 6,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=result[\"output\"],\n",
-                "    input=result[\"input\"],\n",
-                "    agent_trajectory=result[\"intermediate_steps\"],\n",
-                ")\n",
-                "evaluation_result"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "95ce4240-f5a0-4810-8d09-b2f4c9e18b7f",
-            "metadata": {},
-            "source": [
-                "## Providing List of Valid Tools\n",
-                "\n",
-                "By default, the evaluator doesn't take into account the tools the agent is permitted to call. You can provide these to the evaluator via the `agent_tools` argument.\n"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "id": "24c10566-2ef5-45c5-9213-a8fb28e2ca1f",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"trajectory\", agent_tools=[ping, trace_route])"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "id": "7b995786-5b78-4d9e-8e8a-1f2a203113e2",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1.0,\n",
-                            " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the specified website.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. In this case, only one tool was needed to answer the question, and the model chose the correct one.\\n\\niii. The AI language model uses the tool in a helpful way. The 'ping' tool was used to determine the latency of the website, which was the information the user was seeking.\\n\\niv. The AI language model does not use too many steps to answer the question. Only one step was needed and used.\\n\\nv. The appropriate tool was used to answer the question. The 'ping' tool is designed to measure latency, which was the information the user was seeking.\\n\\nGiven these considerations, the AI language model's performance in answering this question is excellent.\"}"
-                        ]
-                    },
-                    "execution_count": 8,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=result[\"output\"],\n",
-                "    input=result[\"input\"],\n",
-                "    agent_trajectory=result[\"intermediate_steps\"],\n",
-                ")\n",
-                "evaluation_result"
-            ]
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "6e5ea1a1-7e74-459b-bf14-688f87d09124",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "# Agent Trajectory\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/trajectory/trajectory_eval.ipynb)\n",
+    "\n",
+    "Agents can be difficult to holistically evaluate due to the breadth of actions and generation they can make. We recommend using multiple evaluation techniques appropriate to your use case. One way to evaluate an agent is to look at the whole trajectory of actions taken along with their responses.\n",
+    "\n",
+    "Evaluators that do this can implement the `AgentTrajectoryEvaluator` interface. This walkthrough will show how to use the `trajectory` evaluator to grade  an OpenAI functions agent.\n",
+    "\n",
+    "For more information, check out the reference docs for the [TrajectoryEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain) for more info."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "149402da-5212-43e2-b7c0-a701727f5293",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation import load_evaluator\n",
+    "\n",
+    "evaluator = load_evaluator(\"trajectory\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b1c64c1a",
+   "metadata": {},
+   "source": [
+    "## Methods\n",
+    "\n",
+    "\n",
+    "The Agent Trajectory Evaluators are used with the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.evaluate_agent_trajectory) (and async [aevaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.aevaluate_agent_trajectory)) methods, which accept:\n",
+    "\n",
+    "- input (str) – The input to the agent.\n",
+    "- prediction (str) – The final predicted response.\n",
+    "- agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory\n",
+    "\n",
+    "They return a dictionary with the following values:\n",
+    "- score: Float from 0 to 1, where 1 would mean \"most effective\" and 0 would mean \"least effective\"\n",
+    "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e733562c-4c17-4942-9647-acfc5ebfaca2",
+   "metadata": {},
+   "source": [
+    "## Capturing Trajectory\n",
+    "\n",
+    "The easiest way to return an agent's trajectory (without using tracing callbacks like those in LangSmith) for evaluation is to initialize the agent with `return_intermediate_steps=True`.\n",
+    "\n",
+    "Below, create an example agent we will call to evaluate."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "451cb0cb-6f42-4abd-aa6d-fb871fce034d",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "from urllib.parse import urlparse\n",
+    "\n",
+    "from langchain.agents import AgentType, initialize_agent\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.tools import tool\n",
+    "from pydantic import HttpUrl\n",
+    "\n",
+    "\n",
+    "@tool\n",
+    "def ping(url: HttpUrl, return_error: bool) -> str:\n",
+    "    \"\"\"Ping the fully specified url. Must include https:// in the url.\"\"\"\n",
+    "    hostname = urlparse(str(url)).netloc\n",
+    "    completed_process = subprocess.run(\n",
+    "        [\"ping\", \"-c\", \"1\", hostname], capture_output=True, text=True\n",
+    "    )\n",
+    "    output = completed_process.stdout\n",
+    "    if return_error and completed_process.returncode != 0:\n",
+    "        return completed_process.stderr\n",
+    "    return output\n",
+    "\n",
+    "\n",
+    "@tool\n",
+    "def trace_route(url: HttpUrl, return_error: bool) -> str:\n",
+    "    \"\"\"Trace the route to the specified url. Must include https:// in the url.\"\"\"\n",
+    "    hostname = urlparse(str(url)).netloc\n",
+    "    completed_process = subprocess.run(\n",
+    "        [\"traceroute\", hostname], capture_output=True, text=True\n",
+    "    )\n",
+    "    output = completed_process.stdout\n",
+    "    if return_error and completed_process.returncode != 0:\n",
+    "        return completed_process.stderr\n",
+    "    return output\n",
+    "\n",
+    "\n",
+    "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0613\", temperature=0)\n",
+    "agent = initialize_agent(\n",
+    "    llm=llm,\n",
+    "    tools=[ping, trace_route],\n",
+    "    agent=AgentType.OPENAI_MULTI_FUNCTIONS,\n",
+    "    return_intermediate_steps=True,  # IMPORTANT!\n",
+    ")\n",
+    "\n",
+    "result = agent(\"What's the latency like for https://langchain.com?\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2df34eed-45a5-4f91-88d3-9aa55f28391a",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "## Evaluate Trajectory\n",
+    "\n",
+    "Pass the input, trajectory, and pass to the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator.evaluate_agent_trajectory) method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "8d2c8703-98ed-4068-8a8b-393f0f1f64ea",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 1.0,\n",
+       " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the website https://langchain.com.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. It uses the 'ping' tool to measure the latency of the website, which is the correct tool for this task.\\n\\niii. The AI language model uses the tool in a helpful way. It inputs the URL into the 'ping' tool and correctly interprets the output to provide the latency in milliseconds.\\n\\niv. The AI language model does not use too many steps to answer the question. It only uses one step, which is appropriate for this type of question.\\n\\nv. The appropriate tool is used to answer the question. The 'ping' tool is the correct tool to measure website latency.\\n\\nGiven these considerations, the AI language model's performance is excellent. It uses the correct tool, interprets the output correctly, and provides a helpful and direct answer to the user's question.\"}"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
+    "    prediction=result[\"output\"],\n",
+    "    input=result[\"input\"],\n",
+    "    agent_trajectory=result[\"intermediate_steps\"],\n",
+    ")\n",
+    "evaluation_result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fc5467c1-ea92-405f-949a-3011388fa9ee",
+   "metadata": {},
+   "source": [
+    "## Configuring the Evaluation LLM\n",
+    "\n",
+    "If you don't select an LLM to use for evaluation, the [load_evaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.loading.load_evaluator.html#langchain.evaluation.loading.load_evaluator) function will use `gpt-4` to power the evaluation chain. You can select any chat model for the agent trajectory evaluator as below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "1f6318f3-642a-4766-bc7a-f91239795ee7",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# %pip install anthropic\n",
+    "# ANTHROPIC_API_KEY=<YOUR ANTHROPIC API KEY>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "b2852289-5df9-402e-95b5-7efebf0fc943",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatAnthropic\n",
+    "\n",
+    "eval_llm = ChatAnthropic(temperature=0)\n",
+    "evaluator = load_evaluator(\"trajectory\", llm=eval_llm)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "ff72d21a-93b9-4c2f-8613-733d9c9330d7",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 1.0,\n",
+       " 'reasoning': \"Here is my detailed evaluation of the AI's response:\\n\\ni. The final answer is helpful, as it directly provides the latency measurement for the requested website.\\n\\nii. The sequence of using the ping tool to measure latency is logical for this question.\\n\\niii. The ping tool is used in a helpful way, with the website URL provided as input and the output latency measurement extracted.\\n\\niv. Only one step is used, which is appropriate for simply measuring latency. More steps are not needed.\\n\\nv. The ping tool is an appropriate choice to measure latency. \\n\\nIn summary, the AI uses an optimal single step approach with the right tool and extracts the needed output. The final answer directly answers the question in a helpful way.\\n\\nOverall\"}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
+    "    prediction=result[\"output\"],\n",
+    "    input=result[\"input\"],\n",
+    "    agent_trajectory=result[\"intermediate_steps\"],\n",
+    ")\n",
+    "evaluation_result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "95ce4240-f5a0-4810-8d09-b2f4c9e18b7f",
+   "metadata": {},
+   "source": [
+    "## Providing List of Valid Tools\n",
+    "\n",
+    "By default, the evaluator doesn't take into account the tools the agent is permitted to call. You can provide these to the evaluator via the `agent_tools` argument.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "24c10566-2ef5-45c5-9213-a8fb28e2ca1f",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation import load_evaluator\n",
+    "\n",
+    "evaluator = load_evaluator(\"trajectory\", agent_tools=[ping, trace_route])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "7b995786-5b78-4d9e-8e8a-1f2a203113e2",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'score': 1.0,\n",
+       " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the specified website.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. In this case, only one tool was needed to answer the question, and the model chose the correct one.\\n\\niii. The AI language model uses the tool in a helpful way. The 'ping' tool was used to determine the latency of the website, which was the information the user was seeking.\\n\\niv. The AI language model does not use too many steps to answer the question. Only one step was needed and used.\\n\\nv. The appropriate tool was used to answer the question. The 'ping' tool is designed to measure latency, which was the information the user was seeking.\\n\\nGiven these considerations, the AI language model's performance in answering this question is excellent.\"}"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
+    "    prediction=result[\"output\"],\n",
+    "    input=result[\"input\"],\n",
+    "    agent_trajectory=result[\"intermediate_steps\"],\n",
+    ")\n",
+    "evaluation_result"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
 }

docs/docs/guides/fallbacks.ipynb

@@ -33,7 +33,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic"
+    "from langchain.chat_models import ChatAnthropic, ChatOpenAI"
    ]
   },
   {
@@ -52,6 +52,7 @@
    "outputs": [],
    "source": [
     "from unittest.mock import patch\n",
+    "\n",
     "from openai.error import RateLimitError"
    ]
   },

docs/docs/guides/local_llms.ipynb

@@ -265,10 +265,16 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "5eba38dc",
-   "metadata": {},
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
    "outputs": [],
    "source": [
-    "CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dirclear"
+    "%env CMAKE_ARGS=\"-DLLAMA_METAL=on\"\n",
+    "%env FORCE_CMAKE=1\n",
+    "%pip install -U llama-cpp-python --no-cache-dirclear`"
    ]
   },
   {
@@ -379,7 +385,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "pip install gpt4all"
+    "pip install gpt4all\n"
    ]
   },
   {
@@ -476,9 +482,9 @@
     }
    ],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.chains import LLMChain\n",
     "from langchain.chains.prompt_selector import ConditionalPromptSelector\n",
+    "from langchain.prompts import PromptTemplate\n",
     "\n",
     "DEFAULT_LLAMA_SEARCH_PROMPT = PromptTemplate(\n",
     "    input_variables=[\"question\"],\n",

docs/docs/guides/model_laboratory.ipynb

@@ -19,8 +19,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import LLMChain\nfrom langchain.llms import OpenAI, Cohere, HuggingFaceHub\nfrom langchain.prompts import PromptTemplate\n",
-    "from langchain.model_laboratory import ModelLaboratory"
+    "from langchain.llms import Cohere, HuggingFaceHub, OpenAI\n",
+    "from langchain.model_laboratory import ModelLaboratory\n",
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {
@@ -139,7 +140,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import SelfAskWithSearchChain\nfrom langchain.utilities import SerpAPIWrapper\n",
+    "from langchain.chains import SelfAskWithSearchChain\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
     "\n",
     "open_ai_llm = OpenAI(temperature=0)\n",
     "search = SerpAPIWrapper()\n",

docs/docs/guides/privacy/presidio_data_anonymization/index.ipynb

@@ -98,7 +98,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "text = f\"\"\"Slim Shady recently lost his wallet. \n",
+    "text = \"\"\"Slim Shady recently lost his wallet. \n",
     "Inside is some cash and his credit card with the number 4916 0387 9536 0861. \n",
     "If you would find it, please call at 313-666-7440 or write an email here: real.slim.shady@gmail.com.\"\"\""
    ]
@@ -127,8 +127,8 @@
     }
    ],
    "source": [
-    "from langchain.prompts.prompt import PromptTemplate\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts.prompt import PromptTemplate\n",
     "\n",
     "anonymizer = PresidioAnonymizer()\n",
     "\n",
@@ -286,7 +286,6 @@
     "# Define the regex pattern in a Presidio `Pattern` object:\n",
     "from presidio_analyzer import Pattern, PatternRecognizer\n",
     "\n",
-    "\n",
     "polish_phone_numbers_pattern = Pattern(\n",
     "    name=\"polish_phone_numbers_pattern\",\n",
     "    regex=\"(?<!\\w)(\\(?(\\+|00)?48\\)?)?[ -]?\\d{3}[ -]?\\d{3}[ -]?\\d{3}(?!\\w)\",\n",

docs/docs/guides/privacy/presidio_data_anonymization/qa_privacy_protection.ipynb

@@ -233,7 +233,6 @@
     "# Define the regex pattern in a Presidio `Pattern` object:\n",
     "from presidio_analyzer import Pattern, PatternRecognizer\n",
     "\n",
-    "\n",
     "polish_id_pattern = Pattern(\n",
     "    name=\"polish_id_pattern\",\n",
     "    regex=\"[A-Z]{3}\\d{6}\",\n",
@@ -638,8 +637,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
     "from langchain.vectorstores import FAISS\n",
     "\n",
     "# 2. Load the data: In our case data's already loaded\n",
@@ -664,12 +663,11 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
+    "\n",
     "from langchain.chat_models.openai import ChatOpenAI\n",
-    "from langchain.schema.runnable import RunnableMap\n",
     "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "from langchain.schema.runnable import RunnableLambda\n",
+    "from langchain.schema.runnable import RunnableLambda, RunnableMap, RunnablePassthrough\n",
     "\n",
     "# 6. Create anonymizer chain\n",
     "template = \"\"\"Answer the question based only on the following context:\n",

docs/docs/guides/safety/amazon_comprehend_chain.ipynb

@@ -60,9 +60,10 @@
    },
    "outputs": [],
    "source": [
-    "import boto3\n",
     "import os\n",
     "\n",
+    "import boto3\n",
+    "\n",
     "comprehend_client = boto3.client(\"comprehend\", region_name=\"us-east-1\")"
    ]
   },
@@ -78,7 +79,8 @@
     "from langchain_experimental.comprehend_moderation import AmazonComprehendModerationChain\n",
     "\n",
     "comprehend_moderation = AmazonComprehendModerationChain(\n",
-    "    client=comprehend_client, verbose=True  # optional\n",
+    "    client=comprehend_client,\n",
+    "    verbose=True,  # optional\n",
     ")"
    ]
   },
@@ -101,8 +103,8 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.llms.fake import FakeListLLM\n",
+    "from langchain.prompts import PromptTemplate\n",
     "from langchain_experimental.comprehend_moderation.base_moderation_exceptions import (\n",
     "    ModerationPiiError,\n",
     ")\n",
@@ -173,8 +175,8 @@
    "source": [
     "from langchain_experimental.comprehend_moderation import (\n",
     "    BaseModerationConfig,\n",
-    "    ModerationPromptSafetyConfig,\n",
     "    ModerationPiiConfig,\n",
+    "    ModerationPromptSafetyConfig,\n",
     "    ModerationToxicityConfig,\n",
     ")\n",
     "\n",
@@ -238,8 +240,8 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.llms.fake import FakeListLLM\n",
+    "from langchain.prompts import PromptTemplate\n",
     "\n",
     "template = \"\"\"Question: {question}\n",
     "\n",
@@ -401,8 +403,8 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.llms.fake import FakeListLLM\n",
+    "from langchain.prompts import PromptTemplate\n",
     "\n",
     "template = \"\"\"Question: {question}\n",
     "\n",
@@ -690,10 +692,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import json\n",
+    "\n",
     "from langchain.llms import SagemakerEndpoint\n",
     "from langchain.llms.sagemaker_endpoint import LLMContentHandler\n",
     "from langchain.prompts import PromptTemplate\n",
-    "import json\n",
     "\n",
     "\n",
     "class ContentHandler(LLMContentHandler):\n",

docs/docs/guides/safety/hugging_face_prompt_injection.ipynb

@@ -144,8 +144,8 @@
     }
    ],
    "source": [
+    "from langchain.agents import AgentType, initialize_agent\n",
     "from langchain.llms import OpenAI\n",
-    "from langchain.agents import initialize_agent, AgentType\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "agent = initialize_agent(\n",

docs/docs/integrations/callbacks/argilla.ipynb

@@ -147,7 +147,7 @@
     "    api_key=os.environ[\"ARGILLA_API_KEY\"],\n",
     ")\n",
     "\n",
-    "dataset.push_to_argilla(\"langchain-dataset\");"
+    "dataset.push_to_argilla(\"langchain-dataset\")"
    ]
   },
   {
@@ -281,8 +281,8 @@
    ],
    "source": [
     "from langchain.callbacks import ArgillaCallbackHandler, StdOutCallbackHandler\n",
-    "from langchain.llms import OpenAI\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
     "\n",
     "argilla_callback = ArgillaCallbackHandler(\n",

docs/docs/integrations/callbacks/confident.ipynb

@@ -118,7 +118,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
     "from langchain.callbacks.confident_callback import DeepEvalCallbackHandler\n",
     "\n",
     "deepeval_callback = DeepEvalCallbackHandler(\n",

docs/docs/integrations/callbacks/context.ipynb

@@ -36,14 +36,10 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "vscode": {
-     "languageId": "shellscript"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "$ pip install context-python --upgrade"
+    "!pip install context-python --upgrade"
    ]
   },
   {
@@ -108,12 +104,12 @@
    "source": [
     "import os\n",
     "\n",
+    "from langchain.callbacks import ContextCallbackHandler\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
-    "    SystemMessage,\n",
     "    HumanMessage,\n",
+    "    SystemMessage,\n",
     ")\n",
-    "from langchain.callbacks import ContextCallbackHandler\n",
     "\n",
     "token = os.environ[\"CONTEXT_API_TOKEN\"]\n",
     "\n",
@@ -166,14 +162,14 @@
    "source": [
     "import os\n",
     "\n",
-    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.callbacks import ContextCallbackHandler\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
     "from langchain.prompts.chat import (\n",
     "    ChatPromptTemplate,\n",
     "    HumanMessagePromptTemplate,\n",
     ")\n",
-    "from langchain.callbacks import ContextCallbackHandler\n",
     "\n",
     "token = os.environ[\"CONTEXT_API_TOKEN\"]\n",
     "\n",