merge

.devcontainer/README.md

@@ -15,11 +15,7 @@ You may use the button above, or follow these steps to open this repo in a Codes
 For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
   
 ## VS Code Dev Containers
-[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/langchain-ai/langchain)
-
-Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name: 
-https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/<yourusername>/<yourclonedreponame>
-
+[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 
 If you already have VS Code and Docker installed, you can use the button above to get started. This will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
 
@@ -29,7 +25,7 @@ You can also follow these steps to open this repo in a container using the VS Co
 
 2. Open a locally cloned copy of the code:
 
-   - Fork and Clone this repository to your local filesystem.
+   - Clone this repository to your local filesystem.
    - Press <kbd>F1</kbd> and select the **Dev Containers: Open Folder in Container...** command.
    - Select the cloned copy of this folder, wait for the container to start, and try things out!
 

.github/workflows/_release.yml

@@ -37,7 +37,6 @@ jobs:
           echo version=$(poetry version --short) >> $GITHUB_OUTPUT
       - name: Create Release
         uses: ncipollo/release-action@v1
-        if: ${{ inputs.working-directory == 'libs/langchain' }}
         with:
           artifacts: "dist/*"
           token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/codespell.yml

@@ -20,5 +20,3 @@ jobs:
         uses: actions/checkout@v3
       - name: Codespell
         uses: codespell-project/actions-codespell@v2
-        with:
-          skip: guide_imports.json

.gitignore

@@ -162,7 +162,6 @@ docs/.docusaurus/
 docs/.cache-loader/
 docs/_dist
 docs/api_reference/api_reference.rst
-docs/api_reference/experimental_api_reference.rst
 docs/api_reference/_build
 docs/api_reference/*/
 !docs/api_reference/_static/

MIGRATE.md

@@ -43,10 +43,6 @@ Now:
 
 `from langchain_experimental.sql import SQLDatabaseChain`
 
-Alternatively, if you are just interested in using the query generation part of the SQL chain, you can check out [`create_sql_query_chain`](https://github.com/langchain-ai/langchain/blob/master/docs/extras/use_cases/tabular/sql_query.ipynb)
-
-`from langchain.chains import create_sql_query_chain`
-
 ## `load_prompt` for Python files
 
 Note: this only applies if you want to load Python files as prompts.

README.md

@@ -12,14 +12,14 @@
 [![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hwchase17/langchain)
 [![GitHub star chart](https://img.shields.io/github/stars/hwchase17/langchain?style=social)](https://star-history.com/#hwchase17/langchain)
-[![Dependency Status](https://img.shields.io/librariesio/github/langchain-ai/langchain)](https://libraries.io/github/langchain-ai/langchain)
+[![Dependency Status](https://img.shields.io/librariesio/github/hwchase17/langchain)](https://libraries.io/github/hwchase17/langchain)
 [![Open Issues](https://img.shields.io/github/issues-raw/hwchase17/langchain)](https://github.com/hwchase17/langchain/issues)
 
 
 Looking for the JS/TS version? Check out [LangChain.js](https://github.com/hwchase17/langchainjs).
 
 **Production Support:** As you move your LangChains into production, we'd love to offer more comprehensive support.
-Please fill out [this form](https://6w1pwbss0py.typeform.com/to/rrbrdTH2) and we'll set up a dedicated support Slack channel.
+Please fill out [this form](https://forms.gle/57d8AmXBYp8PP8tZA) and we'll set up a dedicated support Slack channel.
 
 ## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28
 

docs/.local_build.sh

@@ -13,6 +13,5 @@ cp -r {docs_skeleton,snippets} _dist
 cp -r extras/* _dist/docs_skeleton/docs
 cd _dist/docs_skeleton
 poetry run nbdoc_build
-poetry run python generate_api_reference_links.py
 yarn install
 yarn start

docs/api_reference/conf.py

@@ -7,67 +7,20 @@
 
 # -- Path setup --------------------------------------------------------------
 
-import json
-import os
-import sys
-from pathlib import Path
-
-import toml
-from docutils import nodes
-from sphinx.util.docutils import SphinxDirective
-
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+import toml
 
-_DIR = Path(__file__).parent.absolute()
 sys.path.insert(0, os.path.abspath("."))
 sys.path.insert(0, os.path.abspath("../../libs/langchain"))
-sys.path.insert(0, os.path.abspath("../../libs/experimental"))
 
-with (_DIR.parents[1] / "libs" / "langchain" / "pyproject.toml").open("r") as f:
+with open("../../libs/langchain/pyproject.toml") as f:
     data = toml.load(f)
-with (_DIR / "guide_imports.json").open("r") as f:
-    imported_classes = json.load(f)
-
-
-class ExampleLinksDirective(SphinxDirective):
-    """Directive to generate a list of links to examples.
-
-    We have a script that extracts links to API reference docs
-    from our notebook examples. This directive uses that information
-    to backlink to the examples from the API reference docs."""
-
-    has_content = False
-    required_arguments = 1
-
-    def run(self):
-        """Run the directive.
-
-        Called any time :example_links:`ClassName` is used
-        in the template *.rst files."""
-        class_or_func_name = self.arguments[0]
-        links = imported_classes.get(class_or_func_name, {})
-        list_node = nodes.bullet_list()
-        for doc_name, link in links.items():
-            item_node = nodes.list_item()
-            para_node = nodes.paragraph()
-            link_node = nodes.reference()
-            link_node["refuri"] = link
-            link_node.append(nodes.Text(doc_name))
-            para_node.append(link_node)
-            item_node.append(para_node)
-            list_node.append(item_node)
-        if list_node.children:
-            title_node = nodes.title()
-            title_node.append(nodes.Text(f"Examples using {class_or_func_name}"))
-            return [title_node, list_node]
-        return [list_node]
-
-
-def setup(app):
-    app.add_directive("example_links", ExampleLinksDirective)
-
 
 # -- Project information -----------------------------------------------------
 

docs/api_reference/create_api_rst.py

@@ -5,15 +5,13 @@ from pathlib import Path
 
 ROOT_DIR = Path(__file__).parents[2].absolute()
 PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
-EXP_DIR = ROOT_DIR / "libs" / "experimental" / "langchain_experimental"
 WRITE_FILE = Path(__file__).parent / "api_reference.rst"
-EXP_WRITE_FILE = Path(__file__).parent / "experimental_api_reference.rst"
 
 
-def load_members(dir: Path) -> dict:
+def load_members() -> dict:
     members: dict = {}
-    for py in glob.glob(str(dir) + "/**/*.py", recursive=True):
-        module = py[len(str(dir)) + 1 :].replace(".py", "").replace("/", ".")
+    for py in glob.glob(str(PKG_DIR) + "/**/*.py", recursive=True):
+        module = py[len(str(PKG_DIR)) + 1 :].replace(".py", "").replace("/", ".")
         top_level = module.split(".")[0]
         if top_level not in members:
             members[top_level] = {"classes": [], "functions": []}
@@ -28,10 +26,12 @@ def load_members(dir: Path) -> dict:
     return members
 
 
-def construct_doc(pkg: str, members: dict) -> str:
-    full_doc = f"""\
+def construct_doc(members: dict) -> str:
+    full_doc = """\
+.. _api_reference:
+
 =============
-``{pkg}`` API Reference
+API Reference
 =============
 
 """
@@ -40,12 +40,16 @@ def construct_doc(pkg: str, members: dict) -> str:
         functions = _members["functions"]
         if not (classes or functions):
             continue
-        section = f":mod:`{pkg}.{module}`"
+
+        module_title = module.replace("_", " ").title()
+        if module_title == "Llms":
+            module_title = "LLMs"
+        section = f":mod:`langchain.{module}`: {module_title}"
         full_doc += f"""\
 {section}
 {'=' * (len(section) + 1)}
 
-.. automodule:: {pkg}.{module}
+.. automodule:: langchain.{module}
     :no-members:
     :no-inherited-members:
 
@@ -56,7 +60,7 @@ def construct_doc(pkg: str, members: dict) -> str:
             full_doc += f"""\
 Classes
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
@@ -70,11 +74,10 @@ Classes
             full_doc += f"""\
 Functions
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
-    :template: function.rst
 
     {fstring}
 
@@ -83,14 +86,10 @@ Functions
 
 
 def main() -> None:
-    lc_members = load_members(PKG_DIR)
-    lc_doc = ".. _api_reference:\n\n" + construct_doc("langchain", lc_members)
+    members = load_members()
+    full_doc = construct_doc(members)
     with open(WRITE_FILE, "w") as f:
-        f.write(lc_doc)
-    exp_members = load_members(EXP_DIR)
-    exp_doc = ".. _experimental_api_reference:\n\n" + construct_doc("langchain_experimental", exp_members)
-    with open(EXP_WRITE_FILE, "w") as f:
-        f.write(exp_doc)
+        f.write(full_doc)
 
 
 if __name__ == "__main__":

docs/api_reference/modules/evaluation.rst

@@ -0,0 +1,9 @@
+Evaluation
+=======================
+
+LangChain has a number of convenient evaluation chains you can use off the shelf to grade your models' oupputs.
+
+.. automodule:: langchain.evaluation
+   :members:
+   :undoc-members:
+   :inherited-members:

docs/api_reference/requirements.txt

@@ -10,4 +10,6 @@ sphinx-panels
 toml
 myst_nb
 sphinx_copybutton
-pydata-sphinx-theme==0.13.1
+pydata-sphinx-theme==0.13.1
+nbdoc
+urllib3<2

docs/api_reference/templates/class.rst

@@ -26,5 +26,3 @@
    {%- endfor %}
    {% endif %}
    {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/templates/function.rst

@@ -1,8 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autofunction:: {{ objname }}
-
-.. example_links:: {{ objname }}

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -45,9 +45,6 @@
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
         </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('experimental_api_reference') }}">Experimental</a>
-        </li>
         <li class="nav-item">
           <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
         </li>

docs/api_reference/themes/scikit-learn-modern/static/css/theme.css

@@ -745,11 +745,6 @@ span.descname {
   background-color: transparent;
   padding: 0;
   font-family: monospace;
-  font-size: 1.2rem;
-}
-
-em.property {
-  font-weight: normal;
 }
 
 span.descclassname {

docs/docs_skeleton/_link_ungenerator.py

@@ -0,0 +1,44 @@
+import os
+import re
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+# Directory where generated markdown files are stored
+docs_dir = "docs_skeleton/docs"
+
+# Regular expression to match Python import lines
+link_re = re.compile(r"\[(from langchain.+?)\]\(.+?\)")
+
+
+def find_files(path):
+    """Find all MDX files in the given path"""
+    for root, _, files in os.walk(path):
+        for file in files:
+            if file.endswith(".mdx") or file.endswith(".md"):
+                yield os.path.join(root, file)
+
+
+def unreplace_imports(file):
+    """Replace import links in a markdown file with just the import text"""
+    with open(file, "r") as f:
+        data = f.read()
+
+    for match in link_re.finditer(data):
+        # Replace the markdown link with just the link text
+        data = data.replace(match.group(0), match.group(1))
+
+    with open(file, "w") as f:
+        f.write(data)
+
+
+def main():
+    """Main function"""
+    for file in find_files(docs_dir):
+        print(f"Removing links from imports in {file}")
+        unreplace_imports(file)
+
+
+if __name__ == "__main__":
+    main()

docs/docs_skeleton/docs/guides/evaluation/comparison/index.mdx

@@ -3,22 +3,6 @@ sidebar_position: 3
 ---
 # Comparison Evaluators
 
-Comparison evaluators in LangChain help measure two different chain or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.
-
-These evaluators inherit from the `PairwiseStringEvaluator` class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.
-
-To create a custom comparison evaluator, inherit from the `PairwiseStringEvaluator` class and overwrite the `_evaluate_string_pairs` method. If you require asynchronous evaluation, also overwrite the `_aevaluate_string_pairs` method.
-
-Here's a summary of the key methods and properties of a comparison evaluator:
-
-- `evaluate_string_pairs`: Evaluate the output string pairs. This function should be overwritten when creating custom evaluators.
-- `aevaluate_string_pairs`: Asynchronously evaluate the output string pairs. This function should be overwritten for asynchronous evaluation.
-- `requires_input`: This property indicates whether this evaluator requires an input string.
-- `requires_reference`: This property specifies whether this evaluator requires a reference label.
-
-Detailed information about creating custom evaluators and the available built-in comparison evaluators are provided in the following sections.
-
 import DocCardList from "@theme/DocCardList";
 
-<DocCardList />
-
+<DocCardList />

docs/docs_skeleton/docs/guides/evaluation/index.mdx

@@ -6,26 +6,23 @@ import DocCardList from "@theme/DocCardList";
 
 # Evaluation
 
-Building applications with language models involves many moving parts. One of the most critical components is ensuring that the outcomes produced by your models are reliable and useful across a broad array of inputs, and that they work well with your application's other software components. Ensuring reliability usually boils down to some combination of application design, testing & evaluation, and runtime checks. 
+Language models can be unpredictable. This makes it challenging to ship reliable applications to production, where repeatable, useful outcomes across diverse inputs are a minimum requirement. Tests help demonstrate each component in an LLM application can produce the required or expected functionality. These tests also safeguard against regressions while you improve interconnected pieces of an integrated system. However, measuring the quality of generated text can be challenging. It can be hard to agree on the right set of metrics for your application, and it can be difficult to translate those into better performance. Furthermore, it's common to lack sufficient evaluation data to adequately test the range of inputs and expected outputs for each component when you're just getting started. The LangChain community is building open source tools and guides to help address these challenges.
 
-The guides in this section review the APIs and functionality LangChain provides to help yous better evaluate your applications. Evaluation and testing are both critical when thinking about deploying LLM applications, since production environments require repeatable and useful outcomes.
+LangChain exposes different types of evaluators for common types of evaluation. Each type has off-the-shelf implementations you can use to get started, as well as an
+ extensible API so you can create your own or contribute improvements for everyone to use. The following sections have example notebooks for you to get started.
 
-LangChain offers various types of evaluators to help you measure performance and integrity on diverse data, and we hope to encourage the the community to create and share other useful evaluators so everyone can improve. These docs will introduce the evaluator types, how to use them, and provide some examples of their use in real-world scenarios.
+- [String Evaluators](/docs/guides/evaluation/string/): Evaluate the predicted string for a given input, usually against a reference string
+- [Trajectory Evaluators](/docs/guides/evaluation/trajectory/): Evaluate the whole trajectory of agent actions
+- [Comparison Evaluators](/docs/guides/evaluation/comparison/): Compare predictions from two runs on a common input
 
-Each evaluator type in LangChain comes with ready-to-use implementations and an extensible API that allows for customization according to your unique requirements. Here are some of the types of evaluators we offer:
 
-- [String Evaluators](/docs/guides/evaluation/string/): These evaluators assess the predicted string for a given input, usually comparing it against a reference string.
-- [Trajectory Evaluators](/docs/guides/evaluation/trajectory/): These are used to evaluate the entire trajectory of agent actions.
-- [Comparison Evaluators](/docs/guides/evaluation/comparison/): These evaluators are designed to compare predictions from two runs on a common input.
+This section also provides some additional examples of how you could use these evaluators for different scenarios or apply to different chain implementations in the LangChain library. Some examples include:
 
-These evaluators can be used across various scenarios and can be applied to different chain and LLM implementations in the LangChain library.
+- [Preference Scoring Chain Outputs](/docs/guides/evaluation/examples/comparisons): An example using a comparison evaluator on different models or prompts to select statistically significant differences in aggregate preference scores
 
-We also are working to share guides and cookbooks that demonstrate how to use these evaluators in real-world scenarios, such as:
-
-- [Chain Comparisons](/docs/guides/evaluation/examples/comparisons): This example uses a comparison evaluator to predict the preferred output. It reviews ways to measure confidence intervals to select statistically significant differences in aggregate preference scores across different models or prompts.
 
 ## Reference Docs
 
-For detailed information on the available evaluators, including how to instantiate, configure, and customize them, check out the [reference documentation](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.evaluation) directly.
+For detailed information of the available evaluators, including how to instantiate, configure, and customize them. Check out the [reference documentation](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.evaluation) directly.
 
 <DocCardList />

docs/docs_skeleton/docs/guides/evaluation/string/index.mdx

@@ -3,25 +3,6 @@ sidebar_position: 2
 ---
 # String Evaluators
 
-A string evaluator is a component within LangChain designed to assess the performance of a language model by comparing its generated outputs (predictions) to a reference string or an input. This comparison is a crucial step in the evaluation of language models, providing a measure of the accuracy or quality of the generated text.
-
-In practice, string evaluators are typically used to evaluate a predicted string against a given input, such as a question or a prompt. Often, a reference label or context string is provided to define what a correct or ideal response would look like. These evaluators can be customized to tailor the evaluation process to fit your application's specific requirements.
-
-To create a custom string evaluator, inherit from the `StringEvaluator` class and implement the `_evaluate_strings` method. If you require asynchronous support, also implement the `_aevaluate_strings` method.
-
-Here's a summary of the key attributes and methods associated with a string evaluator:
-
-- `evaluation_name`: Specifies the name of the evaluation.
-- `requires_input`: Boolean attribute that indicates whether the evaluator requires an input string. If True, the evaluator will raise an error when the input isn't provided. If False, a warning will be logged if an input _is_ provided, indicating that it will not be considered in the evaluation.
-- `requires_reference`: Boolean attribute specifying whether the evaluator requires a reference label. If True, the evaluator will raise an error when the reference isn't provided. If False, a warning will be logged if a reference _is_ provided, indicating that it will not be considered in the evaluation.
-
-String evaluators also implement the following methods:
-
-- `aevaluate_strings`: Asynchronously evaluates the output of the Chain or Language Model, with support for optional input and label.
-- `evaluate_strings`: Synchronously evaluates the output of the Chain or Language Model, with support for optional input and label.
-
-The following sections provide detailed information on available string evaluator implementations as well as how to create a custom string evaluator.
-
 import DocCardList from "@theme/DocCardList";
 
-<DocCardList />
+<DocCardList />

docs/docs_skeleton/docs/guides/evaluation/trajectory/index.mdx

@@ -3,26 +3,6 @@ sidebar_position: 4
 ---
 # Trajectory Evaluators
 
-Trajectory Evaluators in LangChain provide a more holistic approach to evaluating an agent. These evaluators assess the full sequence of actions taken by an agent and their corresponding responses, which we refer to as the "trajectory". This allows you to better measure an agent's effectiveness and capabilities.
-
-A Trajectory Evaluator implements the `AgentTrajectoryEvaluator` interface, which requires two main methods:
-
-- `evaluate_agent_trajectory`: This method synchronously evaluates an agent's trajectory.
-- `aevaluate_agent_trajectory`: This asynchronous counterpart allows evaluations to be run in parallel for efficiency.
-
-Both methods accept three main parameters:
-
-- `input`: The initial input given to the agent.
-- `prediction`: The final predicted response from the agent.
-- `agent_trajectory`: The intermediate steps taken by the agent, given as a list of tuples.
-
-These methods return a dictionary. It is recommended that custom implementations return a `score` (a float indicating the effectiveness of the agent) and `reasoning` (a string explaining the reasoning behind the score).
-
-You can capture an agent's trajectory by initializing the agent with the `return_intermediate_steps=True` parameter. This lets you collect all intermediate steps without relying on special callbacks.
-
-For a deeper dive into the implementation and use of Trajectory Evaluators, refer to the sections below.
-
 import DocCardList from "@theme/DocCardList";
 
-<DocCardList />
-
+<DocCardList />

docs/docs_skeleton/docs/guides/expression_language/index.mdx

@@ -1,9 +0,0 @@
-# LangChain Expression Language
-
-import DocCardList from "@theme/DocCardList";
-
-LangChain Expression Language is a declarative way to easily compose chains together.
-Any chain constructed this way will automatically have full sync, async, and streaming support.
-See guides below for how to interact with chains constructed this way as well as cookbook examples.
-
-<DocCardList />

docs/docs_skeleton/docs/guides/safety/index.mdx

@@ -1,6 +0,0 @@
-# Preventing harmful outputs
-
-One of the key concerns with using LLMs is that they may generate harmful or unethical text. This is an area of active research in the field. Here we present some built-in chains inspired by this research, which are intended to make the outputs of LLMs safer.
-
-- [Moderation chain](/docs/use_cases/safety/moderation): Explicitly check if any output text is harmful and flag it.
-- [Constitutional chain](/docs/use_cases/safety/constitutional_chain): Prompt the model with a set of principles which should guide it's behavior.

docs/docs_skeleton/docs/modules/chains/additional/index.mdx

@@ -0,0 +1,8 @@
+---
+sidebar_position: 4
+---
+# Additional
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/chains/additional/multi_prompt_router.mdx

@@ -0,0 +1,7 @@
+# Dynamically selecting from multiple prompts
+
+This notebook demonstrates how to use the `RouterChain` paradigm to create a chain that dynamically selects the prompt to use for a given input. Specifically we show how to use the `MultiPromptChain` to create a question-answering chain that selects the prompt which is most relevant for a given question, and then answers the question using that prompt.
+
+import Example from "@snippets/modules/chains/additional/multi_prompt_router.mdx"
+
+<Example/>

docs/docs_skeleton/docs/modules/chains/additional/multi_retrieval_qa_router.mdx

@@ -1,4 +1,4 @@
-# Dynamically select from multiple retrievers
+# Dynamically selecting from multiple retrievers
 
 This notebook demonstrates how to use the `RouterChain` paradigm to create a chain that dynamically selects which Retrieval system to use. Specifically we show how to use the `MultiRetrievalQAChain` to create a question-answering chain that selects the retrieval QA chain which is most relevant for a given question, and then answers the question using it.
 

docs/docs_skeleton/docs/modules/chains/additional/question_answering.mdx

@@ -1,4 +1,4 @@
-# QA over in-memory documents
+# Document QA
 
 Here we walk through how to use LangChain for question answering over a list of documents. Under the hood we'll be using our [Document chains](/docs/modules/chains/document/).
 

docs/docs_skeleton/docs/modules/chains/foundational/sequential_chains.mdx

@@ -1,6 +1,6 @@
 # Sequential
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 The next step after calling a language model is make a series of calls to a language model. This is particularly useful when you want to take the output from one call and use it as the input to another.
 

docs/docs_skeleton/docs/modules/chains/popular/chat_vector_db.mdx

@@ -2,7 +2,7 @@
 sidebar_position: 2
 ---
 
-# Store and reference chat history
+# Conversational Retrieval QA
 The ConversationalRetrievalQA chain builds on RetrievalQAChain to provide a chat history component.
 
 It first combines the chat history (either explicitly passed in or retrieved from the provided memory) and the question into a standalone question, then looks up relevant documents from the retriever, and finally passes those documents and the question to a question answering chain to return a response.

docs/docs_skeleton/docs/modules/chains/popular/index.mdx

@@ -0,0 +1,8 @@
+---
+sidebar_position: 3
+---
+# Popular
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/chains/popular/vector_db_qa.mdx

@@ -1,7 +1,7 @@
 ---
 sidebar_position: 1
 ---
-# QA using a Retriever
+# Retrieval QA
 
 This example showcases question answering over an index.
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/integrations/_category_.yml

@@ -0,0 +1 @@
+label: 'Integrations'

docs/docs_skeleton/docs/modules/memory/buffer.mdx

@@ -4,6 +4,6 @@ This notebook shows how to use `ConversationBufferMemory`. This memory allows fo
 
 We can first extract it as a string.
 
-import Example from "@snippets/modules/memory/types/buffer.mdx"
+import Example from "@snippets/modules/memory/how_to/buffer.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/buffer_window.mdx

@@ -4,6 +4,6 @@
 
 Let's first explore the basic functionality of this type of memory.
 
-import Example from "@snippets/modules/memory/types/buffer_window.mdx"
+import Example from "@snippets/modules/memory/how_to/buffer_window.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/chat_messages/index.mdx

@@ -1,17 +0,0 @@
----
-sidebar_position: 1
----
-# Chat Messages
-
-:::info
-Head to [Integrations](/docs/integrations/memory/) for documentation on built-in memory integrations with 3rd-party databases and tools.
-:::
-
-One of the core utility classes underpinning most (if not all) memory modules is the `ChatMessageHistory` class.
-This is a super lightweight wrapper which exposes convenience methods for saving Human messages, AI messages, and then fetching them all.
-
-You may want to use this class directly if you are managing memory outside of a chain.
-
-import GetStarted from "@snippets/modules/memory/chat_messages/get_started.mdx"
-
-<GetStarted/>

docs/docs_skeleton/docs/modules/memory/entity_summary_memory.mdx

@@ -4,6 +4,6 @@ Entity Memory remembers given facts about specific entities in a conversation. I
 
 Let's first walk through using this functionality.
 
-import Example from "@snippets/modules/memory/types/entity_summary_memory.mdx"
+import Example from "@snippets/modules/memory/how_to/entity_summary_memory.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/index.mdx

@@ -1,62 +1,34 @@
 ---
 sidebar_position: 3
 ---
+
 # Memory
 
-Most LLM applications have a conversational interface. An essential component of a conversation is being able to refer to information introduced earlier in the conversation.
-At bare minimum, a conversational system should be able to access some window of past messages directly.
-A more complex system will need to have a world model that it is constantly updating, which allows it to do things like maintain information about entities and their relationships.
+🚧 _Docs under construction_ 🚧
 
-We call this ability to store information about past interactions "memory".
-LangChain provides a lot of utilities for adding memory to a system.
-These utilities can be used by themselves or incorporated seamlessly into a chain.
+:::info
+Head to [Integrations](/docs/integrations/memory/) for documentation on built-in memory integrations with 3rd-party tools.
+:::
 
-A memory system needs to support two basic actions: reading and writing.
-Recall that every chain defines some core execution logic that expects certain inputs.
-Some of these inputs come directly from the user, but some of these inputs can come from memory.
-A chain will interact with its memory system twice in a given run.
-1. AFTER receiving the initial user inputs but BEFORE executing the core logic, a chain will READ from its memory system and augment the user inputs.
-2. AFTER executing the core logic but BEFORE returning the answer, a chain will WRITE the inputs and outputs of the current run to memory, so that they can be referred to in future runs.
+By default, Chains and Agents are stateless,
+meaning that they treat each incoming query independently (like the underlying LLMs and chat models themselves).
+In some applications, like chatbots, it is essential
+to remember previous interactions, both in the short and long-term.
+The **Memory** class does exactly that.
 
-![memory-diagram](/img/memory_diagram.png)
-
-
-## Building memory into a system
-The two core design decisions in any memory system are:
-- How state is stored
-- How state is queried
-
-### Storing: List of chat messages
-Underlying any memory is a history of all chat interactions.
-Even if these are not all used directly, they need to be stored in some form.
-One of the key parts of the LangChain memory module is a series of integrations for storing these chat messages,
-from in-memory lists to persistent databases.
-
-- [Chat message storage](/docs/modules/memory/chat_messages/): How to work with Chat Messages, and the various integrations offered
-
-### Querying: Data structures and algorithms on top of chat messages
-Keeping a list of chat messages is fairly straight-forward.
-What is less straight-forward are the data structures and algorithms built on top of chat messages that serve a view of those messages that is most useful.
-
-A very simply memory system might just return the most recent messages each run. A slightly more complex memory system might return a succinct summary of the past K messages.
-An even more sophisticated system might extract entities from stored messages and only return information about entities referenced in the current run.
-
-Each application can have different requirements for how memory is queried. The memory module should make it easy to both get started with simple memory systems and write your own custom systems if needed.
-
-- [Memory types](/docs/modules/memory/types/): The various data structures and algorithms that make up the memory types LangChain supports
+LangChain provides memory components in two forms.
+First, LangChain provides helper utilities for managing and manipulating previous chat messages.
+These are designed to be modular and useful regardless of how they are used.
+Secondly, LangChain provides easy ways to incorporate these utilities into chains.
 
 ## Get started
 
-Let's take a look at what Memory actually looks like in LangChain.
-Here we'll cover the basics of interacting with an arbitrary memory class.
+Memory involves keeping a concept of state around throughout a user's interactions with an language model. A user's interactions with a language model are captured in the concept of ChatMessages, so this boils down to ingesting, capturing, transforming and extracting knowledge from a sequence of chat messages. There are many different ways to do this, each of which exists as its own memory type.
+
+In general, for each type of memory there are two ways to understanding using memory. These are the standalone functions which extract information from a sequence of messages, and then there is the way you can use this type of memory in a chain.
+
+Memory can return multiple pieces of information (for example, the most recent N messages and a summary of all previous messages). The returned information can either be a string or a list of messages.
 
 import GetStarted from "@snippets/modules/memory/get_started.mdx"
 
 <GetStarted/>
-
-## Next steps
-
-And that's it for getting started!
-Please see the other sections for walkthroughs of more advanced topics,
-like custom memory, multiple memories, and more.
-

docs/docs_skeleton/docs/modules/memory/summary.mdx

@@ -4,6 +4,6 @@ Conversation summary memory summarizes the conversation as it happens and stores
 
 Let's first explore the basic functionality of this type of memory.
 
-import Example from "@snippets/modules/memory/types/summary.mdx"
+import Example from "@snippets/modules/memory/how_to/summary.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/types/index.mdx

@@ -1,8 +0,0 @@
----
-sidebar_position: 2
----
-# Memory Types
-
-There are many different types of memory.
-Each have their own parameters, their own return types, and are useful in different scenarios.
-Please see their individual page for more detail on each one.

docs/docs_skeleton/docs/modules/memory/vectorstore_retriever_memory.mdx

@@ -6,6 +6,6 @@ This differs from most of the other Memory classes in that it doesn't explicitly
 
 In this case, the "docs" are previous conversation snippets. This can be useful to refer to relevant pieces of information that the AI was told earlier in the conversation.
 
-import Example from "@snippets/modules/memory/types/vectorstore_retriever_memory.mdx"
+import Example from "@snippets/modules/memory/how_to/vectorstore_retriever_memory.mdx"
 
 <Example/>

docs/docs_skeleton/docs/use_cases/question_answering/how_to/_category_.yml

@@ -1 +0,0 @@
-label: 'How to'

docs/docs_skeleton/generate_api_reference_links.py

@@ -1,183 +0,0 @@
-import importlib
-import inspect
-import json
-import logging
-import os
-import re
-from pathlib import Path
-import argparse
-
-logging.basicConfig(level=logging.INFO)
-logger = logging.getLogger(__name__)
-# Base URL for all class documentation
-_BASE_URL = "https://api.python.langchain.com/en/latest/"
-
-# Regular expression to match Python code blocks
-code_block_re = re.compile(r"^(```python\n)(.*?)(```\n)", re.DOTALL | re.MULTILINE)
-# Regular expression to match langchain import lines
-_IMPORT_RE = re.compile(
-    r"from\s+(langchain\.\w+(\.\w+)*?)\s+import\s+"
-    r"((?:\w+(?:,\s*)?)*"  # Match zero or more words separated by a comma+optional ws
-    r"(?:\s*\(.*?\))?)",  # Match optional parentheses block
-    re.DOTALL,  # Match newlines as well
-)
-
-_CURRENT_PATH = Path(__file__).parent.absolute()
-# Directory where generated markdown files are stored
-_DOCS_DIR = _CURRENT_PATH / "docs"
-_JSON_PATH = _CURRENT_PATH.parent / "api_reference" / "guide_imports.json"
-
-
-def find_files(path):
-    """Find all MDX files in the given path"""
-    # Check if is file first
-    if os.path.isfile(path):
-        yield path
-        return
-    for root, _, files in os.walk(path):
-        for file in files:
-            if file.endswith(".mdx") or file.endswith(".md"):
-                yield os.path.join(root, file)
-
-
-def get_full_module_name(module_path, class_name):
-    """Get full module name using inspect"""
-    module = importlib.import_module(module_path)
-    class_ = getattr(module, class_name)
-    return inspect.getmodule(class_).__name__
-
-
-def get_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "--docs_dir",
-        type=str,
-        default=_DOCS_DIR,
-        help="Directory where generated markdown files are stored",
-    )
-    return parser.parse_args()
-
-
-def main():
-    """Main function"""
-    args = get_args()
-    global_imports = {}
-
-    for file in find_files(args.docs_dir):
-        print(f"Adding links for imports in {file}")
-        file_imports = replace_imports(file)
-
-        if file_imports:
-            # Use relative file path as key
-            relative_path = (
-                os.path.relpath(file, _DOCS_DIR).replace(".mdx", "").replace(".md", "")
-            )
-
-            doc_url = f"https://python.langchain.com/docs/{relative_path}"
-            for import_info in file_imports:
-                doc_title = import_info["title"]
-                class_name = import_info["imported"]
-                if class_name not in global_imports:
-                    global_imports[class_name] = {}
-                global_imports[class_name][doc_title] = doc_url
-
-    # Write the global imports information to a JSON file
-    _JSON_PATH.parent.mkdir(parents=True, exist_ok=True)
-    with _JSON_PATH.open("w") as f:
-        json.dump(global_imports, f)
-
-
-def _get_doc_title(data: str, file_name: str) -> str:
-    try:
-        return re.findall(r"^#\s+(.*)", data, re.MULTILINE)[0]
-    except IndexError:
-        pass
-    # Parse the rst-style titles
-    try:
-        return re.findall(r"^(.*)\n=+\n", data, re.MULTILINE)[0]
-    except IndexError:
-        return file_name
-
-
-def replace_imports(file):
-    """Replace imports in each Python code block with links to their
-    documentation and append the import info in a comment"""
-    all_imports = []
-    with open(file, "r") as f:
-        data = f.read()
-
-    file_name = os.path.basename(file)
-    _DOC_TITLE = _get_doc_title(data, file_name)
-
-    def replacer(match):
-        # Extract the code block content
-        code = match.group(2)
-        # Replace if any import comment exists
-        # TODO: Use our own custom <code> component rather than this
-        # injection method
-        existing_comment_re = re.compile(r"^<!--IMPORTS:.*?-->\n", re.MULTILINE)
-        code = existing_comment_re.sub("", code)
-
-        # Process imports in the code block
-        imports = []
-        for import_match in _IMPORT_RE.finditer(code):
-            module = import_match.group(1)
-            imports_str = (
-                import_match.group(3).replace("(\n", "").replace("\n)", "")
-            )  # Handle newlines within parentheses
-            # remove any newline and spaces, then split by comma
-            imported_classes = [
-                imp.strip()
-                for imp in re.split(r",\s*", imports_str.replace("\n", ""))
-                if imp.strip()
-            ]
-            for class_name in imported_classes:
-                try:
-                    module_path = get_full_module_name(module, class_name)
-                except AttributeError as e:
-                    logger.warning(f"Could not find module for {class_name}, {e}")
-                    continue
-                except ImportError as e:
-                    logger.warning(f"Failed to load for class {class_name}, {e}")
-                    continue
-
-                url = (
-                    _BASE_URL
-                    + module_path.split(".")[1]
-                    + "/"
-                    + module_path
-                    + "."
-                    + class_name
-                    + ".html"
-                )
-
-                # Add the import information to our list
-                imports.append(
-                    {
-                        "imported": class_name,
-                        "source": module,
-                        "docs": url,
-                        "title": _DOC_TITLE,
-                    }
-                )
-
-        if imports:
-            all_imports.extend(imports)
-            # Create a unique comment containing the import information
-            import_comment = f"<!--IMPORTS:{json.dumps(imports)}-->"
-            # Inject the import comment at the start of the code block
-            return match.group(1) + import_comment + "\n" + code + match.group(3)
-        else:
-            # If there are no imports, return the original match
-            return match.group(0)
-
-    # Use re.sub to replace each Python code block
-    data = code_block_re.sub(replacer, data)
-
-    with open(file, "w") as f:
-        f.write(data)
-    return all_imports
-
-
-if __name__ == "__main__":
-    main()

docs/docs_skeleton/link_generator.py

@@ -0,0 +1,75 @@
+import importlib
+import inspect
+import logging
+import os
+import re
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+# Base URL for all class documentation
+base_url = "https://api.python.langchain.com/en/latest/"
+
+# Directory where generated markdown files are stored
+docs_dir = "docs_skeleton/docs"
+
+# Regular expression to match Python import lines
+import_re = re.compile(r"(from\s+(langchain\.\w+(\.\w+)*?)\s+import\s+)(\w+)")
+
+
+def find_files(path):
+    """Find all MDX files in the given path"""
+    for root, _, files in os.walk(path):
+        for file in files:
+            if file.endswith(".mdx") or file.endswith(".md"):
+                yield os.path.join(root, file)
+
+
+def get_full_module_name(module_path, class_name):
+    """Get full module name using inspect"""
+    module = importlib.import_module(module_path)
+    class_ = getattr(module, class_name)
+    return inspect.getmodule(class_).__name__
+
+
+def replace_imports(file):
+    """Replace imports in a markdown file with links to their documentation"""
+    with open(file, "r") as f:
+        data = f.read()
+
+    for match in import_re.finditer(data):
+        # Create the URL for this class's documentation
+        class_name = match.group(4)
+        try:
+            module_path = get_full_module_name(match.group(2), class_name)
+        except AttributeError as e:
+            logger.error(f"Could not find module for {class_name}", e)
+            continue
+        module_path_parts = module_path.replace(".", "/")
+
+        url = (
+            base_url
+            + module_path_parts
+            + "/"
+            + module_path_parts
+            + "."
+            + class_name
+            + ".html"
+        )
+
+        # Replace the class name with a markdown link
+        link = f"[{match.group(1)}{class_name}]({url})"
+        data = data.replace(match.group(0), f"{match.group(0)}\n{link}")
+
+    with open(file, "w") as f:
+        f.write(data)
+
+
+def main():
+    """Main function"""
+    for file in find_files(docs_dir):
+        print(f"Adding links for imports in {file}")
+        replace_imports(file)
+
+
+if __name__ == "__main__":
+    main()

docs/docs_skeleton/settings.ini

@@ -5,7 +5,7 @@ tst_flags = notest
 user = hwchase17
 doc_host = https://python.langchain.com
 doc_baseurl = /docs
-module_baseurls = metaflow=https://github.com/Netflix/metaflow/tree/master/
+module_baseurls = langchain=https://github.com/hwchase17/langchain/tree/master
 	fastcore=https://github.com/fastcore/tree/master
 host = github
 

docs/docs_skeleton/src/theme/CodeBlock/Container/index.js

@@ -0,0 +1,21 @@
+import React from 'react';
+import clsx from 'clsx';
+import {ThemeClassNames, usePrismTheme} from '@docusaurus/theme-common';
+import {getPrismCssVariables} from '@docusaurus/theme-common/internal';
+import styles from './styles.module.css';
+export default function CodeBlockContainer({as: As, ...props}) {
+  const prismTheme = usePrismTheme();
+  const prismCssVariables = getPrismCssVariables(prismTheme);
+  return (
+    <As
+      // Polymorphic components are hard to type, without `oneOf` generics
+      {...props}
+      style={prismCssVariables}
+      className={clsx(
+        props.className,
+        styles.codeBlockContainer,
+        ThemeClassNames.common.codeBlock,
+      )}
+    />
+  );
+}

docs/docs_skeleton/src/theme/CodeBlock/Container/styles.module.css

@@ -0,0 +1,7 @@
+.codeBlockContainer {
+  background: var(--prism-background-color);
+  color: var(--prism-color);
+  margin-bottom: var(--ifm-leading);
+  box-shadow: var(--ifm-global-shadow-lw);
+  border-radius: var(--ifm-code-border-radius);
+}

docs/docs_skeleton/src/theme/CodeBlock/Content/Element.js

@@ -0,0 +1,17 @@
+import React from 'react';
+import clsx from 'clsx';
+import Container from '@theme/CodeBlock/Container';
+import styles from './styles.module.css';
+// <pre> tags in markdown map to CodeBlocks. They may contain JSX children. When
+// the children is not a simple string, we just return a styled block without
+// actually highlighting.
+export default function CodeBlockJSX({children, className}) {
+  return (
+    <Container
+      as="pre"
+      tabIndex={0}
+      className={clsx(styles.codeBlockStandalone, 'thin-scrollbar', className)}>
+      <code className={styles.codeBlockLines}>{children}</code>
+    </Container>
+  );
+}

docs/docs_skeleton/src/theme/CodeBlock/Content/String.js

@@ -0,0 +1,97 @@
+import React from 'react';
+import clsx from 'clsx';
+import {useThemeConfig, usePrismTheme} from '@docusaurus/theme-common';
+import {
+  parseCodeBlockTitle,
+  parseLanguage,
+  parseLines,
+  containsLineNumbers,
+  useCodeWordWrap,
+} from '@docusaurus/theme-common/internal';
+import Highlight, {defaultProps} from 'prism-react-renderer';
+import Line from '@theme/CodeBlock/Line';
+import CopyButton from '@theme/CodeBlock/CopyButton';
+import WordWrapButton from '@theme/CodeBlock/WordWrapButton';
+import Container from '@theme/CodeBlock/Container';
+import styles from './styles.module.css';
+export default function CodeBlockString({
+  children,
+  className: blockClassName = '',
+  metastring,
+  title: titleProp,
+  showLineNumbers: showLineNumbersProp,
+  language: languageProp,
+}) {
+  const {
+    prism: {defaultLanguage, magicComments},
+  } = useThemeConfig();
+  const language =
+    languageProp ?? parseLanguage(blockClassName) ?? defaultLanguage;
+  const prismTheme = usePrismTheme();
+  const wordWrap = useCodeWordWrap();
+  // We still parse the metastring in case we want to support more syntax in the
+  // future. Note that MDX doesn't strip quotes when parsing metastring:
+  // "title=\"xyz\"" => title: "\"xyz\""
+  const title = parseCodeBlockTitle(metastring) || titleProp;
+  const {lineClassNames, code} = parseLines(children, {
+    metastring,
+    language,
+    magicComments,
+  });
+  const showLineNumbers =
+    showLineNumbersProp ?? containsLineNumbers(metastring);
+  return (
+    <Container
+      as="div"
+      className={clsx(
+        blockClassName,
+        language &&
+          !blockClassName.includes(`language-${language}`) &&
+          `language-${language}`,
+      )}>
+      {title && <div className={styles.codeBlockTitle}>{title}</div>}
+      <div className={styles.codeBlockContent}>
+        <Highlight
+          {...defaultProps}
+          theme={prismTheme}
+          code={code}
+          language={language ?? 'text'}>
+          {({className, tokens, getLineProps, getTokenProps}) => (
+            <pre
+              /* eslint-disable-next-line jsx-a11y/no-noninteractive-tabindex */
+              tabIndex={0}
+              ref={wordWrap.codeBlockRef}
+              className={clsx(className, styles.codeBlock, 'thin-scrollbar')}>
+              <code
+                className={clsx(
+                  styles.codeBlockLines,
+                  showLineNumbers && styles.codeBlockLinesWithNumbering,
+                )}>
+                {tokens.map((line, i) => (
+                  <Line
+                    key={i}
+                    line={line}
+                    getLineProps={getLineProps}
+                    getTokenProps={getTokenProps}
+                    classNames={lineClassNames[i]}
+                    showLineNumbers={showLineNumbers}
+                  />
+                ))}
+              </code>
+            </pre>
+          )}
+        </Highlight>
+        <div className={styles.buttonGroup}>
+          {(wordWrap.isEnabled || wordWrap.isCodeScrollable) && (
+            <WordWrapButton
+              className={styles.codeButton}
+              onClick={() => wordWrap.toggle()}
+              isEnabled={wordWrap.isEnabled}
+            />
+          )}
+          <CopyButton className={styles.codeButton} code={code} />
+        </div>
+      </div>
+    </Container>
+  );
+}

docs/docs_skeleton/src/theme/CodeBlock/Content/styles.module.css

@@ -0,0 +1,80 @@
+.codeBlockContent {
+  position: relative;
+  /* rtl:ignore */
+  direction: ltr;
+  border-radius: inherit;
+}
+
+.codeBlockTitle {
+  border-bottom: 1px solid var(--ifm-color-emphasis-300);
+  font-size: var(--ifm-code-font-size);
+  font-weight: 500;
+  padding: 0.75rem var(--ifm-pre-padding);
+  border-top-left-radius: inherit;
+  border-top-right-radius: inherit;
+}
+
+.codeBlock {
+  --ifm-pre-background: var(--prism-background-color);
+  margin: 0;
+  padding: 0;
+}
+
+.codeBlockTitle + .codeBlockContent .codeBlock {
+  border-top-left-radius: 0;
+  border-top-right-radius: 0;
+}
+
+.codeBlockStandalone {
+  padding: 0;
+}
+
+.codeBlockLines {
+  font: inherit;
+  /* rtl:ignore */
+  float: left;
+  min-width: 100%;
+  padding: var(--ifm-pre-padding);
+}
+
+.codeBlockLinesWithNumbering {
+  display: table;
+  padding: var(--ifm-pre-padding) 0;
+}
+
+@media print {
+  .codeBlockLines {
+    white-space: pre-wrap;
+  }
+}
+
+.buttonGroup {
+  display: flex;
+  column-gap: 0.2rem;
+  position: absolute;
+  /* rtl:ignore */
+  right: calc(var(--ifm-pre-padding) / 2);
+  top: calc(var(--ifm-pre-padding) / 2);
+}
+
+.buttonGroup button {
+  display: flex;
+  align-items: center;
+  background: var(--prism-background-color);
+  color: var(--prism-color);
+  border: 1px solid var(--ifm-color-emphasis-300);
+  border-radius: var(--ifm-global-radius);
+  padding: 0.4rem;
+  line-height: 0;
+  transition: opacity var(--ifm-transition-fast) ease-in-out;
+  opacity: 0;
+}
+
+.buttonGroup button:focus-visible,
+.buttonGroup button:hover {
+  opacity: 1 !important;
+}
+
+:global(.theme-code-block:hover) .buttonGroup button {
+  opacity: 0.4;
+}

docs/docs_skeleton/src/theme/CodeBlock/CopyButton/index.js

@@ -0,0 +1,56 @@
+import React, {useCallback, useState, useRef, useEffect} from 'react';
+import clsx from 'clsx';
+// @ts-expect-error: TODO, we need to make theme-classic have type: module
+import copy from 'copy-text-to-clipboard';
+import {translate} from '@docusaurus/Translate';
+import styles from './styles.module.css';
+export default function CopyButton({code, className}) {
+  const [isCopied, setIsCopied] = useState(false);
+  const copyTimeout = useRef(undefined);
+  const handleCopyCode = useCallback(() => {
+    copy(code);
+    setIsCopied(true);
+    copyTimeout.current = window.setTimeout(() => {
+      setIsCopied(false);
+    }, 1000);
+  }, [code]);
+  useEffect(() => () => window.clearTimeout(copyTimeout.current), []);
+  return (
+    <button
+      type="button"
+      aria-label={
+        isCopied
+          ? translate({
+              id: 'theme.CodeBlock.copied',
+              message: 'Copied',
+              description: 'The copied button label on code blocks',
+            })
+          : translate({
+              id: 'theme.CodeBlock.copyButtonAriaLabel',
+              message: 'Copy code to clipboard',
+              description: 'The ARIA label for copy code blocks button',
+            })
+      }
+      title={translate({
+        id: 'theme.CodeBlock.copy',
+        message: 'Copy',
+        description: 'The copy button label on code blocks',
+      })}
+      className={clsx(
+        'clean-btn',
+        className,
+        styles.copyButton,
+        isCopied && styles.copyButtonCopied,
+      )}
+      onClick={handleCopyCode}>
+      <span className={styles.copyButtonIcons} aria-hidden="true">
+        <svg className={styles.copyButtonIcon} viewBox="0 0 24 24">
+          <path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z" />
+        </svg>
+        <svg className={styles.copyButtonSuccessIcon} viewBox="0 0 24 24">
+          <path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z" />
+        </svg>
+      </span>
+    </button>
+  );
+}

docs/docs_skeleton/src/theme/CodeBlock/CopyButton/styles.module.css

@@ -0,0 +1,40 @@
+:global(.theme-code-block:hover) .copyButtonCopied {
+  opacity: 1 !important;
+}
+
+.copyButtonIcons {
+  position: relative;
+  width: 1.125rem;
+  height: 1.125rem;
+}
+
+.copyButtonIcon,
+.copyButtonSuccessIcon {
+  position: absolute;
+  top: 0;
+  left: 0;
+  fill: currentColor;
+  opacity: inherit;
+  width: inherit;
+  height: inherit;
+  transition: all var(--ifm-transition-fast) ease;
+}
+
+.copyButtonSuccessIcon {
+  top: 50%;
+  left: 50%;
+  transform: translate(-50%, -50%) scale(0.33);
+  opacity: 0;
+  color: #00d600;
+}
+
+.copyButtonCopied .copyButtonIcon {
+  transform: scale(0.33);
+  opacity: 0;
+}
+
+.copyButtonCopied .copyButtonSuccessIcon {
+  transform: translate(-50%, -50%) scale(1);
+  opacity: 1;
+  transition-delay: 0.075s;
+}

docs/docs_skeleton/src/theme/CodeBlock/Line/index.js

@@ -0,0 +1,34 @@
+import React from 'react';
+import clsx from 'clsx';
+import styles from './styles.module.css';
+export default function CodeBlockLine({
+  line,
+  classNames,
+  showLineNumbers,
+  getLineProps,
+  getTokenProps,
+}) {
+  if (line.length === 1 && line[0].content === '\n') {
+    line[0].content = '';
+  }
+  const lineProps = getLineProps({
+    line,
+    className: clsx(classNames, showLineNumbers && styles.codeLine),
+  });
+  const lineTokens = line.map((token, key) => (
+    <span key={key} {...getTokenProps({token, key})} />
+  ));
+  return (
+    <span {...lineProps}>
+      {showLineNumbers ? (
+        <>
+          <span className={styles.codeLineNumber} />
+          <span className={styles.codeLineContent}>{lineTokens}</span>
+        </>
+      ) : (
+        lineTokens
+      )}
+      <br />
+    </span>
+  );
+}

docs/docs_skeleton/src/theme/CodeBlock/Line/styles.module.css

@@ -0,0 +1,45 @@
+/* Intentionally has zero specificity, so that to be able to override
+the background in custom CSS file due bug https://github.com/facebook/docusaurus/issues/3678 */
+:where(:root) {
+  --docusaurus-highlighted-code-line-bg: rgb(72 77 91);
+}
+
+:where([data-theme='dark']) {
+  --docusaurus-highlighted-code-line-bg: rgb(100 100 100);
+}
+
+:global(.theme-code-block-highlighted-line) {
+  background-color: var(--docusaurus-highlighted-code-line-bg);
+  display: block;
+  margin: 0 calc(-1 * var(--ifm-pre-padding));
+  padding: 0 var(--ifm-pre-padding);
+}
+
+.codeLine {
+  display: table-row;
+  counter-increment: line-count;
+}
+
+.codeLineNumber {
+  display: table-cell;
+  text-align: right;
+  width: 1%;
+  position: sticky;
+  left: 0;
+  padding: 0 var(--ifm-pre-padding);
+  background: var(--ifm-pre-background);
+  overflow-wrap: normal;
+}
+
+.codeLineNumber::before {
+  content: counter(line-count);
+  opacity: 0.4;
+}
+
+:global(.theme-code-block-highlighted-line) .codeLineNumber::before {
+  opacity: 0.8;
+}
+
+.codeLineContent {
+  padding-right: var(--ifm-pre-padding);
+}

docs/docs_skeleton/src/theme/CodeBlock/WordWrapButton/index.js

@@ -0,0 +1,34 @@
+import React from 'react';
+import clsx from 'clsx';
+import {translate} from '@docusaurus/Translate';
+import styles from './styles.module.css';
+export default function WordWrapButton({className, onClick, isEnabled}) {
+  const title = translate({
+    id: 'theme.CodeBlock.wordWrapToggle',
+    message: 'Toggle word wrap',
+    description:
+      'The title attribute for toggle word wrapping button of code block lines',
+  });
+  return (
+    <button
+      type="button"
+      onClick={onClick}
+      className={clsx(
+        'clean-btn',
+        className,
+        isEnabled && styles.wordWrapButtonEnabled,
+      )}
+      aria-label={title}
+      title={title}>
+      <svg
+        className={styles.wordWrapButtonIcon}
+        viewBox="0 0 24 24"
+        aria-hidden="true">
+        <path
+          fill="currentColor"
+          d="M4 19h6v-2H4v2zM20 5H4v2h16V5zm-3 6H4v2h13.25c1.1 0 2 .9 2 2s-.9 2-2 2H15v-2l-3 3l3 3v-2h2c2.21 0 4-1.79 4-4s-1.79-4-4-4z"
+        />
+      </svg>
+    </button>
+  );
+}

docs/docs_skeleton/src/theme/CodeBlock/WordWrapButton/styles.module.css

@@ -0,0 +1,8 @@
+.wordWrapButtonIcon {
+  width: 1.2rem;
+  height: 1.2rem;
+}
+
+.wordWrapButtonEnabled .wordWrapButtonIcon {
+  color: var(--ifm-color-primary);
+}

docs/docs_skeleton/src/theme/CodeBlock/index.js

@@ -1,58 +1,32 @@
-/* eslint-disable react/jsx-props-no-spreading */
-import React from "react";
-import CodeBlock from "@theme-original/CodeBlock";
-
-function Imports({ imports }) {
+import React, {isValidElement} from 'react';
+import useIsBrowser from '@docusaurus/useIsBrowser';
+import ElementContent from '@theme/CodeBlock/Content/Element';
+import StringContent from '@theme/CodeBlock/Content/String';
+/**
+ * Best attempt to make the children a plain string so it is copyable. If there
+ * are react elements, we will not be able to copy the content, and it will
+ * return `children` as-is; otherwise, it concatenates the string children
+ * together.
+ */
+function maybeStringifyChildren(children) {
+  if (React.Children.toArray(children).some((el) => isValidElement(el))) {
+    return children;
+  }
+  // The children is now guaranteed to be one/more plain strings
+  return Array.isArray(children) ? children.join('') : children;
+}
+export default function CodeBlock({children: rawChildren, ...props}) {
+  // The Prism theme on SSR is always the default theme but the site theme can
+  // be in a different mode. React hydration doesn't update DOM styles that come
+  // from SSR. Hence force a re-render after mounting to apply the current
+  // relevant styles.
+  const isBrowser = useIsBrowser();
+  const children = maybeStringifyChildren(rawChildren);
+  const CodeBlockComp =
+    typeof children === 'string' ? StringContent : ElementContent;
   return (
-    <div
-      style={{
-        paddingTop: "1.3rem",
-        background: "var(--prism-background-color)",
-        color: "var(--prism-color)",
-        marginTop: "calc(-1 * var(--ifm-leading) - 5px)",
-        marginBottom: "var(--ifm-leading)",
-        boxShadow: "var(--ifm-global-shadow-lw)",
-        borderBottomLeftRadius: "var(--ifm-code-border-radius)",
-        borderBottomRightRadius: "var(--ifm-code-border-radius)",
-      }}
-    >
-      <h4 style={{ paddingLeft: "0.65rem", marginBottom: "0.45rem" }}>
-        API Reference:
-      </h4>
-      <ul style={{ paddingBottom: "1rem" }}>
-        {imports.map(({ imported, source, docs }) => (
-          <li key={imported}>
-            <a href={docs}>
-              <span>{imported}</span>
-            </a>{" "}
-            from <code>{source}</code>
-          </li>
-        ))}
-      </ul>
-    </div>
+    <CodeBlockComp key={String(isBrowser)} {...props}>
+      {children}
+    </CodeBlockComp>
   );
 }
-
-export default function CodeBlockWrapper({ children, ...props }) {
-  // Initialize imports as an empty array
-  let imports = [];
-
-  // Check if children is a string
-  if (typeof children === "string") {
-    // Search for an IMPORTS comment in the code
-    const match = /<!--IMPORTS:(.*?)-->\n/.exec(children);
-    if (match) {
-      imports = JSON.parse(match[1]);
-      children = children.replace(match[0], "");
-    }
-  } else if (children.imports) {
-    imports = children.imports;
-  }
-
-  return (
-    <>
-      <CodeBlock {...props}>{children}</CodeBlock>
-      {imports.length > 0 && <Imports imports={imports} />}
-    </>
-  );
-}

docs/docs_skeleton/vercel.json

@@ -1610,59 +1610,59 @@
     },
     {
       "source": "/en/latest/modules/chains/examples/flare.html",
-      "destination": "/docs/use_cases/question_answering/how_to/flare"
+      "destination": "/docs/modules/chains/additional/flare"
     },
     {
       "source": "/en/latest/modules/chains/examples/graph_cypher_qa.html",
-      "destination": "/docs/use_cases/graph/graph_cypher_qa"
+      "destination": "/docs/modules/chains/additional/graph_cypher_qa"
     },
     {
       "source": "/en/latest/modules/chains/examples/graph_nebula_qa.html",
-      "destination": "/docs/use_cases/graph/graph_nebula_qa"
+      "destination": "/docs/modules/chains/additional/graph_nebula_qa"
     },
     {
       "source": "/en/latest/modules/chains/index_examples/graph_qa.html",
-      "destination": "/docs/use_cases/graph/graph_qa"
+      "destination": "/docs/modules/chains/additional/graph_qa"
     },
     {
       "source": "/en/latest/modules/chains/index_examples/hyde.html",
-      "destination": "/docs/use_cases/question_answering/how_to/hyde"
+      "destination": "/docs/modules/chains/additional/hyde"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_bash.html",
-      "destination": "/docs/use_cases/code_writing/llm_bash"
+      "destination": "/docs/modules/chains/additional/llm_bash"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_checker.html",
-      "destination": "/docs/use_cases/self_check/llm_checker"
+      "destination": "/docs/modules/chains/additional/llm_checker"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_math.html",
-      "destination": "/docs/use_cases/code_writing/llm_math"
+      "destination": "/docs/modules/chains/additional/llm_math"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_requests.html",
-      "destination": "/docs/use_cases/apis/llm_requests"
+      "destination": "/docs/modules/chains/additional/llm_requests"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_summarization_checker.html",
-      "destination": "/docs/use_cases/self_check/llm_summarization_checker"
+      "destination": "/docs/modules/chains/additional/llm_summarization_checker"
     },
     {
       "source": "/en/latest/modules/chains/examples/openapi.html",
-      "destination": "/docs/use_cases/apis/openapi"
+      "destination": "/docs/modules/chains/additional/openapi"
     },
     {
       "source": "/en/latest/modules/chains/examples/pal.html",
-      "destination": "/docs/use_cases/code_writing/pal"
+      "destination": "/docs/modules/chains/additional/pal"
     },
     {
       "source": "/en/latest/modules/chains/examples/tagging.html",
-      "destination": "/docs/use_cases/tagging"
+      "destination": "/docs/modules/chains/additional/tagging"
     },
     {
       "source": "/en/latest/modules/chains/index_examples/vector_db_text_generation.html",
-      "destination": "/docs/use_cases/question_answering/how_to/vector_db_text_generation"
+      "destination": "/docs/modules/chains/additional/vector_db_text_generation"
     },
     {
       "source": "/en/latest/modules/chains/generic/router.html",
@@ -3448,10 +3448,6 @@
       "source": "/docs/modules/model_io/models/llms/integrations/writer",
       "destination": "/docs/integrations/llms/writer"
     },
-    {
-      "source": "/en/latest/modules/prompts.html",
-      "destination": "/docs/modules/model_io/prompts"
-    },
     {
       "source": "/en/latest/modules/prompts/output_parsers.html",
       "destination": "/docs/modules/model_io/output_parsers/"
@@ -3476,10 +3472,6 @@
       "source": "/en/latest/modules/prompts/output_parsers/examples/retry.html",
       "destination": "/docs/modules/model_io/output_parsers/retry"
     },
-    {
-      "source": "/en/latest/modules/prompts/example_selectors.html",
-      "destination": "/docs/modules/model_io/example_selectors"
-    },
     {
       "source": "/en/latest/modules/prompts/example_selectors/examples/custom_example_selector.html",
       "destination": "/docs/modules/model_io/prompts/example_selectors/custom_example_selector"
@@ -3492,10 +3484,6 @@
       "source": "/en/latest/modules/prompts/example_selectors/examples/ngram_overlap.html",
       "destination": "/docs/modules/model_io/prompts/example_selectors/ngram_overlap"
     },
-    {
-      "source": "/en/latest/modules/prompts/prompt_templates.html",
-      "destination": "/docs/modules/model_io/prompt_templates"
-    },
     {
       "source": "/en/latest/modules/prompts/prompt_templates/examples/connecting_to_a_feature_store.html",
       "destination": "/docs/modules/model_io/prompts/prompt_templates/connecting_to_a_feature_store"
@@ -3748,10 +3736,6 @@
       "source": "/docs/modules/evaluation/:path*(/?)",
       "destination": "/docs/guides/evaluation/:path*"
     },
-    {
-      "source": "/en/latest/modules/indexes.html",
-      "destination": "/docs/modules/data_connection"
-    },
     {
       "source": "/en/latest/modules/indexes/:path*",
       "destination": "/docs/modules/data_connection/:path*"
@@ -3787,170 +3771,6 @@
     {
       "source": "/en/latest/:path*",
       "destination": "/docs/:path*"
-    },
-    {
-      "source": "/docs/modules/chains/additional/constitutional_chain",
-      "destination": "/docs/guides/safety/constitutional_chain"
-    },
-    {
-      "source": "/docs/modules/chains/additional/moderation",
-      "destination": "/docs/guides/safety/moderation"
-    },
-    {
-      "source": "/docs/modules/chains/popular/api",
-      "destination": "/docs/use_cases/apis/api"
-    },
-    {
-      "source": "/docs/modules/chains/additional/analyze_document",
-      "destination": "/docs/use_cases/question_answering/how_to/analyze_document"
-    },
-    {
-      "source": "/docs/modules/chains/popular/chat_vector_db",
-      "destination": "/docs/use_cases/question_answering/how_to/chat_vector_db"
-    },
-    {
-      "source": "/docs/modules/chains/additional/multi_retrieval_qa_router",
-      "destination": "/docs/use_cases/question_answering/how_to/multi_retrieval_qa_router"
-    },
-    {
-      "source": "/docs/modules/chains/additional/question_answering",
-      "destination": "/docs/use_cases/question_answering/how_to/question_answering"
-    },
-    {
-      "source": "/docs/modules/chains/popular/vector_db_qa",
-      "destination": "/docs/use_cases/question_answering/how_to/vector_db_qa"
-    },
-    {
-      "source": "/docs/modules/chains/popular/summarize",
-      "destination": "/docs/use_cases/summarization/summarize"
-    },
-    {
-      "source": "/docs/modules/chains/popular/sqlite",
-      "destination": "/docs/use_cases/tabular/sqlite"
-    },
-    {
-      "source": "/docs/modules/chains/popular/openai_functions",
-      "destination": "/docs/modules/chains/how_to/openai_functions"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_requests",
-      "destination": "/docs/use_cases/apis/llm_requests"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openai_openapi",
-      "destination": "/docs/use_cases/apis/openai_openapi"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openapi",
-      "destination": "/docs/use_cases/apis/openapi"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openapi_openai",
-      "destination": "/docs/use_cases/apis/openapi_openai"
-    },
-    {
-      "source": "/docs/modules/chains/additional/cpal",
-      "destination": "/docs/use_cases/code_writing/cpal"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_bash",
-      "destination": "/docs/use_cases/code_writing/llm_bash"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_math",
-      "destination": "/docs/use_cases/code_writing/llm_math"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_symbolic_math",
-      "destination": "/docs/use_cases/code_writing/llm_symbolic_math"
-    },
-    {
-      "source": "/docs/modules/chains/additional/pal",
-      "destination": "/docs/use_cases/code_writing/pal"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_arangodb_qa",
-      "destination": "/docs/use_cases/graph/graph_arangodb_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_cypher_qa",
-      "destination": "/docs/use_cases/graph/graph_cypher_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_hugegraph_qa",
-      "destination": "/docs/use_cases/graph/graph_hugegraph_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_kuzu_qa",
-      "destination": "/docs/use_cases/graph/graph_kuzu_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_nebula_qa",
-      "destination": "/docs/use_cases/graph/graph_nebula_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_qa",
-      "destination": "/docs/use_cases/graph/graph_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_sparql_qa",
-      "destination": "/docs/use_cases/graph/graph_sparql_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/neptune_cypher_qa",
-      "destination": "/docs/use_cases/graph/neptune_cypher_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/tot",
-      "destination": "/docs/use_cases/graph/tot"
-    },
-    {
-      "source": "/docs/use_cases/question_answering//document-context-aware-QA",
-      "destination": "/docs/use_cases/question_answering/how_to/document-context-aware-QA"
-    },
-    {
-      "source": "/docs/modules/chains/additional/flare",
-      "destination": "/docs/use_cases/question_answering/how_to/flare"
-    },
-    {
-      "source": "/docs/modules/chains/additional/hyde",
-      "destination": "/docs/use_cases/question_answering/how_to/hyde"
-    },
-    {
-      "source": "/docs/use_cases/question_answering//local_retrieval_qa",
-      "destination": "/docs/use_cases/question_answering/how_to/local_retrieval_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/qa_citations",
-      "destination": "/docs/use_cases/question_answering/how_to/qa_citations"
-    },
-    {
-      "source": "/docs/modules/chains/additional/vector_db_text_generation",
-      "destination": "/docs/use_cases/question_answering/how_to/vector_db_text_generation"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openai_functions_retrieval_qa",
-      "destination": "/docs/use_cases/question_answering/integrations/openai_functions_retrieval_qa"
-    },
-    {
-      "source": "/docs/use_cases/question_answering//semantic-search-over-chat",
-      "destination": "/docs/use_cases/question_answering/integrations/semantic-search-over-chat"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_checker",
-      "destination": "/docs/use_cases/self_check/llm_checker"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_summarization_checker",
-      "destination": "/docs/use_cases/self_check/llm_summarization_checker"
-    },
-    {
-      "source": "/docs/modules/chains/additional/elasticsearch_database",
-      "destination": "/docs/use_cases/tabular/elasticsearch_database"
-    },
-    {
-      "source": "/docs/modules/chains/additional/tagging",
-      "destination": "/docs/use_cases/tagging"
     }
   ]
 }

docs/docs_skeleton/vercel_build.sh

@@ -1,53 +1,11 @@
 #!/bin/bash
 
-version_compare() {
-    local v1=(${1//./ })
-    local v2=(${2//./ })
-    for i in {0..2}; do
-        if (( ${v1[i]} < ${v2[i]} )); then
-            return 1
-        fi
-    done
-    return 0
-}
-
-openssl_version=$(openssl version | awk '{print $2}')
-required_openssl_version="1.1.1"
-
-python_version=$(python3 --version 2>&1 | awk '{print $2}')
-required_python_version="3.10"
-
-echo "OpenSSL Version"
-echo $openssl_version
-echo "Python Version"
-echo $python_version
-# If openssl version is less than 1.1.1 AND python version is less than 3.10
-if ! version_compare $openssl_version $required_openssl_version && ! version_compare $python_version $required_python_version; then
-### See: https://github.com/urllib3/urllib3/issues/2168
-# Requests lib breaks for old SSL versions,
-# which are defaults on Amazon Linux 2 (which Vercel uses for builds)
-    yum -y update
-    yum remove openssl-devel -y
-    yum install gcc bzip2-devel libffi-devel zlib-devel wget tar -y
-    yum install openssl11 -y
-    yum install openssl11-devel -y
-
-    wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz
-    tar xzf Python-3.11.4.tgz
-    cd Python-3.11.4
-    ./configure
-    make altinstall
-    echo "Python Version"
-    python3.11 --version
-    cd ..
-fi
-
 cd ..
-python3.11 -m venv .venv
+python3 --version
+python3 -m venv .venv
 source .venv/bin/activate
-python3.11 -m pip install --upgrade pip
-python3.11 -m pip install -r vercel_requirements.txt
+python3 -m pip install -r vercel_requirements.txt
 cp -r extras/* docs_skeleton/docs
+python3 docs_skeleton/link_generator.py
 cd docs_skeleton
 nbdoc_build
-python3.11 generate_api_reference_links.py

docs/extras/additional_resources/tutorials.mdx

@@ -1,6 +1,5 @@
 # Tutorials
 
-Below are links to video tutorials and courses on LangChain. For written guides on common use cases for LangChain, check out the [use cases guides](/docs/use_cases).
 
 ⛓ icon marks a new addition [last update 2023-07-05]
 

docs/extras/guides/debugging.md

@@ -4,7 +4,7 @@ If you're building with LLMs, at some point something will break, and you'll nee
 
 Here's a few different tools and functionalities to aid in debugging.
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 ## Tracing
 

docs/extras/guides/evaluation/comparison/pairwise_string.ipynb

@@ -43,7 +43,7 @@
     {
      "data": {
       "text/plain": [
-       "{'reasoning': 'Both responses are relevant to the question asked, as they both provide a numerical answer to the question about the number of dogs in the park. However, Response A is incorrect according to the reference answer, which states that there are four dogs. Response B, on the other hand, is correct as it matches the reference answer. Neither response demonstrates depth of thought, as they both simply provide a numerical answer without any additional information or context. \\n\\nBased on these criteria, Response B is the better response.\\n',\n",
+       "{'reasoning': 'Response A is incorrect as it states there are three dogs in the park, which contradicts the reference answer of four. Response B, on the other hand, is accurate as it matches the reference answer. Although Response B is not as detailed or elaborate as Response A, it is more important that the response is accurate. \\n\\nFinal Decision: [[B]]\\n',\n",
        " 'value': 'B',\n",
        " 'score': 0}"
       ]
@@ -62,27 +62,6 @@
     ")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "7491d2e6-4e77-4b17-be6b-7da966785c1d",
-   "metadata": {},
-   "source": [
-    "## Methods\n",
-    "\n",
-    "\n",
-    "The pairwise string evaluator can be called using [evaluate_string_pairs](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.evaluate_string_pairs) (or async [aevaluate_string_pairs](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.aevaluate_string_pairs)) methods, which accept:\n",
-    "\n",
-    "- prediction (str) – The predicted response of the first model, chain, or prompt.\n",
-    "- prediction_b (str) – The predicted response of the second model, chain, or prompt.\n",
-    "- input (str) – The input question, prompt, or other text.\n",
-    "- reference (str) – (Only for the labeled_pairwise_string variant) The reference response.\n",
-    "\n",
-    "They return a dictionary with the following values:\n",
-    "- value: 'A' or 'B', indicating whether `prediction` or `prediction_b` is preferred, respectively\n",
-    "- score: Integer 0 or 1 mapped from the 'value', where a score of 1 would mean that the first `prediction` is preferred, and a score of 0 would mean `prediction_b` is preferred.\n",
-    "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "ed353b93-be71-4479-b9c0-8c97814c2e58",
@@ -120,7 +99,7 @@
     {
      "data": {
       "text/plain": [
-       "{'reasoning': 'Both responses are correct and relevant to the question. However, Response B is more helpful and insightful as it provides a more detailed explanation of what addition is. Response A is correct but lacks depth as it does not explain what the operation of addition entails. \\n\\nFinal Decision: [[B]]',\n",
+       "{'reasoning': \"Response A is accurate but lacks depth and detail. It simply states that addition is a mathematical operation without explaining what it does or how it works. \\n\\nResponse B, on the other hand, provides a more detailed explanation. It not only identifies addition as a mathematical operation, but also explains that it involves adding two numbers to create a third number, the 'sum'. This response is more helpful and informative, providing a clearer understanding of what addition is.\\n\\nTherefore, the better response is B.\\n\",\n",
        " 'value': 'B',\n",
        " 'score': 0}"
       ]
@@ -138,74 +117,6 @@
     ")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "4a09b21d-9851-47e8-93d3-90044b2945b0",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "## Defining the Criteria\n",
-    "\n",
-    "By default, the LLM is instructed to select the 'preferred' response based on helpfulness, relevance, correctness, and depth of thought. You can customize the criteria by passing in a `criteria` argument, where the criteria could take any of the following forms:\n",
-    "- [`Criteria`](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.Criteria.html#langchain.evaluation.criteria.eval_chain.Criteria) enum or its string value - to use one of the default criteria and their descriptions\n",
-    "- [Constitutional principal](https://api.python.langchain.com/en/latest/chains/langchain.chains.constitutional_ai.models.ConstitutionalPrinciple.html#langchain.chains.constitutional_ai.models.ConstitutionalPrinciple) - use one any of the constitutional principles defined in langchain\n",
-    "- Dictionary: a list of custom criteria, where the key is the name of the criteria, and the value is the description.\n",
-    "- A list of criteria or constitutional principles - to combine multiple criteria in one.\n",
-    "\n",
-    "Below is an example for determining preferred writing responses based on a custom style."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "8539e7d9-f7b0-4d32-9c45-593a7915c093",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "custom_criteria = {\n",
-    "    \"simplicity\": \"Is the language straightforward and unpretentious?\",\n",
-    "    \"clarity\": \"Are the sentences clear and easy to understand?\",\n",
-    "    \"precision\": \"Is the writing precise, with no unnecessary words or details?\",\n",
-    "    \"truthfulness\": \"Does the writing feel honest and sincere?\",\n",
-    "    \"subtext\": \"Does the writing suggest deeper meanings or themes?\",\n",
-    "}\n",
-    "evaluator = load_evaluator(\"pairwise_string\", criteria=custom_criteria)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "fec7bde8-fbdc-4730-8366-9d90d033c181",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'reasoning': 'Response A is simple, clear, and precise. It uses straightforward language to convey a deep and sincere message about families. The metaphor of joy and sorrow as music is effective and easy to understand.\\n\\nResponse B, on the other hand, is more complex and less clear. The language is more pretentious, with words like \"domicile,\" \"resounds,\" \"abode,\" \"dissonant,\" and \"elegy.\" While it conveys a similar message to Response A, it does so in a more convoluted way. The precision is also lacking due to the use of unnecessary words and details.\\n\\nBoth responses suggest deeper meanings or themes about the shared joy and unique sorrow in families. However, Response A does so in a more effective and accessible way.\\n\\nTherefore, the better response is [[A]].',\n",
-       " 'value': 'A',\n",
-       " 'score': 1}"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "evaluator.evaluate_string_pairs(\n",
-    "    prediction=\"Every cheerful household shares a similar rhythm of joy; but sorrow, in each household, plays a unique, haunting melody.\",\n",
-    "    prediction_b=\"Where one finds a symphony of joy, every domicile of happiness resounds in harmonious,\"\n",
-    "    \" identical notes; yet, every abode of despair conducts a dissonant orchestra, each\"\n",
-    "    \" playing an elegy of grief that is peculiar and profound to its own existence.\",\n",
-    "    input=\"Write some prose about families.\",\n",
-    ")"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "a25b60b2-627c-408a-be4b-a2e5cbc10726",
@@ -218,7 +129,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 5,
    "id": "de84a958-1330-482b-b950-68bcf23f9e35",
    "metadata": {},
    "outputs": [],
@@ -232,7 +143,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 6,
    "id": "e162153f-d50a-4a7c-a033-019dabbc954c",
    "metadata": {
     "tags": []
@@ -241,12 +152,12 @@
     {
      "data": {
       "text/plain": [
-       "{'reasoning': 'Here is my assessment:\\n\\nResponse B is more helpful, insightful, and accurate than Response A. Response B simply states \"4\", which directly answers the question by providing the exact number of dogs mentioned in the reference answer. In contrast, Response A states \"there are three dogs\", which is incorrect according to the reference answer. \\n\\nIn terms of helpfulness, Response B gives the precise number while Response A provides an inaccurate guess. For relevance, both refer to dogs in the park from the question. However, Response B is more correct and factual based on the reference answer. Response A shows some attempt at reasoning but is ultimately incorrect. Response B requires less depth of thought to simply state the factual number.\\n\\nIn summary, Response B is superior in terms of helpfulness, relevance, correctness, and depth. My final decision is: [[B]]\\n',\n",
+       "{'reasoning': 'Here is my assessment:\\n\\nResponse B is better because it directly answers the question by stating the number \"4\", which matches the ground truth reference answer. Response A provides an incorrect number of dogs, stating there are three dogs when the reference says there are four. \\n\\nResponse B is more helpful, relevant, accurate and provides the right level of detail by simply stating the number that was asked for. Response A provides an inaccurate number, so is less helpful and accurate.\\n\\nIn summary, Response B better followed the instructions and answered the question correctly per the reference answer.\\n\\n[[B]]',\n",
        " 'value': 'B',\n",
        " 'score': 0}"
       ]
      },
-     "execution_count": 8,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -274,7 +185,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 7,
    "id": "fb817efa-3a4d-439d-af8c-773b89d97ec9",
    "metadata": {
     "tags": []
@@ -284,9 +195,7 @@
     "from langchain.prompts import PromptTemplate\n",
     "\n",
     "prompt_template = PromptTemplate.from_template(\n",
-    "    \"\"\"Given the input context, which do you prefer: A or B?\n",
-    "Evaluate based on the following criteria:\n",
-    "{criteria}\n",
+    "    \"\"\"Given the input context, which is most similar to the reference label: A or B?\n",
     "Reason step by step and finally, respond with either [[A]] or [[B]] on its own line.\n",
     "\n",
     "DATA\n",
@@ -307,7 +216,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 8,
    "id": "d40aa4f0-cfd5-4cb4-83c8-8d2300a04c2f",
    "metadata": {
     "tags": []
@@ -317,7 +226,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "input_variables=['prediction', 'reference', 'prediction_b', 'input'] output_parser=None partial_variables={'criteria': 'helpfulness: Is the submission helpful, insightful, and appropriate?\\nrelevance: Is the submission referring to a real quote from the text?\\ncorrectness: Is the submission correct, accurate, and factual?\\ndepth: Does the submission demonstrate depth of thought?'} template='Given the input context, which do you prefer: A or B?\\nEvaluate based on the following criteria:\\n{criteria}\\nReason step by step and finally, respond with either [[A]] or [[B]] on its own line.\\n\\nDATA\\n----\\ninput: {input}\\nreference: {reference}\\nA: {prediction}\\nB: {prediction_b}\\n---\\nReasoning:\\n\\n' template_format='f-string' validate_template=True\n"
+      "input_variables=['input', 'prediction', 'prediction_b', 'reference'] output_parser=None partial_variables={} template='Given the input context, which is most similar to the reference label: A or B?\\nReason step by step and finally, respond with either [[A]] or [[B]] on its own line.\\n\\nDATA\\n----\\ninput: {input}\\nreference: {reference}\\nA: {prediction}\\nB: {prediction_b}\\n---\\nReasoning:\\n\\n' template_format='f-string' validate_template=True\n"
      ]
     }
    ],
@@ -328,7 +237,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 9,
    "id": "9467bb42-7a31-4071-8f66-9ed2c6f06dcd",
    "metadata": {
     "tags": []
@@ -337,12 +246,12 @@
     {
      "data": {
       "text/plain": [
-       "{'reasoning': 'Helpfulness: Both A and B are helpful as they provide a direct answer to the question.\\nRelevance: A is relevant as it refers to the correct name of the dog from the text. B is not relevant as it provides a different name.\\nCorrectness: A is correct as it accurately states the name of the dog. B is incorrect as it provides a different name.\\nDepth: Both A and B demonstrate a similar level of depth as they both provide a straightforward answer to the question.\\n\\nGiven these evaluations, the preferred response is:\\n',\n",
+       "{'reasoning': 'Option A is more similar to the reference label because it mentions the same dog\\'s name, \"fido\". Option B mentions a different name, \"spot\". Therefore, A is more similar to the reference label. \\n',\n",
        " 'value': 'A',\n",
        " 'score': 1}"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/extras/guides/evaluation/examples/agent_vectordb_sota_pg.ipynb

@@ -0,0 +1,511 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Agent VectorDB Question Answering Benchmarking\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task using an agent to route between multiple vectordatabases.\n",
+    "\n",
+    "It is highly recommended that you do any evaluation/benchmarking with tracing enabled. See [here](https://python.langchain.com/guides/tracing/) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/qt/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-vectordb-qa-sota-pg-d3ae24016b514f92/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)\n",
+      "100%|██████████| 1/1 [00:00<00:00, 414.42it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"agent-vectordb-qa-sota-pg\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "61375342",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'steps': [{'tool': 'State of Union QA System', 'tool_input': None},\n",
+       "  {'tool': None, 'tool_input': 'What is the purpose of the NATO Alliance?'}]}"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "02500304",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of YC?',\n",
+       " 'answer': 'The purpose of YC is to cause startups to be founded that would not otherwise have existed.',\n",
+       " 'steps': [{'tool': 'Paul Graham QA System', 'tool_input': None},\n",
+       "  {'tool': None, 'tool_input': 'What is the purpose of YC?'}]}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[-1]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to create some pipelines for doing question answering. Step one in that is creating indexes over the data in question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "\n",
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "7f0de2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes import VectorstoreIndexCreator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "ef84ff99",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Using embedded DuckDB without persistence: data will be transient\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore_sota = (\n",
+    "    VectorstoreIndexCreator(vectorstore_kwargs={\"collection_name\": \"sota\"})\n",
+    "    .from_loaders([loader])\n",
+    "    .vectorstore\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import RetrievalQA\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "573719a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_sota = RetrievalQA.from_chain_type(\n",
+    "    llm=OpenAI(temperature=0),\n",
+    "    chain_type=\"stuff\",\n",
+    "    retriever=vectorstore_sota.as_retriever(),\n",
+    "    input_key=\"question\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e48b03d8",
+   "metadata": {},
+   "source": [
+    "Now we do the same for the Paul Graham data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "c2dbb014",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = TextLoader(\"../../modules/paul_graham_essay.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "98d16f08",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Using embedded DuckDB without persistence: data will be transient\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore_pg = (\n",
+    "    VectorstoreIndexCreator(vectorstore_kwargs={\"collection_name\": \"paul_graham\"})\n",
+    "    .from_loaders([loader])\n",
+    "    .vectorstore\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "ec0aab02",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain_pg = RetrievalQA.from_chain_type(\n",
+    "    llm=OpenAI(temperature=0),\n",
+    "    chain_type=\"stuff\",\n",
+    "    retriever=vectorstore_pg.as_retriever(),\n",
+    "    input_key=\"question\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "76b5f8fb",
+   "metadata": {},
+   "source": [
+    "We can now set up an agent to route between them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "ade1aafa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import initialize_agent, Tool\n",
+    "from langchain.agents import AgentType\n",
+    "\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name=\"State of Union QA System\",\n",
+    "        func=chain_sota.run,\n",
+    "        description=\"useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question.\",\n",
+    "    ),\n",
+    "    Tool(\n",
+    "        name=\"Paul Graham System\",\n",
+    "        func=chain_pg.run,\n",
+    "        description=\"useful for when you need to answer questions about Paul Graham. Input should be a fully formed question.\",\n",
+    "    ),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "id": "104853f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = initialize_agent(\n",
+    "    tools,\n",
+    "    OpenAI(temperature=0),\n",
+    "    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
+    "    max_iterations=4,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f036641",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "id": "4664e79f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'"
+      ]
+     },
+     "execution_count": 35,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(dataset[0][\"question\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "id": "799f6c17",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = []\n",
+    "predicted_dataset = []\n",
+    "error_dataset = []\n",
+    "for data in dataset:\n",
+    "    new_data = {\"input\": data[\"question\"], \"answer\": data[\"answer\"]}\n",
+    "    try:\n",
+    "        predictions.append(agent(new_data))\n",
+    "        predicted_dataset.append(new_data)\n",
+    "    except Exception:\n",
+    "        error_dataset.append(new_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'output': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'}"
+      ]
+     },
+     "execution_count": 37,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 38,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(\n",
+    "    predicted_dataset, predictions, question_key=\"input\", prediction_key=\"output\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction[\"grade\"] = graded_outputs[i][\"text\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 28, ' INCORRECT': 5})"
+      ]
+     },
+     "execution_count": 41,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "Counter([pred[\"grade\"] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 42,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred[\"grade\"] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': 'What are the four common sense steps that the author suggests to move forward safely?',\n",
+       " 'answer': 'The four common sense steps suggested by the author to move forward safely are: stay protected with vaccines and treatments, prepare for new variants, end the shutdown of schools and businesses, and stay vigilant.',\n",
+       " 'output': 'The four common sense steps suggested in the most recent State of the Union address are: cutting the cost of prescription drugs, providing a pathway to citizenship for Dreamers, revising laws so businesses have the workers they need and families don’t wait decades to reunite, and protecting access to health care and preserving a woman’s right to choose.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 43,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/examples/comparisons.ipynb

@@ -24,15 +24,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 2,
    "metadata": {
     "tags": []
    },
    "outputs": [],
    "source": [
-    "from langchain.evaluation import load_evaluator\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.evaluation.comparison import PairwiseStringEvalChain\n",
     "\n",
-    "eval_chain = load_evaluator(\"pairwise_string\")"
+    "llm = ChatOpenAI(model=\"gpt-4\")\n",
+    "\n",
+    "eval_chain = PairwiseStringEvalChain.from_llm(llm=llm)"
    ]
   },
   {
@@ -47,7 +50,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 3,
    "metadata": {
     "tags": []
    },
@@ -56,13 +59,13 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7)\n"
+      "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
      ]
     },
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "a2358d37246640ce95e0f9940194590a",
+       "model_id": "d852a1884480457292c90d8bd9d4f1e6",
        "version_major": 2,
        "version_minor": 0
       },
@@ -91,7 +94,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "metadata": {
     "tags": []
    },
@@ -124,7 +127,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "metadata": {
     "tags": []
    },
@@ -149,7 +152,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "metadata": {
     "tags": []
    },
@@ -157,7 +160,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "87277cb39a1a4726bb7cc533a24e2ea4",
+       "model_id": "b076d6bf6680422aa9082d4bad4d98a3",
        "version_major": 2,
        "version_minor": 0
       },
@@ -167,6 +170,14 @@
      },
      "metadata": {},
      "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n",
+      "Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n"
+     ]
     }
    ],
    "source": [
@@ -204,7 +215,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 9,
    "metadata": {
     "tags": []
    },
@@ -241,7 +252,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -259,7 +270,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 11,
    "metadata": {
     "tags": []
    },
@@ -268,8 +279,8 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "OpenAI Functions Agent: 95.00%\n",
-      "None: 5.00%\n"
+      "OpenAI Functions Agent: 90.00%\n",
+      "Structured Chat Agent: 10.00%\n"
      ]
     }
    ],
@@ -299,7 +310,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 12,
    "metadata": {
     "tags": []
    },
@@ -338,7 +349,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 13,
    "metadata": {
     "tags": []
    },
@@ -347,8 +358,8 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "The \"OpenAI Functions Agent\" would be preferred between 83.18% and 100.00% percent of the time (with 95% confidence).\n",
-      "The \"Structured Chat Agent\" would be preferred between 0.00% and 16.82% percent of the time (with 95% confidence).\n"
+      "The \"OpenAI Functions Agent\" would be preferred between 69.90% and 97.21% percent of the time (with 95% confidence).\n",
+      "The \"Structured Chat Agent\" would be preferred between 2.79% and 30.10% percent of the time (with 95% confidence).\n"
      ]
     }
    ],
@@ -369,7 +380,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 18,
    "metadata": {
     "tags": []
    },
@@ -378,17 +389,9 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "The p-value is 0.00000. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
-      "then there is a 0.00038% chance of observing the OpenAI Functions Agent be preferred at least 19\n",
-      "times out of 19 trials.\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_15978/384907688.py:6: DeprecationWarning: 'binom_test' is deprecated in favour of 'binomtest' from version 1.7.0 and will be removed in Scipy 1.12.0.\n",
-      "  p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n"
+      "The p-value is 0.00040. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+      "then there is a 0.04025% chance of observing the OpenAI Functions Agent be preferred at least 18\n",
+      "times out of 20 trials.\n"
      ]
     }
    ],

docs/extras/guides/evaluation/examples/data_augmented_question_answering.ipynb

@@ -0,0 +1,445 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e78b7bb1",
+   "metadata": {},
+   "source": [
+    "# Data Augmented Question Answering\n",
+    "\n",
+    "This notebook uses some generic prompts/language models to evaluate an question answering system that uses other sources of data besides what is in the model. For example, this can be used to evaluate a question answering system over your proprietary data.\n",
+    "\n",
+    "## Setup\n",
+    "Let's set up an example with our favorite example - the state of the union address."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ab4a6931",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import Chroma\n",
+    "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.chains import RetrievalQA"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "4fdc211d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "\n",
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")\n",
+    "documents = loader.load()\n",
+    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
+    "texts = text_splitter.split_documents(documents)\n",
+    "\n",
+    "embeddings = OpenAIEmbeddings()\n",
+    "docsearch = Chroma.from_documents(texts, embeddings)\n",
+    "qa = RetrievalQA.from_llm(llm=OpenAI(), retriever=docsearch.as_retriever())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30fd72f2",
+   "metadata": {},
+   "source": [
+    "## Examples\n",
+    "Now we need some examples to evaluate. We can do this in two ways:\n",
+    "\n",
+    "1. Hard code some examples ourselves\n",
+    "2. Generate examples automatically, using a language model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "3459b001",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Hard-coded examples\n",
+    "examples = [\n",
+    "    {\n",
+    "        \"query\": \"What did the president say about Ketanji Brown Jackson\",\n",
+    "        \"answer\": \"He praised her legal ability and said he nominated her for the supreme court.\",\n",
+    "    },\n",
+    "    {\"query\": \"What did the president say about Michael Jackson\", \"answer\": \"Nothing\"},\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "b9c3fa75",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Generated examples\n",
+    "from langchain.evaluation.qa import QAGenerateChain\n",
+    "\n",
+    "example_gen_chain = QAGenerateChain.from_llm(OpenAI())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "c24543a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "new_examples = example_gen_chain.apply_and_parse([{\"doc\": t} for t in texts[:5]])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "a2d27560",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'query': 'According to the document, what did Vladimir Putin miscalculate?',\n",
+       "  'answer': 'He miscalculated that he could roll into Ukraine and the world would roll over.'},\n",
+       " {'query': 'Who is the Ukrainian Ambassador to the United States?',\n",
+       "  'answer': 'The Ukrainian Ambassador to the United States is here tonight.'},\n",
+       " {'query': 'How many countries were part of the coalition formed to confront Putin?',\n",
+       "  'answer': '27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.'},\n",
+       " {'query': 'What action is the U.S. Department of Justice taking to target Russian oligarchs?',\n",
+       "  'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.'},\n",
+       " {'query': 'How much direct assistance is the United States providing to Ukraine?',\n",
+       "  'answer': 'The United States is providing more than $1 Billion in direct assistance to Ukraine.'}]"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "new_examples"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "558da6f3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Combine examples\n",
+    "examples += new_examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "443dc34e",
+   "metadata": {},
+   "source": [
+    "## Evaluate\n",
+    "Now that we have examples, we can use the question answering evaluator to evaluate our question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "782169a5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "1bb77416",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = qa.apply(examples)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "bcd0ad7f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "2e6af79a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "graded_outputs = eval_chain.evaluate(examples, predictions)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "32fac2dc",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Example 0:\n",
+      "Question: What did the president say about Ketanji Brown Jackson\n",
+      "Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
+      "Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.\n",
+      "Predicted Grade:  CORRECT\n",
+      "\n",
+      "Example 1:\n",
+      "Question: What did the president say about Michael Jackson\n",
+      "Real Answer: Nothing\n",
+      "Predicted Answer:  The president did not mention Michael Jackson in this speech.\n",
+      "Predicted Grade:  CORRECT\n",
+      "\n",
+      "Example 2:\n",
+      "Question: According to the document, what did Vladimir Putin miscalculate?\n",
+      "Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.\n",
+      "Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.\n",
+      "Predicted Grade:  CORRECT\n",
+      "\n",
+      "Example 3:\n",
+      "Question: Who is the Ukrainian Ambassador to the United States?\n",
+      "Real Answer: The Ukrainian Ambassador to the United States is here tonight.\n",
+      "Predicted Answer:  I don't know.\n",
+      "Predicted Grade:  INCORRECT\n",
+      "\n",
+      "Example 4:\n",
+      "Question: How many countries were part of the coalition formed to confront Putin?\n",
+      "Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Grade:  INCORRECT\n",
+      "\n",
+      "Example 5:\n",
+      "Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?\n",
+      "Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Grade:  INCORRECT\n",
+      "\n",
+      "Example 6:\n",
+      "Question: How much direct assistance is the United States providing to Ukraine?\n",
+      "Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.\n",
+      "Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.\n",
+      "Predicted Grade:  CORRECT\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i, eg in enumerate(examples):\n",
+    "    print(f\"Example {i}:\")\n",
+    "    print(\"Question: \" + predictions[i][\"query\"])\n",
+    "    print(\"Real Answer: \" + predictions[i][\"answer\"])\n",
+    "    print(\"Predicted Answer: \" + predictions[i][\"result\"])\n",
+    "    print(\"Predicted Grade: \" + graded_outputs[i][\"text\"])\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "50a9e845",
+   "metadata": {},
+   "source": [
+    "## Evaluate with Other Metrics\n",
+    "\n",
+    "In addition to predicting whether the answer is correct or incorrect using a language model, we can also use other metrics to get a more nuanced view on the quality of the answers. To do so, we can use the [Critique](https://docs.inspiredco.ai/critique/) library, which allows for simple calculation of various metrics over generated text.\n",
+    "\n",
+    "First you can get an API key from the [Inspired Cognition Dashboard](https://dashboard.inspiredco.ai) and do some setup:\n",
+    "\n",
+    "```bash\n",
+    "export INSPIREDCO_API_KEY=\"...\"\n",
+    "pip install inspiredco\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "bd0b01dc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import inspiredco.critique\n",
+    "import os\n",
+    "\n",
+    "critique = inspiredco.critique.Critique(api_key=os.environ[\"INSPIREDCO_API_KEY\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4f52629e",
+   "metadata": {},
+   "source": [
+    "Then run the following code to set up the configuration and calculate the [ROUGE](https://docs.inspiredco.ai/critique/metric_rouge.html), [chrf](https://docs.inspiredco.ai/critique/metric_chrf.html), [BERTScore](https://docs.inspiredco.ai/critique/metric_bert_score.html), and [UniEval](https://docs.inspiredco.ai/critique/metric_uni_eval.html) (you can choose [other metrics](https://docs.inspiredco.ai/critique/metrics.html) too):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "84a0ba21",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = {\n",
+    "    \"rouge\": {\n",
+    "        \"metric\": \"rouge\",\n",
+    "        \"config\": {\"variety\": \"rouge_l\"},\n",
+    "    },\n",
+    "    \"chrf\": {\n",
+    "        \"metric\": \"chrf\",\n",
+    "        \"config\": {},\n",
+    "    },\n",
+    "    \"bert_score\": {\n",
+    "        \"metric\": \"bert_score\",\n",
+    "        \"config\": {\"model\": \"bert-base-uncased\"},\n",
+    "    },\n",
+    "    \"uni_eval\": {\n",
+    "        \"metric\": \"uni_eval\",\n",
+    "        \"config\": {\"task\": \"summarization\", \"evaluation_aspect\": \"relevance\"},\n",
+    "    },\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "3b9a4056",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "critique_data = [\n",
+    "    {\"target\": pred[\"result\"], \"references\": [pred[\"answer\"]]} for pred in predictions\n",
+    "]\n",
+    "eval_results = {\n",
+    "    k: critique.evaluate(dataset=critique_data, metric=v[\"metric\"], config=v[\"config\"])\n",
+    "    for k, v in metrics.items()\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f0ae799",
+   "metadata": {},
+   "source": [
+    "Finally, we can print out the results. We can see that overall the scores are higher when the output is semantically correct, and also when the output closely matches with the gold-standard answer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "b51edcf4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Example 0:\n",
+      "Question: What did the president say about Ketanji Brown Jackson\n",
+      "Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
+      "Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.\n",
+      "Predicted Scores: rouge=0.0941, chrf=0.2001, bert_score=0.5219, uni_eval=0.9043\n",
+      "\n",
+      "Example 1:\n",
+      "Question: What did the president say about Michael Jackson\n",
+      "Real Answer: Nothing\n",
+      "Predicted Answer:  The president did not mention Michael Jackson in this speech.\n",
+      "Predicted Scores: rouge=0.0000, chrf=0.1087, bert_score=0.3486, uni_eval=0.7802\n",
+      "\n",
+      "Example 2:\n",
+      "Question: According to the document, what did Vladimir Putin miscalculate?\n",
+      "Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.\n",
+      "Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.\n",
+      "Predicted Scores: rouge=0.5185, chrf=0.6955, bert_score=0.8421, uni_eval=0.9578\n",
+      "\n",
+      "Example 3:\n",
+      "Question: Who is the Ukrainian Ambassador to the United States?\n",
+      "Real Answer: The Ukrainian Ambassador to the United States is here tonight.\n",
+      "Predicted Answer:  I don't know.\n",
+      "Predicted Scores: rouge=0.0000, chrf=0.0375, bert_score=0.3159, uni_eval=0.7493\n",
+      "\n",
+      "Example 4:\n",
+      "Question: How many countries were part of the coalition formed to confront Putin?\n",
+      "Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
+      "Predicted Scores: rouge=0.7419, chrf=0.8602, bert_score=0.8388, uni_eval=0.0669\n",
+      "\n",
+      "Example 5:\n",
+      "Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?\n",
+      "Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.\n",
+      "Predicted Scores: rouge=0.9412, chrf=0.8687, bert_score=0.9607, uni_eval=0.9718\n",
+      "\n",
+      "Example 6:\n",
+      "Question: How much direct assistance is the United States providing to Ukraine?\n",
+      "Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.\n",
+      "Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.\n",
+      "Predicted Scores: rouge=1.0000, chrf=0.9483, bert_score=1.0000, uni_eval=0.9734\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i, eg in enumerate(examples):\n",
+    "    score_string = \", \".join(\n",
+    "        [f\"{k}={v['examples'][i]['value']:.4f}\" for k, v in eval_results.items()]\n",
+    "    )\n",
+    "    print(f\"Example {i}:\")\n",
+    "    print(\"Question: \" + predictions[i][\"query\"])\n",
+    "    print(\"Real Answer: \" + predictions[i][\"answer\"])\n",
+    "    print(\"Predicted Answer: \" + predictions[i][\"result\"])\n",
+    "    print(\"Predicted Scores: \" + score_string)\n",
+    "    print()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/examples/openapi_eval.ipynb

@@ -0,0 +1,975 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "692f3256",
+   "metadata": {},
+   "source": [
+    "# Evaluating an OpenAPI Chain\n",
+    "\n",
+    "This notebook goes over ways to semantically evaluate an [OpenAPI Chain](/docs/modules/chains/additional/openapi.html), which calls an endpoint defined by the OpenAPI specification using purely natural language."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "a457106d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.tools import OpenAPISpec, APIOperation\n",
+    "from langchain.chains import OpenAPIEndpointChain, LLMChain\n",
+    "from langchain.requests import Requests\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2c3b0954",
+   "metadata": {},
+   "source": [
+    "## Load the API Chain\n",
+    "\n",
+    "Load a wrapper of the spec (so we can work with it more easily). You can load from a url or from a local file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "794142ba",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load and parse the OpenAPI Spec\n",
+    "spec = OpenAPISpec.from_url(\n",
+    "    \"https://www.klarna.com/us/shopping/public/openai/v0/api-docs/\"\n",
+    ")\n",
+    "# Load a single endpoint operation\n",
+    "operation = APIOperation.from_openapi_spec(spec, \"/public/openai/v0/products\", \"get\")\n",
+    "verbose = False\n",
+    "# Select any LangChain LLM\n",
+    "llm = OpenAI(temperature=0, max_tokens=1000)\n",
+    "# Create the endpoint chain\n",
+    "api_chain = OpenAPIEndpointChain.from_api_operation(\n",
+    "    operation,\n",
+    "    llm,\n",
+    "    requests=Requests(),\n",
+    "    verbose=verbose,\n",
+    "    return_intermediate_steps=True,  # Return request and response text\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c05ba5b",
+   "metadata": {},
+   "source": [
+    "### *Optional*: Generate Input Questions and Request Ground Truth Queries\n",
+    "\n",
+    "See [Generating Test Datasets](#Generating-Test-Datasets) at the end of this notebook for more details."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "a0c0cb7e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import re\n",
+    "# from langchain.prompts import PromptTemplate\n",
+    "\n",
+    "# template = \"\"\"Below is a service description:\n",
+    "\n",
+    "# {spec}\n",
+    "\n",
+    "# Imagine you're a new user trying to use {operation} through a search bar. What are 10 different things you want to request?\n",
+    "# Wants/Questions:\n",
+    "# 1. \"\"\"\n",
+    "\n",
+    "# prompt = PromptTemplate.from_template(template)\n",
+    "\n",
+    "# generation_chain = LLMChain(llm=llm, prompt=prompt)\n",
+    "\n",
+    "# questions_ = generation_chain.run(spec=operation.to_typescript(), operation=operation.operation_id).split('\\n')\n",
+    "# # Strip preceding numeric bullets\n",
+    "# questions = [re.sub(r'^\\d+\\. ', '', q).strip() for q in questions_]\n",
+    "# questions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "f3d767ef",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ground_truths = [\n",
+    "# {\"q\": ...} # What are the best queries for each input?\n",
+    "# ]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "81098a05",
+   "metadata": {},
+   "source": [
+    "## Run the API Chain\n",
+    "\n",
+    "The two simplest questions a user of the API Chain are:\n",
+    "- Did the chain succesfully access the endpoint?\n",
+    "- Did the action accomplish the correct result?\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "64bc7ed9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import defaultdict\n",
+    "\n",
+    "# Collect metrics to report at completion\n",
+    "scores = defaultdict(list)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "dfd2d09f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--openapi-chain-klarna-products-get-5d03362007667626/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "10932c9c139941d1a8be1a798f29e923",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"openapi-chain-klarna-products-get\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "e08191a7",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'question': 'What iPhone models are available?',\n",
+       "  'expected_query': {'max_price': None, 'q': 'iPhone'}},\n",
+       " {'question': 'Are there any budget laptops?',\n",
+       "  'expected_query': {'max_price': 300, 'q': 'laptop'}},\n",
+       " {'question': 'Show me the cheapest gaming PC.',\n",
+       "  'expected_query': {'max_price': 500, 'q': 'gaming pc'}},\n",
+       " {'question': 'Are there any tablets under $400?',\n",
+       "  'expected_query': {'max_price': 400, 'q': 'tablet'}},\n",
+       " {'question': 'What are the best headphones?',\n",
+       "  'expected_query': {'max_price': None, 'q': 'headphones'}},\n",
+       " {'question': 'What are the top rated laptops?',\n",
+       "  'expected_query': {'max_price': None, 'q': 'laptop'}},\n",
+       " {'question': 'I want to buy some shoes. I like Adidas and Nike.',\n",
+       "  'expected_query': {'max_price': None, 'q': 'shoe'}},\n",
+       " {'question': 'I want to buy a new skirt',\n",
+       "  'expected_query': {'max_price': None, 'q': 'skirt'}},\n",
+       " {'question': 'My company is asking me to get a professional Deskopt PC - money is no object.',\n",
+       "  'expected_query': {'max_price': 10000, 'q': 'professional desktop PC'}},\n",
+       " {'question': 'What are the best budget cameras?',\n",
+       "  'expected_query': {'max_price': 300, 'q': 'camera'}}]"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "7ee71384",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "questions = [d[\"question\"] for d in dataset]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "00511f7a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Run the the API chain itself\n",
+    "raise_error = False  # Stop on first failed example - useful for development\n",
+    "chain_outputs = []\n",
+    "failed_examples = []\n",
+    "for question in questions:\n",
+    "    try:\n",
+    "        chain_outputs.append(api_chain(question))\n",
+    "        scores[\"completed\"].append(1.0)\n",
+    "    except Exception as e:\n",
+    "        if raise_error:\n",
+    "            raise e\n",
+    "        failed_examples.append({\"q\": question, \"error\": e})\n",
+    "        scores[\"completed\"].append(0.0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "f3c9729f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[]"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# If the chain failed to run, show the failing examples\n",
+    "failed_examples"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "914e7587",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['There are currently 10 Apple iPhone models available: Apple iPhone 14 Pro Max 256GB, Apple iPhone 12 128GB, Apple iPhone 13 128GB, Apple iPhone 14 Pro 128GB, Apple iPhone 14 Pro 256GB, Apple iPhone 14 Pro Max 128GB, Apple iPhone 13 Pro Max 128GB, Apple iPhone 14 128GB, Apple iPhone 12 Pro 512GB, and Apple iPhone 12 mini 64GB.',\n",
+       " 'Yes, there are several budget laptops in the API response. For example, the HP 14-dq0055dx and HP 15-dw0083wm are both priced at $199.99 and $244.99 respectively.',\n",
+       " 'The cheapest gaming PC available is the Alarco Gaming PC (X_BLACK_GTX750) for $499.99. You can find more information about it here: https://www.klarna.com/us/shopping/pl/cl223/3203154750/Desktop-Computers/Alarco-Gaming-PC-%28X_BLACK_GTX750%29/?utm_source=openai&ref-site=openai_plugin',\n",
+       " 'Yes, there are several tablets under $400. These include the Apple iPad 10.2\" 32GB (2019), Samsung Galaxy Tab A8 10.5 SM-X200 32GB, Samsung Galaxy Tab A7 Lite 8.7 SM-T220 32GB, Amazon Fire HD 8\" 32GB (10th Generation), and Amazon Fire HD 10 32GB.',\n",
+       " 'It looks like you are looking for the best headphones. Based on the API response, it looks like the Apple AirPods Pro (2nd generation) 2022, Apple AirPods Max, and Bose Noise Cancelling Headphones 700 are the best options.',\n",
+       " 'The top rated laptops based on the API response are the Apple MacBook Pro (2021) M1 Pro 8C CPU 14C GPU 16GB 512GB SSD 14\", Apple MacBook Pro (2022) M2 OC 10C GPU 8GB 256GB SSD 13.3\", Apple MacBook Air (2022) M2 OC 8C GPU 8GB 256GB SSD 13.6\", and Apple MacBook Pro (2023) M2 Pro OC 16C GPU 16GB 512GB SSD 14.2\".',\n",
+       " \"I found several Nike and Adidas shoes in the API response. Here are the links to the products: Nike Dunk Low M - Black/White: https://www.klarna.com/us/shopping/pl/cl337/3200177969/Shoes/Nike-Dunk-Low-M-Black-White/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 4 Retro M - Midnight Navy: https://www.klarna.com/us/shopping/pl/cl337/3202929835/Shoes/Nike-Air-Jordan-4-Retro-M-Midnight-Navy/?utm_source=openai&ref-site=openai_plugin, Nike Air Force 1 '07 M - White: https://www.klarna.com/us/shopping/pl/cl337/3979297/Shoes/Nike-Air-Force-1-07-M-White/?utm_source=openai&ref-site=openai_plugin, Nike Dunk Low W - White/Black: https://www.klarna.com/us/shopping/pl/cl337/3200134705/Shoes/Nike-Dunk-Low-W-White-Black/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 1 Retro High M - White/University Blue/Black: https://www.klarna.com/us/shopping/pl/cl337/3200383658/Shoes/Nike-Air-Jordan-1-Retro-High-M-White-University-Blue-Black/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 1 Retro High OG M - True Blue/Cement Grey/White: https://www.klarna.com/us/shopping/pl/cl337/3204655673/Shoes/Nike-Air-Jordan-1-Retro-High-OG-M-True-Blue-Cement-Grey-White/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 11 Retro Cherry - White/Varsity Red/Black: https://www.klarna.com/us/shopping/pl/cl337/3202929696/Shoes/Nike-Air-Jordan-11-Retro-Cherry-White-Varsity-Red-Black/?utm_source=openai&ref-site=openai_plugin, Nike Dunk High W - White/Black: https://www.klarna.com/us/shopping/pl/cl337/3201956448/Shoes/Nike-Dunk-High-W-White-Black/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 5 Retro M - Black/Taxi/Aquatone: https://www.klarna.com/us/shopping/pl/cl337/3204923084/Shoes/Nike-Air-Jordan-5-Retro-M-Black-Taxi-Aquatone/?utm_source=openai&ref-site=openai_plugin, Nike Court Legacy Lift W: https://www.klarna.com/us/shopping/pl/cl337/3202103728/Shoes/Nike-Court-Legacy-Lift-W/?utm_source=openai&ref-site=openai_plugin\",\n",
+       " \"I found several skirts that may interest you. Please take a look at the following products: Avenue Plus Size Denim Stretch Skirt, LoveShackFancy Ruffled Mini Skirt - Antique White, Nike Dri-Fit Club Golf Skirt - Active Pink, Skims Soft Lounge Ruched Long Skirt, French Toast Girl's Front Pleated Skirt with Tabs, Alexia Admor Women's Harmonie Mini Skirt Pink Pink, Vero Moda Long Skirt, Nike Court Dri-FIT Victory Flouncy Tennis Skirt Women - White/Black, Haoyuan Mini Pleated Skirts W, and Zimmermann Lyre Midi Skirt.\",\n",
+       " 'Based on the API response, you may want to consider the Skytech Archangel Gaming Computer PC Desktop, the CyberPowerPC Gamer Master Gaming Desktop, or the ASUS ROG Strix G10DK-RS756, as they all offer powerful processors and plenty of RAM.',\n",
+       " 'Based on the API response, the best budget cameras are the DJI Mini 2 Dog Camera ($448.50), Insta360 Sphere with Landing Pad ($429.99), DJI FPV Gimbal Camera ($121.06), Parrot Camera & Body ($36.19), and DJI FPV Air Unit ($179.00).']"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "answers = [res[\"output\"] for res in chain_outputs]\n",
+    "answers"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "484f0587",
+   "metadata": {},
+   "source": [
+    "## Evaluate the requests chain\n",
+    "\n",
+    "The API Chain has two main components:\n",
+    "1. Translate the user query to an API request (request synthesizer)\n",
+    "2. Translate the API response to a natural language response\n",
+    "\n",
+    "Here, we construct an evaluation chain to grade the request synthesizer against selected human queries "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "3ea5afd7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "truth_queries = [json.dumps(data[\"expected_query\"]) for data in dataset]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "e055f24b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Collect the API queries generated by the chain\n",
+    "predicted_queries = [\n",
+    "    output[\"intermediate_steps\"][\"request_args\"] for output in chain_outputs\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "7d4f2b88",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import PromptTemplate\n",
+    "\n",
+    "template = \"\"\"You are trying to answer the following question by querying an API:\n",
+    "\n",
+    "> Question: {question}\n",
+    "\n",
+    "The query you know you should be executing against the API is:\n",
+    "\n",
+    "> Query: {truth_query}\n",
+    "\n",
+    "Is the following predicted query semantically the same (eg likely to produce the same answer)?\n",
+    "\n",
+    "> Predicted Query: {predict_query}\n",
+    "\n",
+    "Please give the Predicted Query a grade of either an A, B, C, D, or F, along with an explanation of why. End the evaluation with 'Final Grade: <the letter>'\n",
+    "\n",
+    "> Explanation: Let's think step by step.\"\"\"\n",
+    "\n",
+    "prompt = PromptTemplate.from_template(template)\n",
+    "\n",
+    "eval_chain = LLMChain(llm=llm, prompt=prompt, verbose=verbose)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "8cc1b1db",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[' The original query is asking for all iPhone models, so the \"q\" parameter is correct. The \"max_price\" parameter is also correct, as it is set to null, meaning that no maximum price is set. The predicted query adds two additional parameters, \"size\" and \"min_price\". The \"size\" parameter is not necessary, as it is not relevant to the question being asked. The \"min_price\" parameter is also not necessary, as it is not relevant to the question being asked and it is set to 0, which is the default value. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',\n",
+       " ' The original query is asking for laptops with a maximum price of 300. The predicted query is asking for laptops with a minimum price of 0 and a maximum price of 500. This means that the predicted query is likely to return more results than the original query, as it is asking for a wider range of prices. Therefore, the predicted query is not semantically the same as the original query, and it is not likely to produce the same answer. Final Grade: F',\n",
+       " \" The first two parameters are the same, so that's good. The third parameter is different, but it's not necessary for the query, so that's not a problem. The fourth parameter is the problem. The original query specifies a maximum price of 500, while the predicted query specifies a maximum price of null. This means that the predicted query will not limit the results to the cheapest gaming PCs, so it is not semantically the same as the original query. Final Grade: F\",\n",
+       " ' The original query is asking for tablets under $400, so the first two parameters are correct. The predicted query also includes the parameters \"size\" and \"min_price\", which are not necessary for the original query. The \"size\" parameter is not relevant to the question, and the \"min_price\" parameter is redundant since the original query already specifies a maximum price. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',\n",
+       " ' The original query is asking for headphones with no maximum price, so the predicted query is not semantically the same because it has a maximum price of 500. The predicted query also has a size of 10, which is not specified in the original query. Therefore, the predicted query is not semantically the same as the original query. Final Grade: F',\n",
+       " \" The original query is asking for the top rated laptops, so the 'size' parameter should be set to 10 to get the top 10 results. The 'min_price' parameter should be set to 0 to get results from all price ranges. The 'max_price' parameter should be set to null to get results from all price ranges. The 'q' parameter should be set to 'laptop' to get results related to laptops. All of these parameters are present in the predicted query, so it is semantically the same as the original query. Final Grade: A\",\n",
+       " ' The original query is asking for shoes, so the predicted query is asking for the same thing. The original query does not specify a size, so the predicted query is not adding any additional information. The original query does not specify a price range, so the predicted query is adding additional information that is not necessary. Therefore, the predicted query is not semantically the same as the original query and is likely to produce different results. Final Grade: D',\n",
+       " ' The original query is asking for a skirt, so the predicted query is asking for the same thing. The predicted query also adds additional parameters such as size and price range, which could help narrow down the results. However, the size parameter is not necessary for the query to be successful, and the price range is too narrow. Therefore, the predicted query is not as effective as the original query. Final Grade: C',\n",
+       " ' The first part of the query is asking for a Desktop PC, which is the same as the original query. The second part of the query is asking for a size of 10, which is not relevant to the original query. The third part of the query is asking for a minimum price of 0, which is not relevant to the original query. The fourth part of the query is asking for a maximum price of null, which is not relevant to the original query. Therefore, the Predicted Query does not semantically match the original query and is not likely to produce the same answer. Final Grade: F',\n",
+       " ' The original query is asking for cameras with a maximum price of 300. The predicted query is asking for cameras with a maximum price of 500. This means that the predicted query is likely to return more results than the original query, which may include cameras that are not within the budget range. Therefore, the predicted query is not semantically the same as the original query and does not answer the original question. Final Grade: F']"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "request_eval_results = []\n",
+    "for question, predict_query, truth_query in list(\n",
+    "    zip(questions, predicted_queries, truth_queries)\n",
+    "):\n",
+    "    eval_output = eval_chain.run(\n",
+    "        question=question,\n",
+    "        truth_query=truth_query,\n",
+    "        predict_query=predict_query,\n",
+    "    )\n",
+    "    request_eval_results.append(eval_output)\n",
+    "request_eval_results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "0d76f8ba",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "from typing import List\n",
+    "\n",
+    "\n",
+    "# Parse the evaluation chain responses into a rubric\n",
+    "def parse_eval_results(results: List[str]) -> List[float]:\n",
+    "    rubric = {\"A\": 1.0, \"B\": 0.75, \"C\": 0.5, \"D\": 0.25, \"F\": 0}\n",
+    "    return [rubric[re.search(r\"Final Grade: (\\w+)\", res).group(1)] for res in results]\n",
+    "\n",
+    "\n",
+    "parsed_results = parse_eval_results(request_eval_results)\n",
+    "# Collect the scores for a final evaluation table\n",
+    "scores[\"request_synthesizer\"].extend(parsed_results)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f3ee8ea",
+   "metadata": {},
+   "source": [
+    "## Evaluate the Response Chain\n",
+    "\n",
+    "The second component translated the structured API response to a natural language response.\n",
+    "Evaluate this against the user's original question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "8b97847c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import PromptTemplate\n",
+    "\n",
+    "template = \"\"\"You are trying to answer the following question by querying an API:\n",
+    "\n",
+    "> Question: {question}\n",
+    "\n",
+    "The API returned a response of:\n",
+    "\n",
+    "> API result: {api_response}\n",
+    "\n",
+    "Your response to the user: {answer}\n",
+    "\n",
+    "Please evaluate the accuracy and utility of your response to the user's original question, conditioned on the information available.\n",
+    "Give a letter grade of either an A, B, C, D, or F, along with an explanation of why. End the evaluation with 'Final Grade: <the letter>'\n",
+    "\n",
+    "> Explanation: Let's think step by step.\"\"\"\n",
+    "\n",
+    "prompt = PromptTemplate.from_template(template)\n",
+    "\n",
+    "eval_chain = LLMChain(llm=llm, prompt=prompt, verbose=verbose)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "642852ce",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Extract the API responses from the chain\n",
+    "api_responses = [\n",
+    "    output[\"intermediate_steps\"][\"response_text\"] for output in chain_outputs\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "08a5eb4f",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[' The original query is asking for all iPhone models, so the \"q\" parameter is correct. The \"max_price\" parameter is also correct, as it is set to null, meaning that no maximum price is set. The predicted query adds two additional parameters, \"size\" and \"min_price\". The \"size\" parameter is not necessary, as it is not relevant to the question being asked. The \"min_price\" parameter is also not necessary, as it is not relevant to the question being asked and it is set to 0, which is the default value. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',\n",
+       " ' The original query is asking for laptops with a maximum price of 300. The predicted query is asking for laptops with a minimum price of 0 and a maximum price of 500. This means that the predicted query is likely to return more results than the original query, as it is asking for a wider range of prices. Therefore, the predicted query is not semantically the same as the original query, and it is not likely to produce the same answer. Final Grade: F',\n",
+       " \" The first two parameters are the same, so that's good. The third parameter is different, but it's not necessary for the query, so that's not a problem. The fourth parameter is the problem. The original query specifies a maximum price of 500, while the predicted query specifies a maximum price of null. This means that the predicted query will not limit the results to the cheapest gaming PCs, so it is not semantically the same as the original query. Final Grade: F\",\n",
+       " ' The original query is asking for tablets under $400, so the first two parameters are correct. The predicted query also includes the parameters \"size\" and \"min_price\", which are not necessary for the original query. The \"size\" parameter is not relevant to the question, and the \"min_price\" parameter is redundant since the original query already specifies a maximum price. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',\n",
+       " ' The original query is asking for headphones with no maximum price, so the predicted query is not semantically the same because it has a maximum price of 500. The predicted query also has a size of 10, which is not specified in the original query. Therefore, the predicted query is not semantically the same as the original query. Final Grade: F',\n",
+       " \" The original query is asking for the top rated laptops, so the 'size' parameter should be set to 10 to get the top 10 results. The 'min_price' parameter should be set to 0 to get results from all price ranges. The 'max_price' parameter should be set to null to get results from all price ranges. The 'q' parameter should be set to 'laptop' to get results related to laptops. All of these parameters are present in the predicted query, so it is semantically the same as the original query. Final Grade: A\",\n",
+       " ' The original query is asking for shoes, so the predicted query is asking for the same thing. The original query does not specify a size, so the predicted query is not adding any additional information. The original query does not specify a price range, so the predicted query is adding additional information that is not necessary. Therefore, the predicted query is not semantically the same as the original query and is likely to produce different results. Final Grade: D',\n",
+       " ' The original query is asking for a skirt, so the predicted query is asking for the same thing. The predicted query also adds additional parameters such as size and price range, which could help narrow down the results. However, the size parameter is not necessary for the query to be successful, and the price range is too narrow. Therefore, the predicted query is not as effective as the original query. Final Grade: C',\n",
+       " ' The first part of the query is asking for a Desktop PC, which is the same as the original query. The second part of the query is asking for a size of 10, which is not relevant to the original query. The third part of the query is asking for a minimum price of 0, which is not relevant to the original query. The fourth part of the query is asking for a maximum price of null, which is not relevant to the original query. Therefore, the Predicted Query does not semantically match the original query and is not likely to produce the same answer. Final Grade: F',\n",
+       " ' The original query is asking for cameras with a maximum price of 300. The predicted query is asking for cameras with a maximum price of 500. This means that the predicted query is likely to return more results than the original query, which may include cameras that are not within the budget range. Therefore, the predicted query is not semantically the same as the original query and does not answer the original question. Final Grade: F',\n",
+       " ' The user asked a question about what iPhone models are available, and the API returned a response with 10 different models. The response provided by the user accurately listed all 10 models, so the accuracy of the response is A+. The utility of the response is also A+ since the user was able to get the exact information they were looking for. Final Grade: A+',\n",
+       " \" The API response provided a list of laptops with their prices and attributes. The user asked if there were any budget laptops, and the response provided a list of laptops that are all priced under $500. Therefore, the response was accurate and useful in answering the user's question. Final Grade: A\",\n",
+       " \" The API response provided the name, price, and URL of the product, which is exactly what the user asked for. The response also provided additional information about the product's attributes, which is useful for the user to make an informed decision. Therefore, the response is accurate and useful. Final Grade: A\",\n",
+       " \" The API response provided a list of tablets that are under $400. The response accurately answered the user's question. Additionally, the response provided useful information such as the product name, price, and attributes. Therefore, the response was accurate and useful. Final Grade: A\",\n",
+       " \" The API response provided a list of headphones with their respective prices and attributes. The user asked for the best headphones, so the response should include the best headphones based on the criteria provided. The response provided a list of headphones that are all from the same brand (Apple) and all have the same type of headphone (True Wireless, In-Ear). This does not provide the user with enough information to make an informed decision about which headphones are the best. Therefore, the response does not accurately answer the user's question. Final Grade: F\",\n",
+       " ' The API response provided a list of laptops with their attributes, which is exactly what the user asked for. The response provided a comprehensive list of the top rated laptops, which is what the user was looking for. The response was accurate and useful, providing the user with the information they needed. Final Grade: A',\n",
+       " ' The API response provided a list of shoes from both Adidas and Nike, which is exactly what the user asked for. The response also included the product name, price, and attributes for each shoe, which is useful information for the user to make an informed decision. The response also included links to the products, which is helpful for the user to purchase the shoes. Therefore, the response was accurate and useful. Final Grade: A',\n",
+       " \" The API response provided a list of skirts that could potentially meet the user's needs. The response also included the name, price, and attributes of each skirt. This is a great start, as it provides the user with a variety of options to choose from. However, the response does not provide any images of the skirts, which would have been helpful for the user to make a decision. Additionally, the response does not provide any information about the availability of the skirts, which could be important for the user. \\n\\nFinal Grade: B\",\n",
+       " ' The user asked for a professional desktop PC with no budget constraints. The API response provided a list of products that fit the criteria, including the Skytech Archangel Gaming Computer PC Desktop, the CyberPowerPC Gamer Master Gaming Desktop, and the ASUS ROG Strix G10DK-RS756. The response accurately suggested these three products as they all offer powerful processors and plenty of RAM. Therefore, the response is accurate and useful. Final Grade: A',\n",
+       " \" The API response provided a list of cameras with their prices, which is exactly what the user asked for. The response also included additional information such as features and memory cards, which is not necessary for the user's question but could be useful for further research. The response was accurate and provided the user with the information they needed. Final Grade: A\"]"
+      ]
+     },
+     "execution_count": 19,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Run the grader chain\n",
+    "response_eval_results = []\n",
+    "for question, api_response, answer in list(zip(questions, api_responses, answers)):\n",
+    "    request_eval_results.append(\n",
+    "        eval_chain.run(question=question, api_response=api_response, answer=answer)\n",
+    "    )\n",
+    "request_eval_results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "a144aa9d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Reusing the rubric from above, parse the evaluation chain responses\n",
+    "parsed_response_results = parse_eval_results(request_eval_results)\n",
+    "# Collect the scores for a final evaluation table\n",
+    "scores[\"result_synthesizer\"].extend(parsed_response_results)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "e95042bc",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Metric              \tMin       \tMean      \tMax       \n",
+      "completed           \t1.00      \t1.00      \t1.00      \n",
+      "request_synthesizer \t0.00      \t0.23      \t1.00      \n",
+      "result_synthesizer  \t0.00      \t0.55      \t1.00      \n"
+     ]
+    }
+   ],
+   "source": [
+    "# Print out Score statistics for the evaluation session\n",
+    "header = \"{:<20}\\t{:<10}\\t{:<10}\\t{:<10}\".format(\"Metric\", \"Min\", \"Mean\", \"Max\")\n",
+    "print(header)\n",
+    "for metric, metric_scores in scores.items():\n",
+    "    mean_scores = (\n",
+    "        sum(metric_scores) / len(metric_scores)\n",
+    "        if len(metric_scores) > 0\n",
+    "        else float(\"nan\")\n",
+    "    )\n",
+    "    row = \"{:<20}\\t{:<10.2f}\\t{:<10.2f}\\t{:<10.2f}\".format(\n",
+    "        metric, min(metric_scores), mean_scores, max(metric_scores)\n",
+    "    )\n",
+    "    print(row)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "03fe96af",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[]"
+      ]
+     },
+     "execution_count": 22,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Re-show the examples for which the chain failed to complete\n",
+    "failed_examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2bb3636d",
+   "metadata": {},
+   "source": [
+    "## Generating Test Datasets\n",
+    "\n",
+    "To evaluate a chain against your own endpoint, you'll want to generate a test dataset that's conforms to the API.\n",
+    "\n",
+    "This section provides an overview of how to bootstrap the process.\n",
+    "\n",
+    "First, we'll parse the OpenAPI Spec. For this example, we'll [Speak](https://www.speak.com/)'s OpenAPI specification."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "a453eb93",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.\n",
+      "Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load and parse the OpenAPI Spec\n",
+    "spec = OpenAPISpec.from_url(\"https://api.speak.com/openapi.yaml\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "bb65ffe8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['/v1/public/openai/explain-phrase',\n",
+       " '/v1/public/openai/explain-task',\n",
+       " '/v1/public/openai/translate']"
+      ]
+     },
+     "execution_count": 24,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# List the paths in the OpenAPI Spec\n",
+    "paths = sorted(spec.paths.keys())\n",
+    "paths"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "0988f01b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['post']"
+      ]
+     },
+     "execution_count": 25,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# See which HTTP Methods are available for a given path\n",
+    "methods = spec.get_methods_for_path(\"/v1/public/openai/explain-task\")\n",
+    "methods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "e9ef0a77",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "type explainTask = (_: {\n",
+      "/* Description of the task that the user wants to accomplish or do. For example, \"tell the waiter they messed up my order\" or \"compliment someone on their shirt\" */\n",
+      "  task_description?: string,\n",
+      "/* The foreign language that the user is learning and asking about. The value can be inferred from question - for example, if the user asks \"how do i ask a girl out in mexico city\", the value should be \"Spanish\" because of Mexico City. Always use the full name of the language (e.g. Spanish, French). */\n",
+      "  learning_language?: string,\n",
+      "/* The user's native language. Infer this value from the language the user asked their question in. Always use the full name of the language (e.g. Spanish, French). */\n",
+      "  native_language?: string,\n",
+      "/* A description of any additional context in the user's question that could affect the explanation - e.g. setting, scenario, situation, tone, speaking style and formality, usage notes, or any other qualifiers. */\n",
+      "  additional_context?: string,\n",
+      "/* Full text of the user's question. */\n",
+      "  full_query?: string,\n",
+      "}) => any;\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load a single endpoint operation\n",
+    "operation = APIOperation.from_openapi_spec(\n",
+    "    spec, \"/v1/public/openai/explain-task\", \"post\"\n",
+    ")\n",
+    "\n",
+    "# The operation can be serialized as typescript\n",
+    "print(operation.to_typescript())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "id": "f1186b6d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Compress the service definition to avoid leaking too much input structure to the sample data\n",
+    "template = \"\"\"In 20 words or less, what does this service accomplish?\n",
+    "{spec}\n",
+    "\n",
+    "Function: It's designed to \"\"\"\n",
+    "prompt = PromptTemplate.from_template(template)\n",
+    "generation_chain = LLMChain(llm=llm, prompt=prompt)\n",
+    "purpose = generation_chain.run(spec=operation.to_typescript())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "a594406a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[\"Can you explain how to say 'hello' in Spanish?\",\n",
+       " \"I need help understanding the French word for 'goodbye'.\",\n",
+       " \"Can you tell me how to say 'thank you' in German?\",\n",
+       " \"I'm trying to learn the Italian word for 'please'.\",\n",
+       " \"Can you help me with the pronunciation of 'yes' in Portuguese?\",\n",
+       " \"I'm looking for the Dutch word for 'no'.\",\n",
+       " \"Can you explain the meaning of 'hello' in Japanese?\",\n",
+       " \"I need help understanding the Russian word for 'thank you'.\",\n",
+       " \"Can you tell me how to say 'goodbye' in Chinese?\",\n",
+       " \"I'm trying to learn the Arabic word for 'please'.\"]"
+      ]
+     },
+     "execution_count": 28,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "template = \"\"\"Write a list of {num_to_generate} unique messages users might send to a service designed to{purpose} They must each be completely unique.\n",
+    "\n",
+    "1.\"\"\"\n",
+    "\n",
+    "\n",
+    "def parse_list(text: str) -> List[str]:\n",
+    "    # Match lines starting with a number then period\n",
+    "    # Strip leading and trailing whitespace\n",
+    "    matches = re.findall(r\"^\\d+\\. \", text)\n",
+    "    return [re.sub(r\"^\\d+\\. \", \"\", q).strip().strip('\"') for q in text.split(\"\\n\")]\n",
+    "\n",
+    "\n",
+    "num_to_generate = 10  # How many examples to use for this test set.\n",
+    "prompt = PromptTemplate.from_template(template)\n",
+    "generation_chain = LLMChain(llm=llm, prompt=prompt)\n",
+    "text = generation_chain.run(purpose=purpose, num_to_generate=num_to_generate)\n",
+    "# Strip preceding numeric bullets\n",
+    "queries = parse_list(text)\n",
+    "queries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "8dc60f43",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['{\"task_description\": \"say \\'hello\\'\", \"learning_language\": \"Spanish\", \"native_language\": \"English\", \"full_query\": \"Can you explain how to say \\'hello\\' in Spanish?\"}',\n",
+       " '{\"task_description\": \"understanding the French word for \\'goodbye\\'\", \"learning_language\": \"French\", \"native_language\": \"English\", \"full_query\": \"I need help understanding the French word for \\'goodbye\\'.\"}',\n",
+       " '{\"task_description\": \"say \\'thank you\\'\", \"learning_language\": \"German\", \"native_language\": \"English\", \"full_query\": \"Can you tell me how to say \\'thank you\\' in German?\"}',\n",
+       " '{\"task_description\": \"Learn the Italian word for \\'please\\'\", \"learning_language\": \"Italian\", \"native_language\": \"English\", \"full_query\": \"I\\'m trying to learn the Italian word for \\'please\\'.\"}',\n",
+       " '{\"task_description\": \"Help with pronunciation of \\'yes\\' in Portuguese\", \"learning_language\": \"Portuguese\", \"native_language\": \"English\", \"full_query\": \"Can you help me with the pronunciation of \\'yes\\' in Portuguese?\"}',\n",
+       " '{\"task_description\": \"Find the Dutch word for \\'no\\'\", \"learning_language\": \"Dutch\", \"native_language\": \"English\", \"full_query\": \"I\\'m looking for the Dutch word for \\'no\\'.\"}',\n",
+       " '{\"task_description\": \"Explain the meaning of \\'hello\\' in Japanese\", \"learning_language\": \"Japanese\", \"native_language\": \"English\", \"full_query\": \"Can you explain the meaning of \\'hello\\' in Japanese?\"}',\n",
+       " '{\"task_description\": \"understanding the Russian word for \\'thank you\\'\", \"learning_language\": \"Russian\", \"native_language\": \"English\", \"full_query\": \"I need help understanding the Russian word for \\'thank you\\'.\"}',\n",
+       " '{\"task_description\": \"say goodbye\", \"learning_language\": \"Chinese\", \"native_language\": \"English\", \"full_query\": \"Can you tell me how to say \\'goodbye\\' in Chinese?\"}',\n",
+       " '{\"task_description\": \"Learn the Arabic word for \\'please\\'\", \"learning_language\": \"Arabic\", \"native_language\": \"English\", \"full_query\": \"I\\'m trying to learn the Arabic word for \\'please\\'.\"}']"
+      ]
+     },
+     "execution_count": 29,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Define the generation chain to get hypotheses\n",
+    "api_chain = OpenAPIEndpointChain.from_api_operation(\n",
+    "    operation,\n",
+    "    llm,\n",
+    "    requests=Requests(),\n",
+    "    verbose=verbose,\n",
+    "    return_intermediate_steps=True,  # Return request and response text\n",
+    ")\n",
+    "\n",
+    "predicted_outputs = [api_chain(query) for query in queries]\n",
+    "request_args = [\n",
+    "    output[\"intermediate_steps\"][\"request_args\"] for output in predicted_outputs\n",
+    "]\n",
+    "\n",
+    "# Show the generated request\n",
+    "request_args"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "b727e28e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## AI Assisted Correction\n",
+    "correction_template = \"\"\"Correct the following API request based on the user's feedback. If the user indicates no changes are needed, output the original without making any changes.\n",
+    "\n",
+    "REQUEST: {request}\n",
+    "\n",
+    "User Feedback / requested changes: {user_feedback}\n",
+    "\n",
+    "Finalized Request: \"\"\"\n",
+    "\n",
+    "prompt = PromptTemplate.from_template(correction_template)\n",
+    "correction_chain = LLMChain(llm=llm, prompt=prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "c1f4d71f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Query: Can you explain how to say 'hello' in Spanish?\n",
+      "Request: {\"task_description\": \"say 'hello'\", \"learning_language\": \"Spanish\", \"native_language\": \"English\", \"full_query\": \"Can you explain how to say 'hello' in Spanish?\"}\n",
+      "Requested changes: \n",
+      "Query: I need help understanding the French word for 'goodbye'.\n",
+      "Request: {\"task_description\": \"understanding the French word for 'goodbye'\", \"learning_language\": \"French\", \"native_language\": \"English\", \"full_query\": \"I need help understanding the French word for 'goodbye'.\"}\n",
+      "Requested changes: \n",
+      "Query: Can you tell me how to say 'thank you' in German?\n",
+      "Request: {\"task_description\": \"say 'thank you'\", \"learning_language\": \"German\", \"native_language\": \"English\", \"full_query\": \"Can you tell me how to say 'thank you' in German?\"}\n",
+      "Requested changes: \n",
+      "Query: I'm trying to learn the Italian word for 'please'.\n",
+      "Request: {\"task_description\": \"Learn the Italian word for 'please'\", \"learning_language\": \"Italian\", \"native_language\": \"English\", \"full_query\": \"I'm trying to learn the Italian word for 'please'.\"}\n",
+      "Requested changes: \n",
+      "Query: Can you help me with the pronunciation of 'yes' in Portuguese?\n",
+      "Request: {\"task_description\": \"Help with pronunciation of 'yes' in Portuguese\", \"learning_language\": \"Portuguese\", \"native_language\": \"English\", \"full_query\": \"Can you help me with the pronunciation of 'yes' in Portuguese?\"}\n",
+      "Requested changes: \n",
+      "Query: I'm looking for the Dutch word for 'no'.\n",
+      "Request: {\"task_description\": \"Find the Dutch word for 'no'\", \"learning_language\": \"Dutch\", \"native_language\": \"English\", \"full_query\": \"I'm looking for the Dutch word for 'no'.\"}\n",
+      "Requested changes: \n",
+      "Query: Can you explain the meaning of 'hello' in Japanese?\n",
+      "Request: {\"task_description\": \"Explain the meaning of 'hello' in Japanese\", \"learning_language\": \"Japanese\", \"native_language\": \"English\", \"full_query\": \"Can you explain the meaning of 'hello' in Japanese?\"}\n",
+      "Requested changes: \n",
+      "Query: I need help understanding the Russian word for 'thank you'.\n",
+      "Request: {\"task_description\": \"understanding the Russian word for 'thank you'\", \"learning_language\": \"Russian\", \"native_language\": \"English\", \"full_query\": \"I need help understanding the Russian word for 'thank you'.\"}\n",
+      "Requested changes: \n",
+      "Query: Can you tell me how to say 'goodbye' in Chinese?\n",
+      "Request: {\"task_description\": \"say goodbye\", \"learning_language\": \"Chinese\", \"native_language\": \"English\", \"full_query\": \"Can you tell me how to say 'goodbye' in Chinese?\"}\n",
+      "Requested changes: \n",
+      "Query: I'm trying to learn the Arabic word for 'please'.\n",
+      "Request: {\"task_description\": \"Learn the Arabic word for 'please'\", \"learning_language\": \"Arabic\", \"native_language\": \"English\", \"full_query\": \"I'm trying to learn the Arabic word for 'please'.\"}\n",
+      "Requested changes: \n"
+     ]
+    }
+   ],
+   "source": [
+    "ground_truth = []\n",
+    "for query, request_arg in list(zip(queries, request_args)):\n",
+    "    feedback = input(f\"Query: {query}\\nRequest: {request_arg}\\nRequested changes: \")\n",
+    "    if feedback == \"n\" or feedback == \"none\" or not feedback:\n",
+    "        ground_truth.append(request_arg)\n",
+    "        continue\n",
+    "    resolved = correction_chain.run(request=request_arg, user_feedback=feedback)\n",
+    "    ground_truth.append(resolved.strip())\n",
+    "    print(\"Updated request:\", resolved)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "19d68882",
+   "metadata": {},
+   "source": [
+    "**Now you can use the `ground_truth` as shown above in [Evaluate the Requests Chain](#Evaluate-the-requests-chain)!**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "id": "5a596176",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['{\"task_description\": \"say \\'hello\\'\", \"learning_language\": \"Spanish\", \"native_language\": \"English\", \"full_query\": \"Can you explain how to say \\'hello\\' in Spanish?\"}',\n",
+       " '{\"task_description\": \"understanding the French word for \\'goodbye\\'\", \"learning_language\": \"French\", \"native_language\": \"English\", \"full_query\": \"I need help understanding the French word for \\'goodbye\\'.\"}',\n",
+       " '{\"task_description\": \"say \\'thank you\\'\", \"learning_language\": \"German\", \"native_language\": \"English\", \"full_query\": \"Can you tell me how to say \\'thank you\\' in German?\"}',\n",
+       " '{\"task_description\": \"Learn the Italian word for \\'please\\'\", \"learning_language\": \"Italian\", \"native_language\": \"English\", \"full_query\": \"I\\'m trying to learn the Italian word for \\'please\\'.\"}',\n",
+       " '{\"task_description\": \"Help with pronunciation of \\'yes\\' in Portuguese\", \"learning_language\": \"Portuguese\", \"native_language\": \"English\", \"full_query\": \"Can you help me with the pronunciation of \\'yes\\' in Portuguese?\"}',\n",
+       " '{\"task_description\": \"Find the Dutch word for \\'no\\'\", \"learning_language\": \"Dutch\", \"native_language\": \"English\", \"full_query\": \"I\\'m looking for the Dutch word for \\'no\\'.\"}',\n",
+       " '{\"task_description\": \"Explain the meaning of \\'hello\\' in Japanese\", \"learning_language\": \"Japanese\", \"native_language\": \"English\", \"full_query\": \"Can you explain the meaning of \\'hello\\' in Japanese?\"}',\n",
+       " '{\"task_description\": \"understanding the Russian word for \\'thank you\\'\", \"learning_language\": \"Russian\", \"native_language\": \"English\", \"full_query\": \"I need help understanding the Russian word for \\'thank you\\'.\"}',\n",
+       " '{\"task_description\": \"say goodbye\", \"learning_language\": \"Chinese\", \"native_language\": \"English\", \"full_query\": \"Can you tell me how to say \\'goodbye\\' in Chinese?\"}',\n",
+       " '{\"task_description\": \"Learn the Arabic word for \\'please\\'\", \"learning_language\": \"Arabic\", \"native_language\": \"English\", \"full_query\": \"I\\'m trying to learn the Arabic word for \\'please\\'.\"}']"
+      ]
+     },
+     "execution_count": 32,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Now you have a new ground truth set to use as shown above!\n",
+    "ground_truth"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b7fe9dfa",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/examples/qa_benchmarking_pg.ipynb

@@ -0,0 +1,372 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Question Answering Benchmarking: Paul Graham Essay\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task over a Paul Graham essay.\n",
+    "\n",
+    "It is highly recommended that you do any evaluation/benchmarking with tracing enabled. See [here](https://python.langchain.com/docs/modules/callbacks/how_to/tracing) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-paul-graham-76e8f711e038d742/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "9264acfe710b4faabf060f0fcf4f7308",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"question-answering-paul-graham\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "\n",
+    "loader = TextLoader(\"../../modules/paul_graham_essay.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "7f0de2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes import VectorstoreIndexCreator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "ef84ff99",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import RetrievalQA\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "573719a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = RetrievalQA.from_chain_type(\n",
+    "    llm=OpenAI(),\n",
+    "    chain_type=\"stuff\",\n",
+    "    retriever=vectorstore.as_retriever(),\n",
+    "    input_key=\"question\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53b5aa23",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "3f81d951",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What were the two main things the author worked on before college?',\n",
+       " 'answer': 'The two main things the author worked on before college were writing and programming.',\n",
+       " 'result': ' Writing and programming.'}"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain(dataset[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = chain.apply(dataset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What were the two main things the author worked on before college?',\n",
+       " 'answer': 'The two main things the author worked on before college were writing and programming.',\n",
+       " 'result': ' Writing and programming.'}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(\n",
+    "    dataset, predictions, question_key=\"question\", prediction_key=\"result\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction[\"grade\"] = graded_outputs[i][\"text\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 12, ' INCORRECT': 10})"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "Counter([pred[\"grade\"] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred[\"grade\"] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What did the author write their dissertation on?',\n",
+       " 'answer': 'The author wrote their dissertation on applications of continuations.',\n",
+       " 'result': ' The author does not mention what their dissertation was on, so it is not known.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/examples/qa_benchmarking_sota.ipynb

@@ -0,0 +1,385 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# Question Answering Benchmarking: State of the Union Address\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task over a state of the union address.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f127fb04",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-state-of-the-union-a7e5a3b2db4f440d/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"question-answering-state-of-the-union\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ab6a716",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c18680b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader\n",
+    "\n",
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "7f0de2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.indexes import VectorstoreIndexCreator"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "ef84ff99",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Running Chroma using direct local API.\n",
+      "Using DuckDB in-memory for database. Data will be transient.\n"
+     ]
+    }
+   ],
+   "source": [
+    "vectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a question answering chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains import RetrievalQA\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "573719a0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = RetrievalQA.from_chain_type(\n",
+    "    llm=OpenAI(),\n",
+    "    chain_type=\"stuff\",\n",
+    "    retriever=vectorstore.as_retriever(),\n",
+    "    input_key=\"question\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "37d669e9",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "3089e409",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'result': ' The NATO Alliance was created to secure peace and stability in Europe after World War 2.'}"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain(dataset[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = chain.apply(dataset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "49d969fb",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. The first thing we can do is look at them by eye."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "1d583f03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the purpose of the NATO Alliance?',\n",
+       " 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',\n",
+       " 'result': ' The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'}"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "Next, we can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(\n",
+    "    dataset, predictions, question_key=\"question\", prediction_key=\"result\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction[\"grade\"] = graded_outputs[i][\"text\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 7, ' INCORRECT': 4})"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "Counter([pred[\"grade\"] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred[\"grade\"] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',\n",
+       " 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.',\n",
+       " 'result': ' The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and is naming a chief prosecutor for pandemic fraud.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/examples/qa_generation.ipynb

@@ -2,100 +2,93 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "c04293ac",
+   "id": "ee2a3a21",
    "metadata": {},
    "source": [
-    "# SQL Query\n",
-    "\n",
-    "This notebook walks through how to load and run a chain that constructs SQL queries that can be run against your database to answer a question. Note that this ONLY constructs the query and does not run it."
+    "# QA Generation\n",
+    "This notebook shows how to use the `QAGenerationChain` to come up with question-answer pairs over a specific document.\n",
+    "This is important because often times you may not have data to evaluate your question-answer system over, so this is a cheap and lightweight way to generate it!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "33d3f0b4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import TextLoader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "2029a29c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "87edb84c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "doc = loader.load()[0]"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "e9063a93",
+   "id": "04125b6d",
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import create_sql_query_chain\n",
-    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.utilities import SQLDatabase"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "a1ff5cee",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "db = SQLDatabase.from_uri(\"sqlite:///../../../../notebooks/Chinook.db\")"
+    "from langchain.chains import QAGenerationChain\n",
+    "\n",
+    "chain = QAGenerationChain.from_llm(ChatOpenAI(temperature=0))"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "cb04579f",
+   "id": "4f1593e4",
    "metadata": {},
    "outputs": [],
    "source": [
-    "chain = create_sql_query_chain(ChatOpenAI(temperature=0), db)"
+    "qa = chain.run(doc.page_content)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "id": "744e6210",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "response = chain.invoke({\"question\":\"How many employees are there\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "28f984f1",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "SELECT COUNT(*) FROM Employee\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(response)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "08de511c",
+   "execution_count": 10,
+   "id": "ee831f92",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "'[(8,)]'"
+       "{'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',\n",
+       " 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.'}"
       ]
      },
-     "execution_count": 15,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "db.run(response)"
+    "qa[1]"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e3a006a7",
+   "id": "7028754e",
    "metadata": {},
    "outputs": [],
    "source": []

docs/extras/guides/evaluation/examples/question_answering.ipynb

@@ -0,0 +1,445 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "480b7cf8",
+   "metadata": {},
+   "source": [
+    "# Question Answering\n",
+    "\n",
+    "This notebook covers how to evaluate generic question answering problems. This is a situation where you have an example containing a question and its corresponding ground truth answer, and you want to measure how well the language model does at answering those questions."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "78e3023b",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "For demonstration purposes, we will just evaluate a simple question answering system that only evaluates the model's internal knowledge. Please see other notebooks for examples where it evaluates how the model does at question answering over data not present in what the model was trained on."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "96710d50",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "e33ccf00",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt = PromptTemplate(\n",
+    "    template=\"Question: {question}\\nAnswer:\", input_variables=[\"question\"]\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "172d993a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(model_name=\"text-davinci-003\", temperature=0)\n",
+    "chain = LLMChain(llm=llm, prompt=prompt)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0c584440",
+   "metadata": {},
+   "source": [
+    "## Examples\n",
+    "For this purpose, we will just use two simple hardcoded examples, but see other notebooks for tips on how to get and/or generate these examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "87de1d84",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "examples = [\n",
+    "    {\n",
+    "        \"question\": \"Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?\",\n",
+    "        \"answer\": \"11\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"question\": 'Is the following sentence plausible? \"Joao Moutinho caught the screen pass in the NFC championship.\"',\n",
+    "        \"answer\": \"No\",\n",
+    "    },\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "143b1155",
+   "metadata": {},
+   "source": [
+    "## Predictions\n",
+    "\n",
+    "We can now make and inspect the predictions for these questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "c7bd809c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = chain.apply(examples)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "f06dceab",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'text': ' 11 tennis balls'},\n",
+       " {'text': ' No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.'}]"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "45cc2f9d",
+   "metadata": {},
+   "source": [
+    "## Evaluation\n",
+    "\n",
+    "We can see that if we tried to just do exact match on the answer answers (`11` and `No`) they would not match what the language model answered. However, semantically the language model is correct in both cases. In order to account for this, we can use a language model itself to evaluate the answers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "0cacc65a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "5aa6cd65",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(\n",
+    "    examples, predictions, question_key=\"question\", prediction_key=\"text\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "63780020",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Example 0:\n",
+      "Question: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?\n",
+      "Real Answer: 11\n",
+      "Predicted Answer:  11 tennis balls\n",
+      "Predicted Grade:  CORRECT\n",
+      "\n",
+      "Example 1:\n",
+      "Question: Is the following sentence plausible? \"Joao Moutinho caught the screen pass in the NFC championship.\"\n",
+      "Real Answer: No\n",
+      "Predicted Answer:  No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.\n",
+      "Predicted Grade:  CORRECT\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i, eg in enumerate(examples):\n",
+    "    print(f\"Example {i}:\")\n",
+    "    print(\"Question: \" + eg[\"question\"])\n",
+    "    print(\"Real Answer: \" + eg[\"answer\"])\n",
+    "    print(\"Predicted Answer: \" + predictions[i][\"text\"])\n",
+    "    print(\"Predicted Grade: \" + graded_outputs[i][\"text\"])\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "782ae8c8",
+   "metadata": {},
+   "source": [
+    "## Customize Prompt\n",
+    "\n",
+    "You can also customize the prompt that is used. Here is an example prompting it using a score from 0 to 10.\n",
+    "The custom prompt requires 3 input variables: \"query\", \"answer\" and \"result\". Where \"query\" is the question, \"answer\" is the ground truth answer, and \"result\" is the predicted answer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "153425c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts.prompt import PromptTemplate\n",
+    "\n",
+    "_PROMPT_TEMPLATE = \"\"\"You are an expert professor specialized in grading students' answers to questions.\n",
+    "You are grading the following question:\n",
+    "{query}\n",
+    "Here is the real answer:\n",
+    "{answer}\n",
+    "You are grading the following predicted answer:\n",
+    "{result}\n",
+    "What grade do you give from 0 to 10, where 0 is the lowest (very low similarity) and 10 is the highest (very high similarity)?\n",
+    "\"\"\"\n",
+    "\n",
+    "PROMPT = PromptTemplate(\n",
+    "    input_variables=[\"query\", \"answer\", \"result\"], template=_PROMPT_TEMPLATE\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0a3b0fb7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "evalchain = QAEvalChain.from_llm(llm=llm, prompt=PROMPT)\n",
+    "evalchain.evaluate(\n",
+    "    examples,\n",
+    "    predictions,\n",
+    "    question_key=\"question\",\n",
+    "    answer_key=\"answer\",\n",
+    "    prediction_key=\"text\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb1cf335",
+   "metadata": {},
+   "source": [
+    "## Evaluation without Ground Truth\n",
+    "Its possible to evaluate question answering systems without ground truth. You would need a `\"context\"` input that reflects what the information the LLM uses to answer the question. This context can be obtained by any retreival system. Here's an example of how it works:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6c59293f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "context_examples = [\n",
+    "    {\n",
+    "        \"question\": \"How old am I?\",\n",
+    "        \"context\": \"I am 30 years old. I live in New York and take the train to work everyday.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"question\": 'Who won the NFC championship game in 2023?\"',\n",
+    "        \"context\": \"NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7\",\n",
+    "    },\n",
+    "]\n",
+    "QA_PROMPT = \"Answer the question based on the  context\\nContext:{context}\\nQuestion:{question}\\nAnswer:\"\n",
+    "template = PromptTemplate(input_variables=[\"context\", \"question\"], template=QA_PROMPT)\n",
+    "qa_chain = LLMChain(llm=llm, prompt=template)\n",
+    "predictions = qa_chain.apply(context_examples)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "e500d0cc",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'text': 'You are 30 years old.'},\n",
+       " {'text': ' The Philadelphia Eagles won the NFC championship game in 2023.'}]"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "6d8cbc1d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import ContextQAEvalChain\n",
+    "\n",
+    "eval_chain = ContextQAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(\n",
+    "    context_examples, predictions, question_key=\"question\", prediction_key=\"text\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "6c5262d0",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'text': ' CORRECT'}, {'text': ' CORRECT'}]"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "graded_outputs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aaa61f0c",
+   "metadata": {},
+   "source": [
+    "## Comparing to other evaluation metrics\n",
+    "We can compare the evaluation results we get to other common evaluation metrics. To do this, let's load some evaluation metrics from HuggingFace's `evaluate` package."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "d851453b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Some data munging to get the examples in the right format\n",
+    "for i, eg in enumerate(examples):\n",
+    "    eg[\"id\"] = str(i)\n",
+    "    eg[\"answers\"] = {\"text\": [eg[\"answer\"]], \"answer_start\": [0]}\n",
+    "    predictions[i][\"id\"] = str(i)\n",
+    "    predictions[i][\"prediction_text\"] = predictions[i][\"text\"]\n",
+    "\n",
+    "for p in predictions:\n",
+    "    del p[\"text\"]\n",
+    "\n",
+    "new_examples = examples.copy()\n",
+    "for eg in new_examples:\n",
+    "    del eg[\"question\"]\n",
+    "    del eg[\"answer\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "c38eb3e9",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "from evaluate import load\n",
+    "\n",
+    "squad_metric = load(\"squad\")\n",
+    "results = squad_metric.compute(\n",
+    "    references=new_examples,\n",
+    "    predictions=predictions,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "07d68f85",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'exact_match': 0.0, 'f1': 28.125}"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3b775150",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "53f3bc57609c7a84333bb558594977aa5b4026b1d6070b93987956689e367341"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/examples/sql_qa_benchmarking_chinook.ipynb

@@ -0,0 +1,428 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "984169ca",
+   "metadata": {},
+   "source": [
+    "# SQL Question Answering Benchmarking: Chinook\n",
+    "\n",
+    "Here we go over how to benchmark performance on a question answering task over a SQL database.\n",
+    "\n",
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "44874486",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Comment this out if you are NOT using tracing\n",
+    "import os\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_HANDLER\"] = \"langchain\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f66405e",
+   "metadata": {},
+   "source": [
+    "## Loading the data\n",
+    "\n",
+    "First, let's load the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "0df1393f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b220d07ee5d14909bc842b4545cdc0de",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading readme:   0%|          | 0.00/21.0 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Downloading and preparing dataset json/LangChainDatasets--sql-qa-chinook to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "e89e3c8ef76f49889c4b39c624828c71",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a8421df6c26045e8978c7086cb418222",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading data:   0%|          | 0.00/1.44k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d1fb6becc3324a85bf039a53caf30924",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Generating train split: 0 examples [00:00, ? examples/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Dataset json downloaded and prepared to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51. Subsequent calls will reuse this data.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "9d68ad1b3e4a4bd79f92597aac4d3cc9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"sql-qa-chinook\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "ab44d504",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'How many employees are there?', 'answer': '8'}"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a16b75d",
+   "metadata": {},
+   "source": [
+    "## Setting up a chain\n",
+    "This uses the example Chinook database.\n",
+    "To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the `.db` file in a notebooks folder at the root of this repository.\n",
+    "\n",
+    "Note that here we load a simple chain. If you want to experiment with more complex chains, or an agent, just create the `chain` object in a different way."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5b2d5e98",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain import OpenAI, SQLDatabase, SQLDatabaseChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "33cdcbfc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db = SQLDatabase.from_uri(\"sqlite:///../../../notebooks/Chinook.db\")\n",
+    "llm = OpenAI(temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0b5d8f6",
+   "metadata": {},
+   "source": [
+    "Now we can create a SQL database chain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "8843cb0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = SQLDatabaseChain.from_llm(llm, db, input_key=\"question\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6c0062e7",
+   "metadata": {},
+   "source": [
+    "## Make a prediction\n",
+    "\n",
+    "First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "id": "d28c5e7d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'How many employees are there?',\n",
+       " 'answer': '8',\n",
+       " 'result': ' There are 8 employees.'}"
+      ]
+     },
+     "execution_count": 27,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain(dataset[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0c16cd7",
+   "metadata": {},
+   "source": [
+    "## Make many predictions\n",
+    "Now we can make predictions. Note that we add a try-except because this chain can sometimes error (if SQL is written incorrectly, etc)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "24b4c66e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = []\n",
+    "predicted_dataset = []\n",
+    "error_dataset = []\n",
+    "for data in dataset:\n",
+    "    try:\n",
+    "        predictions.append(chain(data))\n",
+    "        predicted_dataset.append(data)\n",
+    "    except:\n",
+    "        error_dataset.append(data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4783344b",
+   "metadata": {},
+   "source": [
+    "## Evaluate performance\n",
+    "Now we can evaluate the predictions. We can use a language model to score them programatically"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "d0a9341d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa import QAEvalChain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "1612dec1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "eval_chain = QAEvalChain.from_llm(llm)\n",
+    "graded_outputs = eval_chain.evaluate(\n",
+    "    predicted_dataset, predictions, question_key=\"question\", prediction_key=\"result\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "79587806",
+   "metadata": {},
+   "source": [
+    "We can add in the graded output to the `predictions` dict and then get a count of the grades."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "2a689df5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i, prediction in enumerate(predictions):\n",
+    "    prediction[\"grade\"] = graded_outputs[i][\"text\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "27b61215",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Counter({' CORRECT': 3, ' INCORRECT': 4})"
+      ]
+     },
+     "execution_count": 24,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "Counter([pred[\"grade\"] for pred in predictions])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12fe30f4",
+   "metadata": {},
+   "source": [
+    "We can also filter the datapoints to the incorrect examples and look at them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "47c692a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "incorrect = [pred for pred in predictions if pred[\"grade\"] == \" INCORRECT\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "0ef976c1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'question': 'How many employees are also customers?',\n",
+       " 'answer': 'None',\n",
+       " 'result': ' 59 employees are also customers.',\n",
+       " 'grade': ' INCORRECT'}"
+      ]
+     },
+     "execution_count": 26,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "incorrect[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7710401a",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/string/Untitled.ipynb

@@ -1,318 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "bce7335e-f3b2-44f3-90cc-8c0a23a89a21",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from langchain.agents import load_tools\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.utilities import GoogleSearchAPIWrapper\n",
-    "from langchain.schema import (\n",
-    "    SystemMessage,\n",
-    "    HumanMessage,\n",
-    "    AIMessage\n",
-    ")\n",
-    "\n",
-    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
-    "# os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
-    "# os.environ[\"LANGCHAIN_API_KEY\"] = \"******\"\n",
-    "# os.environ[\"LANGCHAIN_PROJECT\"] = \"Jarvis\"\n",
-    "\n",
-    "\n",
-    "prefix_messages = [{\"role\": \"system\", \"content\": \"You are a helpful discord Chatbot.\"}]\n",
-    "\n",
-    "llm = ChatOpenAI(model_name='gpt-3.5-turbo', \n",
-    "             temperature=0.5, \n",
-    "             max_tokens = 2000)\n",
-    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
-    "agent = initialize_agent(tools,\n",
-    "                         llm,\n",
-    "                         agent=\"zero-shot-react-description\",\n",
-    "                         verbose=True,\n",
-    "                         handle_parsing_errors=True\n",
-    "                         )\n",
-    "\n",
-    "\n",
-    "async def on_ready():\n",
-    "    print(f'{bot.user} has connected to Discord!')\n",
-    "\n",
-    "async def on_message(message):\n",
-    "\n",
-    "    print(\"Detected bot name in message:\", message.content)\n",
-    "\n",
-    "    # Capture the output of agent.run() in the response variable\n",
-    "    response = agent.run(message.content)\n",
-    "\n",
-    "    while response:\n",
-    "        print(response)\n",
-    "        chunk, response = response[:2000], response[2000:]\n",
-    "        print(f\"Chunk: {chunk}\")\n",
-    "    print(\"Response sent.\")\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "1551ce9f-b6de-4035-b6d6-825722823b48",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from dataclasses import dataclass\n",
-    "@dataclass\n",
-    "class Message:\n",
-    "    content: str"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "id": "6e6859ec-8544-4407-9663-6b53c0092903",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Detected bot name in message: Hi AI, how are you today?\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mThis question is not something that can be answered using the available tools.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to follow the correct format for answering questions.\n",
-      "Action: N/A\u001b[0m\n",
-      "Observation: Invalid Format: Missing 'Action Input:' after 'Action:'\n",
-      "Thought:\u001b[32;1m\u001b[1;3m\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "Agent stopped due to iteration limit or time limit.\n",
-      "Chunk: Agent stopped due to iteration limit or time limit.\n",
-      "Response sent.\n"
-     ]
-    }
-   ],
-   "source": [
-    "await on_message(Message(content=\"Hi AI, how are you today?\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "b850294c-7f8f-4e79-adcf-47e4e3a898df",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langsmith import Client\n",
-    "\n",
-    "client = Client()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "6d089ddc-69bc-45a8-b8db-9962e4f1f5ee",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from itertools import islice\n",
-    "\n",
-    "runs = list(islice(client.list_runs(), 10))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 38,
-   "id": "f0349fac-5a98-400f-ba03-61ed4e1332be",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "runs = sorted(runs, key=lambda x: x.start_time, reverse=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "02f133f0-39ee-4b46-b443-12c1f9b76fff",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "ids = [run.id for run in runs]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 39,
-   "id": "3366dce4-0c38-4a7d-8111-046a58b24917",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "runs2 = list(client.list_runs(id=ids))\n",
-    "runs2 = sorted(runs2, key=lambda x: x.start_time, reverse=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "82915b90-39a0-47d6-9121-56a13f210f52",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['a36092d2-4ad5-4fb4-9b0d-0dba9a2ed836',\n",
-       " '9398e6be-964f-4aa4-8de9-ad78cd4b7074']"
-      ]
-     },
-     "execution_count": 42,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "[str(x) for x in ids[:2]]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 48,
-   "id": "f610ec91-dc48-4a17-91c5-5c4675c77abc",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langsmith.run_helpers import traceable\n",
-    "\n",
-    "@traceable(run_type=\"llm\", name=\"\"\"<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/dQw4w9WgXcQ?start=5\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen></iframe>\"\"\")\n",
-    "def foo():\n",
-    "    return \"bar\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 49,
-   "id": "bd317bd7-8b2a-433a-8ec3-098a84ba8e64",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'bar'"
-      ]
-     },
-     "execution_count": 49,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "foo()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 52,
-   "id": "b142519b-6885-415c-83b9-4a346fb90589",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.llms import AzureOpenAI"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "5c50bb2b-72b8-4322-9b16-d857ecd9f347",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/guides/evaluation/string/criteria_eval_chain.ipynb

@@ -5,13 +5,16 @@
    "id": "4cf569a7-9a1d-4489-934e-50e57760c907",
    "metadata": {},
    "source": [
-    "# Criteria Evaluation\n",
+    "# Evaluating Custom Criteria\n",
     "\n",
-    "In scenarios where you wish to assess a model's output using a specific rubric or criteria set, the `criteria` evaluator proves to be a handy tool. It allows you to verify if an LLM or Chain's output complies with a defined set of criteria.\n",
+    "Suppose you want to test a model's output against a custom rubric or custom set of criteria, how would you go about testing this?\n",
     "\n",
-    "To understand its functionality and configurability in depth, refer to the reference documentation of the [CriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain) class.\n",
+    "The `criteria` evaluator is a convenient way to predict whether an LLM or Chain's output complies with a set of criteria, so long as you can\n",
+    "properly define those criteria.\n",
     "\n",
-    "### Usage without references\n",
+    "For more details, check out the reference docs for the [CriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain)'s class definition.\n",
+    "\n",
+    "### Without References\n",
     "\n",
     "In this example, you will use the `CriteriaEvalChain` to check whether an output is concise. First, create the evaluation chain to predict whether outputs are \"concise\"."
    ]
@@ -59,24 +62,6 @@
     "print(eval_result)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "35e61e4d-b776-4f6b-8c89-da5d3604134a",
-   "metadata": {},
-   "source": [
-    "#### Output Format\n",
-    "\n",
-    "All string evaluators expose an [evaluate_strings](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html?highlight=evaluate_strings#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.evaluate_strings) (or async [aevaluate_strings](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html?highlight=evaluate_strings#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.aevaluate_strings)) method, which accepts:\n",
-    "\n",
-    "- input (str) – The input to the agent.\n",
-    "- prediction (str) – The predicted response.\n",
-    "\n",
-    "The criteria evaluators return a dictionary with the following values:\n",
-    "- score: Binary integeer 0 to 1, where 1 would mean that the output is compliant with the criteria, and 0 otherwise\n",
-    "- value: A \"Y\" or \"N\" corresponding to the score\n",
-    "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "c40b1ac7-8f95-48ed-89a2-623bcc746461",
@@ -84,7 +69,7 @@
    "source": [
     "## Using Reference Labels\n",
     "\n",
-    "Some criteria (such as correctness) require reference labels to work correctly. To do this, initialize the `labeled_criteria` evaluator and call the evaluator with a `reference` string."
+    "Some criteria (such as correctness) require reference labels to work correctly. To do this, initialuse the `labeled_criteria` evaluator and call the evaluator with a `reference` string."
    ]
   },
   {
@@ -123,7 +108,7 @@
     "**Default Criteria**\n",
     "\n",
     "Most of the time, you'll want to define your own custom criteria (see below), but we also provide some common criteria you can load with a single string.\n",
-    "Here's a list of pre-implemented criteria. Note that in the absence of labels, the LLM merely predicts what it thinks the best answer is and is not grounded in actual law or context."
+    "Here's a list of pre-implemented criteria:"
    ]
   },
   {
@@ -169,23 +154,20 @@
     "\n",
     "To evaluate outputs against your own custom criteria, or to be more explicit the definition of any of the default criteria, pass in a dictionary of `\"criterion_name\": \"criterion_description\"`\n",
     "\n",
-    "Note: it's recommended that you create a single evaluator per criterion. This way, separate feedback can be provided for each aspect. Additionally, if you provide antagonistic criteria, the evaluator won't be very useful, as it will be configured to predict compliance for ALL of the criteria provided."
+    "Note: the evaluator still predicts whether the output complies with ALL of the criteria provided. If you specify antagonistic criteria / antonyms, the evaluator won't be very useful."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 8,
    "id": "bafa0a11-2617-4663-84bf-24df7d0736be",
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "{'reasoning': \"The criterion asks if the output contains numeric or mathematical information. The joke in the submission does contain mathematical information. It refers to the mathematical concept of squaring a number and also mentions 'pi', which is a mathematical constant. Therefore, the submission does meet the criterion.\\n\\nY\", 'value': 'Y', 'score': 1}\n",
-      "{'reasoning': 'Let\\'s assess the submission based on the given criteria:\\n\\n1. Numeric: The output does not contain any explicit numeric information. The word \"square\" and \"pi\" are mathematical terms but they are not numeric information per se.\\n\\n2. Mathematical: The output does contain mathematical information. The terms \"square\" and \"pi\" are mathematical terms. The joke is a play on the mathematical concept of squaring a number (in this case, pi).\\n\\n3. Grammatical: The output is grammatically correct. The sentence structure, punctuation, and word usage are all correct.\\n\\n4. Logical: The output is logical. It makes sense within the context of the joke. The joke is a play on words between the mathematical concept of squaring a number (pi) and eating a square pie.\\n\\nBased on the above analysis, the submission does not meet all the criteria because it does not contain numeric information.\\nN', 'value': 'N', 'score': 0}\n"
+      "{'reasoning': 'The criterion asks if the output contains numeric or mathematical information. \\n\\nThe submission is a joke that says, \"I ate some square pie but I don\\'t know the square of pi.\" \\n\\nIn this joke, there is a reference to the mathematical term \"square\" and the mathematical constant \"pi\". \\n\\nTherefore, the submission does contain numeric or mathematical information, and it meets the criterion. \\n\\nY', 'value': 'Y', 'score': 1}\n"
      ]
     }
    ],
@@ -199,22 +181,6 @@
     "query = \"Tell me a joke\"\n",
     "prediction = \"I ate some square pie but I don't know the square of pi.\"\n",
     "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
-    "print(eval_result)\n",
-    "\n",
-    "# If you wanted to specify multiple criteria. Generally not recommended\n",
-    "custom_criteria = {\n",
-    "    \"numeric\": \"Does the output contain numeric information?\",\n",
-    "    \"mathematical\": \"Does the output contain mathematical information?\",\n",
-    "    \"grammatical\": \"Is the output grammatically correct?\",\n",
-    "    \"logical\": \"Is the output logical?\",\n",
-    "}\n",
-    "\n",
-    "eval_chain = load_evaluator(\n",
-    "    EvaluatorType.CRITERIA,\n",
-    "    criteria=custom_criteria,\n",
-    ")\n",
-    "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
-    "print(\"Multi-criteria evaluation\")\n",
     "print(eval_result)"
    ]
   },
@@ -436,12 +402,6 @@
     "\n",
     "Remember when selecting criteria to decide whether they ought to require ground truth labels or not. Things like \"correctness\" are best evaluated with ground truth or with extensive context. Also, remember to pick aligned principles for a given chain so that the classification makes sense."
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a684e2f1",
-   "metadata": {},
-   "source": []
   }
  ],
  "metadata": {

docs/extras/guides/evaluation/string/qa.ipynb

@@ -0,0 +1,227 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d63696a8-d035-4cf7-9605-c3210f0b551d",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "# QA Correctness\n",
+    "\n",
+    "When thinking about a QA system, one of the most important questions to ask is whether the final generated result is correct. The `\"qa\"` evaluator compares a question-answering model's response to a reference answer to provide this level of information. If you are able to annotate a test dataset, this evaluator will be useful.\n",
+    "\n",
+    "For more details, check out the reference docs for the [QAEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.qa.eval_chain.QAEvalChain.html#langchain.evaluation.qa.eval_chain.QAEvalChain)'s class definition."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9672fdb9-b53f-41e4-8f72-f21d11edbeac",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.evaluation import load_evaluator\n",
+    "\n",
+    "llm = ChatOpenAI(model=\"gpt-4\", temperature=0)\n",
+    "\n",
+    "# Note: the eval_llm is optional. A gpt-4 model will be provided by default if not specified\n",
+    "evaluator = load_evaluator(\"qa\", eval_llm=llm)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "b4db474a-9c9d-473f-81b1-55070ee584a6",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'reasoning': None, 'value': 'CORRECT', 'score': 1}"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "evaluator.evaluate_strings(\n",
+    "    input=\"What's last quarter's sales numbers?\",\n",
+    "    prediction=\"Last quarter we sold 600,000 total units of product.\",\n",
+    "    reference=\"Last quarter we sold 100,000 units of product A, 210,000 units of product B, and 300,000 units of product C.\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a5b345aa-7f45-4eea-bedf-9b0d5e824be3",
+   "metadata": {},
+   "source": [
+    "## SQL Correctness\n",
+    "\n",
+    "You can use an LLM to check the equivalence of a SQL query against a reference SQL query using the sql prompt."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "6c803b8c-fe1f-4fb7-8ea0-d9c67b855eb3",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.evaluation.qa.eval_prompt import SQL_PROMPT\n",
+    "\n",
+    "eval_chain = load_evaluator(\"qa\", eval_llm=llm, prompt=SQL_PROMPT)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "e28b8d07-248f-405c-bcef-e0ebe3a05c3e",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'reasoning': 'The expert answer and the submission are very similar in their structure and logic. Both queries are trying to calculate the sum of sales amounts for the last quarter. They both use the SUM function to add up the sale_amount from the sales table. They also both use the same WHERE clause to filter the sales data to only include sales from the last quarter. The WHERE clause uses the DATEADD function to subtract 1 quarter from the current date (GETDATE()) and only includes sales where the sale_date is greater than or equal to this date and less than the current date.\\n\\nThe main difference between the two queries is that the expert answer uses a subquery to first select the sale_amount from the sales table with the appropriate date filter, and then sums these amounts in the outer query. The submission, on the other hand, does not use a subquery and instead sums the sale_amount directly in the main query with the same date filter.\\n\\nHowever, this difference does not affect the result of the query. Both queries will return the same result, which is the sum of the sales amounts for the last quarter.\\n\\nCORRECT',\n",
+       " 'value': 'CORRECT',\n",
+       " 'score': 1}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "eval_chain.evaluate_strings(\n",
+    "    input=\"What's last quarter's sales numbers?\",\n",
+    "    prediction=\"\"\"SELECT SUM(sale_amount) AS last_quarter_sales\n",
+    "FROM sales\n",
+    "WHERE sale_date >= DATEADD(quarter, -1, GETDATE()) AND sale_date < GETDATE();\n",
+    "\"\"\",\n",
+    "    reference=\"\"\"SELECT SUM(sub.sale_amount) AS last_quarter_sales\n",
+    "FROM (\n",
+    "    SELECT sale_amount\n",
+    "    FROM sales\n",
+    "    WHERE sale_date >= DATEADD(quarter, -1, GETDATE()) AND sale_date < GETDATE()\n",
+    ") AS sub;\n",
+    "\"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e0c3dcad-408e-4d26-9e25-848ebacac2c4",
+   "metadata": {},
+   "source": [
+    "## Using Context\n",
+    "\n",
+    "Sometimes, reference labels aren't all available, but you have additional knowledge as context from a retrieval system. Often there may be additional information that isn't available to the model you want to evaluate. For this type of scenario, you can use the [ContextQAEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.qa.eval_chain.ContextQAEvalChain.html#langchain.evaluation.qa.eval_chain.ContextQAEvalChain)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "9f3ae116-3a2f-461d-ba6f-7352b42c1b0c",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'reasoning': None, 'value': 'CORRECT', 'score': 1}"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "eval_chain = load_evaluator(\"context_qa\", eval_llm=llm)\n",
+    "\n",
+    "eval_chain.evaluate_strings(\n",
+    "    input=\"Who won the NFC championship game in 2023?\",\n",
+    "    prediction=\"Eagles\",\n",
+    "    reference=\"NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ba5eac17-08b6-4e4f-a896-79e7fc637018",
+   "metadata": {},
+   "source": [
+    "## CoT With Context\n",
+    "\n",
+    "The same prompt strategies such as chain of thought can be used to make the evaluation results more reliable.\n",
+    "The [CotQAEvalChain's](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.qa.eval_chain.CotQAEvalChain.html#langchain.evaluation.qa.eval_chain.CotQAEvalChain) default prompt instructs the model to do this."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "26e3b686-98f4-45a5-9854-7071ec2893f1",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'reasoning': 'The student\\'s answer is \"Eagles\". The context states that the Philadelphia Eagles won the NFC championship game in 2023. Therefore, the student\\'s answer matches the information provided in the context.',\n",
+       " 'value': 'GRADE: CORRECT',\n",
+       " 'score': 1}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "eval_chain = load_evaluator(\"cot_qa\", eval_llm=llm)\n",
+    "\n",
+    "eval_chain.evaluate_strings(\n",
+    "    input=\"Who won the NFC championship game in 2023?\",\n",
+    "    prediction=\"Eagles\",\n",
+    "    reference=\"NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7\",\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/trajectory/custom.ipynb

@@ -67,7 +67,7 @@
    "id": "297dea4b-fb28-4292-b6e0-1c769cfb9cbd",
    "metadata": {},
    "source": [
-    "The example above will return a score of 1 if the language model predicts that any of the actions were unnecessary, and it returns a score of 0 if all of them were predicted to be necessary. It returns the string 'decision' as the 'value', and includes the rest of the generated text as 'reasoning' to let you audit the decision.\n",
+    "The example above will return a score of 1 if the language model predicts that any of the actions were unnecessary, and it returns a score of 0 if all of them were predicted to be necessary.\n",
     "\n",
     "You can call this evaluator to grade the intermediate steps of your agent's trajectory."
    ]

docs/extras/guides/evaluation/trajectory/trajectory_eval.ipynb

@@ -30,25 +30,6 @@
     "evaluator = load_evaluator(\"trajectory\")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "b1c64c1a",
-   "metadata": {},
-   "source": [
-    "## Methods\n",
-    "\n",
-    "\n",
-    "The Agent Trajectory Evaluators are used with the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.evaluate_agent_trajectory) (and async [aevaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.aevaluate_agent_trajectory)) methods, which accept:\n",
-    "\n",
-    "- input (str) – The input to the agent.\n",
-    "- prediction (str) – The final predicted response.\n",
-    "- agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory\n",
-    "\n",
-    "They return a dictionary with the following values:\n",
-    "- score: Float from 0 to 1, where 1 would mean \"most effective\" and 0 would mean \"least effective\"\n",
-    "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "e733562c-4c17-4942-9647-acfc5ebfaca2",
@@ -71,13 +52,11 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "import subprocess\n",
-    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.tools import tool\n",
     "from langchain.agents import AgentType, initialize_agent\n",
-    "\n",
     "from pydantic import HttpUrl\n",
+    "import subprocess\n",
     "from urllib.parse import urlparse\n",
     "\n",
     "\n",
@@ -138,11 +117,17 @@
     "tags": []
    },
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Type <class 'langchain.agents.openai_functions_multi_agent.base._FunctionsAgentAction'> not serializable\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
-       "{'score': 1.0,\n",
-       " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the website https://langchain.com.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. It uses the 'ping' tool to measure the latency of the website, which is the correct tool for this task.\\n\\niii. The AI language model uses the tool in a helpful way. It inputs the URL into the 'ping' tool and correctly interprets the output to provide the latency in milliseconds.\\n\\niv. The AI language model does not use too many steps to answer the question. It only uses one step, which is appropriate for this type of question.\\n\\nv. The appropriate tool is used to answer the question. The 'ping' tool is the correct tool to measure website latency.\\n\\nGiven these considerations, the AI language model's performance is excellent. It uses the correct tool, interprets the output correctly, and provides a helpful and direct answer to the user's question.\"}"
+       "1.0"
       ]
      },
      "execution_count": 3,
@@ -156,7 +141,7 @@
     "    input=result[\"input\"],\n",
     "    agent_trajectory=result[\"intermediate_steps\"],\n",
     ")\n",
-    "evaluation_result"
+    "evaluation_result[\"score\"]"
    ]
   },
   {
@@ -208,8 +193,7 @@
     {
      "data": {
       "text/plain": [
-       "{'score': 1.0,\n",
-       " 'reasoning': \"Here is my detailed evaluation of the AI's response:\\n\\ni. The final answer is helpful, as it directly provides the latency measurement for the requested website.\\n\\nii. The sequence of using the ping tool to measure latency is logical for this question.\\n\\niii. The ping tool is used in a helpful way, with the website URL provided as input and the output latency measurement extracted.\\n\\niv. Only one step is used, which is appropriate for simply measuring latency. More steps are not needed.\\n\\nv. The ping tool is an appropriate choice to measure latency. \\n\\nIn summary, the AI uses an optimal single step approach with the right tool and extracts the needed output. The final answer directly answers the question in a helpful way.\\n\\nOverall\"}"
+       "1.0"
       ]
      },
      "execution_count": 6,
@@ -223,7 +207,7 @@
     "    input=result[\"input\"],\n",
     "    agent_trajectory=result[\"intermediate_steps\"],\n",
     ")\n",
-    "evaluation_result"
+    "evaluation_result[\"score\"]"
    ]
   },
   {
@@ -261,8 +245,7 @@
     {
      "data": {
       "text/plain": [
-       "{'score': 1.0,\n",
-       " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the specified website.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. In this case, only one tool was needed to answer the question, and the model chose the correct one.\\n\\niii. The AI language model uses the tool in a helpful way. The 'ping' tool was used to determine the latency of the website, which was the information the user was seeking.\\n\\niv. The AI language model does not use too many steps to answer the question. Only one step was needed and used.\\n\\nv. The appropriate tool was used to answer the question. The 'ping' tool is designed to measure latency, which was the information the user was seeking.\\n\\nGiven these considerations, the AI language model's performance in answering this question is excellent.\"}"
+       "1.0"
       ]
      },
      "execution_count": 8,
@@ -276,7 +259,7 @@
     "    input=result[\"input\"],\n",
     "    agent_trajectory=result[\"intermediate_steps\"],\n",
     ")\n",
-    "evaluation_result"
+    "evaluation_result[\"score\"]"
    ]
   }
  ],

docs/extras/guides/expression_language/interface.ipynb

@@ -1,269 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "9a9acd2e",
-   "metadata": {},
-   "source": [
-    "# Interface\n",
-    "\n",
-    "In an effort to make it as easy as possible to create custom chains, we've implemented a [\"Runnable\"](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.Runnable.html#langchain.schema.runnable.Runnable) protocol that most components implement. This is a standard interface with a few different methods, which makes it easy to define custom chains as well as making it possible to invoke them in a standard way. The standard interface exposed includes:\n",
-    "\n",
-    "- `stream`: stream back chunks of the response\n",
-    "- `invoke`: call the chain on an input\n",
-    "- `batch`: call the chain on a list of inputs\n",
-    "\n",
-    "These also have corresponding async methods:\n",
-    "\n",
-    "- `astream`: stream back chunks of the response async\n",
-    "- `ainvoke`: call the chain on an input async\n",
-    "- `abatch`: call the chain on a list of inputs async\n",
-    "\n",
-    "The type of the input varies by component. For a prompt it is a dictionary, for a retriever it is a single string, for a model either a single string, a list of chat messages, or a PromptValue.\n",
-    "\n",
-    "The output type also varies by component. For an LLM it is a string, for a ChatModel it's a ChatMessage, for a prompt it's a PromptValue, for a retriever it's a list of documents.\n",
-    "\n",
-    "Let's take a look at these methods! To do so, we'll create a super simple PromptTemplate + ChatModel chain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "466b65b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "3c634ef0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = ChatOpenAI()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "d1850a1f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "56d0669f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "daf2b2b2",
-   "metadata": {},
-   "source": [
-    "## Stream"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "bea9639d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Sure, here's a bear-themed joke for you:\n",
-      "\n",
-      "Why don't bears wear shoes?\n",
-      "\n",
-      "Because they have bear feet!"
-     ]
-    }
-   ],
-   "source": [
-    "for s in chain.stream({\"topic\": \"bears\"}):\n",
-    "    print(s.content, end=\"\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cbf1c782",
-   "metadata": {},
-   "source": [
-    "## Invoke"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "470e483f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "88f0c279",
-   "metadata": {},
-   "source": [
-    "## Batch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "9685de67",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears ever wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False),\n",
-       " AIMessage(content=\"Why don't cats play poker in the wild?\\n\\nToo many cheetahs!\", additional_kwargs={}, example=False)]"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b960cbfe",
-   "metadata": {},
-   "source": [
-    "## Async Stream"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "ea35eee4",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Why don't bears wear shoes?\n",
-      "\n",
-      "Because they have bear feet!"
-     ]
-    }
-   ],
-   "source": [
-    "async for s in chain.astream({\"topic\": \"bears\"}):\n",
-    "    print(s.content, end=\"\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "04cb3324",
-   "metadata": {},
-   "source": [
-    "## Async Invoke"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "ef8c9b20",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Sure, here you go:\\n\\nWhy don't bears wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "await chain.ainvoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3da288d5",
-   "metadata": {},
-   "source": [
-    "## Async Batch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "eba2a103",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False)]"
-      ]
-     },
-     "execution_count": 18,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "await chain.abatch([{\"topic\": \"bears\"}])"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/guides/model_laboratory.ipynb

@@ -5,7 +5,7 @@
    "id": "920a3c1a",
    "metadata": {},
    "source": [
-    "# Model comparison\n",
+    "# Model Comparison\n",
     "\n",
     "Constructing your language model application will likely involved choosing between many different options of prompts, models, and even chains to use. When doing so, you will want to compare these different options on different inputs in an easy, flexible, and intuitive way. \n",
     "\n",
@@ -254,7 +254,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.10.9"
   }
  },
  "nbformat": 4,

docs/extras/integrations/callbacks/argilla.ipynb

@@ -14,7 +14,7 @@
     "> using both human and machine feedback. We provide support for each step in the MLOps cycle, \n",
     "> from data labeling to model monitoring.\n",
     "\n",
-    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/integrations/callbacks/argilla.html\">\n",
+    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/modules/callbacks/integrations/argilla.html\">\n",
     "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
     "</a>"
    ]

docs/extras/integrations/chat/anthropic_functions.ipynb

@@ -1,287 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "5125a1e3",
-   "metadata": {},
-   "source": [
-    "# Anthropic Functions\n",
-    "\n",
-    "This notebook shows how to use an experimental wrapper around Anthropic that gives it the same API as OpenAI Functions."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "378be79b",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/harrisonchase/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.14) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.\n",
-      "  warnings.warn(\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain_experimental.llms.anthropic_functions import AnthropicFunctions"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "65499965",
-   "metadata": {},
-   "source": [
-    "## Initialize Model\n",
-    "\n",
-    "You can initialize this wrapper the same way you'd initialize ChatAnthropic"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "e1d535f6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = AnthropicFunctions(model='claude-2')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "fcc9eaf4",
-   "metadata": {},
-   "source": [
-    "## Passing in functions\n",
-    "\n",
-    "You can now pass in functions in a similar way"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "0779c320",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "functions=[\n",
-    "    {\n",
-    "      \"name\": \"get_current_weather\",\n",
-    "      \"description\": \"Get the current weather in a given location\",\n",
-    "      \"parameters\": {\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\n",
-    "          \"location\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"description\": \"The city and state, e.g. San Francisco, CA\"\n",
-    "          },\n",
-    "          \"unit\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"enum\": [\"celsius\", \"fahrenheit\"]\n",
-    "          }\n",
-    "        },\n",
-    "        \"required\": [\"location\"]\n",
-    "      }\n",
-    "    }\n",
-    "  ]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "ad75a933",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema import HumanMessage"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "fc703085",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "response = model.predict_messages(\n",
-    "    [HumanMessage(content=\"whats the weater in boston?\")], \n",
-    "    functions=functions\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "04d7936a",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' ', additional_kwargs={'function_call': {'name': 'get_current_weather', 'arguments': '{\"location\": \"Boston, MA\", \"unit\": \"fahrenheit\"}'}}, example=False)"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "response"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0072fdba",
-   "metadata": {},
-   "source": [
-    "## Using for extraction\n",
-    "\n",
-    "You can now use this for extraction."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "7af5c567",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import create_extraction_chain\n",
-    "schema = {\n",
-    "    \"properties\": {\n",
-    "        \"name\": {\"type\": \"string\"},\n",
-    "        \"height\": {\"type\": \"integer\"},\n",
-    "        \"hair_color\": {\"type\": \"string\"},\n",
-    "    },\n",
-    "    \"required\": [\"name\", \"height\"],\n",
-    "}\n",
-    "inp = \"\"\"\n",
-    "Alex is 5 feet tall. Claudia is 1 feet taller Alex and jumps higher than him. Claudia is a brunette and Alex is blonde.\n",
-    "        \"\"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "bd01082a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = create_extraction_chain(schema, model)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "b5a23e9f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[{'name': 'Alex', 'height': '5', 'hair_color': 'blonde'},\n",
-       " {'name': 'Claudia', 'height': '6', 'hair_color': 'brunette'}]"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.run(inp)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "90ec959e",
-   "metadata": {},
-   "source": [
-    "## Using for tagging\n",
-    "\n",
-    "You can now use this for tagging"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "03c1eb0d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import create_tagging_chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "581c0ece",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "schema = {\n",
-    "    \"properties\": {\n",
-    "        \"sentiment\": {\"type\": \"string\"},\n",
-    "        \"aggressiveness\": {\"type\": \"integer\"},\n",
-    "        \"language\": {\"type\": \"string\"},\n",
-    "    }\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "d9a8570e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = create_tagging_chain(schema, model)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "cf37d679",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'sentiment': 'positive', 'aggressiveness': '0', 'language': 'english'}"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.run(\"this is really cool\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/document_loaders/Etherscan.ipynb

@@ -13,12 +13,12 @@
     "You will need a Etherscan api key to proceed. The free api key has 5 calls per seconds quota.\n",
     "\n",
     "The loader supports the following six functinalities:\n",
-    "* Retrieve normal transactions under specific account on Ethereum Mainet\n",
-    "* Retrieve internal transactions under specific account on Ethereum Mainet\n",
-    "* Retrieve erc20 transactions under specific account on Ethereum Mainet\n",
-    "* Retrieve erc721 transactions under specific account on Ethereum Mainet\n",
-    "* Retrieve erc1155 transactions under specific account on Ethereum Mainet\n",
-    "* Retrieve ethereum balance in wei under specific account on Ethereum Mainet\n",
+    "* Retrieve normal transactions under specifc account on Ethereum Mainet\n",
+    "* Retrieve internal transactions under specifc account on Ethereum Mainet\n",
+    "* Retrieve erc20 transactions under specifc account on Ethereum Mainet\n",
+    "* Retrieve erc721 transactions under specifc account on Ethereum Mainet\n",
+    "* Retrieve erc1155 transactions under specifc account on Ethereum Mainet\n",
+    "* Retrieve ethereum balance in wei under specifc account on Ethereum Mainet\n",
     "\n",
     "\n",
     "If the account does not have corresponding transactions, the loader will a list with one document. The content of document is ''.\n",

docs/extras/integrations/document_loaders/chatgpt_loader.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# ChatGPT Data\n",
+    "### ChatGPT Data\n",
     "\n",
     ">[ChatGPT](https://chat.openai.com) is an artificial intelligence (AI) chatbot developed by OpenAI.\n",
     "\n",

docs/extras/integrations/document_loaders/example_data/notebook.md

@@ -2,7 +2,7 @@
 
 This notebook covers how to load data from an .ipynb notebook into a format suitable by LangChain.
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 
 ```python

docs/extras/integrations/document_loaders/geopandas.ipynb

@@ -46,7 +46,7 @@
    "id": "04981332",
    "metadata": {},
    "source": [
-    "Create a GeoPandas dataframe from [`Open City Data`](https://python.langchain.com/docs/integrations/document_loaders/open_city_data) as an example input."
+    "Create a GeoPandas dataframe from [`Open City Data`](https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/open_city_data) as an example input."
    ]
   },
   {

docs/extras/integrations/document_loaders/mediawikidump.ipynb

@@ -57,13 +57,7 @@
     }
    ],
    "source": [
-    "loader = MWDumpLoader(\n",
-    "    file_path = \"example_data/testmw_pages_current.xml\", \n",
-    "    encoding=\"utf8\",\n",
-    "    #namespaces = [0,2,3] Optional list to load only specific namespaces. Loads all namespaces by default.\n",
-    "    skip_redirects = True, #will skip over pages that just redirect to other pages (or not if False)\n",
-    "    stop_on_error = False #will skip over pages that cause parsing errors (or not if False)\n",
-    "     )\n",
+    "loader = MWDumpLoader(\"example_data/testmw_pages_current.xml\", encoding=\"utf8\")\n",
     "documents = loader.load()\n",
     "print(f\"You have {len(documents)} document(s) in your data \")"
    ]

docs/extras/integrations/document_loaders/tomarkdown.ipynb

@@ -113,7 +113,7 @@
       "\n",
       "The modules are (from least to most complex):\n",
       "\n",
-      "- [Models](https://python.langchain.com/docs/modules/model_io/models/): Supported model types and integrations.\n",
+      "- [Models](https://python.langchain.com/en/latest/modules/models.html): Supported model types and integrations.\n",
       "\n",
       "- [Prompts](https://python.langchain.com/en/latest/modules/prompts.html): Prompt management, optimization, and serialization.\n",
       "\n",

docs/extras/integrations/document_loaders/trello.ipynb

@@ -44,7 +44,7 @@
    },
    "outputs": [],
    "source": [
-    "#!pip install py-trello beautifulsoup4 lxml"
+    "#!pip install py-trello beautifulsoup4"
    ]
   },
   {

docs/extras/integrations/llms/jsonformer_experimental.ipynb

@@ -216,7 +216,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain_experimental.llms import JsonFormer\n",
+    "from langchain.experimental.llms import JsonFormer\n",
     "\n",
     "json_former = JsonFormer(json_schema=decoder_schema, pipeline=hf_model)"
    ]

docs/extras/integrations/llms/minimax.ipynb

@@ -1,176 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Minimax\n",
-    "\n",
-    "[Minimax](https://api.minimax.chat) is a Chinese startup that provides natural language processing models for companies and individuals.\n",
-    "\n",
-    "This example demonstrates using Langchain to interact with Minimax."
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Setup\n",
-    "\n",
-    "To run this notebook, you'll need a [Minimax account](https://api.minimax.chat), an [API key](https://api.minimax.chat/user-center/basic-information/interface-key), and a [Group ID](https://api.minimax.chat/user-center/basic-information)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Single model call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 33,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.llms import Minimax"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 34,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Load the model\n",
-    "minimax = Minimax(minimax_api_key=\"YOUR_API_KEY\", minimax_group_id=\"YOUR_GROUP_ID\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "# Prompt the model\n",
-    "minimax(\"What is the difference between panda and bear?\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Chained model calls"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "# get api_key and group_id: https://api.minimax.chat/user-center/basic-information\n",
-    "# We need `MINIMAX_API_KEY` and `MINIMAX_GROUP_ID`\n",
-    "\n",
-    "import os\n",
-    "\n",
-    "os.environ[\"MINIMAX_API_KEY\"] = \"YOUR_API_KEY\"\n",
-    "os.environ[\"MINIMAX_GROUP_ID\"] = \"YOUR_GROUP_ID\""
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "from langchain.llms import Minimax\n",
-    "from langchain import PromptTemplate, LLMChain"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Question: {question}\n",
-    "\n",
-    "Answer: Let's think step by step.\"\"\"\n",
-    "\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "llm = Minimax()"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "question = \"What NBA team won the Championship in the year Jay Zhou was born?\"\n",
-    "\n",
-    "llm_chain.run(question)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.4"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/llms/octoai.ipynb

@@ -5,7 +5,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# OctoAI Compute Service\n",
+    "## OctoAI Compute Service\n",
     "This example goes over how to use LangChain to interact with `OctoAI` [LLM endpoints](https://octoai.cloud/templates)\n",
     "## Environment setup\n",
     "\n",

docs/extras/integrations/llms/petals_example.ipynb

@@ -16,9 +16,7 @@
    "metadata": {},
    "source": [
     "## Install petals\n",
-    "The `petals` package is required to use the Petals API. Install `petals` using `pip3 install petals`.\n",
-    "\n",
-    "For Apple Silicon(M1/M2) users please follow this guide [https://github.com/bigscience-workshop/petals/issues/147#issuecomment-1365379642](https://github.com/bigscience-workshop/petals/issues/147#issuecomment-1365379642) to install petals "
+    "The `petals` package is required to use the Petals API. Install `petals` using `pip3 install petals`."
    ]
   },
   {
@@ -64,7 +62,7 @@
    },
    "outputs": [
     {
-     "name": "stdout",
+     "name": "stdin",
      "output_type": "stream",
      "text": [
       " ········\n"

docs/extras/integrations/llms/rellm_experimental.ipynb

@@ -162,7 +162,7 @@
     }
    ],
    "source": [
-    "from langchain_experimental.llms import RELLM\n",
+    "from langchain.experimental.llms import RELLM\n",
     "\n",
     "model = RELLM(pipeline=hf_model, regex=pattern, max_new_tokens=200)\n",
     "\n",