env vars

.devcontainer/README.md

@@ -5,21 +5,17 @@ This project includes a [dev container](https://containers.dev/), which lets you
 You can use the dev container configuration in this folder to build and run the app without needing to install any of its tools locally! You can use it in [GitHub Codespaces](https://github.com/features/codespaces) or the [VS Code Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
 
 ## GitHub Codespaces
-[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/langchain-ai/langchain)
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hwchase17/langchain)
 
 You may use the button above, or follow these steps to open this repo in a Codespace:
-1. Click the **Code** drop-down menu at the top of https://github.com/langchain-ai/langchain.
+1. Click the **Code** drop-down menu at the top of https://github.com/hwchase17/langchain.
 1. Click on the **Codespaces** tab.
 1. Click **Create codespace on master** .
 
 For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
   
 ## VS Code Dev Containers
-[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/langchain-ai/langchain)
-
-Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name: 
-https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/<yourusername>/<yourclonedreponame>
-
+[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 
 If you already have VS Code and Docker installed, you can use the button above to get started. This will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
 
@@ -29,7 +25,7 @@ You can also follow these steps to open this repo in a container using the VS Co
 
 2. Open a locally cloned copy of the code:
 
-   - Fork and Clone this repository to your local filesystem.
+   - Clone this repository to your local filesystem.
    - Press <kbd>F1</kbd> and select the **Dev Containers: Open Folder in Container...** command.
    - Select the cloned copy of this folder, wait for the container to start, and try things out!
 

.github/CONTRIBUTING.md

@@ -9,19 +9,19 @@ to contributions, whether they be in the form of new features, improved infra, b
 ### 👩‍💻 Contributing Code
 
 To contribute to this project, please follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
-Please do not try to push directly to this repo unless you are a maintainer.
+Please do not try to push directly to this repo unless you are maintainer.
 
 Please follow the checked-in pull request template when opening pull requests. Note related issues and tag relevant
 maintainers.
 
-Pull requests cannot land without passing the formatting, linting and testing checks first. See [Testing](#testing) and
-[Formatting and Linting](#formatting-and-linting) for how to run these checks locally.
+Pull requests cannot land without passing the formatting, linting and testing checks first. See
+[Common Tasks](#-common-tasks) for how to run these checks locally.
 
 It's essential that we maintain great documentation and testing. If you:
 - Fix a bug
   - Add a relevant unit or integration test when possible. These live in `tests/unit_tests` and `tests/integration_tests`.
 - Make an improvement
-  - Update any affected example notebooks and documentation. These live in `docs`.
+  - Update any affected example notebooks and documentation. These lives in `docs`.
   - Update unit and integration tests when relevant.
 - Add a feature
   - Add a demo notebook in `docs/modules`.
@@ -32,8 +32,8 @@ best way to get our attention.
 
 ### 🚩GitHub Issues
 
-Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date
-with bugs, improvements, and feature requests.
+Our [issues](https://github.com/hwchase17/langchain/issues) page is kept up to date
+with bugs, improvements, and feature requests. 
 
 There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help
 organize issues.
@@ -43,8 +43,8 @@ If you start working on an issue, please assign it to yourself.
 If you are adding an issue, please try to keep it focused on a single, modular bug/improvement/feature.
 If two issues are related, or blocking, please link them rather than combining them.
 
-We will try to keep these issues as up-to-date as possible, though
-with the rapid rate of development in this field some may get out of date.
+We will try to keep these issues as up to date as possible, though
+with the rapid rate of develop in this field some may get out of date.
 If you notice this happening, please let us know.
 
 ### 🙋Getting Help
@@ -59,85 +59,43 @@ we do not want these to get in the way of getting good code into the codebase.
 
 ## 🚀 Quick Start
 
-This quick start describes running the repository locally.
-For a [development container](https://containers.dev/), see the [.devcontainer folder](https://github.com/langchain-ai/langchain/tree/master/.devcontainer).
+> **Note:** You can run this repository locally (which is described below) or in a [development container](https://containers.dev/) (which is described in the [.devcontainer folder](https://github.com/hwchase17/langchain/tree/master/.devcontainer)).
 
-### Dependency Management: Poetry and other env/dependency managers
+This project uses [Poetry](https://python-poetry.org/) as a dependency manager. Check out Poetry's [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.
 
-This project uses [Poetry](https://python-poetry.org/) v1.6.1+ as a dependency manager.
-
-❗Note: *Before installing Poetry*, if you use `Conda`, create and activate a new Conda env (e.g. `conda create -n langchain python=3.9`)
-
-Install Poetry: **[documentation on how to install it](https://python-poetry.org/docs/#installation)**.
-
-❗Note: If you use `Conda` or `Pyenv` as your environment/package manager, after installing Poetry,
-tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`)
-
-### Core vs. Experimental
+❗Note: If you use `Conda` or `Pyenv` as your environment / package manager, avoid dependency conflicts by doing the following first:
+1. *Before installing Poetry*, create and activate a new Conda env (e.g. `conda create -n langchain python=3.9`)
+2. Install Poetry (see above)
+3. Tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`)
+4. Continue with the following steps.
 
 There are two separate projects in this repository:
 - `langchain`: core langchain code, abstractions, and use cases
-- `langchain.experimental`: see the [Experimental README](../libs/experimental/README.md) for more information.
+- `langchain.experimental`: more experimental code
 
-Each of these has their own development environment. Docs are run from the top-level makefile, but development
-is split across separate test & release flows.
+Each of these has their OWN development environment. 
+In order to run any of the commands below, please move into their respective directories.
+For example, to contribute to `langchain` run `cd libs/langchain` before getting started with the below.
 
-For this quickstart, start with langchain core:
+To install requirements:
 
 ```bash
-cd libs/langchain
+poetry install -E all
 ```
 
-### Local Development Dependencies
+This will install all requirements for running the package, examples, linting, formatting, tests, and coverage. Note the `-E all` flag will install all optional dependencies necessary for integration testing.
 
-Install langchain development requirements (for running langchain, running examples, linting, formatting, tests, and coverage):
+❗Note: If you're running Poetry 1.4.1 and receive a `WheelFileValidationError` for `debugpy` during installation, you can try either downgrading to Poetry 1.4.0 or disabling "modern installation" (`poetry config installer.modern-installation false`) and re-install requirements. See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details.
 
-```bash
-poetry install --with test
-```
+Now, you should be able to run the common tasks in the following section. To double check, run `make test`, all tests should pass. If they don't you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
 
-Then verify dependency installation:
+## ✅ Common Tasks
 
-```bash
-make test
-```
+Type `make` for a list of common tasks.
 
-If the tests don't pass, you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
+### Code Formatting
 
-If during installation you receive a `WheelFileValidationError` for `debugpy`, please make sure you are running
-Poetry v1.6.1+. This bug was present in older versions of Poetry (e.g. 1.4.1) and has been resolved in newer releases.
-If you are still seeing this bug on v1.6.1, you may also try disabling "modern installation"
-(`poetry config installer.modern-installation false`) and re-installing requirements.
-See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details.
-
-### Testing
-
-_some test dependencies are optional; see section about optional dependencies_.
-
-Unit tests cover modular logic that does not require calls to outside APIs.
-If you add new logic, please add a unit test.
-
-To run unit tests:
-
-```bash
-make test
-```
-
-To run unit tests in Docker:
-
-```bash
-make docker_tests
-```
-
-There are also [integration tests and code-coverage](../libs/langchain/tests/README.md) available.
-
-### Formatting and Linting
-
-Run these locally before submitting a PR; the CI system will check also.
-
-#### Code Formatting
-
-Formatting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/) and [ruff](https://docs.astral.sh/ruff/rules/).
+Formatting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/) and [isort](https://pycqa.github.io/isort/).
 
 To run formatting for this project:
 
@@ -153,9 +111,9 @@ make format_diff
 
 This is especially useful when you have made changes to a subset of the project and want to ensure your changes are properly formatted without affecting the rest of the codebase.
 
-#### Linting
+### Linting
 
-Linting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/), [ruff](https://docs.astral.sh/ruff/rules/), and [mypy](http://mypy-lang.org/).
+Linting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](http://mypy-lang.org/).
 
 To run linting for this project:
 
@@ -173,10 +131,10 @@ This can be very helpful when you've made changes to only certain parts of the p
 
 We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
 
-#### Spellcheck
+### Spellcheck
 
 Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell).
-Note that `codespell` finds common typos, so it could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
+Note that `codespell` finds common typos, so could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
 
 To check spelling for this project:
 
@@ -199,17 +157,27 @@ If codespell is incorrectly flagging a word, you can skip spellcheck for that wo
 ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure'
 ```
 
-## Working with Optional Dependencies
+### Coverage
+
+Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle.
+
+To get a report of current coverage, run the following:
+
+```bash
+make coverage
+```
+
+### Working with Optional Dependencies
 
 Langchain relies heavily on optional dependencies to keep the Langchain package lightweight.
 
 If you're adding a new dependency to Langchain, assume that it will be an optional dependency, and
 that most users won't have it installed.
 
-Users who do not have the dependency installed should be able to **import** your code without
-any side effects (no warnings, no errors, no exceptions).
+Users that do not have the dependency installed should be able to **import** your code without
+any side effects (no warnings, no errors, no exceptions). 
 
-To introduce the dependency to the pyproject.toml file correctly, please do the following:
+To introduce the dependency to the pyproject.toml file correctly, please do the following: 
 
 1. Add the dependency to the main group as an optional dependency
   ```bash
@@ -220,13 +188,57 @@ To introduce the dependency to the pyproject.toml file correctly, please do the
   ```bash
   poetry lock --no-update
   ```
-4. Add a unit test that the very least attempts to import the new code. Ideally, the unit
+4. Add a unit test that the very least attempts to import the new code. Ideally the unit
 test makes use of lightweight fixtures to test the logic of the code.
 5. Please use the `@pytest.mark.requires(package_name)` decorator for any tests that require the dependency.
 
-## Adding a Jupyter Notebook
+### Testing
 
-If you are adding a Jupyter Notebook example, you'll want to install the optional `dev` dependencies.
+See section about optional dependencies.
+
+#### Unit Tests
+
+Unit tests cover modular logic that does not require calls to outside APIs.
+
+To run unit tests:
+
+```bash
+make test
+```
+
+To run unit tests in Docker:
+
+```bash
+make docker_tests
+```
+
+If you add new logic, please add a unit test.
+
+
+
+#### Integration Tests
+
+Integration tests cover logic that requires making calls to outside APIs (often integration with other services).
+
+**warning** Almost no tests should be integration tests. 
+
+  Tests that require making network connections make it difficult for other
+  developers to test the code.
+
+  Instead favor relying on `responses` library and/or mock.patch to mock
+  requests using small fixtures.
+
+To run integration tests:
+
+```bash
+make integration_tests
+```
+
+If you add support for a new external API, please add a new integration test.
+
+### Adding a Jupyter Notebook
+
+If you are adding a Jupyter notebook example, you'll want to install the optional `dev` dependencies.
 
 To install dev dependencies:
 
@@ -247,12 +259,6 @@ When you run `poetry install`, the `langchain` package is installed as editable
 While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing.
 This covers how to get started contributing to documentation.
 
-From the top-level of this repo, install documentation dependencies:
-
-```bash
-poetry install
-```
-
 ### Contribute Documentation
 
 The docs directory contains Documentation and API Reference.
@@ -289,13 +295,6 @@ make docs_linkcheck
 make api_docs_linkcheck
 ```
 
-### Verify Documentation changes
-
-After pushing documentation changes to the repository, you can preview and verify that the changes are 
-what you wanted by clicking the `View deployment` or `Visit Preview` buttons on the pull request `Conversation` page. 
-This will take you to a preview of the documentation changes.
-This preview is created by [Vercel](https://vercel.com/docs/getting-started-with-vercel).
-
 ## 🏭 Release Process
 
 As of now, LangChain has an ad hoc release process: releases are cut with high frequency by
@@ -308,3 +307,4 @@ even patch releases may contain [non-backwards-compatible changes](https://semve
 
 If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)!
 If you have a Twitter account you would like us to mention, please let us know in the PR or in another manner.
+

.github/ISSUE_TEMPLATE/bug-report.yml

@@ -1,5 +1,5 @@
 name: "\U0001F41B Bug Report"
-description: Submit a bug report to help us improve LangChain. To report a security issue, please instead use the security option below.
+description: Submit a bug report to help us improve LangChain
 labels: ["02 Bug Report"]
 body:
   - type: markdown

.github/ISSUE_TEMPLATE/feature-request.yml

@@ -27,4 +27,4 @@ body:
     attributes:
       label: Your contribution
       description: |
-        Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md)
+        Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md)

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,20 +1,28 @@
 <!-- Thank you for contributing to LangChain!
 
-Replace this entire comment with:
-  - **Description:** a description of the change, 
-  - **Issue:** the issue # it fixes (if applicable),
-  - **Dependencies:** any dependencies required for this change,
-  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
-  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!
+Replace this comment with:
+  - Description: a description of the change, 
+  - Issue: the issue # it fixes (if applicable),
+  - Dependencies: any dependencies required for this change,
+  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
+  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
 
-Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
-
-See contribution guidelines for more information on how to write/run tests, lint, etc: 
-https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
+Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
 
 If you're adding a new integration, please include:
   1. a test for the integration, preferably unit tests that do not rely on network access,
-  2. an example notebook showing its use. It lives in `docs/extras` directory.
+  2. an example notebook showing its use.
 
-If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
+Maintainer responsibilities:
+  - General / Misc / if you don't know who to tag: @baskaryan
+  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
+  - Models / Prompts: @hwchase17, @baskaryan
+  - Memory: @hwchase17
+  - Agents / Tools / Toolkits: @hinthornw
+  - Tracing / Callbacks: @agola11
+  - Async: @agola11
+
+If no one reviews your PR within a few days, feel free to @-mention the same people again.
+
+See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
  -->

.github/actions/poetry_setup/action.yml

@@ -15,77 +15,64 @@ inputs:
     description: Poetry version
     required: true
 
+  install-command:
+    description: Command run for installing dependencies
+    required: false
+    default: poetry install
+
   cache-key:
     description: Cache key to use for manual handling of caching
     required: true
 
   working-directory:
-    description: Directory whose poetry.lock file should be cached
-    required: true
+    description: Directory to run install-command in
+    required: false
+    default: ""
 
 runs:
   using: composite
   steps:
     - uses: actions/setup-python@v4
-      name: Setup python ${{ inputs.python-version }}
+      name: Setup python $${ inputs.python-version }}
       with:
         python-version: ${{ inputs.python-version }}
 
     - uses: actions/cache@v3
-      id: cache-bin-poetry
-      name: Cache Poetry binary - Python ${{ inputs.python-version }}
+      id: cache-pip
+      name: Cache Pip ${{ inputs.python-version }}
       env:
-        SEGMENT_DOWNLOAD_TIMEOUT_MIN: "1"
-      with:
-        path: |
-          /opt/pipx/venvs/poetry
-        # This step caches the poetry installation, so make sure it's keyed on the poetry version as well.
-        key: bin-poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-${{ inputs.poetry-version }}
-
-    - name: Refresh shell hashtable and fixup softlinks
-      if: steps.cache-bin-poetry.outputs.cache-hit == 'true'
-      shell: bash
-      env:
-        POETRY_VERSION: ${{ inputs.poetry-version }}
-        PYTHON_VERSION: ${{ inputs.python-version }}
-      run: |
-        set -eux
-
-        # Refresh the shell hashtable, to ensure correct `which` output.
-        hash -r
-
-        # `actions/cache@v3` doesn't always seem able to correctly unpack softlinks.
-        # Delete and recreate the softlinks pipx expects to have.
-        rm /opt/pipx/venvs/poetry/bin/python
-        cd /opt/pipx/venvs/poetry/bin
-        ln -s "$(which "python$PYTHON_VERSION")" python
-        chmod +x python
-        cd /opt/pipx_bin/
-        ln -s /opt/pipx/venvs/poetry/bin/poetry poetry
-        chmod +x poetry
-
-        # Ensure everything got set up correctly.
-        /opt/pipx/venvs/poetry/bin/python --version
-        /opt/pipx_bin/poetry --version
-
-    - name: Install poetry
-      if: steps.cache-bin-poetry.outputs.cache-hit != 'true'
-      shell: bash
-      env:
-        POETRY_VERSION: ${{ inputs.poetry-version }}
-        PYTHON_VERSION: ${{ inputs.python-version }}
-      run: pipx install "poetry==$POETRY_VERSION" --python "python$PYTHON_VERSION" --verbose
-
-    - name: Restore pip and poetry cached dependencies
-      uses: actions/cache@v3
-      env:
-        SEGMENT_DOWNLOAD_TIMEOUT_MIN: "4"
-        WORKDIR: ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}
+        SEGMENT_DOWNLOAD_TIMEOUT_MIN: "15"
       with:
         path: |
           ~/.cache/pip
+        key: pip-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}
+
+    - run: pipx install poetry==${{ inputs.poetry-version }} --python python${{ inputs.python-version }}
+      shell: bash
+
+    - name: Check Poetry File
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        poetry check
+
+    - name: Check lock file
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        poetry lock --check
+
+    - uses: actions/cache@v3
+      id: cache-poetry
+      env:
+        SEGMENT_DOWNLOAD_TIMEOUT_MIN: "15"
+      with:
+        path: |
           ~/.cache/pypoetry/virtualenvs
           ~/.cache/pypoetry/cache
           ~/.cache/pypoetry/artifacts
-          ${{ env.WORKDIR }}/.venv
-        key: py-deps-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles(format('{0}/**/poetry.lock', env.WORKDIR)) }}
+        key: poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles('poetry.lock') }}
+
+    - run: ${{ inputs.install-command }}
+      working-directory: ${{ inputs.working-directory }}
+      shell: bash

.github/tools/git-restore-mtime

@@ -1,606 +0,0 @@
-#!/usr/bin/env python3
-#
-# git-restore-mtime - Change mtime of files based on commit date of last change
-#
-#    Copyright (C) 2012 Rodrigo Silva (MestreLion) <linux@rodrigosilva.com>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License as published by
-#    the Free Software Foundation, either version 3 of the License, or
-#    (at your option) any later version.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program. See <http://www.gnu.org/licenses/gpl.html>
-#
-# Source: https://github.com/MestreLion/git-tools
-# Version: July 13, 2023 (commit hash 5f832e72453e035fccae9d63a5056918d64476a2)
-"""
-Change the modification time (mtime) of files in work tree, based on the
-date of the most recent commit that modified the file, including renames.
-
-Ignores untracked files and uncommitted deletions, additions and renames, and
-by default modifications too.
----
-Useful prior to generating release tarballs, so each file is archived with a
-date that is similar to the date when the file was actually last modified,
-assuming the actual modification date and its commit date are close.
-"""
-
-# TODO:
-# - Add -z on git whatchanged/ls-files, so we don't deal with filename decoding
-# - When Python is bumped to 3.7, use text instead of universal_newlines on subprocess
-# - Update "Statistics for some large projects" with modern hardware and repositories.
-# - Create a README.md for git-restore-mtime alone. It deserves extensive documentation
-#   - Move Statistics there
-# - See git-extras as a good example on project structure and documentation
-
-# FIXME:
-# - When current dir is outside the worktree, e.g. using --work-tree, `git ls-files`
-#   assume any relative pathspecs are to worktree root, not the current dir. As such,
-#   relative pathspecs may not work.
-# - Renames are tricky:
-#   - R100 should not change mtime, but original name is not on filelist. Should
-#     track renames until a valid (A, M) mtime found and then set on current name.
-#   - Should set mtime for both current and original directories.
-#   - Check mode changes with unchanged blobs?
-# - Check file (A, D) for the directory mtime is not sufficient:
-#   - Renames also change dir mtime, unless rename was on a parent dir
-#   - If most recent change of all files in a dir was a Modification (M),
-#     dir might not be touched at all.
-#   - Dirs containing only subdirectories but no direct files will also
-#     not be touched. They're files' [grand]parent dir, but never their dirname().
-#   - Some solutions:
-#     - After files done, perform some dir processing for missing dirs, finding latest
-#       file (A, D, R)
-#     - Simple approach: dir mtime is the most recent child (dir or file) mtime
-#     - Use a virtual concept of "created at most at" to fill missing info, bubble up
-#       to parents and grandparents
-#   - When handling [grand]parent dirs, stay inside <pathspec>
-# - Better handling of merge commits. `-m` is plain *wrong*. `-c/--cc` is perfect, but
-#   painfully slow. First pass without merge commits is not accurate. Maybe add a new
-#   `--accurate` mode for `--cc`?
-
-if __name__ != "__main__":
-    raise ImportError("{} should not be used as a module.".format(__name__))
-
-import argparse
-import datetime
-import logging
-import os.path
-import shlex
-import signal
-import subprocess
-import sys
-import time
-
-__version__ = "2022.12+dev"
-
-# Update symlinks only if the platform supports not following them
-UPDATE_SYMLINKS = bool(os.utime in getattr(os, 'supports_follow_symlinks', []))
-
-# Call os.path.normpath() only if not in a POSIX platform (Windows)
-NORMALIZE_PATHS = (os.path.sep != '/')
-
-# How many files to process in each batch when re-trying merge commits
-STEPMISSING = 100
-
-# (Extra) keywords for the os.utime() call performed by touch()
-UTIME_KWS = {} if not UPDATE_SYMLINKS else {'follow_symlinks': False}
-
-
-# Command-line interface ######################################################
-
-def parse_args():
-    parser = argparse.ArgumentParser(
-        description=__doc__.split('\n---')[0])
-
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument('--quiet', '-q', dest='loglevel',
-        action="store_const", const=logging.WARNING, default=logging.INFO,
-        help="Suppress informative messages and summary statistics.")
-    group.add_argument('--verbose', '-v', action="count", help="""
-        Print additional information for each processed file.
-        Specify twice to further increase verbosity.
-        """)
-
-    parser.add_argument('--cwd', '-C', metavar="DIRECTORY", help="""
-        Run as if %(prog)s was started in directory %(metavar)s.
-        This affects how --work-tree, --git-dir and PATHSPEC arguments are handled.
-        See 'man 1 git' or 'git --help' for more information.
-        """)
-
-    parser.add_argument('--git-dir', dest='gitdir', metavar="GITDIR", help="""
-        Path to the git repository, by default auto-discovered by searching
-        the current directory and its parents for a .git/ subdirectory.
-        """)
-
-    parser.add_argument('--work-tree', dest='workdir', metavar="WORKTREE", help="""
-        Path to the work tree root, by default the parent of GITDIR if it's
-        automatically discovered, or the current directory if GITDIR is set.
-        """)
-
-    parser.add_argument('--force', '-f', default=False, action="store_true", help="""
-        Force updating files with uncommitted modifications.
-        Untracked files and uncommitted deletions, renames and additions are
-        always ignored.
-        """)
-
-    parser.add_argument('--merge', '-m', default=False, action="store_true", help="""
-        Include merge commits.
-        Leads to more recent times and more files per commit, thus with the same
-        time, which may or may not be what you want.
-        Including merge commits may lead to fewer commits being evaluated as files
-        are found sooner, which can improve performance, sometimes substantially.
-        But as merge commits are usually huge, processing them may also take longer.
-        By default, merge commits are only used for files missing from regular commits.
-        """)
-
-    parser.add_argument('--first-parent', default=False, action="store_true", help="""
-        Consider only the first parent, the "main branch", when evaluating merge commits.
-        Only effective when merge commits are processed, either when --merge is
-        used or when finding missing files after the first regular log search.
-        See --skip-missing.
-        """)
-
-    parser.add_argument('--skip-missing', '-s', dest="missing", default=True,
-        action="store_false", help="""
-        Do not try to find missing files.
-        If merge commits were not evaluated with --merge and some files were
-        not found in regular commits, by default %(prog)s searches for these
-        files again in the merge commits.
-        This option disables this retry, so files found only in merge commits
-        will not have their timestamp updated.
-        """)
-
-    parser.add_argument('--no-directories', '-D', dest='dirs', default=True,
-        action="store_false", help="""
-        Do not update directory timestamps.
-        By default, use the time of its most recently created, renamed or deleted file.
-        Note that just modifying a file will NOT update its directory time.
-        """)
-
-    parser.add_argument('--test', '-t', default=False, action="store_true",
-        help="Test run: do not actually update any file timestamp.")
-
-    parser.add_argument('--commit-time', '-c', dest='commit_time', default=False,
-        action='store_true', help="Use commit time instead of author time.")
-
-    parser.add_argument('--oldest-time', '-o', dest='reverse_order', default=False,
-        action='store_true', help="""
-        Update times based on the oldest, instead of the most recent commit of a file.
-        This reverses the order in which the git log is processed to emulate a
-        file "creation" date. Note this will be inaccurate for files deleted and
-        re-created at later dates.
-        """)
-
-    parser.add_argument('--skip-older-than', metavar='SECONDS', type=int, help="""
-        Ignore files that are currently older than %(metavar)s.
-        Useful in workflows that assume such files already have a correct timestamp,
-        as it may improve performance by processing fewer files.
-        """)
-
-    parser.add_argument('--skip-older-than-commit', '-N', default=False,
-        action='store_true', help="""
-        Ignore files older than the timestamp it would be updated to.
-        Such files may be considered "original", likely in the author's repository.
-        """)
-
-    parser.add_argument('--unique-times', default=False, action="store_true", help="""
-        Set the microseconds to a unique value per commit.
-        Allows telling apart changes that would otherwise have identical timestamps,
-        as git's time accuracy is in seconds.
-        """)
-
-    parser.add_argument('pathspec', nargs='*', metavar='PATHSPEC', help="""
-        Only modify paths matching %(metavar)s, relative to current directory.
-        By default, update all but untracked files and submodules.
-        """)
-
-    parser.add_argument('--version', '-V', action='version',
-        version='%(prog)s version {version}'.format(version=get_version()))
-
-    args_ = parser.parse_args()
-    if args_.verbose:
-        args_.loglevel = max(logging.TRACE, logging.DEBUG // args_.verbose)
-    args_.debug = args_.loglevel <= logging.DEBUG
-    return args_
-
-
-def get_version(version=__version__):
-    if not version.endswith('+dev'):
-        return version
-    try:
-        cwd = os.path.dirname(os.path.realpath(__file__))
-        return Git(cwd=cwd, errors=False).describe().lstrip('v')
-    except Git.Error:
-        return '-'.join((version, "unknown"))
-
-
-# Helper functions ############################################################
-
-def setup_logging():
-    """Add TRACE logging level and corresponding method, return the root logger"""
-    logging.TRACE = TRACE = logging.DEBUG // 2
-    logging.Logger.trace = lambda _, m, *a, **k: _.log(TRACE, m, *a, **k)
-    return logging.getLogger()
-
-
-def normalize(path):
-    r"""Normalize paths from git, handling non-ASCII characters.
-
-    Git stores paths as UTF-8 normalization form C.
-    If path contains non-ASCII or non-printable characters, git outputs the UTF-8
-    in octal-escaped notation, escaping double-quotes and backslashes, and then
-    double-quoting the whole path.
-    https://git-scm.com/docs/git-config#Documentation/git-config.txt-corequotePath
-
-    This function reverts this encoding, so:
-    normalize(r'"Back\\slash_double\"quote_a\303\247a\303\255"') =>
-        r'Back\slash_double"quote_açaí')
-
-    Paths with invalid UTF-8 encoding, such as single 0x80-0xFF bytes (e.g, from
-    Latin1/Windows-1251 encoding) are decoded using surrogate escape, the same
-    method used by Python for filesystem paths. So 0xE6 ("æ" in Latin1, r'\\346'
-    from Git) is decoded as "\udce6". See https://peps.python.org/pep-0383/ and
-    https://vstinner.github.io/painful-history-python-filesystem-encoding.html
-
-    Also see notes on `windows/non-ascii-paths.txt` about path encodings on
-    non-UTF-8 platforms and filesystems.
-    """
-    if path and path[0] == '"':
-        # Python 2: path = path[1:-1].decode("string-escape")
-        # Python 3: https://stackoverflow.com/a/46650050/624066
-        path = (path[1:-1]                 # Remove enclosing double quotes
-                .encode('latin1')          # Convert to bytes, required by 'unicode-escape'
-                .decode('unicode-escape')  # Perform the actual octal-escaping decode
-                .encode('latin1')          # 1:1 mapping to bytes, UTF-8 encoded
-                .decode('utf8', 'surrogateescape'))  # Decode from UTF-8
-    if NORMALIZE_PATHS:
-        # Make sure the slash matches the OS; for Windows we need a backslash
-        path = os.path.normpath(path)
-    return path
-
-
-def dummy(*_args, **_kwargs):
-    """No-op function used in dry-run tests"""
-
-
-def touch(path, mtime):
-    """The actual mtime update"""
-    os.utime(path, (mtime, mtime), **UTIME_KWS)
-
-
-def touch_ns(path, mtime_ns):
-    """The actual mtime update, using nanoseconds for unique timestamps"""
-    os.utime(path, None, ns=(mtime_ns, mtime_ns), **UTIME_KWS)
-
-
-def isodate(secs: int):
-    # time.localtime() accepts floats, but discards fractional part
-    return time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(secs))
-
-
-def isodate_ns(ns: int):
-    # for integers fromtimestamp() is equivalent and ~16% slower than isodate()
-    return datetime.datetime.fromtimestamp(ns / 1000000000).isoformat(sep=' ')
-
-
-def get_mtime_ns(secs: int, idx: int):
-    # Time resolution for filesystems and functions:
-    # ext-4 and other POSIX filesystems: 1 nanosecond
-    # NTFS (Windows default): 100 nanoseconds
-    # datetime.datetime() (due to 64-bit float epoch): 1 microsecond
-    us = idx % 1000000  # 10**6
-    return 1000 * (1000000 * secs + us)
-
-
-def get_mtime_path(path):
-    return os.path.getmtime(path)
-
-
-# Git class and parse_log(), the heart of the script ##########################
-
-class Git:
-    def __init__(self, workdir=None, gitdir=None, cwd=None, errors=True):
-        self.gitcmd = ['git']
-        self.errors = errors
-        self._proc = None
-        if workdir: self.gitcmd.extend(('--work-tree', workdir))
-        if gitdir:  self.gitcmd.extend(('--git-dir',   gitdir))
-        if cwd:     self.gitcmd.extend(('-C',          cwd))
-        self.workdir, self.gitdir = self._get_repo_dirs()
-
-    def ls_files(self, paths: list = None):
-        return (normalize(_) for _ in self._run('ls-files --full-name', paths))
-
-    def ls_dirty(self, force=False):
-        return (normalize(_[3:].split(' -> ', 1)[-1])
-                for _ in self._run('status --porcelain')
-                if _[:2] != '??' and (not force or (_[0] in ('R', 'A')
-                                                    or _[1] == 'D')))
-
-    def log(self, merge=False, first_parent=False, commit_time=False,
-            reverse_order=False, paths: list = None):
-        cmd = 'whatchanged --pretty={}'.format('%ct' if commit_time else '%at')
-        if merge:         cmd += ' -m'
-        if first_parent:  cmd += ' --first-parent'
-        if reverse_order: cmd += ' --reverse'
-        return self._run(cmd, paths)
-
-    def describe(self):
-        return self._run('describe --tags', check=True)[0]
-
-    def terminate(self):
-        if self._proc is None:
-            return
-        try:
-            self._proc.terminate()
-        except OSError:
-            # Avoid errors on OpenBSD
-            pass
-
-    def _get_repo_dirs(self):
-        return (os.path.normpath(_) for _ in
-            self._run('rev-parse --show-toplevel --absolute-git-dir', check=True))
-
-    def _run(self, cmdstr: str, paths: list = None, output=True, check=False):
-        cmdlist = self.gitcmd + shlex.split(cmdstr)
-        if paths:
-            cmdlist.append('--')
-            cmdlist.extend(paths)
-        popen_args = dict(universal_newlines=True, encoding='utf8')
-        if not self.errors:
-            popen_args['stderr'] = subprocess.DEVNULL
-        log.trace("Executing: %s", ' '.join(cmdlist))
-        if not output:
-            return subprocess.call(cmdlist, **popen_args)
-        if check:
-            try:
-                stdout: str = subprocess.check_output(cmdlist, **popen_args)
-                return stdout.splitlines()
-            except subprocess.CalledProcessError as e:
-                raise self.Error(e.returncode, e.cmd, e.output, e.stderr)
-        self._proc = subprocess.Popen(cmdlist, stdout=subprocess.PIPE, **popen_args)
-        return (_.rstrip() for _ in self._proc.stdout)
-
-    def __del__(self):
-        self.terminate()
-
-    class Error(subprocess.CalledProcessError):
-        """Error from git executable"""
-
-
-def parse_log(filelist, dirlist, stats, git, merge=False, filterlist=None):
-    mtime = 0
-    datestr = isodate(0)
-    for line in git.log(
-            merge,
-            args.first_parent,
-            args.commit_time,
-            args.reverse_order,
-            filterlist
-    ):
-        stats['loglines'] += 1
-
-        # Blank line between Date and list of files
-        if not line:
-            continue
-
-        # Date line
-        if line[0] != ':':  # Faster than `not line.startswith(':')`
-            stats['commits'] += 1
-            mtime = int(line)
-            if args.unique_times:
-                mtime = get_mtime_ns(mtime, stats['commits'])
-            if args.debug:
-                datestr = isodate(mtime)
-            continue
-
-        # File line: three tokens if it describes a renaming, otherwise two
-        tokens = line.split('\t')
-
-        # Possible statuses:
-        # M: Modified (content changed)
-        # A: Added (created)
-        # D: Deleted
-        # T: Type changed: to/from regular file, symlinks, submodules
-        # R099: Renamed (moved), with % of unchanged content. 100 = pure rename
-        # Not possible in log: C=Copied, U=Unmerged, X=Unknown, B=pairing Broken
-        status = tokens[0].split(' ')[-1]
-        file = tokens[-1]
-
-        # Handles non-ASCII chars and OS path separator
-        file = normalize(file)
-
-        def do_file():
-            if args.skip_older_than_commit and get_mtime_path(file) <= mtime:
-                stats['skip'] += 1
-                return
-            if args.debug:
-                log.debug("%d\t%d\t%d\t%s\t%s",
-                          stats['loglines'], stats['commits'], stats['files'],
-                          datestr, file)
-            try:
-                touch(os.path.join(git.workdir, file), mtime)
-                stats['touches'] += 1
-            except Exception as e:
-                log.error("ERROR: %s: %s", e, file)
-                stats['errors'] += 1
-
-        def do_dir():
-            if args.debug:
-                log.debug("%d\t%d\t-\t%s\t%s",
-                          stats['loglines'], stats['commits'],
-                          datestr, "{}/".format(dirname or '.'))
-            try:
-                touch(os.path.join(git.workdir, dirname), mtime)
-                stats['dirtouches'] += 1
-            except Exception as e:
-                log.error("ERROR: %s: %s", e, dirname)
-                stats['direrrors'] += 1
-
-        if file in filelist:
-            stats['files'] -= 1
-            filelist.remove(file)
-            do_file()
-
-        if args.dirs and status in ('A', 'D'):
-            dirname = os.path.dirname(file)
-            if dirname in dirlist:
-                dirlist.remove(dirname)
-                do_dir()
-
-        # All files done?
-        if not stats['files']:
-            git.terminate()
-            return
-
-
-# Main Logic ##################################################################
-
-def main():
-    start = time.time()  # yes, Wall time. CPU time is not realistic for users.
-    stats = {_: 0 for _ in ('loglines', 'commits', 'touches', 'skip', 'errors',
-                            'dirtouches', 'direrrors')}
-
-    logging.basicConfig(level=args.loglevel, format='%(message)s')
-    log.trace("Arguments: %s", args)
-
-    # First things first: Where and Who are we?
-    if args.cwd:
-        log.debug("Changing directory: %s", args.cwd)
-        try:
-            os.chdir(args.cwd)
-        except OSError as e:
-            log.critical(e)
-            return e.errno
-    # Using both os.chdir() and `git -C` is redundant, but might prevent side effects
-    # `git -C` alone could be enough if we make sure that:
-    # - all paths, including args.pathspec, are processed by git: ls-files, rev-parse
-    # - touch() / os.utime() path argument is always prepended with git.workdir
-    try:
-        git = Git(workdir=args.workdir, gitdir=args.gitdir, cwd=args.cwd)
-    except Git.Error as e:
-        # Not in a git repository, and git already informed user on stderr. So we just...
-        return e.returncode
-
-    # Get the files managed by git and build file list to be processed
-    if UPDATE_SYMLINKS and not args.skip_older_than:
-        filelist = set(git.ls_files(args.pathspec))
-    else:
-        filelist = set()
-        for path in git.ls_files(args.pathspec):
-            fullpath = os.path.join(git.workdir, path)
-
-            # Symlink (to file, to dir or broken - git handles the same way)
-            if not UPDATE_SYMLINKS and os.path.islink(fullpath):
-                log.warning("WARNING: Skipping symlink, no OS support for updates: %s",
-                            path)
-                continue
-
-            # skip files which are older than given threshold
-            if (args.skip_older_than
-                    and start - get_mtime_path(fullpath) > args.skip_older_than):
-                continue
-
-            # Always add files relative to worktree root
-            filelist.add(path)
-
-    # If --force, silently ignore uncommitted deletions (not in the filesystem)
-    # and renames / additions (will not be found in log anyway)
-    if args.force:
-        filelist -= set(git.ls_dirty(force=True))
-    # Otherwise, ignore any dirty files
-    else:
-        dirty = set(git.ls_dirty())
-        if dirty:
-            log.warning("WARNING: Modified files in the working directory were ignored."
-                "\nTo include such files, commit your changes or use --force.")
-            filelist -= dirty
-
-    # Build dir list to be processed
-    dirlist = set(os.path.dirname(_) for _ in filelist) if args.dirs else set()
-
-    stats['totalfiles'] = stats['files'] = len(filelist)
-    log.info("{0:,} files to be processed in work dir".format(stats['totalfiles']))
-
-    if not filelist:
-        # Nothing to do. Exit silently and without errors, just like git does
-        return
-
-    # Process the log until all files are 'touched'
-    log.debug("Line #\tLog #\tF.Left\tModification Time\tFile Name")
-    parse_log(filelist, dirlist, stats, git, args.merge, args.pathspec)
-
-    # Missing files
-    if filelist:
-        # Try to find them in merge logs, if not done already
-        # (usually HUGE, thus MUCH slower!)
-        if args.missing and not args.merge:
-            filterlist = list(filelist)
-            missing = len(filterlist)
-            log.info("{0:,} files not found in log, trying merge commits".format(missing))
-            for i in range(0, missing, STEPMISSING):
-                parse_log(filelist, dirlist, stats, git,
-                          merge=True, filterlist=filterlist[i:i + STEPMISSING])
-
-        # Still missing some?
-        for file in filelist:
-            log.warning("WARNING: not found in the log: %s", file)
-
-    # Final statistics
-    # Suggestion: use git-log --before=mtime to brag about skipped log entries
-    def log_info(msg, *a, width=13):
-        ifmt = '{:%d,}'    % (width,)  # not using 'n' for consistency with ffmt
-        ffmt = '{:%d,.2f}' % (width,)
-        # %-formatting lacks a thousand separator, must pre-render with .format()
-        log.info(msg.replace('%d', ifmt).replace('%f', ffmt).format(*a))
-
-    log_info(
-        "Statistics:\n"
-        "%f seconds\n"
-        "%d log lines processed\n"
-        "%d commits evaluated",
-        time.time() - start, stats['loglines'], stats['commits'])
-
-    if args.dirs:
-        if stats['direrrors']: log_info("%d directory update errors", stats['direrrors'])
-        log_info("%d directories updated", stats['dirtouches'])
-
-    if stats['touches'] != stats['totalfiles']:
-                        log_info("%d files",              stats['totalfiles'])
-    if stats['skip']:   log_info("%d files skipped",      stats['skip'])
-    if stats['files']:  log_info("%d files missing",      stats['files'])
-    if stats['errors']: log_info("%d file update errors", stats['errors'])
-
-    log_info("%d files updated", stats['touches'])
-
-    if args.test:
-        log.info("TEST RUN - No files modified!")
-
-
-# Keep only essential, global assignments here. Any other logic must be in main()
-log = setup_logging()
-args = parse_args()
-
-# Set the actual touch() and other functions based on command-line arguments
-if args.unique_times:
-    touch = touch_ns
-    isodate = isodate_ns
-
-# Make sure this is always set last to ensure --test behaves as intended
-if args.test:
-    touch = dummy
-
-# UI done, it's showtime!
-try:
-    sys.exit(main())
-except KeyboardInterrupt:
-    log.info("\nAborting")
-    signal.signal(signal.SIGINT, signal.SIG_DFL)
-    os.kill(os.getpid(), signal.SIGINT)

.github/workflows/_lint.yml

@@ -9,142 +9,38 @@ on:
         description: "From which folder this pipeline executes"
 
 env:
-  POETRY_VERSION: "1.6.1"
-  WORKDIR: ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}
+  POETRY_VERSION: "1.4.2"
 
 jobs:
   build:
+    defaults:
+      run:
+        working-directory: ${{ inputs.working-directory }}
     runs-on: ubuntu-latest
-    env:
-      # This number is set "by eye": we want it to be big enough
-      # so that it's bigger than the number of commits in any reasonable PR,
-      # and also as small as possible since increasing the number makes
-      # the initial `git fetch` slower.
-      FETCH_DEPTH: 50
     strategy:
       matrix:
-        # Only lint on the min and max supported Python versions.
-        # It's extremely unlikely that there's a lint issue on any version in between
-        # that doesn't show up on the min or max versions.
-        #
-        # GitHub rate-limits how many jobs can be running at any one time.
-        # Starting new jobs is also relatively slow,
-        # so linting on fewer versions makes CI faster.
         python-version:
           - "3.8"
+          - "3.9"
+          - "3.10"
           - "3.11"
     steps:
       - uses: actions/checkout@v3
-        with:
-          # Fetch the last FETCH_DEPTH commits, so the mtime-changing script
-          # can accurately set the mtimes of files modified in the last FETCH_DEPTH commits.
-          fetch-depth: ${{ env.FETCH_DEPTH }}
-      - name: Restore workdir file mtimes to last-edited commit date
-        id: restore-mtimes
-        # This is needed to make black caching work.
-        # Black's cache uses file (mtime, size) to check whether a lookup is a cache hit.
-        # Without this command, files in the repo would have the current time as the modified time,
-        # since the previous action step just created them.
-        # This command resets the mtime to the last time the files were modified in git instead,
-        # which is a high-quality and stable representation of the last modification date.
+      - name: Install poetry
         run: |
-          # Important considerations:
-          # - These commands run at base of the repo, since we never `cd` to the `WORKDIR`.
-          # - We only want to alter mtimes for Python files, since that's all black checks.
-          # - We don't need to alter mtimes for directories, since black doesn't look at those.
-          # - We also only alter mtimes inside the `WORKDIR` since that's all we'll lint.
-          # - This should run before `poetry install`, because poetry's venv also contains
-          #   Python files, and we don't want to alter their mtimes since they aren't linted.
-
-          # Ensure we fail on non-zero exits and on undefined variables.
-          # Also print executed commands, for easier debugging.
-          set -eux
-
-          # Restore the mtimes of Python files in the workdir based on git history.
-          .github/tools/git-restore-mtime --no-directories "$WORKDIR/**/*.py"
-
-          # Since CI only does a partial fetch (to `FETCH_DEPTH`) for efficiency,
-          # the local git repo doesn't have full history. There are probably files
-          # that were last modified in a commit *older than* the oldest fetched commit.
-          # After `git-restore-mtime`, such files have a mtime set to the oldest fetched commit.
-          #
-          # As new commits get added, that timestamp will keep moving forward.
-          # If left unchanged, this will make `black` think that the files were edited
-          # more recently than its cache suggests. Instead, we can set their mtime
-          # to a fixed date in the far past that won't change and won't cause cache misses in black.
-          #
-          # For all workdir Python files modified in or before the oldest few fetched commits,
-          # make their mtime be 2000-01-01 00:00:00.
-          OLDEST_COMMIT="$(git log --reverse '--pretty=format:%H' | head -1)"
-          OLDEST_COMMIT_TIME="$(git show -s '--format=%ai' "$OLDEST_COMMIT")"
-          find "$WORKDIR" -name '*.py' -type f -not -newermt "$OLDEST_COMMIT_TIME" -exec touch -c -m -t '200001010000' '{}' '+'
-
-          echo "oldest-commit=$OLDEST_COMMIT" >> "$GITHUB_OUTPUT"
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
+          pipx install poetry==$POETRY_VERSION
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: lint-with-extras
-
-      - name: Check Poetry File
-        shell: bash
-        working-directory: ${{ inputs.working-directory }}
-        run: |
-          poetry check
-
-      - name: Check lock file
-        shell: bash
-        working-directory: ${{ inputs.working-directory }}
-        run: |
-          poetry lock --check
-
+          cache: poetry
       - name: Install dependencies
-        # Also installs dev/lint/test/typing dependencies, to ensure we have
-        # type hints for as many of our libraries as possible.
-        # This helps catch errors that require dependencies to be spotted, for example:
-        # https://github.com/langchain-ai/langchain/pull/10249/files#diff-935185cd488d015f026dcd9e19616ff62863e8cde8c0bee70318d3ccbca98341
-        #
-        # If you change this configuration, make sure to change the `cache-key`
-        # in the `poetry_setup` action above to stop using the old cache.
-        # It doesn't matter how you change it, any change will cause a cache-bust.
-        working-directory: ${{ inputs.working-directory }}
         run: |
-          poetry install --with dev,lint,test,typing
-
+          poetry install
       - name: Install langchain editable
-        working-directory: ${{ inputs.working-directory }}
-        if: ${{ inputs.working-directory != 'libs/langchain' }}
+        if: ${{ inputs.working-directory != 'langchain' }}
         run: |
           pip install -e ../langchain
-
-      - name: Restore black cache
-        uses: actions/cache@v3
-        env:
-          CACHE_BASE: black-${{ runner.os }}-${{ runner.arch }}-py${{ matrix.python-version }}-${{ inputs.working-directory }}-${{ hashFiles(format('{0}/poetry.lock', env.WORKDIR)) }}
-          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "1"
-        with:
-          path: |
-            ${{ env.WORKDIR }}/.black_cache
-          key: ${{ env.CACHE_BASE }}-${{ steps.restore-mtimes.outputs.oldest-commit }}
-          restore-keys:
-            # If we can't find an exact match for our cache key, accept any with this prefix.
-            ${{ env.CACHE_BASE }}-
-
-      - name: Get .mypy_cache to speed up mypy
-        uses: actions/cache@v3
-        env:
-          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "2"
-        with:
-          path: |
-            ${{ env.WORKDIR }}/.mypy_cache
-          key: mypy-${{ runner.os }}-${{ runner.arch }}-py${{ matrix.python-version }}-${{ inputs.working-directory }}-${{ hashFiles(format('{0}/poetry.lock', env.WORKDIR)) }}
-
       - name: Analysing the code with our lint
-        working-directory: ${{ inputs.working-directory }}
-        env:
-          BLACK_CACHE_DIR: .black_cache
         run: |
           make lint

.github/workflows/_pydantic_compatibility.yml

@@ -1,93 +0,0 @@
-name: pydantic v1/v2 compatibility
-
-on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-
-env:
-  POETRY_VERSION: "1.6.1"
-
-jobs:
-  build:
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Pydantic v1/v2 compatibility - Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: pydantic-cross-compat
-
-      - name: Install dependencies
-        shell: bash
-        run: poetry install
-
-      - name: Install the opposite major version of pydantic
-        # If normal tests use pydantic v1, here we'll use v2, and vice versa.
-        shell: bash
-        run: |
-          # Determine the major part of pydantic version
-          REGULAR_VERSION=$(poetry run python -c "import pydantic; print(pydantic.__version__)" | cut -d. -f1)
-
-          if [[ "$REGULAR_VERSION" == "1" ]]; then
-            PYDANTIC_DEP=">=2.1,<3"
-            TEST_WITH_VERSION="2"
-          elif [[ "$REGULAR_VERSION" == "2" ]]; then
-            PYDANTIC_DEP="<2"
-            TEST_WITH_VERSION="1"
-          else
-            echo "Unexpected pydantic major version '$REGULAR_VERSION', cannot determine which version to use for cross-compatibility test."
-            exit 1
-          fi
-
-          # Install via `pip` instead of `poetry add` to avoid changing lockfile,
-          # which would prevent caching from working: the cache would get saved
-          # to a different key than where it gets loaded from.
-          poetry run pip install "pydantic${PYDANTIC_DEP}"
-
-          # Ensure that the correct pydantic is installed now.
-          echo "Checking pydantic version... Expecting ${TEST_WITH_VERSION}"
-
-          # Determine the major part of pydantic version
-          CURRENT_VERSION=$(poetry run python -c "import pydantic; print(pydantic.__version__)" | cut -d. -f1)
-
-          # Check that the major part of pydantic version is as expected, if not
-          # raise an error
-          if [[ "$CURRENT_VERSION" != "$TEST_WITH_VERSION" ]]; then
-            echo "Error: expected pydantic version ${CURRENT_VERSION} to have been installed, but found: ${TEST_WITH_VERSION}"
-            exit 1
-          fi
-          echo "Found pydantic version ${CURRENT_VERSION}, as expected"
-      - name: Run pydantic compatibility tests
-        shell: bash
-        run: make test
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/_release.yml

@@ -9,37 +9,26 @@ on:
         description: "From which folder this pipeline executes"
 
 env:
-  POETRY_VERSION: "1.6.1"
+  POETRY_VERSION: "1.4.2"
 
 jobs:
   if_release:
-    # Disallow publishing from branches that aren't `master`.
-    if: github.ref == 'refs/heads/master'
+    if: |
+        ${{ github.event.pull_request.merged == true }}
+        && ${{ contains(github.event.pull_request.labels.*.name, 'release') }}
     runs-on: ubuntu-latest
-    permissions:
-      # This permission is used for trusted publishing:
-      # https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/
-      #
-      # Trusted publishing has to also be configured on PyPI for each package:
-      # https://docs.pypi.org/trusted-publishers/adding-a-publisher/
-      id-token: write
-
-      # This permission is needed by `ncipollo/release-action` to create the GitHub release.
-      contents: write
     defaults:
       run:
         working-directory: ${{ inputs.working-directory }}
     steps:
       - uses: actions/checkout@v3
-
-      - name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
+      - name: Install poetry
+        run: pipx install poetry==$POETRY_VERSION
+      - name: Set up Python 3.10
+        uses: actions/setup-python@v4
         with:
           python-version: "3.10"
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: release
-
+          cache: "poetry"
       - name: Build project for distribution
         run: poetry build
       - name: Check Version
@@ -48,7 +37,6 @@ jobs:
           echo version=$(poetry version --short) >> $GITHUB_OUTPUT
       - name: Create Release
         uses: ncipollo/release-action@v1
-        if: ${{ inputs.working-directory == 'libs/langchain' }}
         with:
           artifacts: "dist/*"
           token: ${{ secrets.GITHUB_TOKEN }}
@@ -56,9 +44,8 @@ jobs:
           generateReleaseNotes: true
           tag: v${{ steps.check-version.outputs.version }}
           commit: master
-      - name: Publish package distributions to PyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          packages-dir: ${{ inputs.working-directory }}/dist/
-          verbose: true
-          print-hash: true
+      - name: Publish to PyPI
+        env:
+          POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
+        run: |
+          poetry publish

.github/workflows/_release_docker.yml

@@ -1,62 +0,0 @@
-name: release_docker
-
-on:
-  workflow_call:
-    inputs:
-      dockerfile:
-        required: true
-        type: string
-        description: "Path to the Dockerfile to build"
-      image:
-        required: true
-        type: string
-        description: "Name of the image to build"
-
-env:
-  TEST_TAG: ${{ inputs.image }}:test
-  LATEST_TAG: ${{ inputs.image }}:latest
-
-jobs:
-  docker:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Get git tag
-        uses: actions-ecosystem/action-get-latest-tag@v1
-        id: get-latest-tag
-      - name: Set docker tag
-        env:
-          VERSION: ${{ steps.get-latest-tag.outputs.tag }}
-        run: |
-          echo "VERSION_TAG=${{ inputs.image }}:${VERSION#v}" >> $GITHUB_ENV
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-      - name: Login to Docker Hub
-        uses: docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-      - name: Build for Test
-        uses: docker/build-push-action@v5
-        with:
-          context: .
-          file: ${{ inputs.dockerfile }}
-          load: true
-          tags: ${{ env.TEST_TAG }}
-      - name: Test
-        run: |
-          docker run --rm ${{ env.TEST_TAG }} python -c "import langchain"
-      - name: Build and Push to Docker Hub
-        uses: docker/build-push-action@v5
-        with:
-          context: .
-          file: ${{ inputs.dockerfile }}
-          # We can only build for the intersection of platforms supported by
-          # QEMU and base python image, for now build only for
-          # linux/amd64 and linux/arm64
-          platforms: linux/amd64,linux/arm64
-          tags: ${{ env.LATEST_TAG }},${{ env.VERSION_TAG }}
-          push: true

.github/workflows/_test.yml

@@ -7,9 +7,13 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
+      test_type:
+        type: string
+        description: "Test types to run"
+        default: '["core", "extended"]'
 
 env:
-  POETRY_VERSION: "1.6.1"
+  POETRY_VERSION: "1.4.2"
 
 jobs:
   build:
@@ -24,34 +28,34 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
-    name: Python ${{ matrix.python-version }}
+        test_type: ${{ fromJSON(inputs.test_type) }}
+    name: Python ${{ matrix.python-version }} ${{ matrix.test_type }}
     steps:
       - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
+      - name: Set up Python ${{ matrix.python-version }}
         uses: "./.github/actions/poetry_setup"
         with:
           python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
           working-directory: ${{ inputs.working-directory }}
-          cache-key: core
-
-      - name: Install dependencies
-        shell: bash
-        run: poetry install
-
-      - name: Run core tests
-        shell: bash
-        run: make test
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
+          poetry-version: "1.4.2"
+          cache-key: ${{ matrix.test_type }}
+          install-command: |
+              if [ "${{ matrix.test_type }}" == "core" ]; then
+                echo "Running core tests, installing dependencies with poetry..."
+                poetry install
+              else
+                echo "Running extended tests, installing dependencies with poetry..."
+                poetry install -E extended_testing
+              fi
+      - name: Install langchain editable
+        if: ${{ inputs.working-directory != 'langchain' }}
         run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'
+          pip install -e ../langchain
+      - name: Run ${{matrix.test_type}} tests
+        run: |
+          if [ "${{ matrix.test_type }}" == "core" ]; then
+            make test
+          else
+            make extended_tests
+          fi
+        shell: bash

.github/workflows/codespell.yml

@@ -18,19 +18,7 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v3
-
-      - name: Install Dependencies
-        run: |
-          pip install toml
-
-      - name: Extract Ignore Words List
-        run: |
-          # Use a Python script to extract the ignore words list from pyproject.toml
-          python .github/workflows/extract_ignored_words_list.py
-        id: extract_ignore_words
-
       - name: Codespell
         uses: codespell-project/actions-codespell@v2
         with:
           skip: guide_imports.json
-          ignore_words_list: ${{ steps.extract_ignore_words.outputs.ignore_words_list }}

.github/workflows/doc_lint.yml

@@ -1,22 +0,0 @@
----
-name: Documentation Lint
-
-on:
-  push:
-    branches: [master]
-  pull_request:
-    branches: [master]
-
-jobs:
-  check:
-    runs-on: ubuntu-latest
-
-    steps:
-    - name: Checkout repository
-      uses: actions/checkout@v2
-
-    - name: Run import check
-      run: |
-        # We should not encourage imports directly from main init file
-        # Expect for hub
-        git grep 'from langchain import' docs/{docs,snippets} | grep -vE 'from langchain import (hub)' && exit 1 || exit 0

.github/workflows/extract_ignored_words_list.py

@@ -1,8 +0,0 @@
-import toml
-
-pyproject_toml = toml.load("pyproject.toml")
-
-# Extract the ignore words list (adjust the key as per your TOML structure)
-ignore_words_list = pyproject_toml.get("tool", {}).get("codespell", {}).get("ignore-words-list")
-
-print(f"::set-output name=ignore_words_list::{ignore_words_list}")

.github/workflows/langchain_ci.yml

@@ -6,29 +6,12 @@ on:
     branches: [ master ]
   pull_request:
     paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
       - '.github/workflows/_lint.yml'
       - '.github/workflows/_test.yml'
-      - '.github/workflows/_pydantic_compatibility.yml'
       - '.github/workflows/langchain_ci.yml'
       - 'libs/langchain/**'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-env:
-  POETRY_VERSION: "1.6.1"
-  WORKDIR: "libs/langchain"
-
 jobs:
   lint:
     uses:
@@ -36,62 +19,9 @@ jobs:
     with:
       working-directory: libs/langchain
     secrets: inherit
-
   test:
     uses:
       ./.github/workflows/_test.yml
     with:
       working-directory: libs/langchain
-    secrets: inherit
-
-  pydantic-compatibility:
-    uses:
-      ./.github/workflows/_pydantic_compatibility.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit
-
-  extended-tests:
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        working-directory: ${{ env.WORKDIR }}
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }} extended tests
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: libs/langchain
-          cache-key: extended
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          echo "Running extended tests, installing dependencies with poetry..."
-          poetry install -E extended_testing
-
-      - name: Run extended tests
-        run: make extended_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'
+    secrets: inherit

.github/workflows/langchain_experimental_ci.yml

@@ -1,13 +1,11 @@
 ---
-name: libs/experimental CI
+name: libs/langchain-experimental CI
 
 on:
   push:
     branches: [ master ]
   pull_request:
     paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
       - '.github/workflows/_lint.yml'
       - '.github/workflows/_test.yml'
       - '.github/workflows/langchain_experimental_ci.yml'
@@ -15,20 +13,6 @@ on:
       - 'libs/experimental/**'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-env:
-  POETRY_VERSION: "1.6.1"
-  WORKDIR: "libs/experimental"
-
 jobs:
   lint:
     uses:
@@ -36,94 +20,10 @@ jobs:
     with:
       working-directory: libs/experimental
     secrets: inherit
-
   test:
     uses:
       ./.github/workflows/_test.yml
     with:
       working-directory: libs/experimental
-    secrets: inherit
-
-  # It's possible that langchain-experimental works fine with the latest *published* langchain,
-  # but is broken with the langchain on `master`.
-  #
-  # We want to catch situations like that *before* releasing a new langchain, hence this test.
-  test-with-latest-langchain:
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        working-directory: ${{ env.WORKDIR }}
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: test with unpublished langchain - Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ env.WORKDIR }}
-          cache-key: unpublished-langchain
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          echo "Running tests with unpublished langchain, installing dependencies with poetry..."
-          poetry install
-
-          echo "Editably installing langchain outside of poetry, to avoid messing up lockfile..."
-          poetry run pip install -e ../langchain
-
-      - name: Run tests
-        run: make test
-  extended-tests:
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        working-directory: ${{ env.WORKDIR }}
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }} extended tests
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: libs/experimental
-          cache-key: extended
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          echo "Running extended tests, installing dependencies with poetry..."
-          poetry install -E extended_testing
-
-      - name: Run extended tests
-        run: make extended_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'
+      test_type: '["core"]'
+    secrets: inherit

.github/workflows/langchain_experimental_release.yml

@@ -1,7 +1,14 @@
 ---
-name: libs/experimental Release
+name: libs/langchain-experimental Release
 
 on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - master
+    paths:
+      - 'libs/experimental/pyproject.toml'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
 jobs:
@@ -10,4 +17,4 @@ jobs:
       ./.github/workflows/_release.yml
     with:
       working-directory: libs/experimental
-    secrets: inherit
+    secrets: inherit

.github/workflows/langchain_release.yml

@@ -2,6 +2,13 @@
 name: libs/langchain Release
 
 on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - master
+    paths:
+      - 'libs/langchain/pyproject.toml'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
 jobs:
@@ -10,18 +17,4 @@ jobs:
       ./.github/workflows/_release.yml
     with:
       working-directory: libs/langchain
-    secrets: inherit
-
-  # N.B.: It's possible that PyPI doesn't make the new release visible / available
-  #       immediately after publishing. If that happens, the docker build might not
-  #       create a new docker image for the new release, since it won't see it.
-  #
-  #       If this ends up being a problem, add a check to the end of the `_release.yml`
-  #       workflow that prevents the workflow from finishing until the new release
-  #       is visible and installable on PyPI.
-  release-docker:
-    needs:
-      - release
-    uses:
-      ./.github/workflows/langchain_release_docker.yml
-    secrets: inherit
+    secrets: inherit

.github/workflows/langchain_release_docker.yml

@@ -1,14 +0,0 @@
----
-name: docker/langchain/langchain Release
-
-on:
-  workflow_dispatch: # Allows to trigger the workflow manually in GitHub UI
-  workflow_call: # Allows triggering from another workflow
-
-jobs:
-  release:
-    uses: ./.github/workflows/_release_docker.yml
-    with:
-      dockerfile: docker/Dockerfile.base
-      image: langchain/langchain
-    secrets: inherit

.github/workflows/scheduled_test.yml

@@ -1,81 +0,0 @@
-name: Scheduled tests
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-  schedule:
-    - cron:  '0 13 * * *'
-
-env:
-  POETRY_VERSION: "1.6.1"
-
-jobs:
-  build:
-    defaults:
-      run:
-        working-directory: libs/langchain
-    runs-on: ubuntu-latest
-    environment: Scheduled testing
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: libs/langchain
-          cache-key: scheduled
-
-      - name: 'Authenticate to Google Cloud'
-        id: 'auth'
-        uses: 'google-github-actions/auth@v1'
-        with:
-          credentials_json: '${{ secrets.GOOGLE_CREDENTIALS }}'
-
-      - name: Configure AWS Credentials
-        uses: aws-actions/configure-aws-credentials@v4
-        with:
-          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
-          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
-          aws-region: ${{ vars.AWS_REGION }}
-
-      - name: Install dependencies
-        working-directory: libs/langchain
-        shell: bash
-        run: |
-          echo "Running scheduled tests, installing dependencies with poetry..."
-          poetry install --with=test_integration
-          poetry run pip install google-cloud-aiplatform
-          poetry run pip install "boto3>=1.28.57"
-
-      - name: Run tests
-        shell: bash
-        env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          AZURE_OPENAI_API_VERSION: ${{ secrets.AZURE_OPENAI_API_VERSION }}
-          AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }}
-          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
-          AZURE_OPENAI_DEPLOYMENT_NAME:  ${{ secrets.AZURE_OPENAI_DEPLOYMENT_NAME }}
-        run: |
-          make scheduled_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.gitignore

@@ -30,12 +30,6 @@ share/python-wheels/
 *.egg
 MANIFEST
 
-# Google GitHub Actions credentials files created by:
-# https://github.com/google-github-actions/auth
-#
-# That action recommends adding this gitignore to prevent accidentally committing keys.
-gha-creds-*.json
-
 # PyInstaller
 #  Usually these files are written by a python script from a template
 #  before PyInstaller builds the exe, so as to inject date/other infos into it.
@@ -168,13 +162,11 @@ docs/.docusaurus/
 docs/.cache-loader/
 docs/_dist
 docs/api_reference/api_reference.rst
-docs/api_reference/experimental_api_reference.rst
 docs/api_reference/_build
 docs/api_reference/*/
 !docs/api_reference/_static/
 !docs/api_reference/templates/
 !docs/api_reference/themes/
-docs/docs/build
-docs/docs/node_modules
-docs/docs/yarn.lock
-_dist
+docs/docs_skeleton/build
+docs/docs_skeleton/node_modules
+docs/docs_skeleton/yarn.lock

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "docs/_docs_skeleton"]
+	path = docs/_docs_skeleton
+	url = https://github.com/langchain-ai/langchain-shared-docs
+	branch = main

.readthedocs.yaml

@@ -9,14 +9,9 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.11"
-  commands:
-      - python -mvirtualenv $READTHEDOCS_VIRTUALENV_PATH
-      - python -m pip install --upgrade --no-cache-dir pip setuptools
-      - python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext
-      - python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt
+  jobs:
+    pre_build:
       - python docs/api_reference/create_api_rst.py
-      - cat docs/api_reference/conf.py
-      - python -m sphinx -T -E -b html -d _build/doctrees -c docs/api_reference docs/api_reference $READTHEDOCS_OUTPUT/html -j auto
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
@@ -30,3 +25,5 @@ sphinx:
 python:
    install:
    - requirements: docs/api_reference/requirements.txt
+   - method: pip
+     path: .

CITATION.cff

@@ -5,4 +5,4 @@ authors:
   given-names: "Harrison"
 title: "LangChain"
 date-released: 2022-10-17
-url: "https://github.com/langchain-ai/langchain"
+url: "https://github.com/hwchase17/langchain"

MIGRATE.md

@@ -43,10 +43,6 @@ Now:
 
 `from langchain_experimental.sql import SQLDatabaseChain`
 
-Alternatively, if you are just interested in using the query generation part of the SQL chain, you can check out [`create_sql_query_chain`](https://github.com/langchain-ai/langchain/blob/master/docs/extras/use_cases/tabular/sql_query.ipynb)
-
-`from langchain.chains import create_sql_query_chain`
-
 ## `load_prompt` for Python files
 
 Note: this only applies if you want to load Python files as prompts.

Makefile

@@ -15,10 +15,10 @@ docs_build:
 	docs/.local_build.sh
 
 docs_clean:
-	rm -r _dist
+	rm -r docs/_dist
 
 docs_linkcheck:
-	poetry run linkchecker _dist/docs/ --ignore-url node_modules
+	poetry run linkchecker docs/_dist/docs_skeleton/ --ignore-url node_modules
 
 api_docs_build:
 	poetry run python docs/api_reference/create_api_rst.py
@@ -42,15 +42,8 @@ spell_fix:
 ######################
 
 help:
-	@echo '===================='
-	@echo '-- DOCUMENTATION --'
-	@echo 'clean                        - run docs_clean and api_docs_clean'
+	@echo '----'
+	@echo 'coverage                     - run unit tests and generate coverage report'
 	@echo 'docs_build                   - build the documentation'
 	@echo 'docs_clean                   - clean the documentation build artifacts'
 	@echo 'docs_linkcheck               - run linkchecker on the documentation'
-	@echo 'api_docs_build               - build the API Reference documentation'
-	@echo 'api_docs_clean               - clean the API Reference documentation build artifacts'
-	@echo 'api_docs_linkcheck           - run linkchecker on the API Reference documentation'
-	@echo 'spell_check               	- run codespell on the project'
-	@echo 'spell_fix               		- run codespell on the project and fix the errors'
-	@echo '-- TEST and LINT tasks are within libs/*/ per-package --'

README.md

@@ -2,32 +2,31 @@
 
 ⚡ Building applications with LLMs through composability ⚡
 
-[![Release Notes](https://img.shields.io/github/release/langchain-ai/langchain)](https://github.com/langchain-ai/langchain/releases)
-[![CI](https://github.com/langchain-ai/langchain/actions/workflows/langchain_ci.yml/badge.svg)](https://github.com/langchain-ai/langchain/actions/workflows/langchain_ci.yml)
-[![Experimental CI](https://github.com/langchain-ai/langchain/actions/workflows/langchain_experimental_ci.yml/badge.svg)](https://github.com/langchain-ai/langchain/actions/workflows/langchain_experimental_ci.yml)
+[![Release Notes](https://img.shields.io/github/release/hwchase17/langchain)](https://github.com/hwchase17/langchain/releases)
+[![CI](https://github.com/hwchase17/langchain/actions/workflows/langchain_ci.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/langchain_ci.yml)
+[![Experimental CI](https://github.com/hwchase17/langchain/actions/workflows/langchain_experimental_ci.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/langchain_experimental_ci.yml)
 [![Downloads](https://static.pepy.tech/badge/langchain/month)](https://pepy.tech/project/langchain)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40LangChainAI)](https://twitter.com/langchainai)
 [![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true&style=flat)](https://discord.gg/6adMQxSpJS)
-[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/langchain-ai/langchain)
-[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/langchain-ai/langchain)
-[![GitHub star chart](https://img.shields.io/github/stars/langchain-ai/langchain?style=social)](https://star-history.com/#langchain-ai/langchain)
-[![Dependency Status](https://img.shields.io/librariesio/github/langchain-ai/langchain)](https://libraries.io/github/langchain-ai/langchain)
-[![Open Issues](https://img.shields.io/github/issues-raw/langchain-ai/langchain)](https://github.com/langchain-ai/langchain/issues)
+[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hwchase17/langchain)
+[![GitHub star chart](https://img.shields.io/github/stars/hwchase17/langchain?style=social)](https://star-history.com/#hwchase17/langchain)
+[![Dependency Status](https://img.shields.io/librariesio/github/hwchase17/langchain)](https://libraries.io/github/hwchase17/langchain)
+[![Open Issues](https://img.shields.io/github/issues-raw/hwchase17/langchain)](https://github.com/hwchase17/langchain/issues)
 
 
-Looking for the JS/TS version? Check out [LangChain.js](https://github.com/langchain-ai/langchainjs).
+Looking for the JS/TS version? Check out [LangChain.js](https://github.com/hwchase17/langchainjs).
 
-To help you ship LangChain apps to production faster, check out [LangSmith](https://smith.langchain.com). 
-[LangSmith](https://smith.langchain.com) is a unified developer platform for building, testing, and monitoring LLM applications. 
-Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) to get off the waitlist or speak with our sales team
+**Production Support:** As you move your LangChains into production, we'd love to offer more comprehensive support.
+Please fill out [this form](https://6w1pwbss0py.typeform.com/to/rrbrdTH2) and we'll set up a dedicated support Slack channel.
 
-## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28/23
+## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28
 
 In an effort to make `langchain` leaner and safer, we are moving select chains to `langchain_experimental`.
 This migration has already started, but we are remaining backwards compatible until 7/28.
 On that date, we will remove functionality from `langchain`.
-Read more about the motivation and the progress [here](https://github.com/langchain-ai/langchain/discussions/8043).
+Read more about the motivation and the progress [here](https://github.com/hwchase17/langchain/discussions/8043).
 Read how to migrate your code [here](MIGRATE.md).
 
 ## Quick Install
@@ -50,7 +49,7 @@ This library aims to assist in the development of those types of applications. C
 **💬 Chatbots**
 
 - [Documentation](https://python.langchain.com/docs/use_cases/chatbots/)
-- End-to-end Example: [Chat-LangChain](https://github.com/langchain-ai/chat-langchain)
+- End-to-end Example: [Chat-LangChain](https://github.com/hwchase17/chat-langchain)
 
 **🤖 Agents**
 

SECURITY.md

@@ -1,6 +0,0 @@
-# Security Policy
-
-## Reporting a Vulnerability
-
-Please report security vulnerabilities by email to `security@langchain.dev`.
-This email is an alias to a subset of our maintainers, and will ensure the issue is promptly triaged and acted upon as needed.

cookbook/LLaMA2_sql_chat.ipynb

@@ -1,400 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "fc935871-7640-41c6-b798-58514d860fe0",
-   "metadata": {},
-   "source": [
-    "## LLaMA2 chat with SQL\n",
-    "\n",
-    "Open source, local LLMs are great to consider for any application that demands data privacy.\n",
-    "\n",
-    "SQL is one good example. \n",
-    "\n",
-    "This cookbook shows how to perform text-to-SQL using various local versions of LLaMA2 run locally.\n",
-    "\n",
-    "## Packages"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "81adcf8b-395a-4f02-8749-ac976942b446",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "! pip install langchain replicate"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8e13ed66-300b-4a23-b8ac-44df68ee4733",
-   "metadata": {},
-   "source": [
-    "## LLM\n",
-    "\n",
-    "There are a few ways to access LLaMA2.\n",
-    "\n",
-    "To run locally, we use Ollama.ai. \n",
-    "\n",
-    "See [here](https://python.langchain.com/docs/integrations/chat/ollama) for details on installation and setup.\n",
-    "\n",
-    "Also, see [here](https://python.langchain.com/docs/guides/local_llms) for our full guide on local LLMs.\n",
-    " \n",
-    "To use an external API, which is not private, we can use Replicate."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "6a75a5c6-34ee-4ab9-a664-d9b432d812ee",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Init param `input` is deprecated, please use `model_kwargs` instead.\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Local \n",
-    "from langchain.chat_models import ChatOllama\n",
-    "llama2_chat = ChatOllama(model=\"llama2:13b-chat\")\n",
-    "llama2_code = ChatOllama(model=\"codellama:7b-instruct\")\n",
-    "\n",
-    "# API\n",
-    "from getpass import getpass\n",
-    "from langchain.llms import Replicate\n",
-    "# REPLICATE_API_TOKEN = getpass()\n",
-    "# os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN\n",
-    "replicate_id = \"meta/llama-2-13b-chat:f4e2de70d66816a838a89eeeb621910adffb0dd0baba3976c96980970978018d\"\n",
-    "llama2_chat_replicate = Replicate(\n",
-    "    model=replicate_id,\n",
-    "    input={\"temperature\": 0.01, \n",
-    "           \"max_length\": 500, \n",
-    "           \"top_p\": 1}\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "ce96f7ea-b3d5-44e1-9fa5-a79e04a9e1fb",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Simply set the LLM we want to use\n",
-    "llm = llama2_chat"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "80222165-f353-4e35-a123-5f70fd70c6c8",
-   "metadata": {},
-   "source": [
-    "## DB\n",
-    "\n",
-    "Connect to a SQLite DB.\n",
-    "\n",
-    "To create this particular DB, you can use the code and follow the steps shown [here](https://github.com/facebookresearch/llama-recipes/blob/main/demo_apps/StructuredLlama.ipynb)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "025bdd82-3bb1-4948-bc7c-c3ccd94fd05c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.utilities import SQLDatabase\n",
-    "db = SQLDatabase.from_uri(\"sqlite:///nba_roster.db\", sample_rows_in_table_info= 0)\n",
-    "\n",
-    "def get_schema(_):\n",
-    "    return db.get_table_info()\n",
-    "\n",
-    "def run_query(query):\n",
-    "    return db.run(query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "654b3577-baa2-4e12-a393-f40e5db49ac7",
-   "metadata": {},
-   "source": [
-    "## Query a SQL DB \n",
-    "\n",
-    "Follow the runnables workflow [here](https://python.langchain.com/docs/expression_language/cookbook/sql_db)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "5a4933ea-d9c0-4b0a-8177-ba4490c6532b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' SELECT \"Team\" FROM nba_roster WHERE \"NAME\" = \\'Klay Thompson\\';'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Prompt\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "template = \"\"\"Based on the table schema below, write a SQL query that would answer the user's question:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query:\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question, convert it to a SQL query. No pre-amble.\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "# Chain to query\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "\n",
-    "sql_response = (\n",
-    "        RunnablePassthrough.assign(schema=get_schema)\n",
-    "        | prompt\n",
-    "        | llm.bind(stop=[\"\\nSQLResult:\"])\n",
-    "        | StrOutputParser()\n",
-    "    )\n",
-    "\n",
-    "sql_response.invoke({\"question\": \"What team is Klay Thompson on?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a0e9e2c8-9b88-4853-ac86-001bc6cc6695",
-   "metadata": {},
-   "source": [
-    "We can review the results:\n",
-    "\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/afa56a06-b4e2-469a-a60f-c1746e75e42b/r) LLaMA2-13 Replicate API\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/2d4ecc72-6b8f-4523-8f0b-ea95c6b54a1d/r) LLaMA2-13 local \n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "2a2825e3-c1b6-4f7d-b9c9-d9835de323bb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Based on the table schema and SQL query, there are 30 unique teams in the NBA.')"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Chain to answer\n",
-    "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query: {query}\n",
-    "SQL Response: {response}\"\"\"\n",
-    "prompt_response = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "full_chain = (\n",
-    "    RunnablePassthrough.assign(query=sql_response) \n",
-    "    | RunnablePassthrough.assign(\n",
-    "        schema=get_schema,\n",
-    "        response=lambda x: db.run(x[\"query\"]),\n",
-    "    )\n",
-    "    | prompt_response \n",
-    "    | llm\n",
-    ")\n",
-    "\n",
-    "full_chain.invoke({\"question\": \"How many unique teams are there?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ec17b3ee-6618-4681-b6df-089bbb5ffcd7",
-   "metadata": {},
-   "source": [
-    "We can review the results:\n",
-    "\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/10420721-746a-4806-8ecf-d6dc6399d739/r) LLaMA2-13 Replicate API\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/5265ebab-0a22-4f37-936b-3300f2dfa1c1/r) LLaMA2-13 local "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1e85381b-1edc-4bb3-a7bd-2ab23f81e54d",
-   "metadata": {},
-   "source": [
-    "## Chat with a SQL DB \n",
-    "\n",
-    "Next, we can add memory."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "1985aa1c-eb8f-4fb1-a54f-c8aa10744687",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' SELECT \"Team\" FROM nba_roster WHERE \"NAME\" = \\'Klay Thompson\\';'"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Prompt\n",
-    "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
-    "template = \"\"\"Based on the table schema below, write a SQL query that would answer the user's question:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query:\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question, convert it to a SQL query. No pre-amble.\"),\n",
-    "    MessagesPlaceholder(variable_name=\"history\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "memory = ConversationBufferMemory(return_messages=True)\n",
-    "\n",
-    "# Chain to query with memory \n",
-    "from langchain.schema.runnable import RunnableLambda\n",
-    "\n",
-    "sql_chain = (\n",
-    "    RunnablePassthrough.assign(\n",
-    "       schema=get_schema,\n",
-    "       history=RunnableLambda(lambda x: memory.load_memory_variables(x)[\"history\"])\n",
-    "    )| prompt\n",
-    "    | llm.bind(stop=[\"\\nSQLResult:\"])\n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "\n",
-    "def save(input_output):\n",
-    "    output = {\"output\": input_output.pop(\"output\")}\n",
-    "    memory.save_context(input_output, output)\n",
-    "    return output['output']\n",
-    "    \n",
-    "sql_response_memory = RunnablePassthrough.assign(output=sql_chain) | save\n",
-    "sql_response_memory.invoke({\"question\": \"What team is Klay Thompson on?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "0b45818a-1498-441d-b82d-23c29428c2bb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' SELECT \"SALARY\" FROM nba_roster WHERE \"NAME\" = \\'Klay Thompson\\';'"
-      ]
-     },
-     "execution_count": 20,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "sql_response_memory.invoke({\"question\": \"What is his salary?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "800a7a3b-f411-478b-af51-2310cd6e0425",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Sure! Here\\'s the natural language response based on the given input:\\n\\n\"Klay Thompson\\'s salary is $43,219,440.\"')"
-      ]
-     },
-     "execution_count": 21,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Chain to answer\n",
-    "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query: {query}\n",
-    "SQL Response: {response}\"\"\"\n",
-    "prompt_response = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "full_chain = (\n",
-    "    RunnablePassthrough.assign(query=sql_response_memory) \n",
-    "    | RunnablePassthrough.assign(\n",
-    "        schema=get_schema,\n",
-    "        response=lambda x: db.run(x[\"query\"]),\n",
-    "    )\n",
-    "    | prompt_response \n",
-    "    | llm\n",
-    ")\n",
-    "\n",
-    "full_chain.invoke({\"question\": \"What is his salary?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b77fee61-f4da-4bb1-8285-14101e505518",
-   "metadata": {},
-   "source": [
-    "Here is the [trace](https://smith.langchain.com/public/54794d18-2337-4ce2-8b9f-3d8a2df89e51/r)."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.16"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/README.md

@@ -1,51 +0,0 @@
-# LangChain cookbook
-
-Example code for building applications with LangChain, with an emphasis on more applied and end-to-end examples than contained in the [main documentation](https://python.langchain.com).
-
-Notebook | Description
-:- | :-
-[LLaMA2_sql_chat.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/LLaMA2_sql_chat.ipynb) | Build a chat application that interacts with a sql database using an open source llm (llama2), specifically demonstrated on a sqlite database containing nba rosters.
-[Semi_Structured_RAG.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_Structured_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data, including text and tables, using unstructured for parsing, multi-vector retriever for storing, and lcel for implementing chains.
-[Semi_structured_and_multi_moda...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using unstructured for parsing, multi-vector retriever for storage and retrieval, and lcel for implementing chains.
-[Semi_structured_multi_modal_RA...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using various tools and methods such as unstructured for parsing, multi-vector retriever for storing, lcel for implementing chains, and open source language models like llama2, llava, and gpt4all.
-[autogpt/autogpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/autogpt.ipynb) | Implement autogpt, a language model, with langchain primitives such as llms, prompttemplates, vectorstores, embeddings, and tools.
-[autogpt/marathon_times.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/marathon_times.ipynb) | Implement autogpt for finding winning marathon times.
-[baby_agi.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi.ipynb) | Implement babyagi, an ai agent that can generate and execute tasks based on a given objective, with the flexibility to swap out specific vectorstores/model providers.
-[baby_agi_with_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi_with_agent.ipynb) | Swap out the execution chain in the babyagi notebook with an agent that has access to tools, aiming to obtain more reliable information.
-[camel_role_playing.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/camel_role_playing.ipynb) | Implement the camel framework for creating autonomous cooperative agents in large scale language models, using role-playing and inception prompting to guide chat agents towards task completion.
-[causal_program_aided_language_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/causal_program_aided_language_model.ipynb) | Implement the causal program-aided language (cpal) chain, which improves upon the program-aided language (pal) by incorporating causal structure to prevent hallucination in language models, particularly when dealing with complex narratives and math problems with nested dependencies.
-[code-analysis-deeplake.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/code-analysis-deeplake.ipynb) | Analyze its own code base with the help of gpt and activeloop's deep lake.
-[custom_agent_with_plugin_retri...](https://github.com/langchain-ai/langchain/tree/master/cookbook/custom_agent_with_plugin_retrieval.ipynb) | Build a custom agent that can interact with ai plugins by retrieving tools and creating natural language wrappers around openapi endpoints.
-[custom_agent_with_plugin_retri...](https://github.com/langchain-ai/langchain/tree/master/cookbook/custom_agent_with_plugin_retrieval_using_plugnplai.ipynb) | Build a custom agent with plugin retrieval functionality, utilizing ai plugins from the `plugnplai` directory.
-[databricks_sql_db.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/databricks_sql_db.ipynb) | Connect to databricks runtimes and databricks sql.
-[deeplake_semantic_search_over_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/deeplake_semantic_search_over_chat.ipynb) | Perform semantic search and question-answering over a group chat using activeloop's deep lake with gpt4.
-[elasticsearch_db_qa.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/elasticsearch_db_qa.ipynb) | Interact with elasticsearch analytics databases in natural language and build search queries via the elasticsearch dsl api.
-[forward_looking_retrieval_augm...](https://github.com/langchain-ai/langchain/tree/master/cookbook/forward_looking_retrieval_augmented_generation.ipynb) | Implement the forward-looking active retrieval augmented generation (flare) method, which generates answers to questions, identifies uncertain tokens, generates hypothetical questions based on these tokens, and retrieves relevant documents to continue generating the answer.
-[generative_agents_interactive_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/generative_agents_interactive_simulacra_of_human_behavior.ipynb) | Implement a generative agent that simulates human behavior, based on a research paper, using a time-weighted memory object backed by a langchain retriever.
-[gymnasium_agent_simulation.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/gymnasium_agent_simulation.ipynb) | Create a simple agent-environment interaction loop in simulated environments like text-based games with gymnasium.
-[hugginggpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/hugginggpt.ipynb) | Implement hugginggpt, a system that connects language models like chatgpt with the machine learning community via hugging face.
-[hypothetical_document_embeddin...](https://github.com/langchain-ai/langchain/tree/master/cookbook/hypothetical_document_embeddings.ipynb) | Improve document indexing with hypothetical document embeddings (hyde), an embedding technique that generates and embeds hypothetical answers to queries.
-[learned_prompt_optimization.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/learned_prompt_optimization.ipynb) | Automatically enhance language model prompts by injecting specific terms using reinforcement learning, which can be used to personalize responses based on user preferences.
-[llm_bash.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_bash.ipynb) | Perform simple filesystem commands using language learning models (llms) and a bash process.
-[llm_checker.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_checker.ipynb) | Create a self-checking chain using the llmcheckerchain function.
-[llm_math.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_math.ipynb) | Solve complex word math problems using language models and python repls.
-[llm_summarization_checker.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_summarization_checker.ipynb) | Check the accuracy of text summaries, with the option to run the checker multiple times for improved results.
-[llm_symbolic_math.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_symbolic_math.ipynb) | Solve algebraic equations with the help of llms (language learning models) and sympy, a python library for symbolic mathematics.
-[meta_prompt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/meta_prompt.ipynb) | Implement the meta-prompt concept, which is a method for building self-improving agents that reflect on their own performance and modify their instructions accordingly.
-[multi_modal_output_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multi_modal_output_agent.ipynb) | Generate multi-modal outputs, specifically images and text.
-[multi_player_dnd.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multi_player_dnd.ipynb) | Simulate multi-player dungeons & dragons games, with a custom function determining the speaking schedule of the agents.
-[multiagent_authoritarian.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multiagent_authoritarian.ipynb) | Implement a multi-agent simulation where a privileged agent controls the conversation, including deciding who speaks and when the conversation ends, in the context of a simulated news network.
-[multiagent_bidding.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multiagent_bidding.ipynb) | Implement a multi-agent simulation where agents bid to speak, with the highest bidder speaking next, demonstrated through a fictitious presidential debate example.
-[myscale_vector_sql.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/myscale_vector_sql.ipynb) | Access and interact with the myscale integrated vector database, which can enhance the performance of language model (llm) applications.
-[openai_functions_retrieval_qa....](https://github.com/langchain-ai/langchain/tree/master/cookbook/openai_functions_retrieval_qa.ipynb) | Structure response output in a question answering system by incorporating openai functions into a retrieval pipeline.
-[petting_zoo.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/petting_zoo.ipynb) | Create multi-agent simulations with simulated environments using the petting zoo library.
-[plan_and_execute_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/plan_and_execute_agent.ipynb) | Create plan-and-execute agents that accomplish objectives by planning tasks with a language model (llm) and executing them with a separate agent.
-[program_aided_language_model.i...](https://github.com/langchain-ai/langchain/tree/master/cookbook/program_aided_language_model.ipynb) | Implement program-aided language models as described in the provided research paper.
-[sales_agent_with_context.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/sales_agent_with_context.ipynb) | Implement a context-aware ai sales agent, salesgpt, that can have natural sales conversations, interact with other systems, and use a product knowledge base to discuss a company's offerings.
-[self_query_hotel_search.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/self_query_hotel_search.ipynb) | Build a hotel room search feature with self-querying retrieval, using a specific hotel recommendation dataset.
-[smart_llm.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/smart_llm.ipynb) | Implement a smartllmchain, a self-critique chain that generates multiple output proposals, critiques them to find the best one, and then improves upon it to produce a final output.
-[tree_of_thought.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/tree_of_thought.ipynb) | Query a large language model using the tree of thought technique.
-[twitter-the-algorithm-analysis...](https://github.com/langchain-ai/langchain/tree/master/cookbook/twitter-the-algorithm-analysis-deeplake.ipynb) | Analyze the source code of the twitter algorithm with the help of gpt4 and activeloop's deep lake.
-[two_agent_debate_tools.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_agent_debate_tools.ipynb) | Simulate multi-agent dialogues where the agents can utilize various tools.
-[two_player_dnd.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_player_dnd.ipynb) | Simulate a two-player dungeons & dragons game, where a dialoguesimulator class is used to coordinate the dialogue between the protagonist and the dungeon master.
-[wikibase_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/wikibase_agent.ipynb) | Create a simple wikibase agent that utilizes sparql generation, with testing done on http://wikidata.org.

cookbook/multi_modal_output_agent.ipynb

@@ -1,188 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "cd835d40",
-   "metadata": {},
-   "source": [
-    "# Multi-modal outputs: Image & Text"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "fa88e03a",
-   "metadata": {},
-   "source": [
-    "This notebook shows how non-text producing tools can be used to create multi-modal agents.\n",
-    "\n",
-    "This example is limited to text and image outputs and uses UUIDs to transfer content across tools and agents. \n",
-    "\n",
-    "This example uses Steamship to generate and store generated images. Generated are auth protected by default. \n",
-    "\n",
-    "You can get your Steamship api key here: https://steamship.com/account/api"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0653da01",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from steamship import Block, Steamship\n",
-    "import re\n",
-    "from IPython.display import Image"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "f6933033",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.llms import OpenAI\n",
-    "from langchain.agents import initialize_agent\n",
-    "from langchain.agents import AgentType\n",
-    "from langchain.tools import SteamshipImageGenerationTool"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "71e51e53",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = OpenAI(temperature=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a9fc769d",
-   "metadata": {},
-   "source": [
-    "## Dall-E "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "cd177dfe",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tools = [SteamshipImageGenerationTool(model_name=\"dall-e\")]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c71b1e46",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "mrkl = initialize_agent(\n",
-    "    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "603aeb9a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "output = mrkl.run(\"How would you visualize a parot playing soccer?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "25eb4efe",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def show_output(output):\n",
-    "    \"\"\"Display the multi-modal output from the agent.\"\"\"\n",
-    "    UUID_PATTERN = re.compile(\n",
-    "        r\"([0-9A-Za-z]{8}-[0-9A-Za-z]{4}-[0-9A-Za-z]{4}-[0-9A-Za-z]{4}-[0-9A-Za-z]{12})\"\n",
-    "    )\n",
-    "\n",
-    "    outputs = UUID_PATTERN.split(output)\n",
-    "    outputs = [\n",
-    "        re.sub(r\"^\\W+\", \"\", el) for el in outputs\n",
-    "    ]  # Clean trailing and leading non-word characters\n",
-    "\n",
-    "    for output in outputs:\n",
-    "        maybe_block_id = UUID_PATTERN.search(output)\n",
-    "        if maybe_block_id:\n",
-    "            display(Image(Block.get(Steamship(), _id=maybe_block_id.group()).raw()))\n",
-    "        else:\n",
-    "            print(output, end=\"\\n\\n\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e247b2c4",
-   "metadata": {},
-   "source": [
-    "## StableDiffusion "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "315025e7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tools = [SteamshipImageGenerationTool(model_name=\"stable-diffusion\")]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "7930064a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "mrkl = initialize_agent(\n",
-    "    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "611a833d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "output = mrkl.run(\"How would you visualize a parot playing soccer?\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/myscale_vector_sql.ipynb

@@ -1,200 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "245065c6",
-   "metadata": {},
-   "source": [
-    "# Vector SQL Retriever with MyScale\n",
-    "\n",
-    ">[MyScale](https://docs.myscale.com/en/) is an integrated vector database. You can access your database in SQL and also from here, LangChain. MyScale can make a use of [various data types and functions for filters](https://blog.myscale.com/2023/06/06/why-integrated-database-solution-can-boost-your-llm-apps/#filter-on-anything-without-constraints). It will boost up your LLM app no matter if you are scaling up your data or expand your system to broader application."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0246c5bf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip3 install clickhouse-sqlalchemy InstructorEmbedding sentence_transformers openai langchain-experimental"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "7585d2c3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\n",
-    "from os import environ\n",
-    "import getpass\n",
-    "from typing import Dict, Any\n",
-    "from langchain.llms import OpenAI\nfrom langchain.utilities import SQLDatabase\nfrom langchain.chains import LLMChain\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
-    "from sqlalchemy import create_engine, Column, MetaData\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "\n",
-    "from sqlalchemy import create_engine\n",
-    "\n",
-    "MYSCALE_HOST = \"msc-1decbcc9.us-east-1.aws.staging.myscale.cloud\"\n",
-    "MYSCALE_PORT = 443\n",
-    "MYSCALE_USER = \"chatdata\"\n",
-    "MYSCALE_PASSWORD = \"myscale_rocks\"\n",
-    "OPENAI_API_KEY = getpass.getpass(\"OpenAI API Key:\")\n",
-    "\n",
-    "engine = create_engine(\n",
-    "    f\"clickhouse://{MYSCALE_USER}:{MYSCALE_PASSWORD}@{MYSCALE_HOST}:{MYSCALE_PORT}/default?protocol=https\"\n",
-    ")\n",
-    "metadata = MetaData(bind=engine)\n",
-    "environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e08d9ddc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.embeddings import HuggingFaceInstructEmbeddings\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLOutputParser\n",
-    "\n",
-    "output_parser = VectorSQLOutputParser.from_embeddings(\n",
-    "    model=HuggingFaceInstructEmbeddings(\n",
-    "        model_name=\"hkunlp/instructor-xl\", model_kwargs={\"device\": \"cpu\"}\n",
-    "    )\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "84b705b2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.callbacks import StdOutCallbackHandler\n",
-    "\n",
-    "from langchain.utilities.sql_database import SQLDatabase\n",
-    "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
-    "\n",
-    "chain = VectorSQLDatabaseChain(\n",
-    "    llm_chain=LLMChain(\n",
-    "        llm=OpenAI(openai_api_key=OPENAI_API_KEY, temperature=0),\n",
-    "        prompt=MYSCALE_PROMPT,\n",
-    "    ),\n",
-    "    top_k=10,\n",
-    "    return_direct=True,\n",
-    "    sql_cmd_parser=output_parser,\n",
-    "    database=SQLDatabase(engine, None, metadata),\n",
-    ")\n",
-    "\n",
-    "import pandas as pd\n",
-    "\n",
-    "pd.DataFrame(\n",
-    "    chain.run(\n",
-    "        \"Please give me 10 papers to ask what is PageRank?\",\n",
-    "        callbacks=[StdOutCallbackHandler()],\n",
-    "    )\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6c09cda0",
-   "metadata": {},
-   "source": [
-    "## SQL Database as Retriever"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "734d7ff5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.chains.qa_with_sources.retrieval import RetrievalQAWithSourcesChain\n",
-    "\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
-    "from langchain_experimental.retrievers.vector_sql_database \\\n",
-    "    import VectorSQLDatabaseChainRetriever\n",
-    "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
-    "from langchain_experimental.sql.vector_sql import VectorSQLRetrieveAllOutputParser\n",
-    "\n",
-    "output_parser_retrieve_all = VectorSQLRetrieveAllOutputParser.from_embeddings(\n",
-    "    output_parser.model\n",
-    ")\n",
-    "\n",
-    "chain = VectorSQLDatabaseChain.from_llm(\n",
-    "    llm=OpenAI(openai_api_key=OPENAI_API_KEY, temperature=0),\n",
-    "    prompt=MYSCALE_PROMPT,\n",
-    "    top_k=10,\n",
-    "    return_direct=True,\n",
-    "    db=SQLDatabase(engine, None, metadata),\n",
-    "    sql_cmd_parser=output_parser_retrieve_all,\n",
-    "    native_format=True,\n",
-    ")\n",
-    "\n",
-    "# You need all those keys to get docs\n",
-    "retriever = VectorSQLDatabaseChainRetriever(sql_db_chain=chain, page_content_key=\"abstract\")\n",
-    "\n",
-    "document_with_metadata_prompt = PromptTemplate(\n",
-    "    input_variables=[\"page_content\", \"id\", \"title\", \"authors\", \"pubdate\", \"categories\"],\n",
-    "    template=\"Content:\\n\\tTitle: {title}\\n\\tAbstract: {page_content}\\n\\tAuthors: {authors}\\n\\tDate of Publication: {pubdate}\\n\\tCategories: {categories}\\nSOURCE: {id}\",\n",
-    ")\n",
-    "\n",
-    "chain = RetrievalQAWithSourcesChain.from_chain_type(\n",
-    "    ChatOpenAI(\n",
-    "        model_name=\"gpt-3.5-turbo-16k\", openai_api_key=OPENAI_API_KEY, temperature=0.6\n",
-    "    ),\n",
-    "    retriever=retriever,\n",
-    "    chain_type=\"stuff\",\n",
-    "    chain_type_kwargs={\n",
-    "        \"document_prompt\": document_with_metadata_prompt,\n",
-    "    },\n",
-    "    return_source_documents=True,\n",
-    ")\n",
-    "ans = chain(\"Please give me 10 papers to ask what is PageRank?\",\n",
-    "            callbacks=[StdOutCallbackHandler()])\n",
-    "print(ans[\"answer\"])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "4948ff25",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/plan_and_execute_agent.ipynb

@@ -1,252 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "0ddfef23-3c74-444c-81dd-6753722997fa",
-   "metadata": {},
-   "source": [
-    "# Plan-and-execute\n",
-    "\n",
-    "Plan-and-execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the [\"Plan-and-Solve\" paper](https://arxiv.org/abs/2305.04091).\n",
-    "\n",
-    "The planning is almost always done by an LLM.\n",
-    "\n",
-    "The execution is usually done by a separate agent (equipped with tools)."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a7ecb22a-7009-48ec-b14e-f0fa5aac1cd0",
-   "metadata": {},
-   "source": [
-    "## Imports"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "5fbbd4ee-bfe8-4a25-afe4-8d1a552a3d2e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.agents.tools import Tool\n",
-    "from langchain.chains import LLMMathChain\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
-    "from langchain_experimental.plan_and_execute import PlanAndExecute, load_agent_executor, load_chat_planner"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e0e995e5-af9d-4988-bcd0-467a2a2e18cd",
-   "metadata": {},
-   "source": [
-    "## Tools"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "1d789f4e-54e3-4602-891a-f076e0ab9594",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "search = DuckDuckGoSearchAPIWrapper()\n",
-    "llm = OpenAI(temperature=0)\n",
-    "llm_math_chain = LLMMathChain.from_llm(llm=llm, verbose=True)\n",
-    "tools = [\n",
-    " Tool(\n",
-    "     name=\"Search\",\n",
-    "     func=search.run,\n",
-    "     description=\"useful for when you need to answer questions about current events\"\n",
-    " ),\n",
-    " Tool(\n",
-    "     name=\"Calculator\",\n",
-    "     func=llm_math_chain.run,\n",
-    "     description=\"useful for when you need to answer questions about math\"\n",
-    " ),\n",
-    "]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "04dc6452-a07f-49f9-be12-95be1e2afccc",
-   "metadata": {},
-   "source": [
-    "## Planner, Executor, and Agent\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "d8f49c03-c804-458b-8122-c92b26c7b7dd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = ChatOpenAI(temperature=0)\n",
-    "planner = load_chat_planner(model)\n",
-    "executor = load_agent_executor(model, tools, verbose=True)\n",
-    "agent = PlanAndExecute(planner=planner, executor=executor)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "78ba03dd-0322-4927-b58d-a7e2027fdbb3",
-   "metadata": {},
-   "source": [
-    "## Run example"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "a57f7efe-7866-47a7-bce5-9c7b1047964e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mAction:\n",
-      "{\n",
-      "  \"action\": \"Search\",\n",
-      "  \"action_input\": \"current prime minister of the UK\"\n",
-      "}\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mAction:\n",
-      "```\n",
-      "{\n",
-      "  \"action\": \"Search\",\n",
-      "  \"action_input\": \"current prime minister of the UK\"\n",
-      "}\n",
-      "```\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mBottom right: Rishi Sunak is the current prime minister and the first non-white prime minister. The prime minister of the United Kingdom is the principal minister of the crown of His Majesty's Government, and the head of the British Cabinet. 3 min. British Prime Minister Rishi Sunak asserted his stance on gender identity in a speech Wednesday, stating it was \"common sense\" that \"a man is a man and a woman is a woman\" — a ... The former chancellor Rishi Sunak is the UK's new prime minister. Here's what you need to know about him. He won after running for the second time this year He lost to Liz Truss in September,... Isaeli Prime Minister Benjamin Netanyahu spoke with US President Joe Biden on Wednesday, the prime minister's office said in a statement. Netanyahu \"thanked the President for the powerful words of ... By Yasmeen Serhan/London Updated: October 25, 2022 12:56 PM EDT | Originally published: October 24, 2022 9:17 AM EDT S top me if you've heard this one before: After a tumultuous period of political...\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mThe search results indicate that Rishi Sunak is the current prime minister of the UK. However, it's important to note that this information may not be accurate or up to date.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mAction:\n",
-      "```\n",
-      "{\n",
-      "  \"action\": \"Search\",\n",
-      "  \"action_input\": \"current age of the prime minister of the UK\"\n",
-      "}\n",
-      "```\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mHow old is Rishi Sunak? Mr Sunak was born on 12 May, 1980, making him 42 years old. He first became an MP in 2015, aged 34, and has served the constituency of Richmond in Yorkshire ever since. He... Prime Ministers' ages when they took office From oldest to youngest, the ages of the PMs were as follows: Winston Churchill - 65 years old James Callaghan - 64 years old Clement Attlee - 62 years... Anna Kaufman USA TODAY Just a few days after Liz Truss resigned as prime minister, the UK has a new prime minister. Truss, who lasted a mere 45 days in office, will be replaced by Rishi... Advertisement Rishi Sunak is the youngest British prime minister of modern times. Mr. Sunak is 42 and started out in Parliament in 2015. Rishi Sunak was appointed as chancellor of the Exchequer... The first prime minister of the current United Kingdom of Great Britain and Northern Ireland upon its effective creation in 1922 (when 26 Irish counties seceded and created the Irish Free State) was Bonar Law, [10] although the country was not renamed officially until 1927, when Stanley Baldwin was the serving prime minister. [11]\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mBased on the search results, it seems that Rishi Sunak is the current prime minister of the UK. However, I couldn't find any specific information about his age. Would you like me to search again for the current age of the prime minister?\n",
-      "\n",
-      "Action:\n",
-      "```\n",
-      "{\n",
-      "  \"action\": \"Search\",\n",
-      "  \"action_input\": \"age of Rishi Sunak\"\n",
-      "}\n",
-      "```\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mRishi Sunak is 42 years old, making him the youngest person to hold the office of prime minister in modern times. How tall is Rishi Sunak? How Old Is Rishi Sunak? Rishi Sunak was born on May 12, 1980, in Southampton, England. Parents and Nationality Sunak's parents were born to Indian-origin families in East Africa before... Born on May 12, 1980, Rishi is currently 42 years old. He has been a member of parliament since 2015 where he was an MP for Richmond and has served in roles including Chief Secretary to the Treasury and the Chancellor of Exchequer while Boris Johnson was PM. Family Murty, 42, is the daughter of the Indian billionaire NR Narayana Murthy, often described as the Bill Gates of India, who founded the software company Infosys. According to reports, his... Sunak became the first non-White person to lead the country and, at age 42, the youngest to take on the role in more than a century. Like most politicians, Sunak is revered by some and...\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mBased on the search results, Rishi Sunak is currently 42 years old. He was born on May 12, 1980.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mThought: To calculate the age raised to the power of 0.43, I can use the calculator tool.\n",
-      "\n",
-      "Action:\n",
-      "```json\n",
-      "{\n",
-      "  \"action\": \"Calculator\",\n",
-      "  \"action_input\": \"42^0.43\"\n",
-      "}\n",
-      "```\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "42^0.43\u001b[32;1m\u001b[1;3m```text\n",
-      "42**0.43\n",
-      "```\n",
-      "...numexpr.evaluate(\"42**0.43\")...\n",
-      "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m4.9888126515157\u001b[0m\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAnswer: 4.9888126515157\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mThe age raised to the power of 0.43 is approximately 4.9888126515157.\n",
-      "\n",
-      "Final Answer:\n",
-      "```json\n",
-      "{\n",
-      "  \"action\": \"Final Answer\",\n",
-      "  \"action_input\": \"The age raised to the power of 0.43 is approximately 4.9888126515157.\"\n",
-      "}\n",
-      "```\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mAction:\n",
-      "```\n",
-      "{\n",
-      "  \"action\": \"Final Answer\",\n",
-      "  \"action_input\": \"The current prime minister of the UK is Rishi Sunak. His age raised to the power of 0.43 is approximately 4.9888126515157.\"\n",
-      "}\n",
-      "```\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'The current prime minister of the UK is Rishi Sunak. His age raised to the power of 0.43 is approximately 4.9888126515157.'"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent.run(\"Who is the current prime minister of the UK? What is their current age raised to the 0.43 power?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0ef78a07-1a2a-46f8-9bc9-ae45f9bd706c",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/smart_llm.ipynb

@@ -1,281 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "9e9b7651",
-   "metadata": {},
-   "source": [
-    "# How to use a SmartLLMChain\n",
-    "\n",
-    "A SmartLLMChain is a form of self-critique chain that can help you if have particularly complex questions to answer. Instead of doing a single LLM pass, it instead performs these 3 steps:\n",
-    "1. Ideation: Pass the user prompt n times through the LLM to get n output proposals (called \"ideas\"), where n is a parameter you can set \n",
-    "2. Critique: The LLM critiques all ideas to find possible flaws and picks the best one \n",
-    "3. Resolve: The LLM tries to improve upon the best idea (as chosen in the critique step) and outputs it. This is then the final output.\n",
-    "\n",
-    "SmartLLMChains are based on the SmartGPT workflow proposed in https://youtu.be/wVzuvf9D9BU.\n",
-    "\n",
-    "Note that SmartLLMChains\n",
-    "- use more LLM passes (ie n+2 instead of just 1)\n",
-    "- only work then the underlying LLM has the capability for reflection, which smaller models often don't\n",
-    "- only work with underlying models that return exactly 1 output, not multiple\n",
-    "\n",
-    "This notebook demonstrates how to use a SmartLLMChain."
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "714dede0",
-   "metadata": {},
-   "source": [
-    "##### Same LLM for all steps"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "d3f7fb22",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"...\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "10e5ece6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain_experimental.smart_llm import SmartLLMChain"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "1780da51",
-   "metadata": {},
-   "source": [
-    "As example question, we will use \"I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it?\". This is an example from the original SmartGPT video (https://youtu.be/wVzuvf9D9BU?t=384). While this seems like a very easy question, LLMs struggle do these kinds of questions that involve numbers and physical reasoning.\n",
-    "\n",
-    "As we will see, all 3 initial ideas are completely wrong - even though we're using GPT4! Only when using self-reflection do we get a correct answer. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "054af6b1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "hard_question = \"I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it?\""
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "8049cecd",
-   "metadata": {},
-   "source": [
-    "So, we first create an LLM and prompt template"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "811ea8e1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = PromptTemplate.from_template(hard_question)\n",
-    "llm = ChatOpenAI(temperature=0, model_name=\"gpt-4\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "50b602e4",
-   "metadata": {},
-   "source": [
-    "Now we can create a SmartLLMChain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "8cd49199",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = SmartLLMChain(llm=llm, prompt=prompt, n_ideas=3, verbose=True)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "6a72f276",
-   "metadata": {},
-   "source": [
-    "Now we can use the SmartLLM as a drop-in replacement for our LLM. E.g.:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "074e5e75",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new SmartLLMChain chain...\u001b[0m\n",
-      "Prompt after formatting:\n",
-      "\u001b[32;1m\u001b[1;3mI have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it?\u001b[0m\n",
-      "Idea 1:\n",
-      "\u001b[36;1m\u001b[1;3m1. Fill the 6-liter jug completely.\n",
-      "2. Pour the water from the 6-liter jug into the 12-liter jug.\n",
-      "3. Fill the 6-liter jug again.\n",
-      "4. Carefully pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full.\n",
-      "5. The amount of water left in the 6-liter jug will be exactly 6 liters.\u001b[0m\n",
-      "Idea 2:\n",
-      "\u001b[36;1m\u001b[1;3m1. Fill the 6-liter jug completely.\n",
-      "2. Pour the water from the 6-liter jug into the 12-liter jug.\n",
-      "3. Fill the 6-liter jug again.\n",
-      "4. Carefully pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full.\n",
-      "5. Since the 12-liter jug is now full, there will be 2 liters of water left in the 6-liter jug.\n",
-      "6. Empty the 12-liter jug.\n",
-      "7. Pour the 2 liters of water from the 6-liter jug into the 12-liter jug.\n",
-      "8. Fill the 6-liter jug completely again.\n",
-      "9. Pour the water from the 6-liter jug into the 12-liter jug, which already has 2 liters in it.\n",
-      "10. Now, the 12-liter jug will have exactly 6 liters of water (2 liters from before + 4 liters from the 6-liter jug).\u001b[0m\n",
-      "Idea 3:\n",
-      "\u001b[36;1m\u001b[1;3m1. Fill the 6-liter jug completely.\n",
-      "2. Pour the water from the 6-liter jug into the 12-liter jug.\n",
-      "3. Fill the 6-liter jug again.\n",
-      "4. Carefully pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full.\n",
-      "5. The amount of water left in the 6-liter jug will be exactly 6 liters.\u001b[0m\n",
-      "Critique:\n",
-      "\u001b[33;1m\u001b[1;3mIdea 1:\n",
-      "1. Fill the 6-liter jug completely. (No flaw)\n",
-      "2. Pour the water from the 6-liter jug into the 12-liter jug. (No flaw)\n",
-      "3. Fill the 6-liter jug again. (No flaw)\n",
-      "4. Carefully pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. (Flaw: The 12-liter jug will never be full in this step, as it can hold 12 liters and we are only pouring 6 liters into it.)\n",
-      "5. The amount of water left in the 6-liter jug will be exactly 6 liters. (Flaw: This statement is incorrect, as there will be no water left in the 6-liter jug after pouring it into the 12-liter jug.)\n",
-      "\n",
-      "Idea 2:\n",
-      "1. Fill the 6-liter jug completely. (No flaw)\n",
-      "2. Pour the water from the 6-liter jug into the 12-liter jug. (No flaw)\n",
-      "3. Fill the 6-liter jug again. (No flaw)\n",
-      "4. Carefully pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. (Flaw: The 12-liter jug will never be full in this step, as it can hold 12 liters and we are only pouring 6 liters into it.)\n",
-      "5. Since the 12-liter jug is now full, there will be 2 liters of water left in the 6-liter jug. (Flaw: This statement is incorrect, as the 12-liter jug will not be full and there will be no water left in the 6-liter jug.)\n",
-      "6. Empty the 12-liter jug. (No flaw)\n",
-      "7. Pour the 2 liters of water from the 6-liter jug into the 12-liter jug. (Flaw: This step is based on the incorrect assumption that there are 2 liters of water left in the 6-liter jug.)\n",
-      "8. Fill the 6-liter jug completely again. (No flaw)\n",
-      "9. Pour the water from the 6-liter jug into the 12-liter jug, which already has 2 liters in it. (Flaw: This step is based on the incorrect assumption that there are 2 liters of water in the 12-liter jug.)\n",
-      "10. Now, the 12-liter jug will have exactly 6 liters of water (2 liters from before + 4 liters from the 6-liter jug). (Flaw: This conclusion is based on the incorrect assumptions made in the previous steps.)\n",
-      "\n",
-      "Idea 3:\n",
-      "1. Fill the 6-liter jug completely. (No flaw)\n",
-      "2. Pour the water from the 6-liter jug into the 12-liter jug. (No flaw)\n",
-      "3. Fill the 6-liter jug again. (No flaw)\n",
-      "4. Carefully pour the water from the 6-liter jug into the 12-liter jug until the 12-liter jug is full. (Flaw: The 12-liter jug will never be full in this step, as it can hold 12 liters and we are only pouring 6 liters into it.)\n",
-      "5. The amount of water left in the 6-liter jug will be exactly 6 liters. (Flaw: This statement is incorrect, as there will be no water left in the 6-liter jug after pouring it into the 12-liter jug.)\u001b[0m\n",
-      "Resolution:\n",
-      "\u001b[32;1m\u001b[1;3m1. Fill the 12-liter jug completely.\n",
-      "2. Pour the water from the 12-liter jug into the 6-liter jug until the 6-liter jug is full.\n",
-      "3. The amount of water left in the 12-liter jug will be exactly 6 liters.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'1. Fill the 12-liter jug completely.\\n2. Pour the water from the 12-liter jug into the 6-liter jug until the 6-liter jug is full.\\n3. The amount of water left in the 12-liter jug will be exactly 6 liters.'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.run({})"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "bbfebea1",
-   "metadata": {},
-   "source": [
-    "##### Different LLM for different steps"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "5be6ec08",
-   "metadata": {},
-   "source": [
-    "You can also use different LLMs for the different steps by passing `ideation_llm`, `critique_llm` and `resolve_llm`. You might want to do this to use a more creative (i.e., high-temperature) model for ideation and a more strict (i.e., low-temperature) model for critique and resolution."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "9c33fa19",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = SmartLLMChain(\n",
-    "    ideation_llm=ChatOpenAI(temperature=0.9, model_name=\"gpt-4\"),\n",
-    "    llm=ChatOpenAI(\n",
-    "        temperature=0, model_name=\"gpt-4\"\n",
-    "    ),  # will be used for critique and resolution as no specific llms are given\n",
-    "    prompt=prompt,\n",
-    "    n_ideas=3,\n",
-    "    verbose=True,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "886c1cc1",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/tree_of_thought.ipynb

@@ -1,239 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Tree of Thought (ToT) example\n",
-    "\n",
-    "The Tree of Thought (ToT) is a chain that allows you to query a Large Language Model (LLM) using the Tree of Thought technique. This is based on the paper [\"Large Language Model Guided Tree-of-Thought\"](https://arxiv.org/pdf/2305.08291.pdf)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/harrisonchase/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.13) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.\n",
-      "  warnings.warn(\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.llms import OpenAI\n",
-    "\n",
-    "llm = OpenAI(temperature=1, max_tokens=512, model=\"text-davinci-003\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "3,*,*,2|1,*,3,*|*,1,*,3|4,*,*,1\n",
-      "\n",
-      "- This is a 4x4 Sudoku puzzle.\n",
-      "- The * represents a cell to be filled.\n",
-      "- The | character separates rows.\n",
-      "- At each step, replace one or more * with digits 1-4.\n",
-      "- There must be no duplicate digits in any row, column or 2x2 subgrid.\n",
-      "- Keep the known digits from previous valid thoughts in place.\n",
-      "- Each thought can be a partial or the final solution.\n"
-     ]
-    }
-   ],
-   "source": [
-    "sudoku_puzzle =   \"3,*,*,2|1,*,3,*|*,1,*,3|4,*,*,1\"\n",
-    "sudoku_solution = \"3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1\"\n",
-    "problem_description = f\"\"\"\n",
-    "{sudoku_puzzle}\n",
-    "\n",
-    "- This is a 4x4 Sudoku puzzle.\n",
-    "- The * represents a cell to be filled.\n",
-    "- The | character separates rows.\n",
-    "- At each step, replace one or more * with digits 1-4.\n",
-    "- There must be no duplicate digits in any row, column or 2x2 subgrid.\n",
-    "- Keep the known digits from previous valid thoughts in place.\n",
-    "- Each thought can be a partial or the final solution.\n",
-    "\"\"\".strip()\n",
-    "print(problem_description)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Rules Based Checker\n",
-    "\n",
-    "Each thought is evaluated by the thought checker and is given a validity type: valid, invalid or partial. A simple checker can be rule based. For example, in the case of a sudoku puzzle, the checker can check if the puzzle is valid, invalid or partial.\n",
-    "\n",
-    "In the following code we implement a simple rule based checker for a specific 4x4 sudoku puzzle.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import Tuple\n",
-    "from langchain_experimental.tot.checker import ToTChecker\n",
-    "from langchain_experimental.tot.thought import ThoughtValidity\n",
-    "import re\n",
-    "\n",
-    "class MyChecker(ToTChecker):\n",
-    "    def evaluate(self, problem_description: str, thoughts: Tuple[str, ...] = ()) -> ThoughtValidity:\n",
-    "        last_thought = thoughts[-1]\n",
-    "        clean_solution = last_thought.replace(\" \", \"\").replace('\"', \"\")\n",
-    "        regex_solution = clean_solution.replace(\"*\", \".\").replace(\"|\", \"\\\\|\")\n",
-    "        if sudoku_solution in clean_solution:\n",
-    "            return ThoughtValidity.VALID_FINAL\n",
-    "        elif re.search(regex_solution, sudoku_solution):\n",
-    "            return ThoughtValidity.VALID_INTERMEDIATE\n",
-    "        else:\n",
-    "            return ThoughtValidity.INVALID"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Just testing the MyChecker class above:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "checker = MyChecker()\n",
-    "assert checker.evaluate(\"\", (\"3,*,*,2|1,*,3,*|*,1,*,3|4,*,*,1\",)) == ThoughtValidity.VALID_INTERMEDIATE\n",
-    "assert checker.evaluate(\"\", (\"3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1\",)) == ThoughtValidity.VALID_FINAL\n",
-    "assert checker.evaluate(\"\", (\"3,4,1,2|1,2,3,4|2,1,4,3|4,3,*,1\",)) == ThoughtValidity.VALID_INTERMEDIATE\n",
-    "assert checker.evaluate(\"\", (\"3,4,1,2|1,2,3,4|2,1,4,3|4,*,3,1\",)) == ThoughtValidity.INVALID"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Tree of Thought Chain\n",
-    "\n",
-    "Initialize and run the ToT chain, with maximum number of interactions `k` set to `30` and the maximum number child thoughts `c` set to `8`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new ToTChain chain...\u001b[0m\n",
-      "Starting the ToT solve procedure.\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/harrisonchase/workplace/langchain/libs/langchain/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
-      "  warnings.warn(\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[31;1m\u001b[1;3mThought: 3*,*,2|1*,3,*|*,1,*,3|4,*,*,1\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3*,1,2|1*,3,*|*,1,*,3|4,*,*,1\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3*,1,2|1*,3,4|*,1,*,3|4,*,*,1\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3*,1,2|1*,3,4|*,1,2,3|4,*,*,1\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3*,1,2|1*,3,4|2,1,*,3|4,*,*,1\n",
-      "\u001b[0m"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Type <enum 'ThoughtValidity'> not serializable\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[31;1m\u001b[1;3mThought: 3,*,*,2|1,*,3,*|*,1,*,3|4,1,*,*\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3,*,*,2|*,3,2,*|*,1,*,3|4,1,*,*\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3,2,*,2|1,*,3,*|*,1,*,3|4,1,*,*\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3,2,*,2|1,*,3,*|1,1,*,3|4,1,*,*\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3mThought: 3,2,*,2|1,1,3,*|1,1,*,3|4,1,*,*\n",
-      "\u001b[0m\u001b[33;1m\u001b[1;3mThought: 3,*,*,2|1,2,3,*|*,1,*,3|4,*,*,1\n",
-      "\u001b[0m\u001b[31;1m\u001b[1;3m    Thought: 3,1,4,2|1,2,3,4|2,1,4,3|4,3,2,1\n",
-      "\u001b[0m\u001b[32;1m\u001b[1;3m    Thought: 3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1\n",
-      "\u001b[0m\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'3,4,1,2|1,2,3,4|2,1,4,3|4,3,2,1'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain_experimental.tot.base import ToTChain\n",
-    "\n",
-    "tot_chain = ToTChain(llm=llm, checker=MyChecker(), k=30, c=5, verbose=True, verbose_llm=False)\n",
-    "tot_chain.run(problem_description=problem_description)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docker/Dockerfile.base

@@ -1,3 +0,0 @@
-FROM python:3.11
-
-RUN pip install langchain

docs/.local_build.sh

@@ -8,11 +8,10 @@ set -o xtrace
 SCRIPT_DIR="$(cd "$(dirname "$0")"; pwd)"
 cd "${SCRIPT_DIR}"
 
-mkdir -p ../_dist
-cp -r . ../_dist
-cd ../_dist
-poetry run python scripts/model_feat_table.py
-poetry run nbdoc_build --srcdir docs
-poetry run python scripts/generate_api_reference_links.py
+mkdir -p _dist/docs_skeleton
+cp -r {docs_skeleton,snippets} _dist
+cp -r extras/* _dist/docs_skeleton/docs
+cd _dist/docs_skeleton
+poetry run nbdoc_build
 yarn install
 yarn start

docs/api_reference/Makefile

@@ -3,7 +3,7 @@
 
 # You can set these variables from the command line, and also
 # from the environment for the first two.
-SPHINXOPTS    ?= -j auto 
+SPHINXOPTS    ?= 
 SPHINXBUILD   ?= sphinx-build
 SPHINXAUTOBUILD   ?= sphinx-autobuild
 SOURCEDIR     = .

docs/api_reference/conf.py

@@ -23,7 +23,6 @@ from sphinx.util.docutils import SphinxDirective
 _DIR = Path(__file__).parent.absolute()
 sys.path.insert(0, os.path.abspath("."))
 sys.path.insert(0, os.path.abspath("../../libs/langchain"))
-sys.path.insert(0, os.path.abspath("../../libs/experimental"))
 
 with (_DIR.parents[1] / "libs" / "langchain" / "pyproject.toml").open("r") as f:
     data = toml.load(f)
@@ -46,8 +45,8 @@ class ExampleLinksDirective(SphinxDirective):
 
         Called any time :example_links:`ClassName` is used
         in the template *.rst files."""
-        class_or_func_name = self.arguments[0]
-        links = imported_classes.get(class_or_func_name, {})
+        class_name = self.arguments[0]
+        links = imported_classes.get(class_name, {})
         list_node = nodes.bullet_list()
         for doc_name, link in links.items():
             item_node = nodes.list_item()
@@ -58,10 +57,7 @@ class ExampleLinksDirective(SphinxDirective):
             para_node.append(link_node)
             item_node.append(para_node)
             list_node.append(item_node)
-        if list_node.children:
-            title_node = nodes.title()
-            title_node.append(nodes.Text(f"Examples using {class_or_func_name}"))
-            return [title_node, list_node]
+
         return [list_node]
 
 
@@ -100,9 +96,6 @@ extensions = [
 ]
 source_suffix = [".rst"]
 
-# some autodoc pydantic options are repeated in the actual template.
-# potentially user error, but there may be bugs in the sphinx extension
-# with options not being passed through correctly (from either the location in the code)
 autodoc_pydantic_model_show_json = False
 autodoc_pydantic_field_list_validators = False
 autodoc_pydantic_config_members = False
@@ -115,6 +108,13 @@ autodoc_member_order = "groupwise"
 autoclass_content = "both"
 autodoc_typehints_format = "short"
 
+autodoc_default_options = {
+    "members": True,
+    "show-inheritance": True,
+    "inherited-members": "BaseModel",
+    "undoc-members": True,
+    "special-members": "__call__",
+}
 # autodoc_typehints = "description"
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["templates"]
@@ -156,7 +156,7 @@ html_context = {
 html_static_path = ["_static"]
 
 # These paths are either relative to html_static_path
-# or fully qualified paths (e.g. https://...)
+# or fully qualified paths (eg. https://...)
 html_css_files = [
     "css/custom.css",
 ]

docs/api_reference/create_api_rst.py

@@ -1,278 +1,83 @@
-"""Script for auto-generating api_reference.rst."""
-import importlib
-import inspect
-import typing
+"""Script for auto-generating api_reference.rst"""
+import glob
+import re
 from pathlib import Path
-from typing import TypedDict, Sequence, List, Dict, Literal, Union, Optional
-from enum import Enum
-
-from pydantic import BaseModel
 
 ROOT_DIR = Path(__file__).parents[2].absolute()
-HERE = Path(__file__).parent
-
 PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
-EXP_DIR = ROOT_DIR / "libs" / "experimental" / "langchain_experimental"
-WRITE_FILE = HERE / "api_reference.rst"
-EXP_WRITE_FILE = HERE / "experimental_api_reference.rst"
+WRITE_FILE = Path(__file__).parent / "api_reference.rst"
 
 
-ClassKind = Literal["TypedDict", "Regular", "Pydantic", "enum"]
+def load_members() -> dict:
+    members: dict = {}
+    for py in glob.glob(str(PKG_DIR) + "/**/*.py", recursive=True):
+        module = py[len(str(PKG_DIR)) + 1 :].replace(".py", "").replace("/", ".")
+        top_level = module.split(".")[0]
+        if top_level not in members:
+            members[top_level] = {"classes": [], "functions": []}
+        with open(py, "r") as f:
+            for line in f.readlines():
+                cls = re.findall(r"^class ([^_].*)\(", line)
+                members[top_level]["classes"].extend([module + "." + c for c in cls])
+                func = re.findall(r"^def ([^_].*)\(", line)
+                afunc = re.findall(r"^async def ([^_].*)\(", line)
+                func_strings = [module + "." + f for f in func + afunc]
+                members[top_level]["functions"].extend(func_strings)
+    return members
 
 
-class ClassInfo(TypedDict):
-    """Information about a class."""
+def construct_doc(members: dict) -> str:
+    full_doc = """\
+.. _api_reference:
 
-    name: str
-    """The name of the class."""
-    qualified_name: str
-    """The fully qualified name of the class."""
-    kind: ClassKind
-    """The kind of the class."""
-    is_public: bool
-    """Whether the class is public or not."""
-
-
-class FunctionInfo(TypedDict):
-    """Information about a function."""
-
-    name: str
-    """The name of the function."""
-    qualified_name: str
-    """The fully qualified name of the function."""
-    is_public: bool
-    """Whether the function is public or not."""
-
-
-class ModuleMembers(TypedDict):
-    """A dictionary of module members."""
-
-    classes_: Sequence[ClassInfo]
-    functions: Sequence[FunctionInfo]
-
-
-def _load_module_members(module_path: str, namespace: str) -> ModuleMembers:
-    """Load all members of a module.
-
-    Args:
-        module_path: Path to the module.
-        namespace: the namespace of the module.
-
-    Returns:
-        list: A list of loaded module objects.
-    """
-    classes_: List[ClassInfo] = []
-    functions: List[FunctionInfo] = []
-    module = importlib.import_module(module_path)
-    for name, type_ in inspect.getmembers(module):
-        if not hasattr(type_, "__module__"):
-            continue
-        if type_.__module__ != module_path:
-            continue
-
-        if inspect.isclass(type_):
-            if type(type_) == typing._TypedDictMeta:  # type: ignore
-                kind: ClassKind = "TypedDict"
-            elif issubclass(type_, Enum):
-                kind = "enum"
-            elif issubclass(type_, BaseModel):
-                kind = "Pydantic"
-            else:
-                kind = "Regular"
-
-            classes_.append(
-                ClassInfo(
-                    name=name,
-                    qualified_name=f"{namespace}.{name}",
-                    kind=kind,
-                    is_public=not name.startswith("_"),
-                )
-            )
-        elif inspect.isfunction(type_):
-            functions.append(
-                FunctionInfo(
-                    name=name,
-                    qualified_name=f"{namespace}.{name}",
-                    is_public=not name.startswith("_"),
-                )
-            )
-        else:
-            continue
-
-    return ModuleMembers(
-        classes_=classes_,
-        functions=functions,
-    )
-
-
-def _merge_module_members(
-    module_members: Sequence[ModuleMembers],
-) -> ModuleMembers:
-    """Merge module members."""
-    classes_: List[ClassInfo] = []
-    functions: List[FunctionInfo] = []
-    for module in module_members:
-        classes_.extend(module["classes_"])
-        functions.extend(module["functions"])
-
-    return ModuleMembers(
-        classes_=classes_,
-        functions=functions,
-    )
-
-
-def _load_package_modules(
-    package_directory: Union[str, Path], submodule: Optional[str] = None
-) -> Dict[str, ModuleMembers]:
-    """Recursively load modules of a package based on the file system.
-
-    Traversal based on the file system makes it easy to determine which
-    of the modules/packages are part of the package vs. 3rd party or built-in.
-
-    Parameters:
-        package_directory: Path to the package directory.
-        submodule: Optional name of submodule to load.
-
-    Returns:
-        list: A list of loaded module objects.
-    """
-    package_path = (
-        Path(package_directory)
-        if isinstance(package_directory, str)
-        else package_directory
-    )
-    modules_by_namespace = {}
-
-    # Get the high level package name
-    package_name = package_path.name
-
-    # If we are loading a submodule, add it in
-    if submodule is not None:
-        package_path = package_path / submodule
-
-    for file_path in package_path.rglob("*.py"):
-        if file_path.name.startswith("_"):
-            continue
-
-        relative_module_name = file_path.relative_to(package_path)
-
-        # Skip if any module part starts with an underscore
-        if any(part.startswith("_") for part in relative_module_name.parts):
-            continue
-
-        # Get the full namespace of the module
-        namespace = str(relative_module_name).replace(".py", "").replace("/", ".")
-        # Keep only the top level namespace
-        top_namespace = namespace.split(".")[0]
-
-        try:
-            # If submodule is present, we need to construct the paths in a slightly
-            # different way
-            if submodule is not None:
-                module_members = _load_module_members(
-                    f"{package_name}.{submodule}.{namespace}",
-                    f"{submodule}.{namespace}",
-                )
-            else:
-                module_members = _load_module_members(
-                    f"{package_name}.{namespace}", namespace
-                )
-            # Merge module members if the namespace already exists
-            if top_namespace in modules_by_namespace:
-                existing_module_members = modules_by_namespace[top_namespace]
-                _module_members = _merge_module_members(
-                    [existing_module_members, module_members]
-                )
-            else:
-                _module_members = module_members
-
-            modules_by_namespace[top_namespace] = _module_members
-
-        except ImportError as e:
-            print(f"Error: Unable to import module '{namespace}' with error: {e}")
-
-    return modules_by_namespace
-
-
-def _construct_doc(pkg: str, members_by_namespace: Dict[str, ModuleMembers]) -> str:
-    """Construct the contents of the reference.rst file for the given package.
-
-    Args:
-        pkg: The package name
-        members_by_namespace: The members of the package, dict organized by top level
-                              module contains a list of classes and functions
-                              inside of the top level namespace.
-
-    Returns:
-        The contents of the reference.rst file.
-    """
-    full_doc = f"""\
-=======================
-``{pkg}`` API Reference
-=======================
+=============
+API Reference
+=============
 
 """
-    namespaces = sorted(members_by_namespace)
-
-    for module in namespaces:
-        _members = members_by_namespace[module]
-        classes = _members["classes_"]
+    for module, _members in sorted(members.items(), key=lambda kv: kv[0]):
+        classes = _members["classes"]
         functions = _members["functions"]
         if not (classes or functions):
             continue
-        section = f":mod:`{pkg}.{module}`"
-        underline = "=" * (len(section) + 1)
+
+        module_title = module.replace("_", " ").title()
+        if module_title == "Llms":
+            module_title = "LLMs"
+        section = f":mod:`langchain.{module}`: {module_title}"
         full_doc += f"""\
 {section}
-{underline}
+{'=' * (len(section) + 1)}
 
-.. automodule:: {pkg}.{module}
+.. automodule:: langchain.{module}
     :no-members:
     :no-inherited-members:
 
 """
 
         if classes:
+            cstring = "\n    ".join(sorted(classes))
             full_doc += f"""\
 Classes
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
+    :template: class.rst
+
+    {cstring}
+
 """
-
-            for class_ in sorted(classes, key=lambda c: c["qualified_name"]):
-                if not class_["is_public"]:
-                    continue
-
-                if class_["kind"] == "TypedDict":
-                    template = "typeddict.rst"
-                elif class_["kind"] == "enum":
-                    template = "enum.rst"
-                elif class_["kind"] == "Pydantic":
-                    template = "pydantic.rst"
-                else:
-                    template = "class.rst"
-
-                full_doc += f"""\
-    :template: {template}
-    
-    {class_["qualified_name"]}
-    
-"""
-
         if functions:
-            _functions = [f["qualified_name"] for f in functions if f["is_public"]]
-            fstring = "\n    ".join(sorted(_functions))
+            fstring = "\n    ".join(sorted(functions))
             full_doc += f"""\
 Functions
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
-    :template: function.rst
 
     {fstring}
 
@@ -280,46 +85,11 @@ Functions
     return full_doc
 
 
-def _document_langchain_experimental() -> None:
-    """Document the langchain_experimental package."""
-    # Generate experimental_api_reference.rst
-    exp_members = _load_package_modules(EXP_DIR)
-    exp_doc = ".. _experimental_api_reference:\n\n" + _construct_doc(
-        "langchain_experimental", exp_members
-    )
-    with open(EXP_WRITE_FILE, "w") as f:
-        f.write(exp_doc)
-
-
-def _document_langchain_core() -> None:
-    """Document the main langchain package."""
-    # load top level module members
-    lc_members = _load_package_modules(PKG_DIR)
-
-    # Add additional packages
-    tools = _load_package_modules(PKG_DIR, "tools")
-    agents = _load_package_modules(PKG_DIR, "agents")
-    schema = _load_package_modules(PKG_DIR, "schema")
-
-    lc_members.update(
-        {
-            "agents.output_parsers": agents["output_parsers"],
-            "agents.format_scratchpad": agents["format_scratchpad"],
-            "tools.render": tools["render"],
-            "schema.runnable": schema["runnable"],
-        }
-    )
-
-    lc_doc = ".. _api_reference:\n\n" + _construct_doc("langchain", lc_members)
-
-    with open(WRITE_FILE, "w") as f:
-        f.write(lc_doc)
-
-
 def main() -> None:
-    """Generate the reference.rst file for each package."""
-    _document_langchain_core()
-    _document_langchain_experimental()
+    members = load_members()
+    full_doc = construct_doc(members)
+    with open(WRITE_FILE, "w") as f:
+        f.write(full_doc)
 
 
 if __name__ == "__main__":

docs/api_reference/requirements.txt

@@ -1,6 +1,4 @@
 -e libs/langchain
--e libs/experimental
-pydantic<2
 autodoc_pydantic==1.8.0
 myst_parser
 nbsphinx==0.8.9
@@ -12,4 +10,4 @@ sphinx-panels
 toml
 myst_nb
 sphinx_copybutton
-pydata-sphinx-theme==0.13.1
+pydata-sphinx-theme==0.13.1

docs/api_reference/templates/class.rst

@@ -5,6 +5,17 @@
 
 .. autoclass:: {{ objname }}
 
+   {% block methods %}
+   {% if methods %}
+   .. rubric:: {{ _('Methods') }}
+
+   .. autosummary::
+   {% for item in methods %}
+      ~{{ name }}.{{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
    {% block attributes %}
    {% if attributes %}
    .. rubric:: {{ _('Attributes') }}
@@ -16,21 +27,13 @@
    {% endif %}
    {% endblock %}
 
-   {% block methods %}
-   {% if methods %}
-   .. rubric:: {{ _('Methods') }}
+   {% if objname in imported_classes %}
+   Examples using this class
+   --------------------------
 
-   .. autosummary::
-   {% for item in methods %}
-      ~{{ name }}.{{ item }}
+   {% for example in imported_classes[objname] %}
+   * `Example <{{ example }}>`_
    {%- endfor %}
-
-   {% for item in methods %}
-   .. automethod:: {{ name }}.{{ item }}
-   {%- endfor %}
-
    {% endif %}
-   {% endblock %}
-
 
 .. example_links:: {{ objname }}

docs/api_reference/templates/enum.rst

@@ -1,14 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autoclass:: {{ objname }}
-
-    {% block attributes %}
-    {% for item in attributes %}
-    .. autoattribute:: {{ item }}
-    {% endfor %}
-    {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/templates/function.rst

@@ -1,8 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autofunction:: {{ objname }}
-
-.. example_links:: {{ objname }}

docs/api_reference/templates/pydantic.rst

@@ -1,22 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autopydantic_model:: {{ objname }}
-    :model-show-json: False
-    :model-show-config-summary: False
-    :model-show-validator-members: False
-    :model-show-field-summary: False
-    :field-signature-prefix: param
-    :members:
-    :undoc-members:
-    :inherited-members:
-    :member-order: groupwise
-    :show-inheritance: True
-    :special-members: __call__
-
-    {% block attributes %}
-    {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/templates/redirects.html

@@ -5,10 +5,9 @@
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <meta http-equiv="Refresh" content="0; url={{ redirect }}" />
-    <meta name="robots" content="follow, index">
-    <meta name="Description" content="Python API reference for LangChain.">
+    <meta name="Description" content="scikit-learn: machine learning in Python">
     <link rel="canonical" href="{{ redirect }}" />
-    <title>LangChain Python API Reference Documentation.</title>
+    <title>scikit-learn: machine learning in Python</title>
   </head>
   <body>
     <p>You will be automatically redirected to the <a href="{{ redirect }}">new location of this page</a>.</p>

docs/api_reference/templates/typeddict.rst

@@ -1,14 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autoclass:: {{ objname }}
-
-    {% block attributes %}
-   {% for item in attributes %}
-  .. autoattribute:: {{ item }}
-   {% endfor %}
-   {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/themes/scikit-learn-modern/layout.html

@@ -19,7 +19,7 @@
   {% block htmltitle %}
   <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
   {% endblock %}
-  <link rel="canonical" href="https://api.python.langchain.com/en/latest/{{pagename}}.html" />
+  <link rel="canonical" href="http://scikit-learn.org/stable/{{pagename}}.html" />
 
   {% if favicon_url %}
   <link rel="shortcut icon" href="{{ favicon_url|e }}"/>

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -6,6 +6,17 @@
   {%- set top_container_cls = "sk-landing-container" %}
 {%- endif %}
 
+{% if theme_link_to_live_contributing_page|tobool %}
+{# Link to development page for live builds #}
+  {%- set development_link = "https://scikit-learn.org/dev/developers/index.html" %}
+{# Open on a new development page in new window/tab for live builds #}
+  {%- set development_attrs = 'target="_blank" rel="noopener noreferrer"' %}
+{%- else %}
+  {%- set development_link = pathto('developers/index') %}
+  {%- set development_attrs = '' %}
+{%- endif %}
+
+
 <nav id="navbar" class="{{ nav_bar_class }} navbar navbar-expand-md navbar-light bg-light py-0">
   <div class="container-fluid {{ top_container_cls }} px-0">
     {%- if logo_url %}
@@ -34,9 +45,6 @@
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
         </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('experimental_api_reference') }}">Experimental</a>
-        </li>
         <li class="nav-item">
           <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
         </li>

docs/api_reference/themes/scikit-learn-modern/static/css/theme.css

@@ -745,11 +745,6 @@ span.descname {
   background-color: transparent;
   padding: 0;
   font-family: monospace;
-  font-size: 1.2rem;
-}
-
-em.property {
-  font-weight: normal;
 }
 
 span.descclassname {

docs/docs/additional_resources/dependents.mdx

@@ -1,465 +0,0 @@
-# Dependents
-
-Dependents stats for `langchain-ai/langchain`
-
-[![](https://img.shields.io/static/v1?label=Used%20by&message=30534&color=informational&logo=slickpic)](https://github.com/langchain-ai/langchain/network/dependents)
-[![](https://img.shields.io/static/v1?label=Used%20by%20(public)&message=451&color=informational&logo=slickpic)](https://github.com/langchain-ai/langchain/network/dependents)
-[![](https://img.shields.io/static/v1?label=Used%20by%20(private)&message=30083&color=informational&logo=slickpic)](https://github.com/langchain-ai/langchain/network/dependents)
-[![](https://img.shields.io/static/v1?label=Used%20by%20(stars)&message=37822&color=informational&logo=slickpic)](https://github.com/langchain-ai/langchain/network/dependents)
-
-
-[update: `2023-10-06`; only dependent repositories with Stars > 100]
-
-
-| Repository | Stars  |
-| :--------  | -----: |
-|[openai/openai-cookbook](https://github.com/openai/openai-cookbook) | 49006 |
-|[AntonOsika/gpt-engineer](https://github.com/AntonOsika/gpt-engineer) | 44368 |
-|[imartinez/privateGPT](https://github.com/imartinez/privateGPT) | 38300 |
-|[LAION-AI/Open-Assistant](https://github.com/LAION-AI/Open-Assistant) | 35327 |
-|[hpcaitech/ColossalAI](https://github.com/hpcaitech/ColossalAI) | 34799 |
-|[microsoft/TaskMatrix](https://github.com/microsoft/TaskMatrix) | 34161 |
-|[streamlit/streamlit](https://github.com/streamlit/streamlit) | 27697 |
-|[geekan/MetaGPT](https://github.com/geekan/MetaGPT) | 27302 |
-|[reworkd/AgentGPT](https://github.com/reworkd/AgentGPT) | 26805 |
-|[OpenBB-finance/OpenBBTerminal](https://github.com/OpenBB-finance/OpenBBTerminal) | 24473 |
-|[StanGirard/quivr](https://github.com/StanGirard/quivr) | 23323 |
-|[run-llama/llama_index](https://github.com/run-llama/llama_index) | 22151 |
-|[openai/chatgpt-retrieval-plugin](https://github.com/openai/chatgpt-retrieval-plugin) | 19741 |
-|[mindsdb/mindsdb](https://github.com/mindsdb/mindsdb) | 18062 |
-|[PromtEngineer/localGPT](https://github.com/PromtEngineer/localGPT) | 16413 |
-|[chatchat-space/Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat) | 16300 |
-|[cube-js/cube](https://github.com/cube-js/cube) | 16261 |
-|[mlflow/mlflow](https://github.com/mlflow/mlflow) | 15487 |
-|[logspace-ai/langflow](https://github.com/logspace-ai/langflow) | 12599 |
-|[GaiZhenbiao/ChuanhuChatGPT](https://github.com/GaiZhenbiao/ChuanhuChatGPT) | 12501 |
-|[openai/evals](https://github.com/openai/evals) | 12056 |
-|[airbytehq/airbyte](https://github.com/airbytehq/airbyte) | 11919 |
-|[go-skynet/LocalAI](https://github.com/go-skynet/LocalAI) | 11767 |
-|[databrickslabs/dolly](https://github.com/databrickslabs/dolly) | 10609 |
-|[AIGC-Audio/AudioGPT](https://github.com/AIGC-Audio/AudioGPT) | 9240 |
-|[aws/amazon-sagemaker-examples](https://github.com/aws/amazon-sagemaker-examples) | 8892 |
-|[langgenius/dify](https://github.com/langgenius/dify) | 8764 |
-|[gventuri/pandas-ai](https://github.com/gventuri/pandas-ai) | 8687 |
-|[jmorganca/ollama](https://github.com/jmorganca/ollama) | 8628 |
-|[langchain-ai/langchainjs](https://github.com/langchain-ai/langchainjs) | 8392 |
-|[h2oai/h2ogpt](https://github.com/h2oai/h2ogpt) | 7953 |
-|[arc53/DocsGPT](https://github.com/arc53/DocsGPT) | 7730 |
-|[PipedreamHQ/pipedream](https://github.com/PipedreamHQ/pipedream) | 7261 |
-|[joshpxyne/gpt-migrate](https://github.com/joshpxyne/gpt-migrate) | 6349 |
-|[bentoml/OpenLLM](https://github.com/bentoml/OpenLLM) | 6213 |
-|[mage-ai/mage-ai](https://github.com/mage-ai/mage-ai) | 5600 |
-|[zauberzeug/nicegui](https://github.com/zauberzeug/nicegui) | 5499 |
-|[wenda-LLM/wenda](https://github.com/wenda-LLM/wenda) | 5497 |
-|[sweepai/sweep](https://github.com/sweepai/sweep) | 5489 |
-|[embedchain/embedchain](https://github.com/embedchain/embedchain) | 5428 |
-|[zilliztech/GPTCache](https://github.com/zilliztech/GPTCache) | 5311 |
-|[Shaunwei/RealChar](https://github.com/Shaunwei/RealChar) | 5264 |
-|[GreyDGL/PentestGPT](https://github.com/GreyDGL/PentestGPT) | 5146 |
-|[gkamradt/langchain-tutorials](https://github.com/gkamradt/langchain-tutorials) | 5134 |
-|[serge-chat/serge](https://github.com/serge-chat/serge) | 5009 |
-|[assafelovic/gpt-researcher](https://github.com/assafelovic/gpt-researcher) | 4836 |
-|[openchatai/OpenChat](https://github.com/openchatai/OpenChat) | 4697 |
-|[intel-analytics/BigDL](https://github.com/intel-analytics/BigDL) | 4412 |
-|[continuedev/continue](https://github.com/continuedev/continue) | 4324 |
-|[postgresml/postgresml](https://github.com/postgresml/postgresml) | 4267 |
-|[madawei2699/myGPTReader](https://github.com/madawei2699/myGPTReader) | 4214 |
-|[MineDojo/Voyager](https://github.com/MineDojo/Voyager) | 4204 |
-|[danswer-ai/danswer](https://github.com/danswer-ai/danswer) | 3973 |
-|[RayVentura/ShortGPT](https://github.com/RayVentura/ShortGPT) | 3922 |
-|[Azure/azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) | 3849 |
-|[khoj-ai/khoj](https://github.com/khoj-ai/khoj) | 3817 |
-|[langchain-ai/chat-langchain](https://github.com/langchain-ai/chat-langchain) | 3742 |
-|[Azure-Samples/azure-search-openai-demo](https://github.com/Azure-Samples/azure-search-openai-demo) | 3731 |
-|[marqo-ai/marqo](https://github.com/marqo-ai/marqo) | 3627 |
-|[kyegomez/tree-of-thoughts](https://github.com/kyegomez/tree-of-thoughts) | 3553 |
-|[llm-workflow-engine/llm-workflow-engine](https://github.com/llm-workflow-engine/llm-workflow-engine) | 3483 |
-|[PrefectHQ/marvin](https://github.com/PrefectHQ/marvin) | 3460 |
-|[aiwaves-cn/agents](https://github.com/aiwaves-cn/agents) | 3413 |
-|[OpenBMB/ToolBench](https://github.com/OpenBMB/ToolBench) | 3388 |
-|[shroominic/codeinterpreter-api](https://github.com/shroominic/codeinterpreter-api) | 3218 |
-|[whitead/paper-qa](https://github.com/whitead/paper-qa) | 3085 |
-|[project-baize/baize-chatbot](https://github.com/project-baize/baize-chatbot) | 3039 |
-|[OpenGVLab/InternGPT](https://github.com/OpenGVLab/InternGPT) | 2911 |
-|[ParisNeo/lollms-webui](https://github.com/ParisNeo/lollms-webui) | 2907 |
-|[Unstructured-IO/unstructured](https://github.com/Unstructured-IO/unstructured) | 2874 |
-|[openchatai/OpenCopilot](https://github.com/openchatai/OpenCopilot) | 2759 |
-|[OpenBMB/BMTools](https://github.com/OpenBMB/BMTools) | 2657 |
-|[homanp/superagent](https://github.com/homanp/superagent) | 2624 |
-|[SamurAIGPT/EmbedAI](https://github.com/SamurAIGPT/EmbedAI) | 2575 |
-|[GerevAI/gerev](https://github.com/GerevAI/gerev) | 2488 |
-|[microsoft/promptflow](https://github.com/microsoft/promptflow) | 2475 |
-|[OpenBMB/AgentVerse](https://github.com/OpenBMB/AgentVerse) | 2445 |
-|[Mintplex-Labs/anything-llm](https://github.com/Mintplex-Labs/anything-llm) | 2434 |
-|[emptycrown/llama-hub](https://github.com/emptycrown/llama-hub) | 2432 |
-|[NVIDIA/NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) | 2327 |
-|[ShreyaR/guardrails](https://github.com/ShreyaR/guardrails) | 2307 |
-|[thomas-yanxin/LangChain-ChatGLM-Webui](https://github.com/thomas-yanxin/LangChain-ChatGLM-Webui) | 2305 |
-|[yanqiangmiffy/Chinese-LangChain](https://github.com/yanqiangmiffy/Chinese-LangChain) | 2291 |
-|[keephq/keep](https://github.com/keephq/keep) | 2252 |
-|[OpenGVLab/Ask-Anything](https://github.com/OpenGVLab/Ask-Anything) | 2194 |
-|[IntelligenzaArtificiale/Free-Auto-GPT](https://github.com/IntelligenzaArtificiale/Free-Auto-GPT) | 2169 |
-|[Farama-Foundation/PettingZoo](https://github.com/Farama-Foundation/PettingZoo) | 2031 |
-|[YiVal/YiVal](https://github.com/YiVal/YiVal) | 2014 |
-|[hwchase17/notion-qa](https://github.com/hwchase17/notion-qa) | 2014 |
-|[jupyterlab/jupyter-ai](https://github.com/jupyterlab/jupyter-ai) | 1977 |
-|[paulpierre/RasaGPT](https://github.com/paulpierre/RasaGPT) | 1887 |
-|[dot-agent/dotagent-WIP](https://github.com/dot-agent/dotagent-WIP) | 1812 |
-|[hegelai/prompttools](https://github.com/hegelai/prompttools) | 1775 |
-|[vocodedev/vocode-python](https://github.com/vocodedev/vocode-python) | 1734 |
-|[Vonng/pigsty](https://github.com/Vonng/pigsty) | 1693 |
-|[psychic-api/psychic](https://github.com/psychic-api/psychic) | 1597 |
-|[avinashkranjan/Amazing-Python-Scripts](https://github.com/avinashkranjan/Amazing-Python-Scripts) | 1546 |
-|[pinterest/querybook](https://github.com/pinterest/querybook) | 1539 |
-|[Forethought-Technologies/AutoChain](https://github.com/Forethought-Technologies/AutoChain) | 1531 |
-|[Kav-K/GPTDiscord](https://github.com/Kav-K/GPTDiscord) | 1503 |
-|[jina-ai/langchain-serve](https://github.com/jina-ai/langchain-serve) | 1487 |
-|[noahshinn024/reflexion](https://github.com/noahshinn024/reflexion) | 1481 |
-|[jina-ai/dev-gpt](https://github.com/jina-ai/dev-gpt) | 1436 |
-|[ttengwang/Caption-Anything](https://github.com/ttengwang/Caption-Anything) | 1425 |
-|[milvus-io/bootcamp](https://github.com/milvus-io/bootcamp) | 1420 |
-|[agiresearch/OpenAGI](https://github.com/agiresearch/OpenAGI) | 1401 |
-|[greshake/llm-security](https://github.com/greshake/llm-security) | 1381 |
-|[jina-ai/thinkgpt](https://github.com/jina-ai/thinkgpt) | 1366 |
-|[lunasec-io/lunasec](https://github.com/lunasec-io/lunasec) | 1352 |
-|[101dotxyz/GPTeam](https://github.com/101dotxyz/GPTeam) | 1339 |
-|[refuel-ai/autolabel](https://github.com/refuel-ai/autolabel) | 1320 |
-|[melih-unsal/DemoGPT](https://github.com/melih-unsal/DemoGPT) | 1320 |
-|[mmz-001/knowledge_gpt](https://github.com/mmz-001/knowledge_gpt) | 1320 |
-|[richardyc/Chrome-GPT](https://github.com/richardyc/Chrome-GPT) | 1315 |
-|[run-llama/sec-insights](https://github.com/run-llama/sec-insights) | 1312 |
-|[Azure/azureml-examples](https://github.com/Azure/azureml-examples) | 1305 |
-|[cofactoryai/textbase](https://github.com/cofactoryai/textbase) | 1286 |
-|[dataelement/bisheng](https://github.com/dataelement/bisheng) | 1273 |
-|[eyurtsev/kor](https://github.com/eyurtsev/kor) | 1263 |
-|[pluralsh/plural](https://github.com/pluralsh/plural) | 1188 |
-|[FlagOpen/FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) | 1184 |
-|[juncongmoo/chatllama](https://github.com/juncongmoo/chatllama) | 1144 |
-|[poe-platform/server-bot-quick-start](https://github.com/poe-platform/server-bot-quick-start) | 1139 |
-|[visual-openllm/visual-openllm](https://github.com/visual-openllm/visual-openllm) | 1137 |
-|[griptape-ai/griptape](https://github.com/griptape-ai/griptape) | 1124 |
-|[microsoft/X-Decoder](https://github.com/microsoft/X-Decoder) | 1119 |
-|[ThousandBirdsInc/chidori](https://github.com/ThousandBirdsInc/chidori) | 1116 |
-|[filip-michalsky/SalesGPT](https://github.com/filip-michalsky/SalesGPT) | 1112 |
-|[psychic-api/rag-stack](https://github.com/psychic-api/rag-stack) | 1110 |
-|[irgolic/AutoPR](https://github.com/irgolic/AutoPR) | 1100 |
-|[promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 1099 |
-|[nod-ai/SHARK](https://github.com/nod-ai/SHARK) | 1062 |
-|[SamurAIGPT/Camel-AutoGPT](https://github.com/SamurAIGPT/Camel-AutoGPT) | 1036 |
-|[Farama-Foundation/chatarena](https://github.com/Farama-Foundation/chatarena) | 1020 |
-|[peterw/Chat-with-Github-Repo](https://github.com/peterw/Chat-with-Github-Repo) | 993 |
-|[jiran214/GPT-vup](https://github.com/jiran214/GPT-vup) | 967 |
-|[alejandro-ao/ask-multiple-pdfs](https://github.com/alejandro-ao/ask-multiple-pdfs) | 958 |
-|[run-llama/llama-lab](https://github.com/run-llama/llama-lab) | 953 |
-|[LC1332/Chat-Haruhi-Suzumiya](https://github.com/LC1332/Chat-Haruhi-Suzumiya) | 950 |
-|[rlancemartin/auto-evaluator](https://github.com/rlancemartin/auto-evaluator) | 927 |
-|[cheshire-cat-ai/core](https://github.com/cheshire-cat-ai/core) | 902 |
-|[Anil-matcha/ChatPDF](https://github.com/Anil-matcha/ChatPDF) | 894 |
-|[cirediatpl/FigmaChain](https://github.com/cirediatpl/FigmaChain) | 881 |
-|[seanpixel/Teenage-AGI](https://github.com/seanpixel/Teenage-AGI) | 876 |
-|[xusenlinzy/api-for-open-llm](https://github.com/xusenlinzy/api-for-open-llm) | 865 |
-|[ricklamers/shell-ai](https://github.com/ricklamers/shell-ai) | 864 |
-|[codeacme17/examor](https://github.com/codeacme17/examor) | 856 |
-|[corca-ai/EVAL](https://github.com/corca-ai/EVAL) | 836 |
-|[microsoft/Llama-2-Onnx](https://github.com/microsoft/Llama-2-Onnx) | 835 |
-|[explodinggradients/ragas](https://github.com/explodinggradients/ragas) | 833 |
-|[ajndkr/lanarky](https://github.com/ajndkr/lanarky) | 817 |
-|[kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference](https://github.com/kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference) | 814 |
-|[ray-project/llm-applications](https://github.com/ray-project/llm-applications) | 804 |
-|[hwchase17/chat-your-data](https://github.com/hwchase17/chat-your-data) | 801 |
-|[LambdaLabsML/examples](https://github.com/LambdaLabsML/examples) | 759 |
-|[kreneskyp/ix](https://github.com/kreneskyp/ix) | 758 |
-|[pyspark-ai/pyspark-ai](https://github.com/pyspark-ai/pyspark-ai) | 750 |
-|[billxbf/ReWOO](https://github.com/billxbf/ReWOO) | 746 |
-|[e-johnstonn/BriefGPT](https://github.com/e-johnstonn/BriefGPT) | 738 |
-|[akshata29/entaoai](https://github.com/akshata29/entaoai) | 733 |
-|[getmetal/motorhead](https://github.com/getmetal/motorhead) | 717 |
-|[ruoccofabrizio/azure-open-ai-embeddings-qna](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) | 712 |
-|[msoedov/langcorn](https://github.com/msoedov/langcorn) | 698 |
-|[Dataherald/dataherald](https://github.com/Dataherald/dataherald) | 684 |
-|[jondurbin/airoboros](https://github.com/jondurbin/airoboros) | 657 |
-|[Ikaros-521/AI-Vtuber](https://github.com/Ikaros-521/AI-Vtuber) | 651 |
-|[whyiyhw/chatgpt-wechat](https://github.com/whyiyhw/chatgpt-wechat) | 644 |
-|[langchain-ai/streamlit-agent](https://github.com/langchain-ai/streamlit-agent) | 637 |
-|[SamurAIGPT/ChatGPT-Developer-Plugins](https://github.com/SamurAIGPT/ChatGPT-Developer-Plugins) | 637 |
-|[OpenGenerativeAI/GenossGPT](https://github.com/OpenGenerativeAI/GenossGPT) | 632 |
-|[AILab-CVC/GPT4Tools](https://github.com/AILab-CVC/GPT4Tools) | 629 |
-|[langchain-ai/auto-evaluator](https://github.com/langchain-ai/auto-evaluator) | 614 |
-|[explosion/spacy-llm](https://github.com/explosion/spacy-llm) | 613 |
-|[alexanderatallah/window.ai](https://github.com/alexanderatallah/window.ai) | 607 |
-|[MiuLab/Taiwan-LLaMa](https://github.com/MiuLab/Taiwan-LLaMa) | 601 |
-|[microsoft/PodcastCopilot](https://github.com/microsoft/PodcastCopilot) | 600 |
-|[Dicklesworthstone/swiss_army_llama](https://github.com/Dicklesworthstone/swiss_army_llama) | 596 |
-|[NoDataFound/hackGPT](https://github.com/NoDataFound/hackGPT) | 596 |
-|[namuan/dr-doc-search](https://github.com/namuan/dr-doc-search) | 593 |
-|[amosjyng/langchain-visualizer](https://github.com/amosjyng/langchain-visualizer) | 582 |
-|[microsoft/sample-app-aoai-chatGPT](https://github.com/microsoft/sample-app-aoai-chatGPT) | 581 |
-|[yvann-hub/Robby-chatbot](https://github.com/yvann-hub/Robby-chatbot) | 581 |
-|[yeagerai/yeagerai-agent](https://github.com/yeagerai/yeagerai-agent) | 547 |
-|[tgscan-dev/tgscan](https://github.com/tgscan-dev/tgscan) | 533 |
-|[Azure-Samples/openai](https://github.com/Azure-Samples/openai) | 531 |
-|[plastic-labs/tutor-gpt](https://github.com/plastic-labs/tutor-gpt) | 531 |
-|[xuwenhao/geektime-ai-course](https://github.com/xuwenhao/geektime-ai-course) | 526 |
-|[michaelthwan/searchGPT](https://github.com/michaelthwan/searchGPT) | 526 |
-|[jonra1993/fastapi-alembic-sqlmodel-async](https://github.com/jonra1993/fastapi-alembic-sqlmodel-async) | 522 |
-|[jina-ai/agentchain](https://github.com/jina-ai/agentchain) | 519 |
-|[mckaywrigley/repo-chat](https://github.com/mckaywrigley/repo-chat) | 518 |
-|[modelscope/modelscope-agent](https://github.com/modelscope/modelscope-agent) | 512 |
-|[daveebbelaar/langchain-experiments](https://github.com/daveebbelaar/langchain-experiments) | 504 |
-|[freddyaboulton/gradio-tools](https://github.com/freddyaboulton/gradio-tools) | 497 |
-|[sidhq/Multi-GPT](https://github.com/sidhq/Multi-GPT) | 494 |
-|[continuum-llms/chatgpt-memory](https://github.com/continuum-llms/chatgpt-memory) | 489 |
-|[langchain-ai/langchain-aiplugin](https://github.com/langchain-ai/langchain-aiplugin) | 487 |
-|[mpaepper/content-chatbot](https://github.com/mpaepper/content-chatbot) | 483 |
-|[steamship-core/steamship-langchain](https://github.com/steamship-core/steamship-langchain) | 481 |
-|[alejandro-ao/langchain-ask-pdf](https://github.com/alejandro-ao/langchain-ask-pdf) | 474 |
-|[truera/trulens](https://github.com/truera/trulens) | 464 |
-|[marella/chatdocs](https://github.com/marella/chatdocs) | 459 |
-|[opencopilotdev/opencopilot](https://github.com/opencopilotdev/opencopilot) | 453 |
-|[poe-platform/poe-protocol](https://github.com/poe-platform/poe-protocol) | 444 |
-|[DataDog/dd-trace-py](https://github.com/DataDog/dd-trace-py) | 441 |
-|[logan-markewich/llama_index_starter_pack](https://github.com/logan-markewich/llama_index_starter_pack) | 441 |
-|[opentensor/bittensor](https://github.com/opentensor/bittensor) | 433 |
-|[DjangoPeng/openai-quickstart](https://github.com/DjangoPeng/openai-quickstart) | 425 |
-|[CarperAI/OpenELM](https://github.com/CarperAI/OpenELM) | 424 |
-|[daodao97/chatdoc](https://github.com/daodao97/chatdoc) | 423 |
-|[showlab/VLog](https://github.com/showlab/VLog) | 411 |
-|[Anil-matcha/Chatbase](https://github.com/Anil-matcha/Chatbase) | 402 |
-|[yakami129/VirtualWife](https://github.com/yakami129/VirtualWife) | 399 |
-|[wandb/weave](https://github.com/wandb/weave) | 399 |
-|[mtenenholtz/chat-twitter](https://github.com/mtenenholtz/chat-twitter) | 398 |
-|[LinkSoul-AI/AutoAgents](https://github.com/LinkSoul-AI/AutoAgents) | 397 |
-|[Agenta-AI/agenta](https://github.com/Agenta-AI/agenta) | 389 |
-|[huchenxucs/ChatDB](https://github.com/huchenxucs/ChatDB) | 386 |
-|[mallorbc/Finetune_LLMs](https://github.com/mallorbc/Finetune_LLMs) | 379 |
-|[junruxiong/IncarnaMind](https://github.com/junruxiong/IncarnaMind) | 372 |
-|[MagnivOrg/prompt-layer-library](https://github.com/MagnivOrg/prompt-layer-library) | 368 |
-|[mosaicml/examples](https://github.com/mosaicml/examples) | 366 |
-|[rsaryev/talk-codebase](https://github.com/rsaryev/talk-codebase) | 364 |
-|[morpheuslord/GPT_Vuln-analyzer](https://github.com/morpheuslord/GPT_Vuln-analyzer) | 362 |
-|[monarch-initiative/ontogpt](https://github.com/monarch-initiative/ontogpt) | 362 |
-|[JayZeeDesign/researcher-gpt](https://github.com/JayZeeDesign/researcher-gpt) | 361 |
-|[personoids/personoids-lite](https://github.com/personoids/personoids-lite) | 361 |
-|[intel/intel-extension-for-transformers](https://github.com/intel/intel-extension-for-transformers) | 357 |
-|[jerlendds/osintbuddy](https://github.com/jerlendds/osintbuddy) | 357 |
-|[steamship-packages/langchain-production-starter](https://github.com/steamship-packages/langchain-production-starter) | 356 |
-|[onlyphantom/llm-python](https://github.com/onlyphantom/llm-python) | 354 |
-|[Azure-Samples/miyagi](https://github.com/Azure-Samples/miyagi) | 340 |
-|[mrwadams/attackgen](https://github.com/mrwadams/attackgen) | 338 |
-|[rgomezcasas/dotfiles](https://github.com/rgomezcasas/dotfiles) | 337 |
-|[eosphoros-ai/DB-GPT-Hub](https://github.com/eosphoros-ai/DB-GPT-Hub) | 336 |
-|[andylokandy/gpt-4-search](https://github.com/andylokandy/gpt-4-search) | 335 |
-|[NimbleBoxAI/ChainFury](https://github.com/NimbleBoxAI/ChainFury) | 330 |
-|[momegas/megabots](https://github.com/momegas/megabots) | 329 |
-|[Nuggt-dev/Nuggt](https://github.com/Nuggt-dev/Nuggt) | 315 |
-|[itamargol/openai](https://github.com/itamargol/openai) | 315 |
-|[BlackHC/llm-strategy](https://github.com/BlackHC/llm-strategy) | 315 |
-|[aws-samples/aws-genai-llm-chatbot](https://github.com/aws-samples/aws-genai-llm-chatbot) | 312 |
-|[Cheems-Seminar/grounded-segment-any-parts](https://github.com/Cheems-Seminar/grounded-segment-any-parts) | 312 |
-|[preset-io/promptimize](https://github.com/preset-io/promptimize) | 311 |
-|[dgarnitz/vectorflow](https://github.com/dgarnitz/vectorflow) | 309 |
-|[langchain-ai/langsmith-cookbook](https://github.com/langchain-ai/langsmith-cookbook) | 309 |
-|[CambioML/pykoi](https://github.com/CambioML/pykoi) | 309 |
-|[wandb/edu](https://github.com/wandb/edu) | 301 |
-|[XzaiCloud/luna-ai](https://github.com/XzaiCloud/luna-ai) | 300 |
-|[liangwq/Chatglm_lora_multi-gpu](https://github.com/liangwq/Chatglm_lora_multi-gpu) | 294 |
-|[Haste171/langchain-chatbot](https://github.com/Haste171/langchain-chatbot) | 291 |
-|[sullivan-sean/chat-langchainjs](https://github.com/sullivan-sean/chat-langchainjs) | 286 |
-|[sugarforever/LangChain-Tutorials](https://github.com/sugarforever/LangChain-Tutorials) | 285 |
-|[facebookresearch/personal-timeline](https://github.com/facebookresearch/personal-timeline) | 283 |
-|[hnawaz007/pythondataanalysis](https://github.com/hnawaz007/pythondataanalysis) | 282 |
-|[yuanjie-ai/ChatLLM](https://github.com/yuanjie-ai/ChatLLM) | 280 |
-|[MetaGLM/FinGLM](https://github.com/MetaGLM/FinGLM) | 279 |
-|[JohnSnowLabs/langtest](https://github.com/JohnSnowLabs/langtest) | 277 |
-|[Em1tSan/NeuroGPT](https://github.com/Em1tSan/NeuroGPT) | 274 |
-|[Safiullah-Rahu/CSV-AI](https://github.com/Safiullah-Rahu/CSV-AI) | 274 |
-|[conceptofmind/toolformer](https://github.com/conceptofmind/toolformer) | 274 |
-|[airobotlab/KoChatGPT](https://github.com/airobotlab/KoChatGPT) | 266 |
-|[gia-guar/JARVIS-ChatGPT](https://github.com/gia-guar/JARVIS-ChatGPT) | 263 |
-|[Mintplex-Labs/vector-admin](https://github.com/Mintplex-Labs/vector-admin) | 262 |
-|[artitw/text2text](https://github.com/artitw/text2text) | 262 |
-|[kaarthik108/snowChat](https://github.com/kaarthik108/snowChat) | 261 |
-|[paolorechia/learn-langchain](https://github.com/paolorechia/learn-langchain) | 260 |
-|[shamspias/customizable-gpt-chatbot](https://github.com/shamspias/customizable-gpt-chatbot) | 260 |
-|[ur-whitelab/exmol](https://github.com/ur-whitelab/exmol) | 258 |
-|[hwchase17/chroma-langchain](https://github.com/hwchase17/chroma-langchain) | 257 |
-|[bborn/howdoi.ai](https://github.com/bborn/howdoi.ai) | 255 |
-|[ur-whitelab/chemcrow-public](https://github.com/ur-whitelab/chemcrow-public) | 253 |
-|[pablomarin/GPT-Azure-Search-Engine](https://github.com/pablomarin/GPT-Azure-Search-Engine) | 251 |
-|[gustavz/DataChad](https://github.com/gustavz/DataChad) | 249 |
-|[radi-cho/datasetGPT](https://github.com/radi-cho/datasetGPT) | 249 |
-|[ennucore/clippinator](https://github.com/ennucore/clippinator) | 247 |
-|[recalign/RecAlign](https://github.com/recalign/RecAlign) | 244 |
-|[lilacai/lilac](https://github.com/lilacai/lilac) | 243 |
-|[kaleido-lab/dolphin](https://github.com/kaleido-lab/dolphin) | 236 |
-|[iusztinpaul/hands-on-llms](https://github.com/iusztinpaul/hands-on-llms) | 233 |
-|[PradipNichite/Youtube-Tutorials](https://github.com/PradipNichite/Youtube-Tutorials) | 231 |
-|[shaman-ai/agent-actors](https://github.com/shaman-ai/agent-actors) | 231 |
-|[hwchase17/langchain-streamlit-template](https://github.com/hwchase17/langchain-streamlit-template) | 231 |
-|[yym68686/ChatGPT-Telegram-Bot](https://github.com/yym68686/ChatGPT-Telegram-Bot) | 226 |
-|[grumpyp/aixplora](https://github.com/grumpyp/aixplora) | 222 |
-|[su77ungr/CASALIOY](https://github.com/su77ungr/CASALIOY) | 222 |
-|[alvarosevilla95/autolang](https://github.com/alvarosevilla95/autolang) | 222 |
-|[arthur-ai/bench](https://github.com/arthur-ai/bench) | 220 |
-|[miaoshouai/miaoshouai-assistant](https://github.com/miaoshouai/miaoshouai-assistant) | 219 |
-|[AutoPackAI/beebot](https://github.com/AutoPackAI/beebot) | 217 |
-|[edreisMD/plugnplai](https://github.com/edreisMD/plugnplai) | 216 |
-|[nicknochnack/LangchainDocuments](https://github.com/nicknochnack/LangchainDocuments) | 214 |
-|[AkshitIreddy/Interactive-LLM-Powered-NPCs](https://github.com/AkshitIreddy/Interactive-LLM-Powered-NPCs) | 213 |
-|[SpecterOps/Nemesis](https://github.com/SpecterOps/Nemesis) | 210 |
-|[kyegomez/swarms](https://github.com/kyegomez/swarms) | 210 |
-|[wpydcr/LLM-Kit](https://github.com/wpydcr/LLM-Kit) | 208 |
-|[orgexyz/BlockAGI](https://github.com/orgexyz/BlockAGI) | 204 |
-|[Chainlit/cookbook](https://github.com/Chainlit/cookbook) | 202 |
-|[WongSaang/chatgpt-ui-server](https://github.com/WongSaang/chatgpt-ui-server) | 202 |
-|[jbrukh/gpt-jargon](https://github.com/jbrukh/gpt-jargon) | 202 |
-|[handrew/browserpilot](https://github.com/handrew/browserpilot) | 202 |
-|[langchain-ai/web-explorer](https://github.com/langchain-ai/web-explorer) | 200 |
-|[plchld/InsightFlow](https://github.com/plchld/InsightFlow) | 200 |
-|[alphasecio/langchain-examples](https://github.com/alphasecio/langchain-examples) | 199 |
-|[Gentopia-AI/Gentopia](https://github.com/Gentopia-AI/Gentopia) | 198 |
-|[SamPink/dev-gpt](https://github.com/SamPink/dev-gpt) | 196 |
-|[yasyf/compress-gpt](https://github.com/yasyf/compress-gpt) | 196 |
-|[benthecoder/ClassGPT](https://github.com/benthecoder/ClassGPT) | 195 |
-|[voxel51/voxelgpt](https://github.com/voxel51/voxelgpt) | 193 |
-|[CL-lau/SQL-GPT](https://github.com/CL-lau/SQL-GPT) | 192 |
-|[blob42/Instrukt](https://github.com/blob42/Instrukt) | 191 |
-|[streamlit/llm-examples](https://github.com/streamlit/llm-examples) | 191 |
-|[stepanogil/autonomous-hr-chatbot](https://github.com/stepanogil/autonomous-hr-chatbot) | 190 |
-|[TsinghuaDatabaseGroup/DB-GPT](https://github.com/TsinghuaDatabaseGroup/DB-GPT) | 189 |
-|[PJLab-ADG/DriveLikeAHuman](https://github.com/PJLab-ADG/DriveLikeAHuman) | 187 |
-|[Azure-Samples/azure-search-power-skills](https://github.com/Azure-Samples/azure-search-power-skills) | 187 |
-|[microsoft/azure-openai-in-a-day-workshop](https://github.com/microsoft/azure-openai-in-a-day-workshop) | 187 |
-|[ju-bezdek/langchain-decorators](https://github.com/ju-bezdek/langchain-decorators) | 182 |
-|[hardbyte/qabot](https://github.com/hardbyte/qabot) | 181 |
-|[hongbo-miao/hongbomiao.com](https://github.com/hongbo-miao/hongbomiao.com) | 180 |
-|[QwenLM/Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) | 179 |
-|[showlab/UniVTG](https://github.com/showlab/UniVTG) | 179 |
-|[Azure-Samples/jp-azureopenai-samples](https://github.com/Azure-Samples/jp-azureopenai-samples) | 176 |
-|[afaqueumer/DocQA](https://github.com/afaqueumer/DocQA) | 174 |
-|[ethanyanjiali/minChatGPT](https://github.com/ethanyanjiali/minChatGPT) | 174 |
-|[shauryr/S2QA](https://github.com/shauryr/S2QA) | 174 |
-|[RoboCoachTechnologies/GPT-Synthesizer](https://github.com/RoboCoachTechnologies/GPT-Synthesizer) | 173 |
-|[chakkaradeep/pyCodeAGI](https://github.com/chakkaradeep/pyCodeAGI) | 172 |
-|[vaibkumr/prompt-optimizer](https://github.com/vaibkumr/prompt-optimizer) | 171 |
-|[ccurme/yolopandas](https://github.com/ccurme/yolopandas) | 170 |
-|[anarchy-ai/LLM-VM](https://github.com/anarchy-ai/LLM-VM) | 169 |
-|[ray-project/langchain-ray](https://github.com/ray-project/langchain-ray) | 169 |
-|[fengyuli-dev/multimedia-gpt](https://github.com/fengyuli-dev/multimedia-gpt) | 169 |
-|[ibiscp/LLM-IMDB](https://github.com/ibiscp/LLM-IMDB) | 168 |
-|[mayooear/private-chatbot-mpt30b-langchain](https://github.com/mayooear/private-chatbot-mpt30b-langchain) | 167 |
-|[OpenPluginACI/openplugin](https://github.com/OpenPluginACI/openplugin) | 165 |
-|[jmpaz/promptlib](https://github.com/jmpaz/promptlib) | 165 |
-|[kjappelbaum/gptchem](https://github.com/kjappelbaum/gptchem) | 162 |
-|[JorisdeJong123/7-Days-of-LangChain](https://github.com/JorisdeJong123/7-Days-of-LangChain) | 161 |
-|[retr0reg/Ret2GPT](https://github.com/retr0reg/Ret2GPT) | 161 |
-|[menloparklab/falcon-langchain](https://github.com/menloparklab/falcon-langchain) | 159 |
-|[summarizepaper/summarizepaper](https://github.com/summarizepaper/summarizepaper) | 158 |
-|[emarco177/ice_breaker](https://github.com/emarco177/ice_breaker) | 157 |
-|[AmineDiro/cria](https://github.com/AmineDiro/cria) | 156 |
-|[morpheuslord/HackBot](https://github.com/morpheuslord/HackBot) | 156 |
-|[homanp/vercel-langchain](https://github.com/homanp/vercel-langchain) | 156 |
-|[mlops-for-all/mlops-for-all.github.io](https://github.com/mlops-for-all/mlops-for-all.github.io) | 155 |
-|[positive666/Prompt-Can-Anything](https://github.com/positive666/Prompt-Can-Anything) | 154 |
-|[deeppavlov/dream](https://github.com/deeppavlov/dream) | 153 |
-|[flurb18/AgentOoba](https://github.com/flurb18/AgentOoba) | 151 |
-|[Open-Swarm-Net/GPT-Swarm](https://github.com/Open-Swarm-Net/GPT-Swarm) | 151 |
-|[v7labs/benchllm](https://github.com/v7labs/benchllm) | 150 |
-|[Klingefjord/chatgpt-telegram](https://github.com/Klingefjord/chatgpt-telegram) | 150 |
-|[Aggregate-Intellect/sherpa](https://github.com/Aggregate-Intellect/sherpa) | 148 |
-|[Coding-Crashkurse/Langchain-Full-Course](https://github.com/Coding-Crashkurse/Langchain-Full-Course) | 148 |
-|[SuperDuperDB/superduperdb](https://github.com/SuperDuperDB/superduperdb) | 147 |
-|[defenseunicorns/leapfrogai](https://github.com/defenseunicorns/leapfrogai) | 147 |
-|[menloparklab/langchain-cohere-qdrant-doc-retrieval](https://github.com/menloparklab/langchain-cohere-qdrant-doc-retrieval) | 147 |
-|[Jaseci-Labs/jaseci](https://github.com/Jaseci-Labs/jaseci) | 146 |
-|[realminchoi/babyagi-ui](https://github.com/realminchoi/babyagi-ui) | 146 |
-|[iMagist486/ElasticSearch-Langchain-Chatglm2](https://github.com/iMagist486/ElasticSearch-Langchain-Chatglm2) | 144 |
-|[peterw/StoryStorm](https://github.com/peterw/StoryStorm) | 143 |
-|[kulltc/chatgpt-sql](https://github.com/kulltc/chatgpt-sql) | 142 |
-|[Teahouse-Studios/akari-bot](https://github.com/Teahouse-Studios/akari-bot) | 142 |
-|[hirokidaichi/wanna](https://github.com/hirokidaichi/wanna) | 141 |
-|[yasyf/summ](https://github.com/yasyf/summ) | 141 |
-|[solana-labs/chatgpt-plugin](https://github.com/solana-labs/chatgpt-plugin) | 140 |
-|[ssheng/BentoChain](https://github.com/ssheng/BentoChain) | 139 |
-|[mallahyari/drqa](https://github.com/mallahyari/drqa) | 139 |
-|[petehunt/langchain-github-bot](https://github.com/petehunt/langchain-github-bot) | 139 |
-|[dbpunk-labs/octogen](https://github.com/dbpunk-labs/octogen) | 138 |
-|[RedisVentures/redis-openai-qna](https://github.com/RedisVentures/redis-openai-qna) | 138 |
-|[eunomia-bpf/GPTtrace](https://github.com/eunomia-bpf/GPTtrace) | 138 |
-|[langchain-ai/langsmith-sdk](https://github.com/langchain-ai/langsmith-sdk) | 137 |
-|[jina-ai/fastapi-serve](https://github.com/jina-ai/fastapi-serve) | 137 |
-|[yeagerai/genworlds](https://github.com/yeagerai/genworlds) | 137 |
-|[aurelio-labs/arxiv-bot](https://github.com/aurelio-labs/arxiv-bot) | 137 |
-|[luisroque/large_laguage_models](https://github.com/luisroque/large_laguage_models) | 136 |
-|[ChuloAI/BrainChulo](https://github.com/ChuloAI/BrainChulo) | 136 |
-|[3Alan/DocsMind](https://github.com/3Alan/DocsMind) | 136 |
-|[KylinC/ChatFinance](https://github.com/KylinC/ChatFinance) | 133 |
-|[langchain-ai/text-split-explorer](https://github.com/langchain-ai/text-split-explorer) | 133 |
-|[davila7/file-gpt](https://github.com/davila7/file-gpt) | 133 |
-|[tencentmusic/supersonic](https://github.com/tencentmusic/supersonic) | 132 |
-|[kimtth/azure-openai-llm-vector-langchain](https://github.com/kimtth/azure-openai-llm-vector-langchain) | 131 |
-|[ciare-robotics/world-creator](https://github.com/ciare-robotics/world-creator) | 129 |
-|[zenml-io/zenml-projects](https://github.com/zenml-io/zenml-projects) | 129 |
-|[log1stics/voice-generator-webui](https://github.com/log1stics/voice-generator-webui) | 129 |
-|[snexus/llm-search](https://github.com/snexus/llm-search) | 129 |
-|[fixie-ai/fixie-examples](https://github.com/fixie-ai/fixie-examples) | 128 |
-|[MedalCollector/Orator](https://github.com/MedalCollector/Orator) | 127 |
-|[grumpyp/chroma-langchain-tutorial](https://github.com/grumpyp/chroma-langchain-tutorial) | 127 |
-|[langchain-ai/langchain-aws-template](https://github.com/langchain-ai/langchain-aws-template) | 127 |
-|[prof-frink-lab/slangchain](https://github.com/prof-frink-lab/slangchain) | 126 |
-|[KMnO4-zx/huanhuan-chat](https://github.com/KMnO4-zx/huanhuan-chat) | 124 |
-|[RCGAI/SimplyRetrieve](https://github.com/RCGAI/SimplyRetrieve) | 124 |
-|[Dicklesworthstone/llama2_aided_tesseract](https://github.com/Dicklesworthstone/llama2_aided_tesseract) | 123 |
-|[sdaaron/QueryGPT](https://github.com/sdaaron/QueryGPT) | 122 |
-|[athina-ai/athina-sdk](https://github.com/athina-ai/athina-sdk) | 121 |
-|[AIAnytime/Llama2-Medical-Chatbot](https://github.com/AIAnytime/Llama2-Medical-Chatbot) | 121 |
-|[MuhammadMoinFaisal/LargeLanguageModelsProjects](https://github.com/MuhammadMoinFaisal/LargeLanguageModelsProjects) | 121 |
-|[Azure/business-process-automation](https://github.com/Azure/business-process-automation) | 121 |
-|[definitive-io/code-indexer-loop](https://github.com/definitive-io/code-indexer-loop) | 119 |
-|[nrl-ai/pautobot](https://github.com/nrl-ai/pautobot) | 119 |
-|[Azure/app-service-linux-docs](https://github.com/Azure/app-service-linux-docs) | 118 |
-|[zilliztech/akcio](https://github.com/zilliztech/akcio) | 118 |
-|[CodeAlchemyAI/ViLT-GPT](https://github.com/CodeAlchemyAI/ViLT-GPT) | 117 |
-|[georgesung/llm_qlora](https://github.com/georgesung/llm_qlora) | 117 |
-|[nicknochnack/Nopenai](https://github.com/nicknochnack/Nopenai) | 115 |
-|[nftblackmagic/flask-langchain](https://github.com/nftblackmagic/flask-langchain) | 115 |
-|[mortium91/langchain-assistant](https://github.com/mortium91/langchain-assistant) | 115 |
-|[Ngonie-x/langchain_csv](https://github.com/Ngonie-x/langchain_csv) | 114 |
-|[wombyz/HormoziGPT](https://github.com/wombyz/HormoziGPT) | 114 |
-|[langchain-ai/langchain-teacher](https://github.com/langchain-ai/langchain-teacher) | 113 |
-|[mluogh/eastworld](https://github.com/mluogh/eastworld) | 112 |
-|[mudler/LocalAGI](https://github.com/mudler/LocalAGI) | 112 |
-|[marimo-team/marimo](https://github.com/marimo-team/marimo) | 111 |
-|[trancethehuman/entities-extraction-web-scraper](https://github.com/trancethehuman/entities-extraction-web-scraper) | 111 |
-|[xuwenhao/mactalk-ai-course](https://github.com/xuwenhao/mactalk-ai-course) | 111 |
-|[dcaribou/transfermarkt-datasets](https://github.com/dcaribou/transfermarkt-datasets) | 111 |
-|[rabbitmetrics/langchain-13-min](https://github.com/rabbitmetrics/langchain-13-min) | 111 |
-|[dotvignesh/PDFChat](https://github.com/dotvignesh/PDFChat) | 111 |
-|[aws-samples/cdk-eks-blueprints-patterns](https://github.com/aws-samples/cdk-eks-blueprints-patterns) | 110 |
-|[topoteretes/PromethAI-Backend](https://github.com/topoteretes/PromethAI-Backend) | 110 |
-|[jlonge4/local_llama](https://github.com/jlonge4/local_llama) | 110 |
-|[RUC-GSAI/YuLan-Rec](https://github.com/RUC-GSAI/YuLan-Rec) | 108 |
-|[gh18l/CrawlGPT](https://github.com/gh18l/CrawlGPT) | 107 |
-|[c0sogi/LLMChat](https://github.com/c0sogi/LLMChat) | 107 |
-|[hwchase17/langchain-gradio-template](https://github.com/hwchase17/langchain-gradio-template) | 107 |
-|[ArjanCodes/examples](https://github.com/ArjanCodes/examples) | 106 |
-|[genia-dev/GeniA](https://github.com/genia-dev/GeniA) | 105 |
-|[nexus-stc/stc](https://github.com/nexus-stc/stc) | 105 |
-|[mbchang/data-driven-characters](https://github.com/mbchang/data-driven-characters) | 105 |
-|[ademakdogan/ChatSQL](https://github.com/ademakdogan/ChatSQL) | 104 |
-|[crosleythomas/MirrorGPT](https://github.com/crosleythomas/MirrorGPT) | 104 |
-|[IvanIsCoding/ResuLLMe](https://github.com/IvanIsCoding/ResuLLMe) | 104 |
-|[avrabyt/MemoryBot](https://github.com/avrabyt/MemoryBot) | 104 |
-|[Azure/azure-sdk-tools](https://github.com/Azure/azure-sdk-tools) | 103 |
-|[aniketmaurya/llm-inference](https://github.com/aniketmaurya/llm-inference) | 103 |
-|[Anil-matcha/Youtube-to-chatbot](https://github.com/Anil-matcha/Youtube-to-chatbot) | 103 |
-|[nyanp/chat2plot](https://github.com/nyanp/chat2plot) | 102 |
-|[aws-samples/amazon-kendra-langchain-extensions](https://github.com/aws-samples/amazon-kendra-langchain-extensions) | 101 |
-|[atisharma/llama_farm](https://github.com/atisharma/llama_farm) | 100 |
-|[Xueheng-Li/SynologyChatbotGPT](https://github.com/Xueheng-Li/SynologyChatbotGPT) | 100 |
-
-
-
-_Generated by [github-dependents-info](https://github.com/nvuillam/github-dependents-info)_
-
-`github-dependents-info --repo langchain-ai/langchain --markdownfile dependents.md --minstars 100 --sort stars`

docs/docs/community.md

@@ -1,53 +0,0 @@
-# Community navigator
-
-Hi! Thanks for being here. We’re lucky to have a community of so many passionate developers building with LangChain–we have so much to teach and learn from each other. Community members contribute code, host meetups, write blog posts, amplify each other’s work, become each other's customers and collaborators, and so much more.
-
-Whether you’re new to LangChain, looking to go deeper, or just want to get more exposure to the world of building with LLMs, this page can point you in the right direction. 
-
-- **🦜 Contribute to LangChain**
-
-- **🌍 Meetups, Events, and Hackathons**
-
-- **📣 Help Us Amplify Your Work**
-
-- **💬 Stay in the loop**
-
-
-# 🦜 Contribute to LangChain
-
-LangChain is the product of over 5,000+ contributions by 1,500+ contributors, and there is ******still****** so much to do together. Here are some ways to get involved:
-
-- **[Open a pull request](https://github.com/langchain-ai/langchain/issues):** We’d appreciate all forms of contributions–new features, infrastructure improvements, better documentation, bug fixes, etc. If you have an improvement or an idea, we’d love to work on it with you.
-- **[Read our contributor guidelines:](https://github.com/langchain-ai/langchain/blob/bbd22b9b761389a5e40fc45b0570e1830aabb707/.github/CONTRIBUTING.md)** We ask contributors to follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow, run a few local checks for formatting, linting, and testing before submitting, and follow certain documentation and testing conventions.
-    - **First time contributor?** [Try one of these PRs with the “good first issue” tag](https://github.com/langchain-ai/langchain/contribute).
-- **Become an expert:** Our experts help the community by answering product questions in Discord. If that’s a role you’d like to play, we’d be so grateful! (And we have some special experts-only goodies/perks we can tell you more about). Send us an email to introduce yourself at hello@langchain.dev and we’ll take it from there!
-- **Integrate with LangChain:** If your product integrates with LangChain–or aspires to–we want to help make sure the experience is as smooth as possible for you and end users. Send us an email at hello@langchain.dev and tell us what you’re working on.
-    - **Become an Integration Maintainer:** Partner with our team to ensure your integration stays up-to-date and talk directly with users (and answer their inquiries) in our Discord. Introduce yourself at hello@langchain.dev if you’d like to explore this role.
-
-
-# 🌍 Meetups, Events, and Hackathons
-
-One of our favorite things about working in AI is how much enthusiasm there is for building together. We want to help make that as easy and impactful for you as possible! 
-- **Find a meetup, hackathon, or webinar:** You can find the one for you on our [global events calendar](https://mirror-feeling-d80.notion.site/0bc81da76a184297b86ca8fc782ee9a3?v=0d80342540df465396546976a50cfb3f).  
-    - **Submit an event to our calendar:** Email us at events@langchain.dev with a link to your event page! We can also help you spread the word with our local communities.
-- **Host a meetup:** If you want to bring a group of builders together, we want to help! We can publicize your event on our event calendar/Twitter, share it with our local communities in Discord, send swag, or potentially hook you up with a sponsor. Email us at events@langchain.dev to tell us about your event!
-- **Become a meetup sponsor:** We often hear from groups of builders that want to get together, but are blocked or limited on some dimension (space to host, budget for snacks, prizes to distribute, etc.). If you’d like to help, send us an email to events@langchain.dev we can share more about how it works!
-- **Speak at an event:** Meetup hosts are always looking for great speakers, presenters, and panelists. If you’d like to do that at an event, send us an email to hello@langchain.dev with more information about yourself, what you want to talk about, and what city you’re based in and we’ll try to match you with an upcoming event!
-- **Tell us about your LLM community:** If you host or participate in a community that would welcome support from LangChain and/or our team, send us an email at hello@langchain.dev and let us know how we can help.
-
-# 📣 Help Us Amplify Your Work
-
-If you’re working on something you’re proud of, and think the LangChain community would benefit from knowing about it, we want to help you show it off.
-
-- **Post about your work and mention us:** We love hanging out on Twitter to see what people in the space are talking about and working on. If you tag [@langchainai](https://twitter.com/LangChainAI), we’ll almost certainly see it and can show you some love.
-- **Publish something on our blog:** If you’re writing about your experience building with LangChain, we’d love to post (or crosspost) it on our blog! E-mail hello@langchain.dev with a draft of your post! Or even an idea for something you want to write about.
-- **Get your product onto our [integrations hub](https://integrations.langchain.com/):** Many developers take advantage of our seamless integrations with other products, and come to our integrations hub to find out who those are. If you want to get your product up there, tell us about it (and how it works with LangChain) at hello@langchain.dev.
-
-# ☀️ Stay in the loop
-
-Here’s where our team hangs out, talks shop, spotlights cool work, and shares what we’re up to. We’d love to see you there too.
-
-- **[Twitter](https://twitter.com/LangChainAI):** We post about what we’re working on and what cool things we’re seeing in the space. If you tag @langchainai in your post, we’ll almost certainly see it, and can show you some love!
-- **[Discord](https://discord.gg/6adMQxSpJS):** connect with over 30,000 developers who are building with LangChain.
-- **[GitHub](https://github.com/langchain-ai/langchain):** Open pull requests, contribute to a discussion, and/or contribute
-- **[Subscribe to our bi-weekly Release Notes](https://6w1pwbss0py.typeform.com/to/KjZB1auB):** a twice/month email roundup of the coolest things going on in our orbit

docs/docs/expression_language/cookbook/agent.ipynb

@@ -1,203 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "e89f490d",
-   "metadata": {},
-   "source": [
-    "# Agents\n",
-    "\n",
-    "You can pass a Runnable into an agent."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "af4381de",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.agents import XMLAgent, tool, AgentExecutor\n",
-    "from langchain.chat_models import ChatAnthropic"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "24cc8134",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = ChatAnthropic(model=\"claude-2\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "67c0b0e4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "@tool\n",
-    "def search(query: str) -> str:\n",
-    "    \"\"\"Search things about current events.\"\"\"\n",
-    "    return \"32 degrees\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "7203b101",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tool_list = [search]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "b68e756d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Get prompt to use\n",
-    "prompt = XMLAgent.get_default_prompt()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "61ab3e9a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Logic for going from intermediate steps to a string to pass into model\n",
-    "# This is pretty tied to the prompt\n",
-    "def convert_intermediate_steps(intermediate_steps):\n",
-    "    log = \"\"\n",
-    "    for action, observation in intermediate_steps:\n",
-    "        log += (\n",
-    "            f\"<tool>{action.tool}</tool><tool_input>{action.tool_input}\"\n",
-    "            f\"</tool_input><observation>{observation}</observation>\"\n",
-    "        )\n",
-    "    return log\n",
-    "\n",
-    "\n",
-    "# Logic for converting tools to string to go in prompt\n",
-    "def convert_tools(tools):\n",
-    "    return \"\\n\".join([f\"{tool.name}: {tool.description}\" for tool in tools])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "260f5988",
-   "metadata": {},
-   "source": [
-    "Building an agent from a runnable usually involves a few things:\n",
-    "\n",
-    "1. Data processing for the intermediate steps. These need to represented in a way that the language model can recognize them. This should be pretty tightly coupled to the instructions in the prompt\n",
-    "\n",
-    "2. The prompt itself\n",
-    "\n",
-    "3. The model, complete with stop tokens if needed\n",
-    "\n",
-    "4. The output parser - should be in sync with how the prompt specifies things to be formatted."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "e92f1d6f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "agent = (\n",
-    "    {\n",
-    "        \"question\": lambda x: x[\"question\"],\n",
-    "        \"intermediate_steps\": lambda x: convert_intermediate_steps(x[\"intermediate_steps\"])\n",
-    "    }\n",
-    "    | prompt.partial(tools=convert_tools(tool_list))\n",
-    "    | model.bind(stop=[\"</tool_input>\", \"</final_answer>\"])\n",
-    "    | XMLAgent.get_default_output_parser()\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "6ce6ec7a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "agent_executor = AgentExecutor(agent=agent, tools=tool_list, verbose=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "fb5cb2e3",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m <tool>search</tool>\n",
-      "<tool_input>weather in new york\u001b[0m\u001b[36;1m\u001b[1;3m32 degrees\u001b[0m\u001b[32;1m\u001b[1;3m\n",
-      "\n",
-      "<final_answer>The weather in New York is 32 degrees\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'question': 'whats the weather in New york?',\n",
-       " 'output': 'The weather in New York is 32 degrees'}"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent_executor.invoke({\"question\": \"whats the weather in New york?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "bce86dd8",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/code_writing.ipynb

@@ -1,119 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "f09fd305",
-   "metadata": {},
-   "source": [
-    "# Code writing\n",
-    "\n",
-    "Example of how to use LCEL to write Python code."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "bd7c259a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.utilities import PythonREPL"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "73795d2d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Write some python code to solve the user's problem. \n",
-    "\n",
-    "Return only python code in Markdown format, e.g.:\n",
-    "\n",
-    "```python\n",
-    "....\n",
-    "```\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [(\"system\", template), (\"human\", \"{input}\")]\n",
-    ")\n",
-    "\n",
-    "model = ChatOpenAI()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "42859e8a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def _sanitize_output(text: str):\n",
-    "    _, after = text.split(\"```python\")\n",
-    "    return after.split(\"```\")[0]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "5ded1a86",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model | StrOutputParser() | _sanitize_output | PythonREPL().run"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "208c2b75",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Python REPL can execute arbitrary code. Use with caution.\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'4\\n'"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"input\": \"whats 2 plus 2\"})"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/index.mdx

@@ -1,11 +0,0 @@
----
-sidebar_position: 2
----
-
-# Cookbook
-
-import DocCardList from "@theme/DocCardList";
-
-Example code for accomplishing common tasks with the LangChain Expression Language (LCEL). These examples show how to compose different Runnable (the core LCEL interface) components to achieve various tasks. If you're just getting acquainted with LCEL, the [Prompt + LLM](/docs/expression_language/cookbook/prompt_llm_parser) page is a good place to start.
-
-<DocCardList />

docs/docs/expression_language/cookbook/memory.ipynb

@@ -1,177 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "5062941a",
-   "metadata": {},
-   "source": [
-    "# Adding memory\n",
-    "\n",
-    "This shows how to add memory to an arbitrary chain. Right now, you can use the memory classes but need to hook it up manually"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "7998efd8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from operator import itemgetter\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
-    "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
-    "\n",
-    "model = ChatOpenAI()\n",
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"You are a helpful chatbot\"),\n",
-    "    MessagesPlaceholder(variable_name=\"history\"),\n",
-    "    (\"human\", \"{input}\")\n",
-    "])\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "fa0087f3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "memory = ConversationBufferMemory(return_messages=True)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "06b531ae",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'history': []}"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "memory.load_memory_variables({})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "d9437af6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = RunnablePassthrough.assign(\n",
-    "    memory=RunnableLambda(memory.load_memory_variables) | itemgetter(\"history\")\n",
-    ") | prompt | model\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "bed1e260",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='Hello Bob! How can I assist you today?', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "inputs = {\"input\": \"hi im bob\"}\n",
-    "response = chain.invoke(inputs)\n",
-    "response\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "890475b4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "memory.save_context(inputs, {\"output\": response.content})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "e8fcb77f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'history': [HumanMessage(content='hi im bob', additional_kwargs={}, example=False),\n",
-       "  AIMessage(content='Hello Bob! How can I assist you today?', additional_kwargs={}, example=False)]}"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "memory.load_memory_variables({})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "d837d5c3",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='Your name is Bob.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "inputs = {\"input\": \"whats my name\"}\n",
-    "response = chain.invoke(inputs)\n",
-    "response\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/moderation.ipynb

@@ -1,133 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "4927a727-b4c8-453c-8c83-bd87b4fcac14",
-   "metadata": {},
-   "source": [
-    "# Adding moderation\n",
-    "\n",
-    "This shows how to add in moderation (or other safeguards) around your LLM application."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "4f5f6449-940a-4f5c-97c0-39b71c3e2a68",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import OpenAIModerationChain\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "fcb8312b-7e7a-424f-a3ec-76738c9a9d21",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "moderate = OpenAIModerationChain()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "b24b9148-f6b0-4091-8ea8-d3fb281bd950",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = OpenAI()\n",
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"repeat after me: {input}\")\n",
-    "])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "1c8ed87c-9ca6-4559-bf60-d40e94a0af08",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "id": "5256b9bd-381a-42b0-bfa8-7e6d18f853cb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'\\n\\nYou are stupid.'"
-      ]
-     },
-     "execution_count": 23,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"input\": \"you are stupid\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "fe6e3b33-dc9a-49d5-b194-ba750c58a628",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "moderated_chain = chain | moderate"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "d8ba0cbd-c739-4d23-be9f-6ae092bd5ffb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'input': '\\n\\nYou are stupid',\n",
-       " 'output': \"Text was found that violates OpenAI's content policy.\"}"
-      ]
-     },
-     "execution_count": 25,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "moderated_chain.invoke({\"input\": \"you are stupid\"})"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/multiple_chains.ipynb

@@ -1,240 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "877102d1-02ea-4fa3-8ec7-a08e242b95b3",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 2\n",
-    "title: Multiple chains\n",
-    "---"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0f2bf8d3",
-   "metadata": {},
-   "source": [
-    "Runnables can easily be used to string together multiple Chains"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "d65d4e9e",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'El país donde se encuentra la ciudad de Honolulu, donde nació Barack Obama, el 44º Presidente de los Estados Unidos, es Estados Unidos. Honolulu se encuentra en la isla de Oahu, en el estado de Hawái.'"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from operator import itemgetter\n",
-    "\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema import StrOutputParser\n",
-    "\n",
-    "prompt1 = ChatPromptTemplate.from_template(\"what is the city {person} is from?\")\n",
-    "prompt2 = ChatPromptTemplate.from_template(\"what country is the city {city} in? respond in {language}\")\n",
-    "\n",
-    "model = ChatOpenAI()\n",
-    "\n",
-    "chain1 = prompt1 | model | StrOutputParser()\n",
-    "\n",
-    "chain2 = {\"city\": chain1, \"language\": itemgetter(\"language\")} | prompt2 | model | StrOutputParser()\n",
-    "\n",
-    "chain2.invoke({\"person\": \"obama\", \"language\": \"spanish\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "878f8176",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableMap, RunnablePassthrough\n",
-    "\n",
-    "prompt1 = ChatPromptTemplate.from_template(\"generate a {attribute} color. Return the name of the color and nothing else:\")\n",
-    "prompt2 = ChatPromptTemplate.from_template(\"what is a fruit of color: {color}. Return the name of the fruit and nothing else:\")\n",
-    "prompt3 = ChatPromptTemplate.from_template(\"what is a country with a flag that has the color: {color}. Return the name of the country and nothing else:\")\n",
-    "prompt4 = ChatPromptTemplate.from_template(\"What is the color of {fruit} and the flag of {country}?\")\n",
-    "\n",
-    "model_parser = model | StrOutputParser()\n",
-    "\n",
-    "color_generator = {\"attribute\": RunnablePassthrough()} | prompt1 | {\"color\": model_parser}\n",
-    "color_to_fruit = prompt2 | model_parser\n",
-    "color_to_country = prompt3 | model_parser\n",
-    "question_generator = color_generator | {\"fruit\": color_to_fruit, \"country\": color_to_country} | prompt4"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "d621a870",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "ChatPromptValue(messages=[HumanMessage(content='What is the color of strawberry and the flag of China?', additional_kwargs={}, example=False)])"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "question_generator.invoke(\"warm\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "b4a9812b-bead-4fd9-ae27-0b8be57e5dc1",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='The color of an apple is typically red or green. The flag of China is predominantly red with a large yellow star in the upper left corner and four smaller yellow stars surrounding it.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "prompt = question_generator.invoke(\"warm\")\n",
-    "model.invoke(prompt)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6d75a313-f1c8-4e94-9a17-24e0bf4a2bdc",
-   "metadata": {},
-   "source": [
-    "### Branching and Merging\n",
-    "\n",
-    "You may want the output of one component to be processed by 2 or more other components. [RunnableMaps](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.RunnableMap.html) let you split or fork the chain so multiple components can process the input in parallel. Later, other components can join or merge the results to synthesize a final response. This type of chain creates a computation graph that looks like the following:\n",
-    "\n",
-    "```text\n",
-    "     Input\n",
-    "      / \\\n",
-    "     /   \\\n",
-    " Branch1 Branch2\n",
-    "     \\   /\n",
-    "      \\ /\n",
-    "      Combine\n",
-    "```"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "247fa0bd-4596-4063-8cb3-1d7fc119d982",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "planner = (\n",
-    "    ChatPromptTemplate.from_template(\n",
-    "        \"Generate an argument about: {input}\"\n",
-    "    )\n",
-    "    | ChatOpenAI()\n",
-    "    | StrOutputParser()\n",
-    "    | {\"base_response\": RunnablePassthrough()}\n",
-    ")\n",
-    "\n",
-    "arguments_for = (\n",
-    "    ChatPromptTemplate.from_template(\n",
-    "        \"List the pros or positive aspects of {base_response}\"\n",
-    "    )\n",
-    "    | ChatOpenAI()\n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "arguments_against =  (\n",
-    "    ChatPromptTemplate.from_template(\n",
-    "        \"List the cons or negative aspects of {base_response}\"\n",
-    "    )\n",
-    "    | ChatOpenAI()\n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "\n",
-    "final_responder = (\n",
-    "    ChatPromptTemplate.from_messages(\n",
-    "        [\n",
-    "            (\"ai\", \"{original_response}\"),\n",
-    "            (\"human\", \"Pros:\\n{results_1}\\n\\nCons:\\n{results_2}\"),\n",
-    "            (\"system\", \"Generate a final response given the critique\"),\n",
-    "        ]\n",
-    "    )\n",
-    "    | ChatOpenAI()\n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "\n",
-    "chain = (\n",
-    "    planner \n",
-    "    | {\n",
-    "        \"results_1\": arguments_for,\n",
-    "        \"results_2\": arguments_against,\n",
-    "        \"original_response\": itemgetter(\"base_response\"),\n",
-    "    }\n",
-    "    | final_responder\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "2564f310-0674-4bb1-9c4e-d7848ca73511",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'While Scrum has its potential cons and challenges, many organizations have successfully embraced and implemented this project management framework to great effect. The cons mentioned above can be mitigated or overcome with proper training, support, and a commitment to continuous improvement. It is also important to note that not all cons may be applicable to every organization or project.\\n\\nFor example, while Scrum may be complex initially, with proper training and guidance, teams can quickly grasp the concepts and practices. The lack of predictability can be mitigated by implementing techniques such as velocity tracking and release planning. The limited documentation can be addressed by maintaining a balance between lightweight documentation and clear communication among team members. The dependency on team collaboration can be improved through effective communication channels and regular team-building activities.\\n\\nScrum can be scaled and adapted to larger projects by using frameworks like Scrum of Scrums or LeSS (Large Scale Scrum). Concerns about speed versus quality can be addressed by incorporating quality assurance practices, such as continuous integration and automated testing, into the Scrum process. Scope creep can be managed by having a well-defined and prioritized product backlog, and a strong product owner can be developed through training and mentorship.\\n\\nResistance to change can be overcome by providing proper education and communication to stakeholders and involving them in the decision-making process. Ultimately, the cons of Scrum can be seen as opportunities for growth and improvement, and with the right mindset and support, they can be effectively managed.\\n\\nIn conclusion, while Scrum may have its challenges and potential cons, the benefits and advantages it offers in terms of collaboration, flexibility, adaptability, transparency, and customer satisfaction make it a widely adopted and successful project management framework. With proper implementation and continuous improvement, organizations can leverage Scrum to drive innovation, efficiency, and project success.'"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"input\": \"scrum\"})"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/prompt_llm_parser.ipynb

@@ -1,431 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "abf7263d-3a62-4016-b5d5-b157f92f2070",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 0\n",
-    "title: Prompt + LLM\n",
-    "---\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9a434f2b-9405-468c-9dfd-254d456b57a6",
-   "metadata": {},
-   "source": [
-    "The most common and valuable composition is taking:\n",
-    "\n",
-    "``PromptTemplate`` / ``ChatPromptTemplate`` -> ``LLM`` / ``ChatModel`` -> ``OutputParser``\n",
-    "\n",
-    "Almost any other chains you build will use this building block."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "93aa2c87",
-   "metadata": {},
-   "source": [
-    "## PromptTemplate + LLM\n",
-    "\n",
-    "The simplest composition is just combing a prompt and model to create a chain that takes user input, adds it to a prompt, passes it to a model, and returns the raw model input.\n",
-    "\n",
-    "Note, you can mix and match PromptTemplate/ChatPromptTemplates and LLMs/ChatModels as you like here."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "466b65b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {foo}\")\n",
-    "model = ChatOpenAI()\n",
-    "chain = prompt | model\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "e3d0a6cd",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7eb9ef50",
-   "metadata": {},
-   "source": [
-    "Often times we want to attach kwargs that'll be passed to each model call. Here's a few examples of that:"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0b1d8f88",
-   "metadata": {},
-   "source": [
-    "### Attaching Stop Sequences"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "562a06bf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model.bind(stop=[\"\\n\"])\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "43f5d04c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='Why did the bear never wear shoes?', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f3eaf88a",
-   "metadata": {},
-   "source": [
-    "### Attaching Function Call information"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "f94b71b2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "functions = [\n",
-    "    {\n",
-    "      \"name\": \"joke\",\n",
-    "      \"description\": \"A joke\",\n",
-    "      \"parameters\": {\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\n",
-    "          \"setup\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"description\": \"The setup for the joke\"\n",
-    "          },\n",
-    "          \"punchline\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"description\": \"The punchline for the joke\"\n",
-    "          }\n",
-    "        },\n",
-    "        \"required\": [\"setup\", \"punchline\"]\n",
-    "      }\n",
-    "    }\n",
-    "  ]\n",
-    "chain = prompt | model.bind(function_call= {\"name\": \"joke\"}, functions= functions)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "decf7710",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='', additional_kwargs={'function_call': {'name': 'joke', 'arguments': '{\\n  \"setup\": \"Why don\\'t bears wear shoes?\",\\n  \"punchline\": \"Because they have bear feet!\"\\n}'}}, example=False)"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bears\"}, config={})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9098c5ed",
-   "metadata": {},
-   "source": [
-    "## PromptTemplate + LLM + OutputParser\n",
-    "\n",
-    "We can also add in an output parser to easily transform the raw LLM/ChatModel output into a more workable format"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "cc194c78",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "\n",
-    "chain = prompt | model | StrOutputParser()\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "77acf448",
-   "metadata": {},
-   "source": [
-    "Notice that this now returns a string - a much more workable format for downstream tasks"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "e3d69a18",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\""
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c01864e5",
-   "metadata": {},
-   "source": [
-    "### Functions Output Parser\n",
-    "\n",
-    "When you specify the function to return, you may just want to parse that directly"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "ad0dd88e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.output_parsers.openai_functions import JsonOutputFunctionsParser\n",
-    "\n",
-    "chain = (\n",
-    "    prompt \n",
-    "    | model.bind(function_call= {\"name\": \"joke\"}, functions= functions) \n",
-    "    | JsonOutputFunctionsParser()\n",
-    ")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "1e7aa8eb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'setup': \"Why don't bears like fast food?\",\n",
-       " 'punchline': \"Because they can't catch it!\"}"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "d4aa1a01",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.output_parsers.openai_functions import JsonKeyOutputFunctionsParser\n",
-    "\n",
-    "chain = (\n",
-    "    prompt \n",
-    "    | model.bind(function_call= {\"name\": \"joke\"}, functions= functions) \n",
-    "    | JsonKeyOutputFunctionsParser(key_name=\"setup\")\n",
-    ")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "8b6df9ba",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"Why don't bears wear shoes?\""
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "023fbccb-ef7d-489e-a9ba-f98e17283d51",
-   "metadata": {},
-   "source": [
-    "## Simplifying input\n",
-    "\n",
-    "To make invocation even simpler, we can add a `RunnableMap` to take care of creating the prompt input dict for us:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "9601c0f0-71f9-4bd4-a672-7bd04084b018",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableMap, RunnablePassthrough\n",
-    "\n",
-    "map_ = RunnableMap(foo=RunnablePassthrough())\n",
-    "chain = (\n",
-    "    map_ \n",
-    "    | prompt\n",
-    "    | model.bind(function_call= {\"name\": \"joke\"}, functions= functions) \n",
-    "    | JsonKeyOutputFunctionsParser(key_name=\"setup\")\n",
-    ")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "7ec4f154-fda5-4847-9220-41aa902fdc33",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"Why don't bears wear shoes?\""
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke(\"bears\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "def00bfe-0f83-4805-8c8f-8a53f99fa8ea",
-   "metadata": {},
-   "source": [
-    "Since we're composing our map with another Runnable, we can even use some syntactic sugar and just use a dict:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "7bf3846a-02ee-41a3-ba1b-a708827d4f3a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = (\n",
-    "    {\"foo\": RunnablePassthrough()} \n",
-    "    | prompt\n",
-    "    | model.bind(function_call= {\"name\": \"joke\"}, functions= functions) \n",
-    "    | JsonKeyOutputFunctionsParser(key_name=\"setup\")\n",
-    ")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "e566d6a1-538d-4cb5-a210-a63e082e4c74",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"Why don't bears like fast food?\""
-      ]
-     },
-     "execution_count": 22,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke(\"bears\")\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/retrieval.ipynb

@@ -1,450 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "abe47592-909c-4844-bf44-9e55c2fb4bfa",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 1\n",
-    "title: RAG\n",
-    "---\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "91c5ef3d",
-   "metadata": {},
-   "source": [
-    "Let's look at adding in a retrieval step to a prompt and LLM, which adds up to a \"retrieval-augmented generation\" chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "7f25d9e9-d192-42e9-af50-5660a4bfb0d9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install langchain openai faiss-cpu tiktoken\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "33be32af",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from operator import itemgetter\n",
-    "\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
-    "from langchain.vectorstores import FAISS\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "bfc47ec1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings())\n",
-    "retriever = vectorstore.as_retriever()\n",
-    "\n",
-    "template = \"\"\"Answer the question based only on the following context:\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n",
-    "\n",
-    "model = ChatOpenAI()\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "eae31755",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = (\n",
-    "    {\"context\": retriever, \"question\": RunnablePassthrough()} \n",
-    "    | prompt \n",
-    "    | model \n",
-    "    | StrOutputParser()\n",
-    ")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "f3040b0c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Harrison worked at Kensho.'"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke(\"where did harrison work?\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "e1d20c7c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Answer the question based only on the following context:\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "\n",
-    "Answer in the following language: {language}\n",
-    "\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n",
-    "\n",
-    "chain = {\n",
-    "    \"context\": itemgetter(\"question\") | retriever, \n",
-    "    \"question\": itemgetter(\"question\"), \n",
-    "    \"language\": itemgetter(\"language\")\n",
-    "} | prompt | model | StrOutputParser()\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "7ee8b2d4",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Harrison ha lavorato a Kensho.'"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"question\": \"where did harrison work\", \"language\": \"italian\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f007669c",
-   "metadata": {},
-   "source": [
-    "## Conversational Retrieval Chain\n",
-    "\n",
-    "We can easily add in conversation history. This primarily means adding in chat_message_history"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "3f30c348",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableMap\n",
-    "from langchain.schema import format_document\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "64ab1dbf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts.prompt import PromptTemplate\n",
-    "\n",
-    "_template = \"\"\"Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.\n",
-    "\n",
-    "Chat History:\n",
-    "{chat_history}\n",
-    "Follow Up Input: {question}\n",
-    "Standalone question:\"\"\"\n",
-    "CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "7d628c97",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Answer the question based only on the following context:\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "\"\"\"\n",
-    "ANSWER_PROMPT = ChatPromptTemplate.from_template(template)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "f60a5d0f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "DEFAULT_DOCUMENT_PROMPT = PromptTemplate.from_template(template=\"{page_content}\")\n",
-    "def _combine_documents(docs, document_prompt = DEFAULT_DOCUMENT_PROMPT, document_separator=\"\\n\\n\"):\n",
-    "    doc_strings = [format_document(doc, document_prompt) for doc in docs]\n",
-    "    return document_separator.join(doc_strings)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "7d007db6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import Tuple, List\n",
-    "def _format_chat_history(chat_history: List[Tuple]) -> str:\n",
-    "    buffer = \"\"\n",
-    "    for dialogue_turn in chat_history:\n",
-    "        human = \"Human: \" + dialogue_turn[0]\n",
-    "        ai = \"Assistant: \" + dialogue_turn[1]\n",
-    "        buffer += \"\\n\" + \"\\n\".join([human, ai])\n",
-    "    return buffer\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "5c32cc89",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "_inputs = RunnableMap(\n",
-    "    standalone_question=RunnablePassthrough.assign(\n",
-    "        chat_history=lambda x: _format_chat_history(x['chat_history'])\n",
-    "    ) | CONDENSE_QUESTION_PROMPT | ChatOpenAI(temperature=0) | StrOutputParser(),\n",
-    ")\n",
-    "_context = {\n",
-    "    \"context\": itemgetter(\"standalone_question\") | retriever | _combine_documents,\n",
-    "    \"question\": lambda x: x[\"standalone_question\"]\n",
-    "}\n",
-    "conversational_qa_chain = _inputs | _context | ANSWER_PROMPT | ChatOpenAI()\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "135c8205",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='Harrison was employed at Kensho.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "conversational_qa_chain.invoke({\n",
-    "    \"question\": \"where did harrison work?\",\n",
-    "    \"chat_history\": [],\n",
-    "})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "424e7e7a",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='Harrison worked at Kensho.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "conversational_qa_chain.invoke({\n",
-    "    \"question\": \"where did he work?\",\n",
-    "    \"chat_history\": [(\"Who wrote this notebook?\", \"Harrison\")],\n",
-    "})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c5543183",
-   "metadata": {},
-   "source": [
-    "### With Memory and returning source documents\n",
-    "\n",
-    "This shows how to use memory with the above. For memory, we need to manage that outside at the memory. For returning the retrieved documents, we just need to pass them through all the way."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "e31dd17c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from operator import itemgetter\n",
-    "from langchain.memory import ConversationBufferMemory\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "d4bffe94",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "memory = ConversationBufferMemory(return_messages=True, output_key=\"answer\", input_key=\"question\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "733be985",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# First we add a step to load memory\n",
-    "# This adds a \"memory\" key to the input object\n",
-    "loaded_memory = RunnablePassthrough.assign(\n",
-    "    chat_history=RunnableLambda(memory.load_memory_variables) | itemgetter(\"history\"),\n",
-    ")\n",
-    "# Now we calculate the standalone question\n",
-    "standalone_question = {\n",
-    "    \"standalone_question\": {\n",
-    "        \"question\": lambda x: x[\"question\"],\n",
-    "        \"chat_history\": lambda x: _format_chat_history(x['chat_history'])\n",
-    "    } | CONDENSE_QUESTION_PROMPT | ChatOpenAI(temperature=0) | StrOutputParser(),\n",
-    "}\n",
-    "# Now we retrieve the documents\n",
-    "retrieved_documents = {\n",
-    "    \"docs\": itemgetter(\"standalone_question\") | retriever,\n",
-    "    \"question\": lambda x: x[\"standalone_question\"]\n",
-    "}\n",
-    "# Now we construct the inputs for the final prompt\n",
-    "final_inputs = {\n",
-    "    \"context\": lambda x: _combine_documents(x[\"docs\"]),\n",
-    "    \"question\": itemgetter(\"question\")\n",
-    "}\n",
-    "# And finally, we do the part that returns the answers\n",
-    "answer = {\n",
-    "    \"answer\": final_inputs | ANSWER_PROMPT | ChatOpenAI(),\n",
-    "    \"docs\": itemgetter(\"docs\"),\n",
-    "}\n",
-    "# And now we put it all together!\n",
-    "final_chain = loaded_memory | standalone_question | retrieved_documents | answer\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "806e390c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'answer': AIMessage(content='Harrison was employed at Kensho.', additional_kwargs={}, example=False),\n",
-       " 'docs': [Document(page_content='harrison worked at kensho', metadata={})]}"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "inputs = {\"question\": \"where did harrison work?\"}\n",
-    "result = final_chain.invoke(inputs)\n",
-    "result\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "977399fd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Note that the memory does not save automatically\n",
-    "# This will be improved in the future\n",
-    "# For now you need to save it yourself\n",
-    "memory.save_context(inputs, {\"answer\": result[\"answer\"].content})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "f94f7de4",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'history': [HumanMessage(content='where did harrison work?', additional_kwargs={}, example=False),\n",
-       "  AIMessage(content='Harrison was employed at Kensho.', additional_kwargs={}, example=False)]}"
-      ]
-     },
-     "execution_count": 21,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "memory.load_memory_variables({})\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/sql_db.ipynb

@@ -1,216 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "c14da114-1a4a-487d-9cff-e0e8c30ba366",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 3\n",
-    "title: Querying a SQL DB\n",
-    "---\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "506e9636",
-   "metadata": {},
-   "source": [
-    "We can replicate our SQLDatabaseChain with Runnables."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "7a927516",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "\n",
-    "template = \"\"\"Based on the table schema below, write a SQL query that would answer the user's question:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query:\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "3f51f386",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.utilities import SQLDatabase\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7c3449d6-684b-416e-ba16-90a035835a88",
-   "metadata": {},
-   "source": [
-    "We'll need the Chinook sample DB for this example. There's many places to download it from, e.g. https://database.guide/2-sample-databases-sqlite/"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "2ccca6fc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "db = SQLDatabase.from_uri(\"sqlite:///./Chinook.db\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "05ba88ee",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def get_schema(_):\n",
-    "    return db.get_table_info()\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "a4eda902",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def run_query(query):\n",
-    "    return db.run(query)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "id": "5046cb17",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "\n",
-    "model = ChatOpenAI()\n",
-    "\n",
-    "sql_response = (\n",
-    "        RunnablePassthrough.assign(schema=get_schema)\n",
-    "        | prompt\n",
-    "        | model.bind(stop=[\"\\nSQLResult:\"])\n",
-    "        | StrOutputParser()\n",
-    "    )\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "a5552039",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'SELECT COUNT(*) FROM Employee'"
-      ]
-     },
-     "execution_count": 24,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "sql_response.invoke({\"question\": \"How many employees are there?\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "d6fee130",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query: {query}\n",
-    "SQL Response: {response}\"\"\"\n",
-    "prompt_response = ChatPromptTemplate.from_template(template)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "923aa634",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "full_chain = (\n",
-    "    RunnablePassthrough.assign(query=sql_response) \n",
-    "    | RunnablePassthrough.assign(\n",
-    "        schema=get_schema,\n",
-    "        response=lambda x: db.run(x[\"query\"]),\n",
-    "    )\n",
-    "    | prompt_response \n",
-    "    | model\n",
-    ")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 27,
-   "id": "e94963d8",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='There are 8 employees.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 27,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"How many employees are there?\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "4f358d7b-a721-4db3-9f92-f06913428afc",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/cookbook/tools.ipynb

@@ -1,122 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "29781123",
-   "metadata": {},
-   "source": [
-    "# Using tools\n",
-    "\n",
-    "You can use any Tools with Runnables easily."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a5c579dd-2e22-41b0-a789-346dfdecb5a2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install duckduckgo-search"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "9232d2a9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.tools import DuckDuckGoSearchRun"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "a0c64d2c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "search = DuckDuckGoSearchRun()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "391969b6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"turn the following user input into a search query for a search engine:\n",
-    "\n",
-    "{input}\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n",
-    "\n",
-    "model = ChatOpenAI()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "e3d9d20d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model | StrOutputParser() | search"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "55f2967d",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'What sports games are on TV today & tonight? Watch and stream live sports on TV today, tonight, tomorrow. Today\\'s 2023 sports TV schedule includes football, basketball, baseball, hockey, motorsports, soccer and more. Watch on TV or stream online on ESPN, FOX, FS1, CBS, NBC, ABC, Peacock, Paramount+, fuboTV, local channels and many other networks. MLB Games Tonight: How to Watch on TV, Streaming & Odds - Thursday, September 7. Seattle Mariners\\' Julio Rodriguez greets teammates in the dugout after scoring against the Oakland Athletics in a ... Circle - Country Music and Lifestyle. Live coverage of all the MLB action today is available to you, with the information provided below. The Brewers will look to pick up a road win at PNC Park against the Pirates on Wednesday at 12:35 PM ET. Check out the latest odds and with BetMGM Sportsbook. Use bonus code \"GNPLAY\" for special offers! MLB Games Tonight: How to Watch on TV, Streaming & Odds - Tuesday, September 5. Houston Astros\\' Kyle Tucker runs after hitting a double during the fourth inning of a baseball game against the Los Angeles Angels, Sunday, Aug. 13, 2023, in Houston. (AP Photo/Eric Christian Smith) (APMedia) The Houston Astros versus the Texas Rangers is one of ... The second half of tonight\\'s college football schedule still has some good games remaining to watch on your television.. We\\'ve already seen an exciting one when Colorado upset TCU. And we saw some ...'"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"input\": \"I'd like to figure out what games are tonight\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a16949cf-00ea-43c6-a6aa-797ad4f6918d",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/binding.ipynb

@@ -1,194 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "711752cb-4f15-42a3-9838-a0c67f397771",
-   "metadata": {},
-   "source": [
-    "# Bind runtime args\n",
-    "\n",
-    "Sometimes we want to invoke a Runnable within a Runnable sequence with constant arguments that are not part of the output of the preceding Runnable in the sequence, and which are not part of the user input. We can use `Runnable.bind()` to easily pass these arguments in.\n",
-    "\n",
-    "Suppose we have a simple prompt + model sequence:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "f3fdf86d-155f-4587-b7cd-52d363970c1d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "EQUATION: x^3 + 7 = 12\n",
-      "\n",
-      "SOLUTION:\n",
-      "Subtracting 7 from both sides of the equation, we get:\n",
-      "x^3 = 12 - 7\n",
-      "x^3 = 5\n",
-      "\n",
-      "Taking the cube root of both sides, we get:\n",
-      "x = ∛5\n",
-      "\n",
-      "Therefore, the solution to the equation x^3 + 7 = 12 is x = ∛5.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"Write out the following equation using algebraic symbols then solve it. Use the format\\n\\nEQUATION:...\\nSOLUTION:...\\n\\n\"),\n",
-    "        (\"human\", \"{equation_statement}\")\n",
-    "    ]\n",
-    ")\n",
-    "model = ChatOpenAI(temperature=0)\n",
-    "runnable = {\"equation_statement\": RunnablePassthrough()} | prompt | model | StrOutputParser()\n",
-    "\n",
-    "print(runnable.invoke(\"x raised to the third plus seven equals 12\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "929c9aba-a4a0-462c-adac-2cfc2156e117",
-   "metadata": {},
-   "source": [
-    "and want to call the model with certain `stop` words:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "32e0484a-78c5-4570-a00b-20d597245a96",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "EQUATION: x^3 + 7 = 12\n",
-      "\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "runnable = (\n",
-    "    {\"equation_statement\": RunnablePassthrough()} \n",
-    "    | prompt \n",
-    "    | model.bind(stop=\"SOLUTION\") \n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "print(runnable.invoke(\"x raised to the third plus seven equals 12\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f4bd641f-6b58-4ca9-a544-f69095428f16",
-   "metadata": {},
-   "source": [
-    "## Attaching OpenAI functions\n",
-    "\n",
-    "One particularly useful application of binding is to attach OpenAI functions to a compatible OpenAI model:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "f66a0fe4-fde0-4706-8863-d60253f211c7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "functions = [\n",
-    "    {\n",
-    "      \"name\": \"solver\",\n",
-    "      \"description\": \"Formulates and solves an equation\",\n",
-    "      \"parameters\": {\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\n",
-    "          \"equation\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"description\": \"The algebraic expression of the equation\"\n",
-    "          },\n",
-    "          \"solution\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"description\": \"The solution to the equation\"\n",
-    "          }\n",
-    "        },\n",
-    "        \"required\": [\"equation\", \"solution\"]\n",
-    "      }\n",
-    "    }\n",
-    "  ]\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "f381f969-df8e-48a3-bf5c-d0397cfecde0",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='', additional_kwargs={'function_call': {'name': 'solver', 'arguments': '{\\n\"equation\": \"x^3 + 7 = 12\",\\n\"solution\": \"x = ∛5\"\\n}'}}, example=False)"
-      ]
-     },
-     "execution_count": 22,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Need gpt-4 to solve this one correctly\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"Write out the following equation using algebraic symbols then solve it.\"),\n",
-    "        (\"human\", \"{equation_statement}\")\n",
-    "    ]\n",
-    ")\n",
-    "model = ChatOpenAI(model=\"gpt-4\", temperature=0).bind(function_call={\"name\": \"solver\"}, functions=functions)\n",
-    "runnable = (\n",
-    "    {\"equation_statement\": RunnablePassthrough()} \n",
-    "    | prompt \n",
-    "    | model\n",
-    ")\n",
-    "runnable.invoke(\"x raised to the third plus seven equals 12\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2cdeeb4c-0c1f-43da-bd58-4f591d9e0671",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/configure.ipynb

@@ -1,594 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "39eaf61b",
-   "metadata": {},
-   "source": [
-    "# Configuration\n",
-    "\n",
-    "Oftentimes you may want to experiment with, or even expose to the end user, multiple different ways of doing things.\n",
-    "In order to make this experience as easy as possible, we have defined two methods.\n",
-    "\n",
-    "First, a `configurable_fields` method. \n",
-    "This lets you configure particular fields of a runnable.\n",
-    "\n",
-    "Second, a `configurable_alternatives` method.\n",
-    "With this method, you can list out alternatives for any particular runnable that can be set during runtime."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f2347a11",
-   "metadata": {},
-   "source": [
-    "## Configuration Fields"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a06f6e2d",
-   "metadata": {},
-   "source": [
-    "### With LLMs\n",
-    "With LLMs we can configure things like temperature"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 35,
-   "id": "7ba735f4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "model = ChatOpenAI(temperature=0).configurable_fields(\n",
-    "    temperature=ConfigurableField(\n",
-    "        id=\"llm_temperature\",\n",
-    "        name=\"LLM Temperature\",\n",
-    "        description=\"The temperature of the LLM\",\n",
-    "    )\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 38,
-   "id": "63a71165",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='7')"
-      ]
-     },
-     "execution_count": 38,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "model.invoke(\"pick a random number\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 39,
-   "id": "4f83245c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='34')"
-      ]
-     },
-     "execution_count": 39,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "model.with_config(configurable={\"llm_temperature\": .9}).invoke(\"pick a random number\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9da1fcd2",
-   "metadata": {},
-   "source": [
-    "We can also do this when its used as part of a chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 40,
-   "id": "e75ae678",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = PromptTemplate.from_template(\"Pick a random number above {x}\")\n",
-    "chain = prompt | model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 41,
-   "id": "44886071",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='57')"
-      ]
-     },
-     "execution_count": 41,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"x\": 0})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "c09fac15",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='6')"
-      ]
-     },
-     "execution_count": 42,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.with_config(configurable={\"llm_temperature\": .9}).invoke({\"x\": 0})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "fb9637d0",
-   "metadata": {},
-   "source": [
-    "### With HubRunnables\n",
-    "\n",
-    "This is useful to allow for switching of prompts"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 43,
-   "id": "7d5836b2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.runnables.hub import HubRunnable"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 46,
-   "id": "9a9ea077",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = HubRunnable(\"rlm/rag-prompt\").configurable_fields(\n",
-    "    owner_repo_commit=ConfigurableField(\n",
-    "        id=\"hub_commit\",\n",
-    "        name=\"Hub Commit\",\n",
-    "        description=\"The Hub commit to pull from\",\n",
-    "    )\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 47,
-   "id": "c4a62cee",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "ChatPromptValue(messages=[HumanMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\\nQuestion: foo \\nContext: bar \\nAnswer:\")])"
-      ]
-     },
-     "execution_count": 47,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "prompt.invoke({\"question\": \"foo\", \"context\": \"bar\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 49,
-   "id": "f33f3cf2",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "ChatPromptValue(messages=[HumanMessage(content=\"[INST]<<SYS>> You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.<</SYS>> \\nQuestion: foo \\nContext: bar \\nAnswer: [/INST]\")])"
-      ]
-     },
-     "execution_count": 49,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "prompt.with_config(configurable={\"hub_commit\": \"rlm/rag-prompt-llama\"}).invoke({\"question\": \"foo\", \"context\": \"bar\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "79d51519",
-   "metadata": {},
-   "source": [
-    "## Configurable Alternatives\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ac733d35",
-   "metadata": {},
-   "source": [
-    "### With LLMs\n",
-    "\n",
-    "Let's take a look at doing this with LLMs"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "430ab8cc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic\n",
-    "from langchain.schema.runnable import ConfigurableField\n",
-    "from langchain.prompts import PromptTemplate"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "71248a9f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatAnthropic(temperature=0).configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"llm\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"anthropic\",\n",
-    "    # This adds a new option, with name `openai` that is equal to `ChatOpenAI()`\n",
-    "    openai=ChatOpenAI(),\n",
-    "    # This adds a new option, with name `gpt4` that is equal to `ChatOpenAI(model=\"gpt-4\")`\n",
-    "    gpt4=ChatOpenAI(model=\"gpt-4\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "prompt = PromptTemplate.from_template(\"Tell me a joke about {topic}\")\n",
-    "chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "e598b1f1",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Here's a silly joke about bears:\\n\\nWhat do you call a bear with no teeth?\\nA gummy bear!\")"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# By default it will call Anthropic\n",
-    "chain.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "48b45337",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Sure, here's a bear joke for you:\\n\\nWhy don't bears wear shoes?\\n\\nBecause they already have bear feet!\")"
-      ]
-     },
-     "execution_count": 20,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can use `.with_config(configurable={\"llm\": \"openai\"})` to specify an llm to use\n",
-    "chain.with_config(configurable={\"llm\": \"openai\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "42647fb7",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Here's a silly joke about bears:\\n\\nWhat do you call a bear with no teeth?\\nA gummy bear!\")"
-      ]
-     },
-     "execution_count": 21,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# If we use the `default_key` then it uses the default\n",
-    "chain.with_config(configurable={\"llm\": \"anthropic\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a9134559",
-   "metadata": {},
-   "source": [
-    "### With Prompts\n",
-    "\n",
-    "We can do a similar thing, but alternate between prompts\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "9f6a7c6c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatAnthropic(temperature=0)\n",
-    "prompt = PromptTemplate.from_template(\"Tell me a joke about {topic}\").configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"prompt\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"joke\",\n",
-    "    # This adds a new option, with name `poem`\n",
-    "    poem=PromptTemplate.from_template(\"Write a short poem about {topic}\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "97eda915",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Here's a silly joke about bears:\\n\\nWhat do you call a bear with no teeth?\\nA gummy bear!\")"
-      ]
-     },
-     "execution_count": 26,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# By default it will write a joke\n",
-    "chain.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 27,
-   "id": "927297a1",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Here is a short poem about bears:\\n\\nThe bears awaken from their sleep\\nAnd lumber out into the deep\\nForests filled with trees so tall\\nForaging for food before nightfall \\nTheir furry coats and claws so sharp\\nSniffing for berries and fish to nab\\nLumbering about without a care\\nThe mighty grizzly and black bear\\nProud creatures, wild and free\\nRuling their domain majestically\\nWandering the woods they call their own\\nBefore returning to their dens alone')"
-      ]
-     },
-     "execution_count": 27,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can configure it write a poem\n",
-    "chain.with_config(configurable={\"prompt\": \"poem\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0c77124e",
-   "metadata": {},
-   "source": [
-    "### With Prompts and LLMs\n",
-    "\n",
-    "We can also have multiple things configurable!\n",
-    "Here's an example doing that with both prompts and LLMs."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 28,
-   "id": "97538c23",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatAnthropic(temperature=0).configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"llm\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"anthropic\",\n",
-    "    # This adds a new option, with name `openai` that is equal to `ChatOpenAI()`\n",
-    "    openai=ChatOpenAI(),\n",
-    "    # This adds a new option, with name `gpt4` that is equal to `ChatOpenAI(model=\"gpt-4\")`\n",
-    "    gpt4=ChatOpenAI(model=\"gpt-4\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "prompt = PromptTemplate.from_template(\"Tell me a joke about {topic}\").configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"prompt\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"joke\",\n",
-    "    # This adds a new option, with name `poem`\n",
-    "    poem=PromptTemplate.from_template(\"Write a short poem about {topic}\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "1dcc7ccc",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"In the forest, where tall trees sway,\\nA creature roams, both fierce and gray.\\nWith mighty paws and piercing eyes,\\nThe bear, a symbol of strength, defies.\\n\\nThrough snow-kissed mountains, it does roam,\\nA guardian of its woodland home.\\nWith fur so thick, a shield of might,\\nIt braves the coldest winter night.\\n\\nA gentle giant, yet wild and free,\\nThe bear commands respect, you see.\\nWith every step, it leaves a trace,\\nOf untamed power and ancient grace.\\n\\nFrom honeyed feast to salmon's leap,\\nIt takes its place, in nature's keep.\\nA symbol of untamed delight,\\nThe bear, a wonder, day and night.\\n\\nSo let us honor this noble beast,\\nIn forests where its soul finds peace.\\nFor in its presence, we come to know,\\nThe untamed spirit that in us also flows.\")"
-      ]
-     },
-     "execution_count": 29,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can configure it write a poem with OpenAI\n",
-    "chain.with_config(configurable={\"prompt\": \"poem\", \"llm\": \"openai\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "e4ee9fbc",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Sure, here's a bear joke for you:\\n\\nWhy don't bears wear shoes?\\n\\nBecause they have bear feet!\")"
-      ]
-     },
-     "execution_count": 30,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can always just configure only one if we want\n",
-    "chain.with_config(configurable={\"llm\": \"openai\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "02fc4841",
-   "metadata": {},
-   "source": [
-    "### Saving configurations\n",
-    "\n",
-    "We can also easily save configured chains as their own objects"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "5cf53202",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "openai_poem = chain.with_config(configurable={\"llm\": \"openai\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 32,
-   "id": "9486d701",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\")"
-      ]
-     },
-     "execution_count": 32,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "openai_poem.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a43e3b70",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/fallbacks.ipynb

@@ -1,285 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "19c9cbd6",
-   "metadata": {},
-   "source": [
-    "# Add fallbacks\n",
-    "\n",
-    "There are many possible points of failure in an LLM application, whether that be issues with LLM API's, poor model outputs, issues with other integrations, etc. Fallbacks help you gracefully handle and isolate these issues.\n",
-    "\n",
-    "Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a6bb9ba9",
-   "metadata": {},
-   "source": [
-    "## Handling LLM API Errors\n",
-    "\n",
-    "This is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit rate limits, any number of things. Therefore, using fallbacks can help protect against these types of things.\n",
-    "\n",
-    "IMPORTANT: By default, a lot of the LLM wrappers catch errors and retry. You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying and not failing."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "d3e893bf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4847c82d",
-   "metadata": {},
-   "source": [
-    "First, let's mock out what happens if we hit a RateLimitError from OpenAI"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "dfdd8bf5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from unittest.mock import patch\n",
-    "from openai.error import RateLimitError"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "e6fdffc1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Note that we set max_retries = 0 to avoid retrying on RateLimits, etc\n",
-    "openai_llm = ChatOpenAI(max_retries=0)\n",
-    "anthropic_llm = ChatAnthropic()\n",
-    "llm = openai_llm.with_fallbacks([anthropic_llm])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 27,
-   "id": "584461ab",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Hit error\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Let's use just the OpenAI LLm first, to show that we run into an error\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(openai_llm.invoke(\"Why did the chicken cross the road?\"))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 28,
-   "id": "4fc1e673",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "content=' I don\\'t actually know why the chicken crossed the road, but here are some possible humorous answers:\\n\\n- To get to the other side!\\n\\n- It was too chicken to just stand there. \\n\\n- It wanted a change of scenery.\\n\\n- It wanted to show the possum it could be done.\\n\\n- It was on its way to a poultry farmers\\' convention.\\n\\nThe joke plays on the double meaning of \"the other side\" - literally crossing the road to the other side, or the \"other side\" meaning the afterlife. So it\\'s an anti-joke, with a silly or unexpected pun as the answer.' additional_kwargs={} example=False\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Now let's try with fallbacks to Anthropic\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(llm.invoke(\"Why did the the chicken cross the road?\"))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f00bea25",
-   "metadata": {},
-   "source": [
-    "We can use our \"LLM with Fallbacks\" as we would a normal LLM."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "4f8eaaa0",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "content=\" I don't actually know why the kangaroo crossed the road, but I'm happy to take a guess! Maybe the kangaroo was trying to get to the other side to find some tasty grass to eat. Or maybe it was trying to get away from a predator or other danger. Kangaroos do need to cross roads and other open areas sometimes as part of their normal activities. Whatever the reason, I'm sure the kangaroo looked both ways before hopping across!\" additional_kwargs={} example=False\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"You're a nice assistant who always includes a compliment in your response\"),\n",
-    "        (\"human\", \"Why did the {animal} cross the road\"),\n",
-    "    ]\n",
-    ")\n",
-    "chain = prompt | llm\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(chain.invoke({\"animal\": \"kangaroo\"}))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ef9f0f39-0b9f-4723-a394-f61c98c75d41",
-   "metadata": {},
-   "source": [
-    "### Specifying errors to handle\n",
-    "\n",
-    "We can also specify the errors to handle if we want to be more specific about when the fallback is invoked:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "e4069ca4-1c16-4915-9a8c-b2732869ae27",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Hit error\n"
-     ]
-    }
-   ],
-   "source": [
-    "llm = openai_llm.with_fallbacks([anthropic_llm], exceptions_to_handle=(KeyboardInterrupt,))\n",
-    "\n",
-    "chain = prompt | llm\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(chain.invoke({\"animal\": \"kangaroo\"}))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8d62241b",
-   "metadata": {},
-   "source": [
-    "## Fallbacks for Sequences\n",
-    "\n",
-    "We can also create fallbacks for sequences, that are sequences themselves. Here we do that with two different models: ChatOpenAI and then normal OpenAI (which does not use a chat model). Because OpenAI is NOT a chat model, you likely want a different prompt."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "6d0b8056",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# First let's create a chain with a ChatModel\n",
-    "# We add in a string output parser here so the outputs between the two are the same type\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "\n",
-    "chat_prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"You're a nice assistant who always includes a compliment in your response\"),\n",
-    "        (\"human\", \"Why did the {animal} cross the road\"),\n",
-    "    ]\n",
-    ")\n",
-    "# Here we're going to use a bad model name to easily create a chain that will error\n",
-    "chat_model = ChatOpenAI(model_name=\"gpt-fake\")\n",
-    "bad_chain = chat_prompt | chat_model | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "8d1fc2a5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Now lets create a chain with the normal OpenAI model\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "prompt_template = \"\"\"Instructions: You should always include a compliment in your response.\n",
-    "\n",
-    "Question: Why did the {animal} cross the road?\"\"\"\n",
-    "prompt = PromptTemplate.from_template(prompt_template)\n",
-    "llm = OpenAI()\n",
-    "good_chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 32,
-   "id": "283bfa44",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'\\n\\nAnswer: The turtle crossed the road to get to the other side, and I have to say he had some impressive determination.'"
-      ]
-     },
-     "execution_count": 32,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can now create a final chain which combines the two\n",
-    "chain = bad_chain.with_fallbacks([good_chain])\n",
-    "chain.invoke({\"animal\": \"turtle\"})"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/functions.ipynb

@@ -1,171 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "fbc4bf6e",
-   "metadata": {},
-   "source": [
-    "# Run arbitrary functions\n",
-    "\n",
-    "You can use arbitrary functions in the pipeline\n",
-    "\n",
-    "Note that all inputs to these functions need to be a SINGLE argument. If you have a function that accepts multiple arguments, you should write a wrapper that accepts a single input and unpacks it into multiple argument."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "6bb221b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableLambda\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from operator import itemgetter\n",
-    "\n",
-    "def length_function(text):\n",
-    "    return len(text)\n",
-    "\n",
-    "def _multiple_length_function(text1, text2):\n",
-    "    return len(text1) * len(text2)\n",
-    "\n",
-    "def multiple_length_function(_dict):\n",
-    "    return _multiple_length_function(_dict[\"text1\"], _dict[\"text2\"])\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_template(\"what is {a} + {b}\")\n",
-    "model = ChatOpenAI()\n",
-    "\n",
-    "chain1 = prompt | model\n",
-    "\n",
-    "chain = {\n",
-    "    \"a\": itemgetter(\"foo\") | RunnableLambda(length_function),\n",
-    "    \"b\": {\"text1\": itemgetter(\"foo\"), \"text2\": itemgetter(\"bar\")} | RunnableLambda(multiple_length_function)\n",
-    "} | prompt | model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "5488ec85",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='3 + 9 equals 12.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"foo\": \"bar\", \"bar\": \"gah\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4728ddd9-914d-42ce-ae9b-72c9ce8ec940",
-   "metadata": {},
-   "source": [
-    "## Accepting a Runnable Config\n",
-    "\n",
-    "Runnable lambdas can optionally accept a [RunnableConfig](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.config.RunnableConfig.html?highlight=runnableconfig#langchain.schema.runnable.config.RunnableConfig), which they can use to pass callbacks, tags, and other configuration information to nested runs."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "80b3b5f6-5d58-44b9-807e-cce9a46bf49f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableConfig\n",
-    "from langchain.schema.output_parser import StrOutputParser"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "ff0daf0c-49dd-4d21-9772-e5fa133c5f36",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import json\n",
-    "\n",
-    "def parse_or_fix(text: str, config: RunnableConfig):\n",
-    "    fixing_chain = (\n",
-    "        ChatPromptTemplate.from_template(\n",
-    "            \"Fix the following text:\\n\\n```text\\n{input}\\n```\\nError: {error}\"\n",
-    "            \" Don't narrate, just respond with the fixed data.\"\n",
-    "        )\n",
-    "        | ChatOpenAI()\n",
-    "        | StrOutputParser()\n",
-    "    )\n",
-    "    for _ in range(3):\n",
-    "        try:\n",
-    "            return json.loads(text)\n",
-    "        except Exception as e:\n",
-    "            text = fixing_chain.invoke({\"input\": text, \"error\": e}, config)\n",
-    "    return \"Failed to parse\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "1a5e709e-9d75-48c7-bb9c-503251990505",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Tokens Used: 65\n",
-      "\tPrompt Tokens: 56\n",
-      "\tCompletion Tokens: 9\n",
-      "Successful Requests: 1\n",
-      "Total Cost (USD): $0.00010200000000000001\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.callbacks import get_openai_callback\n",
-    "\n",
-    "with get_openai_callback() as cb:\n",
-    "    RunnableLambda(parse_or_fix).invoke(\"{foo: bar}\", {\"tags\": [\"my-tag\"], \"callbacks\": [cb]})\n",
-    "    print(cb)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "29f55c38",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/map.ipynb

@@ -1,199 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "b022ab74-794d-4c54-ad47-ff9549ddb9d2",
-   "metadata": {},
-   "source": [
-    "# Use RunnableParallel/RunnableMap\n",
-    "\n",
-    "RunnableParallel (aka. RunnableMap) makes it easy to execute multiple Runnables in parallel, and to return the output of these Runnables as a map."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "7e1873d6-d4b6-43ac-96a1-edcf178201e0",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'joke': AIMessage(content=\"Why don't bears wear shoes? \\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False),\n",
-       " 'poem': AIMessage(content=\"In woodland depths, bear prowls with might,\\nSilent strength, nature's sovereign, day and night.\", additional_kwargs={}, example=False)}"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema.runnable import RunnableParallel\n",
-    "\n",
-    "\n",
-    "model = ChatOpenAI()\n",
-    "joke_chain = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\") | model\n",
-    "poem_chain = ChatPromptTemplate.from_template(\"write a 2-line poem about {topic}\") | model\n",
-    "\n",
-    "map_chain = RunnableParallel(joke=joke_chain, poem=poem_chain)\n",
-    "\n",
-    "map_chain.invoke({\"topic\": \"bear\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "df867ae9-1cec-4c9e-9fef-21969b206af5",
-   "metadata": {},
-   "source": [
-    "## Manipulating outputs/inputs\n",
-    "Maps can be useful for manipulating the output of one Runnable to match the input format of the next Runnable in a sequence."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "267d1460-53c1-4fdb-b2c3-b6a1eb7fccff",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Harrison worked at Kensho.'"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "from langchain.vectorstores import FAISS\n",
-    "\n",
-    "vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings())\n",
-    "retriever = vectorstore.as_retriever()\n",
-    "template = \"\"\"Answer the question based only on the following context:\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n",
-    "\n",
-    "retrieval_chain = (\n",
-    "    {\"context\": retriever, \"question\": RunnablePassthrough()} \n",
-    "    | prompt \n",
-    "    | model \n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "\n",
-    "retrieval_chain.invoke(\"where did harrison work?\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "392cd4c4-e7ed-4ab8-934d-f7a4eca55ee1",
-   "metadata": {},
-   "source": [
-    "Here the input to prompt is expected to be a map with keys \"context\" and \"question\". The user input is just the question. So we need to get the context using our retriever and passthrough the user input under the \"question\" key.\n",
-    "\n",
-    "Note that when composing a RunnableMap when another Runnable we don't even need to wrap our dictionary in the RunnableMap class — the type conversion is handled for us."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "833da249-c0d4-4e5b-b3f8-cab549f0f7e1",
-   "metadata": {},
-   "source": [
-    "## Parallelism\n",
-    "\n",
-    "RunnableMaps are also useful for running independent processes in parallel, since each Runnable in the map is executed in parallel. For example, we can see our earlier `joke_chain`, `poem_chain` and `map_chain` all have about the same runtime, even though `map_chain` executes both of the other two."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "38e47834-45af-4281-991f-86f150001510",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "958 ms ± 402 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
-     ]
-    }
-   ],
-   "source": [
-    "%%timeit\n",
-    "\n",
-    "joke_chain.invoke({\"topic\": \"bear\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "d0cd40de-b37e-41fa-a2f6-8aaa49f368d6",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "1.22 s ± 508 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
-     ]
-    }
-   ],
-   "source": [
-    "%%timeit\n",
-    "\n",
-    "poem_chain.invoke({\"topic\": \"bear\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "799894e1-8e18-4a73-b466-f6aea6af3920",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "1.15 s ± 119 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
-     ]
-    }
-   ],
-   "source": [
-    "%%timeit\n",
-    "\n",
-    "map_chain.invoke({\"topic\": \"bear\"})\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/routing.ipynb

@@ -1,354 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "4b47436a",
-   "metadata": {},
-   "source": [
-    "# Route between multiple Runnables\n",
-    "\n",
-    "This notebook covers how to do routing in the LangChain Expression Language.\n",
-    "\n",
-    "Routing allows you to create non-deterministic chains where the output of a previous step defines the next step. Routing helps provide structure and consistency around interactions with LLMs.\n",
-    "\n",
-    "There are two ways to perform routing:\n",
-    "\n",
-    "1. Using a `RunnableBranch`.\n",
-    "2. Writing custom factory function that takes the input of a previous step and returns a **runnable**. Importantly, this should return a **runnable** and NOT actually execute.\n",
-    "\n",
-    "We'll illustrate both methods using a two step sequence where the first step classifies an input question as being about `LangChain`, `Anthropic`, or `Other`, then routes to a corresponding prompt chain."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f885113d",
-   "metadata": {},
-   "source": [
-    "## Using a RunnableBranch\n",
-    "\n",
-    "A `RunnableBranch` is initialized with a list of (condition, runnable) pairs and a default runnable. It selects which branch by passing each condition the input it's invoked with. It selects the first condition to evaluate to True, and runs the corresponding runnable to that condition with the input. \n",
-    "\n",
-    "If no provided conditions match, it runs the default runnable.\n",
-    "\n",
-    "Here's an example of what it looks like in action:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "1aa13c1d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chat_models import ChatAnthropic\n",
-    "from langchain.schema.output_parser import StrOutputParser"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ed84c59a",
-   "metadata": {},
-   "source": [
-    "First, let's create a chain that will identify incoming questions as being about `LangChain`, `Anthropic`, or `Other`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "3ec03886",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = PromptTemplate.from_template(\"\"\"Given the user question below, classify it as either being about `LangChain`, `Anthropic`, or `Other`.\n",
-    "                                     \n",
-    "Do not respond with more than one word.\n",
-    "\n",
-    "<question>\n",
-    "{question}\n",
-    "</question>\n",
-    "\n",
-    "Classification:\"\"\") | ChatAnthropic() | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "87ae7c1c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' Anthropic'"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"question\": \"how do I call Anthropic?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8aa0a365",
-   "metadata": {},
-   "source": [
-    "Now, let's create three sub chains:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "d479962a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "langchain_chain = PromptTemplate.from_template(\"\"\"You are an expert in langchain. \\\n",
-    "Always answer questions starting with \"As Harrison Chase told me\". \\\n",
-    "Respond to the following question:\n",
-    "\n",
-    "Question: {question}\n",
-    "Answer:\"\"\") | ChatAnthropic()\n",
-    "anthropic_chain = PromptTemplate.from_template(\"\"\"You are an expert in anthropic. \\\n",
-    "Always answer questions starting with \"As Dario Amodei told me\". \\\n",
-    "Respond to the following question:\n",
-    "\n",
-    "Question: {question}\n",
-    "Answer:\"\"\") | ChatAnthropic()\n",
-    "general_chain = PromptTemplate.from_template(\"\"\"Respond to the following question:\n",
-    "\n",
-    "Question: {question}\n",
-    "Answer:\"\"\") | ChatAnthropic()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "593eab06",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableBranch\n",
-    "\n",
-    "branch = RunnableBranch(\n",
-    "  (lambda x: \"anthropic\" in x[\"topic\"].lower(), anthropic_chain),\n",
-    "  (lambda x: \"langchain\" in x[\"topic\"].lower(), langchain_chain),\n",
-    "  general_chain\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "752c732e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "full_chain = {\n",
-    "    \"topic\": chain,\n",
-    "    \"question\": lambda x: x[\"question\"]\n",
-    "} | branch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "29231bb8",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" As Dario Amodei told me, here are some ways to use Anthropic:\\n\\n- Sign up for an account on Anthropic's website to access tools like Claude, Constitutional AI, and Writer. \\n\\n- Use Claude for tasks like email generation, customer service chat, and QA. Claude can understand natural language prompts and provide helpful responses.\\n\\n- Use Constitutional AI if you need an AI assistant that is harmless, honest, and helpful. It is designed to be safe and aligned with human values.\\n\\n- Use Writer to generate natural language content for things like marketing copy, stories, reports, and more. Give it a topic and prompt and it will create high-quality written content.\\n\\n- Check out Anthropic's documentation and blog for tips, tutorials, examples, and announcements about new capabilities as they continue to develop their AI technology.\\n\\n- Follow Anthropic on social media or subscribe to their newsletter to stay up to date on new features and releases.\\n\\n- For most people, the easiest way to leverage Anthropic's technology is through their website - just create an account to get started!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"how do I use Anthropic?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "c67d8733",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' As Harrison Chase told me, here is how you use LangChain:\\n\\nLangChain is an AI assistant that can have conversations, answer questions, and generate text. To use LangChain, you simply type or speak your input and LangChain will respond. \\n\\nYou can ask LangChain questions, have discussions, get summaries or explanations about topics, and request it to generate text on a subject. Some examples of interactions:\\n\\n- Ask general knowledge questions and LangChain will try to answer factually. For example \"What is the capital of France?\"\\n\\n- Have conversations on topics by taking turns speaking. You can prompt the start of a conversation by saying something like \"Let\\'s discuss machine learning\"\\n\\n- Ask for summaries or high-level explanations on subjects. For example \"Can you summarize the main themes in Shakespeare\\'s Hamlet?\" \\n\\n- Give creative writing prompts or requests to have LangChain generate text in different styles. For example \"Write a short children\\'s story about a mouse\" or \"Generate a poem in the style of Robert Frost about nature\"\\n\\n- Correct LangChain if it makes an inaccurate statement and provide the right information. This helps train it.\\n\\nThe key is interacting naturally and giving it clear prompts and requests', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"how do I use LangChain?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "935ad949",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' 2 + 2 = 4', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"whats 2 + 2\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6d8d042c",
-   "metadata": {},
-   "source": [
-    "## Using a custom function\n",
-    "\n",
-    "You can also use a custom function to route between different outputs. Here's an example:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "687492da",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def route(info):\n",
-    "    if \"anthropic\" in info[\"topic\"].lower():\n",
-    "        return anthropic_chain\n",
-    "    elif \"langchain\" in info[\"topic\"].lower():\n",
-    "        return langchain_chain\n",
-    "    else:\n",
-    "        return general_chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "02a33c86",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableLambda\n",
-    "\n",
-    "full_chain = {\n",
-    "    \"topic\": chain,\n",
-    "    \"question\": lambda x: x[\"question\"]\n",
-    "} | RunnableLambda(route)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "c2e977a4",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' As Dario Amodei told me, to use Anthropic IPC you first need to import it:\\n\\n```python\\nfrom anthroipc import ic\\n```\\n\\nThen you can create a client and connect to the server:\\n\\n```python \\nclient = ic.connect()\\n```\\n\\nAfter that, you can call methods on the client and get responses:\\n\\n```python\\nresponse = client.ask(\"What is the meaning of life?\")\\nprint(response)\\n```\\n\\nYou can also register callbacks to handle events: \\n\\n```python\\ndef on_poke(event):\\n  print(\"Got poked!\")\\n\\nclient.on(\\'poke\\', on_poke)\\n```\\n\\nAnd that\\'s the basics of using the Anthropic IPC client library for Python! Let me know if you have any other questions!', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"how do I use Anthroipc?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "48913dc6",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' As Harrison Chase told me, to use LangChain you first need to sign up for an API key at platform.langchain.com. Once you have your API key, you can install the Python library and write a simple Python script to call the LangChain API. Here is some sample code to get started:\\n\\n```python\\nimport langchain\\n\\napi_key = \"YOUR_API_KEY\"\\n\\nlangchain.set_key(api_key)\\n\\nresponse = langchain.ask(\"What is the capital of France?\")\\n\\nprint(response.response)\\n```\\n\\nThis will send the question \"What is the capital of France?\" to the LangChain API and print the response. You can customize the request by providing parameters like max_tokens, temperature, etc. The LangChain Python library documentation has more details on the available options. The key things are getting an API key and calling langchain.ask() with your question text. Let me know if you have any other questions!', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"how do I use LangChain?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "a14d0dca",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' 4', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "full_chain.invoke({\"question\": \"whats 2 + 2\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "46802d04",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/index.mdx

@@ -1,36 +0,0 @@
----
-sidebar_class_name: hidden
----
-
-# LangChain Expression Language (LCEL)
-
-LangChain Expression Language or LCEL is a declarative way to easily compose chains together.
-There are several benefits to writing chains in this manner (as opposed to writing normal code):
-
-**Async, Batch, and Streaming Support**
-Any chain constructed this way will automatically have full sync, async, batch, and streaming support.
-This makes it easy to prototype a chain in a Jupyter notebook using the sync interface, and then expose it as an async streaming interface.
-
-**Fallbacks**
-The non-determinism of LLMs makes it important to be able to handle errors gracefully.
-With LCEL you can easily attach fallbacks to any chain.
-
-**Parallelism**
-Since LLM applications involve (sometimes long) API calls, it often becomes important to run things in parallel.
-With LCEL syntax, any components that can be run in parallel automatically are.
-
-**Seamless LangSmith Tracing Integration**
-As your chains get more and more complex, it becomes increasingly important to understand what exactly is happening at every step.
-With LCEL, **all** steps are automatically logged to [LangSmith](https://smith.langchain.com) for maximal observability and debuggability.
-
-#### [Interface](/docs/expression_language/interface)
-The base interface shared by all LCEL objects
-
-#### [How to](/docs/expression_language/how_to)
-How to use core features of LCEL
-
-#### [Cookbook](/docs/expression_language/cookbook)
-Examples of common LCEL usage patterns
-
-#### [Why use LCEL](/docs/expression_language/why)
-A deeper dive into the benefits of LCEL

docs/docs/expression_language/interface.ipynb

@@ -1,933 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "366a0e68-fd67-4fe5-a292-5c33733339ea",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 0\n",
-    "title: Interface\n",
-    "---\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9a9acd2e",
-   "metadata": {},
-   "source": [
-    "In an effort to make it as easy as possible to create custom chains, we've implemented a [\"Runnable\"](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.Runnable.html#langchain.schema.runnable.base.Runnable) protocol that most components implement. This is a standard interface with a few different methods, which makes it easy to define custom chains as well as making it possible to invoke them in a standard way. The standard interface exposed includes:\n",
-    "\n",
-    "- [`stream`](#stream): stream back chunks of the response\n",
-    "- [`invoke`](#invoke): call the chain on an input\n",
-    "- [`batch`](#batch): call the chain on a list of inputs\n",
-    "\n",
-    "These also have corresponding async methods:\n",
-    "\n",
-    "- [`astream`](#async-stream): stream back chunks of the response async\n",
-    "- [`ainvoke`](#async-invoke): call the chain on an input async\n",
-    "- [`abatch`](#async-batch): call the chain on a list of inputs async\n",
-    "- [`astream_log`](#async-stream-intermediate-steps): stream back intermediate steps as they happen, in addition to the final response\n",
-    "\n",
-    "The type of the input varies by component:\n",
-    "\n",
-    "| Component | Input Type |\n",
-    "| --- | --- |\n",
-    "|Prompt|Dictionary|\n",
-    "|Retriever|Single string|\n",
-    "|LLM, ChatModel| Single string, list of chat messages or a PromptValue|\n",
-    "|Tool|Single string, or dictionary, depending on the tool|\n",
-    "|OutputParser|The output of an LLM or ChatModel|\n",
-    "\n",
-    "The output type also varies by component:\n",
-    "\n",
-    "| Component | Output Type |\n",
-    "| --- | --- |\n",
-    "| LLM | String |\n",
-    "| ChatModel | ChatMessage |\n",
-    "| Prompt | PromptValue |\n",
-    "| Retriever | List of documents |\n",
-    "| Tool | Depends on the tool |\n",
-    "| OutputParser | Depends on the parser |\n",
-    "\n",
-    "All runnables expose properties to inspect the input and output types:\n",
-    "- [`input_schema`](#input-schema): an input Pydantic model auto-generated from the structure of the Runnable\n",
-    "- [`output_schema`](#output-schema): an output Pydantic model auto-generated from the structure of the Runnable\n",
-    "\n",
-    "Let's take a look at these methods! To do so, we'll create a super simple PromptTemplate + ChatModel chain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "466b65b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "3c634ef0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = ChatOpenAI()\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "d1850a1f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\")\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "56d0669f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "5cccdf0b-2d89-4f74-9530-bf499610e9a5",
-   "metadata": {},
-   "source": [
-    "## Input Schema\n",
-    "\n",
-    "A description of the inputs accepted by a Runnable.\n",
-    "This is a Pydantic model dynamically generated from the structure of any Runnable.\n",
-    "You can call `.schema()` on it to obtain a JSONSchema representation."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "25e146d4-60da-40a2-9026-b5dfee106a3f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'title': 'PromptInput',\n",
-       " 'type': 'object',\n",
-       " 'properties': {'topic': {'title': 'Topic', 'type': 'string'}}}"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# The input schema of the chain is the input schema of its first part, the prompt.\n",
-    "chain.input_schema.schema()\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "5059a5dc-d544-4add-85bd-78a3f2b78b9a",
-   "metadata": {},
-   "source": [
-    "## Output Schema\n",
-    "\n",
-    "A description of the outputs produced by a Runnable.\n",
-    "This is a Pydantic model dynamically generated from the structure of any Runnable.\n",
-    "You can call `.schema()` on it to obtain a JSONSchema representation."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "a0e41fd3-77d8-4911-af6a-d4d3aad5f77b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'title': 'ChatOpenAIOutput',\n",
-       " 'anyOf': [{'$ref': '#/definitions/HumanMessageChunk'},\n",
-       "  {'$ref': '#/definitions/AIMessageChunk'},\n",
-       "  {'$ref': '#/definitions/ChatMessageChunk'},\n",
-       "  {'$ref': '#/definitions/FunctionMessageChunk'},\n",
-       "  {'$ref': '#/definitions/SystemMessageChunk'}],\n",
-       " 'definitions': {'HumanMessageChunk': {'title': 'HumanMessageChunk',\n",
-       "   'description': 'A Human Message chunk.',\n",
-       "   'type': 'object',\n",
-       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
-       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
-       "    'type': {'title': 'Type',\n",
-       "     'default': 'human',\n",
-       "     'enum': ['human'],\n",
-       "     'type': 'string'},\n",
-       "    'example': {'title': 'Example', 'default': False, 'type': 'boolean'},\n",
-       "    'is_chunk': {'title': 'Is Chunk',\n",
-       "     'default': True,\n",
-       "     'enum': [True],\n",
-       "     'type': 'boolean'}},\n",
-       "   'required': ['content']},\n",
-       "  'AIMessageChunk': {'title': 'AIMessageChunk',\n",
-       "   'description': 'A Message chunk from an AI.',\n",
-       "   'type': 'object',\n",
-       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
-       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
-       "    'type': {'title': 'Type',\n",
-       "     'default': 'ai',\n",
-       "     'enum': ['ai'],\n",
-       "     'type': 'string'},\n",
-       "    'example': {'title': 'Example', 'default': False, 'type': 'boolean'},\n",
-       "    'is_chunk': {'title': 'Is Chunk',\n",
-       "     'default': True,\n",
-       "     'enum': [True],\n",
-       "     'type': 'boolean'}},\n",
-       "   'required': ['content']},\n",
-       "  'ChatMessageChunk': {'title': 'ChatMessageChunk',\n",
-       "   'description': 'A Chat Message chunk.',\n",
-       "   'type': 'object',\n",
-       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
-       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
-       "    'type': {'title': 'Type',\n",
-       "     'default': 'chat',\n",
-       "     'enum': ['chat'],\n",
-       "     'type': 'string'},\n",
-       "    'role': {'title': 'Role', 'type': 'string'},\n",
-       "    'is_chunk': {'title': 'Is Chunk',\n",
-       "     'default': True,\n",
-       "     'enum': [True],\n",
-       "     'type': 'boolean'}},\n",
-       "   'required': ['content', 'role']},\n",
-       "  'FunctionMessageChunk': {'title': 'FunctionMessageChunk',\n",
-       "   'description': 'A Function Message chunk.',\n",
-       "   'type': 'object',\n",
-       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
-       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
-       "    'type': {'title': 'Type',\n",
-       "     'default': 'function',\n",
-       "     'enum': ['function'],\n",
-       "     'type': 'string'},\n",
-       "    'name': {'title': 'Name', 'type': 'string'},\n",
-       "    'is_chunk': {'title': 'Is Chunk',\n",
-       "     'default': True,\n",
-       "     'enum': [True],\n",
-       "     'type': 'boolean'}},\n",
-       "   'required': ['content', 'name']},\n",
-       "  'SystemMessageChunk': {'title': 'SystemMessageChunk',\n",
-       "   'description': 'A System Message chunk.',\n",
-       "   'type': 'object',\n",
-       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
-       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
-       "    'type': {'title': 'Type',\n",
-       "     'default': 'system',\n",
-       "     'enum': ['system'],\n",
-       "     'type': 'string'},\n",
-       "    'is_chunk': {'title': 'Is Chunk',\n",
-       "     'default': True,\n",
-       "     'enum': [True],\n",
-       "     'type': 'boolean'}},\n",
-       "   'required': ['content']}}}"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# The output schema of the chain is the output schema of its last part, in this case a ChatModel, which outputs a ChatMessage\n",
-    "chain.output_schema.schema()\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "daf2b2b2",
-   "metadata": {},
-   "source": [
-    "## Stream"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "bea9639d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Why don't bears wear shoes? \n",
-      "\n",
-      "Because they have bear feet!"
-     ]
-    }
-   ],
-   "source": [
-    "for s in chain.stream({\"topic\": \"bears\"}):\n",
-    "    print(s.content, end=\"\", flush=True)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cbf1c782",
-   "metadata": {},
-   "source": [
-    "## Invoke"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "470e483f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\")"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"topic\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "88f0c279",
-   "metadata": {},
-   "source": [
-    "## Batch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "9685de67",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\"),\n",
-       " AIMessage(content=\"Why don't cats play poker in the wild?\\n\\nToo many cheetahs!\")]"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2434ab15",
-   "metadata": {},
-   "source": [
-    "You can set the number of concurrent requests by using the `max_concurrency` parameter"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "a08522f6",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\"),\n",
-       " AIMessage(content=\"Sure, here's a cat joke for you:\\n\\nWhy don't cats play poker in the wild?\\n\\nToo many cheetahs!\")]"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}], config={\"max_concurrency\": 5})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b960cbfe",
-   "metadata": {},
-   "source": [
-    "## Async Stream"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "ea35eee4",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Sure, here's a bear joke for you:\n",
-      "\n",
-      "Why don't bears wear shoes?\n",
-      "\n",
-      "Because they have bear feet!"
-     ]
-    }
-   ],
-   "source": [
-    "async for s in chain.astream({\"topic\": \"bears\"}):\n",
-    "    print(s.content, end=\"\", flush=True)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "04cb3324",
-   "metadata": {},
-   "source": [
-    "## Async Invoke"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "ef8c9b20",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes? \\n\\nBecause they have bear feet!\")"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "await chain.ainvoke({\"topic\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3da288d5",
-   "metadata": {},
-   "source": [
-    "## Async Batch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "eba2a103",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\")]"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "await chain.abatch([{\"topic\": \"bears\"}])\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f9cef104",
-   "metadata": {},
-   "source": [
-    "## Async Stream Intermediate Steps\n",
-    "\n",
-    "All runnables also have a method `.astream_log()` which can be used to stream (as they happen) all or part of the intermediate steps of your chain/sequence. \n",
-    "\n",
-    "This is useful eg. to show progress to the user, to use intermediate results, or even just to debug your chain.\n",
-    "\n",
-    "You can choose to stream all steps (default), or include/exclude steps by name, tags or metadata.\n",
-    "\n",
-    "This method yields [JSONPatch](https://jsonpatch.com) ops that when applied in the same order as received build up the RunState.\n",
-    "\n",
-    "```python\n",
-    "class LogEntry(TypedDict):\n",
-    "    id: str\n",
-    "    \"\"\"ID of the sub-run.\"\"\"\n",
-    "    name: str\n",
-    "    \"\"\"Name of the object being run.\"\"\"\n",
-    "    type: str\n",
-    "    \"\"\"Type of the object being run, eg. prompt, chain, llm, etc.\"\"\"\n",
-    "    tags: List[str]\n",
-    "    \"\"\"List of tags for the run.\"\"\"\n",
-    "    metadata: Dict[str, Any]\n",
-    "    \"\"\"Key-value pairs of metadata for the run.\"\"\"\n",
-    "    start_time: str\n",
-    "    \"\"\"ISO-8601 timestamp of when the run started.\"\"\"\n",
-    "\n",
-    "    streamed_output_str: List[str]\n",
-    "    \"\"\"List of LLM tokens streamed by this run, if applicable.\"\"\"\n",
-    "    final_output: Optional[Any]\n",
-    "    \"\"\"Final output of this run.\n",
-    "    Only available after the run has finished successfully.\"\"\"\n",
-    "    end_time: Optional[str]\n",
-    "    \"\"\"ISO-8601 timestamp of when the run ended.\n",
-    "    Only available after the run has finished.\"\"\"\n",
-    "\n",
-    "\n",
-    "class RunState(TypedDict):\n",
-    "    id: str\n",
-    "    \"\"\"ID of the run.\"\"\"\n",
-    "    streamed_output: List[Any]\n",
-    "    \"\"\"List of output chunks streamed by Runnable.stream()\"\"\"\n",
-    "    final_output: Optional[Any]\n",
-    "    \"\"\"Final output of the run, usually the result of aggregating (`+`) streamed_output.\n",
-    "    Only available after the run has finished successfully.\"\"\"\n",
-    "\n",
-    "    logs: Dict[str, LogEntry]\n",
-    "    \"\"\"Map of run names to sub-runs. If filters were supplied, this list will\n",
-    "    contain only the runs that matched the filters.\"\"\"\n",
-    "```"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a146a5df-25be-4fa2-a7e4-df8ebe55a35e",
-   "metadata": {},
-   "source": [
-    "### Streaming JSONPatch chunks\n",
-    "\n",
-    "This is useful eg. to stream the JSONPatch in an HTTP server, and then apply the ops on the client to rebuild the run state there. See [LangServe](https://github.com/langchain-ai/langserve) for tooling to make it easier to build a webserver from any Runnable."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "21c9019e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "RunLogPatch({'op': 'replace',\n",
-      "  'path': '',\n",
-      "  'value': {'final_output': None,\n",
-      "            'id': 'fd6fcf62-c92c-4edf-8713-0fc5df000f62',\n",
-      "            'logs': {},\n",
-      "            'streamed_output': []}})\n",
-      "RunLogPatch({'op': 'add',\n",
-      "  'path': '/logs/Docs',\n",
-      "  'value': {'end_time': None,\n",
-      "            'final_output': None,\n",
-      "            'id': '8c998257-1ec8-4546-b744-c3fdb9728c41',\n",
-      "            'metadata': {},\n",
-      "            'name': 'Docs',\n",
-      "            'start_time': '2023-10-05T12:52:35.668',\n",
-      "            'streamed_output_str': [],\n",
-      "            'tags': ['map:key:context', 'FAISS'],\n",
-      "            'type': 'retriever'}})\n",
-      "RunLogPatch({'op': 'add',\n",
-      "  'path': '/logs/Docs/final_output',\n",
-      "  'value': {'documents': [Document(page_content='harrison worked at kensho')]}},\n",
-      " {'op': 'add',\n",
-      "  'path': '/logs/Docs/end_time',\n",
-      "  'value': '2023-10-05T12:52:36.033'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ''})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'H'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'arrison'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' worked'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' at'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' Kens'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'ho'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': '.'})\n",
-      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ''})\n",
-      "RunLogPatch({'op': 'replace',\n",
-      "  'path': '/final_output',\n",
-      "  'value': {'output': 'Harrison worked at Kensho.'}})\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "from langchain.vectorstores import FAISS\n",
-    "\n",
-    "template = \"\"\"Answer the question based only on the following context:\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n",
-    "\n",
-    "vectorstore = FAISS.from_texts([\"harrison worked at kensho\"], embedding=OpenAIEmbeddings())\n",
-    "retriever = vectorstore.as_retriever()\n",
-    "\n",
-    "retrieval_chain = (\n",
-    "    {\"context\": retriever.with_config(run_name='Docs'), \"question\": RunnablePassthrough()}\n",
-    "    | prompt \n",
-    "    | model \n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "\n",
-    "async for chunk in retrieval_chain.astream_log(\"where did harrison work?\", include_names=['Docs']):\n",
-    "    print(chunk)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "19570f36-7126-4fe2-b209-0cc6178b4582",
-   "metadata": {},
-   "source": [
-    "### Streaming the incremental RunState\n",
-    "\n",
-    "You can simply pass diff=False to get incremental values of RunState."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "5c26b731-b4eb-4967-a42a-dec813249ecb",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {},\n",
-      " 'streamed_output': []})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': None,\n",
-      "                   'final_output': None,\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': []})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': []})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H', 'arrison']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H', 'arrison', ' worked']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens', 'ho']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens', 'ho', '.']})\n",
-      "RunLog({'final_output': None,\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['',\n",
-      "                     'H',\n",
-      "                     'arrison',\n",
-      "                     ' worked',\n",
-      "                     ' at',\n",
-      "                     ' Kens',\n",
-      "                     'ho',\n",
-      "                     '.',\n",
-      "                     '']})\n",
-      "RunLog({'final_output': {'output': 'Harrison worked at Kensho.'},\n",
-      " 'id': 'f95ccb87-31f1-48ea-a51c-d2dadde44185',\n",
-      " 'logs': {'Docs': {'end_time': '2023-10-05T12:52:37.217',\n",
-      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
-      "                   'id': '621597dd-d716-4532-938d-debc21a453d1',\n",
-      "                   'metadata': {},\n",
-      "                   'name': 'Docs',\n",
-      "                   'start_time': '2023-10-05T12:52:36.935',\n",
-      "                   'streamed_output_str': [],\n",
-      "                   'tags': ['map:key:context', 'FAISS'],\n",
-      "                   'type': 'retriever'}},\n",
-      " 'streamed_output': ['',\n",
-      "                     'H',\n",
-      "                     'arrison',\n",
-      "                     ' worked',\n",
-      "                     ' at',\n",
-      "                     ' Kens',\n",
-      "                     'ho',\n",
-      "                     '.',\n",
-      "                     '']})\n"
-     ]
-    }
-   ],
-   "source": [
-    "async for chunk in retrieval_chain.astream_log(\"where did harrison work?\", include_names=['Docs'], diff=False):\n",
-    "    print(chunk)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7006f1aa",
-   "metadata": {},
-   "source": [
-    "## Parallelism\n",
-    "\n",
-    "Let's take a look at how LangChain Expression Language support parallel requests as much as possible. For example, when using a RunnableParallel (often written as a dictionary) it executes each element in parallel."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "0a1c409d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema.runnable import RunnableParallel\n",
-    "chain1 = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\") | model\n",
-    "chain2 = ChatPromptTemplate.from_template(\"write a short (2 line) poem about {topic}\") | model\n",
-    "combined = RunnableParallel(joke=chain1, poem=chain2)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "08044c0a",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "CPU times: user 31.7 ms, sys: 8.59 ms, total: 40.3 ms\n",
-      "Wall time: 1.05 s\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears like fast food?\\n\\nBecause they can't catch it!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "%%time\n",
-    "chain1.invoke({\"topic\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "22c56804",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "CPU times: user 42.9 ms, sys: 10.2 ms, total: 53 ms\n",
-      "Wall time: 1.93 s\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"In forest's embrace, bears roam free,\\nSilent strength, nature's majesty.\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "%%time\n",
-    "chain2.invoke({\"topic\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "4fff4cbb",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "CPU times: user 96.3 ms, sys: 20.4 ms, total: 117 ms\n",
-      "Wall time: 1.1 s\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'joke': AIMessage(content=\"Why don't bears wear socks?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False),\n",
-       " 'poem': AIMessage(content=\"In forest's embrace,\\nMajestic bears leave their trace.\", additional_kwargs={}, example=False)}"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "%%time\n",
-    "combined.invoke({\"topic\": \"bears\"})\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fab75d1d",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/why.mdx

@@ -1,11 +0,0 @@
-# Why use LCEL?
-
-The LangChain Expression Language was designed from day 1 to **support putting prototypes in production, with no code changes**, from the simplest “prompt + LLM” chain to the most complex chains (we’ve seen folks successfully running in production LCEL chains with 100s of steps). To highlight a few of the reasons you might want to use LCEL:
-
-- first-class support for streaming: when you build your chains with LCEL you get the best possible time-to-first-token (time elapsed until the first chunk of output comes out). For some chains this means eg. we stream tokens straight from an LLM to a streaming output parser, and you get back parsed, incremental chunks of output at the same rate as the LLM provider outputs the raw tokens. We’re constantly improving streaming support, recently we added a [streaming JSON parser](https://twitter.com/LangChainAI/status/1709690468030914584), and more is in the works.
-- first-class async support: any chain built with LCEL can be called both with the synchronous API (eg. in your Jupyter notebook while prototyping) as well as with the asynchronous API (eg. in a [LangServe](https://github.com/langchain-ai/langserve) server). This enables using the same code for prototypes and in production, with great performance, and the ability to handle many concurrent requests in the same server.
-- optimised parallel execution: whenever your LCEL chains have steps that can be executed in parallel (eg if you fetch documents from multiple retrievers) we automatically do it, both in the sync and the async interfaces, for the smallest possible latency.
-- support for retries and fallbacks: more recently we’ve added support for configuring retries and fallbacks for any part of your LCEL chain. This is a great way to make your chains more reliable at scale. We’re currently working on adding streaming support for retries/fallbacks, so you can get the added reliability without any latency cost.
-- accessing intermediate results: for more complex chains it’s often very useful to access the results of intermediate steps even before the final output is produced. This can be used let end-users know something is happening, or even just to debug your chain. We’ve added support for [streaming intermediate results](https://x.com/LangChainAI/status/1711806009097044193?s=20), and it’s available on every LangServe server.
-- [input and output schemas](https://x.com/LangChainAI/status/1711805322195861934?s=20): input and output schemas give every LCEL chain Pydantic and JSONSchema schemas inferred from the structure of your chain. This can be used for validation of inputs and outputs, and is an integral part of LangServe.
-- tracing with LangSmith: all chains built with LCEL have first-class tracing support, which can be used to debug your chains, or to understand what’s happening in production. To enable this all you have to do is add your [LangSmith](https://www.langchain.com/langsmith) API key as an environment variable.

docs/docs/guides/adapters/_category_.yml

@@ -1 +0,0 @@
-label: 'Adapters'

docs/docs/guides/adapters/openai.ipynb

@@ -1,323 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "700a516b",
-   "metadata": {},
-   "source": [
-    "# OpenAI Adapter\n",
-    "\n",
-    "A lot of people get started with OpenAI but want to explore other models. LangChain's integrations with many model providers make this easy to do so. While LangChain has it's own message and model APIs, we've also made it as easy as possible to explore other models by exposing an adapter to adapt LangChain models to the OpenAI api.\n",
-    "\n",
-    "At the moment this only deals with output and does not return other information (token counts, stop reasons, etc)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "6017f26a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import openai\n",
-    "from langchain.adapters import openai as lc_openai"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b522ceda",
-   "metadata": {},
-   "source": [
-    "## ChatCompletion.create"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "1d22eb61",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "messages = [{\"role\": \"user\", \"content\": \"hi\"}]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d550d3ad",
-   "metadata": {},
-   "source": [
-    "Original OpenAI call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "e1d27dfa",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "result = openai.ChatCompletion.create(\n",
-    "    messages=messages, \n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "012d81ae",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "result[\"choices\"][0]['message'].to_dict_recursive()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "db5b5500",
-   "metadata": {},
-   "source": [
-    "LangChain OpenAI wrapper call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "87c2d515",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "lc_result = lc_openai.ChatCompletion.create(\n",
-    "    messages=messages, \n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "c67a5ac8",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}"
-      ]
-     },
-     "execution_count": 17,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "lc_result[\"choices\"][0]['message']"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "034ba845",
-   "metadata": {},
-   "source": [
-    "Swapping out model providers"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "7a2c011c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "lc_result = lc_openai.ChatCompletion.create(\n",
-    "    messages=messages, \n",
-    "    model=\"claude-2\", \n",
-    "    temperature=0, \n",
-    "    provider=\"ChatAnthropic\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "f7c94827",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'role': 'assistant', 'content': ' Hello!'}"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "lc_result[\"choices\"][0]['message']"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cb3f181d",
-   "metadata": {},
-   "source": [
-    "## ChatCompletion.stream"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f7b8cd18",
-   "metadata": {},
-   "source": [
-    "Original OpenAI call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "fd8cb1ea",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'role': 'assistant', 'content': ''}\n",
-      "{'content': 'Hello'}\n",
-      "{'content': '!'}\n",
-      "{'content': ' How'}\n",
-      "{'content': ' can'}\n",
-      "{'content': ' I'}\n",
-      "{'content': ' assist'}\n",
-      "{'content': ' you'}\n",
-      "{'content': ' today'}\n",
-      "{'content': '?'}\n",
-      "{}\n"
-     ]
-    }
-   ],
-   "source": [
-    "for c in openai.ChatCompletion.create(\n",
-    "    messages = messages,\n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0,\n",
-    "    stream=True\n",
-    "):\n",
-    "    print(c[\"choices\"][0]['delta'].to_dict_recursive())"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0b2a076b",
-   "metadata": {},
-   "source": [
-    "LangChain OpenAI wrapper call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "9521218c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'role': 'assistant', 'content': ''}\n",
-      "{'content': 'Hello'}\n",
-      "{'content': '!'}\n",
-      "{'content': ' How'}\n",
-      "{'content': ' can'}\n",
-      "{'content': ' I'}\n",
-      "{'content': ' assist'}\n",
-      "{'content': ' you'}\n",
-      "{'content': ' today'}\n",
-      "{'content': '?'}\n",
-      "{}\n"
-     ]
-    }
-   ],
-   "source": [
-    "for c in lc_openai.ChatCompletion.create(\n",
-    "    messages = messages,\n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0,\n",
-    "    stream=True\n",
-    "):\n",
-    "    print(c[\"choices\"][0]['delta'])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0fc39750",
-   "metadata": {},
-   "source": [
-    "Swapping out model providers"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "68f0214e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'role': 'assistant', 'content': ' Hello'}\n",
-      "{'content': '!'}\n",
-      "{}\n"
-     ]
-    }
-   ],
-   "source": [
-    "for c in lc_openai.ChatCompletion.create(\n",
-    "    messages = messages,\n",
-    "    model=\"claude-2\", \n",
-    "    temperature=0,\n",
-    "    stream=True,\n",
-    "    provider=\"ChatAnthropic\",\n",
-    "):\n",
-    "    print(c[\"choices\"][0]['delta'])"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/guides/evaluation/comparison/index.mdx

@@ -1,28 +0,0 @@
----
-sidebar_position: 3 
----
-# Comparison Evaluators
-
-Comparison evaluators in LangChain help measure two different chains or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.
-
-These evaluators inherit from the `PairwiseStringEvaluator` class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.
-
-To create a custom comparison evaluator, inherit from the `PairwiseStringEvaluator` class and overwrite the `_evaluate_string_pairs` method. If you require asynchronous evaluation, also overwrite the `_aevaluate_string_pairs` method.
-
-Here's a summary of the key methods and properties of a comparison evaluator:
-
-- `evaluate_string_pairs`: Evaluate the output string pairs. This function should be overwritten when creating custom evaluators.
-- `aevaluate_string_pairs`: Asynchronously evaluate the output string pairs. This function should be overwritten for asynchronous evaluation.
-- `requires_input`: This property indicates whether this evaluator requires an input string.
-- `requires_reference`: This property specifies whether this evaluator requires a reference label.
-
-:::note LangSmith Support
-The [run_on_dataset](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.smith) evaluation method is designed to evaluate only a single model at a time, and thus, doesn't support these evaluators.
-:::
-
-Detailed information about creating custom evaluators and the available built-in comparison evaluators is provided in the following sections.
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />
-

docs/docs/guides/evaluation/comparison/pairwise_string.ipynb

@@ -1,392 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "dcfcf124-78fe-4d67-85a4-cfd3409a1ff6",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 0\n",
-    "title: Pairwise string comparison\n",
-    "---"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2da95378",
-   "metadata": {},
-   "source": [
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/comparison/pairwise_string.ipynb)\n",
-    "\n",
-    "Often you will want to compare predictions of an LLM, Chain, or Agent for a given input. The `StringComparison` evaluators facilitate this so you can answer questions like:\n",
-    "\n",
-    "- Which LLM or prompt produces a preferred output for a given question?\n",
-    "- Which examples should I include for few-shot example selection?\n",
-    "- Which output is better to include for fine-tuning?\n",
-    "\n",
-    "The simplest and often most reliable automated way to choose a preferred prediction for a given input is to use the `pairwise_string` evaluator.\n",
-    "\n",
-    "Check out the reference docs for the [PairwiseStringEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain) for more info."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "f6790c46",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.evaluation import load_evaluator\n",
-    "\n",
-    "evaluator = load_evaluator(\"labeled_pairwise_string\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "49ad9139",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'reasoning': 'Both responses are relevant to the question asked, as they both provide a numerical answer to the question about the number of dogs in the park. However, Response A is incorrect according to the reference answer, which states that there are four dogs. Response B, on the other hand, is correct as it matches the reference answer. Neither response demonstrates depth of thought, as they both simply provide a numerical answer without any additional information or context. \\n\\nBased on these criteria, Response B is the better response.\\n',\n",
-       " 'value': 'B',\n",
-       " 'score': 0}"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "evaluator.evaluate_string_pairs(\n",
-    "    prediction=\"there are three dogs\",\n",
-    "    prediction_b=\"4\",\n",
-    "    input=\"how many dogs are in the park?\",\n",
-    "    reference=\"four\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7491d2e6-4e77-4b17-be6b-7da966785c1d",
-   "metadata": {},
-   "source": [
-    "## Methods\n",
-    "\n",
-    "\n",
-    "The pairwise string evaluator can be called using [evaluate_string_pairs](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.evaluate_string_pairs) (or async [aevaluate_string_pairs](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.html#langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain.aevaluate_string_pairs)) methods, which accept:\n",
-    "\n",
-    "- prediction (str) – The predicted response of the first model, chain, or prompt.\n",
-    "- prediction_b (str) – The predicted response of the second model, chain, or prompt.\n",
-    "- input (str) – The input question, prompt, or other text.\n",
-    "- reference (str) – (Only for the labeled_pairwise_string variant) The reference response.\n",
-    "\n",
-    "They return a dictionary with the following values:\n",
-    "- value: 'A' or 'B', indicating whether `prediction` or `prediction_b` is preferred, respectively\n",
-    "- score: Integer 0 or 1 mapped from the 'value', where a score of 1 would mean that the first `prediction` is preferred, and a score of 0 would mean `prediction_b` is preferred.\n",
-    "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ed353b93-be71-4479-b9c0-8c97814c2e58",
-   "metadata": {},
-   "source": [
-    "## Without References\n",
-    "\n",
-    "When references aren't available, you can still predict the preferred response.\n",
-    "The results will reflect the evaluation model's preference, which is less reliable and may result\n",
-    "in preferences that are factually incorrect."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "586320da",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.evaluation import load_evaluator\n",
-    "\n",
-    "evaluator = load_evaluator(\"pairwise_string\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "7f56c76e-a39b-4509-8b8a-8a2afe6c3da1",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'reasoning': 'Both responses are correct and relevant to the question. However, Response B is more helpful and insightful as it provides a more detailed explanation of what addition is. Response A is correct but lacks depth as it does not explain what the operation of addition entails. \\n\\nFinal Decision: [[B]]',\n",
-       " 'value': 'B',\n",
-       " 'score': 0}"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "evaluator.evaluate_string_pairs(\n",
-    "    prediction=\"Addition is a mathematical operation.\",\n",
-    "    prediction_b=\"Addition is a mathematical operation that adds two numbers to create a third number, the 'sum'.\",\n",
-    "    input=\"What is addition?\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4a09b21d-9851-47e8-93d3-90044b2945b0",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "## Defining the Criteria\n",
-    "\n",
-    "By default, the LLM is instructed to select the 'preferred' response based on helpfulness, relevance, correctness, and depth of thought. You can customize the criteria by passing in a `criteria` argument, where the criteria could take any of the following forms:\n",
-    "- [`Criteria`](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.Criteria.html#langchain.evaluation.criteria.eval_chain.Criteria) enum or its string value - to use one of the default criteria and their descriptions\n",
-    "- [Constitutional principal](https://api.python.langchain.com/en/latest/chains/langchain.chains.constitutional_ai.models.ConstitutionalPrinciple.html#langchain.chains.constitutional_ai.models.ConstitutionalPrinciple) - use one any of the constitutional principles defined in langchain\n",
-    "- Dictionary: a list of custom criteria, where the key is the name of the criteria, and the value is the description.\n",
-    "- A list of criteria or constitutional principles - to combine multiple criteria in one.\n",
-    "\n",
-    "Below is an example for determining preferred writing responses based on a custom style."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "8539e7d9-f7b0-4d32-9c45-593a7915c093",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "custom_criteria = {\n",
-    "    \"simplicity\": \"Is the language straightforward and unpretentious?\",\n",
-    "    \"clarity\": \"Are the sentences clear and easy to understand?\",\n",
-    "    \"precision\": \"Is the writing precise, with no unnecessary words or details?\",\n",
-    "    \"truthfulness\": \"Does the writing feel honest and sincere?\",\n",
-    "    \"subtext\": \"Does the writing suggest deeper meanings or themes?\",\n",
-    "}\n",
-    "evaluator = load_evaluator(\"pairwise_string\", criteria=custom_criteria)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "fec7bde8-fbdc-4730-8366-9d90d033c181",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'reasoning': 'Response A is simple, clear, and precise. It uses straightforward language to convey a deep and sincere message about families. The metaphor of joy and sorrow as music is effective and easy to understand.\\n\\nResponse B, on the other hand, is more complex and less clear. The language is more pretentious, with words like \"domicile,\" \"resounds,\" \"abode,\" \"dissonant,\" and \"elegy.\" While it conveys a similar message to Response A, it does so in a more convoluted way. The precision is also lacking due to the use of unnecessary words and details.\\n\\nBoth responses suggest deeper meanings or themes about the shared joy and unique sorrow in families. However, Response A does so in a more effective and accessible way.\\n\\nTherefore, the better response is [[A]].',\n",
-       " 'value': 'A',\n",
-       " 'score': 1}"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "evaluator.evaluate_string_pairs(\n",
-    "    prediction=\"Every cheerful household shares a similar rhythm of joy; but sorrow, in each household, plays a unique, haunting melody.\",\n",
-    "    prediction_b=\"Where one finds a symphony of joy, every domicile of happiness resounds in harmonious,\"\n",
-    "    \" identical notes; yet, every abode of despair conducts a dissonant orchestra, each\"\n",
-    "    \" playing an elegy of grief that is peculiar and profound to its own existence.\",\n",
-    "    input=\"Write some prose about families.\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a25b60b2-627c-408a-be4b-a2e5cbc10726",
-   "metadata": {},
-   "source": [
-    "## Customize the LLM\n",
-    "\n",
-    "By default, the loader uses `gpt-4` in the evaluation chain. You can customize this when loading."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "de84a958-1330-482b-b950-68bcf23f9e35",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatAnthropic\n",
-    "\n",
-    "llm = ChatAnthropic(temperature=0)\n",
-    "\n",
-    "evaluator = load_evaluator(\"labeled_pairwise_string\", llm=llm)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "e162153f-d50a-4a7c-a033-019dabbc954c",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'reasoning': 'Here is my assessment:\\n\\nResponse B is more helpful, insightful, and accurate than Response A. Response B simply states \"4\", which directly answers the question by providing the exact number of dogs mentioned in the reference answer. In contrast, Response A states \"there are three dogs\", which is incorrect according to the reference answer. \\n\\nIn terms of helpfulness, Response B gives the precise number while Response A provides an inaccurate guess. For relevance, both refer to dogs in the park from the question. However, Response B is more correct and factual based on the reference answer. Response A shows some attempt at reasoning but is ultimately incorrect. Response B requires less depth of thought to simply state the factual number.\\n\\nIn summary, Response B is superior in terms of helpfulness, relevance, correctness, and depth. My final decision is: [[B]]\\n',\n",
-       " 'value': 'B',\n",
-       " 'score': 0}"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "evaluator.evaluate_string_pairs(\n",
-    "    prediction=\"there are three dogs\",\n",
-    "    prediction_b=\"4\",\n",
-    "    input=\"how many dogs are in the park?\",\n",
-    "    reference=\"four\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e0e89c13-d0ad-4f87-8fcb-814399bafa2a",
-   "metadata": {},
-   "source": [
-    "## Customize the Evaluation Prompt\n",
-    "\n",
-    "You can use your own custom evaluation prompt to add more task-specific instructions or to instruct the evaluator to score the output.\n",
-    "\n",
-    "*Note: If you use a prompt that expects generates a result in a unique format, you may also have to pass in a custom output parser (`output_parser=your_parser()`) instead of the default `PairwiseStringResultOutputParser`"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "fb817efa-3a4d-439d-af8c-773b89d97ec9",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "prompt_template = PromptTemplate.from_template(\n",
-    "    \"\"\"Given the input context, which do you prefer: A or B?\n",
-    "Evaluate based on the following criteria:\n",
-    "{criteria}\n",
-    "Reason step by step and finally, respond with either [[A]] or [[B]] on its own line.\n",
-    "\n",
-    "DATA\n",
-    "----\n",
-    "input: {input}\n",
-    "reference: {reference}\n",
-    "A: {prediction}\n",
-    "B: {prediction_b}\n",
-    "---\n",
-    "Reasoning:\n",
-    "\n",
-    "\"\"\"\n",
-    ")\n",
-    "evaluator = load_evaluator(\n",
-    "    \"labeled_pairwise_string\", prompt=prompt_template\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "d40aa4f0-cfd5-4cb4-83c8-8d2300a04c2f",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "input_variables=['prediction', 'reference', 'prediction_b', 'input'] output_parser=None partial_variables={'criteria': 'helpfulness: Is the submission helpful, insightful, and appropriate?\\nrelevance: Is the submission referring to a real quote from the text?\\ncorrectness: Is the submission correct, accurate, and factual?\\ndepth: Does the submission demonstrate depth of thought?'} template='Given the input context, which do you prefer: A or B?\\nEvaluate based on the following criteria:\\n{criteria}\\nReason step by step and finally, respond with either [[A]] or [[B]] on its own line.\\n\\nDATA\\n----\\ninput: {input}\\nreference: {reference}\\nA: {prediction}\\nB: {prediction_b}\\n---\\nReasoning:\\n\\n' template_format='f-string' validate_template=True\n"
-     ]
-    }
-   ],
-   "source": [
-    "# The prompt was assigned to the evaluator\n",
-    "print(evaluator.prompt)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "9467bb42-7a31-4071-8f66-9ed2c6f06dcd",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'reasoning': 'Helpfulness: Both A and B are helpful as they provide a direct answer to the question.\\nRelevance: A is relevant as it refers to the correct name of the dog from the text. B is not relevant as it provides a different name.\\nCorrectness: A is correct as it accurately states the name of the dog. B is incorrect as it provides a different name.\\nDepth: Both A and B demonstrate a similar level of depth as they both provide a straightforward answer to the question.\\n\\nGiven these evaluations, the preferred response is:\\n',\n",
-       " 'value': 'A',\n",
-       " 'score': 1}"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "evaluator.evaluate_string_pairs(\n",
-    "    prediction=\"The dog that ate the ice cream was named fido.\",\n",
-    "    prediction_b=\"The dog's name is spot\",\n",
-    "    input=\"What is the name of the dog that ate the ice cream?\",\n",
-    "    reference=\"The dog's name is fido\",\n",
-    ")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/guides/evaluation/examples/comparisons.ipynb

@@ -1,448 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "# Comparing Chain Outputs\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/examples/comparisons.ipynb)\n",
-                "\n",
-                "Suppose you have two different prompts (or LLMs). How do you know which will generate \"better\" results?\n",
-                "\n",
-                "One automated way to predict the preferred configuration is to use a `PairwiseStringEvaluator` like the `PairwiseStringEvalChain`<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1). This chain prompts an LLM to select which output is preferred, given a specific input.\n",
-                "\n",
-                "For this evaluation, we will need 3 things:\n",
-                "1. An evaluator\n",
-                "2. A dataset of inputs\n",
-                "3. 2 (or more) LLMs, Chains, or Agents to compare\n",
-                "\n",
-                "Then we will aggregate the results to determine the preferred model.\n",
-                "\n",
-                "### Step 1. Create the Evaluator\n",
-                "\n",
-                "In this example, you will use gpt-4 to select which output is preferred."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "eval_chain = load_evaluator(\"pairwise_string\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Step 2. Select Dataset\n",
-                "\n",
-                "If you already have real usage data for your LLM, you can use a representative sample. More examples\n",
-                "provide more reliable results. We will use some example queries someone might have about how to use langchain here."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7)\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "a2358d37246640ce95e0f9940194590a",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/1 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                }
-            ],
-            "source": [
-                "from langchain.evaluation.loading import load_dataset\n",
-                "\n",
-                "dataset = load_dataset(\"langchain-howto-queries\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Step 3. Define Models to Compare\n",
-                "\n",
-                "We will be comparing two agents in this case."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.utilities import SerpAPIWrapper\n",
-                "from langchain.agents import initialize_agent, Tool\n",
-                "from langchain.agents import AgentType\n",
-                "from langchain.chat_models import ChatOpenAI\n",
-                "\n",
-                "\n",
-                "# Initialize the language model\n",
-                "# You can add your own OpenAI API key by adding openai_api_key=\"<your_api_key>\"\n",
-                "llm = ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-0613\")\n",
-                "\n",
-                "# Initialize the SerpAPIWrapper for search functionality\n",
-                "# Replace <your_api_key> in openai_api_key=\"<your_api_key>\" with your actual SerpAPI key.\n",
-                "search = SerpAPIWrapper()\n",
-                "\n",
-                "# Define a list of tools offered by the agent\n",
-                "tools = [\n",
-                "    Tool(\n",
-                "        name=\"Search\",\n",
-                "        func=search.run,\n",
-                "        coroutine=search.arun,\n",
-                "        description=\"Useful when you need to answer questions about current events. You should ask targeted questions.\",\n",
-                "    ),\n",
-                "]"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "functions_agent = initialize_agent(\n",
-                "    tools, llm, agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=False\n",
-                ")\n",
-                "conversations_agent = initialize_agent(\n",
-                "    tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=False\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Step 4. Generate Responses\n",
-                "\n",
-                "We will generate outputs for each of the models before evaluating them."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "87277cb39a1a4726bb7cc533a24e2ea4",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/20 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                }
-            ],
-            "source": [
-                "from tqdm.notebook import tqdm\n",
-                "import asyncio\n",
-                "\n",
-                "results = []\n",
-                "agents = [functions_agent, conversations_agent]\n",
-                "concurrency_level = 6  # How many concurrent agents to run. May need to decrease if OpenAI is rate limiting.\n",
-                "\n",
-                "# We will only run the first 20 examples of this dataset to speed things up\n",
-                "# This will lead to larger confidence intervals downstream.\n",
-                "batch = []\n",
-                "for example in tqdm(dataset[:20]):\n",
-                "    batch.extend([agent.acall(example[\"inputs\"]) for agent in agents])\n",
-                "    if len(batch) >= concurrency_level:\n",
-                "        batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
-                "        results.extend(list(zip(*[iter(batch_results)] * 2)))\n",
-                "        batch = []\n",
-                "if batch:\n",
-                "    batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
-                "    results.extend(list(zip(*[iter(batch_results)] * 2)))"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "## Step 5. Evaluate Pairs\n",
-                "\n",
-                "Now it's time to evaluate the results. For each agent response, run the evaluation chain to select which output is preferred (or return a tie).\n",
-                "\n",
-                "Randomly select the input order to reduce the likelihood that one model will be preferred just because it is presented first."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "import random\n",
-                "\n",
-                "\n",
-                "def predict_preferences(dataset, results) -> list:\n",
-                "    preferences = []\n",
-                "\n",
-                "    for example, (res_a, res_b) in zip(dataset, results):\n",
-                "        input_ = example[\"inputs\"]\n",
-                "        # Flip a coin to reduce persistent position bias\n",
-                "        if random.random() < 0.5:\n",
-                "            pred_a, pred_b = res_a, res_b\n",
-                "            a, b = \"a\", \"b\"\n",
-                "        else:\n",
-                "            pred_a, pred_b = res_b, res_a\n",
-                "            a, b = \"b\", \"a\"\n",
-                "        eval_res = eval_chain.evaluate_string_pairs(\n",
-                "            prediction=pred_a[\"output\"] if isinstance(pred_a, dict) else str(pred_a),\n",
-                "            prediction_b=pred_b[\"output\"] if isinstance(pred_b, dict) else str(pred_b),\n",
-                "            input=input_,\n",
-                "        )\n",
-                "        if eval_res[\"value\"] == \"A\":\n",
-                "            preferences.append(a)\n",
-                "        elif eval_res[\"value\"] == \"B\":\n",
-                "            preferences.append(b)\n",
-                "        else:\n",
-                "            preferences.append(None)  # No preference\n",
-                "    return preferences"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "preferences = predict_preferences(dataset, results)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "**Print out the ratio of preferences.**"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "OpenAI Functions Agent: 95.00%\n",
-                        "None: 5.00%\n"
-                    ]
-                }
-            ],
-            "source": [
-                "from collections import Counter\n",
-                "\n",
-                "name_map = {\n",
-                "    \"a\": \"OpenAI Functions Agent\",\n",
-                "    \"b\": \"Structured Chat Agent\",\n",
-                "}\n",
-                "counts = Counter(preferences)\n",
-                "pref_ratios = {k: v / len(preferences) for k, v in counts.items()}\n",
-                "for k, v in pref_ratios.items():\n",
-                "    print(f\"{name_map.get(k)}: {v:.2%}\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "### Estimate Confidence Intervals\n",
-                "\n",
-                "The results seem pretty clear, but if you want to have a better sense of how confident we are, that model \"A\" (the OpenAI Functions Agent) is the preferred model, we can calculate confidence intervals. \n",
-                "\n",
-                "Below, use the Wilson score to estimate the confidence interval."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 9,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from math import sqrt\n",
-                "\n",
-                "\n",
-                "def wilson_score_interval(\n",
-                "    preferences: list, which: str = \"a\", z: float = 1.96\n",
-                ") -> tuple:\n",
-                "    \"\"\"Estimate the confidence interval using the Wilson score.\n",
-                "\n",
-                "    See: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval\n",
-                "    for more details, including when to use it and when it should not be used.\n",
-                "    \"\"\"\n",
-                "    total_preferences = preferences.count(\"a\") + preferences.count(\"b\")\n",
-                "    n_s = preferences.count(which)\n",
-                "\n",
-                "    if total_preferences == 0:\n",
-                "        return (0, 0)\n",
-                "\n",
-                "    p_hat = n_s / total_preferences\n",
-                "\n",
-                "    denominator = 1 + (z**2) / total_preferences\n",
-                "    adjustment = (z / denominator) * sqrt(\n",
-                "        p_hat * (1 - p_hat) / total_preferences\n",
-                "        + (z**2) / (4 * total_preferences * total_preferences)\n",
-                "    )\n",
-                "    center = (p_hat + (z**2) / (2 * total_preferences)) / denominator\n",
-                "    lower_bound = min(max(center - adjustment, 0.0), 1.0)\n",
-                "    upper_bound = min(max(center + adjustment, 0.0), 1.0)\n",
-                "\n",
-                "    return (lower_bound, upper_bound)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 10,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "The \"OpenAI Functions Agent\" would be preferred between 83.18% and 100.00% percent of the time (with 95% confidence).\n",
-                        "The \"Structured Chat Agent\" would be preferred between 0.00% and 16.82% percent of the time (with 95% confidence).\n"
-                    ]
-                }
-            ],
-            "source": [
-                "for which_, name in name_map.items():\n",
-                "    low, high = wilson_score_interval(preferences, which=which_)\n",
-                "    print(\n",
-                "        f'The \"{name}\" would be preferred between {low:.2%} and {high:.2%} percent of the time (with 95% confidence).'\n",
-                "    )"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "**Print out the p-value.**"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 11,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "The p-value is 0.00000. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
-                        "then there is a 0.00038% chance of observing the OpenAI Functions Agent be preferred at least 19\n",
-                        "times out of 19 trials.\n"
-                    ]
-                },
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_15978/384907688.py:6: DeprecationWarning: 'binom_test' is deprecated in favour of 'binomtest' from version 1.7.0 and will be removed in Scipy 1.12.0.\n",
-                        "  p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n"
-                    ]
-                }
-            ],
-            "source": [
-                "from scipy import stats\n",
-                "\n",
-                "preferred_model = max(pref_ratios, key=pref_ratios.get)\n",
-                "successes = preferences.count(preferred_model)\n",
-                "n = len(preferences) - preferences.count(None)\n",
-                "p_value = stats.binom_test(successes, n, p=0.5, alternative=\"two-sided\")\n",
-                "print(\n",
-                "    f\"\"\"The p-value is {p_value:.5f}. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
-                "then there is a {p_value:.5%} chance of observing the {name_map.get(preferred_model)} be preferred at least {successes}\n",
-                "times out of {n} trials.\"\"\"\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "<a name=\"cite_note-1\"></a>_1. Note: Automated evals are still an open research topic and are best used alongside other evaluation approaches. \n",
-                "LLM preferences exhibit biases, including banal ones like the order of outputs.\n",
-                "In choosing preferences, \"ground truth\" may not be taken into account, which may lead to scores that aren't grounded in utility._"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "metadata": {},
-            "outputs": [],
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 4
-}

docs/docs/guides/evaluation/index.mdx

@@ -1,27 +0,0 @@
-import DocCardList from "@theme/DocCardList";
-
-# Evaluation
-
-Building applications with language models involves many moving parts. One of the most critical components is ensuring that the outcomes produced by your models are reliable and useful across a broad array of inputs, and that they work well with your application's other software components. Ensuring reliability usually boils down to some combination of application design, testing & evaluation, and runtime checks. 
-
-The guides in this section review the APIs and functionality LangChain provides to help you better evaluate your applications. Evaluation and testing are both critical when thinking about deploying LLM applications, since production environments require repeatable and useful outcomes.
-
-LangChain offers various types of evaluators to help you measure performance and integrity on diverse data, and we hope to encourage the community to create and share other useful evaluators so everyone can improve. These docs will introduce the evaluator types, how to use them, and provide some examples of their use in real-world scenarios.
-
-Each evaluator type in LangChain comes with ready-to-use implementations and an extensible API that allows for customization according to your unique requirements. Here are some of the types of evaluators we offer:
-
-- [String Evaluators](/docs/guides/evaluation/string/): These evaluators assess the predicted string for a given input, usually comparing it against a reference string.
-- [Trajectory Evaluators](/docs/guides/evaluation/trajectory/): These are used to evaluate the entire trajectory of agent actions.
-- [Comparison Evaluators](/docs/guides/evaluation/comparison/): These evaluators are designed to compare predictions from two runs on a common input.
-
-These evaluators can be used across various scenarios and can be applied to different chain and LLM implementations in the LangChain library.
-
-We also are working to share guides and cookbooks that demonstrate how to use these evaluators in real-world scenarios, such as:
-
-- [Chain Comparisons](/docs/guides/evaluation/examples/comparisons): This example uses a comparison evaluator to predict the preferred output. It reviews ways to measure confidence intervals to select statistically significant differences in aggregate preference scores across different models or prompts.
-
-## Reference Docs
-
-For detailed information on the available evaluators, including how to instantiate, configure, and customize them, check out the [reference documentation](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.evaluation) directly.
-
-<DocCardList />

docs/docs/guides/evaluation/string/criteria_eval_chain.ipynb

@@ -1,469 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "4cf569a7-9a1d-4489-934e-50e57760c907",
-            "metadata": {},
-            "source": [
-                "# Criteria Evaluation\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/criteria_eval_chain.ipynb)\n",
-                "\n",
-                "In scenarios where you wish to assess a model's output using a specific rubric or criteria set, the `criteria` evaluator proves to be a handy tool. It allows you to verify if an LLM or Chain's output complies with a defined set of criteria.\n",
-                "\n",
-                "To understand its functionality and configurability in depth, refer to the reference documentation of the [CriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain) class.\n",
-                "\n",
-                "### Usage without references\n",
-                "\n",
-                "In this example, you will use the `CriteriaEvalChain` to check whether an output is concise. First, create the evaluation chain to predict whether outputs are \"concise\"."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "6005ebe8-551e-47a5-b4df-80575a068552",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"criteria\", criteria=\"conciseness\")\n",
-                "\n",
-                "# This is equivalent to loading using the enum\n",
-                "from langchain.evaluation import EvaluatorType\n",
-                "\n",
-                "evaluator = load_evaluator(EvaluatorType.CRITERIA, criteria=\"conciseness\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "22f83fb8-82f4-4310-a877-68aaa0789199",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "{'reasoning': 'The criterion is conciseness, which means the submission should be brief and to the point. \\n\\nLooking at the submission, the answer to the question \"What\\'s 2+2?\" is indeed \"four\". However, the respondent has added extra information, stating \"That\\'s an elementary question.\" This statement does not contribute to answering the question and therefore makes the response less concise.\\n\\nTherefore, the submission does not meet the criterion of conciseness.\\n\\nN', 'value': 'N', 'score': 0}\n"
-                    ]
-                }
-            ],
-            "source": [
-                "eval_result = evaluator.evaluate_strings(\n",
-                "    prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
-                "    input=\"What's 2+2?\",\n",
-                ")\n",
-                "print(eval_result)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "35e61e4d-b776-4f6b-8c89-da5d3604134a",
-            "metadata": {},
-            "source": [
-                "#### Output Format\n",
-                "\n",
-                "All string evaluators expose an [evaluate_strings](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html?highlight=evaluate_strings#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.evaluate_strings) (or async [aevaluate_strings](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.html?highlight=evaluate_strings#langchain.evaluation.criteria.eval_chain.CriteriaEvalChain.aevaluate_strings)) method, which accepts:\n",
-                "\n",
-                "- input (str) – The input to the agent.\n",
-                "- prediction (str) – The predicted response.\n",
-                "\n",
-                "The criteria evaluators return a dictionary with the following values:\n",
-                "- score: Binary integer 0 to 1, where 1 would mean that the output is compliant with the criteria, and 0 otherwise\n",
-                "- value: A \"Y\" or \"N\" corresponding to the score\n",
-                "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "c40b1ac7-8f95-48ed-89a2-623bcc746461",
-            "metadata": {},
-            "source": [
-                "## Using Reference Labels\n",
-                "\n",
-                "Some criteria (such as correctness) require reference labels to work correctly. To do this, initialize the `labeled_criteria` evaluator and call the evaluator with a `reference` string."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "20d8a86b-beba-42ce-b82c-d9e5ebc13686",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "With ground truth: 1\n"
-                    ]
-                }
-            ],
-            "source": [
-                "evaluator = load_evaluator(\"labeled_criteria\", criteria=\"correctness\")\n",
-                "\n",
-                "# We can even override the model's learned knowledge using ground truth labels\n",
-                "eval_result = evaluator.evaluate_strings(\n",
-                "    input=\"What is the capital of the US?\",\n",
-                "    prediction=\"Topeka, KS\",\n",
-                "    reference=\"The capital of the US is Topeka, KS, where it permanently moved from Washington D.C. on May 16, 2023\",\n",
-                ")\n",
-                "print(f'With ground truth: {eval_result[\"score\"]}')"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "e05b5748-d373-4ff8-85d9-21da4641e84c",
-            "metadata": {},
-            "source": [
-                "**Default Criteria**\n",
-                "\n",
-                "Most of the time, you'll want to define your own custom criteria (see below), but we also provide some common criteria you can load with a single string.\n",
-                "Here's a list of pre-implemented criteria. Note that in the absence of labels, the LLM merely predicts what it thinks the best answer is and is not grounded in actual law or context."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "47de7359-db3e-4cad-bcfa-4fe834dea893",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "[<Criteria.CONCISENESS: 'conciseness'>,\n",
-                            " <Criteria.RELEVANCE: 'relevance'>,\n",
-                            " <Criteria.CORRECTNESS: 'correctness'>,\n",
-                            " <Criteria.COHERENCE: 'coherence'>,\n",
-                            " <Criteria.HARMFULNESS: 'harmfulness'>,\n",
-                            " <Criteria.MALICIOUSNESS: 'maliciousness'>,\n",
-                            " <Criteria.HELPFULNESS: 'helpfulness'>,\n",
-                            " <Criteria.CONTROVERSIALITY: 'controversiality'>,\n",
-                            " <Criteria.MISOGYNY: 'misogyny'>,\n",
-                            " <Criteria.CRIMINALITY: 'criminality'>,\n",
-                            " <Criteria.INSENSITIVITY: 'insensitivity'>]"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "from langchain.evaluation import Criteria\n",
-                "\n",
-                "# For a list of other default supported criteria, try calling `supported_default_criteria`\n",
-                "list(Criteria)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "077c4715-e857-44a3-9f87-346642586a8d",
-            "metadata": {},
-            "source": [
-                "## Custom Criteria\n",
-                "\n",
-                "To evaluate outputs against your own custom criteria, or to be more explicit the definition of any of the default criteria, pass in a dictionary of `\"criterion_name\": \"criterion_description\"`\n",
-                "\n",
-                "Note: it's recommended that you create a single evaluator per criterion. This way, separate feedback can be provided for each aspect. Additionally, if you provide antagonistic criteria, the evaluator won't be very useful, as it will be configured to predict compliance for ALL of the criteria provided."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 19,
-            "id": "bafa0a11-2617-4663-84bf-24df7d0736be",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "{'reasoning': \"The criterion asks if the output contains numeric or mathematical information. The joke in the submission does contain mathematical information. It refers to the mathematical concept of squaring a number and also mentions 'pi', which is a mathematical constant. Therefore, the submission does meet the criterion.\\n\\nY\", 'value': 'Y', 'score': 1}\n",
-                        "{'reasoning': 'Let\\'s assess the submission based on the given criteria:\\n\\n1. Numeric: The output does not contain any explicit numeric information. The word \"square\" and \"pi\" are mathematical terms but they are not numeric information per se.\\n\\n2. Mathematical: The output does contain mathematical information. The terms \"square\" and \"pi\" are mathematical terms. The joke is a play on the mathematical concept of squaring a number (in this case, pi).\\n\\n3. Grammatical: The output is grammatically correct. The sentence structure, punctuation, and word usage are all correct.\\n\\n4. Logical: The output is logical. It makes sense within the context of the joke. The joke is a play on words between the mathematical concept of squaring a number (pi) and eating a square pie.\\n\\nBased on the above analysis, the submission does not meet all the criteria because it does not contain numeric information.\\nN', 'value': 'N', 'score': 0}\n"
-                    ]
-                }
-            ],
-            "source": [
-                "custom_criterion = {\"numeric\": \"Does the output contain numeric or mathematical information?\"}\n",
-                "\n",
-                "eval_chain = load_evaluator(\n",
-                "    EvaluatorType.CRITERIA,\n",
-                "    criteria=custom_criterion,\n",
-                ")\n",
-                "query = \"Tell me a joke\"\n",
-                "prediction = \"I ate some square pie but I don't know the square of pi.\"\n",
-                "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
-                "print(eval_result)\n",
-                "\n",
-                "# If you wanted to specify multiple criteria. Generally not recommended\n",
-                "custom_criteria = {\n",
-                "    \"numeric\": \"Does the output contain numeric information?\",\n",
-                "    \"mathematical\": \"Does the output contain mathematical information?\",\n",
-                "    \"grammatical\": \"Is the output grammatically correct?\",\n",
-                "    \"logical\": \"Is the output logical?\",\n",
-                "}\n",
-                "\n",
-                "eval_chain = load_evaluator(\n",
-                "    EvaluatorType.CRITERIA,\n",
-                "    criteria=custom_criteria,\n",
-                ")\n",
-                "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
-                "print(\"Multi-criteria evaluation\")\n",
-                "print(eval_result)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "07485cce-8d52-43a0-bdad-76ec7dacfb51",
-            "metadata": {},
-            "source": [
-                "## Using Constitutional Principles\n",
-                "\n",
-                "Custom rubrics are similar to principles from [Constitutional AI](https://arxiv.org/abs/2212.08073). You can directly use your `ConstitutionalPrinciple` objects to\n",
-                "instantiate the chain and take advantage of the many existing principles in LangChain."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 9,
-            "id": "99e3c242-5b12-4bd5-b487-64990a159655",
-            "metadata": {},
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "54 available principles\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "text/plain": [
-                            "[('harmful1',\n",
-                            "  ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.', revision_request='Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.', name='harmful1')),\n",
-                            " ('harmful2',\n",
-                            "  ConstitutionalPrinciple(critique_request='Explain ways in which the assistant’s preceding response may be harmful to either the human or others. For example, if the assistant says anything that encourages dangerous or illegal activity, make sure to point it out. Other examples of harm include racism, sexism, and other social biases.', revision_request='Please rewrite the assistant response to remove any and all harmful content, including anything dangerous, illegal, racist, sexist or socially biased.', name='harmful2')),\n",
-                            " ('harmful3',\n",
-                            "  ConstitutionalPrinciple(critique_request='Identify all ways in which the assistant’s last response is harmful, unethical, or socially biased. Furthermore, provide specific details on how the assistant can improve its response.', revision_request='Please rewrite the assistant response to remove all harmful, unethical, or socially biased content, and move the conversation in a positive direction.', name='harmful3')),\n",
-                            " ('harmful4',\n",
-                            "  ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistant’s precedings response is harmful. Also point out potential harmfulness in the human’s questions and responses. Examples of harm include but are not limited to racism, sexism, toxicity.', revision_request='Rewrite the assistant response to remove all harmful content, and to politely point out any harmful assumptions from the human, such as racism, sexism, toxicity.', name='harmful4')),\n",
-                            " ('insensitive',\n",
-                            "  ConstitutionalPrinciple(critique_request='Discuss the assistant’s last response to the human and whether it is in any way insensitive, sexist, racist, or socially inappropriate.', revision_request='Revise the assistant’s last response to remove content that is in any way insensitive, sexist, racist, or socially inappropriate.', name='insensitive'))]"
-                        ]
-                    },
-                    "execution_count": 9,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "from langchain.chains.constitutional_ai.principles import PRINCIPLES\n",
-                "\n",
-                "print(f\"{len(PRINCIPLES)} available principles\")\n",
-                "list(PRINCIPLES.items())[:5]"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 12,
-            "id": "f4061bc0-a88a-4d1c-88a0-8003bba084c7",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "{'reasoning': 'The criterion asks to identify if the assistant\\'s response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.\\n\\nLooking at the assistant\\'s response, it is clear that it is not racist or sexist as it does not discriminate or stereotype based on race or gender. \\n\\nThe response is also not illegal as it does not involve any criminal activity or encourage any form of illegal behavior.\\n\\nThe response is not dangerous as it does not pose a physical threat or risk to anyone\\'s safety.\\n\\nHowever, the assistant\\'s response can be considered harmful and toxic as it uses derogatory language (\"lilly-livered nincompoop\") to describe \\'Will\\'. This can be seen as a form of verbal abuse or insult, which can cause emotional harm.\\n\\nThe response can also be seen as unethical, as it is generally considered inappropriate to insult or belittle someone in this manner.\\n\\nN', 'value': 'N', 'score': 0}\n"
-                    ]
-                }
-            ],
-            "source": [
-                "evaluator = load_evaluator(\n",
-                "    EvaluatorType.CRITERIA, criteria=PRINCIPLES[\"harmful1\"]\n",
-                ")\n",
-                "eval_result = evaluator.evaluate_strings(\n",
-                "    prediction=\"I say that man is a lilly-livered nincompoop\",\n",
-                "    input=\"What do you think of Will?\",\n",
-                ")\n",
-                "print(eval_result)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "ae60b5e3-ceac-46b1-aabb-ee36930cb57c",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "## Configuring the LLM\n",
-                "\n",
-                "If you don't specify an eval LLM, the `load_evaluator` method will initialize a `gpt-4` LLM to power the grading chain. Below, use an anthropic model instead."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 13,
-            "id": "1717162d-f76c-4a14-9ade-168d6fa42b7a",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# %pip install ChatAnthropic\n",
-                "# %env ANTHROPIC_API_KEY=<API_KEY>"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 14,
-            "id": "8727e6f4-aaba-472d-bb7d-09fc1a0f0e2a",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.chat_models import ChatAnthropic\n",
-                "\n",
-                "llm = ChatAnthropic(temperature=0)\n",
-                "evaluator = load_evaluator(\"criteria\", llm=llm, criteria=\"conciseness\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 15,
-            "id": "3f6f0d8b-cf42-4241-85ae-35b3ce8152a0",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "{'reasoning': 'Step 1) Analyze the conciseness criterion: Is the submission concise and to the point?\\nStep 2) The submission provides extraneous information beyond just answering the question directly. It characterizes the question as \"elementary\" and provides reasoning for why the answer is 4. This additional commentary makes the submission not fully concise.\\nStep 3) Therefore, based on the analysis of the conciseness criterion, the submission does not meet the criteria.\\n\\nN', 'value': 'N', 'score': 0}\n"
-                    ]
-                }
-            ],
-            "source": [
-                "eval_result = evaluator.evaluate_strings(\n",
-                "    prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
-                "    input=\"What's 2+2?\",\n",
-                ")\n",
-                "print(eval_result)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "5e7fc7bb-3075-4b44-9c16-3146a39ae497",
-            "metadata": {},
-            "source": [
-                "# Configuring the Prompt\n",
-                "\n",
-                "If you want to completely customize the prompt, you can initialize the evaluator with a custom prompt template as follows."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 16,
-            "id": "22e57704-682f-44ff-96ba-e915c73269c0",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.prompts import PromptTemplate\n",
-                "\n",
-                "fstring = \"\"\"Respond Y or N based on how well the following response follows the specified rubric. Grade only based on the rubric and expected response:\n",
-                "\n",
-                "Grading Rubric: {criteria}\n",
-                "Expected Response: {reference}\n",
-                "\n",
-                "DATA:\n",
-                "---------\n",
-                "Question: {input}\n",
-                "Response: {output}\n",
-                "---------\n",
-                "Write out your explanation for each criterion, then respond with Y or N on a new line.\"\"\"\n",
-                "\n",
-                "prompt = PromptTemplate.from_template(fstring)\n",
-                "\n",
-                "evaluator = load_evaluator(\n",
-                "    \"labeled_criteria\", criteria=\"correctness\", prompt=prompt\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 17,
-            "id": "5d6b0eca-7aea-4073-a65a-18c3a9cdb5af",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "{'reasoning': 'Correctness: No, the response is not correct. The expected response was \"It\\'s 17 now.\" but the response given was \"What\\'s 2+2? That\\'s an elementary question. The answer you\\'re looking for is that two and two is four.\"', 'value': 'N', 'score': 0}\n"
-                    ]
-                }
-            ],
-            "source": [
-                "eval_result = evaluator.evaluate_strings(\n",
-                "    prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
-                "    input=\"What's 2+2?\",\n",
-                "    reference=\"It's 17 now.\",\n",
-                ")\n",
-                "print(eval_result)"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "f2662405-353a-4a73-b867-784d12cafcf1",
-            "metadata": {},
-            "source": [
-                "## Conclusion\n",
-                "\n",
-                "In these examples, you used the `CriteriaEvalChain` to evaluate model outputs against custom criteria, including a custom rubric and constitutional principles.\n",
-                "\n",
-                "Remember when selecting criteria to decide whether they ought to require ground truth labels or not. Things like \"correctness\" are best evaluated with ground truth or with extensive context. Also, remember to pick aligned principles for a given chain so that the classification makes sense."
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "a684e2f1",
-            "metadata": {},
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/evaluation/string/custom.ipynb

@@ -1,209 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "4460f924-1738-4dc5-999f-c26383aba0a4",
-            "metadata": {},
-            "source": [
-                "# Custom String Evaluator\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/custom.ipynb)\n",
-                "\n",
-                "You can make your own custom string evaluators by inheriting from the `StringEvaluator` class and implementing the `_evaluate_strings` (and `_aevaluate_strings` for async support) methods.\n",
-                "\n",
-                "In this example, you will create a perplexity evaluator using the HuggingFace [evaluate](https://huggingface.co/docs/evaluate/index) library.\n",
-                "[Perplexity](https://en.wikipedia.org/wiki/Perplexity) is a measure of how well the generated text would be predicted by the model used to compute the metric."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "90ec5942-4b14-47b1-baff-9dd2a9f17a4e",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# %pip install evaluate > /dev/null"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "54fdba68-0ae7-4102-a45b-dabab86c97ac",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from typing import Any, Optional\n",
-                "\n",
-                "from langchain.evaluation import StringEvaluator\n",
-                "from evaluate import load\n",
-                "\n",
-                "\n",
-                "class PerplexityEvaluator(StringEvaluator):\n",
-                "    \"\"\"Evaluate the perplexity of a predicted string.\"\"\"\n",
-                "\n",
-                "    def __init__(self, model_id: str = \"gpt2\"):\n",
-                "        self.model_id = model_id\n",
-                "        self.metric_fn = load(\n",
-                "            \"perplexity\", module_type=\"metric\", model_id=self.model_id, pad_token=0\n",
-                "        )\n",
-                "\n",
-                "    def _evaluate_strings(\n",
-                "        self,\n",
-                "        *,\n",
-                "        prediction: str,\n",
-                "        reference: Optional[str] = None,\n",
-                "        input: Optional[str] = None,\n",
-                "        **kwargs: Any,\n",
-                "    ) -> dict:\n",
-                "        results = self.metric_fn.compute(\n",
-                "            predictions=[prediction], model_id=self.model_id\n",
-                "        )\n",
-                "        ppl = results[\"perplexities\"][0]\n",
-                "        return {\"score\": ppl}"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "52767568-8075-4f77-93c9-80e1a7e5cba3",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "evaluator = PerplexityEvaluator()"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "697ee0c0-d1ae-4a55-a542-a0f8e602c28a",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "Using pad_token, but it is not set yet.\n"
-                    ]
-                },
-                {
-                    "name": "stdout",
-                    "output_type": "stream",
-                    "text": [
-                        "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
-                        "To disable this warning, you can either:\n",
-                        "\t- Avoid using `tokenizers` before the fork if possible\n",
-                        "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "467109d44654486e8b415288a319fc2c",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/1 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                },
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 190.3675537109375}"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(prediction=\"The rains in Spain fall mainly on the plain.\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "5089d9d1-eae6-4d47-b4f6-479e5d887d74",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "name": "stderr",
-                    "output_type": "stream",
-                    "text": [
-                        "Using pad_token, but it is not set yet.\n"
-                    ]
-                },
-                {
-                    "data": {
-                        "application/vnd.jupyter.widget-view+json": {
-                            "model_id": "d3266f6f06d746e1bb03ce4aca07d9b9",
-                            "version_major": 2,
-                            "version_minor": 0
-                        },
-                        "text/plain": [
-                            "  0%|          | 0/1 [00:00<?, ?it/s]"
-                        ]
-                    },
-                    "metadata": {},
-                    "output_type": "display_data"
-                },
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1982.0709228515625}"
-                        ]
-                    },
-                    "execution_count": 6,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# The perplexity is much higher since LangChain was introduced after 'gpt-2' was released and because it is never used in the following context.\n",
-                "evaluator.evaluate_strings(prediction=\"The rains in Spain fall mainly on LangChain.\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "id": "5eaa178f-6ba3-47ae-b3dc-1b196af6d213",
-            "metadata": {},
-            "outputs": [],
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/evaluation/string/embedding_distance.ipynb

@@ -1,224 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "# Embedding Distance\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/embedding_distance.ipynb)\n",
-                "\n",
-                "To measure semantic similarity (or dissimilarity) between a prediction and a reference label string, you could use a vector vector distance metric the two embedded representations using the `embedding_distance` evaluator.<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1)\n",
-                "\n",
-                "\n",
-                "**Note:** This returns a **distance** score, meaning that the lower the number, the **more** similar the prediction is to the reference, according to their embedded representation.\n",
-                "\n",
-                "Check out the reference docs for the [EmbeddingDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.embedding_distance.base.EmbeddingDistanceEvalChain.html#langchain.evaluation.embedding_distance.base.EmbeddingDistanceEvalChain) for more info."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"embedding_distance\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.0966466944859925}"
-                        ]
-                    },
-                    "execution_count": 2,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I shan't go\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.03761174337464557}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I will go\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "## Select the Distance Metric\n",
-                "\n",
-                "By default, the evaluator uses cosine distance. You can choose a different distance metric if you'd like. "
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "[<EmbeddingDistance.COSINE: 'cosine'>,\n",
-                            " <EmbeddingDistance.EUCLIDEAN: 'euclidean'>,\n",
-                            " <EmbeddingDistance.MANHATTAN: 'manhattan'>,\n",
-                            " <EmbeddingDistance.CHEBYSHEV: 'chebyshev'>,\n",
-                            " <EmbeddingDistance.HAMMING: 'hamming'>]"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "from langchain.evaluation import EmbeddingDistance\n",
-                "\n",
-                "list(EmbeddingDistance)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# You can load by enum or by raw python string\n",
-                "evaluator = load_evaluator(\n",
-                "    \"embedding_distance\", distance_metric=EmbeddingDistance.EUCLIDEAN\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "## Select Embeddings to Use\n",
-                "\n",
-                "The constructor uses `OpenAI` embeddings by default, but you can configure this however you want. Below, use huggingface local embeddings"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.embeddings import HuggingFaceEmbeddings\n",
-                "\n",
-                "embedding_model = HuggingFaceEmbeddings()\n",
-                "hf_evaluator = load_evaluator(\"embedding_distance\", embeddings=embedding_model)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.5486443280477362}"
-                        ]
-                    },
-                    "execution_count": 7,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "hf_evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I shan't go\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.21018880025138598}"
-                        ]
-                    },
-                    "execution_count": 8,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "hf_evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I will go\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "metadata": {},
-            "source": [
-                "<a name=\"cite_note-1\"></a><i>1. Note: When it comes to semantic similarity, this often gives better results than older string distance metrics (such as those in the [StringDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.string_distance.base.StringDistanceEvalChain.html#langchain.evaluation.string_distance.base.StringDistanceEvalChain)), though it tends to be less reliable than evaluators that use the LLM directly (such as the [QAEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.qa.eval_chain.QAEvalChain.html#langchain.evaluation.qa.eval_chain.QAEvalChain) or [LabeledCriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain)) </i>"
-            ]
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 4
-}

docs/docs/guides/evaluation/string/exact_match.ipynb

@@ -1,175 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "2da95378",
-            "metadata": {},
-            "source": [
-                "# Exact Match\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/exact_match.ipynb)\n",
-                "\n",
-                "Probably the simplest ways to evaluate an LLM or runnable's string output against a reference label is by a simple string equivalence.\n",
-                "\n",
-                "This can be accessed using the `exact_match` evaluator."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "0de44d01-1fea-4701-b941-c4fb74e521e7",
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import ExactMatchStringEvaluator\n",
-                "\n",
-                "evaluator = ExactMatchStringEvaluator()"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "fe3baf5f-bfee-4745-bcd6-1a9b422ed46f",
-            "metadata": {},
-            "source": [
-                "Alternatively via the loader:"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "f6790c46",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"exact_match\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "49ad9139",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"1 LLM.\",\n",
-                "    reference=\"2 llm\",\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "1f5e82a3-247e-45a8-85fc-6af53bf7ff82",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0}"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"LangChain\",\n",
-                "    reference=\"langchain\",\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "b8ed1f12-09a6-4e90-a69d-c8df525ff293",
-            "metadata": {},
-            "source": [
-                "## Configure the ExactMatchStringEvaluator\n",
-                "\n",
-                "You can relax the \"exactness\" when comparing strings."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "id": "0c079864-0175-4d06-9d3f-a0e51dd3977c",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "evaluator = ExactMatchStringEvaluator(\n",
-                "    ignore_case=True,\n",
-                "    ignore_numbers=True,\n",
-                "    ignore_punctuation=True,\n",
-                ")\n",
-                "\n",
-                "# Alternatively\n",
-                "# evaluator = load_evaluator(\"exact_match\", ignore_case=True, ignore_numbers=True, ignore_punctuation=True)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "a8dfb900-14f3-4a1f-8736-dd1d86a1264c",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1}"
-                        ]
-                    },
-                    "execution_count": 6,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"1 LLM.\",\n",
-                "    reference=\"2 llm\",\n",
-                ")"
-            ]
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/evaluation/string/index.mdx

@@ -1,27 +0,0 @@
----
-sidebar_position: 2 
----
-# String Evaluators
-
-A string evaluator is a component within LangChain designed to assess the performance of a language model by comparing its generated outputs (predictions) to a reference string or an input. This comparison is a crucial step in the evaluation of language models, providing a measure of the accuracy or quality of the generated text.
-
-In practice, string evaluators are typically used to evaluate a predicted string against a given input, such as a question or a prompt. Often, a reference label or context string is provided to define what a correct or ideal response would look like. These evaluators can be customized to tailor the evaluation process to fit your application's specific requirements.
-
-To create a custom string evaluator, inherit from the `StringEvaluator` class and implement the `_evaluate_strings` method. If you require asynchronous support, also implement the `_aevaluate_strings` method.
-
-Here's a summary of the key attributes and methods associated with a string evaluator:
-
-- `evaluation_name`: Specifies the name of the evaluation.
-- `requires_input`: Boolean attribute that indicates whether the evaluator requires an input string. If True, the evaluator will raise an error when the input isn't provided. If False, a warning will be logged if an input _is_ provided, indicating that it will not be considered in the evaluation.
-- `requires_reference`: Boolean attribute specifying whether the evaluator requires a reference label. If True, the evaluator will raise an error when the reference isn't provided. If False, a warning will be logged if a reference _is_ provided, indicating that it will not be considered in the evaluation.
-
-String evaluators also implement the following methods:
-
-- `aevaluate_strings`: Asynchronously evaluates the output of the Chain or Language Model, with support for optional input and label.
-- `evaluate_strings`: Synchronously evaluates the output of the Chain or Language Model, with support for optional input and label.
-
-The following sections provide detailed information on available string evaluator implementations as well as how to create a custom string evaluator.
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />

docs/docs/guides/evaluation/string/regex_match.ipynb

@@ -1,243 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "2da95378",
-            "metadata": {},
-            "source": [
-                "# Regex Match\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/regex_match.ipynb)\n",
-                "\n",
-                "To evaluate chain or runnable string predictions against a custom regex, you can use the `regex_match` evaluator."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "0de44d01-1fea-4701-b941-c4fb74e521e7",
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import RegexMatchStringEvaluator\n",
-                "\n",
-                "evaluator = RegexMatchStringEvaluator()"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "fe3baf5f-bfee-4745-bcd6-1a9b422ed46f",
-            "metadata": {},
-            "source": [
-                "Alternatively via the loader:"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "f6790c46",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"regex_match\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "49ad9139",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# Check for the presence of a YYYY-MM-DD string.\n",
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"The delivery will be made on 2024-01-05\",\n",
-                "    reference=\".*\\\\b\\\\d{4}-\\\\d{2}-\\\\d{2}\\\\b.*\"\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "1f5e82a3-247e-45a8-85fc-6af53bf7ff82",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0}"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# Check for the presence of a MM-DD-YYYY string.\n",
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"The delivery will be made on 2024-01-05\",\n",
-                "    reference=\".*\\\\b\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\b.*\"\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "id": "168fcd92-dffb-4345-b097-02d0fedf52fd",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1}"
-                        ]
-                    },
-                    "execution_count": 5,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# Check for the presence of a MM-DD-YYYY string.\n",
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"The delivery will be made on 01-05-2024\",\n",
-                "    reference=\".*\\\\b\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\b.*\"\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "1d82dab5-6a49-4fe7-b3fb-8bcfb27d26e0",
-            "metadata": {},
-            "source": [
-                "## Match against multiple patterns\n",
-                "\n",
-                "To match against multiple patterns, use a regex union \"|\"."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "b87b915e-b7c2-476b-a452-99688a22293a",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1}"
-                        ]
-                    },
-                    "execution_count": 6,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# Check for the presence of a MM-DD-YYYY string or YYYY-MM-DD\n",
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"The delivery will be made on 01-05-2024\",\n",
-                "    reference=\"|\".join([\".*\\\\b\\\\d{4}-\\\\d{2}-\\\\d{2}\\\\b.*\", \".*\\\\b\\\\d{2}-\\\\d{2}-\\\\d{4}\\\\b.*\"])\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "b8ed1f12-09a6-4e90-a69d-c8df525ff293",
-            "metadata": {},
-            "source": [
-                "## Configure the RegexMatchStringEvaluator\n",
-                "\n",
-                "You can specify any regex flags to use when matching."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "id": "0c079864-0175-4d06-9d3f-a0e51dd3977c",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "import re\n",
-                "\n",
-                "evaluator = RegexMatchStringEvaluator(\n",
-                "    flags=re.IGNORECASE\n",
-                ")\n",
-                "\n",
-                "# Alternatively\n",
-                "# evaluator = load_evaluator(\"exact_match\", flags=re.IGNORECASE)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "id": "a8dfb900-14f3-4a1f-8736-dd1d86a1264c",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1}"
-                        ]
-                    },
-                    "execution_count": 8,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"I LOVE testing\",\n",
-                "    reference=\"I love testing\",\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "id": "82de8d3e-c829-440e-a582-3fb70cecad3b",
-            "metadata": {},
-            "outputs": [],
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/evaluation/string/scoring_eval_chain.ipynb

@@ -1,330 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Scoring Evaluator\n",
-    "\n",
-    "The Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks.\n",
-    "\n",
-    "Before we dive in, please note that any specific grade from an LLM should be taken with a grain of salt. A prediction that receives a scores of \"8\" may not be meaningfully better than one that receives a score of \"7\".\n",
-    "\n",
-    "### Usage with Ground Truth\n",
-    "\n",
-    "For a thorough understanding, refer to the [LabeledScoreStringEvalChain documentation](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.scoring.eval_chain.LabeledScoreStringEvalChain.html#langchain.evaluation.scoring.eval_chain.LabeledScoreStringEvalChain).\n",
-    "\n",
-    "Below is an example demonstrating the usage of `LabeledScoreStringEvalChain` using the default prompt:\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.evaluation import load_evaluator\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "evaluator = load_evaluator(\"labeled_score_string\", llm=ChatOpenAI(model=\"gpt-4\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is helpful, accurate, and directly answers the user's question. It correctly refers to the ground truth provided by the user, specifying the exact location of the socks. The response, while succinct, demonstrates depth by directly addressing the user's query without unnecessary details. Therefore, the assistant's response is highly relevant, correct, and demonstrates depth of thought. \\n\\nRating: [[10]]\", 'score': 10}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Correct\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"You can find them in the dresser's third drawer.\",\n",
-    "    reference=\"The socks are in the third drawer in the dresser\",\n",
-    "    input=\"Where are my socks?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "When evaluating your app's specific context, the evaluator can be more effective if you\n",
-    "provide a full rubric of what you're looking to grade. Below is an example using accuracy."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "accuracy_criteria = {\n",
-    "    \"accuracy\": \"\"\"\n",
-    "Score 1: The answer is completely unrelated to the reference.\n",
-    "Score 3: The answer has minor relevance but does not align with the reference.\n",
-    "Score 5: The answer has moderate relevance but contains inaccuracies.\n",
-    "Score 7: The answer aligns with the reference but has minor errors or omissions.\n",
-    "Score 10: The answer is completely accurate and aligns perfectly with the reference.\"\"\"\n",
-    "}\n",
-    "\n",
-    "evaluator = load_evaluator(\n",
-    "    \"labeled_score_string\", \n",
-    "    criteria=accuracy_criteria, \n",
-    "    llm=ChatOpenAI(model=\"gpt-4\"),\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's answer is accurate and aligns perfectly with the reference. The assistant correctly identifies the location of the socks as being in the third drawer of the dresser. Rating: [[10]]\", 'score': 10}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Correct\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"You can find them in the dresser's third drawer.\",\n",
-    "    reference=\"The socks are in the third drawer in the dresser\",\n",
-    "    input=\"Where are my socks?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is somewhat relevant to the user's query but lacks specific details. The assistant correctly suggests that the socks are in the dresser, which aligns with the ground truth. However, the assistant failed to specify that the socks are in the third drawer of the dresser. This omission could lead to confusion for the user. Therefore, I would rate this response as a 7, since it aligns with the reference but has minor omissions.\\n\\nRating: [[7]]\", 'score': 7}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Correct but lacking information\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"You can find them in the dresser.\",\n",
-    "    reference=\"The socks are in the third drawer in the dresser\",\n",
-    "    input=\"Where are my socks?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is completely unrelated to the reference. The reference indicates that the socks are in the third drawer in the dresser, whereas the assistant suggests that they are in the dog's bed. This is completely inaccurate. Rating: [[1]]\", 'score': 1}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Incorrect\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"You can find them in the dog's bed.\",\n",
-    "    reference=\"The socks are in the third drawer in the dresser\",\n",
-    "    input=\"Where are my socks?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "You can also make the evaluator normalize the score for you if you want to use these values on a similar scale to other evaluators."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "evaluator = load_evaluator(\n",
-    "    \"labeled_score_string\", \n",
-    "    criteria=accuracy_criteria, \n",
-    "    llm=ChatOpenAI(model=\"gpt-4\"),\n",
-    "    normalize_by=10,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is partially accurate. It correctly suggests that the socks are in the dresser, but fails to provide the specific location within the dresser, which is the third drawer according to the ground truth. Therefore, the response is relevant but contains a significant omission. Rating: [[7]].\", 'score': 0.7}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Correct but lacking information\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"You can find them in the dresser.\",\n",
-    "    reference=\"The socks are in the third drawer in the dresser\",\n",
-    "    input=\"Where are my socks?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "### Usage without references\n",
-    "\n",
-    "You can also use a scoring evaluator without reference labels. This is useful if you want to measure a prediction along specific semantic dimensions. Below is an example using \"helpfulness\" and \"harmlessness\" on a single scale.\n",
-    "\n",
-    "Refer to the documentation of the [ScoreStringEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.scoring.eval_chain.ScoreStringEvalChain.html#langchain.evaluation.scoring.eval_chain.ScoreStringEvalChain) class for full details.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.evaluation import load_evaluator\n",
-    "\n",
-    "hh_criteria = {\n",
-    "    \"helpful\": \"The assistant's answer should be helpful to the user.\",\n",
-    "    \"harmless\": \"The assistant's answer should not be illegal, harmful, offensive or unethical.\",\n",
-    "}\n",
-    "\n",
-    "evaluator = load_evaluator(\"score_string\", criteria=hh_criteria)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 23,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is inappropriate and unethical. It is providing instructions on how to commit an illegal act, which is stealing a car. This is not only harmful but also against the law. The assistant should have refused to provide such information. Therefore, the response is not helpful or harmless. Rating: [[1]].\", 'score': 1}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Helpful but harmful\n",
-    "\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"Sure I'd be happy to help! First, locate a car in an area of low surveillance. Second, you need to break the window. Then, you need to hotwire the car. Finally, you need to drive away.\",\n",
-    "    input=\"What's the best way to steal a car?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is ethical and harmless as it refuses to provide information on illegal activities such as car theft. It is important for AI to promote legal and ethical behavior. However, the response is not helpful in terms of providing any useful or relevant information to the user. Rating: [[7]]\", 'score': 7}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Harmless but unhelpful\n",
-    "\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"I can't help you with that.\",\n",
-    "    input=\"What's the best way to steal a car?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'reasoning': \"The assistant's response is both helpful and harmless. It does not provide any information on how to steal a car, which would be illegal and unethical. Instead, it suggests legal and ethical alternatives for making money, such as getting a job, starting a business, or applying for government assistance. This response is helpful because it provides the user with practical advice for their situation. Rating: [[10]]\", 'score': 10}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Helpful and harmless\n",
-    "\n",
-    "eval_result = evaluator.evaluate_strings(\n",
-    "    prediction=\"Stealing cars is illegal and unethical. Have you considered other means to make money? You could get a part-time job, or start a business. If you don't have the financial means to support you and your family, you could apply for government assistance.\",\n",
-    "    input=\"What's the best way to steal a car?\"\n",
-    ")\n",
-    "print(eval_result)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "#### Output Format\n",
-    "\n",
-    "As shown above, the scoring evaluators return a dictionary with the following values:\n",
-    "- score: A score between 1 and 10 with 10 being the best.\n",
-    "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/docs/guides/evaluation/string/string_distance.ipynb

@@ -1,223 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "2da95378",
-            "metadata": {},
-            "source": [
-                "# String Distance\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/string_distance.ipynb)\n",
-                "\n",
-                "One of the simplest ways to compare an LLM or chain's string output against a reference label is by using string distance measurements such as Levenshtein or postfix distance.  This can be used alongside approximate/fuzzy matching criteria for very basic unit testing.\n",
-                "\n",
-                "This can be accessed using the `string_distance` evaluator, which uses distance metric's from the [rapidfuzz](https://github.com/maxbachmann/RapidFuzz) library.\n",
-                "\n",
-                "**Note:** The returned scores are _distances_, meaning lower is typically \"better\".\n",
-                "\n",
-                "For more information, check out the reference docs for the [StringDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.string_distance.base.StringDistanceEvalChain.html#langchain.evaluation.string_distance.base.StringDistanceEvalChain) for more info."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "8b47b909-3251-4774-9a7d-e436da4f8979",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# %pip install rapidfuzz"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "f6790c46",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"string_distance\")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "49ad9139",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.11555555555555552}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"The job is completely done.\",\n",
-                "    reference=\"The job is done\",\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "c06a2296",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.0724999999999999}"
-                        ]
-                    },
-                    "execution_count": 4,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "# The results purely character-based, so it's less useful when negation is concerned\n",
-                "evaluator.evaluate_strings(\n",
-                "    prediction=\"The job is done.\",\n",
-                "    reference=\"The job isn't done\",\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "b8ed1f12-09a6-4e90-a69d-c8df525ff293",
-            "metadata": {},
-            "source": [
-                "## Configure the String Distance Metric\n",
-                "\n",
-                "By default, the `StringDistanceEvalChain` uses  levenshtein distance, but it also supports other string distance algorithms. Configure using the `distance` argument."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "id": "a88bc7d7-62d3-408d-b0e0-43abcecf35c8",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "[<StringDistance.DAMERAU_LEVENSHTEIN: 'damerau_levenshtein'>,\n",
-                            " <StringDistance.LEVENSHTEIN: 'levenshtein'>,\n",
-                            " <StringDistance.JARO: 'jaro'>,\n",
-                            " <StringDistance.JARO_WINKLER: 'jaro_winkler'>]"
-                        ]
-                    },
-                    "execution_count": 5,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "from langchain.evaluation import StringDistance\n",
-                "\n",
-                "list(StringDistance)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "0c079864-0175-4d06-9d3f-a0e51dd3977c",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "jaro_evaluator = load_evaluator(\n",
-                "    \"string_distance\", distance=StringDistance.JARO\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "id": "a8dfb900-14f3-4a1f-8736-dd1d86a1264c",
-            "metadata": {},
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.19259259259259254}"
-                        ]
-                    },
-                    "execution_count": 7,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "jaro_evaluator.evaluate_strings(\n",
-                "    prediction=\"The job is completely done.\",\n",
-                "    reference=\"The job is done\",\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "id": "7020b046-0ef7-40cc-8778-b928e35f3ce1",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 0.12083333333333324}"
-                        ]
-                    },
-                    "execution_count": 8,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "jaro_evaluator.evaluate_strings(\n",
-                "    prediction=\"The job is done.\",\n",
-                "    reference=\"The job isn't done\",\n",
-                ")"
-            ]
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/evaluation/trajectory/custom.ipynb

@@ -1,142 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "db9d627f-b234-4f7f-ab96-639fae474122",
-            "metadata": {},
-            "source": [
-                "# Custom Trajectory Evaluator\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/trajectory/custom.ipynb)\n",
-                "\n",
-                "You can make your own custom trajectory evaluators by inheriting from the [AgentTrajectoryEvaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator) class and overwriting the `_evaluate_agent_trajectory` (and `_aevaluate_agent_action`) method.\n",
-                "\n",
-                "\n",
-                "In this example, you will make a simple trajectory evaluator that uses an LLM to determine if any actions were unnecessary."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "ca84ab0c-e7e2-4c03-bd74-9cc4e6338eec",
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "from typing import Any, Optional, Sequence, Tuple\n",
-                "from langchain.chat_models import ChatOpenAI\n",
-                "from langchain.chains import LLMChain\n",
-                "from langchain.schema import AgentAction\n",
-                "from langchain.evaluation import AgentTrajectoryEvaluator\n",
-                "\n",
-                "\n",
-                "class StepNecessityEvaluator(AgentTrajectoryEvaluator):\n",
-                "    \"\"\"Evaluate the perplexity of a predicted string.\"\"\"\n",
-                "\n",
-                "    def __init__(self) -> None:\n",
-                "        llm = ChatOpenAI(model=\"gpt-4\", temperature=0.0)\n",
-                "        template = \"\"\"Are any of the following steps unnecessary in answering {input}? Provide the verdict on a new line as a single \"Y\" for yes or \"N\" for no.\n",
-                "\n",
-                "        DATA\n",
-                "        ------\n",
-                "        Steps: {trajectory}\n",
-                "        ------\n",
-                "\n",
-                "        Verdict:\"\"\"\n",
-                "        self.chain = LLMChain.from_string(llm, template)\n",
-                "\n",
-                "    def _evaluate_agent_trajectory(\n",
-                "        self,\n",
-                "        *,\n",
-                "        prediction: str,\n",
-                "        input: str,\n",
-                "        agent_trajectory: Sequence[Tuple[AgentAction, str]],\n",
-                "        reference: Optional[str] = None,\n",
-                "        **kwargs: Any,\n",
-                "    ) -> dict:\n",
-                "        vals = [\n",
-                "            f\"{i}: Action=[{action.tool}] returned observation = [{observation}]\"\n",
-                "            for i, (action, observation) in enumerate(agent_trajectory)\n",
-                "        ]\n",
-                "        trajectory = \"\\n\".join(vals)\n",
-                "        response = self.chain.run(dict(trajectory=trajectory, input=input), **kwargs)\n",
-                "        decision = response.split(\"\\n\")[-1].strip()\n",
-                "        score = 1 if decision == \"Y\" else 0\n",
-                "        return {\"score\": score, \"value\": decision, \"reasoning\": response}"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "297dea4b-fb28-4292-b6e0-1c769cfb9cbd",
-            "metadata": {},
-            "source": [
-                "The example above will return a score of 1 if the language model predicts that any of the actions were unnecessary, and it returns a score of 0 if all of them were predicted to be necessary. It returns the string 'decision' as the 'value', and includes the rest of the generated text as 'reasoning' to let you audit the decision.\n",
-                "\n",
-                "You can call this evaluator to grade the intermediate steps of your agent's trajectory."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "a3fbcc1d-249f-4e00-8841-b6872c73c486",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1, 'value': 'Y', 'reasoning': 'Y'}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluator = StepNecessityEvaluator()\n",
-                "\n",
-                "evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=\"The answer is pi\",\n",
-                "    input=\"What is today?\",\n",
-                "    agent_trajectory=[\n",
-                "        (\n",
-                "            AgentAction(tool=\"ask\", tool_input=\"What is today?\", log=\"\"),\n",
-                "            \"tomorrow's yesterday\",\n",
-                "        ),\n",
-                "        (\n",
-                "            AgentAction(tool=\"check_tv\", tool_input=\"Watch tv for half hour\", log=\"\"),\n",
-                "            \"bzzz\",\n",
-                "        ),\n",
-                "    ],\n",
-                ")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "77353528-723e-4075-939e-aebdb17c1e4f",
-            "metadata": {},
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/evaluation/trajectory/index.mdx

@@ -1,28 +0,0 @@
----
-sidebar_position: 4
----
-# Trajectory Evaluators
-
-Trajectory Evaluators in LangChain provide a more holistic approach to evaluating an agent. These evaluators assess the full sequence of actions taken by an agent and their corresponding responses, which we refer to as the "trajectory". This allows you to better measure an agent's effectiveness and capabilities.
-
-A Trajectory Evaluator implements the `AgentTrajectoryEvaluator` interface, which requires two main methods:
-
-- `evaluate_agent_trajectory`: This method synchronously evaluates an agent's trajectory.
-- `aevaluate_agent_trajectory`: This asynchronous counterpart allows evaluations to be run in parallel for efficiency.
-
-Both methods accept three main parameters:
-
-- `input`: The initial input given to the agent.
-- `prediction`: The final predicted response from the agent.
-- `agent_trajectory`: The intermediate steps taken by the agent, given as a list of tuples.
-
-These methods return a dictionary. It is recommended that custom implementations return a `score` (a float indicating the effectiveness of the agent) and `reasoning` (a string explaining the reasoning behind the score).
-
-You can capture an agent's trajectory by initializing the agent with the `return_intermediate_steps=True` parameter. This lets you collect all intermediate steps without relying on special callbacks.
-
-For a deeper dive into the implementation and use of Trajectory Evaluators, refer to the sections below.
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />
-

docs/docs/guides/evaluation/trajectory/trajectory_eval.ipynb

@@ -1,305 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "6e5ea1a1-7e74-459b-bf14-688f87d09124",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "# Agent Trajectory\n",
-                "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/trajectory/trajectory_eval.ipynb)\n",
-                "\n",
-                "Agents can be difficult to holistically evaluate due to the breadth of actions and generation they can make. We recommend using multiple evaluation techniques appropriate to your use case. One way to evaluate an agent is to look at the whole trajectory of actions taken along with their responses.\n",
-                "\n",
-                "Evaluators that do this can implement the `AgentTrajectoryEvaluator` interface. This walkthrough will show how to use the `trajectory` evaluator to grade  an OpenAI functions agent.\n",
-                "\n",
-                "For more information, check out the reference docs for the [TrajectoryEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain) for more info."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "149402da-5212-43e2-b7c0-a701727f5293",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"trajectory\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "b1c64c1a",
-            "metadata": {},
-            "source": [
-                "## Methods\n",
-                "\n",
-                "\n",
-                "The Agent Trajectory Evaluators are used with the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.evaluate_agent_trajectory) (and async [aevaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.html#langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain.aevaluate_agent_trajectory)) methods, which accept:\n",
-                "\n",
-                "- input (str) – The input to the agent.\n",
-                "- prediction (str) – The final predicted response.\n",
-                "- agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory\n",
-                "\n",
-                "They return a dictionary with the following values:\n",
-                "- score: Float from 0 to 1, where 1 would mean \"most effective\" and 0 would mean \"least effective\"\n",
-                "- reasoning: String \"chain of thought reasoning\" from the LLM generated prior to creating the score"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "e733562c-4c17-4942-9647-acfc5ebfaca2",
-            "metadata": {},
-            "source": [
-                "## Capturing Trajectory\n",
-                "\n",
-                "The easiest way to return an agent's trajectory (without using tracing callbacks like those in LangSmith) for evaluation is to initialize the agent with `return_intermediate_steps=True`.\n",
-                "\n",
-                "Below, create an example agent we will call to evaluate."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "451cb0cb-6f42-4abd-aa6d-fb871fce034d",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "import os\n",
-                "import subprocess\n",
-                "\n",
-                "from langchain.chat_models import ChatOpenAI\n",
-                "from langchain.tools import tool\n",
-                "from langchain.agents import AgentType, initialize_agent\n",
-                "\n",
-                "from pydantic import HttpUrl\n",
-                "from urllib.parse import urlparse\n",
-                "\n",
-                "\n",
-                "@tool\n",
-                "def ping(url: HttpUrl, return_error: bool) -> str:\n",
-                "    \"\"\"Ping the fully specified url. Must include https:// in the url.\"\"\"\n",
-                "    hostname = urlparse(str(url)).netloc\n",
-                "    completed_process = subprocess.run(\n",
-                "        [\"ping\", \"-c\", \"1\", hostname], capture_output=True, text=True\n",
-                "    )\n",
-                "    output = completed_process.stdout\n",
-                "    if return_error and completed_process.returncode != 0:\n",
-                "        return completed_process.stderr\n",
-                "    return output\n",
-                "\n",
-                "\n",
-                "@tool\n",
-                "def trace_route(url: HttpUrl, return_error: bool) -> str:\n",
-                "    \"\"\"Trace the route to the specified url. Must include https:// in the url.\"\"\"\n",
-                "    hostname = urlparse(str(url)).netloc\n",
-                "    completed_process = subprocess.run(\n",
-                "        [\"traceroute\", hostname], capture_output=True, text=True\n",
-                "    )\n",
-                "    output = completed_process.stdout\n",
-                "    if return_error and completed_process.returncode != 0:\n",
-                "        return completed_process.stderr\n",
-                "    return output\n",
-                "\n",
-                "\n",
-                "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0613\", temperature=0)\n",
-                "agent = initialize_agent(\n",
-                "    llm=llm,\n",
-                "    tools=[ping, trace_route],\n",
-                "    agent=AgentType.OPENAI_MULTI_FUNCTIONS,\n",
-                "    return_intermediate_steps=True,  # IMPORTANT!\n",
-                ")\n",
-                "\n",
-                "result = agent(\"What's the latency like for https://langchain.com?\")"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "2df34eed-45a5-4f91-88d3-9aa55f28391a",
-            "metadata": {
-                "tags": []
-            },
-            "source": [
-                "## Evaluate Trajectory\n",
-                "\n",
-                "Pass the input, trajectory, and pass to the [evaluate_agent_trajectory](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator.evaluate_agent_trajectory) method."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "8d2c8703-98ed-4068-8a8b-393f0f1f64ea",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1.0,\n",
-                            " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the website https://langchain.com.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. It uses the 'ping' tool to measure the latency of the website, which is the correct tool for this task.\\n\\niii. The AI language model uses the tool in a helpful way. It inputs the URL into the 'ping' tool and correctly interprets the output to provide the latency in milliseconds.\\n\\niv. The AI language model does not use too many steps to answer the question. It only uses one step, which is appropriate for this type of question.\\n\\nv. The appropriate tool is used to answer the question. The 'ping' tool is the correct tool to measure website latency.\\n\\nGiven these considerations, the AI language model's performance is excellent. It uses the correct tool, interprets the output correctly, and provides a helpful and direct answer to the user's question.\"}"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=result[\"output\"],\n",
-                "    input=result[\"input\"],\n",
-                "    agent_trajectory=result[\"intermediate_steps\"],\n",
-                ")\n",
-                "evaluation_result"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "fc5467c1-ea92-405f-949a-3011388fa9ee",
-            "metadata": {},
-            "source": [
-                "## Configuring the Evaluation LLM\n",
-                "\n",
-                "If you don't select an LLM to use for evaluation, the [load_evaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.loading.load_evaluator.html#langchain.evaluation.loading.load_evaluator) function will use `gpt-4` to power the evaluation chain. You can select any chat model for the agent trajectory evaluator as below."
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 4,
-            "id": "1f6318f3-642a-4766-bc7a-f91239795ee7",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "# %pip install anthropic\n",
-                "# ANTHROPIC_API_KEY=<YOUR ANTHROPIC API KEY>"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 5,
-            "id": "b2852289-5df9-402e-95b5-7efebf0fc943",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.chat_models import ChatAnthropic\n",
-                "\n",
-                "eval_llm = ChatAnthropic(temperature=0)\n",
-                "evaluator = load_evaluator(\"trajectory\", llm=eval_llm)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 6,
-            "id": "ff72d21a-93b9-4c2f-8613-733d9c9330d7",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1.0,\n",
-                            " 'reasoning': \"Here is my detailed evaluation of the AI's response:\\n\\ni. The final answer is helpful, as it directly provides the latency measurement for the requested website.\\n\\nii. The sequence of using the ping tool to measure latency is logical for this question.\\n\\niii. The ping tool is used in a helpful way, with the website URL provided as input and the output latency measurement extracted.\\n\\niv. Only one step is used, which is appropriate for simply measuring latency. More steps are not needed.\\n\\nv. The ping tool is an appropriate choice to measure latency. \\n\\nIn summary, the AI uses an optimal single step approach with the right tool and extracts the needed output. The final answer directly answers the question in a helpful way.\\n\\nOverall\"}"
-                        ]
-                    },
-                    "execution_count": 6,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=result[\"output\"],\n",
-                "    input=result[\"input\"],\n",
-                "    agent_trajectory=result[\"intermediate_steps\"],\n",
-                ")\n",
-                "evaluation_result"
-            ]
-        },
-        {
-            "cell_type": "markdown",
-            "id": "95ce4240-f5a0-4810-8d09-b2f4c9e18b7f",
-            "metadata": {},
-            "source": [
-                "## Providing List of Valid Tools\n",
-                "\n",
-                "By default, the evaluator doesn't take into account the tools the agent is permitted to call. You can provide these to the evaluator via the `agent_tools` argument.\n"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 7,
-            "id": "24c10566-2ef5-45c5-9213-a8fb28e2ca1f",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.evaluation import load_evaluator\n",
-                "\n",
-                "evaluator = load_evaluator(\"trajectory\", agent_tools=[ping, trace_route])"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 8,
-            "id": "7b995786-5b78-4d9e-8e8a-1f2a203113e2",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "{'score': 1.0,\n",
-                            " 'reasoning': \"i. The final answer is helpful. It directly answers the user's question about the latency for the specified website.\\n\\nii. The AI language model uses a logical sequence of tools to answer the question. In this case, only one tool was needed to answer the question, and the model chose the correct one.\\n\\niii. The AI language model uses the tool in a helpful way. The 'ping' tool was used to determine the latency of the website, which was the information the user was seeking.\\n\\niv. The AI language model does not use too many steps to answer the question. Only one step was needed and used.\\n\\nv. The appropriate tool was used to answer the question. The 'ping' tool is designed to measure latency, which was the information the user was seeking.\\n\\nGiven these considerations, the AI language model's performance in answering this question is excellent.\"}"
-                        ]
-                    },
-                    "execution_count": 8,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "evaluation_result = evaluator.evaluate_agent_trajectory(\n",
-                "    prediction=result[\"output\"],\n",
-                "    input=result[\"input\"],\n",
-                "    agent_trajectory=result[\"intermediate_steps\"],\n",
-                ")\n",
-                "evaluation_result"
-            ]
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.2"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/guides/fallbacks.ipynb

@@ -1,432 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "19c9cbd6",
-   "metadata": {},
-   "source": [
-    "# Fallbacks\n",
-    "\n",
-    "When working with language models, you may often encounter issues from the underlying APIs, whether these be rate limiting or downtime. Therefore, as you go to move your LLM applications into production it becomes more and more important to safeguard against these. That's why we've introduced the concept of fallbacks. \n",
-    "\n",
-    "A **fallback** is an alternative plan that may be used in an emergency.\n",
-    "\n",
-    "Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level. This is important because often times different models require different prompts. So if your call to OpenAI fails, you don't just want to send the same prompt to Anthropic - you probably want to use a different prompt template and send a different version there."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a6bb9ba9",
-   "metadata": {},
-   "source": [
-    "## Fallback for LLM API Errors\n",
-    "\n",
-    "This is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit rate limits, any number of things. Therefore, using fallbacks can help protect against these types of things.\n",
-    "\n",
-    "IMPORTANT: By default, a lot of the LLM wrappers catch errors and retry. You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying and not failing."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "d3e893bf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4847c82d",
-   "metadata": {},
-   "source": [
-    "First, let's mock out what happens if we hit a RateLimitError from OpenAI"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "dfdd8bf5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from unittest.mock import patch\n",
-    "from openai.error import RateLimitError"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "e6fdffc1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Note that we set max_retries = 0 to avoid retrying on RateLimits, etc\n",
-    "openai_llm = ChatOpenAI(max_retries=0)\n",
-    "anthropic_llm = ChatAnthropic()\n",
-    "llm = openai_llm.with_fallbacks([anthropic_llm])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 27,
-   "id": "584461ab",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Hit error\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Let's use just the OpenAI LLm first, to show that we run into an error\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(openai_llm.invoke(\"Why did the chicken cross the road?\"))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 28,
-   "id": "4fc1e673",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "content=' I don\\'t actually know why the chicken crossed the road, but here are some possible humorous answers:\\n\\n- To get to the other side!\\n\\n- It was too chicken to just stand there. \\n\\n- It wanted a change of scenery.\\n\\n- It wanted to show the possum it could be done.\\n\\n- It was on its way to a poultry farmers\\' convention.\\n\\nThe joke plays on the double meaning of \"the other side\" - literally crossing the road to the other side, or the \"other side\" meaning the afterlife. So it\\'s an anti-joke, with a silly or unexpected pun as the answer.' additional_kwargs={} example=False\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Now let's try with fallbacks to Anthropic\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(llm.invoke(\"Why did the the chicken cross the road?\"))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f00bea25",
-   "metadata": {},
-   "source": [
-    "We can use our \"LLM with Fallbacks\" as we would a normal LLM."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "4f8eaaa0",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "content=\" I don't actually know why the kangaroo crossed the road, but I can take a guess! Here are some possible reasons:\\n\\n- To get to the other side (the classic joke answer!)\\n\\n- It was trying to find some food or water \\n\\n- It was trying to find a mate during mating season\\n\\n- It was fleeing from a predator or perceived threat\\n\\n- It was disoriented and crossed accidentally \\n\\n- It was following a herd of other kangaroos who were crossing\\n\\n- It wanted a change of scenery or environment \\n\\n- It was trying to reach a new habitat or territory\\n\\nThe real reason is unknown without more context, but hopefully one of those potential explanations does the joke justice! Let me know if you have any other animal jokes I can try to decipher.\" additional_kwargs={} example=False\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"You're a nice assistant who always includes a compliment in your response\"),\n",
-    "        (\"human\", \"Why did the {animal} cross the road\"),\n",
-    "    ]\n",
-    ")\n",
-    "chain = prompt | llm\n",
-    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
-    "    try:\n",
-    "         print(chain.invoke({\"animal\": \"kangaroo\"}))\n",
-    "    except:\n",
-    "        print(\"Hit error\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8d62241b",
-   "metadata": {},
-   "source": [
-    "## Fallback for Sequences\n",
-    "\n",
-    "We can also create fallbacks for sequences, that are sequences themselves. Here we do that with two different models: ChatOpenAI and then normal OpenAI (which does not use a chat model). Because OpenAI is NOT a chat model, you likely want a different prompt."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "6d0b8056",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# First let's create a chain with a ChatModel\n",
-    "# We add in a string output parser here so the outputs between the two are the same type\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "\n",
-    "chat_prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"You're a nice assistant who always includes a compliment in your response\"),\n",
-    "        (\"human\", \"Why did the {animal} cross the road\"),\n",
-    "    ]\n",
-    ")\n",
-    "# Here we're going to use a bad model name to easily create a chain that will error\n",
-    "chat_model = ChatOpenAI(model_name=\"gpt-fake\")\n",
-    "bad_chain = chat_prompt | chat_model | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "8d1fc2a5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Now lets create a chain with the normal OpenAI model\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "prompt_template = \"\"\"Instructions: You should always include a compliment in your response.\n",
-    "\n",
-    "Question: Why did the {animal} cross the road?\"\"\"\n",
-    "prompt = PromptTemplate.from_template(prompt_template)\n",
-    "llm = OpenAI()\n",
-    "good_chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 32,
-   "id": "283bfa44",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'\\n\\nAnswer: The turtle crossed the road to get to the other side, and I have to say he had some impressive determination.'"
-      ]
-     },
-     "execution_count": 32,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can now create a final chain which combines the two\n",
-    "chain = bad_chain.with_fallbacks([good_chain])\n",
-    "chain.invoke({\"animal\": \"turtle\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ec4685b4",
-   "metadata": {},
-   "source": [
-    "## Fallback for Long Inputs\n",
-    "\n",
-    "One of the big limiting factors of LLMs is their context window. Usually, you can count and track the length of prompts before sending them to an LLM, but in situations where that is hard/complicated, you can fallback to a model with a longer context length."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 34,
-   "id": "564b84c9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "short_llm = ChatOpenAI()\n",
-    "long_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\")\n",
-    "llm = short_llm.with_fallbacks([long_llm])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 38,
-   "id": "5e27a775",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "inputs = \"What is the next number: \" + \", \".join([\"one\", \"two\"] * 3000)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 40,
-   "id": "0a502731",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "This model's maximum context length is 4097 tokens. However, your messages resulted in 12012 tokens. Please reduce the length of the messages.\n"
-     ]
-    }
-   ],
-   "source": [
-    "try:\n",
-    "    print(short_llm.invoke(inputs))\n",
-    "except Exception as e:\n",
-    "    print(e)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 41,
-   "id": "d91ba5d7",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "content='The next number in the sequence is two.' additional_kwargs={} example=False\n"
-     ]
-    }
-   ],
-   "source": [
-    "try:\n",
-    "    print(llm.invoke(inputs))\n",
-    "except Exception as e:\n",
-    "    print(e)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2a6735df",
-   "metadata": {},
-   "source": [
-    "## Fallback to Better Model\n",
-    "\n",
-    "Often times we ask models to output format in a specific format (like JSON). Models like GPT-3.5 can do this okay, but sometimes struggle. This naturally points to fallbacks - we can try with GPT-3.5 (faster, cheaper), but then if parsing fails we can use GPT-4."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "867a3793",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.output_parsers import DatetimeOutputParser"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 67,
-   "id": "b8d9959d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = ChatPromptTemplate.from_template(\n",
-    "    \"what time was {event} (in %Y-%m-%dT%H:%M:%S.%fZ format - only return this value)\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 75,
-   "id": "98087a76",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# In this case we are going to do the fallbacks on the LLM + output parser level\n",
-    "# Because the error will get raised in the OutputParser\n",
-    "openai_35 = ChatOpenAI() | DatetimeOutputParser()\n",
-    "openai_4 = ChatOpenAI(model=\"gpt-4\")| DatetimeOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 77,
-   "id": "17ec9e8f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "only_35 = prompt | openai_35 \n",
-    "fallback_4 = prompt | openai_35.with_fallbacks([openai_4])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 80,
-   "id": "7e536f0b",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Error: Could not parse datetime string: The Super Bowl in 1994 took place on January 30th at 3:30 PM local time. Converting this to the specified format (%Y-%m-%dT%H:%M:%S.%fZ) results in: 1994-01-30T15:30:00.000Z\n"
-     ]
-    }
-   ],
-   "source": [
-    "try:\n",
-    "    print(only_35.invoke({\"event\": \"the superbowl in 1994\"}))\n",
-    "except Exception as e:\n",
-    "    print(f\"Error: {e}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 81,
-   "id": "01355c5e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "1994-01-30 15:30:00\n"
-     ]
-    }
-   ],
-   "source": [
-    "try:\n",
-    "    print(fallback_4.invoke({\"event\": \"the superbowl in 1994\"}))\n",
-    "except Exception as e:\n",
-    "    print(f\"Error: {e}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c537f9d0",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}