Merge branch 'master' into eugene/expand_documentation

.devcontainer/README.md

@@ -15,11 +15,7 @@ You may use the button above, or follow these steps to open this repo in a Codes
 For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
   
 ## VS Code Dev Containers
-[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/langchain-ai/langchain)
-
-Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name: 
-https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/<yourusername>/<yourclonedreponame>
-
+[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 
 If you already have VS Code and Docker installed, you can use the button above to get started. This will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
 
@@ -29,7 +25,7 @@ You can also follow these steps to open this repo in a container using the VS Co
 
 2. Open a locally cloned copy of the code:
 
-   - Fork and Clone this repository to your local filesystem.
+   - Clone this repository to your local filesystem.
    - Press <kbd>F1</kbd> and select the **Dev Containers: Open Folder in Container...** command.
    - Select the cloned copy of this folder, wait for the container to start, and try things out!
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,20 +1,28 @@
 <!-- Thank you for contributing to LangChain!
 
-Replace this entire comment with:
+Replace this comment with:
   - Description: a description of the change, 
   - Issue: the issue # it fixes (if applicable),
   - Dependencies: any dependencies required for this change,
   - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
   - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
 
-Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
-
-See contribution guidelines for more information on how to write/run tests, lint, etc: 
-https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
+Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
 
 If you're adding a new integration, please include:
   1. a test for the integration, preferably unit tests that do not rely on network access,
-  2. an example notebook showing its use. These live is docs/extras directory.
+  2. an example notebook showing its use.
 
-If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
+Maintainer responsibilities:
+  - General / Misc / if you don't know who to tag: @baskaryan
+  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
+  - Models / Prompts: @hwchase17, @baskaryan
+  - Memory: @hwchase17
+  - Agents / Tools / Toolkits: @hinthornw
+  - Tracing / Callbacks: @agola11
+  - Async: @agola11
+
+If no one reviews your PR within a few days, feel free to @-mention the same people again.
+
+See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
  -->

.github/actions/poetry_setup/action.yml

@@ -62,19 +62,6 @@ runs:
       run: |
         poetry lock --check
 
-    - name: Set proper Poetry.lock file
-      shell: bash
-      env:
-        WORKDIR: ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}
-      run: |
-        if [ -f "$WORKDIR/poetry.lock" ]; then
-          echo 'Using working directory poetry.lock in cache key'
-          cp "$WORKDIR/poetry.lock" poetry-lock.cache-key
-        else
-          echo 'Using the top-level poetry.lock in cache key'
-          cp poetry.lock poetry-lock.cache-key
-        fi
-
     - uses: actions/cache@v3
       id: cache-poetry
       env:
@@ -84,8 +71,7 @@ runs:
           ~/.cache/pypoetry/virtualenvs
           ~/.cache/pypoetry/cache
           ~/.cache/pypoetry/artifacts
-          ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}/.venv
-        key: poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles('poetry-lock.cache-key') }}
+        key: poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles('poetry.lock') }}
 
     - run: ${{ inputs.install-command }}
       working-directory: ${{ inputs.working-directory }}

.github/workflows/_lint.yml

@@ -26,27 +26,14 @@ jobs:
           - "3.11"
     steps:
       - uses: actions/checkout@v3
-      - uses: actions/cache@v3
-        id: cache-pip
-        name: Cache langchain editable pip install - ${{ matrix.python-version }}
-        env:
-          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "15"
-        with:
-          path: |
-            ~/.cache/pip
-          key: pip-editable-langchain-deps-${{ runner.os }}-${{ runner.arch }}-py-${{ matrix.python-version }}
       - name: Install poetry
         run: |
           pipx install poetry==$POETRY_VERSION
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v4
-        env:
-          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "15"
         with:
           python-version: ${{ matrix.python-version }}
           cache: poetry
-          cache-dependency-path: |
-            ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}/**/poetry.lock
       - name: Install dependencies
         run: |
           poetry install

.github/workflows/_release.yml

@@ -37,7 +37,6 @@ jobs:
           echo version=$(poetry version --short) >> $GITHUB_OUTPUT
       - name: Create Release
         uses: ncipollo/release-action@v1
-        if: ${{ inputs.working-directory == 'libs/langchain' }}
         with:
           artifacts: "dist/*"
           token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/_test.yml

@@ -10,7 +10,7 @@ on:
       test_type:
         type: string
         description: "Test types to run"
-        default: '["core", "extended", "core-pydantic-2"]'
+        default: '["core", "extended"]'
 
 env:
   POETRY_VERSION: "1.4.2"
@@ -43,42 +43,19 @@ jobs:
               if [ "${{ matrix.test_type }}" == "core" ]; then
                 echo "Running core tests, installing dependencies with poetry..."
                 poetry install
-              elif [ "${{ matrix.test_type }}" == "core-pydantic-2" ]; then
-                echo "Running core-pydantic-v2 tests, installing dependencies with poetry..."
-                poetry install
-                poetry add pydantic@2.1
               else
                 echo "Running extended tests, installing dependencies with poetry..."
                 poetry install -E extended_testing
               fi
-      - name: Verify pydantic version
+      - name: Install langchain editable
+        if: ${{ inputs.working-directory != 'langchain' }}
         run: |
-          if [ "${{ matrix.test_type }}" == "core-pydantic-2" ]; then
-            EXPECTED_VERSION=2
-          else
-            EXPECTED_VERSION=1
-          fi
-          echo "Checking pydantic version... Expecting ${EXPECTED_VERSION}"
-          
-          # Determine the major part of pydantic version
-          VERSION=$(poetry run python -c "import pydantic; print(pydantic.__version__)" | cut -d. -f1)
-          
-          # Check that the major part of pydantic version is as expected, if not
-          # raise an error
-          if [[ "$VERSION" -ne $EXPECTED_VERSION ]]; then
-            echo "Error: pydantic version must be equal to ${EXPECTED_VERSION}; Found: ${VERSION}"
-            exit 1
-          fi
-          echo "Found pydantic version ${VERSION}, as expected"
-        shell: bash
+          pip install -e ../langchain
       - name: Run ${{matrix.test_type}} tests
         run: |
-          case "${{ matrix.test_type }}" in
-              core | core-pydantic-2)
-                  make test
-                  ;;
-              *)
-                  make extended_tests
-                  ;;
-          esac
+          if [ "${{ matrix.test_type }}" == "core" ]; then
+            make test
+          else
+            make extended_tests
+          fi
         shell: bash

.github/workflows/langchain_ci.yml

@@ -24,5 +24,4 @@ jobs:
       ./.github/workflows/_test.yml
     with:
       working-directory: libs/langchain
-      test_type: '["core", "extended", "core-pydantic-2"]'
     secrets: inherit

.github/workflows/langchain_experimental_ci.yml

@@ -1,5 +1,5 @@
 ---
-name: libs/experimental CI
+name: libs/langchain-experimental CI
 
 on:
   push:

.github/workflows/langchain_experimental_release.yml

@@ -1,5 +1,5 @@
 ---
-name: libs/experimental Release
+name: libs/langchain-experimental Release
 
 on:
   pull_request:

.github/workflows/scheduled_test.yml

@@ -1,42 +0,0 @@
-name: Scheduled tests
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-  schedule:
-    - cron:  '0 13 * * *'
-
-env:
-  POETRY_VERSION: "1.4.2"
-
-jobs:
-  build:
-    defaults:
-      run:
-        working-directory: libs/langchain
-    runs-on: ubuntu-latest
-    environment: Scheduled testing
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: "1.4.2"
-          working-directory: libs/langchain
-          install-command: |
-            echo "Running scheduled tests, installing dependencies with poetry..."
-            poetry install --with=test_integration
-      - name: Run tests
-        env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-        run: |
-          make scheduled_tests
-        shell: bash

.gitignore

@@ -162,7 +162,6 @@ docs/.docusaurus/
 docs/.cache-loader/
 docs/_dist
 docs/api_reference/api_reference.rst
-docs/api_reference/experimental_api_reference.rst
 docs/api_reference/_build
 docs/api_reference/*/
 !docs/api_reference/_static/

MIGRATE.md

@@ -43,10 +43,6 @@ Now:
 
 `from langchain_experimental.sql import SQLDatabaseChain`
 
-Alternatively, if you are just interested in using the query generation part of the SQL chain, you can check out [`create_sql_query_chain`](https://github.com/langchain-ai/langchain/blob/master/docs/extras/use_cases/tabular/sql_query.ipynb)
-
-`from langchain.chains import create_sql_query_chain`
-
 ## `load_prompt` for Python files
 
 Note: this only applies if you want to load Python files as prompts.

Makefile

@@ -43,12 +43,7 @@ spell_fix:
 
 help:
 	@echo '----'
-	@echo 'clean                        - run docs_clean and api_docs_clean'
+	@echo 'coverage                     - run unit tests and generate coverage report'
 	@echo 'docs_build                   - build the documentation'
 	@echo 'docs_clean                   - clean the documentation build artifacts'
 	@echo 'docs_linkcheck               - run linkchecker on the documentation'
-	@echo 'api_docs_build               - build the API Reference documentation'
-	@echo 'api_docs_clean               - clean the API Reference documentation build artifacts'
-	@echo 'api_docs_linkcheck           - run linkchecker on the API Reference documentation'
-	@echo 'spell_check               	- run codespell on the project'
-	@echo 'spell_fix               		- run codespell on the project and fix the errors'

README.md

@@ -12,16 +12,16 @@
 [![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hwchase17/langchain)
 [![GitHub star chart](https://img.shields.io/github/stars/hwchase17/langchain?style=social)](https://star-history.com/#hwchase17/langchain)
-[![Dependency Status](https://img.shields.io/librariesio/github/langchain-ai/langchain)](https://libraries.io/github/langchain-ai/langchain)
+[![Dependency Status](https://img.shields.io/librariesio/github/hwchase17/langchain)](https://libraries.io/github/hwchase17/langchain)
 [![Open Issues](https://img.shields.io/github/issues-raw/hwchase17/langchain)](https://github.com/hwchase17/langchain/issues)
 
 
 Looking for the JS/TS version? Check out [LangChain.js](https://github.com/hwchase17/langchainjs).
 
-**Production Support:** As you move your LangChains into production, we'd love to offer more hands-on support.
-Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) to share more about what you're building, and our team will get in touch.
+**Production Support:** As you move your LangChains into production, we'd love to offer more comprehensive support.
+Please fill out [this form](https://6w1pwbss0py.typeform.com/to/rrbrdTH2) and we'll set up a dedicated support Slack channel.
 
-## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28/23
+## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28
 
 In an effort to make `langchain` leaner and safer, we are moving select chains to `langchain_experimental`.
 This migration has already started, but we are remaining backwards compatible until 7/28.

docs/.local_build.sh

@@ -13,6 +13,5 @@ cp -r {docs_skeleton,snippets} _dist
 cp -r extras/* _dist/docs_skeleton/docs
 cd _dist/docs_skeleton
 poetry run nbdoc_build
-poetry run python generate_api_reference_links.py
 yarn install
 yarn start

docs/api_reference/conf.py

@@ -23,7 +23,6 @@ from sphinx.util.docutils import SphinxDirective
 _DIR = Path(__file__).parent.absolute()
 sys.path.insert(0, os.path.abspath("."))
 sys.path.insert(0, os.path.abspath("../../libs/langchain"))
-sys.path.insert(0, os.path.abspath("../../libs/experimental"))
 
 with (_DIR.parents[1] / "libs" / "langchain" / "pyproject.toml").open("r") as f:
     data = toml.load(f)
@@ -100,9 +99,6 @@ extensions = [
 ]
 source_suffix = [".rst"]
 
-# some autodoc pydantic options are repeated in the actual template.
-# potentially user error, but there may be bugs in the sphinx extension
-# with options not being passed through correctly (from either the location in the code)
 autodoc_pydantic_model_show_json = False
 autodoc_pydantic_field_list_validators = False
 autodoc_pydantic_config_members = False
@@ -115,6 +111,13 @@ autodoc_member_order = "groupwise"
 autoclass_content = "both"
 autodoc_typehints_format = "short"
 
+autodoc_default_options = {
+    "members": True,
+    "show-inheritance": True,
+    "inherited-members": "BaseModel",
+    "undoc-members": True,
+    "special-members": "__call__",
+}
 # autodoc_typehints = "description"
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["templates"]

docs/api_reference/create_api_rst.py

@@ -1,260 +1,76 @@
-"""Script for auto-generating api_reference.rst."""
-import importlib
-import inspect
-import typing
+"""Script for auto-generating api_reference.rst"""
+import glob
+import re
 from pathlib import Path
-from typing import TypedDict, Sequence, List, Dict, Literal, Union
-from enum import Enum
-
-from pydantic import BaseModel
 
 ROOT_DIR = Path(__file__).parents[2].absolute()
-HERE = Path(__file__).parent
-
 PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
-EXP_DIR = ROOT_DIR / "libs" / "experimental" / "langchain_experimental"
-WRITE_FILE = HERE / "api_reference.rst"
-EXP_WRITE_FILE = HERE / "experimental_api_reference.rst"
+WRITE_FILE = Path(__file__).parent / "api_reference.rst"
 
 
-ClassKind = Literal["TypedDict", "Regular", "Pydantic", "enum"]
+def load_members() -> dict:
+    members: dict = {}
+    for py in glob.glob(str(PKG_DIR) + "/**/*.py", recursive=True):
+        module = py[len(str(PKG_DIR)) + 1 :].replace(".py", "").replace("/", ".")
+        top_level = module.split(".")[0]
+        if top_level not in members:
+            members[top_level] = {"classes": [], "functions": []}
+        with open(py, "r") as f:
+            for line in f.readlines():
+                cls = re.findall(r"^class ([^_].*)\(", line)
+                members[top_level]["classes"].extend([module + "." + c for c in cls])
+                func = re.findall(r"^def ([^_].*)\(", line)
+                afunc = re.findall(r"^async def ([^_].*)\(", line)
+                func_strings = [module + "." + f for f in func + afunc]
+                members[top_level]["functions"].extend(func_strings)
+    return members
 
 
-class ClassInfo(TypedDict):
-    """Information about a class."""
+def construct_doc(members: dict) -> str:
+    full_doc = """\
+.. _api_reference:
 
-    name: str
-    """The name of the class."""
-    qualified_name: str
-    """The fully qualified name of the class."""
-    kind: ClassKind
-    """The kind of the class."""
-    is_public: bool
-    """Whether the class is public or not."""
-
-
-class FunctionInfo(TypedDict):
-    """Information about a function."""
-
-    name: str
-    """The name of the function."""
-    qualified_name: str
-    """The fully qualified name of the function."""
-    is_public: bool
-    """Whether the function is public or not."""
-
-
-class ModuleMembers(TypedDict):
-    """A dictionary of module members."""
-
-    classes_: Sequence[ClassInfo]
-    functions: Sequence[FunctionInfo]
-
-
-def _load_module_members(module_path: str, namespace: str) -> ModuleMembers:
-    """Load all members of a module.
-
-    Args:
-        module_path: Path to the module.
-        namespace: the namespace of the module.
-
-    Returns:
-        list: A list of loaded module objects.
-    """
-    classes_: List[ClassInfo] = []
-    functions: List[FunctionInfo] = []
-    module = importlib.import_module(module_path)
-    for name, type_ in inspect.getmembers(module):
-        if not hasattr(type_, "__module__"):
-            continue
-        if type_.__module__ != module_path:
-            continue
-
-        if inspect.isclass(type_):
-            if type(type_) == typing._TypedDictMeta:  # type: ignore
-                kind: ClassKind = "TypedDict"
-            elif issubclass(type_, Enum):
-                kind = "enum"
-            elif issubclass(type_, BaseModel):
-                kind = "Pydantic"
-            else:
-                kind = "Regular"
-
-            classes_.append(
-                ClassInfo(
-                    name=name,
-                    qualified_name=f"{namespace}.{name}",
-                    kind=kind,
-                    is_public=not name.startswith("_"),
-                )
-            )
-        elif inspect.isfunction(type_):
-            functions.append(
-                FunctionInfo(
-                    name=name,
-                    qualified_name=f"{namespace}.{name}",
-                    is_public=not name.startswith("_"),
-                )
-            )
-        else:
-            continue
-
-    return ModuleMembers(
-        classes_=classes_,
-        functions=functions,
-    )
-
-
-def _merge_module_members(
-    module_members: Sequence[ModuleMembers],
-) -> ModuleMembers:
-    """Merge module members."""
-    classes_: List[ClassInfo] = []
-    functions: List[FunctionInfo] = []
-    for module in module_members:
-        classes_.extend(module["classes_"])
-        functions.extend(module["functions"])
-
-    return ModuleMembers(
-        classes_=classes_,
-        functions=functions,
-    )
-
-
-def _load_package_modules(
-    package_directory: Union[str, Path]
-) -> Dict[str, ModuleMembers]:
-    """Recursively load modules of a package based on the file system.
-
-    Traversal based on the file system makes it easy to determine which
-    of the modules/packages are part of the package vs. 3rd party or built-in.
-
-    Parameters:
-        package_directory: Path to the package directory.
-
-    Returns:
-        list: A list of loaded module objects.
-    """
-    package_path = (
-        Path(package_directory)
-        if isinstance(package_directory, str)
-        else package_directory
-    )
-    modules_by_namespace = {}
-
-    package_name = package_path.name
-
-    for file_path in package_path.rglob("*.py"):
-        if file_path.name.startswith("_"):
-            continue
-
-        relative_module_name = file_path.relative_to(package_path)
-
-        # Skip if any module part starts with an underscore
-        if any(part.startswith("_") for part in relative_module_name.parts):
-            continue
-
-        # Get the full namespace of the module
-        namespace = str(relative_module_name).replace(".py", "").replace("/", ".")
-        # Keep only the top level namespace
-        top_namespace = namespace.split(".")[0]
-
-        try:
-            module_members = _load_module_members(
-                f"{package_name}.{namespace}", namespace
-            )
-            # Merge module members if the namespace already exists
-            if top_namespace in modules_by_namespace:
-                existing_module_members = modules_by_namespace[top_namespace]
-                _module_members = _merge_module_members(
-                    [existing_module_members, module_members]
-                )
-            else:
-                _module_members = module_members
-
-            modules_by_namespace[top_namespace] = _module_members
-
-        except ImportError as e:
-            print(f"Error: Unable to import module '{namespace}' with error: {e}")
-
-    return modules_by_namespace
-
-
-def _construct_doc(pkg: str, members_by_namespace: Dict[str, ModuleMembers]) -> str:
-    """Construct the contents of the reference.rst file for the given package.
-
-    Args:
-        pkg: The package name
-        members_by_namespace: The members of the package, dict organized by top level
-                              module contains a list of classes and functions
-                              inside of the top level namespace.
-
-    Returns:
-        The contents of the reference.rst file.
-    """
-    full_doc = f"""\
-=======================
-``{pkg}`` API Reference
-=======================
+=============
+API Reference
+=============
 
 """
-    namespaces = sorted(members_by_namespace)
-
-    for module in namespaces:
-        _members = members_by_namespace[module]
-        classes = _members["classes_"]
+    for module, _members in sorted(members.items(), key=lambda kv: kv[0]):
+        classes = _members["classes"]
         functions = _members["functions"]
         if not (classes or functions):
             continue
-        section = f":mod:`{pkg}.{module}`"
-        underline = "=" * (len(section) + 1)
+        section = f":mod:`langchain.{module}`"
         full_doc += f"""\
 {section}
-{underline}
+{'=' * (len(section) + 1)}
 
-.. automodule:: {pkg}.{module}
+.. automodule:: langchain.{module}
     :no-members:
     :no-inherited-members:
 
 """
 
         if classes:
+            cstring = "\n    ".join(sorted(classes))
             full_doc += f"""\
 Classes
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
+    :template: class.rst
+
+    {cstring}
+
 """
-
-            for class_ in classes:
-                if not class_["is_public"]:
-                    continue
-
-                if class_["kind"] == "TypedDict":
-                    template = "typeddict.rst"
-                elif class_["kind"] == "enum":
-                    template = "enum.rst"
-                elif class_["kind"] == "Pydantic":
-                    template = "pydantic.rst"
-                else:
-                    template = "class.rst"
-
-                full_doc += f"""\
-    :template: {template}
-    
-    {class_["qualified_name"]}
-    
-"""
-
         if functions:
-            _functions = [f["qualified_name"] for f in functions if f["is_public"]]
-            fstring = "\n    ".join(sorted(_functions))
+            fstring = "\n    ".join(sorted(functions))
             full_doc += f"""\
 Functions
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
@@ -267,17 +83,10 @@ Functions
 
 
 def main() -> None:
-    """Generate the reference.rst file for each package."""
-    lc_members = _load_package_modules(PKG_DIR)
-    lc_doc = ".. _api_reference:\n\n" + _construct_doc("langchain", lc_members)
+    members = load_members()
+    full_doc = construct_doc(members)
     with open(WRITE_FILE, "w") as f:
-        f.write(lc_doc)
-    exp_members = _load_package_modules(EXP_DIR)
-    exp_doc = ".. _experimental_api_reference:\n\n" + _construct_doc(
-        "langchain_experimental", exp_members
-    )
-    with open(EXP_WRITE_FILE, "w") as f:
-        f.write(exp_doc)
+        f.write(full_doc)
 
 
 if __name__ == "__main__":

docs/api_reference/requirements.txt

@@ -1,5 +1,4 @@
 -e libs/langchain
--e libs/experimental
 autodoc_pydantic==1.8.0
 myst_parser
 nbsphinx==0.8.9
@@ -11,4 +10,4 @@ sphinx-panels
 toml
 myst_nb
 sphinx_copybutton
-pydata-sphinx-theme==0.13.1
+pydata-sphinx-theme==0.13.1

docs/api_reference/templates/class.rst

@@ -5,6 +5,17 @@
 
 .. autoclass:: {{ objname }}
 
+   {% block methods %}
+   {% if methods %}
+   .. rubric:: {{ _('Methods') }}
+
+   .. autosummary::
+   {% for item in methods %}
+      ~{{ name }}.{{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
    {% block attributes %}
    {% if attributes %}
    .. rubric:: {{ _('Attributes') }}
@@ -16,21 +27,4 @@
    {% endif %}
    {% endblock %}
 
-   {% block methods %}
-   {% if methods %}
-   .. rubric:: {{ _('Methods') }}
-
-   .. autosummary::
-   {% for item in methods %}
-      ~{{ name }}.{{ item }}
-   {%- endfor %}
-
-   {% for item in methods %}
-   .. automethod:: {{ name }}.{{ item }}
-   {%- endfor %}
-
-   {% endif %}
-   {% endblock %}
-
-
 .. example_links:: {{ objname }}

docs/api_reference/templates/enum.rst

@@ -1,14 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autoclass:: {{ objname }}
-
-    {% block attributes %}
-    {% for item in attributes %}
-    .. autoattribute:: {{ item }}
-    {% endfor %}
-    {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/templates/pydantic.rst

@@ -1,22 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autopydantic_model:: {{ objname }}
-    :model-show-json: False
-    :model-show-config-summary: False
-    :model-show-validator-members: False
-    :model-show-field-summary: False
-    :field-signature-prefix: param
-    :members:
-    :undoc-members:
-    :inherited-members:
-    :member-order: groupwise
-    :show-inheritance: True
-    :special-members: __call__
-
-    {% block attributes %}
-    {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/templates/typeddict.rst

@@ -1,14 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autoclass:: {{ objname }}
-
-    {% block attributes %}
-   {% for item in attributes %}
-  .. autoattribute:: {{ item }}
-   {% endfor %}
-   {% endblock %}
-
-.. example_links:: {{ objname }}

docs/api_reference/themes/scikit-learn-modern/layout.html

@@ -19,7 +19,7 @@
   {% block htmltitle %}
   <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
   {% endblock %}
-  <link rel="canonical" href="https://api.python.langchain.com/en/latest/{{pagename}}.html" />
+  <link rel="canonical" href="http://scikit-learn.org/stable/{{pagename}}.html" />
 
   {% if favicon_url %}
   <link rel="shortcut icon" href="{{ favicon_url|e }}"/>

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -6,6 +6,17 @@
   {%- set top_container_cls = "sk-landing-container" %}
 {%- endif %}
 
+{% if theme_link_to_live_contributing_page|tobool %}
+{# Link to development page for live builds #}
+  {%- set development_link = "https://scikit-learn.org/dev/developers/index.html" %}
+{# Open on a new development page in new window/tab for live builds #}
+  {%- set development_attrs = 'target="_blank" rel="noopener noreferrer"' %}
+{%- else %}
+  {%- set development_link = pathto('developers/index') %}
+  {%- set development_attrs = '' %}
+{%- endif %}
+
+
 <nav id="navbar" class="{{ nav_bar_class }} navbar navbar-expand-md navbar-light bg-light py-0">
   <div class="container-fluid {{ top_container_cls }} px-0">
     {%- if logo_url %}
@@ -34,9 +45,6 @@
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
         </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('experimental_api_reference') }}">Experimental</a>
-        </li>
         <li class="nav-item">
           <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
         </li>

docs/docs_skeleton/docs/community.md

@@ -1,54 +0,0 @@
-# Community navigator
-
-Hi! Thanks for being here. We’re lucky to have a community of so many passionate developers building with LangChain–we have so much to teach and learn from each other. Community members contribute code, host meetups, write blog posts, amplify each other’s work, become each other's customers and collaborators, and so much more.
-
-Whether you’re new to LangChain, looking to go deeper, or just want to get more exposure to the world of building with LLMs, this page can point you in the right direction. 
-
-- **🦜 Contribute to LangChain**
-
-- **🌍 Meetups, Events, and Hackathons**
-
-- **📣 Help Us Amplify Your Work**
-
-- **💬 Stay in the loop**
-
-
-# 🦜 Contribute to LangChain
-
-LangChain is the product of over 5,000+ contributions by 1,500+ contributors, and there is ******still****** so much to do together. Here are some ways to get involved:
-
-- **[Open a pull request](https://github.com/langchain-ai/langchain/issues):** we’d appreciate all forms of contributions–new features, infrastructure improvements, better documentation, bug fixes, etc. If you have an improvement or an idea, we’d love to work on it with you.
-- **[Read our contributor guidelines:](https://github.com/langchain-ai/langchain/blob/bbd22b9b761389a5e40fc45b0570e1830aabb707/.github/CONTRIBUTING.md)** We ask contributors to follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow, run a few local checks for formatting, linting, and testing before submitting, and follow certain documentation and testing conventions.
-    - **First time contributor?** [Try one of these PRs with the “good first issue” tag](https://github.com/langchain-ai/langchain/contribute).
-- **Become an expert:** our experts help the community by answering product questions in Discord. If that’s a role you’d like to play, we’d be so grateful! (And we have some special experts-only goodies/perks we can tell you more about). Send us an email to introduce yourself at hello@langchain.dev and we’ll take it from there!
-- **Integrate with LangChain:** if your product integrates with LangChain–or aspires to–we want to help make sure the experience is as smooth as possible for you and end users. Send us an email at hello@langchain.dev and tell us what you’re working on.
-    - **Become an Integration Maintainer:** Partner with our team to ensure your integration stays up-to-date and talk directly with users (and answer their inquiries) in our Discord. Introduce yourself at hello@langchain.dev if you’d like to explore this role.
-
-
-# 🌍 Meetups, Events, and Hackathons
-
-One of our favorite things about working in AI is how much enthusiasm there is for building together. We want to help make that as easy and impactful for you as possible! 
-- **Find a meetup, hackathon, or webinar:** you can find the one for you on our [global events calendar](https://mirror-feeling-d80.notion.site/0bc81da76a184297b86ca8fc782ee9a3?v=0d80342540df465396546976a50cfb3f).  
-    - **Submit an event to our calendar:** email us at events@langchain.dev with a link to your event page! We can also help you spread the word with our local communities.
-- **Host a meetup:** If you want to bring a group of builders together, we want to help! We can publicize your event on our event calendar/Twitter, share with our local communities in Discord, send swag, or potentially hook you up with a sponsor. Email us at events@langchain.dev to tell us about your event!
-- **Become a meetup sponsor:** we often hear from groups of builders that want to get together, but are blocked or limited on some dimension (space to host, budget for snacks, prizes to distribute, etc.). If you’d like to help, send us an email to events@langchain.dev we can share more about how it works!
-- **Speak at an event:** meetup hosts are always looking for great speakers, presenters, and panelists. If you’d like to do that at an event, send us an email to hello@langchain.dev with more information about yourself, what you want to talk about, and what city you’re based in and we’ll try to match you with an upcoming event!
-- **Tell us about your LLM community:** If you host or participate in a community that would welcome support from LangChain and/or our team, send us an email at hello@langchain.dev and let us know how we can help.
-
-# 📣 Help Us Amplify Your Work
-
-If you’re working on something you’re proud of, and think the LangChain community would benefit from knowing about it, we want to help you show it off.
-
-- **Post about your work and mention us:** we love hanging out on Twitter to see what people in the space are talking about and working on. If you tag [@langchainai](https://twitter.com/LangChainAI), we’ll almost certainly see it and can show you some love.
-- **Publish something on our blog:** if you’re writing about your experience building with LangChain, we’d love to post (or crosspost) it on our blog! E-mail hello@langchain.dev with a draft of your post! Or even an idea for something you want to write about.
-- **Get your product onto our [integrations hub](https://integrations.langchain.com/):** Many developers take advantage of our seamless integrations with other products, and come to our integrations hub to find out who those are. If you want to get your product up there, tell us about it (and how it works with LangChain) at hello@langchain.dev.
-
-# ☀️ Stay in the loop
-
-Here’s where our team hangs out, talks shop, spotlights cool work, and shares what we’re up to. We’d love to see you there too.
-
-- **[Twitter](https://twitter.com/LangChainAI):** we post about what we’re working on and what cool things we’re seeing in the space. If you tag @langchainai in your post, we’ll almost certainly see it, and can snow you some love!
-- **[Discord](https://discord.gg/6adMQxSpJS):** connect with with >30k developers who are building with LangChain
-- **[GitHub](https://github.com/langchain-ai/langchain):** open pull requests, contribute to a discussion, and/or contribute
-- **[Subscribe to our bi-weekly Release Notes](https://6w1pwbss0py.typeform.com/to/KjZB1auB):** a twice/month email roundup of the coolest things going on in our orbit
-- **Slack:** if you’re building an application in production at your company, we’d love to get into a Slack channel together. Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) and we’ll get in touch about setting one up.

docs/docs_skeleton/docs/guides/evaluation/index.mdx

@@ -8,9 +8,9 @@ import DocCardList from "@theme/DocCardList";
 
 Building applications with language models involves many moving parts. One of the most critical components is ensuring that the outcomes produced by your models are reliable and useful across a broad array of inputs, and that they work well with your application's other software components. Ensuring reliability usually boils down to some combination of application design, testing & evaluation, and runtime checks. 
 
-The guides in this section review the APIs and functionality LangChain provides to help you better evaluate your applications. Evaluation and testing are both critical when thinking about deploying LLM applications, since production environments require repeatable and useful outcomes.
+The guides in this section review the APIs and functionality LangChain provides to help yous better evaluate your applications. Evaluation and testing are both critical when thinking about deploying LLM applications, since production environments require repeatable and useful outcomes.
 
-LangChain offers various types of evaluators to help you measure performance and integrity on diverse data, and we hope to encourage the community to create and share other useful evaluators so everyone can improve. These docs will introduce the evaluator types, how to use them, and provide some examples of their use in real-world scenarios.
+LangChain offers various types of evaluators to help you measure performance and integrity on diverse data, and we hope to encourage the the community to create and share other useful evaluators so everyone can improve. These docs will introduce the evaluator types, how to use them, and provide some examples of their use in real-world scenarios.
 
 Each evaluator type in LangChain comes with ready-to-use implementations and an extensible API that allows for customization according to your unique requirements. Here are some of the types of evaluators we offer:
 

docs/docs_skeleton/docs/guides/expression_language/index.mdx

@@ -1,9 +0,0 @@
-# LangChain Expression Language
-
-import DocCardList from "@theme/DocCardList";
-
-LangChain Expression Language is a declarative way to easily compose chains together.
-Any chain constructed this way will automatically have full sync, async, and streaming support.
-See guides below for how to interact with chains constructed this way as well as cookbook examples.
-
-<DocCardList />

docs/docs_skeleton/docs/guides/langsmith/index.md

@@ -5,8 +5,8 @@ import DocCardList from "@theme/DocCardList";
 LangSmith helps you trace and evaluate your language model applications and intelligent agents to help you
 move from prototype to production.
 
-Check out the [interactive walkthrough](/docs/guides/langsmith/walkthrough) below to get started.
+Check out the [interactive walkthrough](walkthrough) below to get started.
 
 For more information, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/)
 
-<DocCardList />
+<DocCardList />

docs/docs_skeleton/docs/guides/safety/index.mdx

@@ -1,6 +0,0 @@
-# Preventing harmful outputs
-
-One of the key concerns with using LLMs is that they may generate harmful or unethical text. This is an area of active research in the field. Here we present some built-in chains inspired by this research, which are intended to make the outputs of LLMs safer.
-
-- [Moderation chain](/docs/use_cases/safety/moderation): Explicitly check if any output text is harmful and flag it.
-- [Constitutional chain](/docs/use_cases/safety/constitutional_chain): Prompt the model with a set of principles which should guide it's behavior.

docs/docs_skeleton/docs/modules/agents/agent_types/index.mdx

@@ -12,7 +12,7 @@ Here are the agents available in LangChain.
 
 ### [Zero-shot ReAct](/docs/modules/agents/agent_types/react.html)
 
-This agent uses the [ReAct](https://arxiv.org/pdf/2210.03629) framework to determine which tool to use
+This agent uses the [ReAct](https://arxiv.org/pdf/2205.00445.pdf) framework to determine which tool to use
 based solely on the tool's description. Any number of tools can be provided.
 This agent requires that a description is provided for each tool.
 
@@ -28,7 +28,7 @@ navigating around a browser.
 ### [OpenAI Functions](/docs/modules/agents/agent_types/openai_functions_agent.html)
 
 Certain OpenAI models (like gpt-3.5-turbo-0613 and gpt-4-0613) have been explicitly fine-tuned to detect when a
-function should be called and respond with the inputs that should be passed to the function.
+function should to be called and respond with the inputs that should be passed to the function.
 The OpenAI Functions Agent is designed to work with these models.
 
 ### [Conversational](/docs/modules/agents/agent_types/chat_conversation_agent.html)

docs/docs_skeleton/docs/modules/agents/agent_types/openai_functions_agent.mdx

@@ -1,6 +1,6 @@
 # OpenAI functions
 
-Certain OpenAI models (like gpt-3.5-turbo-0613 and gpt-4-0613) have been fine-tuned to detect when a function should be called and respond with the inputs that should be passed to the function.
+Certain OpenAI models (like gpt-3.5-turbo-0613 and gpt-4-0613) have been fine-tuned to detect when a function should to be called and respond with the inputs that should be passed to the function.
 In an API call, you can describe functions and have the model intelligently choose to output a JSON object containing arguments to call those functions.
 The goal of the OpenAI Function APIs is to more reliably return valid and useful function calls than a generic text completion or chat API.
 

docs/docs_skeleton/docs/modules/chains/additional/index.mdx

@@ -0,0 +1,8 @@
+---
+sidebar_position: 4
+---
+# Additional
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/chains/additional/multi_prompt_router.mdx

@@ -0,0 +1,7 @@
+# Dynamically selecting from multiple prompts
+
+This notebook demonstrates how to use the `RouterChain` paradigm to create a chain that dynamically selects the prompt to use for a given input. Specifically we show how to use the `MultiPromptChain` to create a question-answering chain that selects the prompt which is most relevant for a given question, and then answers the question using that prompt.
+
+import Example from "@snippets/modules/chains/additional/multi_prompt_router.mdx"
+
+<Example/>

docs/docs_skeleton/docs/modules/chains/additional/multi_retrieval_qa_router.mdx

@@ -1,4 +1,4 @@
-# Dynamically select from multiple retrievers
+# Dynamically selecting from multiple retrievers
 
 This notebook demonstrates how to use the `RouterChain` paradigm to create a chain that dynamically selects which Retrieval system to use. Specifically we show how to use the `MultiRetrievalQAChain` to create a question-answering chain that selects the retrieval QA chain which is most relevant for a given question, and then answers the question using it.
 

docs/docs_skeleton/docs/modules/chains/additional/question_answering.mdx

@@ -1,4 +1,4 @@
-# QA over in-memory documents
+# Document QA
 
 Here we walk through how to use LangChain for question answering over a list of documents. Under the hood we'll be using our [Document chains](/docs/modules/chains/document/).
 

docs/docs_skeleton/docs/modules/chains/foundational/sequential_chains.mdx

@@ -1,6 +1,6 @@
 # Sequential
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 The next step after calling a language model is make a series of calls to a language model. This is particularly useful when you want to take the output from one call and use it as the input to another.
 

docs/docs_skeleton/docs/modules/chains/popular/api.mdx

@@ -0,0 +1,9 @@
+---
+sidebar_position: 0
+---
+# API chains
+APIChain enables using LLMs to interact with APIs to retrieve relevant information. Construct the chain by providing a question relevant to the provided API documentation.
+
+import Example from "@snippets/modules/chains/popular/api.mdx"
+
+<Example/>

docs/docs_skeleton/docs/modules/chains/popular/chat_vector_db.mdx

@@ -2,7 +2,7 @@
 sidebar_position: 2
 ---
 
-# Store and reference chat history
+# Conversational Retrieval QA
 The ConversationalRetrievalQA chain builds on RetrievalQAChain to provide a chat history component.
 
 It first combines the chat history (either explicitly passed in or retrieved from the provided memory) and the question into a standalone question, then looks up relevant documents from the retriever, and finally passes those documents and the question to a question answering chain to return a response.

docs/docs_skeleton/docs/modules/chains/popular/index.mdx

@@ -0,0 +1,8 @@
+---
+sidebar_position: 3
+---
+# Popular
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/chains/popular/sqlite.mdx

@@ -1,7 +1,7 @@
-# SQL Database Chain
+# SQL
 
 This example demonstrates the use of the `SQLDatabaseChain` for answering questions over a SQL database.
 
 import Example from "@snippets/modules/chains/popular/sqlite.mdx"
 
-<Example/>
+<Example/>

docs/docs_skeleton/docs/modules/chains/popular/summarize.mdx

@@ -0,0 +1,8 @@
+# Summarization
+
+A summarization chain can be used to summarize multiple documents. One way is to input multiple smaller documents, after they have been divided into chunks, and operate over them with a MapReduceDocumentsChain. You can also choose instead for the chain that does summarization to be a StuffDocumentsChain, or a RefineDocumentsChain.
+
+import Example from "@snippets/modules/chains/popular/summarize.mdx"
+
+<Example/>
+

docs/docs_skeleton/docs/modules/chains/popular/vector_db_qa.mdx

@@ -1,7 +1,7 @@
 ---
 sidebar_position: 1
 ---
-# QA using a Retriever
+# Retrieval QA
 
 This example showcases question answering over an index.
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/integrations/_category_.yml

@@ -0,0 +1 @@
+label: 'Integrations'

docs/docs_skeleton/docs/modules/index.mdx

@@ -18,3 +18,5 @@ Let chains choose which tools to use given high-level directives
 Persist application state between runs of a chain
 #### [Callbacks](/docs/modules/callbacks/)
 Log and stream intermediate steps of any chain
+#### [Evaluation](/docs/modules/evaluation/)
+Evaluate the performance of a chain.

docs/docs_skeleton/docs/modules/memory/buffer.mdx

@@ -4,6 +4,6 @@ This notebook shows how to use `ConversationBufferMemory`. This memory allows fo
 
 We can first extract it as a string.
 
-import Example from "@snippets/modules/memory/types/buffer.mdx"
+import Example from "@snippets/modules/memory/how_to/buffer.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/buffer_window.mdx

@@ -4,6 +4,6 @@
 
 Let's first explore the basic functionality of this type of memory.
 
-import Example from "@snippets/modules/memory/types/buffer_window.mdx"
+import Example from "@snippets/modules/memory/how_to/buffer_window.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/chat_messages/index.mdx

@@ -1,17 +0,0 @@
----
-sidebar_position: 1
----
-# Chat Messages
-
-:::info
-Head to [Integrations](/docs/integrations/memory/) for documentation on built-in memory integrations with 3rd-party databases and tools.
-:::
-
-One of the core utility classes underpinning most (if not all) memory modules is the `ChatMessageHistory` class.
-This is a super lightweight wrapper which exposes convenience methods for saving Human messages, AI messages, and then fetching them all.
-
-You may want to use this class directly if you are managing memory outside of a chain.
-
-import GetStarted from "@snippets/modules/memory/chat_messages/get_started.mdx"
-
-<GetStarted/>

docs/docs_skeleton/docs/modules/memory/entity_summary_memory.mdx

@@ -4,6 +4,6 @@ Entity Memory remembers given facts about specific entities in a conversation. I
 
 Let's first walk through using this functionality.
 
-import Example from "@snippets/modules/memory/types/entity_summary_memory.mdx"
+import Example from "@snippets/modules/memory/how_to/entity_summary_memory.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/index.mdx

@@ -1,62 +1,34 @@
 ---
 sidebar_position: 3
 ---
+
 # Memory
 
-Most LLM applications have a conversational interface. An essential component of a conversation is being able to refer to information introduced earlier in the conversation.
-At bare minimum, a conversational system should be able to access some window of past messages directly.
-A more complex system will need to have a world model that it is constantly updating, which allows it to do things like maintain information about entities and their relationships.
+🚧 _Docs under construction_ 🚧
 
-We call this ability to store information about past interactions "memory".
-LangChain provides a lot of utilities for adding memory to a system.
-These utilities can be used by themselves or incorporated seamlessly into a chain.
+:::info
+Head to [Integrations](/docs/integrations/memory/) for documentation on built-in memory integrations with 3rd-party tools.
+:::
 
-A memory system needs to support two basic actions: reading and writing.
-Recall that every chain defines some core execution logic that expects certain inputs.
-Some of these inputs come directly from the user, but some of these inputs can come from memory.
-A chain will interact with its memory system twice in a given run.
-1. AFTER receiving the initial user inputs but BEFORE executing the core logic, a chain will READ from its memory system and augment the user inputs.
-2. AFTER executing the core logic but BEFORE returning the answer, a chain will WRITE the inputs and outputs of the current run to memory, so that they can be referred to in future runs.
+By default, Chains and Agents are stateless,
+meaning that they treat each incoming query independently (like the underlying LLMs and chat models themselves).
+In some applications, like chatbots, it is essential
+to remember previous interactions, both in the short and long-term.
+The **Memory** class does exactly that.
 
-![memory-diagram](/img/memory_diagram.png)
-
-
-## Building memory into a system
-The two core design decisions in any memory system are:
-- How state is stored
-- How state is queried
-
-### Storing: List of chat messages
-Underlying any memory is a history of all chat interactions.
-Even if these are not all used directly, they need to be stored in some form.
-One of the key parts of the LangChain memory module is a series of integrations for storing these chat messages,
-from in-memory lists to persistent databases.
-
-- [Chat message storage](/docs/modules/memory/chat_messages/): How to work with Chat Messages, and the various integrations offered
-
-### Querying: Data structures and algorithms on top of chat messages
-Keeping a list of chat messages is fairly straight-forward.
-What is less straight-forward are the data structures and algorithms built on top of chat messages that serve a view of those messages that is most useful.
-
-A very simply memory system might just return the most recent messages each run. A slightly more complex memory system might return a succinct summary of the past K messages.
-An even more sophisticated system might extract entities from stored messages and only return information about entities referenced in the current run.
-
-Each application can have different requirements for how memory is queried. The memory module should make it easy to both get started with simple memory systems and write your own custom systems if needed.
-
-- [Memory types](/docs/modules/memory/types/): The various data structures and algorithms that make up the memory types LangChain supports
+LangChain provides memory components in two forms.
+First, LangChain provides helper utilities for managing and manipulating previous chat messages.
+These are designed to be modular and useful regardless of how they are used.
+Secondly, LangChain provides easy ways to incorporate these utilities into chains.
 
 ## Get started
 
-Let's take a look at what Memory actually looks like in LangChain.
-Here we'll cover the basics of interacting with an arbitrary memory class.
+Memory involves keeping a concept of state around throughout a user's interactions with an language model. A user's interactions with a language model are captured in the concept of ChatMessages, so this boils down to ingesting, capturing, transforming and extracting knowledge from a sequence of chat messages. There are many different ways to do this, each of which exists as its own memory type.
+
+In general, for each type of memory there are two ways to understanding using memory. These are the standalone functions which extract information from a sequence of messages, and then there is the way you can use this type of memory in a chain.
+
+Memory can return multiple pieces of information (for example, the most recent N messages and a summary of all previous messages). The returned information can either be a string or a list of messages.
 
 import GetStarted from "@snippets/modules/memory/get_started.mdx"
 
 <GetStarted/>
-
-## Next steps
-
-And that's it for getting started!
-Please see the other sections for walkthroughs of more advanced topics,
-like custom memory, multiple memories, and more.
-

docs/docs_skeleton/docs/modules/memory/summary.mdx

@@ -4,6 +4,6 @@ Conversation summary memory summarizes the conversation as it happens and stores
 
 Let's first explore the basic functionality of this type of memory.
 
-import Example from "@snippets/modules/memory/types/summary.mdx"
+import Example from "@snippets/modules/memory/how_to/summary.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/types/index.mdx

@@ -1,8 +0,0 @@
----
-sidebar_position: 2
----
-# Memory Types
-
-There are many different types of memory.
-Each have their own parameters, their own return types, and are useful in different scenarios.
-Please see their individual page for more detail on each one.

docs/docs_skeleton/docs/modules/memory/vectorstore_retriever_memory.mdx

@@ -6,6 +6,6 @@ This differs from most of the other Memory classes in that it doesn't explicitly
 
 In this case, the "docs" are previous conversation snippets. This can be useful to refer to relevant pieces of information that the AI was told earlier in the conversation.
 
-import Example from "@snippets/modules/memory/types/vectorstore_retriever_memory.mdx"
+import Example from "@snippets/modules/memory/how_to/vectorstore_retriever_memory.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/model_io/prompts/index.mdx

@@ -3,12 +3,10 @@ sidebar_position: 0
 ---
 # Prompts
 
-A prompt for a language model is a set of instructions or input provided by a user to
-guide the model's response, helping it understand the context and generate relevant
-and coherent language-based output, such as answering questions, completing sentences,
-or engaging in a conversation.
+The new way of programming models is through prompts.
+A **prompt** refers to the input to the model.
+This input is often constructed from multiple components.
+LangChain provides several classes and functions to make constructing and working with prompts easy.
 
-LangChain provides several classes and functions to help construct and work with prompts.
-
-- [Prompt templates](/docs/modules/model_io/prompts/prompt_templates/): Parametrized model inputs
+- [Prompt templates](/docs/modules/model_io/prompts/prompt_templates/): Parametrize model inputs
 - [Example selectors](/docs/modules/model_io/prompts/example_selectors/): Dynamically select examples to include in prompts

docs/docs_skeleton/docs/modules/model_io/prompts/prompt_templates/index.mdx

@@ -4,15 +4,18 @@ sidebar_position: 0
 
 # Prompt templates
 
-Prompt templates are pre-defined recipes for generating prompts for language models.
+Language models take text as input - that text is commonly referred to as a prompt.
+Typically this is not simply a hardcoded string but rather a combination of a template, some examples, and user input.
+LangChain provides several classes and functions to make constructing and working with prompts easy.
 
-A template may include instructions, few shot examples, and specific context and
-questions appropriate for a given task.
+## What is a prompt template?
 
-LangChain provides tooling to create and work with prompt templates.
+A prompt template refers to a reproducible way to generate a prompt. It contains a text string ("the template"), that can take in a set of parameters from the end user and generates a prompt.
 
-LangChain strives to create model agnostic templates to make it easy to reuse
-existing templates across different language models.
+A prompt template can contain:
+- instructions to the language model,
+- a set of few shot examples to help the language model generate a better response,
+- a question to the language model.
 
 import GetStarted from "@snippets/modules/model_io/prompts/prompt_templates/get_started.mdx"
 

docs/docs_skeleton/docs/use_cases/question_answering/how_to/_category_.yml

@@ -1 +0,0 @@
-label: 'How to'

docs/docs_skeleton/docs/use_cases/web_scraping/index.mdx

@@ -1,9 +0,0 @@
----
-sidebar_position: 3
----
-
-# Web Scraping
-
-Web scraping has historically been a challenging endeavor due to the ever-changing nature of website structures, making it tedious for developers to maintain their scraping scripts. Traditional methods often rely on specific HTML tags and patterns which, when altered, can disrupt data extraction processes.
-
-Enter the LLM-based method for parsing HTML: By leveraging the capabilities of LLMs, and especially OpenAI Functions in LangChain's extraction chain, developers can instruct the model to extract only the desired data in a specified format. This method not only streamlines the extraction process but also significantly reduces the time spent on manual debugging and script modifications. Its adaptability means that even if websites undergo significant design changes, the extraction remains consistent and robust. This level of resilience translates to reduced maintenance efforts, cost savings, and ensures a higher quality of extracted data. Compared to its predecessors, LLM-based approach wins out the web scraping domain by transforming a historically cumbersome task into a more automated and efficient process.

docs/docs_skeleton/docusaurus.config.js

@@ -128,10 +128,6 @@ const config = {
           hideable: true,
         },
       },
-      colorMode: {
-        disableSwitch: false,
-        respectPrefersColorScheme: true,
-      },
       prism: {
         theme: {
           ...baseLightCodeBlockTheme,

docs/docs_skeleton/generate_api_reference_links.py

@@ -5,7 +5,6 @@ import logging
 import os
 import re
 from pathlib import Path
-import argparse
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
@@ -15,12 +14,7 @@ _BASE_URL = "https://api.python.langchain.com/en/latest/"
 # Regular expression to match Python code blocks
 code_block_re = re.compile(r"^(```python\n)(.*?)(```\n)", re.DOTALL | re.MULTILINE)
 # Regular expression to match langchain import lines
-_IMPORT_RE = re.compile(
-    r"from\s+(langchain\.\w+(\.\w+)*?)\s+import\s+"
-    r"((?:\w+(?:,\s*)?)*"  # Match zero or more words separated by a comma+optional ws
-    r"(?:\s*\(.*?\))?)",  # Match optional parentheses block
-    re.DOTALL,  # Match newlines as well
-)
+_IMPORT_RE = re.compile(r"(from\s+(langchain\.\w+(\.\w+)*?)\s+import\s+)(\w+)")
 
 _CURRENT_PATH = Path(__file__).parent.absolute()
 # Directory where generated markdown files are stored
@@ -30,10 +24,6 @@ _JSON_PATH = _CURRENT_PATH.parent / "api_reference" / "guide_imports.json"
 
 def find_files(path):
     """Find all MDX files in the given path"""
-    # Check if is file first
-    if os.path.isfile(path):
-        yield path
-        return
     for root, _, files in os.walk(path):
         for file in files:
             if file.endswith(".mdx") or file.endswith(".md"):
@@ -47,33 +37,20 @@ def get_full_module_name(module_path, class_name):
     return inspect.getmodule(class_).__name__
 
 
-def get_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "--docs_dir",
-        type=str,
-        default=_DOCS_DIR,
-        help="Directory where generated markdown files are stored",
-    )
-    return parser.parse_args()
-
-
 def main():
     """Main function"""
-    args = get_args()
     global_imports = {}
 
-    for file in find_files(args.docs_dir):
+    for file in find_files(_DOCS_DIR):
         print(f"Adding links for imports in {file}")
+
+        # replace_imports now returns the import information rather than writing it to a file
         file_imports = replace_imports(file)
 
         if file_imports:
             # Use relative file path as key
-            relative_path = (
-                os.path.relpath(file, _DOCS_DIR).replace(".mdx", "").replace(".md", "")
-            )
-
-            doc_url = f"https://python.langchain.com/docs/{relative_path}"
+            relative_path = os.path.relpath(file, _DOCS_DIR)
+            doc_url = f"https://python.langchain.com/docs/{relative_path.replace('.mdx', '').replace('.md', '')}"
             for import_info in file_imports:
                 doc_title = import_info["title"]
                 class_name = import_info["imported"]
@@ -82,7 +59,6 @@ def main():
                 global_imports[class_name][doc_title] = doc_url
 
     # Write the global imports information to a JSON file
-    _JSON_PATH.parent.mkdir(parents=True, exist_ok=True)
     with _JSON_PATH.open("w") as f:
         json.dump(global_imports, f)
 
@@ -100,8 +76,7 @@ def _get_doc_title(data: str, file_name: str) -> str:
 
 
 def replace_imports(file):
-    """Replace imports in each Python code block with links to their
-    documentation and append the import info in a comment"""
+    """Replace imports in each Python code block with links to their documentation and append the import info in a comment"""
     all_imports = []
     with open(file, "r") as f:
         data = f.read()
@@ -121,45 +96,37 @@ def replace_imports(file):
         # Process imports in the code block
         imports = []
         for import_match in _IMPORT_RE.finditer(code):
-            module = import_match.group(1)
-            imports_str = (
-                import_match.group(3).replace("(\n", "").replace("\n)", "")
-            )  # Handle newlines within parentheses
-            # remove any newline and spaces, then split by comma
-            imported_classes = [
-                imp.strip()
-                for imp in re.split(r",\s*", imports_str.replace("\n", ""))
-                if imp.strip()
-            ]
-            for class_name in imported_classes:
-                try:
-                    module_path = get_full_module_name(module, class_name)
-                except AttributeError as e:
-                    logger.warning(f"Could not find module for {class_name}, {e}")
-                    continue
-                except ImportError as e:
-                    logger.warning(f"Failed to load for class {class_name}, {e}")
-                    continue
+            class_name = import_match.group(4)
+            try:
+                module_path = get_full_module_name(import_match.group(2), class_name)
+            except AttributeError as e:
+                logger.warning(f"Could not find module for {class_name}, {e}")
+                continue
+            except ImportError as e:
+                # Some CentOS OpenSSL issues can cause this to fail
+                logger.warning(f"Failed to load for class {class_name}, {e}")
+                continue
 
-                url = (
-                    _BASE_URL
-                    + module_path.split(".")[1]
-                    + "/"
-                    + module_path
-                    + "."
-                    + class_name
-                    + ".html"
-                )
+            url = (
+                _BASE_URL
+                + "/"
+                + module_path.split(".")[1]
+                + "/"
+                + module_path
+                + "."
+                + class_name
+                + ".html"
+            )
 
-                # Add the import information to our list
-                imports.append(
-                    {
-                        "imported": class_name,
-                        "source": module,
-                        "docs": url,
-                        "title": _DOC_TITLE,
-                    }
-                )
+            # Add the import information to our list
+            imports.append(
+                {
+                    "imported": class_name,
+                    "source": import_match.group(2),
+                    "docs": url,
+                    "title": _DOC_TITLE,
+                }
+            )
 
         if imports:
             all_imports.extend(imports)

docs/docs_skeleton/package-lock.json

@@ -12,7 +12,7 @@
         "@docusaurus/preset-classic": "2.4.0",
         "@docusaurus/remark-plugin-npm2yarn": "^2.4.0",
         "@mdx-js/react": "^1.6.22",
-        "@mendable/search": "^0.0.150",
+        "@mendable/search": "^0.0.125",
         "clsx": "^1.2.1",
         "json-loader": "^0.5.7",
         "process": "^0.11.10",
@@ -3212,11 +3212,10 @@
       }
     },
     "node_modules/@mendable/search": {
-      "version": "0.0.150",
-      "resolved": "https://registry.npmjs.org/@mendable/search/-/search-0.0.150.tgz",
-      "integrity": "sha512-Eb5SeAWlMxzEim/8eJ/Ysn01Pyh39xlPBzRBw/5OyOBhti0HVLXk4wd1Fq2TKgJC2ppQIvhEKO98PUcj9dNDFw==",
+      "version": "0.0.125",
+      "resolved": "https://registry.npmjs.org/@mendable/search/-/search-0.0.125.tgz",
+      "integrity": "sha512-Mb1J3zDhOyBZV9cXqJocSOBNYGpe8+LQDqd9n9laPWxosSJcSTUewqtlIbMerrYsScBsxskoSiWgRsc7xF5z0Q==",
       "dependencies": {
-        "html-react-parser": "^4.2.0",
         "posthog-js": "^1.45.1"
       },
       "peerDependencies": {
@@ -8333,33 +8332,6 @@
         "safe-buffer": "~5.1.0"
       }
     },
-    "node_modules/html-dom-parser": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/html-dom-parser/-/html-dom-parser-4.0.0.tgz",
-      "integrity": "sha512-TUa3wIwi80f5NF8CVWzkopBVqVAtlawUzJoLwVLHns0XSJGynss4jiY0mTWpiDOsuyw+afP+ujjMgRh9CoZcXw==",
-      "dependencies": {
-        "domhandler": "5.0.3",
-        "htmlparser2": "9.0.0"
-      }
-    },
-    "node_modules/html-dom-parser/node_modules/htmlparser2": {
-      "version": "9.0.0",
-      "resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-9.0.0.tgz",
-      "integrity": "sha512-uxbSI98wmFT/G4P2zXx4OVx04qWUmyFPrD2/CNepa2Zo3GPNaCaaxElDgwUrwYWkK1nr9fft0Ya8dws8coDLLQ==",
-      "funding": [
-        "https://github.com/fb55/htmlparser2?sponsor=1",
-        {
-          "type": "github",
-          "url": "https://github.com/sponsors/fb55"
-        }
-      ],
-      "dependencies": {
-        "domelementtype": "^2.3.0",
-        "domhandler": "^5.0.3",
-        "domutils": "^3.1.0",
-        "entities": "^4.5.0"
-      }
-    },
     "node_modules/html-entities": {
       "version": "2.4.0",
       "resolved": "https://registry.npmjs.org/html-entities/-/html-entities-2.4.0.tgz",
@@ -8403,20 +8375,6 @@
         "node": ">= 12"
       }
     },
-    "node_modules/html-react-parser": {
-      "version": "4.2.0",
-      "resolved": "https://registry.npmjs.org/html-react-parser/-/html-react-parser-4.2.0.tgz",
-      "integrity": "sha512-gzU55AS+FI6qD7XaKe5BLuLFM2Xw0/LodfMWZlxV9uOHe7LCD5Lukx/EgYuBI3c0kLu0XlgFXnSzO0qUUn3Vrg==",
-      "dependencies": {
-        "domhandler": "5.0.3",
-        "html-dom-parser": "4.0.0",
-        "react-property": "2.0.0",
-        "style-to-js": "1.1.3"
-      },
-      "peerDependencies": {
-        "react": "0.14 || 15 || 16 || 17 || 18"
-      }
-    },
     "node_modules/html-tags": {
       "version": "3.3.1",
       "resolved": "https://registry.npmjs.org/html-tags/-/html-tags-3.3.1.tgz",
@@ -11804,11 +11762,6 @@
         "webpack": ">=4.41.1 || 5.x"
       }
     },
-    "node_modules/react-property": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/react-property/-/react-property-2.0.0.tgz",
-      "integrity": "sha512-kzmNjIgU32mO4mmH5+iUyrqlpFQhF8K2k7eZ4fdLSOPFrD1XgEuSBv9LDEgxRXTMBqMd8ppT0x6TIzqE5pdGdw=="
-    },
     "node_modules/react-router": {
       "version": "5.3.4",
       "resolved": "https://registry.npmjs.org/react-router/-/react-router-5.3.4.tgz",
@@ -13174,22 +13127,6 @@
         "url": "https://github.com/sponsors/sindresorhus"
       }
     },
-    "node_modules/style-to-js": {
-      "version": "1.1.3",
-      "resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.3.tgz",
-      "integrity": "sha512-zKI5gN/zb7LS/Vm0eUwjmjrXWw8IMtyA8aPBJZdYiQTXj4+wQ3IucOLIOnF7zCHxvW8UhIGh/uZh/t9zEHXNTQ==",
-      "dependencies": {
-        "style-to-object": "0.4.1"
-      }
-    },
-    "node_modules/style-to-js/node_modules/style-to-object": {
-      "version": "0.4.1",
-      "resolved": "https://registry.npmjs.org/style-to-object/-/style-to-object-0.4.1.tgz",
-      "integrity": "sha512-HFpbb5gr2ypci7Qw+IOhnP2zOU7e77b+rzM+wTzXzfi1PrtBCX0E7Pk4wL4iTLnhzZ+JgEGAhX81ebTg/aYjQw==",
-      "dependencies": {
-        "inline-style-parser": "0.1.1"
-      }
-    },
     "node_modules/style-to-object": {
       "version": "0.3.0",
       "resolved": "https://registry.npmjs.org/style-to-object/-/style-to-object-0.3.0.tgz",

docs/docs_skeleton/package.json

@@ -23,7 +23,7 @@
     "@docusaurus/preset-classic": "2.4.0",
     "@docusaurus/remark-plugin-npm2yarn": "^2.4.0",
     "@mdx-js/react": "^1.6.22",
-    "@mendable/search": "^0.0.150",
+    "@mendable/search": "^0.0.125",
     "clsx": "^1.2.1",
     "json-loader": "^0.5.7",
     "process": "^0.11.10",

docs/docs_skeleton/sidebars.js

@@ -75,7 +75,6 @@ module.exports = {
       slug: "additional_resources",
       },
     },
-    'community'
   ],
   integrations: [
     {

docs/docs_skeleton/vercel_build.sh

@@ -1,47 +1,26 @@
 #!/bin/bash
 
-version_compare() {
-    local v1=(${1//./ })
-    local v2=(${2//./ })
-    for i in {0..2}; do
-        if (( ${v1[i]} < ${v2[i]} )); then
-            return 1
-        fi
-    done
-    return 0
-}
-
-openssl_version=$(openssl version | awk '{print $2}')
-required_openssl_version="1.1.1"
-
-python_version=$(python3 --version 2>&1 | awk '{print $2}')
-required_python_version="3.10"
-
-echo "OpenSSL Version"
-echo $openssl_version
-echo "Python Version"
-echo $python_version
-# If openssl version is less than 1.1.1 AND python version is less than 3.10
-if ! version_compare $openssl_version $required_openssl_version && ! version_compare $python_version $required_python_version; then
 ### See: https://github.com/urllib3/urllib3/issues/2168
 # Requests lib breaks for old SSL versions,
 # which are defaults on Amazon Linux 2 (which Vercel uses for builds)
-    yum -y update
-    yum remove openssl-devel -y
-    yum install gcc bzip2-devel libffi-devel zlib-devel wget tar -y
-    yum install openssl11 -y
-    yum install openssl11-devel -y
-
-    wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz
-    tar xzf Python-3.11.4.tgz
-    cd Python-3.11.4
-    ./configure
-    make altinstall
-    echo "Python Version"
-    python3.11 --version
-    cd ..
-fi
+yum -y update
+yum remove openssl-devel -y
+yum install gcc bzip2-devel libffi-devel zlib-devel wget tar -y
+yum install openssl11 -y
+yum install openssl11-devel -y
+# Install python 3.11 to connect with openSSL 1.1.1
+wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz 
+tar xzf Python-3.11.4.tgz 
+cd Python-3.11.4 
+./configure 
+make altinstall
+# Check python version
+echo "Python Version"
+python3.11 --version
+cd ..
+###
 
+# Install nbdev and generate docs
 cd ..
 python3.11 -m venv .venv
 source .venv/bin/activate

docs/extras/additional_resources/tutorials.mdx

@@ -1,6 +1,5 @@
 # Tutorials
 
-Below are links to video tutorials and courses on LangChain. For written guides on common use cases for LangChain, check out the [use cases guides](/docs/use_cases).
 
 ⛓ icon marks a new addition [last update 2023-07-05]
 
@@ -42,8 +41,8 @@ Below are links to video tutorials and courses on LangChain. For written guides
 ### [LangChain 101](https://www.youtube.com/playlist?list=PLqZXAkvF1bPNQER9mLmDbntNfSpzdDIU5) by [Greg Kamradt (Data Indy)](https://www.youtube.com/@DataIndependent)
 - [What Is LangChain? - LangChain + `ChatGPT` Overview](https://youtu.be/_v_fgW2SkkQ)
 - [Quickstart Guide](https://youtu.be/kYRB-vJFy38)
-- [Beginner's Guide To 7 Essential Concepts](https://youtu.be/2xxziIWmaSA)
-- [Beginner's Guide To 9 Use Cases](https://youtu.be/vGP4pQdCocw)
+- [Beginner Guide To 7 Essential Concepts](https://youtu.be/2xxziIWmaSA)
+- [Beginner Guide To 9 Use Cases](https://youtu.be/vGP4pQdCocw)
 - [Agents Overview + Google Searches](https://youtu.be/Jq9Sf68ozk0)
 - [`OpenAI` + `Wolfram Alpha`](https://youtu.be/UijbzCIJ99g)
 - [Ask Questions On Your Custom (or Private) Files](https://youtu.be/EnT-ZTrcPrg)
@@ -73,7 +72,7 @@ Below are links to video tutorials and courses on LangChain. For written guides
 - [Conversations with Memory (explanation & code walkthrough)](https://youtu.be/X550Zbz_ROE)
 - [Chat with `Flan20B`](https://youtu.be/VW5LBavIfY4)
 - [Using `Hugging Face Models` locally (code walkthrough)](https://youtu.be/Kn7SX2Mx_Jk)
-- [`PAL`: Program-aided Language Models with LangChain code](https://youtu.be/dy7-LvDu-3s)
+- [`PAL` : Program-aided Language Models with LangChain code](https://youtu.be/dy7-LvDu-3s)
 - [Building a Summarization System with LangChain and `GPT-3` - Part 1](https://youtu.be/LNq_2s_H01Y)
 - [Building a Summarization System with LangChain and `GPT-3` - Part 2](https://youtu.be/d-yeHDLgKHw)
 - [Microsoft's `Visual ChatGPT` using LangChain](https://youtu.be/7YEiEyfPF5U)
@@ -85,7 +84,7 @@ Below are links to video tutorials and courses on LangChain. For written guides
 - [`BabyAGI`: Discover the Power of Task-Driven Autonomous Agents!](https://youtu.be/QBcDLSE2ERA)
 - [Improve your `BabyAGI` with LangChain](https://youtu.be/DRgPyOXZ-oE)
 - [Master `PDF` Chat with LangChain - Your essential guide to queries on documents](https://youtu.be/ZzgUqFtxgXI)
-- [Using LangChain with `DuckDuckGO`, `Wikipedia` & `PythonREPL` Tools](https://youtu.be/KerHlb8nuVc)
+- [Using LangChain with `DuckDuckGO` `Wikipedia` & `PythonREPL` Tools](https://youtu.be/KerHlb8nuVc)
 - [Building Custom Tools and Agents with LangChain (gpt-3.5-turbo)](https://youtu.be/biS8G8x8DdA)
 - [LangChain Retrieval QA Over Multiple Files with `ChromaDB`](https://youtu.be/3yPBVii7Ct0)
 - [LangChain Retrieval QA with Instructor Embeddings & `ChromaDB` for PDFs](https://youtu.be/cFCGUjc33aU)
@@ -107,7 +106,7 @@ Below are links to video tutorials and courses on LangChain. For written guides
 - [Working with MULTIPLE `PDF` Files in LangChain: `ChatGPT` for your Data](https://youtu.be/s5LhRdh5fu4)
 - [`ChatGPT` for YOUR OWN `PDF` files with LangChain](https://youtu.be/TLf90ipMzfE)
 - [Talk to YOUR DATA without OpenAI APIs: LangChain](https://youtu.be/wrD-fZvT6UI)
-- [LangChain: PDF Chat App (GUI) | ChatGPT for Your PDF FILES](https://youtu.be/RIWbalZ7sTo)
+- [Langchain: PDF Chat App (GUI) | ChatGPT for Your PDF FILES](https://youtu.be/RIWbalZ7sTo)
 - [LangFlow: Build Chatbots without Writing Code](https://youtu.be/KJ-ux3hre4s)
 - [LangChain: Giving Memory to LLMs](https://youtu.be/dxO6pzlgJiY)
 - [BEST OPEN Alternative to `OPENAI's EMBEDDINGs` for Retrieval QA: LangChain](https://youtu.be/ogEalPMUCSY)
@@ -122,4 +121,4 @@ Below are links to video tutorials and courses on LangChain. For written guides
 
 
 ---------------------
-⛓ icon marks a new addition [last update 2023-07-05]
+⛓ icon marks a new addition [last update 2023-07-05]

docs/extras/guides/adapters/_category_.yml

@@ -1 +0,0 @@
-label: 'Adapters'

docs/extras/guides/adapters/openai.ipynb

@@ -1,323 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "700a516b",
-   "metadata": {},
-   "source": [
-    "# OpenAI Adapter\n",
-    "\n",
-    "A lot of people get started with OpenAI but want to explore other models. LangChain's integrations with many model providers make this easy to do so. While LangChain has it's own message and model APIs, we've also made it as easy as possible to explore other models by exposing an adapter to adapt LangChain models to the OpenAI api.\n",
-    "\n",
-    "At the moment this only deals with output and does not return other information (token counts, stop reasons, etc)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "6017f26a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import openai\n",
-    "from langchain.adapters import openai as lc_openai"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b522ceda",
-   "metadata": {},
-   "source": [
-    "## ChatCompletion.create"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "1d22eb61",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "messages = [{\"role\": \"user\", \"content\": \"hi\"}]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d550d3ad",
-   "metadata": {},
-   "source": [
-    "Original OpenAI call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "e1d27dfa",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "result = openai.ChatCompletion.create(\n",
-    "    messages=messages, \n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "012d81ae",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "result[\"choices\"][0]['message'].to_dict_recursive()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "db5b5500",
-   "metadata": {},
-   "source": [
-    "LangChain OpenAI wrapper call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "87c2d515",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "lc_result = lc_openai.ChatCompletion.create(\n",
-    "    messages=messages, \n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "c67a5ac8",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}"
-      ]
-     },
-     "execution_count": 17,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "lc_result[\"choices\"][0]['message']"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "034ba845",
-   "metadata": {},
-   "source": [
-    "Swapping out model providers"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "7a2c011c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "lc_result = lc_openai.ChatCompletion.create(\n",
-    "    messages=messages, \n",
-    "    model=\"claude-2\", \n",
-    "    temperature=0, \n",
-    "    provider=\"ChatAnthropic\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "f7c94827",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'role': 'assistant', 'content': ' Hello!'}"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "lc_result[\"choices\"][0]['message']"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cb3f181d",
-   "metadata": {},
-   "source": [
-    "## ChatCompletion.stream"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f7b8cd18",
-   "metadata": {},
-   "source": [
-    "Original OpenAI call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 24,
-   "id": "fd8cb1ea",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'role': 'assistant', 'content': ''}\n",
-      "{'content': 'Hello'}\n",
-      "{'content': '!'}\n",
-      "{'content': ' How'}\n",
-      "{'content': ' can'}\n",
-      "{'content': ' I'}\n",
-      "{'content': ' assist'}\n",
-      "{'content': ' you'}\n",
-      "{'content': ' today'}\n",
-      "{'content': '?'}\n",
-      "{}\n"
-     ]
-    }
-   ],
-   "source": [
-    "for c in openai.ChatCompletion.create(\n",
-    "    messages = messages,\n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0,\n",
-    "    stream=True\n",
-    "):\n",
-    "    print(c[\"choices\"][0]['delta'].to_dict_recursive())"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0b2a076b",
-   "metadata": {},
-   "source": [
-    "LangChain OpenAI wrapper call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "9521218c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'role': 'assistant', 'content': ''}\n",
-      "{'content': 'Hello'}\n",
-      "{'content': '!'}\n",
-      "{'content': ' How'}\n",
-      "{'content': ' can'}\n",
-      "{'content': ' I'}\n",
-      "{'content': ' assist'}\n",
-      "{'content': ' you'}\n",
-      "{'content': ' today'}\n",
-      "{'content': '?'}\n",
-      "{}\n"
-     ]
-    }
-   ],
-   "source": [
-    "for c in lc_openai.ChatCompletion.create(\n",
-    "    messages = messages,\n",
-    "    model=\"gpt-3.5-turbo\", \n",
-    "    temperature=0,\n",
-    "    stream=True\n",
-    "):\n",
-    "    print(c[\"choices\"][0]['delta'])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0fc39750",
-   "metadata": {},
-   "source": [
-    "Swapping out model providers"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "68f0214e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'role': 'assistant', 'content': ' Hello'}\n",
-      "{'content': '!'}\n",
-      "{}\n"
-     ]
-    }
-   ],
-   "source": [
-    "for c in lc_openai.ChatCompletion.create(\n",
-    "    messages = messages,\n",
-    "    model=\"claude-2\", \n",
-    "    temperature=0,\n",
-    "    stream=True,\n",
-    "    provider=\"ChatAnthropic\",\n",
-    "):\n",
-    "    print(c[\"choices\"][0]['delta'])"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/guides/debugging.md

@@ -4,7 +4,7 @@ If you're building with LLMs, at some point something will break, and you'll nee
 
 Here's a few different tools and functionalities to aid in debugging.
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 ## Tracing