merge

This PR brings in a vectorstore interface for
[Marqo](https://www.marqo.ai/).

The Marqo vectorstore exposes some of Marqo's functionality in addition
the the VectorStore base class. The Marqo vectorstore also makes the
embedding parameter optional because inference for embeddings is an
inherent part of Marqo.

Docs, notebook examples and integration tests included.

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>

.devcontainer/docker-compose.yaml

@@ -2,7 +2,7 @@ version: '3'
 services:
   langchain:
     build:
-      dockerfile: libs/langchain/dev.Dockerfile
+      dockerfile: dev.Dockerfile
       context: ..
     volumes:
    # Update this to wherever you want VS Code to mount the folder of your project

.github/CONTRIBUTING.md

@@ -69,14 +69,6 @@ This project uses [Poetry](https://python-poetry.org/) as a dependency manager.
 3. Tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`)
 4. Continue with the following steps.
 
-There are two separate projects in this repository:
-- `langchain`: core langchain code, abstractions, and use cases
-- `langchain.experimental`: more experimental code
-
-Each of these has their OWN development environment. 
-In order to run any of the commands below, please move into their respective directories.
-For example, to contribute to `langchain` run `cd libs/langchain` before getting started with the below.
-
 To install requirements:
 
 ```bash
@@ -103,14 +95,6 @@ To run formatting for this project:
 make format
 ```
 
-Additionally, you can run the formatter only on the files that have been modified in your current branch as compared to the master branch using the format_diff command:
-
-```bash
-make format_diff
-```
-
-This is especially useful when you have made changes to a subset of the project and want to ensure your changes are properly formatted without affecting the rest of the codebase.
-
 ### Linting
 
 Linting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](http://mypy-lang.org/).
@@ -121,42 +105,8 @@ To run linting for this project:
 make lint
 ```
 
-In addition, you can run the linter only on the files that have been modified in your current branch as compared to the master branch using the lint_diff command:
-
-```bash
-make lint_diff
-```
-
-This can be very helpful when you've made changes to only certain parts of the project and want to ensure your changes meet the linting standards without having to check the entire codebase.
-
 We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
 
-### Spellcheck
-
-Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell).
-Note that `codespell` finds common typos, so could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
-
-To check spelling for this project:
-
-```bash
-make spell_check
-```
-
-To fix spelling in place:
-
-```bash
-make spell_fix
-```
-
-If codespell is incorrectly flagging a word, you can skip spellcheck for that word by adding it to the codespell config in the `pyproject.toml` file.
-
-```python
-[tool.codespell]
-...
-# Add here:
-ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure'
-```
-
 ### Coverage
 
 Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle.
@@ -256,43 +206,32 @@ When you run `poetry install`, the `langchain` package is installed as editable
 
 ## Documentation
 
-While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing.
-This covers how to get started contributing to documentation.
-
 ### Contribute Documentation
 
-The docs directory contains Documentation and API Reference.
+Docs are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
 
-Documentation is built using [Docusaurus 2](https://docusaurus.io/).
-
-API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
 For that reason, we ask that you add good documentation to all classes and methods.
 
 Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
 
 ### Build Documentation Locally
 
-In the following commands, the prefix `api_` indicates that those are operations for the API Reference.
-
 Before building the documentation, it is always a good idea to clean the build directory:
 
 ```bash
 make docs_clean
-make api_docs_clean
 ```
 
-Next, you can build the documentation as outlined below:
-
-```bash
-make docs_build
-make api_docs_build
-```
-
-Finally, you can run the linkchecker to make sure all links are valid:
+Next, you can run the linkchecker to make sure all links are valid:
 
 ```bash
 make docs_linkcheck
-make api_docs_linkcheck
+```
+
+Finally, you can build the documentation as outlined below:
+
+```bash
+make docs_build
 ```
 
 ## 🏭 Release Process

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,18 +7,16 @@ Replace this comment with:
   - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
   - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
 
-Please make sure you're PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
-
 If you're adding a new integration, please include:
   1. a test for the integration, preferably unit tests that do not rely on network access,
   2. an example notebook showing its use.
 
 Maintainer responsibilities:
-  - General / Misc / if you don't know who to tag: @baskaryan
+  - General / Misc / if you don't know who to tag: @dev2049
   - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
-  - Models / Prompts: @hwchase17, @baskaryan
+  - Models / Prompts: @hwchase17, @dev2049
   - Memory: @hwchase17
-  - Agents / Tools / Toolkits: @hinthornw
+  - Agents / Tools / Toolkits: @vowelparrot
   - Tracing / Callbacks: @agola11
   - Async: @agola11
 

.github/actions/poetry_setup/action.yml

@@ -52,13 +52,11 @@ runs:
 
     - name: Check Poetry File
       shell: bash
-      working-directory: ${{ inputs.working-directory }}
       run: |
         poetry check
 
     - name: Check lock file
       shell: bash
-      working-directory: ${{ inputs.working-directory }}
       run: |
         poetry lock --check
 

.github/workflows/codespell.yml

@@ -1,22 +0,0 @@
----
-name: Codespell
-
-on:
-  push:
-    branches: [master]
-  pull_request:
-    branches: [master]
-
-permissions:
-  contents: read
-
-jobs:
-  codespell:
-    name: Check for spelling errors
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-      - name: Codespell
-        uses: codespell-project/actions-codespell@v2

.github/workflows/langchain_ci.yml

@@ -1,27 +0,0 @@
----
-name: libs/langchain CI
-
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    paths:
-      - '.github/workflows/_lint.yml'
-      - '.github/workflows/_test.yml'
-      - '.github/workflows/langchain_ci.yml'
-      - 'libs/langchain/**'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  lint:
-    uses:
-      ./.github/workflows/_lint.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit
-  test:
-    uses:
-      ./.github/workflows/_test.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit

.github/workflows/langchain_experimental_ci.yml

@@ -1,29 +0,0 @@
----
-name: libs/langchain-experimental CI
-
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    paths:
-      - '.github/workflows/_lint.yml'
-      - '.github/workflows/_test.yml'
-      - '.github/workflows/langchain_experimental_ci.yml'
-      - 'libs/langchain/**'
-      - 'libs/experimental/**'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  lint:
-    uses:
-      ./.github/workflows/_lint.yml
-    with:
-      working-directory: libs/experimental
-    secrets: inherit
-  test:
-    uses:
-      ./.github/workflows/_test.yml
-    with:
-      working-directory: libs/experimental
-      test_type: '["core"]'
-    secrets: inherit

.github/workflows/langchain_experimental_release.yml

@@ -1,20 +0,0 @@
----
-name: libs/langchain-experimental Release
-
-on:
-  pull_request:
-    types:
-      - closed
-    branches:
-      - master
-    paths:
-      - 'libs/experimental/pyproject.toml'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_release.yml
-    with:
-      working-directory: libs/experimental
-    secrets: inherit

.github/workflows/langchain_release.yml

@@ -1,20 +0,0 @@
----
-name: libs/langchain Release
-
-on:
-  pull_request:
-    types:
-      - closed
-    branches:
-      - master
-    paths:
-      - 'libs/langchain/pyproject.toml'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_release.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit

.github/workflows/lint.yml

@@ -1,21 +1,15 @@
 name: lint
 
 on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
+  push:
+    branches: [master]
+  pull_request:
 
 env:
   POETRY_VERSION: "1.4.2"
 
 jobs:
   build:
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -37,10 +31,6 @@ jobs:
       - name: Install dependencies
         run: |
           poetry install
-      - name: Install langchain editable
-        if: ${{ inputs.working-directory != 'langchain' }}
-        run: |
-          pip install -e ../langchain
       - name: Analysing the code with our lint
         run: |
           make lint

.github/workflows/release.yml

@@ -1,12 +1,13 @@
 name: release
 
 on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
+  pull_request:
+    types:
+      - closed
+    branches:
+      - master
+    paths:
+      - 'pyproject.toml'
 
 env:
   POETRY_VERSION: "1.4.2"
@@ -17,9 +18,6 @@ jobs:
         ${{ github.event.pull_request.merged == true }}
         && ${{ contains(github.event.pull_request.labels.*.name, 'release') }}
     runs-on: ubuntu-latest
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
     steps:
       - uses: actions/checkout@v3
       - name: Install poetry

.github/workflows/test.yml

@@ -1,25 +1,16 @@
 name: test
 
 on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-      test_type:
-        type: string
-        description: "Test types to run"
-        default: '["core", "extended"]'
+  push:
+    branches: [master]
+  pull_request:
+  workflow_dispatch:
 
 env:
   POETRY_VERSION: "1.4.2"
 
 jobs:
   build:
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -28,7 +19,9 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
-        test_type: ${{ fromJSON(inputs.test_type) }}
+        test_type:
+          - "core"
+          - "extended"
     name: Python ${{ matrix.python-version }} ${{ matrix.test_type }}
     steps:
       - uses: actions/checkout@v3
@@ -36,7 +29,6 @@ jobs:
         uses: "./.github/actions/poetry_setup"
         with:
           python-version: ${{ matrix.python-version }}
-          working-directory: ${{ inputs.working-directory }}
           poetry-version: "1.4.2"
           cache-key: ${{ matrix.test_type }}
           install-command: |
@@ -47,10 +39,6 @@ jobs:
                 echo "Running extended tests, installing dependencies with poetry..."
                 poetry install -E extended_testing
               fi
-      - name: Install langchain editable
-        if: ${{ inputs.working-directory != 'langchain' }}
-        run: |
-          pip install -e ../langchain
       - name: Run ${{matrix.test_type}} tests
         run: |
           if [ "${{ matrix.test_type }}" == "core" ]; then

.gitignore

@@ -161,12 +161,7 @@ docs/node_modules/
 docs/.docusaurus/
 docs/.cache-loader/
 docs/_dist
-docs/api_reference/api_reference.rst
 docs/api_reference/_build
-docs/api_reference/*/
-!docs/api_reference/_static/
-!docs/api_reference/templates/
-!docs/api_reference/themes/
 docs/docs_skeleton/build
 docs/docs_skeleton/node_modules
 docs/docs_skeleton/yarn.lock

.readthedocs.yaml

@@ -9,9 +9,6 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.11"
-  jobs:
-    pre_build:
-      - python docs/api_reference/create_api_rst.py
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
@@ -24,6 +21,6 @@ sphinx:
 # Optionally declare the Python requirements required to build your docs
 python:
    install:
-   - requirements: docs/api_reference/requirements.txt
+   - requirements: docs/requirements.txt
    - method: pip
      path: .

MIGRATE.md

@@ -1,57 +0,0 @@
-# Migrating to `langchain_experimental`
-
-We are moving any experimental components of LangChain, or components with vulnerability issues, into `langchain_experimental`.
-This guide covers how to migrate.
-
-## Installation
-
-Previously:
-
-`pip install -U langchain`
-
-Now (only if you want to access things in experimental):
-
-`pip install -U langchain langchain_experimental`
-
-## Things in `langchain.experimental`
-
-Previously:
-
-`from langchain.experimental import ...`
-
-Now:
-
-`from langchain_experimental import ...`
-
-## PALChain
-
-Previously:
-
-`from langchain.chains import PALChain`
-
-Now:
-
-`from langchain_experimental.pal_chain import PALChain`
-
-## SQLDatabaseChain
-
-Previously:
-
-`from langchain.chains import SQLDatabaseChain`
-
-Now:
-
-`from langchain_experimental.sql import SQLDatabaseChain`
-
-## `load_prompt` for Python files
-
-Note: this only applies if you want to load Python files as prompts.
-If you want to load json/yaml files, no change is needed.
-
-Previously:
-
-`from langchain.prompts import load_prompt`
-
-Now:
-
-`from langchain_experimental.prompts import load_prompt`

Makefile

@@ -1,45 +1,60 @@
-.PHONY: all clean docs_build docs_clean docs_linkcheck api_docs_build api_docs_clean api_docs_linkcheck
+.PHONY: all clean format lint test tests test_watch integration_tests docker_tests help extended_tests
 
-# Default target executed when no arguments are given to make.
 all: help
 
+coverage:
+	poetry run pytest --cov \
+		--cov-config=.coveragerc \
+		--cov-report xml \
+		--cov-report term-missing:skip-covered
 
-######################
-# DOCUMENTATION
-######################
-
-clean: docs_clean api_docs_clean
+clean: docs_clean
 
+docs_compile:
+	poetry run nbdoc_build --srcdir $(srcdir)
 
 docs_build:
-	docs/.local_build.sh
+	cd docs && poetry run make html
 
 docs_clean:
-	rm -r docs/_dist
+	cd docs && poetry run make clean
 
 docs_linkcheck:
-	poetry run linkchecker docs/_dist/docs_skeleton/ --ignore-url node_modules
+	poetry run linkchecker docs/_build/html/index.html
 
-api_docs_build:
-	poetry run python docs/api_reference/create_api_rst.py
-	cd docs/api_reference && poetry run make html
+format:
+	poetry run black .
+	poetry run ruff --select I --fix .
 
-api_docs_clean:
-	rm -f docs/api_reference/api_reference.rst
-	cd docs/api_reference && poetry run make clean
+PYTHON_FILES=.
+lint: PYTHON_FILES=.
+lint_diff: PYTHON_FILES=$(shell git diff --name-only --diff-filter=d master | grep -E '\.py$$')
 
-api_docs_linkcheck:
-	poetry run linkchecker docs/api_reference/_build/html/index.html
+lint lint_diff:
+	poetry run mypy $(PYTHON_FILES)
+	poetry run black $(PYTHON_FILES) --check
+	poetry run ruff .
 
-spell_check:
-	poetry run codespell --toml pyproject.toml
+TEST_FILE ?= tests/unit_tests/
 
-spell_fix:
-	poetry run codespell --toml pyproject.toml -w
+test:
+	poetry run pytest --disable-socket --allow-unix-socket $(TEST_FILE)
 
-######################
-# HELP
-######################
+tests: 
+	poetry run pytest --disable-socket --allow-unix-socket $(TEST_FILE)
+
+extended_tests:
+	poetry run pytest --disable-socket --allow-unix-socket --only-extended tests/unit_tests
+
+test_watch:
+	poetry run ptw --now . -- tests/unit_tests
+
+integration_tests:
+	poetry run pytest tests/integration_tests
+
+docker_tests:
+	docker build -t my-langchain-image:test .
+	docker run --rm my-langchain-image:test
 
 help:
 	@echo '----'
@@ -47,3 +62,12 @@ help:
 	@echo 'docs_build                   - build the documentation'
 	@echo 'docs_clean                   - clean the documentation build artifacts'
 	@echo 'docs_linkcheck               - run linkchecker on the documentation'
+	@echo 'format                       - run code formatters'
+	@echo 'lint                         - run linters'
+	@echo 'test                         - run unit tests'
+	@echo 'tests                        - run unit tests'
+	@echo 'test TEST_FILE=<test_file>   - run all tests in file'
+	@echo 'extended_tests               - run only extended unit tests'
+	@echo 'test_watch                   - run unit tests in watch mode'
+	@echo 'integration_tests            - run integration tests'
+	@echo 'docker_tests                 - run unit tests in docker'

README.md

@@ -3,8 +3,8 @@
 ⚡ Building applications with LLMs through composability ⚡
 
 [![Release Notes](https://img.shields.io/github/release/hwchase17/langchain)](https://github.com/hwchase17/langchain/releases)
-[![CI](https://github.com/hwchase17/langchain/actions/workflows/langchain_ci.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/langchain_ci.yml)
-[![Experimental CI](https://github.com/hwchase17/langchain/actions/workflows/langchain_experimental_ci.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/langchain_experimental_ci.yml)
+[![lint](https://github.com/hwchase17/langchain/actions/workflows/lint.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/lint.yml)
+[![test](https://github.com/hwchase17/langchain/actions/workflows/test.yml/badge.svg)](https://github.com/hwchase17/langchain/actions/workflows/test.yml)
 [![Downloads](https://static.pepy.tech/badge/langchain/month)](https://pepy.tech/project/langchain)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40LangChainAI)](https://twitter.com/langchainai)
@@ -21,19 +21,11 @@ Looking for the JS/TS version? Check out [LangChain.js](https://github.com/hwcha
 **Production Support:** As you move your LangChains into production, we'd love to offer more comprehensive support.
 Please fill out [this form](https://forms.gle/57d8AmXBYp8PP8tZA) and we'll set up a dedicated support Slack channel.
 
-## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28
-
-In an effort to make `langchain` leaner and safer, we are moving select chains to `langchain_experimental`.
-This migration has already started, but we are remaining backwards compatible until 7/28.
-On that date, we will remove functionality from `langchain`.
-Read more about the motivation and the progress [here](https://github.com/hwchase17/langchain/discussions/8043).
-Read how to migrate your code [here](MIGRATE.md).
-
 ## Quick Install
 
 `pip install langchain`
 or
-`pip install langsmith && conda install langchain -c conda-forge`
+`conda install langchain -c conda-forge`
 
 ## 🤔 What is this?
 

dev.Dockerfile

@@ -37,8 +37,5 @@ ARG PYTHON_VIRTUALENV_HOME
 # Copy only the dependency files for installation
 COPY pyproject.toml poetry.toml ./
 
-# Copy the langchain library for installation
-COPY libs/langchain/ libs/langchain/
-
 # Install the Poetry dependencies (this layer will be cached as long as the dependencies don't change)
 RUN poetry install --no-interaction --no-ansi --with dev,test,docs

docs/.local_build.sh

@@ -1,15 +1,10 @@
-#!/usr/bin/env bash
-
-set -o errexit
-set -o nounset
-set -o pipefail
-set -o xtrace
-
-SCRIPT_DIR="$(cd "$(dirname "$0")"; pwd)"
-cd "${SCRIPT_DIR}"
-
-mkdir -p _dist/docs_skeleton
+mkdir _dist
 cp -r {docs_skeleton,snippets} _dist
+mkdir -p _dist/docs_skeleton/static/api_reference
+cd api_reference
+poetry run make html
+cp -r _build/* ../_dist/docs_skeleton/static/api_reference
+cd ..
 cp -r extras/* _dist/docs_skeleton/docs
 cd _dist/docs_skeleton
 poetry run nbdoc_build

docs/api_reference/_static/js/mendablesearch.js

@@ -0,0 +1,57 @@
+document.addEventListener('DOMContentLoaded', () => {
+  // Load the external dependencies
+  function loadScript(src, onLoadCallback) {
+    const script = document.createElement('script');
+    script.src = src;
+    script.onload = onLoadCallback;
+    document.head.appendChild(script);
+  }
+
+  function createRootElement() {
+    const rootElement = document.createElement('div');
+    rootElement.id = 'my-component-root';
+    document.body.appendChild(rootElement);
+    return rootElement;
+  }
+
+  
+
+  function initializeMendable() {
+    const rootElement = createRootElement();
+    const { MendableFloatingButton } = Mendable;
+    
+
+    const iconSpan1 = React.createElement('span', {
+    }, '🦜');
+
+    const iconSpan2 = React.createElement('span', {
+    }, '🔗');
+
+    const icon = React.createElement('p', {
+      style: { color: '#ffffff', fontSize: '22px',width: '48px', height: '48px', margin: '0px', padding: '0px', display: 'flex', alignItems: 'center', justifyContent: 'center', textAlign: 'center' },
+    }, [iconSpan1, iconSpan2]);
+    
+    const mendableFloatingButton = React.createElement(
+      MendableFloatingButton,
+      {
+        style: { darkMode: false, accentColor: '#010810' },
+        floatingButtonStyle: { color: '#ffffff', backgroundColor: '#010810' },
+        anon_key: '82842b36-3ea6-49b2-9fb8-52cfc4bde6bf', // Mendable Search Public ANON key, ok to be public
+        cmdShortcutKey:'j',
+        messageSettings: {
+          openSourcesInNewTab: false,
+          prettySources: true // Prettify the sources displayed now
+        },
+        icon: icon,
+      }
+    );
+
+    ReactDOM.render(mendableFloatingButton, rootElement);
+  }
+
+  loadScript('https://unpkg.com/react@17/umd/react.production.min.js', () => {
+    loadScript('https://unpkg.com/react-dom@17/umd/react-dom.production.min.js', () => {
+      loadScript('https://unpkg.com/@mendable/search@0.0.102/dist/umd/mendable.min.js', initializeMendable);
+    });
+  });
+});

docs/api_reference/agents.rst

@@ -0,0 +1,12 @@
+Agents
+==============
+
+Reference guide for Agents and associated abstractions.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   modules/agents
+   modules/tools
+   modules/agent_toolkits

docs/api_reference/conf.py

@@ -11,15 +11,13 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-import os
-import sys
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
 
 import toml
 
-sys.path.insert(0, os.path.abspath("."))
-sys.path.insert(0, os.path.abspath("../../libs/langchain"))
-
-with open("../../libs/langchain/pyproject.toml") as f:
+with open("../../pyproject.toml") as f:
     data = toml.load(f)
 
 # -- Project information -----------------------------------------------------
@@ -47,9 +45,11 @@ extensions = [
     "sphinx.ext.napoleon",
     "sphinx.ext.viewcode",
     "sphinxcontrib.autodoc_pydantic",
+    "myst_nb",
     "sphinx_copybutton",
     "sphinx_panels",
     "IPython.sphinxext.ipython_console_highlighting",
+    "sphinx_tabs.tabs",
 ]
 source_suffix = [".rst"]
 
@@ -59,22 +59,24 @@ autodoc_pydantic_config_members = False
 autodoc_pydantic_model_show_config_summary = False
 autodoc_pydantic_model_show_validator_members = False
 autodoc_pydantic_model_show_validator_summary = False
+autodoc_pydantic_model_show_field_summary = False
+autodoc_pydantic_model_members = False
+autodoc_pydantic_model_undoc_members = False
+autodoc_pydantic_model_hide_paramlist = False
 autodoc_pydantic_model_signature_prefix = "class"
-autodoc_pydantic_field_signature_prefix = "param"
-autodoc_member_order = "groupwise"
-autoclass_content = "both"
-autodoc_typehints_format = "short"
-
+autodoc_pydantic_field_signature_prefix = "attribute"
+autodoc_pydantic_model_summary_list_order = "bysource"
+autodoc_member_order = "bysource"
 autodoc_default_options = {
     "members": True,
     "show-inheritance": True,
-    "inherited-members": "BaseModel",
-    "undoc-members": True,
-    "special-members": "__call__",
+    "undoc_members": True,
+    "inherited_members": "BaseModel",
 }
-# autodoc_typehints = "description"
+autodoc_typehints = "description"
+
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ["templates"]
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -87,16 +89,14 @@ exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "scikit-learn-modern"
-html_theme_path = ["themes"]
+html_theme = "sphinx_rtd_theme"
 
-# redirects dictionary maps from old links to new links
-html_additional_pages = {}
-redirects = {
-    "index": "api_reference",
+html_theme_options = {
+    "path_to_docs": "docs",
+    "repository_url": "https://github.com/hwchase17/langchain",
+    "use_repository_button": True,
+    # "style_nav_header_background": "white"
 }
-for old_link in redirects:
-    html_additional_pages[old_link] = "redirects.html"
 
 html_context = {
     "display_github": True,  # Integrate GitHub
@@ -104,7 +104,6 @@ html_context = {
     "github_repo": "langchain",  # Repo name
     "github_version": "master",  # Version
     "conf_py_path": "/docs/api_reference",  # Path in the checkout to the docs root
-    "redirects": redirects,
 }
 
 # Add any paths that contain custom static files (such as style sheets) here,
@@ -117,9 +116,10 @@ html_static_path = ["_static"]
 html_css_files = [
     "css/custom.css",
 ]
-html_use_index = False
 
+html_js_files = [
+    "js/mendablesearch.js",
+]
+
+nb_execution_mode = "off"
 myst_enable_extensions = ["colon_fence"]
-
-# generate autosummary even if no references
-autosummary_generate = True

docs/api_reference/create_api_rst.py

@@ -1,96 +0,0 @@
-"""Script for auto-generating api_reference.rst"""
-import glob
-import re
-from pathlib import Path
-
-ROOT_DIR = Path(__file__).parents[2].absolute()
-PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
-WRITE_FILE = Path(__file__).parent / "api_reference.rst"
-
-
-def load_members() -> dict:
-    members: dict = {}
-    for py in glob.glob(str(PKG_DIR) + "/**/*.py", recursive=True):
-        module = py[len(str(PKG_DIR)) + 1 :].replace(".py", "").replace("/", ".")
-        top_level = module.split(".")[0]
-        if top_level not in members:
-            members[top_level] = {"classes": [], "functions": []}
-        with open(py, "r") as f:
-            for line in f.readlines():
-                cls = re.findall(r"^class ([^_].*)\(", line)
-                members[top_level]["classes"].extend([module + "." + c for c in cls])
-                func = re.findall(r"^def ([^_].*)\(", line)
-                afunc = re.findall(r"^async def ([^_].*)\(", line)
-                func_strings = [module + "." + f for f in func + afunc]
-                members[top_level]["functions"].extend(func_strings)
-    return members
-
-
-def construct_doc(members: dict) -> str:
-    full_doc = """\
-.. _api_reference:
-
-=============
-API Reference
-=============
-
-"""
-    for module, _members in sorted(members.items(), key=lambda kv: kv[0]):
-        classes = _members["classes"]
-        functions = _members["functions"]
-        if not (classes or functions):
-            continue
-
-        module_title = module.replace("_", " ").title()
-        if module_title == "Llms":
-            module_title = "LLMs"
-        section = f":mod:`langchain.{module}`: {module_title}"
-        full_doc += f"""\
-{section}
-{'=' * (len(section) + 1)}
-
-.. automodule:: langchain.{module}
-    :no-members:
-    :no-inherited-members:
-
-"""
-
-        if classes:
-            cstring = "\n    ".join(sorted(classes))
-            full_doc += f"""\
-Classes
---------------
-.. currentmodule:: langchain
-
-.. autosummary::
-    :toctree: {module}
-    :template: class.rst
-
-    {cstring}
-
-"""
-        if functions:
-            fstring = "\n    ".join(sorted(functions))
-            full_doc += f"""\
-Functions
---------------
-.. currentmodule:: langchain
-
-.. autosummary::
-    :toctree: {module}
-
-    {fstring}
-
-"""
-    return full_doc
-
-
-def main() -> None:
-    members = load_members()
-    full_doc = construct_doc(members)
-    with open(WRITE_FILE, "w") as f:
-        f.write(full_doc)
-
-
-if __name__ == "__main__":
-    main()

docs/api_reference/data_connection.rst

@@ -0,0 +1,13 @@
+Data connection
+==============
+LangChain has a number of modules that help you load, structure, store, and retrieve documents.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   modules/document_loaders
+   modules/document_transformers
+   modules/embeddings
+   modules/vectorstores
+   modules/retrievers

docs/api_reference/index.rst

@@ -1,8 +1,29 @@
-=============
-LangChain API
-=============
+API Reference
+==========================
+
+| Full documentation on all methods, classes, and APIs in the LangChain Python package.
 
 .. toctree::
-    :maxdepth: 2
+   :maxdepth: 1
+   :caption: Abstractions
 
-    api_reference.rst
+   ./modules/base_classes.rst
+
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Core
+
+   ./model_io.rst
+   ./data_connection.rst
+   ./modules/chains.rst
+   ./agents.rst
+   ./modules/memory.rst
+   ./modules/callbacks.rst
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Additional
+
+   ./modules/utilities.rst
+   ./modules/experimental.rst

docs/api_reference/model_io.rst

@@ -0,0 +1,12 @@
+Model I/O
+==============
+
+LangChain provides interfaces and integrations for working with language models.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   ./prompts.rst
+   ./models.rst
+   ./modules/output_parsers.rst

docs/api_reference/models.rst

@@ -0,0 +1,11 @@
+Models
+==============
+
+LangChain provides interfaces and integrations for a number of different types of models.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   modules/llms
+   modules/chat_models

docs/api_reference/modules/agent_toolkits.rst

@@ -0,0 +1,7 @@
+Agent Toolkits
+===============================
+
+.. automodule:: langchain.agents.agent_toolkits
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/agents.rst

@@ -0,0 +1,7 @@
+Agents
+===============================
+
+.. automodule:: langchain.agents
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/base_classes.rst

@@ -0,0 +1,5 @@
+Base classes
+========================
+
+.. automodule:: langchain.schema
+   :inherited-members:

docs/api_reference/modules/callbacks.rst

@@ -0,0 +1,7 @@
+Callbacks
+=======================
+
+.. automodule:: langchain.callbacks
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/chains.rst

@@ -0,0 +1,8 @@
+Chains
+=======================
+
+.. automodule:: langchain.chains
+   :members:
+   :undoc-members:
+   :inherited-members: BaseModel
+

docs/api_reference/modules/chat_models.rst

@@ -0,0 +1,7 @@
+Chat Models
+===============================
+
+.. automodule:: langchain.chat_models
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/document_loaders.rst

@@ -0,0 +1,7 @@
+Document Loaders
+===============================
+
+.. automodule:: langchain.document_loaders
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/document_transformers.rst

@@ -0,0 +1,13 @@
+Document Transformers
+===============================
+
+.. automodule:: langchain.document_transformers
+   :members:
+   :undoc-members:
+
+Text Splitters
+------------------------------
+
+.. automodule:: langchain.text_splitter
+   :members:
+   :undoc-members:

docs/api_reference/modules/embeddings.rst

@@ -0,0 +1,5 @@
+Embeddings
+===========================
+
+.. automodule:: langchain.embeddings
+   :members:

docs/api_reference/modules/evaluation.rst

@@ -1,9 +0,0 @@
-Evaluation
-=======================
-
-LangChain has a number of convenient evaluation chains you can use off the shelf to grade your models' oupputs.
-
-.. automodule:: langchain.evaluation
-   :members:
-   :undoc-members:
-   :inherited-members:

docs/api_reference/modules/example_selector.rst

@@ -0,0 +1,5 @@
+Example Selector
+=========================================
+
+.. automodule:: langchain.prompts.example_selector
+   :members:

docs/api_reference/modules/experimental.rst

@@ -0,0 +1,28 @@
+====================
+Experimental
+====================
+
+This module contains experimental modules and reproductions of existing work using LangChain primitives.
+
+Autonomous agents
+------------------
+
+Here, we document the BabyAGI and AutoGPT classes from the langchain.experimental module.
+
+.. autoclass:: langchain.experimental.BabyAGI
+   :members:
+
+.. autoclass:: langchain.experimental.AutoGPT
+   :members:
+
+
+Generative agents
+------------------
+
+Here, we document the GenerativeAgent and GenerativeAgentMemory classes from the langchain.experimental module.
+
+.. autoclass:: langchain.experimental.GenerativeAgent
+   :members:
+
+.. autoclass:: langchain.experimental.GenerativeAgentMemory
+   :members:

docs/api_reference/modules/llms.rst

@@ -0,0 +1,7 @@
+LLMs
+=======================
+
+.. automodule:: langchain.llms
+   :members:
+   :inherited-members:
+   :special-members: __call__

docs/api_reference/modules/memory.rst

@@ -0,0 +1,7 @@
+Memory
+===============================
+
+.. automodule:: langchain.memory
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/output_parsers.rst

@@ -0,0 +1,7 @@
+Output Parsers
+===============================
+
+.. automodule:: langchain.output_parsers
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/prompts.rst

@@ -0,0 +1,6 @@
+Prompt Templates
+========================
+
+.. automodule:: langchain.prompts
+   :members:
+   :undoc-members:

docs/api_reference/modules/retrievers.rst

@@ -0,0 +1,14 @@
+Retrievers
+===============================
+
+.. automodule:: langchain.retrievers
+   :members:
+   :undoc-members:
+
+Document compressors
+-------------------------------
+
+.. automodule:: langchain.retrievers.document_compressors
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/tools.rst

@@ -0,0 +1,7 @@
+Tools
+===============================
+
+.. automodule:: langchain.tools
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/utilities.rst

@@ -0,0 +1,7 @@
+Utilities
+===============================
+
+.. automodule:: langchain.utilities
+   :members:
+   :undoc-members:
+

docs/api_reference/modules/vectorstores.rst

@@ -0,0 +1,6 @@
+Vector Stores
+=============================
+
+.. automodule:: langchain.vectorstores
+   :members:
+   :undoc-members:

docs/api_reference/prompts.rst

@@ -0,0 +1,11 @@
+Prompts
+==============
+
+The reference guides here all relate to objects for working with Prompts.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   modules/prompts
+   modules/example_selector

docs/api_reference/templates/COPYRIGHT.txt

@@ -1,27 +0,0 @@
-Copyright (c) 2007-2023 The scikit-learn developers.
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-* Redistributions of source code must retain the above copyright notice, this
-  list of conditions and the following disclaimer.
-
-* Redistributions in binary form must reproduce the above copyright notice,
-  this list of conditions and the following disclaimer in the documentation
-  and/or other materials provided with the distribution.
-
-* Neither the name of the copyright holder nor the names of its
-  contributors may be used to endorse or promote products derived from
-  this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

docs/api_reference/templates/class.rst

@@ -1,28 +0,0 @@
-:mod:`{{module}}`.{{objname}}
-{{ underline }}==============
-
-.. currentmodule:: {{ module }}
-
-.. autoclass:: {{ objname }}
-
-   {% block methods %}
-   {% if methods %}
-   .. rubric:: {{ _('Methods') }}
-
-   .. autosummary::
-   {% for item in methods %}
-      ~{{ name }}.{{ item }}
-   {%- endfor %}
-   {% endif %}
-   {% endblock %}
-
-   {% block attributes %}
-   {% if attributes %}
-   .. rubric:: {{ _('Attributes') }}
-
-   .. autosummary::
-   {% for item in attributes %}
-      ~{{ name }}.{{ item }}
-   {%- endfor %}
-   {% endif %}
-   {% endblock %}

docs/api_reference/templates/redirects.html

@@ -1,15 +0,0 @@
-{% set redirect = pathto(redirects[pagename]) %}
-<!DOCTYPE html>
-<html>
-  <head>
-    <meta charset="utf-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <meta http-equiv="Refresh" content="0; url={{ redirect }}" />
-    <meta name="Description" content="scikit-learn: machine learning in Python">
-    <link rel="canonical" href="{{ redirect }}" />
-    <title>scikit-learn: machine learning in Python</title>
-  </head>
-  <body>
-    <p>You will be automatically redirected to the <a href="{{ redirect }}">new location of this page</a>.</p>
-  </body>
-</html>

docs/api_reference/themes/COPYRIGHT.txt

@@ -1,27 +0,0 @@
-Copyright (c) 2007-2023 The scikit-learn developers.
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-* Redistributions of source code must retain the above copyright notice, this
-  list of conditions and the following disclaimer.
-
-* Redistributions in binary form must reproduce the above copyright notice,
-  this list of conditions and the following disclaimer in the documentation
-  and/or other materials provided with the distribution.
-
-* Neither the name of the copyright holder nor the names of its
-  contributors may be used to endorse or promote products derived from
-  this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

docs/api_reference/themes/scikit-learn-modern/javascript.html

@@ -1,67 +0,0 @@
-<script>
-$(document).ready(function() {
-    /* Add a [>>>] button on the top-right corner of code samples to hide
-     * the >>> and ... prompts and the output and thus make the code
-     * copyable. */
-    var div = $('.highlight-python .highlight,' +
-                '.highlight-python3 .highlight,' +
-                '.highlight-pycon .highlight,' +
-		'.highlight-default .highlight')
-    var pre = div.find('pre');
-
-    // get the styles from the current theme
-    pre.parent().parent().css('position', 'relative');
-    var hide_text = 'Hide prompts and outputs';
-    var show_text = 'Show prompts and outputs';
-
-    // create and add the button to all the code blocks that contain >>>
-    div.each(function(index) {
-        var jthis = $(this);
-        if (jthis.find('.gp').length > 0) {
-            var button = $('<span class="copybutton">&gt;&gt;&gt;</span>');
-            button.attr('title', hide_text);
-            button.data('hidden', 'false');
-            jthis.prepend(button);
-        }
-        // tracebacks (.gt) contain bare text elements that need to be
-        // wrapped in a span to work with .nextUntil() (see later)
-        jthis.find('pre:has(.gt)').contents().filter(function() {
-            return ((this.nodeType == 3) && (this.data.trim().length > 0));
-        }).wrap('<span>');
-    });
-
-    // define the behavior of the button when it's clicked
-    $('.copybutton').click(function(e){
-        e.preventDefault();
-        var button = $(this);
-        if (button.data('hidden') === 'false') {
-            // hide the code output
-            button.parent().find('.go, .gp, .gt').hide();
-            button.next('pre').find('.gt').nextUntil('.gp, .go').css('visibility', 'hidden');
-            button.css('text-decoration', 'line-through');
-            button.attr('title', show_text);
-            button.data('hidden', 'true');
-        } else {
-            // show the code output
-            button.parent().find('.go, .gp, .gt').show();
-            button.next('pre').find('.gt').nextUntil('.gp, .go').css('visibility', 'visible');
-            button.css('text-decoration', 'none');
-            button.attr('title', hide_text);
-            button.data('hidden', 'false');
-        }
-    });
-
-	/*** Add permalink buttons next to glossary terms ***/
-	$('dl.glossary > dt[id]').append(function() {
-		return ('<a class="headerlink" href="#' +
-			    this.getAttribute('id') +
-			    '" title="Permalink to this term">¶</a>');
-	});
-});
-
-</script>
-{%- if pagename != 'index' and pagename != 'documentation' %}
-    {% if theme_mathjax_path %}
-<script id="MathJax-script" async src="{{ theme_mathjax_path }}"></script>
-    {% endif %}
-{%- endif %}

docs/api_reference/themes/scikit-learn-modern/layout.html

@@ -1,142 +0,0 @@
-{# TEMPLATE VAR SETTINGS #}
-{%- set url_root = pathto('', 1) %}
-{%- if url_root == '#' %}{% set url_root = '' %}{% endif %}
-{%- if not embedded and docstitle %}
-  {%- set titlesuffix = " — "|safe + docstitle|e %}
-{%- else %}
-  {%- set titlesuffix = "" %}
-{%- endif %}
-{%- set lang_attr = 'en' %}
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="{{ lang_attr }}" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="{{ lang_attr }}" > <!--<![endif]-->
-<head>
-  <meta charset="utf-8">
-  {{ metatags }}
-  <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
-  {% block htmltitle %}
-  <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
-  {% endblock %}
-  <link rel="canonical" href="http://scikit-learn.org/stable/{{pagename}}.html" />
-
-  {% if favicon_url %}
-  <link rel="shortcut icon" href="{{ favicon_url|e }}"/>
-  {% endif %}
-
-  <link rel="stylesheet" href="{{ pathto('_static/css/vendor/bootstrap.min.css', 1) }}" type="text/css" />
-  {%- for css in css_files %}
-    {%- if css|attr("rel") %}
-  <link rel="{{ css.rel }}" href="{{ pathto(css.filename, 1) }}" type="text/css"{% if css.title is not none %} title="{{ css.title }}"{% endif %} />
-    {%- else %}
-  <link rel="stylesheet" href="{{ pathto(css, 1) }}" type="text/css" />
-    {%- endif %}
-  {%- endfor %}
-  <link rel="stylesheet" href="{{ pathto('_static/' + style, 1) }}" type="text/css" />
-<script id="documentation_options" data-url_root="{{ pathto('', 1) }}" src="{{ pathto('_static/documentation_options.js', 1) }}"></script>
-<script src="{{ pathto('_static/jquery.js', 1) }}"></script>
-{%- block extrahead %} {% endblock %}
-</head>
-<body>
-{% include "nav.html" %}
-{%- block content %}
-<div class="d-flex" id="sk-doc-wrapper">
-    <input type="checkbox" name="sk-toggle-checkbox" id="sk-toggle-checkbox">
-    <label id="sk-sidemenu-toggle" class="sk-btn-toggle-toc btn sk-btn-primary" for="sk-toggle-checkbox">Toggle Menu</label>
-    <div id="sk-sidebar-wrapper" class="border-right">
-      <div class="sk-sidebar-toc-wrapper">
-        <div class="btn-group w-100 mb-2" role="group" aria-label="rellinks">
-          {%- if prev %}
-            <a href="{{ prev.link|e }}" role="button" class="btn sk-btn-rellink py-1" sk-rellink-tooltip="{{ prev.title|striptags }}">Prev</a>
-          {%- else %}
-            <a href="#" role="button" class="btn sk-btn-rellink py-1 disabled"">Prev</a>
-          {%- endif %}
-          {%- if parents -%}
-            <a href="{{ parents[-1].link|e }}" role="button" class="btn sk-btn-rellink py-1" sk-rellink-tooltip="{{ parents[-1].title|striptags }}">Up</a>
-          {%- else %}
-            <a href="#" role="button" class="btn sk-btn-rellink disabled py-1">Up</a>
-          {%- endif %}
-          {%- if next %}
-            <a href="{{ next.link|e }}" role="button" class="btn sk-btn-rellink py-1" sk-rellink-tooltip="{{ next.title|striptags }}">Next</a>
-          {%- else %}
-            <a href="#" role="button" class="btn sk-btn-rellink py-1 disabled"">Next</a>
-          {%- endif %}
-        </div>
-        {%- if pagename != "install" %}
-        <div class="alert alert-warning p-1 mb-2" role="alert">
-          <p class="text-center mb-0">
-          <strong>LangChain {{ release }}</strong><br/>
-          </p>
-        </div>
-        {%- endif %}
-            {%- if meta and meta['parenttoc']|tobool %}
-            <div class="sk-sidebar-toc">
-            {% set nav = get_nav_object(maxdepth=3, collapse=True, numbered=True) %}
-              <ul>
-              {% for main_nav_item in nav %}
-              {% if main_nav_item.active %}
-              <li>
-                <a href="{{ main_nav_item.url }}" class="sk-toc-active">{{ main_nav_item.title }}</a>
-              </li>
-              <ul>
-              {% for nav_item in main_nav_item.children %}
-                <li>
-                  <a href="{{ nav_item.url }}" class="{% if nav_item.active %}sk-toc-active{% endif %}">{{ nav_item.title }}</a>
-                  {% if nav_item.children %}
-                  <ul>
-                    {% for inner_child in nav_item.children %}
-                      <li class="sk-toctree-l3">
-                        <a href="{{ inner_child.url }}">{{ inner_child.title }}</a>
-                      </li>
-                    {% endfor %}
-                  </ul>
-                  {% endif %}
-                </li>
-              {% endfor %}
-              </ul>
-              {% endif %}
-              {% endfor %}
-              </ul>
-            </div>
-            {%- elif meta and meta['globalsidebartoc']|tobool %}
-            <div class="sk-sidebar-toc sk-sidebar-global-toc">
-              {{ toctree(maxdepth=2, titles_only=True) }}
-            </div>
-            {%- else %}
-            <div class="sk-sidebar-toc">
-              {{ toc }}
-            </div>
-            {%- endif %}
-      </div>
-    </div>
-    <div id="sk-page-content-wrapper">
-      <div class="sk-page-content container-fluid body px-md-3" role="main">
-        {% block body %}{% endblock %}
-      </div>
-    <div class="container">
-      <footer class="sk-content-footer">
-        {%- if pagename != 'index' %}
-        {%- if show_copyright %}
-          {%- if hasdoc('copyright') %}
-            {% trans path=pathto('copyright'), copyright=copyright|e %}&copy; {{ copyright }}.{% endtrans %}
-          {%- else %}
-            {% trans copyright=copyright|e %}&copy; {{ copyright }}.{% endtrans %}
-          {%- endif %}
-        {%- endif %}
-        {%- if last_updated %}
-          {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
-        {%- endif %}
-        {%- if show_source and has_source and sourcename %}
-          <a href="{{ pathto('_sources/' + sourcename, true)|e }}" rel="nofollow">{{ _('Show this page source') }}</a>
-        {%- endif %}
-        {%- endif %}
-      </footer>
-    </div>
-  </div>
-</div>
-{%- endblock %}
-<script src="{{ pathto('_static/js/vendor/bootstrap.min.js', 1) }}"></script>
-{% include "javascript.html" %}
-</body>
-</html>

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -1,69 +0,0 @@
-{%- if pagename != 'index' and pagename != 'documentation' %}
-  {%- set nav_bar_class = "sk-docs-navbar" %}
-  {%- set top_container_cls = "sk-docs-container" %}
-{%- else %}
-  {%- set nav_bar_class = "sk-landing-navbar" %}
-  {%- set top_container_cls = "sk-landing-container" %}
-{%- endif %}
-
-{% if theme_link_to_live_contributing_page|tobool %}
-{# Link to development page for live builds #}
-  {%- set development_link = "https://scikit-learn.org/dev/developers/index.html" %}
-{# Open on a new development page in new window/tab for live builds #}
-  {%- set development_attrs = 'target="_blank" rel="noopener noreferrer"' %}
-{%- else %}
-  {%- set development_link = pathto('developers/index') %}
-  {%- set development_attrs = '' %}
-{%- endif %}
-
-
-<nav id="navbar" class="{{ nav_bar_class }} navbar navbar-expand-md navbar-light bg-light py-0">
-  <div class="container-fluid {{ top_container_cls }} px-0">
-    {%- if logo_url %}
-      <a class="navbar-brand py-0" href="{{ pathto('index') }}">
-        <img
-          class="sk-brand-img"
-          src="{{ logo_url|e }}"
-          alt="logo"/>
-      </a>
-    {%- endif %}
-    <button
-      id="sk-navbar-toggler"
-      class="navbar-toggler"
-      type="button"
-      data-toggle="collapse"
-      data-target="#navbarSupportedContent"
-      aria-controls="navbarSupportedContent"
-      aria-expanded="false"
-      aria-label="Toggle navigation"
-    >
-      <span class="navbar-toggler-icon"></span>
-    </button>
-
-    <div class="sk-navbar-collapse collapse navbar-collapse" id="navbarSupportedContent">
-      <ul class="navbar-nav mr-auto">
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
-        </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
-        </li>
-        {%- for title, link, link_attrs in drop_down_navigation %}
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link nav-more-item-mobile-items" href="{{ link }}" {{ link_attrs }}>{{ title }}</a>
-        </li>
-        {%- endfor %}
-      </ul>
-      {%- if pagename != "search"%}
-      <div id="searchbox" role="search">
-          <div class="searchformwrapper">
-          <form class="search" action="{{ pathto('search') }}" method="get">
-            <input class="sk-search-text-input" type="text" name="q" aria-labelledby="searchlabel" />
-            <input class="sk-search-text-btn" type="submit" value="{{ _('Go') }}" />
-          </form>
-          </div>
-      </div>
-      {%- endif %}
-    </div>
-  </div>
-</nav>

docs/api_reference/themes/scikit-learn-modern/search.html

@@ -1,16 +0,0 @@
-{%- extends "basic/search.html" %}
-{% block extrahead %}
-  <script type="text/javascript" src="{{ pathto('_static/underscore.js', 1) }}"></script>
-  <script type="text/javascript" src="{{ pathto('searchindex.js', 1) }}" defer></script>
-  <script type="text/javascript" src="{{ pathto('_static/doctools.js', 1) }}"></script>
-  <script type="text/javascript" src="{{ pathto('_static/language_data.js', 1) }}"></script>
-  <script type="text/javascript" src="{{ pathto('_static/searchtools.js', 1) }}"></script>
-  <!-- <script type="text/javascript" src="{{ pathto('_static/sphinx_highlight.js', 1) }}"></script> -->
-  <script type="text/javascript">
-    $(document).ready(function() {
-      if (!Search.out) {
-        Search.init();
-      }
-    });
-  </script>
-{% endblock %}

docs/api_reference/themes/scikit-learn-modern/theme.conf

@@ -1,8 +0,0 @@
-[theme]
-inherit = basic
-pygments_style = default
-stylesheet = css/theme.css
-
-[options]
-link_to_live_contributing_page = false
-mathjax_path =

docs/docs_skeleton/docs/ecosystem/integrations/index.mdx

@@ -1,8 +1,7 @@
 ---
 sidebar_position: 0
 ---
-
-# Memory
+# Integrations
 
 import DocCardList from "@theme/DocCardList";
 

docs/docs_skeleton/docs/get_started/introduction.mdx

@@ -51,7 +51,7 @@ Walkthroughs and best-practices for common end-to-end use cases, like:
 Learn best practices for developing with LangChain.
 
 ### [Ecosystem](/docs/ecosystem/)
-LangChain is part of a rich ecosystem of tools that integrate with our framework and build on top of it. Check out our growing list of [integrations](/docs/integrations/) and [dependent repos](/docs/ecosystem/dependents).
+LangChain is part of a rich ecosystem of tools that integrate with our framework and build on top of it. Check out our growing list of [integrations](/docs/ecosystem/integrations/) and [dependent repos](/docs/ecosystem/dependents.html).
 
 ### [Additional resources](/docs/additional_resources/)
 Our community is full of prolific developers, creative builders, and fantastic teachers. Check out [YouTube tutorials](/docs/additional_resources/youtube.html) for great tutorials from folks in the community, and [Gallery](https://github.com/kyrolabs/awesome-langchain) for a list of awesome LangChain projects, compiled by the folks at [KyroLabs](https://kyrolabs.com).

docs/docs_skeleton/docs/get_started/quickstart.mdx

@@ -22,141 +22,137 @@ import OpenAISetup from "@snippets/get_started/quickstart/openai_setup.mdx"
 
 ## Building an application
 
-Now we can start building our language model application. LangChain provides many modules that can be used to build language model applications.
-Modules can be used as stand-alones in simple applications and they can be combined for more complex use cases.
-
-The core building block of LangChain applications is the LLMChain.
-This combines three things:
-- LLM: The language model is the core reasoning engine here. In order to work with LangChain, you need to understand the different types of language models and how to work with them.
-- Prompt Templates: This provides instructions to the language model. This controls what the language model outputs, so understanding how to construct prompts and different prompting strategies is crucial.
-- Output Parsers: These translate the raw response from the LLM to a more workable format, making it easy to use the output downstream.
-
-In this getting started guide we will cover those three components by themselves, and then cover the LLMChain which combines all of them.
-Understanding these concepts will set you up well for being able to use and customize LangChain applications.
-Most LangChain applications allow you to configure the LLM and/or the prompt used, so knowing how to take advantage of this will be a big enabler.
+Now we can start building our language model application. LangChain provides many modules that can be used to build language model applications. Modules can be used as stand-alones in simple applications and they can be combined for more complex use cases.
 
 ## LLMs
+#### Get predictions from a language model
 
-There are two types of language models, which in LangChain are called:
+The basic building block of LangChain is the LLM, which takes in text and generates more text.
 
-- LLMs: this is a language model which takes a string as input and returns a string
-- ChatModels: this is a language model which takes a list of messages as input and returns a message
+As an example, suppose we're building an application that generates a company name based on a company description. In order to do this, we need to initialize an OpenAI model wrapper. In this case, since we want the outputs to be MORE random, we'll initialize our model with a HIGH temperature.
 
-The input/output for LLMs is simple and easy to understand - a string.
-But what about ChatModels? The input there is a list of `ChatMessage`s, and the output is a single `ChatMessage`.
-A `ChatMessage` has two required components:
+import LLM from "@snippets/get_started/quickstart/llm.mdx"
 
-- `content`: This is the content of the message.
-- `role`: This is the role of the entity from which the `ChatMessage` is coming from.
+<LLM/>
 
-LangChain provides several objects to easily distinguish between different roles:
+## Chat models
 
-- `HumanMessage`: A `ChatMessage` coming from a human/user.
-- `AIMessage`: A `ChatMessage` coming from an AI/assistant.
-- `SystemMessage`: A `ChatMessage` coming from the system.
-- `FunctionMessage`: A `ChatMessage` coming from a function call.
+Chat models are a variation on language models. While chat models use language models under the hood, the interface they expose is a bit different: rather than expose a "text in, text out" API, they expose an interface where "chat messages" are the inputs and outputs.
 
-If none of those roles sound right, there is also a `ChatMessage` class where you can specify the role manually.
-For more information on how to use these different messages most effectively, see our prompting guide.
+You can get chat completions by passing one or more messages to the chat model. The response will be a message. The types of messages currently supported in LangChain are `AIMessage`, `HumanMessage`, `SystemMessage`, and `ChatMessage` -- `ChatMessage` takes in an arbitrary role parameter. Most of the time, you'll just be dealing with `HumanMessage`, `AIMessage`, and `SystemMessage`.
 
-LangChain exposes a standard interface for both, but it's useful to understand this difference in order to construct prompts for a given language model.
-The standard interface that LangChain exposes has two methods:
-- `predict`: Takes in a string, returns a string
-- `predict_messages`: Takes in a list of messages, returns a message.
-
-Let's see how to work with these different types of models and these different types of inputs.
-First, let's import an LLM and a ChatModel.
-
-import ImportLLMs from "@snippets/get_started/quickstart/import_llms.mdx"
-
-<ImportLLMs/>
-
-The `OpenAI` and `ChatOpenAI` objects are basically just configuration objects.
-You can initialize them with parameters like `temperature` and others, and pass them around.
-
-Next, let's use the `predict` method to run over a string input.
-
-import InputString from "@snippets/get_started/quickstart/input_string.mdx"
-
-<InputString/>
-
-Finally, let's use the `predict_messages` method to run over a list of messages.
-
-import InputMessages from "@snippets/get_started/quickstart/input_messages.mdx"
-
-<InputMessages/>
-
-For both these methods, you can also pass in parameters as key word arguments.
-For example, you could pass in `temperature=0` to adjust the temperature that is used from what the object was configured with.
-Whatever values are passed in during run time will always override what the object was configured with.
+import ChatModel from "@snippets/get_started/quickstart/chat_model.mdx"
 
+<ChatModel/>
 
 ## Prompt templates
 
-Most LLM applications do not pass user input directly into an LLM. Usually they will add the user input to a larger piece of text, called a prompt template, that provides additional context on the specific task at hand.
+Most LLM applications do not pass user input directly into to an LLM. Usually they will add the user input to a larger piece of text, called a prompt template, that provides additional context on the specific task at hand.
 
 In the previous example, the text we passed to the model contained instructions to generate a company name. For our application, it'd be great if the user only had to provide the description of a company/product, without having to worry about giving the model instructions.
 
-PromptTemplates help with exactly this!
-They bundle up all the logic for going from user input into a fully formatted prompt.
-This can start off very simple - for example, a prompt to produce the above string would just be:
-
 import PromptTemplateLLM from "@snippets/get_started/quickstart/prompt_templates_llms.mdx"
 import PromptTemplateChatModel from "@snippets/get_started/quickstart/prompt_templates_chat_models.mdx"
 
+<Tabs>
+    <TabItem value="llms" label="LLMs" default>
+
+With PromptTemplates this is easy! In this case our template would be very simple:
+
 <PromptTemplateLLM/>
+</TabItem>
+<TabItem value="chat_models" label="Chat models">
 
-However, the advantages of using these over raw string formatting are several.
-You can "partial" out variables - eg you can format only some of the variables at a time.
-You can compose them together, easily combining different templates into a single prompt.
-For explanations of these functionalities, see the [section on prompts](/docs/modules/model_io/prompts) for more detail.
+Similar to LLMs, you can make use of templating by using a `MessagePromptTemplate`. You can build a `ChatPromptTemplate` from one or more `MessagePromptTemplate`s. You can use `ChatPromptTemplate`'s `format_messages` method to generate the formatted messages.
 
-PromptTemplates can also be used to produce a list of messages.
-In this case, the prompt not only contains information about the content, but also each message (its role, its position in the list, etc)
-Here, what happens most often is a ChatPromptTemplate is a list of ChatMessageTemplates.
-Each ChatMessageTemplate contains instructions for how to format that ChatMessage - its role, and then also its content.
-Let's take a look at this below:
+Because this is generating a list of messages, it is slightly more complex than the normal prompt template which is generating only a string. Please see the detailed guides on prompts to understand more options available to you here.
 
 <PromptTemplateChatModel/>
+    </TabItem>
+</Tabs>
 
-ChatPromptTemplates can also include other things besides ChatMessageTemplates - see the [section on prompts](/docs/modules/model_io/prompts) for more detail.
+## Chains
 
-## Output Parsers
+Now that we've got a model and a prompt template, we'll want to combine the two. Chains give us a way to link (or chain) together multiple primitives, like models, prompts, and other chains.
 
-OutputParsers convert the raw output of an LLM into a format that can be used downstream.
-There are few main type of OutputParsers, including:
+import ChainLLM from "@snippets/get_started/quickstart/chains_llms.mdx"
+import ChainChatModel from "@snippets/get_started/quickstart/chains_chat_models.mdx"
 
-- Convert text from LLM -> structured information (eg JSON)
-- Convert a ChatMessage into just a string
-- Convert the extra information returned from a call besides the message (like OpenAI function invocation) into a string.
+<Tabs>
+<TabItem value="llms" label="LLMs" default>
 
-For full information on this, see the [section on output parsers](/docs/modules/model_io/output_parsers)
+The simplest and most common type of chain is an LLMChain, which passes an input first to a PromptTemplate and then to an LLM. We can construct an LLM chain from our existing model and prompt template.
 
-In this getting started guide, we will write our own output parser - one that converts a comma separated list into a list.
+<ChainLLM/>
 
-import OutputParser from "@snippets/get_started/quickstart/output_parser.mdx"
+There we go, our first chain! Understanding how this simple chain works will set you up well for working with more complex chains.
 
-<OutputParser/>
+</TabItem>
+<TabItem value="chat_models" label="Chat models">
 
-## LLMChain
+The `LLMChain` can be used with chat models as well:
 
-We can now combine all these into one chain.
-This chain will take input variables, pass those to a prompt template to create a prompt, pass the prompt to an LLM, and then pass the output through an (optional) output parser.
-This is a convenient way to bundle up a modular piece of logic.
-Let's see it in action!
+<ChainChatModel/>
+</TabItem>
+</Tabs>
 
-import LLMChain from "@snippets/get_started/quickstart/llm_chain.mdx"
+## Agents
 
-<LLMChain/>
+import AgentLLM from "@snippets/get_started/quickstart/agents_llms.mdx"
+import AgentChatModel from "@snippets/get_started/quickstart/agents_chat_models.mdx"
 
-## Next Steps
+Our first chain ran a pre-determined sequence of steps. To handle complex workflows, we need to be able to dynamically choose actions based on inputs.
 
-This is it!
-We've now gone over how to create the core building block of LangChain applications - the LLMChains.
-There is a lot more nuance in all these components (LLMs, prompts, output parsers) and a lot more different components to learn about as well.
-To continue on your journey:
+Agents do just this: they use a language model to determine which actions to take and in what order. Agents are given access to tools, and they repeatedly choose a tool, run the tool, and observe the output until they come up with a final answer.
 
-- [Dive deeper](/docs/modules/model_io) into LLMs, prompts, and output parsers
-- Learn the other [key components](/docs/modules)
-- Check out our [helpful guides](/docs/guides) for detailed walkthroughs on particular topics
-- Explore [end-to-end use cases](/docs/use_cases)
+To load an agent, you need to choose a(n):
+- LLM/Chat model: The language model powering the agent.
+- Tool(s): A function that performs a specific duty. This can be things like: Google Search, Database lookup, Python REPL, other chains. For a list of predefined tools and their specifications, see the [Tools documentation](/docs/modules/agents/tools/).
+- Agent name: A string that references a supported agent class. An agent class is largely parameterized by the prompt the language model uses to determine which action to take. Because this notebook focuses on the simplest, highest level API, this only covers using the standard supported agents. If you want to implement a custom agent, see [here](/docs/modules/agents/how_to/custom_agent.html). For a list of supported agents and their specifications, see [here](/docs/modules/agents/agent_types/).
+
+For this example, we'll be using SerpAPI to query a search engine.
+
+You'll need to install the SerpAPI Python package:
+
+```bash
+pip install google-search-results
+```
+
+And set the `SERPAPI_API_KEY` environment variable.
+
+<Tabs>
+<TabItem value="llms" label="LLMs" default>
+<AgentLLM/>
+</TabItem>
+<TabItem value="chat_models" label="Chat models">
+
+Agents can also be used with chat models, you can initialize one using `AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION` as the agent type.
+
+<AgentChatModel/>
+</TabItem>
+</Tabs>
+
+## Memory
+
+The chains and agents we've looked at so far have been stateless, but for many applications it's necessary to reference past interactions. This is clearly the case with a chatbot for example, where you want it to understand new messages in the context of past messages.
+
+The Memory module gives you a way to maintain application state. The base Memory interface is simple: it lets you update state given the latest run inputs and outputs and it lets you modify (or contextualize) the next input using the stored state.
+
+There are a number of built-in memory systems. The simplest of these are is a buffer memory which just prepends the last few inputs/outputs to the current input - we will use this in the example below.
+
+import MemoryLLM from "@snippets/get_started/quickstart/memory_llms.mdx"
+import MemoryChatModel from "@snippets/get_started/quickstart/memory_chat_models.mdx"
+
+<Tabs>
+<TabItem value="llms" label="LLMs" default>
+
+<MemoryLLM/>
+</TabItem>
+<TabItem value="chat_models" label="Chat models">
+
+You can use Memory with chains and agents initialized with chat models. The main difference between this and Memory for LLMs is that rather than trying to condense all previous messages into a string, we can keep them as their own unique memory object.
+
+<MemoryChatModel/>
+
+</TabItem>
+</Tabs>

docs/docs_skeleton/docs/guides/evaluation/comparison/index.mdx

@@ -1,8 +0,0 @@
----
-sidebar_position: 3 
----
-# Comparison Evaluators
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />

docs/docs_skeleton/docs/guides/evaluation/examples/index.mdx

@@ -1,12 +0,0 @@
----
-sidebar_position: 5
----
-# Examples
-
-🚧 _Docs under construction_ 🚧
-
-Below are some examples for inspecting and checking different chains.
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />

docs/docs_skeleton/docs/guides/evaluation/index.mdx

@@ -1,28 +0,0 @@
----
-sidebar_position: 6
----
-
-import DocCardList from "@theme/DocCardList";
-
-# Evaluation
-
-Language models can be unpredictable. This makes it challenging to ship reliable applications to production, where repeatable, useful outcomes across diverse inputs are a minimum requirement. Tests help demonstrate each component in an LLM application can produce the required or expected functionality. These tests also safeguard against regressions while you improve interconnected pieces of an integrated system. However, measuring the quality of generated text can be challenging. It can be hard to agree on the right set of metrics for your application, and it can be difficult to translate those into better performance. Furthermore, it's common to lack sufficient evaluation data to adequately test the range of inputs and expected outputs for each component when you're just getting started. The LangChain community is building open source tools and guides to help address these challenges.
-
-LangChain exposes different types of evaluators for common types of evaluation. Each type has off-the-shelf implementations you can use to get started, as well as an
- extensible API so you can create your own or contribute improvements for everyone to use. The following sections have example notebooks for you to get started.
-
-- [String Evaluators](/docs/modules/evaluation/string/): Evaluate the predicted string for a given input, usually against a reference string
-- [Trajectory Evaluators](/docs/modules/evaluation/trajectory/): Evaluate the whole trajectory of agent actions
-- [Comparison Evaluators](/docs/modules/evaluation/comparison/): Compare predictions from two runs on a common input
-
-
-This section also provides some additional examples of how you could use these evaluators for different scenarios or apply to different chain implementations in the LangChain library. Some examples include:
-
-- [Preference Scoring Chain Outputs](/docs/modules/evaluation/examples/comparisons): An example using a comparison evaluator on different models or prompts to select statistically significant differences in aggregate preference scores
-
-
-## Reference Docs
-
-For detailed information of the available evaluators, including how to instantiate, configure, and customize them. Check out the [reference documentation](https://api.python.langchain.com/en/latest/api_reference.html#module-langchain.evaluation) directly.
-
-<DocCardList />

docs/docs_skeleton/docs/guides/evaluation/string/index.mdx

@@ -1,8 +0,0 @@
----
-sidebar_position: 2 
----
-# String Evaluators
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />

docs/docs_skeleton/docs/guides/evaluation/trajectory/index.mdx

@@ -1,8 +0,0 @@
----
-sidebar_position: 4
----
-# Trajectory Evaluators
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />

docs/docs_skeleton/docs/guides/langsmith/index.md

@@ -1,12 +0,0 @@
-# LangSmith
-
-import DocCardList from "@theme/DocCardList";
-
-LangSmith helps you trace and evaluate your language model applications and intelligent agents to help you
-move from prototype to production.
-
-Check out the [interactive walkthrough](walkthrough) below to get started.
-
-For more information, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/)
-
-<DocCardList />

docs/docs_skeleton/docs/modules/agents/index.mdx

@@ -3,80 +3,46 @@ sidebar_position: 4
 ---
 # Agents
 
-The core idea of agents is to use an LLM to choose a sequence of actions to take.
-In chains, a sequence of actions is hardcoded (in code).
-In agents, a language model is used as a reasoning engine to determine which actions to take and in which order.
+Some applications require a flexible chain of calls to LLMs and other tools based on user input. The **Agent** interface provides the flexibility for such applications. An agent has access to a suite of tools, and determines which ones to use depending on the user input. Agents can use multiple tools, and use the output of one tool as the input to the next.
 
-There are several key components here:
+There are two main types of agents:
 
-## Agent
+- **Action agents**: at each timestep, decide on the next action using the outputs of all previous actions
+- **Plan-and-execute agents**: decide on the full sequence of actions up front, then execute them all without updating the plan
 
-This is the class responsible for deciding what step to take next.
-This is powered by a language model and a prompt.
-This prompt can include things like:
+Action agents are suitable for small tasks, while plan-and-execute agents are better for complex or long-running tasks that require maintaining long-term objectives and focus. Often the best approach is to combine the dynamism of an action agent with the planning abilities of a plan-and-execute agent by letting the plan-and-execute agent use action agents to execute plans.
 
-1. The personality of the agent (useful for having it respond in a certain way)
-2. Background context for the agent (useful for giving it more context on the types of tasks it's being asked to do)
-3. Prompting strategies to invoke better reasoning (the most famous/widely used being [ReAct](https://arxiv.org/abs/2210.03629))
+For a full list of agent types see [agent types](/docs/modules/agents/agent_types/). Additional abstractions involved in agents are:
+- [**Tools**](/docs/modules/agents/tools/): the actions an agent can take. What tools you give an agent highly depend on what you want the agent to do
+- [**Toolkits**](/docs/modules/agents/toolkits/): wrappers around collections of tools that can be used together a specific use case. For example, in order for an agent to
+  interact with a SQL database it will likely need one tool to execute queries and another to inspect tables
 
-LangChain provides a few different types of agents to get started.
-Even then, you will likely want to customize those agents with parts (1) and (2).
-For a full list of agent types see [agent types](/docs/modules/agents/agent_types/)
+## Action agents
 
-## Tools
+At a high-level an action agent:
+1. Receives user input
+2. Decides which tool, if any, to use and the tool input
+3. Calls the tool and records the output (also known as an "observation")
+4. Decides the next step using the history of tools, tool inputs, and observations
+5. Repeats 3-4 until it determines it can respond directly to the user
 
-Tools are functions that an agent calls.
-There are two important considerations here:
+Action agents are wrapped in **agent executors**, which are responsible for calling the agent, getting back an action and action input, calling the tool that the action references with the generated input, getting the output of the tool, and then passing all that information back into the agent to get the next action it should take.
 
-1. Giving the agent access to the right tools
-2. Describing the tools in a way that is most helpful to the agent
+Although an agent can be constructed in many ways, it typically involves these components:
 
-Without both, the agent you are trying to build will not work.
-If you don't give the agent access to a correct set of tools, it will never be able to accomplish the objective.
-If you don't describe the tools properly, the agent won't know how to properly use them.
+- **Prompt template**: Responsible for taking the user input and previous steps and constructing a prompt
+  to send to the language model
+- **Language model**: Takes the prompt with use input and action history and decides what to do next
+- **Output parser**: Takes the output of the language model and parses it into the next action or a final answer
 
-LangChain provides a wide set of tools to get started, but also makes it easy to define your own (including custom descriptions).
-For a full list of tools, see [here](/docs/modules/agents/tools/)
+## Plan-and-execute agents
 
-## Toolkits
+At a high-level a plan-and-execute agent:
+1. Receives user input
+2. Plans the full sequence of steps to take
+3. Executes the steps in order, passing the outputs of past steps as inputs to future steps
 
-Often the set of tools an agent has access to is more important than a single tool.
-For this LangChain provides the concept of toolkits - groups of tools needed to accomplish specific objectives.
-There are generally around 3-5 tools in a toolkit.
-
-LangChain provides a wide set of toolkits to get started.
-For a full list of toolkits, see [here](/docs/modules/agents/toolkits/)
-
-## AgentExecutor
-
-The agent executor is the runtime for an agent.
-This is what actually calls the agent and executes the actions it chooses.
-Pseudocode for this runtime is below:
-
-```python
-next_action = agent.get_action(...)
-while next_action != AgentFinish:
-    observation = run(next_action)
-    next_action = agent.get_action(..., next_action, observation)
-return next_action
-```
-
-While this may seem simple, there are several complexities this runtime handles for you, including:
-
-1. Handling cases where the agent selects a non-existent tool
-2. Handling cases where the tool errors
-3. Handling cases where the agent produces output that cannot be parsed into a tool invocation
-4. Logging and observability at all levels (agent decisions, tool calls) either to stdout or [LangSmith](https://smith.langchain.com).
-
-## Other types of agent runtimes
-
-The `AgentExecutor` class is the main agent runtime supported by LangChain.
-However, there are other, more experimental runtimes we also support.
-These include:
-
-- [Plan-and-execute Agent](/docs/modules/agents/agent_types/plan_and_execute.html)
-- [Baby AGI](/docs/use_cases/autonomous_agents/baby_agi.html)
-- [Auto GPT](/docs/use_cases/autonomous_agents/autogpt.html)
+The most typical implementation is to have the planner be a language model, and the executor be an action agent. Read more [here](/docs/modules/agents/agent_types/plan_and_execute.html).
 
 ## Get started
 

docs/docs_skeleton/docs/modules/agents/toolkits/index.mdx

@@ -3,8 +3,8 @@ sidebar_position: 3
 ---
 # Toolkits
 
-:::info
-Head to [Integrations](/docs/integrations/toolkits/) for documentation on built-in toolkit integrations.
-:::
-
 Toolkits are collections of tools that are designed to be used together for specific tasks and have convenience loading methods.
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/agents/tools/how_to/_category_.yml

@@ -0,0 +1,2 @@
+label: 'How-to'
+position: 0

docs/docs_skeleton/docs/modules/agents/tools/index.mdx

@@ -3,10 +3,6 @@ sidebar_position: 2
 ---
 # Tools
 
-:::info
-Head to [Integrations](/docs/integrations/tools/) for documentation on built-in tool integrations.
-:::
-
 Tools are interfaces that an agent can use to interact with the world.
 
 ## Get started

docs/docs_skeleton/docs/modules/agents/tools/integrations/_category_.yml

@@ -0,0 +1 @@
+label: 'Integrations'

docs/docs_skeleton/docs/modules/callbacks/how_to/_category_.yml

@@ -0,0 +1,2 @@
+label: 'How-to'
+position: 0

docs/docs_skeleton/docs/modules/callbacks/index.mdx

@@ -3,10 +3,6 @@ sidebar_position: 5
 ---
 # Callbacks
 
-:::info
-Head to [Integrations](/docs/integrations/callbacks/) for documentation on built-in callbacks integrations with 3rd-party tools.
-:::
-
 LangChain provides a callbacks system that allows you to hook into the various stages of your LLM application. This is useful for logging, monitoring, streaming, and other tasks.
 
 import GetStarted from "@snippets/modules/callbacks/get_started.mdx"

docs/docs_skeleton/docs/modules/callbacks/integrations/_category_.yml

@@ -0,0 +1 @@
+label: 'Integrations'

docs/docs_skeleton/docs/modules/data_connection/document_loaders/how_to/_category_.yml

@@ -0,0 +1,2 @@
+label: 'How-to'
+position: 0

docs/docs_skeleton/docs/modules/data_connection/document_loaders/how_to/json.mdx

@@ -2,8 +2,6 @@
 
 >[JSON (JavaScript Object Notation)](https://en.wikipedia.org/wiki/JSON) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other serializable values).
 
->[JSON Lines](https://jsonlines.org/) is a file format where each line is a valid JSON value.
-
 import Example from "@snippets/modules/data_connection/document_loaders/how_to/json.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/data_connection/document_loaders/index.mdx

@@ -3,10 +3,6 @@ sidebar_position: 0
 ---
 # Document loaders
 
-:::info
-Head to [Integrations](/docs/integrations/document_loaders/) for documentation on built-in document loader integrations with 3rd-party tools.
-:::
-
 Use document loaders to load data from a source as `Document`'s. A `Document` is a piece of text
 and associated metadata. For example, there are document loaders for loading a simple `.txt` file, for loading the text
 contents of any web page, or even for loading a transcript of a YouTube video.

docs/docs_skeleton/docs/modules/data_connection/document_loaders/integrations/_category_.yml

@@ -0,0 +1 @@
+label: 'Integrations'

docs/docs_skeleton/docs/modules/data_connection/document_transformers/index.mdx

@@ -3,10 +3,6 @@ sidebar_position: 1
 ---
 # Document transformers
 
-:::info
-Head to [Integrations](/docs/integrations/document_transformers/) for documentation on built-in document transformer integrations with 3rd-party tools.
-:::
-
 Once you've loaded documents, you'll often want to transform them to better suit your application. The simplest example
 is you may want to split a long document into smaller chunks that can fit into your model's context window. LangChain
 has a number of built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.
@@ -28,7 +24,7 @@ That means there are two different axes along which you can customize your text
 1. How the text is split
 2. How the chunk size is measured
 
-### Get started with text splitters
+## Get started with text splitters
 
 import GetStarted from "@snippets/modules/data_connection/document_transformers/get_started.mdx"
 

docs/docs_skeleton/docs/modules/data_connection/document_transformers/text_splitters/_category_.yml

@@ -1,2 +1 @@
 label: 'Text splitters'
-position: 0

docs/docs_skeleton/docs/modules/data_connection/index.mdx

@@ -8,7 +8,7 @@ Many LLM applications require user-specific data that is not part of the model's
 building blocks to load, transform, store and query your data via:
 
 - [Document loaders](/docs/modules/data_connection/document_loaders/): Load documents from many different sources
-- [Document transformers](/docs/modules/data_connection/document_transformers/): Split documents, convert documents into Q&A format, drop redundant documents, and more
+- [Document transformers](/docs/modules/data_connection/document_transformers/): Split documents, drop redundant documents, and more
 - [Text embedding models](/docs/modules/data_connection/text_embedding/): Take unstructured text and turn it into a list of floating point numbers
 - [Vector stores](/docs/modules/data_connection/vectorstores/): Store and search over embedded data
 - [Retrievers](/docs/modules/data_connection/retrievers/): Query your data

docs/docs_skeleton/docs/modules/data_connection/retrievers/how_to/_category_.yml

@@ -0,0 +1,2 @@
+label: 'How-to'
+position: 0

docs/docs_skeleton/docs/modules/data_connection/retrievers/index.mdx

@@ -3,10 +3,6 @@ sidebar_position: 4
 ---
 # Retrievers
 
-:::info
-Head to [Integrations](/docs/integrations/retrievers/) for documentation on built-in retriever integrations with 3rd-party tools.
-:::
-
 A retriever is an interface that returns documents given an unstructured query. It is more general than a vector store.
 A retriever does not need to be able to store documents, only to return (or retrieve) it. Vector stores can be used
 as the backbone of a retriever, but there are other types of retrievers as well.