Merge branch 'master' into erick/skip-release-check-cli

.github/CONTRIBUTING.md

@@ -3,17 +3,31 @@
 Hi there! Thank you for even being interested in contributing to LangChain.
 As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes.
 
-To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/)
-
 ## 🗺️ Guidelines
 
-### 👩‍💻 Ways to contribute
+### 👩‍💻 Contributing Code
 
-There are many ways to contribute to LangChain. Here are some common ways people contribute:
+To contribute to this project, please follow the ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
+Please do not try to push directly to this repo unless you are a maintainer.
 
-- [**Documentation**](https://python.langchain.com/docs/contributing/documentation): Help improve our docs, including this one!
-- [**Code**](https://python.langchain.com/docs/contributing/code): Help us write code, fix bugs, or improve our infrastructure.
-- [**Integrations**](https://python.langchain.com/docs/contributing/integration): Help us integrate with your favorite vendors and tools.
+Please follow the checked-in pull request template when opening pull requests. Note related issues and tag relevant
+maintainers.
+
+Pull requests cannot land without passing the formatting, linting, and testing checks first. See [Testing](#testing) and
+[Formatting and Linting](#formatting-and-linting) for how to run these checks locally.
+
+It's essential that we maintain great documentation and testing. If you:
+- Fix a bug
+  - Add a relevant unit or integration test when possible. These live in `tests/unit_tests` and `tests/integration_tests`.
+- Make an improvement
+  - Update any affected example notebooks and documentation. These live in `docs`.
+  - Update unit and integration tests when relevant.
+- Add a feature
+  - Add a demo notebook in `docs/modules`.
+  - Add unit and integration tests.
+
+We are a small, progress-oriented team. If there's something you'd like to add or change, opening a pull request is the
+best way to get our attention.
 
 ### 🚩GitHub Issues
 
@@ -40,6 +54,268 @@ In a similar vein, we do enforce certain linting, formatting, and documentation
 If you are finding these difficult (or even just annoying) to work with, feel free to contact a maintainer for help -
 we do not want these to get in the way of getting good code into the codebase.
 
-### Contributor Documentation
+## 🚀 Quick Start
 
-To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/)
+This quick start guide explains how to run the repository locally.
+For a [development container](https://containers.dev/), see the [.devcontainer folder](https://github.com/langchain-ai/langchain/tree/master/.devcontainer).
+
+### Dependency Management: Poetry and other env/dependency managers
+
+This project utilizes [Poetry](https://python-poetry.org/) v1.6.1+ as a dependency manager.
+
+❗Note: *Before installing Poetry*, if you use `Conda`, create and activate a new Conda env (e.g. `conda create -n langchain python=3.9`)
+
+Install Poetry: **[documentation on how to install it](https://python-poetry.org/docs/#installation)**.
+
+❗Note: If you use `Conda` or `Pyenv` as your environment/package manager, after installing Poetry,
+tell Poetry to use the virtualenv python environment (`poetry config virtualenvs.prefer-active-python true`)
+
+### Core vs. Experimental
+
+This repository contains two separate projects:
+- `langchain`: core langchain code, abstractions, and use cases.
+- `langchain.experimental`: see the [Experimental README](https://github.com/langchain-ai/langchain/tree/master/libs/experimental/README.md) for more information.
+
+Each of these has its own development environment. Docs are run from the top-level makefile, but development
+is split across separate test & release flows.
+
+For this quickstart, start with langchain core:
+
+```bash
+cd libs/langchain
+```
+
+### Local Development Dependencies
+
+Install langchain development requirements (for running langchain, running examples, linting, formatting, tests, and coverage):
+
+```bash
+poetry install --with test
+```
+
+Then verify dependency installation:
+
+```bash
+make test
+```
+
+If the tests don't pass, you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
+
+If during installation you receive a `WheelFileValidationError` for `debugpy`, please make sure you are running
+Poetry v1.6.1+. This bug was present in older versions of Poetry (e.g. 1.4.1) and has been resolved in newer releases.
+If you are still seeing this bug on v1.6.1, you may also try disabling "modern installation"
+(`poetry config installer.modern-installation false`) and re-installing requirements.
+See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details.
+
+### Testing
+
+_some test dependencies are optional; see section about optional dependencies_.
+
+Unit tests cover modular logic that does not require calls to outside APIs.
+If you add new logic, please add a unit test.
+
+To run unit tests:
+
+```bash
+make test
+```
+
+To run unit tests in Docker:
+
+```bash
+make docker_tests
+```
+
+There are also [integration tests and code-coverage](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests/README.md) available.
+
+### Formatting and Linting
+
+Run these locally before submitting a PR; the CI system will check also.
+
+#### Code Formatting
+
+Formatting for this project is done via [ruff](https://docs.astral.sh/ruff/rules/).
+
+To run formatting for docs, cookbook and templates:
+
+```bash
+make format
+```
+
+To run formatting for a library, run the same command from the relevant library directory:
+
+```bash
+cd libs/{LIBRARY}
+make format
+```
+
+Additionally, you can run the formatter only on the files that have been modified in your current branch as compared to the master branch using the format_diff command:
+
+```bash
+make format_diff
+```
+
+This is especially useful when you have made changes to a subset of the project and want to ensure your changes are properly formatted without affecting the rest of the codebase.
+
+#### Linting
+
+Linting for this project is done via a combination of [ruff](https://docs.astral.sh/ruff/rules/) and [mypy](http://mypy-lang.org/).
+
+To run linting for docs, cookbook and templates:
+
+```bash
+make lint
+```
+
+To run linting for a library, run the same command from the relevant library directory:
+
+```bash
+cd libs/{LIBRARY}
+make lint
+```
+
+In addition, you can run the linter only on the files that have been modified in your current branch as compared to the master branch using the lint_diff command:
+
+```bash
+make lint_diff
+```
+
+This can be very helpful when you've made changes to only certain parts of the project and want to ensure your changes meet the linting standards without having to check the entire codebase.
+
+We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
+
+#### Spellcheck
+
+Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell).
+Note that `codespell` finds common typos, so it could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
+
+To check spelling for this project:
+
+```bash
+make spell_check
+```
+
+To fix spelling in place:
+
+```bash
+make spell_fix
+```
+
+If codespell is incorrectly flagging a word, you can skip spellcheck for that word by adding it to the codespell config in the `pyproject.toml` file.
+
+```python
+[tool.codespell]
+...
+# Add here:
+ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure'
+```
+
+## Working with Optional Dependencies
+
+Langchain relies heavily on optional dependencies to keep the Langchain package lightweight.
+
+If you're adding a new dependency to Langchain, assume that it will be an optional dependency, and
+that most users won't have it installed.
+
+Users who do not have the dependency installed should be able to **import** your code without
+any side effects (no warnings, no errors, no exceptions).
+
+To introduce the dependency to the pyproject.toml file correctly, please do the following:
+
+1. Add the dependency to the main group as an optional dependency
+  ```bash
+  poetry add --optional [package_name]
+  ```
+2. Open pyproject.toml and add the dependency to the `extended_testing` extra
+3. Relock the poetry file to update the extra.
+  ```bash
+  poetry lock --no-update
+  ```
+4. Add a unit test that the very least attempts to import the new code. Ideally, the unit
+test makes use of lightweight fixtures to test the logic of the code.
+5. Please use the `@pytest.mark.requires(package_name)` decorator for any tests that require the dependency.
+
+## Adding a Jupyter Notebook
+
+If you are adding a Jupyter Notebook example, you'll want to install the optional `dev` dependencies.
+
+To install dev dependencies:
+
+```bash
+poetry install --with dev
+```
+
+Launch a notebook:
+
+```bash
+poetry run jupyter notebook
+```
+
+When you run `poetry install`, the `langchain` package is installed as editable in the virtualenv, so your new logic can be imported into the notebook.
+
+## Documentation
+
+While the code is split between `langchain` and `langchain.experimental`, the documentation is one holistic thing.
+This covers how to get started contributing to documentation.
+
+From the top-level of this repo, install documentation dependencies:
+
+```bash
+poetry install
+```
+
+### Contribute Documentation
+
+The docs directory contains Documentation and API Reference.
+
+Documentation is built using [Docusaurus 2](https://docusaurus.io/).
+
+API Reference are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
+For that reason, we ask that you add good documentation to all classes and methods.
+
+Similar to linting, we recognize documentation can be annoying. If you do not want to do it, please contact a project maintainer, and they can help you with it. We do not want this to be a blocker for good code getting contributed.
+
+### Build Documentation Locally
+
+In the following commands, the prefix `api_` indicates that those are operations for the API Reference.
+
+Before building the documentation, it is always a good idea to clean the build directory:
+
+```bash
+make docs_clean
+make api_docs_clean
+```
+
+Next, you can build the documentation as outlined below:
+
+```bash
+make docs_build
+make api_docs_build
+```
+
+Finally, run the link checker to ensure all links are valid:
+
+```bash
+make docs_linkcheck
+make api_docs_linkcheck
+```
+
+### Verify Documentation changes
+
+After pushing documentation changes to the repository, you can preview and verify that the changes are
+what you wanted by clicking the `View deployment` or `Visit Preview` buttons on the pull request `Conversation` page.
+This will take you to a preview of the documentation changes.
+This preview is created by [Vercel](https://vercel.com/docs/getting-started-with-vercel).
+
+## 🏭 Release Process
+
+As of now, LangChain has an ad hoc release process: releases are cut with high frequency by
+a developer and published to [PyPI](https://pypi.org/project/langchain/).
+
+LangChain follows the [semver](https://semver.org/) versioning standard. However, as pre-1.0 software,
+even patch releases may contain [non-backwards-compatible changes](https://semver.org/#spec-item-4).
+
+### 🌟 Recognition
+
+If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)!
+If you have a Twitter account you would like us to mention, please let us know in the PR or through another means.

.github/ISSUE_TEMPLATE/feature-request.yml

@@ -27,4 +27,4 @@ body:
     attributes:
       label: Your contribution
       description: |
-        Is there any way that you could help, e.g. by submitting a PR? Make sure to read the [Contributing Guide](https://python.langchain.com/docs/contributing/)
+        Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md)

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,20 +1,20 @@
 <!-- Thank you for contributing to LangChain!
 
-Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.
-
 Replace this entire comment with:
   - **Description:** a description of the change, 
-  - **Issue:** the issue # it fixes if applicable,
+  - **Issue:** the issue # it fixes (if applicable),
   - **Dependencies:** any dependencies required for this change,
+  - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below),
   - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!
 
-Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.
+Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally.
 
-See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/
+See contribution guidelines for more information on how to write/run tests, lint, etc: 
+https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
 
 If you're adding a new integration, please include:
   1. a test for the integration, preferably unit tests that do not rely on network access,
-  2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.
+  2. an example notebook showing its use. It lives in `docs/extras` directory.
 
 If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
  -->

.github/actions/poetry_setup/action.yml

@@ -26,7 +26,7 @@ inputs:
 runs:
   using: composite
   steps:
-    - uses: actions/setup-python@v5
+    - uses: actions/setup-python@v4
       name: Setup python ${{ inputs.python-version }}
       with:
         python-version: ${{ inputs.python-version }}

.github/scripts/check_diff.py

@@ -1,56 +0,0 @@
-import json
-import sys
-import os
-
-LANGCHAIN_DIRS = {
-    "libs/core",
-    "libs/langchain",
-    "libs/experimental",
-    "libs/community",
-}
-
-if __name__ == "__main__":
-    files = sys.argv[1:]
-    dirs_to_run = set()
-
-    if len(files) == 300:
-        # max diff length is 300 files - there are likely files missing
-        raise ValueError("Max diff reached. Please manually run CI on changed libs.")
-
-    for file in files:
-        if any(
-            file.startswith(dir_)
-            for dir_ in (
-                ".github/workflows",
-                ".github/tools",
-                ".github/actions",
-                "libs/core",
-                ".github/scripts/check_diff.py",
-            )
-        ):
-            dirs_to_run.update(LANGCHAIN_DIRS)
-        elif "libs/community" in file:
-            dirs_to_run.update(
-                ("libs/community", "libs/langchain", "libs/experimental")
-            )
-        elif "libs/partners" in file:
-            partner_dir = file.split("/")[2]
-            if os.path.isdir(f"libs/partners/{partner_dir}"):
-                dirs_to_run.update(
-                    (
-                        f"libs/partners/{partner_dir}",
-                        "libs/langchain",
-                        "libs/experimental",
-                    )
-                )
-            # Skip if the directory was deleted
-        elif "libs/langchain" in file:
-            dirs_to_run.update(("libs/langchain", "libs/experimental"))
-        elif "libs/experimental" in file:
-            dirs_to_run.add("libs/experimental")
-        elif file.startswith("libs/"):
-            dirs_to_run.update(LANGCHAIN_DIRS)
-        else:
-            pass
-    json_output = json.dumps(list(dirs_to_run))
-    print(f"dirs-to-run={json_output}")

.github/workflows/_compile_integration_test.yml

@@ -9,7 +9,7 @@ on:
         description: "From which folder this pipeline executes"
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
 
 jobs:
   build:
@@ -24,7 +24,6 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
-          - "3.12"
     name: Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v4
@@ -39,7 +38,7 @@ jobs:
 
       - name: Install integration dependencies
         shell: bash
-        run: poetry install --with=test_integration,test
+        run: poetry install --with=test_integration
 
       - name: Check integration tests compile
         shell: bash

.github/workflows/_integration_test.yml

@@ -1,60 +0,0 @@
-name: Integration tests
-
-on:
-  workflow_dispatch:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-
-env:
-  POETRY_VERSION: "1.7.1"
-
-jobs:
-  build:
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.12"
-    name: Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: core
-
-      - name: Install dependencies
-        shell: bash
-        run: poetry install --with test,test_integration
-
-      - name: Run integration tests
-        shell: bash
-        env:
-          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
-          TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
-        run: |
-          make integration_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/_lint.yml

@@ -13,7 +13,7 @@ on:
         description: "Relative path to the langchain library folder"
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
   WORKDIR: ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}
 
   # This env var allows us to get inline annotations when ruff has complaints.
@@ -33,7 +33,7 @@ jobs:
         # so linting on fewer versions makes CI faster.
         python-version:
           - "3.8"
-          - "3.12"
+          - "3.11"
     steps:
       - uses: actions/checkout@v4
 
@@ -68,7 +68,7 @@ jobs:
         # It doesn't matter how you change it, any change will cause a cache-bust.
         working-directory: ${{ inputs.working-directory }}
         run: |
-          poetry install --with lint,typing
+          poetry install --with dev,lint,test,typing
 
       - name: Install langchain editable
         working-directory: ${{ inputs.working-directory }}
@@ -76,7 +76,7 @@ jobs:
         env:
           LANGCHAIN_LOCATION: ${{ inputs.langchain-location }}
         run: |
-          poetry run pip install -e "$LANGCHAIN_LOCATION"
+          pip install -e "$LANGCHAIN_LOCATION"
 
       - name: Get .mypy_cache to speed up mypy
         uses: actions/cache@v3
@@ -85,37 +85,9 @@ jobs:
         with:
           path: |
             ${{ env.WORKDIR }}/.mypy_cache
-          key: mypy-lint-${{ runner.os }}-${{ runner.arch }}-py${{ matrix.python-version }}-${{ inputs.working-directory }}-${{ hashFiles(format('{0}/poetry.lock', env.WORKDIR)) }}
-
+          key: mypy-${{ runner.os }}-${{ runner.arch }}-py${{ matrix.python-version }}-${{ inputs.working-directory }}-${{ hashFiles(format('{0}/poetry.lock', env.WORKDIR)) }}
 
       - name: Analysing the code with our lint
         working-directory: ${{ inputs.working-directory }}
         run: |
-          make lint_package
-
-      - name: Install test dependencies
-        # Also installs dev/lint/test/typing dependencies, to ensure we have
-        # type hints for as many of our libraries as possible.
-        # This helps catch errors that require dependencies to be spotted, for example:
-        # https://github.com/langchain-ai/langchain/pull/10249/files#diff-935185cd488d015f026dcd9e19616ff62863e8cde8c0bee70318d3ccbca98341
-        #
-        # If you change this configuration, make sure to change the `cache-key`
-        # in the `poetry_setup` action above to stop using the old cache.
-        # It doesn't matter how you change it, any change will cause a cache-bust.
-        working-directory: ${{ inputs.working-directory }}
-        run: |
-          poetry install --with test
-
-      - name: Get .mypy_cache_test to speed up mypy
-        uses: actions/cache@v3
-        env:
-          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "2"
-        with:
-          path: |
-            ${{ env.WORKDIR }}/.mypy_cache_test
-          key: mypy-test-${{ runner.os }}-${{ runner.arch }}-py${{ matrix.python-version }}-${{ inputs.working-directory }}-${{ hashFiles(format('{0}/poetry.lock', env.WORKDIR)) }}
-
-      - name: Analysing the code with our lint
-        working-directory: ${{ inputs.working-directory }}
-        run: |
-          make lint_tests
+          make lint

.github/workflows/_pydantic_compatibility.yml

@@ -1,4 +1,4 @@
-name: dependencies
+name: pydantic v1/v2 compatibility
 
 on:
   workflow_call:
@@ -7,13 +7,9 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
-      langchain-location:
-        required: false
-        type: string
-        description: "Relative path to the langchain library folder"
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
 
 jobs:
   build:
@@ -28,8 +24,7 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
-          - "3.12"
-    name: dependencies - Python ${{ matrix.python-version }}
+    name: Pydantic v1/v2 compatibility - Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v4
 
@@ -45,22 +40,6 @@ jobs:
         shell: bash
         run: poetry install
 
-      - name: Check imports with base dependencies
-        shell: bash
-        run: poetry run make check_imports
-
-      - name: Install test dependencies
-        shell: bash
-        run: poetry install --with test
-
-      - name: Install langchain editable
-        working-directory: ${{ inputs.working-directory }}
-        if: ${{ inputs.langchain-location }}
-        env:
-          LANGCHAIN_LOCATION: ${{ inputs.langchain-location }}
-        run: |
-          poetry run pip install -e "$LANGCHAIN_LOCATION"
-
       - name: Install the opposite major version of pydantic
         # If normal tests use pydantic v1, here we'll use v2, and vice versa.
         shell: bash

.github/workflows/_release.yml

@@ -7,20 +7,14 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
-  workflow_dispatch:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        default: 'libs/langchain'
 
 env:
   PYTHON_VERSION: "3.10"
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
 
 jobs:
   build:
-    if: github.ref == 'refs/heads/master'
+    # if: github.ref == 'refs/heads/master'
     runs-on: ubuntu-latest
 
     outputs:
@@ -76,101 +70,59 @@ jobs:
       working-directory: ${{ inputs.working-directory }}
     secrets: inherit
 
-  pre-release-checks:
-    needs:
-      - build
-      - test-pypi-publish
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
+  # pre-release-checks:
+  #   needs:
+  #     - build
+  #     - test-pypi-publish
+  #   runs-on: ubuntu-latest
+  #   steps:
+  #     # We explicitly *don't* set up caching here. This ensures our tests are
+  #     # maximally sensitive to catching breakage.
+  #     #
+  #     # For example, here's a way that caching can cause a falsely-passing test:
+  #     # - Make the langchain package manifest no longer list a dependency package
+  #     #   as a requirement. This means it won't be installed by `pip install`,
+  #     #   and attempting to use it would cause a crash.
+  #     # - That dependency used to be required, so it may have been cached.
+  #     #   When restoring the venv packages from cache, that dependency gets included.
+  #     # - Tests pass, because the dependency is present even though it wasn't specified.
+  #     # - The package is published, and it breaks on the missing dependency when
+  #     #   used in the real world.
+  #     - uses: actions/setup-python@v4
+  #       with:
+  #         python-version: ${{ env.PYTHON_VERSION }}
 
-      # We explicitly *don't* set up caching here. This ensures our tests are
-      # maximally sensitive to catching breakage.
-      #
-      # For example, here's a way that caching can cause a falsely-passing test:
-      # - Make the langchain package manifest no longer list a dependency package
-      #   as a requirement. This means it won't be installed by `pip install`,
-      #   and attempting to use it would cause a crash.
-      # - That dependency used to be required, so it may have been cached.
-      #   When restoring the venv packages from cache, that dependency gets included.
-      # - Tests pass, because the dependency is present even though it wasn't specified.
-      # - The package is published, and it breaks on the missing dependency when
-      #   used in the real world.
+  #     - name: Test published package
+  #       shell: bash
+  #       env:
+  #         PKG_NAME: ${{ needs.build.outputs.pkg-name }}
+  #         VERSION: ${{ needs.build.outputs.version }}
+  #       # Here we specify:
+  #       # - The test PyPI index as the *primary* index, meaning that it takes priority.
+  #       # - The regular PyPI index as an extra index, so that any dependencies that
+  #       #   are not found on test PyPI can be resolved and installed anyway.
+  #       #
+  #       # Without the former, we might install the wrong langchain release.
+  #       # Without the latter, we might not be able to install langchain's dependencies.
+  #       #
+  #       # TODO: add more in-depth pre-publish tests after testing that importing works
+  #       run: |
+  #         pip install \
+  #           --index-url https://test.pypi.org/simple/ \
+  #           --extra-index-url https://pypi.org/simple/ \
+  #           "$PKG_NAME==$VERSION"
 
-      - name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
+  #         # Replace all dashes in the package name with underscores,
+  #         # since that's how Python imports packages with dashes in the name.
+  #         IMPORT_NAME="$(echo "$PKG_NAME" | sed s/-/_/g)"
 
-      - name: Import published package
-        shell: bash
-        working-directory: ${{ inputs.working-directory }}
-        env:
-          PKG_NAME: ${{ needs.build.outputs.pkg-name }}
-          VERSION: ${{ needs.build.outputs.version }}
-        # Here we use:
-        # - The default regular PyPI index as the *primary* index, meaning 
-        #   that it takes priority (https://pypi.org/simple)
-        # - The test PyPI index as an extra index, so that any dependencies that
-        #   are not found on test PyPI can be resolved and installed anyway.
-        #   (https://test.pypi.org/simple). This will include the PKG_NAME==VERSION
-        #   package because VERSION will not have been uploaded to regular PyPI yet.
-        #
-        run: |
-          poetry run pip install \
-            --extra-index-url https://test.pypi.org/simple/ \
-            "$PKG_NAME==$VERSION"
-
-          # Replace all dashes in the package name with underscores,
-          # since that's how Python imports packages with dashes in the name.
-          IMPORT_NAME="$(echo "$PKG_NAME" | sed s/-/_/g)"
-
-          poetry run python -c "import $IMPORT_NAME; print(dir($IMPORT_NAME))"
-
-      - name: Import test dependencies
-        run: poetry install --with test,test_integration
-        working-directory: ${{ inputs.working-directory }}
-
-      # Overwrite the local version of the package with the test PyPI version.
-      - name: Import published package (again)
-        working-directory: ${{ inputs.working-directory }}
-        shell: bash
-        env:
-          PKG_NAME: ${{ needs.build.outputs.pkg-name }}
-          VERSION: ${{ needs.build.outputs.version }}
-        run: |
-          poetry run pip install \
-            --extra-index-url https://test.pypi.org/simple/ \
-            "$PKG_NAME==$VERSION"
-
-      - name: Run unit tests
-        run: make tests
-        working-directory: ${{ inputs.working-directory }}
-
-      - name: Run integration tests
-        if: ${{ startsWith(inputs.working-directory, 'libs/partners/') }}
-        env:
-          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
-          TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
-        run: make integration_tests
-        working-directory: ${{ inputs.working-directory }}
-
-      - name: Run unit tests with minimum dependency versions
-        if: ${{ (inputs.working-directory == 'libs/langchain') || (inputs.working-directory == 'libs/community') || (inputs.working-directory == 'libs/experimental') }}
-        run: |
-          poetry run pip install -r _test_minimum_requirements.txt
-          make tests
-        working-directory: ${{ inputs.working-directory }}
+  #         python -c "import $IMPORT_NAME; print(dir($IMPORT_NAME))"
 
   publish:
     needs:
       - build
       - test-pypi-publish
-      - pre-release-checks
+      # - pre-release-checks
     runs-on: ubuntu-latest
     permissions:
       # This permission is used for trusted publishing:
@@ -211,7 +163,7 @@ jobs:
     needs:
       - build
       - test-pypi-publish
-      - pre-release-checks
+      # - pre-release-checks
       - publish
     runs-on: ubuntu-latest
     permissions:

.github/workflows/_test.yml

@@ -7,13 +7,9 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
-      langchain-location:
-        required: false
-        type: string
-        description: "Relative path to the langchain library folder"
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
 
 jobs:
   build:
@@ -28,7 +24,6 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
-          - "3.12"
     name: Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v4
@@ -43,20 +38,11 @@ jobs:
 
       - name: Install dependencies
         shell: bash
-        run: poetry install --with test
-
-      - name: Install langchain editable
-        working-directory: ${{ inputs.working-directory }}
-        if: ${{ inputs.langchain-location }}
-        env:
-          LANGCHAIN_LOCATION: ${{ inputs.langchain-location }}
-        run: |
-          poetry run pip install -e "$LANGCHAIN_LOCATION"
+        run: poetry install
 
       - name: Run core tests
         shell: bash
-        run: |
-          make test
+        run: make test
 
       - name: Ensure the tests did not create any additional files
         shell: bash

.github/workflows/_test_release.yml

@@ -9,12 +9,12 @@ on:
         description: "From which folder this pipeline executes"
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
   PYTHON_VERSION: "3.10"
 
 jobs:
   build:
-    if: github.ref == 'refs/heads/master'
+    # if: github.ref == 'refs/heads/master'
     runs-on: ubuntu-latest
 
     outputs:

.github/workflows/check_diffs.yml

@@ -1,43 +0,0 @@
----
-name: Check library diffs
-
-on:
-  push:
-    branches: [master]
-  pull_request:
-
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: '3.10'
-      - id: files
-        uses: Ana06/get-changed-files@v2.2.0
-      - id: set-matrix
-        run: |
-          python .github/scripts/check_diff.py ${{ steps.files.outputs.all }} >> $GITHUB_OUTPUT
-    outputs:
-      dirs-to-run: ${{ steps.set-matrix.outputs.dirs-to-run }}
-  ci:
-    needs: [ build ]
-    strategy:
-      matrix:
-        working-directory: ${{ fromJson(needs.build.outputs.dirs-to-run) }}
-    uses: ./.github/workflows/_all_ci.yml
-    with:
-      working-directory: ${{ matrix.working-directory }}
-
-

.github/workflows/langchain_ci.yml

@@ -1,25 +1,20 @@
 ---
-name: langchain CI
+name: libs/langchain CI
 
 on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-  workflow_dispatch:
-    inputs:
-      working-directory:
-        required: true
-        type: choice
-        default: 'libs/langchain'
-        options:
-        - libs/langchain
-        - libs/core
-        - libs/experimental
-        - libs/community
-
+  push:
+    branches: [ master ]
+  pull_request:
+    paths:
+      - '.github/actions/poetry_setup/action.yml'
+      - '.github/tools/**'
+      - '.github/workflows/_lint.yml'
+      - '.github/workflows/_test.yml'
+      - '.github/workflows/_pydantic_compatibility.yml'
+      - '.github/workflows/langchain_ci.yml'
+      - 'libs/*'
+      - 'libs/langchain/**'
+  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
 # If another push to the same PR or branch happens while this workflow is still running,
 # cancel the earlier run in favor of the next run.
@@ -28,39 +23,47 @@ on:
 # a limited number of job runners to be active at the same time, so it's better to cancel
 # pointless jobs early so that more useful jobs can run sooner.
 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.working-directory }}
+  group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
+  WORKDIR: "libs/langchain"
 
 jobs:
   lint:
-    uses: ./.github/workflows/_lint.yml
+    uses:
+      ./.github/workflows/_lint.yml
     with:
-      working-directory: ${{ inputs.working-directory }}
+      working-directory: libs/langchain
     secrets: inherit
 
   test:
-    uses: ./.github/workflows/_test.yml
+    uses:
+      ./.github/workflows/_test.yml
     with:
-      working-directory: ${{ inputs.working-directory }}
+      working-directory: libs/langchain
     secrets: inherit
 
   compile-integration-tests:
-    uses: ./.github/workflows/_compile_integration_test.yml
+    uses:
+      ./.github/workflows/_compile_integration_test.yml
     with:
-      working-directory: ${{ inputs.working-directory }}
+      working-directory: libs/langchain
     secrets: inherit
 
-  dependencies:
-    uses: ./.github/workflows/_dependencies.yml
+  pydantic-compatibility:
+    uses:
+      ./.github/workflows/_pydantic_compatibility.yml
     with:
-      working-directory: ${{ inputs.working-directory }}
+      working-directory: libs/langchain
     secrets: inherit
 
   extended-tests:
     runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ${{ env.WORKDIR }}
     strategy:
       matrix:
         python-version:
@@ -68,12 +71,7 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
-          - "3.12"
     name: Python ${{ matrix.python-version }} extended tests
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    if: ${{ ! startsWith(inputs.working-directory, 'libs/partners/') }}
     steps:
       - uses: actions/checkout@v4
 
@@ -82,14 +80,14 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
           poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
+          working-directory: libs/langchain
           cache-key: extended
 
       - name: Install dependencies
         shell: bash
         run: |
           echo "Running extended tests, installing dependencies with poetry..."
-          poetry install -E extended_testing --with test
+          poetry install -E extended_testing
 
       - name: Run extended tests
         run: make extended_tests

.github/workflows/langchain_cli_ci.yml

@@ -0,0 +1,47 @@
+---
+name: libs/cli CI
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    paths:
+      - '.github/actions/poetry_setup/action.yml'
+      - '.github/tools/**'
+      - '.github/workflows/_lint.yml'
+      - '.github/workflows/_test.yml'
+      - '.github/workflows/_pydantic_compatibility.yml'
+      - '.github/workflows/langchain_cli_ci.yml'
+      - 'libs/cli/**'
+      - 'libs/*'
+  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
+
+# If another push to the same PR or branch happens while this workflow is still running,
+# cancel the earlier run in favor of the next run.
+#
+# There's no point in testing an outdated version of the code. GitHub only allows
+# a limited number of job runners to be active at the same time, so it's better to cancel
+# pointless jobs early so that more useful jobs can run sooner.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  POETRY_VERSION: "1.6.1"
+  WORKDIR: "libs/cli"
+
+jobs:
+  lint:
+    uses:
+      ./.github/workflows/_lint.yml
+    with:
+      working-directory: libs/cli
+      langchain-location: ../langchain
+    secrets: inherit
+
+  test:
+    uses:
+      ./.github/workflows/_test.yml
+    with:
+      working-directory: libs/cli
+    secrets: inherit

.github/workflows/langchain_community_release.yml

@@ -1,13 +0,0 @@
----
-name: libs/community Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_release.yml
-    with:
-      working-directory: libs/community
-    secrets: inherit

.github/workflows/langchain_core_release.yml

@@ -1,13 +0,0 @@
----
-name: libs/core Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_release.yml
-    with:
-      working-directory: libs/core
-    secrets: inherit

.github/workflows/langchain_experimental_ci.yml

@@ -0,0 +1,137 @@
+---
+name: libs/experimental CI
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    paths:
+      - '.github/actions/poetry_setup/action.yml'
+      - '.github/tools/**'
+      - '.github/workflows/_lint.yml'
+      - '.github/workflows/_test.yml'
+      - '.github/workflows/langchain_experimental_ci.yml'
+      - 'libs/*'
+      - 'libs/experimental/**'
+  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
+
+# If another push to the same PR or branch happens while this workflow is still running,
+# cancel the earlier run in favor of the next run.
+#
+# There's no point in testing an outdated version of the code. GitHub only allows
+# a limited number of job runners to be active at the same time, so it's better to cancel
+# pointless jobs early so that more useful jobs can run sooner.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  POETRY_VERSION: "1.6.1"
+  WORKDIR: "libs/experimental"
+
+jobs:
+  lint:
+    uses:
+      ./.github/workflows/_lint.yml
+    with:
+      working-directory: libs/experimental
+      langchain-location: ../langchain
+    secrets: inherit
+
+  test:
+    uses:
+      ./.github/workflows/_test.yml
+    with:
+      working-directory: libs/experimental
+    secrets: inherit
+
+  compile-integration-tests:
+    uses:
+      ./.github/workflows/_compile_integration_test.yml
+    with:
+      working-directory: libs/experimental
+    secrets: inherit
+
+  # It's possible that langchain-experimental works fine with the latest *published* langchain,
+  # but is broken with the langchain on `master`.
+  #
+  # We want to catch situations like that *before* releasing a new langchain, hence this test.
+  test-with-latest-langchain:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ${{ env.WORKDIR }}
+    strategy:
+      matrix:
+        python-version:
+          - "3.8"
+          - "3.9"
+          - "3.10"
+          - "3.11"
+    name: test with unpublished langchain - Python ${{ matrix.python-version }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
+        uses: "./.github/actions/poetry_setup"
+        with:
+          python-version: ${{ matrix.python-version }}
+          poetry-version: ${{ env.POETRY_VERSION }}
+          working-directory: ${{ env.WORKDIR }}
+          cache-key: unpublished-langchain
+
+      - name: Install dependencies
+        shell: bash
+        run: |
+          echo "Running tests with unpublished langchain, installing dependencies with poetry..."
+          poetry install
+
+          echo "Editably installing langchain outside of poetry, to avoid messing up lockfile..."
+          poetry run pip install -e ../langchain
+
+      - name: Run tests
+        run: make test
+  extended-tests:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ${{ env.WORKDIR }}
+    strategy:
+      matrix:
+        python-version:
+          - "3.8"
+          - "3.9"
+          - "3.10"
+          - "3.11"
+    name: Python ${{ matrix.python-version }} extended tests
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
+        uses: "./.github/actions/poetry_setup"
+        with:
+          python-version: ${{ matrix.python-version }}
+          poetry-version: ${{ env.POETRY_VERSION }}
+          working-directory: libs/experimental
+          cache-key: extended
+
+      - name: Install dependencies
+        shell: bash
+        run: |
+          echo "Running extended tests, installing dependencies with poetry..."
+          poetry install -E extended_testing
+
+      - name: Run extended tests
+        run: make extended_tests
+
+      - name: Ensure the tests did not create any additional files
+        shell: bash
+        run: |
+          set -eu
+
+          STATUS="$(git status)"
+          echo "$STATUS"
+
+          # grep will exit non-zero if the target message isn't found,
+          # and `set -e` above will cause the step to fail.
+          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/langchain_openai_release.yml

@@ -1,13 +0,0 @@
----
-name: libs/core Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_release.yml
-    with:
-      working-directory: libs/core
-    secrets: inherit

.github/workflows/scheduled_test.yml

@@ -6,7 +6,7 @@ on:
     - cron:  '0 13 * * *'
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
 
 jobs:
   build:
@@ -19,7 +19,9 @@ jobs:
       matrix:
         python-version:
           - "3.8"
-          - "3.12"
+          - "3.9"
+          - "3.10"
+          - "3.11"
     name: Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v4
@@ -34,7 +36,7 @@ jobs:
 
       - name: 'Authenticate to Google Cloud'
         id: 'auth'
-        uses: google-github-actions/auth@v2
+        uses: 'google-github-actions/auth@v1'
         with:
           credentials_json: '${{ secrets.GOOGLE_CREDENTIALS }}'
 
@@ -50,7 +52,13 @@ jobs:
         shell: bash
         run: |
           echo "Running scheduled tests, installing dependencies with poetry..."
-          poetry install --with=test_integration,test
+          poetry install --with=test_integration
+          poetry run pip install google-cloud-aiplatform
+          poetry run pip install "boto3>=1.28.57"
+          if [[ ${{ matrix.python-version }} != "3.8" ]]
+          then
+            poetry run pip install fireworks-ai
+          fi
 
       - name: Run tests
         shell: bash
@@ -60,9 +68,7 @@ jobs:
           AZURE_OPENAI_API_VERSION: ${{ secrets.AZURE_OPENAI_API_VERSION }}
           AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }}
           AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
-          AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
-          AZURE_OPENAI_LLM_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_LLM_DEPLOYMENT_NAME }}
-          AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_NAME }}
           FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
         run: |
           make scheduled_tests

.github/workflows/templates_ci.yml

@@ -24,7 +24,7 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  POETRY_VERSION: "1.7.1"
+  POETRY_VERSION: "1.6.1"
   WORKDIR: "templates"
 
 jobs:
@@ -33,4 +33,5 @@ jobs:
       ./.github/workflows/_lint.yml
     with:
       working-directory: templates
+      langchain-location: ../libs/langchain
     secrets: inherit

.gitignore

@@ -167,7 +167,8 @@ docs/node_modules/
 docs/.docusaurus/
 docs/.cache-loader/
 docs/_dist
-docs/api_reference/*api_reference.rst
+docs/api_reference/api_reference.rst
+docs/api_reference/experimental_api_reference.rst
 docs/api_reference/_build
 docs/api_reference/*/
 !docs/api_reference/_static/

.readthedocs.yaml

@@ -10,10 +10,9 @@ build:
   tools:
     python: "3.11"
   commands:
-      - python -m virtualenv $READTHEDOCS_VIRTUALENV_PATH
+      - python -mvirtualenv $READTHEDOCS_VIRTUALENV_PATH
       - python -m pip install --upgrade --no-cache-dir pip setuptools
       - python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext
-      - python -m pip install ./libs/partners/*
       - python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt
       - python docs/api_reference/create_api_rst.py
       - cat docs/api_reference/conf.py

LICENSE

@@ -1,6 +1,6 @@
-MIT License
+The MIT License
 
-Copyright (c) LangChain, Inc.
+Copyright (c) Harrison Chase
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -9,13 +9,13 @@ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

Makefile

@@ -41,10 +41,9 @@ spell_fix:
 # LINTING AND FORMATTING
 ######################
 
-lint lint_package lint_tests:
+lint:
 	poetry run ruff docs templates cookbook
 	poetry run ruff format docs templates cookbook --diff
-	poetry run ruff --select I docs templates cookbook
 
 format format_diff:
 	poetry run ruff format docs templates cookbook

README.md

@@ -3,7 +3,8 @@
 ⚡ Building applications with LLMs through composability ⚡
 
 [![Release Notes](https://img.shields.io/github/release/langchain-ai/langchain)](https://github.com/langchain-ai/langchain/releases)
-[![CI](https://github.com/langchain-ai/langchain/actions/workflows/check_diffs.yml/badge.svg)](https://github.com/langchain-ai/langchain/actions/workflows/check_diffs.yml)
+[![CI](https://github.com/langchain-ai/langchain/actions/workflows/langchain_ci.yml/badge.svg)](https://github.com/langchain-ai/langchain/actions/workflows/langchain_ci.yml)
+[![Experimental CI](https://github.com/langchain-ai/langchain/actions/workflows/langchain_experimental_ci.yml/badge.svg)](https://github.com/langchain-ai/langchain/actions/workflows/langchain_experimental_ci.yml)
 [![Downloads](https://static.pepy.tech/badge/langchain/month)](https://pepy.tech/project/langchain)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/langchainai.svg?style=social&label=Follow%20%40LangChainAI)](https://twitter.com/langchainai)
@@ -29,7 +30,7 @@ pip install langchain
 
 With conda:
 ```bash
-conda install langchain -c conda-forge
+pip install langsmith && conda install langchain -c conda-forge
 ```
 
 ## 🤔 What is LangChain?
@@ -44,10 +45,7 @@ This framework consists of several parts.
 - **[LangServe](https://github.com/langchain-ai/langserve)**: A library for deploying LangChain chains as a REST API.
 - **[LangSmith](https://smith.langchain.com)**: A developer platform that lets you debug, test, evaluate, and monitor chains built on any LLM framework and seamlessly integrates with LangChain.
 
-The LangChain libraries themselves are made up of several different packages.
-- **[`langchain-core`](libs/core)**: Base abstractions and LangChain Expression Language.
-- **[`langchain-community`](libs/community)**: Third party integrations.
-- **[`langchain`](libs/langchain)**: Chains, agents, and retrieval strategies that make up an application's cognitive architecture.
+**This repo contains the `langchain` ([here](libs/langchain)), `langchain-experimental` ([here](libs/experimental)), and `langchain-cli` ([here](libs/cli)) Python packages, as well as [LangChain Templates](templates).**
 
 ![LangChain Stack](docs/static/img/langchain_stack.png)
 
@@ -95,7 +93,7 @@ Agents involve an LLM making decisions about which Actions to take, taking that
 Please see [here](https://python.langchain.com) for full documentation, which includes:
 
 - [Getting started](https://python.langchain.com/docs/get_started/introduction): installation, setting up the environment, simple examples
-- Overview of the [interfaces](https://python.langchain.com/docs/expression_language/), [modules](https://python.langchain.com/docs/modules/), and [integrations](https://python.langchain.com/docs/integrations/providers)
+- Overview of the [interfaces](https://python.langchain.com/docs/expression_language/), [modules](https://python.langchain.com/docs/modules/) and [integrations](https://python.langchain.com/docs/integrations/providers)
 - [Use case](https://python.langchain.com/docs/use_cases/qa_structured/sql) walkthroughs and best practice [guides](https://python.langchain.com/docs/guides/adapters/openai)
 - [LangSmith](https://python.langchain.com/docs/langsmith/), [LangServe](https://python.langchain.com/docs/langserve), and [LangChain Template](https://python.langchain.com/docs/templates/) overviews
 - [Reference](https://api.python.langchain.com): full API docs
@@ -105,8 +103,4 @@ Please see [here](https://python.langchain.com) for full documentation, which in
 
 As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
 
-For detailed information on how to contribute, see [here](https://python.langchain.com/docs/contributing/).
-
-## 🌟 Contributors
-
-[![langchain contributors](https://contrib.rocks/image?repo=langchain-ai/langchain&max=2000)](https://github.com/langchain-ai/langchain/graphs/contributors)
+For detailed information on how to contribute, see [here](.github/CONTRIBUTING.md).

cookbook/LLaMA2_sql_chat.ipynb

@@ -61,13 +61,13 @@
    ],
    "source": [
     "# Local\n",
-    "from langchain_community.chat_models import ChatOllama\n",
+    "from langchain.chat_models import ChatOllama\n",
     "\n",
     "llama2_chat = ChatOllama(model=\"llama2:13b-chat\")\n",
     "llama2_code = ChatOllama(model=\"codellama:7b-instruct\")\n",
     "\n",
     "# API\n",
-    "from langchain_community.llms import Replicate\n",
+    "from langchain.llms import Replicate\n",
     "\n",
     "# REPLICATE_API_TOKEN = getpass()\n",
     "# os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN\n",
@@ -107,7 +107,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.utilities import SQLDatabase\n",
+    "from langchain.utilities import SQLDatabase\n",
     "\n",
     "db = SQLDatabase.from_uri(\"sqlite:///nba_roster.db\", sample_rows_in_table_info=0)\n",
     "\n",
@@ -125,7 +125,7 @@
    "id": "654b3577-baa2-4e12-a393-f40e5db49ac7",
    "metadata": {},
    "source": [
-    "## Query a SQL Database \n",
+    "## Query a SQL DB \n",
     "\n",
     "Follow the runnables workflow [here](https://python.langchain.com/docs/expression_language/cookbook/sql_db)."
    ]
@@ -151,7 +151,6 @@
     "# Prompt\n",
     "from langchain.prompts import ChatPromptTemplate\n",
     "\n",
-    "# Update the template based on the type of SQL Database like MySQL, Microsoft SQL Server and so on\n",
     "template = \"\"\"Based on the table schema below, write a SQL query that would answer the user's question:\n",
     "{schema}\n",
     "\n",
@@ -165,8 +164,8 @@
     ")\n",
     "\n",
     "# Chain to query\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnablePassthrough\n",
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "sql_response = (\n",
     "    RunnablePassthrough.assign(schema=get_schema)\n",
@@ -218,7 +217,7 @@
     "    [\n",
     "        (\n",
     "            \"system\",\n",
-    "            \"Given an input question and SQL response, convert it to a natural language answer. No pre-amble.\",\n",
+    "            \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\",\n",
     "        ),\n",
     "        (\"human\", template),\n",
     "    ]\n",
@@ -294,7 +293,7 @@
     "memory = ConversationBufferMemory(return_messages=True)\n",
     "\n",
     "# Chain to query with memory\n",
-    "from langchain_core.runnables import RunnableLambda\n",
+    "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
     "sql_chain = (\n",
     "    RunnablePassthrough.assign(\n",
@@ -346,7 +345,7 @@
     "    [\n",
     "        (\n",
     "            \"system\",\n",
-    "            \"Given an input question and SQL response, convert it to a natural language answer. No pre-amble.\",\n",
+    "            \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\",\n",
     "        ),\n",
     "        (\"human\", template),\n",
     "    ]\n",

cookbook/README.md

@@ -8,7 +8,6 @@ Notebook | Description
 [Semi_Structured_RAG.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_Structured_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data, including text and tables, using unstructured for parsing, multi-vector retriever for storing, and lcel for implementing chains.
 [Semi_structured_and_multi_moda...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using unstructured for parsing, multi-vector retriever for storage and retrieval, and lcel for implementing chains.
 [Semi_structured_multi_modal_RA...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using various tools and methods such as unstructured for parsing, multi-vector retriever for storing, lcel for implementing chains, and open source language models like llama2, llava, and gpt4all.
-[analyze_document.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/analyze_document.ipynb) | Analyze a single long document.
 [autogpt/autogpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/autogpt.ipynb) | Implement autogpt, a language model, with langchain primitives such as llms, prompttemplates, vectorstores, embeddings, and tools.
 [autogpt/marathon_times.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/marathon_times.ipynb) | Implement autogpt for finding winning marathon times.
 [baby_agi.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi.ipynb) | Implement babyagi, an ai agent that can generate and execute tasks based on a given objective, with the flexibility to swap out specific vectorstores/model providers.
@@ -45,7 +44,6 @@ Notebook | Description
 [plan_and_execute_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/plan_and_execute_agent.ipynb) | Create plan-and-execute agents that accomplish objectives by planning tasks with a language model (llm) and executing them with a separate agent.
 [press_releases.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/press_releases.ipynb) | Retrieve and query company press release data powered by [Kay.ai](https://kay.ai).
 [program_aided_language_model.i...](https://github.com/langchain-ai/langchain/tree/master/cookbook/program_aided_language_model.ipynb) | Implement program-aided language models as described in the provided research paper.
-[qa_citations.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/qa_citations.ipynb) | Different ways to get a model to cite its sources.
 [retrieval_in_sql.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/retrieval_in_sql.ipynb) | Perform retrieval-augmented-generation (rag) on a PostgreSQL database using pgvector.
 [sales_agent_with_context.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/sales_agent_with_context.ipynb) | Implement a context-aware ai sales agent, salesgpt, that can have natural sales conversations, interact with other systems, and use a product knowledge base to discuss a company's offerings.
 [self_query_hotel_search.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/self_query_hotel_search.ipynb) | Build a hotel room search feature with self-querying retrieval, using a specific hotel recommendation dataset.

cookbook/Semi_Structured_RAG.ipynb

@@ -235,9 +235,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser"
+    "from langchain.schema.output_parser import StrOutputParser"
    ]
   },
   {
@@ -318,11 +318,11 @@
    "source": [
     "import uuid\n",
     "\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.storage import InMemoryStore\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import Chroma\n",
-    "from langchain_core.documents import Document\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(collection_name=\"summaries\", embedding_function=OpenAIEmbeddings())\n",
@@ -374,7 +374,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_core.runnables import RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",
     "template = \"\"\"Answer the question based only on the following context, which can include text and tables:\n",

cookbook/Semi_structured_and_multi_modal_RAG.ipynb

@@ -211,9 +211,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser"
+    "from langchain.schema.output_parser import StrOutputParser"
    ]
   },
   {
@@ -373,11 +373,11 @@
    "source": [
     "import uuid\n",
     "\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.storage import InMemoryStore\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import Chroma\n",
-    "from langchain_core.documents import Document\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(collection_name=\"summaries\", embedding_function=OpenAIEmbeddings())\n",
@@ -646,7 +646,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_core.runnables import RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",
     "template = \"\"\"Answer the question based only on the following context, which can include text and tables:\n",

cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb

@@ -209,9 +209,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOllama\n",
     "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain_community.chat_models import ChatOllama\n",
-    "from langchain_core.output_parsers import StrOutputParser"
+    "from langchain.schema.output_parser import StrOutputParser"
    ]
   },
   {
@@ -376,11 +376,11 @@
    "source": [
     "import uuid\n",
     "\n",
+    "from langchain.embeddings import GPT4AllEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.storage import InMemoryStore\n",
-    "from langchain_community.embeddings import GPT4AllEmbeddings\n",
-    "from langchain_community.vectorstores import Chroma\n",
-    "from langchain_core.documents import Document\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(\n",
@@ -532,7 +532,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_core.runnables import RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",
     "template = \"\"\"Answer the question based only on the following context, which can include text and tables:\n",

cookbook/advanced_rag_eval.ipynb

@@ -62,7 +62,7 @@
     "path = \"/Users/rlm/Desktop/cpi/\"\n",
     "\n",
     "# Load\n",
-    "from langchain_community.document_loaders import PyPDFLoader\n",
+    "from langchain.document_loaders import PyPDFLoader\n",
     "\n",
     "loader = PyPDFLoader(path + \"cpi.pdf\")\n",
     "pdf_pages = loader.load()\n",
@@ -132,8 +132,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import Chroma\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "baseline = Chroma.from_texts(\n",
     "    texts=all_splits_pypdf_texts,\n",
@@ -160,9 +160,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
+    "from langchain.schema.output_parser import StrOutputParser\n",
     "\n",
     "# Prompt\n",
     "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\\n",
@@ -202,7 +202,7 @@
     "import os\n",
     "from io import BytesIO\n",
     "\n",
-    "from langchain_core.messages import HumanMessage\n",
+    "from langchain.schema.messages import HumanMessage\n",
     "from PIL import Image\n",
     "\n",
     "\n",
@@ -273,8 +273,8 @@
     "from base64 import b64decode\n",
     "\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.storage import InMemoryStore\n",
-    "from langchain_core.documents import Document\n",
     "\n",
     "\n",
     "def create_multi_vector_retriever(\n",
@@ -475,7 +475,7 @@
    "source": [
     "from operator import itemgetter\n",
     "\n",
-    "from langchain_core.runnables import RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt\n",
     "template = \"\"\"Answer the question based only on the following context, which can include text and tables:\n",
@@ -521,7 +521,7 @@
     "import re\n",
     "\n",
     "from langchain.schema import Document\n",
-    "from langchain_core.runnables import RunnableLambda\n",
+    "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
     "\n",
     "def looks_like_base64(sb):\n",

cookbook/analyze_document.ipynb

@@ -1,105 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "f69d4a4c-137d-47e9-bea1-786afce9c1c0",
-   "metadata": {},
-   "source": [
-    "# Analyze a single long document\n",
-    "\n",
-    "The AnalyzeDocumentChain takes in a single document, splits it up, and then runs it through a CombineDocumentsChain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "2a0707ce-6d2d-471b-bc33-64da32a7b3f0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with open(\"../docs/docs/modules/state_of_the_union.txt\") as f:\n",
-    "    state_of_the_union = f.read()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "ca14d161-2d5b-4a6c-a296-77d8ce4b28cd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import AnalyzeDocumentChain\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "\n",
-    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "9f97406c-85a9-45fb-99ce-9138c0ba3731",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains.question_answering import load_qa_chain\n",
-    "\n",
-    "qa_chain = load_qa_chain(llm, chain_type=\"map_reduce\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "0871a753-f5bb-4b4f-a394-f87f2691f659",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "qa_document_chain = AnalyzeDocumentChain(combine_docs_chain=qa_chain)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "e6f86428-3c2c-46a0-a57c-e22826fdbf91",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'The President said, \"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.\"'"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "qa_document_chain.run(\n",
-    "    input_document=state_of_the_union,\n",
-    "    question=\"what did the president say about justice breyer?\",\n",
-    ")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/autogpt/autogpt.ipynb

@@ -28,9 +28,9 @@
    "outputs": [],
    "source": [
     "from langchain.agents import Tool\n",
-    "from langchain_community.tools.file_management.read import ReadFileTool\n",
-    "from langchain_community.tools.file_management.write import WriteFileTool\n",
-    "from langchain_community.utilities import SerpAPIWrapper\n",
+    "from langchain.tools.file_management.read import ReadFileTool\n",
+    "from langchain.tools.file_management.write import WriteFileTool\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
     "\n",
     "search = SerpAPIWrapper()\n",
     "tools = [\n",
@@ -62,8 +62,8 @@
    "outputs": [],
    "source": [
     "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import FAISS"
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {
@@ -100,7 +100,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain_experimental.autonomous_agents import AutoGPT"
    ]
   },
@@ -167,7 +167,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain_community.chat_message_histories import FileChatMessageHistory\n",
+    "from langchain.memory.chat_message_histories import FileChatMessageHistory\n",
     "\n",
     "agent = AutoGPT.from_llm_and_tools(\n",
     "    ai_name=\"Tom\",\n",

cookbook/autogpt/marathon_times.ipynb

@@ -39,9 +39,9 @@
     "\n",
     "import nest_asyncio\n",
     "import pandas as pd\n",
+    "from langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.docstore.document import Document\n",
-    "from langchain_community.agent_toolkits.pandas.base import create_pandas_dataframe_agent\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
     "from langchain_experimental.autonomous_agents import AutoGPT\n",
     "\n",
     "# Needed synce jupyter runs an async eventloop\n",
@@ -93,8 +93,8 @@
     "from typing import Optional\n",
     "\n",
     "from langchain.agents import tool\n",
-    "from langchain_community.tools.file_management.read import ReadFileTool\n",
-    "from langchain_community.tools.file_management.write import WriteFileTool\n",
+    "from langchain.tools.file_management.read import ReadFileTool\n",
+    "from langchain.tools.file_management.write import WriteFileTool\n",
     "\n",
     "ROOT_DIR = \"./data/\"\n",
     "\n",
@@ -311,8 +311,8 @@
     "# Memory\n",
     "import faiss\n",
     "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import FAISS\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import FAISS\n",
     "\n",
     "embeddings_model = OpenAIEmbeddings()\n",
     "embedding_size = 1536\n",

cookbook/baby_agi.ipynb

@@ -31,8 +31,8 @@
    "source": [
     "from typing import Optional\n",
     "\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.autonomous_agents import BabyAGI"
    ]
   },
@@ -54,7 +54,7 @@
    "outputs": [],
    "source": [
     "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain_community.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {

cookbook/baby_agi_with_agent.ipynb

@@ -28,9 +28,9 @@
     "from typing import Optional\n",
     "\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.llms import OpenAI\n",
     "from langchain_experimental.autonomous_agents import BabyAGI"
    ]
   },
@@ -63,7 +63,7 @@
     "%pip install faiss-cpu > /dev/null\n",
     "%pip install google-search-results > /dev/null\n",
     "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain_community.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {
@@ -108,8 +108,8 @@
    "source": [
     "from langchain.agents import AgentExecutor, Tool, ZeroShotAgent\n",
     "from langchain.chains import LLMChain\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.utilities import SerpAPIWrapper\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.utilities import SerpAPIWrapper\n",
     "\n",
     "todo_prompt = PromptTemplate.from_template(\n",
     "    \"You are a planner who is an expert at coming up with a todo list for a given objective. Come up with a todo list for this objective: {objective}\"\n",

cookbook/camel_role_playing.ipynb

@@ -36,6 +36,7 @@
    "source": [
     "from typing import List\n",
     "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts.chat import (\n",
     "    HumanMessagePromptTemplate,\n",
     "    SystemMessagePromptTemplate,\n",
@@ -45,8 +46,7 @@
     "    BaseMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/causal_program_aided_language_model.ipynb

@@ -47,7 +47,7 @@
    "outputs": [],
    "source": [
     "from IPython.display import SVG\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.cpal.base import CPALChain\n",
     "from langchain_experimental.pal_chain import PALChain\n",
     "\n",

cookbook/code-analysis-deeplake.ipynb

@@ -23,9 +23,9 @@
    "metadata": {},
    "source": [
     "1. Prepare data:\n",
-    "   1. Upload all python project files using the `langchain_community.document_loaders.TextLoader`. We will call these files the **documents**.\n",
+    "   1. Upload all python project files using the `langchain.document_loaders.TextLoader`. We will call these files the **documents**.\n",
     "   2. Split all documents to chunks using the `langchain.text_splitter.CharacterTextSplitter`.\n",
-    "   3. Embed chunks and upload them into the DeepLake using `langchain.embeddings.openai.OpenAIEmbeddings` and `langchain_community.vectorstores.DeepLake`\n",
+    "   3. Embed chunks and upload them into the DeepLake using `langchain.embeddings.openai.OpenAIEmbeddings` and `langchain.vectorstores.DeepLake`\n",
     "2. Question-Answering:\n",
     "   1. Build a chain from `langchain.chat_models.ChatOpenAI` and `langchain.chains.ConversationalRetrievalChain`\n",
     "   2. Prepare questions.\n",
@@ -166,7 +166,7 @@
     }
    ],
    "source": [
-    "from langchain_community.document_loaders import TextLoader\n",
+    "from langchain.document_loaders import TextLoader\n",
     "\n",
     "root_dir = \"../../../../../../libs\"\n",
     "\n",
@@ -648,7 +648,7 @@
     {
      "data": {
       "text/plain": [
-       "OpenAIEmbeddings(client=<class 'openai.api_resources.embedding.Embedding'>, model='text-embedding-ada-002', deployment='text-embedding-ada-002', openai_api_version='', openai_api_base='', openai_api_type='', openai_proxy='', embedding_ctx_length=8191, openai_api_key='', openai_organization='', allowed_special=set(), disallowed_special='all', chunk_size=1000, max_retries=6, request_timeout=None, headers=None, tiktoken_model_name=None, show_progress_bar=False, model_kwargs={})"
+       "OpenAIEmbeddings(client=<class 'openai.api_resources.embedding.Embedding'>, model='text-embedding-ada-002', deployment='text-embedding-ada-002', openai_api_version='', openai_api_base='', openai_api_type='', openai_proxy='', embedding_ctx_length=8191, openai_api_key='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', openai_organization='', allowed_special=set(), disallowed_special='all', chunk_size=1000, max_retries=6, request_timeout=None, headers=None, tiktoken_model_name=None, show_progress_bar=False, model_kwargs={})"
       ]
      },
      "execution_count": 13,
@@ -657,7 +657,7 @@
     }
    ],
    "source": [
-    "from langchain_community.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "\n",
     "embeddings = OpenAIEmbeddings()\n",
     "embeddings"
@@ -706,7 +706,7 @@
     {
      "data": {
       "text/plain": [
-       "<langchain_community.vectorstores.deeplake.DeepLake at 0x7fe1b67d7a30>"
+       "<langchain.vectorstores.deeplake.DeepLake at 0x7fe1b67d7a30>"
       ]
      },
      "execution_count": 15,
@@ -715,7 +715,7 @@
     }
    ],
    "source": [
-    "from langchain_community.vectorstores import DeepLake\n",
+    "from langchain.vectorstores import DeepLake\n",
     "\n",
     "username = \"<USERNAME_OR_ORG>\"\n",
     "\n",
@@ -740,7 +740,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# from langchain_community.vectorstores import DeepLake\n",
+    "# from langchain.vectorstores import DeepLake\n",
     "\n",
     "# db = DeepLake.from_documents(\n",
     "#     texts, embeddings, dataset_path=f\"hub://{<org_id>}/langchain-code\", runtime={\"tensor_db\": True}\n",
@@ -834,7 +834,7 @@
    "outputs": [],
    "source": [
     "from langchain.chains import ConversationalRetrievalChain\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "model = ChatOpenAI(\n",
     "    model_name=\"gpt-3.5-turbo-0613\"\n",

cookbook/custom_agent_with_plugin_retrieval.ipynb

@@ -40,12 +40,12 @@
     "    AgentOutputParser,\n",
     "    LLMSingleActionAgent,\n",
     ")\n",
+    "from langchain.agents.agent_toolkits import NLAToolkit\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import StringPromptTemplate\n",
     "from langchain.schema import AgentAction, AgentFinish\n",
-    "from langchain_community.agent_toolkits import NLAToolkit\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.tools.plugin import AIPlugin"
+    "from langchain.tools.plugin import AIPlugin"
    ]
   },
   {
@@ -114,9 +114,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.schema import Document\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {

cookbook/custom_agent_with_plugin_retrieval_using_plugnplai.ipynb

@@ -65,12 +65,12 @@
     "    AgentOutputParser,\n",
     "    LLMSingleActionAgent,\n",
     ")\n",
+    "from langchain.agents.agent_toolkits import NLAToolkit\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import StringPromptTemplate\n",
     "from langchain.schema import AgentAction, AgentFinish\n",
-    "from langchain_community.agent_toolkits import NLAToolkit\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.tools.plugin import AIPlugin"
+    "from langchain.tools.plugin import AIPlugin"
    ]
   },
   {
@@ -138,9 +138,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.schema import Document\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {

cookbook/databricks_sql_db.ipynb

@@ -80,7 +80,7 @@
    "outputs": [],
    "source": [
     "# Connecting to Databricks with SQLDatabase wrapper\n",
-    "from langchain_community.utilities import SQLDatabase\n",
+    "from langchain.utilities import SQLDatabase\n",
     "\n",
     "db = SQLDatabase.from_databricks(catalog=\"samples\", schema=\"nyctaxi\")"
    ]
@@ -93,7 +93,7 @@
    "outputs": [],
    "source": [
     "# Creating a OpenAI Chat LLM wrapper\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "llm = ChatOpenAI(temperature=0, model_name=\"gpt-4\")"
    ]
@@ -115,7 +115,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.utilities import SQLDatabaseChain\n",
+    "from langchain.utilities import SQLDatabaseChain\n",
     "\n",
     "db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)"
    ]
@@ -177,7 +177,7 @@
    "outputs": [],
    "source": [
     "from langchain.agents import create_sql_agent\n",
-    "from langchain_community.agent_toolkits import SQLDatabaseToolkit\n",
+    "from langchain.agents.agent_toolkits import SQLDatabaseToolkit\n",
     "\n",
     "toolkit = SQLDatabaseToolkit(db=db, llm=llm)\n",
     "agent = create_sql_agent(llm=llm, toolkit=toolkit, verbose=True)"

cookbook/deeplake_semantic_search_over_chat.ipynb

@@ -52,13 +52,13 @@
     "import os\n",
     "\n",
     "from langchain.chains import RetrievalQA\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.text_splitter import (\n",
     "    CharacterTextSplitter,\n",
     "    RecursiveCharacterTextSplitter,\n",
     ")\n",
-    "from langchain_community.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.vectorstores import DeepLake\n",
+    "from langchain.vectorstores import DeepLake\n",
     "\n",
     "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
     "activeloop_token = getpass.getpass(\"Activeloop Token:\")\n",

cookbook/elasticsearch_db_qa.ipynb

@@ -39,7 +39,7 @@
    "source": [
     "from elasticsearch import Elasticsearch\n",
     "from langchain.chains.elasticsearch_database import ElasticsearchDatabaseChain\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    "from langchain.chat_models import ChatOpenAI"
    ]
   },
   {

cookbook/extraction_openai_tools.ipynb

@@ -22,8 +22,8 @@
     "from typing import List, Optional\n",
     "\n",
     "from langchain.chains.openai_tools import create_extraction_chain_pydantic\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.pydantic_v1 import BaseModel"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.pydantic_v1 import BaseModel"
    ]
   },
   {
@@ -151,11 +151,11 @@
     "\n",
     "from langchain.output_parsers.openai_tools import PydanticToolsParser\n",
     "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
-    "from langchain_core.runnables import Runnable\n",
-    "from langchain_core.pydantic_v1 import BaseModel\n",
+    "from langchain.schema.runnable import Runnable\n",
+    "from langchain.pydantic_v1 import BaseModel\n",
     "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain_core.messages import SystemMessage\n",
-    "from langchain_core.language_models import BaseLanguageModel\n",
+    "from langchain.schema.messages import SystemMessage\n",
+    "from langchain.schema.language_model import BaseLanguageModel\n",
     "\n",
     "_EXTRACTION_TEMPLATE = \"\"\"Extract and save the relevant entities mentioned \\\n",
     "in the following passage together with their properties.\n",

cookbook/fake_llm.ipynb

@@ -20,7 +20,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.llms.fake import FakeListLLM"
+    "from langchain.llms.fake import FakeListLLM"
    ]
   },
   {

cookbook/forward_looking_retrieval_augmented_generation.ipynb

@@ -73,10 +73,10 @@
     "    AsyncCallbackManagerForRetrieverRun,\n",
     "    CallbackManagerForRetrieverRun,\n",
     ")\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.schema import BaseRetriever, Document\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.utilities import GoogleSerperAPIWrapper"
+    "from langchain.utilities import GoogleSerperAPIWrapper"
    ]
   },
   {

cookbook/generative_agents_interactive_simulacra_of_human_behavior.ipynb

@@ -47,11 +47,11 @@
     "from datetime import datetime, timedelta\n",
     "from typing import List\n",
     "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.docstore import InMemoryDocstore\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers import TimeWeightedVectorStoreRetriever\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import FAISS\n",
+    "from langchain.vectorstores import FAISS\n",
     "from termcolor import colored"
    ]
   },

cookbook/hugginggpt.ipynb

@@ -75,7 +75,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.autonomous_agents import HuggingGPT\n",
     "\n",
     "# %env OPENAI_API_BASE=http://localhost:8000/v1"

cookbook/human_input_chat_model.ipynb

@@ -20,7 +20,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models.human import HumanInputChatModel"
+    "from langchain.chat_models.human import HumanInputChatModel"
    ]
   },
   {

cookbook/human_input_llm.ipynb

@@ -19,7 +19,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.llms.human import HumanInputLLM"
+    "from langchain.llms.human import HumanInputLLM"
    ]
   },
   {

cookbook/hypothetical_document_embeddings.ipynb

@@ -21,9 +21,9 @@
    "outputs": [],
    "source": [
     "from langchain.chains import HypotheticalDocumentEmbedder, LLMChain\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.llms import OpenAI"
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {
@@ -172,7 +172,7 @@
    "outputs": [],
    "source": [
     "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain_community.vectorstores import Chroma\n",
+    "from langchain.vectorstores import Chroma\n",
     "\n",
     "with open(\"../../state_of_the_union.txt\") as f:\n",
     "    state_of_the_union = f.read()\n",

cookbook/learned_prompt_optimization.ipynb

@@ -49,9 +49,9 @@
    "source": [
     "# pick and configure the LLM of your choice\n",
     "\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
-    "llm = OpenAI(model=\"gpt-3.5-turbo-instruct\")"
+    "llm = OpenAI(model=\"text-davinci-003\")"
    ]
   },
   {

cookbook/llm_bash.ipynb

@@ -43,7 +43,7 @@
     }
    ],
    "source": [
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.llm_bash.base import LLMBashChain\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
@@ -69,8 +69,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chains.llm_bash.prompt import BashOutputParser\n",
     "from langchain.prompts.prompt import PromptTemplate\n",
-    "from langchain_experimental.llm_bash.prompt import BashOutputParser\n",
     "\n",
     "_PROMPT_TEMPLATE = \"\"\"If someone asks you to perform a task, your job is to come up with a series of bash commands that will perform the task. There is no need to put \"#!/bin/bash\" in your answer. Make sure to reason step by step, using this format:\n",
     "Question: \"copy the files in the directory named 'target' into a new directory at the same level as target called 'myNewDirectory'\"\n",

cookbook/llm_checker.ipynb

@@ -42,7 +42,7 @@
    ],
    "source": [
     "from langchain.chains import LLMCheckerChain\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0.7)\n",
     "\n",

cookbook/llm_math.ipynb

@@ -46,7 +46,7 @@
    ],
    "source": [
     "from langchain.chains import LLMMathChain\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "llm_math = LLMMathChain.from_llm(llm, verbose=True)\n",

cookbook/llm_summarization_checker.ipynb

@@ -331,7 +331,7 @@
    ],
    "source": [
     "from langchain.chains import LLMSummarizationCheckerChain\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "checker_chain = LLMSummarizationCheckerChain.from_llm(llm, verbose=True, max_checks=2)\n",
@@ -822,7 +822,7 @@
    ],
    "source": [
     "from langchain.chains import LLMSummarizationCheckerChain\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "checker_chain = LLMSummarizationCheckerChain.from_llm(llm, verbose=True, max_checks=3)\n",
@@ -1096,7 +1096,7 @@
    ],
    "source": [
     "from langchain.chains import LLMSummarizationCheckerChain\n",
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "checker_chain = LLMSummarizationCheckerChain.from_llm(llm, max_checks=3, verbose=True)\n",

cookbook/llm_symbolic_math.ipynb

@@ -14,7 +14,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.llm_symbolic_math.base import LLMSymbolicMathChain\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",

cookbook/meta_prompt.ipynb

@@ -57,9 +57,9 @@
    "outputs": [],
    "source": [
     "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.memory import ConversationBufferWindowMemory\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.llms import OpenAI"
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {

cookbook/multi_modal_QA.ipynb

@@ -91,8 +91,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.messages import HumanMessage, SystemMessage"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema.messages import HumanMessage, SystemMessage"
    ]
   },
   {

cookbook/multi_modal_RAG_chroma.ipynb

@@ -42,7 +42,7 @@
     "* We will use Open Clip multi-modal embeddings.\n",
     "* We will use [Chroma](https://www.trychroma.com/) with support for multi-modal.\n",
     "\n",
-    "A separate cookbook highlights `Options 2 and 3` [here](https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb).\n",
+    "A seperate cookbook highlights `Options 2 and 3` [here](https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb).\n",
     "\n",
     "![chroma_multimodal.png](attachment:1920fda3-1808-407c-9820-f518c9c6f566.png)\n",
     "\n",
@@ -187,7 +187,7 @@
     "\n",
     "import chromadb\n",
     "import numpy as np\n",
-    "from langchain_community.vectorstores import Chroma\n",
+    "from langchain.vectorstores import Chroma\n",
     "from langchain_experimental.open_clip import OpenCLIPEmbeddings\n",
     "from PIL import Image as _PILImage\n",
     "\n",
@@ -315,10 +315,10 @@
    "source": [
     "from operator import itemgetter\n",
     "\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.messages import HumanMessage, SystemMessage\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema.messages import HumanMessage, SystemMessage\n",
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
     "\n",
     "\n",
     "def prompt_func(data_dict):\n",

cookbook/multi_modal_output_agent.ipynb

@@ -31,7 +31,7 @@
    "source": [
     "import re\n",
     "\n",
-    "from IPython.display import Image, display\n",
+    "from IPython.display import Image\n",
     "from steamship import Block, Steamship"
    ]
   },
@@ -43,8 +43,8 @@
    "outputs": [],
    "source": [
     "from langchain.agents import AgentType, initialize_agent\n",
-    "from langchain.tools import SteamshipImageGenerationTool\n",
-    "from langchain_community.llms import OpenAI"
+    "from langchain.llms import OpenAI\n",
+    "from langchain.tools import SteamshipImageGenerationTool"
    ]
   },
   {
@@ -180,7 +180,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.11.3"
   }
  },
  "nbformat": 4,

cookbook/multi_player_dnd.ipynb

@@ -28,11 +28,11 @@
    "source": [
     "from typing import Callable, List\n",
     "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/multiagent_authoritarian.ipynb

@@ -33,6 +33,7 @@
     "from typing import Callable, List\n",
     "\n",
     "import tenacity\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.output_parsers import RegexParser\n",
     "from langchain.prompts import (\n",
     "    PromptTemplate,\n",
@@ -40,8 +41,7 @@
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/multiagent_bidding.ipynb

@@ -27,13 +27,13 @@
     "from typing import Callable, List\n",
     "\n",
     "import tenacity\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.output_parsers import RegexParser\n",
     "from langchain.prompts import PromptTemplate\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/myscale_vector_sql.ipynb

@@ -31,9 +31,9 @@
     "from os import environ\n",
     "\n",
     "from langchain.chains import LLMChain\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.utilities import SQLDatabase\n",
+    "from langchain.utilities import SQLDatabase\n",
     "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
     "from sqlalchemy import MetaData, create_engine\n",
     "\n",
@@ -57,7 +57,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.embeddings import HuggingFaceInstructEmbeddings\n",
+    "from langchain.embeddings import HuggingFaceInstructEmbeddings\n",
     "from langchain_experimental.sql.vector_sql import VectorSQLOutputParser\n",
     "\n",
     "output_parser = VectorSQLOutputParser.from_embeddings(\n",
@@ -75,8 +75,8 @@
    "outputs": [],
    "source": [
     "from langchain.callbacks import StdOutCallbackHandler\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.utilities.sql_database import SQLDatabase\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.utilities.sql_database import SQLDatabase\n",
     "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
     "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
     "\n",
@@ -117,7 +117,7 @@
    "outputs": [],
    "source": [
     "from langchain.chains.qa_with_sources.retrieval import RetrievalQAWithSourcesChain\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain_experimental.retrievers.vector_sql_database import (\n",
     "    VectorSQLDatabaseChainRetriever,\n",
     ")\n",

cookbook/openai_functions_retrieval_qa.ipynb

@@ -20,10 +20,10 @@
    "outputs": [],
    "source": [
     "from langchain.chains import RetrievalQA\n",
+    "from langchain.document_loaders import TextLoader\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain_community.document_loaders import TextLoader\n",
-    "from langchain_community.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import Chroma"
+    "from langchain.vectorstores import Chroma"
    ]
   },
   {
@@ -52,8 +52,8 @@
    "source": [
     "from langchain.chains import create_qa_with_sources_chain\n",
     "from langchain.chains.combine_documents.stuff import StuffDocumentsChain\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {

cookbook/openai_v1_cookbook.ipynb

@@ -28,8 +28,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.messages import HumanMessage, SystemMessage"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema.messages import HumanMessage, SystemMessage"
    ]
   },
   {
@@ -252,7 +252,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_core.agents import AgentFinish\n",
+    "from langchain.schema.agent import AgentFinish\n",
     "\n",
     "\n",
     "def execute_agent(agent, tools, input):\n",
@@ -414,7 +414,7 @@
     "BREAKING CHANGES:\n",
     "- To use Azure embeddings with OpenAI V1, you'll need to use the new `AzureOpenAIEmbeddings` instead of the existing `OpenAIEmbeddings`. `OpenAIEmbeddings` continue to work when using Azure with `openai<1`.\n",
     "```python\n",
-    "from langchain_community.embeddings import AzureOpenAIEmbeddings\n",
+    "from langchain.embeddings import AzureOpenAIEmbeddings\n",
     "```\n",
     "\n",
     "\n",
@@ -457,8 +457,8 @@
     "\n",
     "from langchain.output_parsers.openai_tools import PydanticToolsParser\n",
     "from langchain.prompts import ChatPromptTemplate\n",
+    "from langchain.pydantic_v1 import BaseModel, Field\n",
     "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
-    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
     "\n",
     "\n",
     "class GetCurrentWeather(BaseModel):\n",

cookbook/petting_zoo.ipynb

@@ -47,12 +47,12 @@
     "import inspect\n",
     "\n",
     "import tenacity\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.output_parsers import RegexParser\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/plan_and_execute_agent.ipynb

@@ -29,11 +29,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.agents.tools import Tool\n",
     "from langchain.chains import LLMMathChain\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_community.llms import OpenAI\n",
-    "from langchain_community.utilities import DuckDuckGoSearchAPIWrapper\n",
-    "from langchain_core.tools import Tool\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.llms import OpenAI\n",
+    "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
     "from langchain_experimental.plan_and_execute import (\n",
     "    PlanAndExecute,\n",
     "    load_agent_executor,\n",

cookbook/press_releases.ipynb

@@ -81,8 +81,8 @@
    "outputs": [],
    "source": [
     "from langchain.chains import ConversationalRetrievalChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.retrievers import KayAiRetriever\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
     "\n",
     "model = ChatOpenAI(model_name=\"gpt-3.5-turbo\")\n",
     "retriever = KayAiRetriever.create(\n",

cookbook/program_aided_language_model.ipynb

@@ -17,7 +17,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.pal_chain import PALChain"
    ]
   },

cookbook/qianfan_baidu_elasticesearch_RAG.ipynb

@@ -37,8 +37,7 @@
    "source": [
     "#!pip install qianfan\n",
     "#!pip install bce-python-sdk\n",
-    "#!pip install elasticsearch == 7.11.0\n",
-    "#!pip install sentence-transformers"
+    "#!pip install elasticsearch == 7.11.0"
    ]
   },
   {
@@ -55,17 +54,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import sentence_transformers\n",
     "from baidubce.auth.bce_credentials import BceCredentials\n",
     "from baidubce.bce_client_configuration import BceClientConfiguration\n",
-    "from langchain.chains.retrieval_qa import RetrievalQA\n",
+    "from langchain.document_loaders.baiducloud_bos_directory import BaiduBOSDirectoryLoader\n",
+    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
+    "from langchain.llms.baidu_qianfan_endpoint import QianfanLLMEndpoint\n",
     "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "from langchain_community.document_loaders.baiducloud_bos_directory import (\n",
-    "    BaiduBOSDirectoryLoader,\n",
-    ")\n",
-    "from langchain_community.embeddings.huggingface import HuggingFaceEmbeddings\n",
-    "from langchain_community.llms.baidu_qianfan_endpoint import QianfanLLMEndpoint\n",
-    "from langchain_community.vectorstores import BESVectorStore"
+    "from langchain.vectorstores import BESVectorStore"
    ]
   },
   {
@@ -166,22 +161,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
    "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.9.17"
   },
+  "orig_nbformat": 4,
   "vscode": {
    "interpreter": {
     "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
@@ -189,5 +177,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 4
+ "nbformat_minor": 2
 }

cookbook/rag_fusion.ipynb

@@ -30,8 +30,8 @@
    "outputs": [],
    "source": [
     "import pinecone\n",
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import Pinecone\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import Pinecone\n",
     "\n",
     "pinecone.init(api_key=\"...\", environment=\"...\")"
    ]
@@ -86,8 +86,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser"
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema.output_parser import StrOutputParser"
    ]
   },
   {

cookbook/retrieval_in_sql.ipynb

@@ -42,8 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.sql_database import SQLDatabase\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
     "\n",
     "CONNECTION_STRING = \"postgresql+psycopg2://postgres:test@localhost:5432/vectordb\"  # Replace with your own\n",
     "db = SQLDatabase.from_uri(CONNECTION_STRING)"
@@ -88,7 +88,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "\n",
     "embeddings_model = OpenAIEmbeddings()"
    ]
@@ -133,7 +133,7 @@
     "from tqdm import tqdm\n",
     "\n",
     "for i in tqdm(range(len(title_embeddings))):\n",
-    "    title = song_titles[i].replace(\"'\", \"''\")\n",
+    "    title = titles[i].replace(\"'\", \"''\")\n",
     "    embedding = title_embeddings[i]\n",
     "    sql_command = (\n",
     "        f'UPDATE \"Track\" SET \"embeddings\" = ARRAY{embedding} WHERE \"Name\" ='\n",
@@ -267,9 +267,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnablePassthrough\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "db = SQLDatabase.from_uri(\n",
     "    CONNECTION_STRING\n",
@@ -324,7 +324,7 @@
    "source": [
     "import re\n",
     "\n",
-    "from langchain_core.runnables import RunnableLambda\n",
+    "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
     "\n",
     "def replace_brackets(match):\n",
@@ -681,9 +681,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.8.18"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 4
+ "nbformat_minor": 2
 }

cookbook/rewrite.ipynb

@@ -31,11 +31,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_community.utilities import DuckDuckGoSearchAPIWrapper\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnablePassthrough"
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
+    "from langchain.utilities import DuckDuckGoSearchAPIWrapper"
    ]
   },
   {

cookbook/sales_agent_with_context.ipynb

@@ -49,14 +49,14 @@
     "from langchain.agents.conversational.prompt import FORMAT_INSTRUCTIONS\n",
     "from langchain.chains import LLMChain, RetrievalQA\n",
     "from langchain.chains.base import Chain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.llms import BaseLLM, OpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
     "from langchain.prompts.base import StringPromptTemplate\n",
     "from langchain.schema import AgentAction, AgentFinish\n",
     "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_community.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain_community.llms import BaseLLM, OpenAI\n",
-    "from langchain_community.vectorstores import Chroma\n",
+    "from langchain.vectorstores import Chroma\n",
     "from pydantic import BaseModel, Field"
    ]
   },

cookbook/selecting_llms_based_on_context_length.ipynb

@@ -17,10 +17,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.prompt_values import PromptValue"
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.prompt import PromptValue"
    ]
   },
   {

cookbook/self_query_hotel_search.ipynb

@@ -255,7 +255,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "model = ChatOpenAI(model=\"gpt-4\")\n",
     "res = model.predict(\n",
@@ -1083,8 +1083,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.embeddings import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import ElasticsearchStore\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import ElasticsearchStore\n",
     "\n",
     "embeddings = OpenAIEmbeddings()"
    ]

cookbook/smart_llm.ipynb

@@ -51,8 +51,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
     "from langchain_experimental.smart_llm import SmartLLMChain"
    ]
   },

cookbook/sql_db_qa.mdx

@@ -9,8 +9,8 @@ To set it up, follow the instructions on https://database.guide/2-sample-databas
 
 
 ```python
-from langchain_community.llms import OpenAI
-from langchain_community.utilities import SQLDatabase
+from langchain.llms import OpenAI
+from langchain.utilities import SQLDatabase
 from langchain_experimental.sql import SQLDatabaseChain
 ```
 
@@ -200,8 +200,8 @@ result["intermediate_steps"]
 How to add memory to a SQLDatabaseChain:
 
 ```python
-from langchain_community.llms import OpenAI
-from langchain_community.utilities import SQLDatabase
+from langchain.llms import OpenAI
+from langchain.utilities import SQLDatabase
 from langchain_experimental.sql import SQLDatabaseChain
 ```
 
@@ -647,7 +647,7 @@ Sometimes you may not have the luxury of using OpenAI or other service-hosted la
 import logging
 import torch
 from transformers import AutoTokenizer, GPT2TokenizerFast, pipeline, AutoModelForSeq2SeqLM, AutoModelForCausalLM
-from langchain_community.llms import HuggingFacePipeline
+from langchain.llms import HuggingFacePipeline
 
 # Note: This model requires a large GPU, e.g. an 80GB A100. See documentation for other ways to run private non-OpenAI models.
 model_id = "google/flan-ul2"
@@ -679,7 +679,7 @@ local_llm = HuggingFacePipeline(pipeline=pipe)
 
 
 ```python
-from langchain_community.utilities import SQLDatabase
+from langchain.utilities import SQLDatabase
 from langchain_experimental.sql import SQLDatabaseChain
 
 db = SQLDatabase.from_uri("sqlite:///../../../../notebooks/Chinook.db", include_tables=['Customer'])
@@ -994,9 +994,9 @@ Now that you have some examples (with manually corrected output SQL), you can do
 ```python
 from langchain.prompts import FewShotPromptTemplate, PromptTemplate
 from langchain.chains.sql_database.prompt import _sqlite_prompt, PROMPT_SUFFIX
-from langchain_community.embeddings.huggingface import HuggingFaceEmbeddings
+from langchain.embeddings.huggingface import HuggingFaceEmbeddings
 from langchain.prompts.example_selector.semantic_similarity import SemanticSimilarityExampleSelector
-from langchain_community.vectorstores import Chroma
+from langchain.vectorstores import Chroma
 
 example_prompt = PromptTemplate(
     input_variables=["table_info", "input", "sql_cmd", "sql_result", "answer"],

cookbook/stepback-qa.ipynb

@@ -23,10 +23,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnableLambda"
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnableLambda"
    ]
   },
   {
@@ -129,7 +129,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.utilities import DuckDuckGoSearchAPIWrapper\n",
+    "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
     "\n",
     "search = DuckDuckGoSearchAPIWrapper(max_results=4)\n",
     "\n",

cookbook/tree_of_thought.ipynb

@@ -24,9 +24,9 @@
     }
    ],
    "source": [
-    "from langchain_community.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
-    "llm = OpenAI(temperature=1, max_tokens=512, model=\"gpt-3.5-turbo-instruct\")"
+    "llm = OpenAI(temperature=1, max_tokens=512, model=\"text-davinci-003\")"
    ]
   },
   {

cookbook/twitter-the-algorithm-analysis-deeplake.ipynb

@@ -37,8 +37,8 @@
     "import getpass\n",
     "import os\n",
     "\n",
-    "from langchain_community.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain_community.vectorstores import DeepLake\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.vectorstores import DeepLake\n",
     "\n",
     "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
     "activeloop_token = getpass.getpass(\"Activeloop Token:\")\n",
@@ -110,7 +110,7 @@
    "source": [
     "import os\n",
     "\n",
-    "from langchain_community.document_loaders import TextLoader\n",
+    "from langchain.document_loaders import TextLoader\n",
     "\n",
     "root_dir = \"./the-algorithm\"\n",
     "docs = []\n",
@@ -3809,7 +3809,7 @@
    "outputs": [],
    "source": [
     "from langchain.chains import ConversationalRetrievalChain\n",
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "model = ChatOpenAI(model_name=\"gpt-3.5-turbo-0613\")  # switch to 'gpt-4'\n",
     "qa = ConversationalRetrievalChain.from_llm(model, retriever=retriever)"

cookbook/two_agent_debate_tools.ipynb

@@ -24,13 +24,13 @@
    "source": [
     "from typing import Callable, List\n",
     "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.memory import ConversationBufferMemory\n",
     "from langchain.schema import (\n",
     "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/two_player_dnd.ipynb

@@ -24,11 +24,11 @@
    "source": [
     "from typing import Callable, List\n",
     "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")\n",
-    "from langchain_community.chat_models import ChatOpenAI"
+    ")"
    ]
   },
   {

cookbook/wikibase_agent.ipynb

@@ -187,7 +187,7 @@
     "    for key in path:\n",
     "        try:\n",
     "            current = current[key]\n",
-    "        except KeyError:\n",
+    "        except:\n",
     "            return None\n",
     "    return current\n",
     "\n",
@@ -599,7 +599,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatOpenAI\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "llm = ChatOpenAI(model_name=\"gpt-4\", temperature=0)"
    ]

docs/.local_build.sh

@@ -9,15 +9,13 @@ SCRIPT_DIR="$(cd "$(dirname "$0")"; pwd)"
 cd "${SCRIPT_DIR}"
 
 mkdir -p ../_dist
-rsync -ruv --exclude node_modules --exclude api_reference --exclude .venv --exclude .docusaurus . ../_dist
+cp -r . ../_dist
 cd ../_dist
 poetry run python scripts/model_feat_table.py
+poetry run nbdoc_build --srcdir docs
 cp ../cookbook/README.md src/pages/cookbook.mdx
-mkdir -p docs/templates
-cp ../templates/docs/INDEX.md docs/templates/index.md
-poetry run python scripts/copy_templates.py
+cp ../.github/CONTRIBUTING.md docs/contributing.md
 wget https://raw.githubusercontent.com/langchain-ai/langserve/main/README.md -O docs/langserve.md
-
-yarn
-
-quarto preview docs
+poetry run python scripts/generate_api_reference_links.py
+yarn install
+yarn start

docs/README.md

@@ -1,3 +1,49 @@
-# LangChain Documentation
+# Website
 
-For more information on contributing to our documentation, see the [Documentation Contributing Guide](https://python.langchain.com/docs/contributing/documentation)
+This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
+
+### Installation
+
+```
+$ yarn
+```
+
+### Local Development
+
+```
+$ yarn start
+```
+
+This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
+
+### Build
+
+```
+$ yarn build
+```
+
+This command generates static content into the `build` directory and can be served using any static contents hosting service.
+
+### Deployment
+
+Using SSH:
+
+```
+$ USE_SSH=true yarn deploy
+```
+
+Not using SSH:
+
+```
+$ GIT_USER=<Your GitHub username> yarn deploy
+```
+
+If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.
+
+### Continuous Integration
+
+Some common defaults for linting/formatting have been set for you. If you integrate your project with an open-source Continuous Integration system (e.g. Travis CI, CircleCI), you may check for issues using the following command.
+
+```
+$ yarn ci
+```

docs/api_reference/_static/css/custom.css

@@ -15,11 +15,3 @@ pre {
 #my-component-root *, #headlessui-portal-root * {
   z-index: 10000;
 }
-
-table.longtable code {
-  white-space: normal;
-}
-
-table.longtable td {
-  max-width: 600px;
-}

docs/api_reference/conf.py

@@ -72,8 +72,8 @@ def setup(app):
 # -- Project information -----------------------------------------------------
 
 project = "🦜🔗 LangChain"
-copyright = "2023, LangChain, Inc."
-author = "LangChain, Inc."
+copyright = "2023, Harrison Chase"
+author = "Harrison Chase"
 
 version = data["tool"]["poetry"]["version"]
 release = version
@@ -136,25 +136,18 @@ html_theme_path = ["themes"]
 # redirects dictionary maps from old links to new links
 html_additional_pages = {}
 redirects = {
-    "index": "langchain_api_reference",
+    "index": "api_reference",
 }
 for old_link in redirects:
     html_additional_pages[old_link] = "redirects.html"
 
-partners_dir = Path(__file__).parent.parent.parent / "libs/partners"
-partners = [
-    (p.name, p.name.replace("-", "_") + "_api_reference")
-    for p in partners_dir.iterdir()
-]
-
 html_context = {
     "display_github": True,  # Integrate GitHub
-    "github_user": "langchain-ai",  # Username
+    "github_user": "hwchase17",  # Username
     "github_repo": "langchain",  # Repo name
     "github_version": "master",  # Version
     "conf_py_path": "/docs/api_reference",  # Path in the checkout to the docs root
     "redirects": redirects,
-    "partners": partners,
 }
 
 # Add any paths that contain custom static files (such as style sheets) here,

docs/api_reference/create_api_rst.py

@@ -1,18 +1,21 @@
 """Script for auto-generating api_reference.rst."""
 import importlib
 import inspect
-import os
 import typing
 from enum import Enum
 from pathlib import Path
 from typing import Dict, List, Literal, Optional, Sequence, TypedDict, Union
 
-import toml
 from pydantic import BaseModel
 
 ROOT_DIR = Path(__file__).parents[2].absolute()
 HERE = Path(__file__).parent
 
+PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
+EXP_DIR = ROOT_DIR / "libs" / "experimental" / "langchain_experimental"
+WRITE_FILE = HERE / "api_reference.rst"
+EXP_WRITE_FILE = HERE / "experimental_api_reference.rst"
+
 
 ClassKind = Literal["TypedDict", "Regular", "Pydantic", "enum"]
 
@@ -191,15 +194,11 @@ def _load_package_modules(
     return modules_by_namespace
 
 
-def _construct_doc(
-    package_namespace: str,
-    members_by_namespace: Dict[str, ModuleMembers],
-    package_version: str,
-) -> str:
+def _construct_doc(pkg: str, members_by_namespace: Dict[str, ModuleMembers]) -> str:
     """Construct the contents of the reference.rst file for the given package.
 
     Args:
-        package_namespace: The package top level namespace
+        pkg: The package name
         members_by_namespace: The members of the package, dict organized by top level
                               module contains a list of classes and functions
                               inside of the top level namespace.
@@ -209,7 +208,7 @@ def _construct_doc(
     """
     full_doc = f"""\
 =======================
-``{package_namespace}`` {package_version}
+``{pkg}`` API Reference
 =======================
 
 """
@@ -221,13 +220,13 @@ def _construct_doc(
         functions = _members["functions"]
         if not (classes or functions):
             continue
-        section = f":mod:`{package_namespace}.{module}`"
+        section = f":mod:`{pkg}.{module}`"
         underline = "=" * (len(section) + 1)
         full_doc += f"""\
 {section}
 {underline}
 
-.. automodule:: {package_namespace}.{module}
+.. automodule:: {pkg}.{module}
     :no-members:
     :no-inherited-members:
 
@@ -237,7 +236,7 @@ def _construct_doc(
             full_doc += f"""\
 Classes
 --------------
-.. currentmodule:: {package_namespace}
+.. currentmodule:: {pkg}
 
 .. autosummary::
     :toctree: {module}
@@ -269,7 +268,7 @@ Classes
             full_doc += f"""\
 Functions
 --------------
-.. currentmodule:: {package_namespace}
+.. currentmodule:: {pkg}
 
 .. autosummary::
     :toctree: {module}
@@ -281,71 +280,46 @@ Functions
     return full_doc
 
 
-def _build_rst_file(package_name: str = "langchain") -> None:
-    """Create a rst file for building of documentation.
-
-    Args:
-        package_name: Can be either "langchain" or "core" or "experimental".
-    """
-    package_dir = _package_dir(package_name)
-    package_members = _load_package_modules(package_dir)
-    package_version = _get_package_version(package_dir)
-    with open(_out_file_path(package_name), "w") as f:
-        f.write(
-            _doc_first_line(package_name)
-            + _construct_doc(
-                _package_namespace(package_name), package_members, package_version
-            )
-        )
+def _document_langchain_experimental() -> None:
+    """Document the langchain_experimental package."""
+    # Generate experimental_api_reference.rst
+    exp_members = _load_package_modules(EXP_DIR)
+    exp_doc = ".. _experimental_api_reference:\n\n" + _construct_doc(
+        "langchain_experimental", exp_members
+    )
+    with open(EXP_WRITE_FILE, "w") as f:
+        f.write(exp_doc)
 
 
-def _package_namespace(package_name: str) -> str:
-    return (
-        package_name
-        if package_name == "langchain"
-        else f"langchain_{package_name.replace('-', '_')}"
+def _document_langchain_core() -> None:
+    """Document the main langchain package."""
+    # load top level module members
+    lc_members = _load_package_modules(PKG_DIR)
+
+    # Add additional packages
+    tools = _load_package_modules(PKG_DIR, "tools")
+    agents = _load_package_modules(PKG_DIR, "agents")
+    schema = _load_package_modules(PKG_DIR, "schema")
+
+    lc_members.update(
+        {
+            "agents.output_parsers": agents["output_parsers"],
+            "agents.format_scratchpad": agents["format_scratchpad"],
+            "tools.render": tools["render"],
+            "schema.runnable": schema["runnable"],
+        }
     )
 
+    lc_doc = ".. _api_reference:\n\n" + _construct_doc("langchain", lc_members)
 
-def _package_dir(package_name: str = "langchain") -> Path:
-    """Return the path to the directory containing the documentation."""
-    if package_name in ("langchain", "experimental", "community", "core", "cli"):
-        return ROOT_DIR / "libs" / package_name / _package_namespace(package_name)
-    else:
-        return (
-            ROOT_DIR
-            / "libs"
-            / "partners"
-            / package_name
-            / _package_namespace(package_name)
-        )
-
-
-def _get_package_version(package_dir: Path) -> str:
-    with open(package_dir.parent / "pyproject.toml", "r") as f:
-        pyproject = toml.load(f)
-    return pyproject["tool"]["poetry"]["version"]
-
-
-def _out_file_path(package_name: str = "langchain") -> Path:
-    """Return the path to the file containing the documentation."""
-    return HERE / f"{package_name.replace('-', '_')}_api_reference.rst"
-
-
-def _doc_first_line(package_name: str = "langchain") -> str:
-    """Return the path to the file containing the documentation."""
-    return f".. {package_name.replace('-', '_')}_api_reference:\n\n"
+    with open(WRITE_FILE, "w") as f:
+        f.write(lc_doc)
 
 
 def main() -> None:
-    """Generate the api_reference.rst file for each package."""
-    for dir in os.listdir(ROOT_DIR / "libs"):
-        if dir in ("cli", "partners"):
-            continue
-        else:
-            _build_rst_file(package_name=dir)
-    for dir in os.listdir(ROOT_DIR / "libs" / "partners"):
-        _build_rst_file(package_name=dir)
+    """Generate the reference.rst file for each package."""
+    _document_langchain_core()
+    _document_langchain_experimental()
 
 
 if __name__ == "__main__":

docs/api_reference/requirements.txt

@@ -1,7 +1,5 @@
--e libs/experimental
 -e libs/langchain
--e libs/core
--e libs/community
+-e libs/experimental
 pydantic<2
 autodoc_pydantic==1.8.0
 myst_parser
@@ -14,4 +12,4 @@ sphinx-panels
 toml
 myst_nb
 sphinx_copybutton
-pydata-sphinx-theme==0.13.1
+pydata-sphinx-theme==0.13.1

docs/api_reference/themes/scikit-learn-modern/layout.html

@@ -63,6 +63,13 @@
             <a href="#" role="button" class="btn sk-btn-rellink py-1 disabled"">Next</a>
           {%- endif %}
         </div>
+        {%- if pagename != "install" %}
+        <div class="alert alert-warning p-1 mb-2" role="alert">
+          <p class="text-center mb-0">
+          <strong>LangChain {{ release }}</strong><br/>
+          </p>
+        </div>
+        {%- endif %}
             {%- if meta and meta['parenttoc']|tobool %}
             <div class="sk-sidebar-toc">
             {% set nav = get_nav_object(maxdepth=3, collapse=True, numbered=True) %}

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -32,33 +32,19 @@
     <div class="sk-navbar-collapse collapse navbar-collapse" id="navbarSupportedContent">
       <ul class="navbar-nav mr-auto">
         <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('langchain_api_reference') }}">LangChain</a>
-        </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('core_api_reference') }}">Core</a>
-        </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('community_api_reference') }}">Community</a>
+          <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
         </li>
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('experimental_api_reference') }}">Experimental</a>
         </li>
-        {%- for title, pathname in partners %}
         <li class="nav-item">
-          <a class="sk-nav-link nav-link nav-more-item-mobile-items" href="{{ pathto(pathname) }}">{{ title }}</a>
+          <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
+        </li>
+        {%- for title, link, link_attrs in drop_down_navigation %}
+        <li class="nav-item">
+          <a class="sk-nav-link nav-link nav-more-item-mobile-items" href="{{ link }}" {{ link_attrs }}>{{ title }}</a>
         </li>
         {%- endfor %}
-        <li class="nav-item dropdown nav-more-item-dropdown">
-          <a class="sk-nav-link nav-link dropdown-toggle" href="#" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Partner libs</a>
-          <div class="dropdown-menu" aria-labelledby="navbarDropdown">
-            {%- for title, pathname in partners %}
-              <a class="sk-nav-dropdown-item dropdown-item" href="{{ pathto(pathname) }}">{{ title }}</a>
-            {%- endfor %}
-          </div>
-        </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Docs</a>
-        </li>
       </ul>
       {%- if pagename != "search"%}
       <div id="searchbox" role="search">

docs/docs/_templates/integration.mdx

@@ -32,7 +32,7 @@ There isn't any special setup for it.
 See a [usage example](/docs/integrations/llms/INCLUDE_REAL_NAME).
 
 ```python
-from langchain_community.llms import integration_class_REPLACE_ME
+from langchain.llms import integration_class_REPLACE_ME
 ```
 
 ## Text Embedding Models
@@ -40,7 +40,7 @@ from langchain_community.llms import integration_class_REPLACE_ME
 See a [usage example](/docs/integrations/text_embedding/INCLUDE_REAL_NAME)
 
 ```python
-from langchain_community.embeddings import integration_class_REPLACE_ME
+from langchain.embeddings import integration_class_REPLACE_ME
 ```
 
 ## Chat models
@@ -48,7 +48,7 @@ from langchain_community.embeddings import integration_class_REPLACE_ME
 See a [usage example](/docs/integrations/chat/INCLUDE_REAL_NAME)
 
 ```python
-from langchain_community.chat_models import integration_class_REPLACE_ME
+from langchain.chat_models import integration_class_REPLACE_ME
 ```
 
 ## Document Loader
@@ -56,5 +56,5 @@ from langchain_community.chat_models import integration_class_REPLACE_ME
 See a [usage example](/docs/integrations/document_loaders/INCLUDE_REAL_NAME).
 
 ```python
-from langchain_community.document_loaders import integration_class_REPLACE_ME
+from langchain.document_loaders import integration_class_REPLACE_ME
 ```