x

.github/CONTRIBUTING.md

@@ -44,7 +44,7 @@ If you are adding an issue, please try to keep it focused on a single, modular b
 If two issues are related, or blocking, please link them rather than combining them.
 
 We will try to keep these issues as up to date as possible, though
-with the rapid rate of development in this field some may get out of date.
+with the rapid rate of develop in this field some may get out of date.
 If you notice this happening, please let us know.
 
 ### 🙋Getting Help
@@ -80,14 +80,14 @@ For example, to contribute to `langchain` run `cd libs/langchain` before getting
 To install requirements:
 
 ```bash
-poetry install --with test
+poetry install -E all
 ```
 
-This will install all requirements for running the package, examples, linting, formatting, tests, and coverage.
+This will install all requirements for running the package, examples, linting, formatting, tests, and coverage. Note the `-E all` flag will install all optional dependencies necessary for integration testing.
 
 ❗Note: If during installation you receive a `WheelFileValidationError` for `debugpy`, please make sure you are running Poetry v1.5.1. This bug was present in older versions of Poetry (e.g. 1.4.1) and has been resolved in newer releases. If you are still seeing this bug on v1.5.1, you may also try disabling "modern installation" (`poetry config installer.modern-installation false`) and re-installing requirements. See [this `debugpy` issue](https://github.com/microsoft/debugpy/issues/1246) for more details.
 
-Now assuming `make` and `pytest` are installed, you should be able to run the common tasks in the following section. To double check, run `make test` under `libs/langchain`, all tests should pass. If they don't, you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
+Now, you should be able to run the common tasks in the following section. To double check, run `make test`, all tests should pass. If they don't you may need to pip install additional dependencies, such as `numexpr` and `openapi_schema_pydantic`.
 
 ## ✅ Common Tasks
 
@@ -134,7 +134,7 @@ We recognize linting can be annoying - if you do not want to do it, please conta
 ### Spellcheck
 
 Spellchecking for this project is done via [codespell](https://github.com/codespell-project/codespell).
-Note that `codespell` finds common typos, so it could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
+Note that `codespell` finds common typos, so could have false-positive (correctly spelled but rarely used) and false-negatives (not finding misspelled) words.
 
 To check spelling for this project:
 

.github/actions/poetry_setup/action.yml

@@ -15,77 +15,70 @@ inputs:
     description: Poetry version
     required: true
 
+  install-command:
+    description: Command run for installing dependencies
+    required: false
+    default: poetry install
+
   cache-key:
     description: Cache key to use for manual handling of caching
     required: true
 
   working-directory:
-    description: Directory whose poetry.lock file should be cached
-    required: true
+    description: Directory to run install-command in
+    required: false
+    default: ""
 
 runs:
   using: composite
   steps:
     - uses: actions/setup-python@v4
-      name: Setup python ${{ inputs.python-version }}
+      name: Setup python $${ inputs.python-version }}
       with:
         python-version: ${{ inputs.python-version }}
 
     - uses: actions/cache@v3
-      id: cache-bin-poetry
-      name: Cache Poetry binary - Python ${{ inputs.python-version }}
+      id: cache-pip
+      name: Cache Pip ${{ inputs.python-version }}
       env:
-        SEGMENT_DOWNLOAD_TIMEOUT_MIN: "1"
+        SEGMENT_DOWNLOAD_TIMEOUT_MIN: "4"
       with:
         path: |
-          /opt/pipx/venvs/poetry
-        # This step caches the poetry installation, so make sure it's keyed on the poetry version as well.
-        key: bin-poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-${{ inputs.poetry-version }}
-
-    - name: Refresh shell hashtable and fixup softlinks
-      if: steps.cache-bin-poetry.outputs.cache-hit == 'true'
-      shell: bash
-      env:
-        POETRY_VERSION: ${{ inputs.poetry-version }}
-        PYTHON_VERSION: ${{ inputs.python-version }}
-      run: |
-        set -eux
-
-        # Refresh the shell hashtable, to ensure correct `which` output.
-        hash -r
-
-        # `actions/cache@v3` doesn't always seem able to correctly unpack softlinks.
-        # Delete and recreate the softlinks pipx expects to have.
-        rm /opt/pipx/venvs/poetry/bin/python
-        cd /opt/pipx/venvs/poetry/bin
-        ln -s "$(which "python$PYTHON_VERSION")" python
-        chmod +x python
-        cd /opt/pipx_bin/
-        ln -s /opt/pipx/venvs/poetry/bin/poetry poetry
-        chmod +x poetry
-
-        # Ensure everything got set up correctly.
-        /opt/pipx/venvs/poetry/bin/python --version
-        /opt/pipx_bin/poetry --version
+          ~/.cache/pip
+        key: pip-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}
 
     - name: Install poetry
-      if: steps.cache-bin-poetry.outputs.cache-hit != 'true'
       shell: bash
       env:
         POETRY_VERSION: ${{ inputs.poetry-version }}
         PYTHON_VERSION: ${{ inputs.python-version }}
       run: pipx install "poetry==$POETRY_VERSION" --python "python$PYTHON_VERSION" --verbose
 
-    - name: Restore pip and poetry cached dependencies
-      uses: actions/cache@v3
+    - name: Check Poetry File
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        poetry check
+
+    - name: Check lock file
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      run: |
+        poetry lock --check
+
+    - uses: actions/cache@v3
+      id: cache-poetry
       env:
         SEGMENT_DOWNLOAD_TIMEOUT_MIN: "4"
         WORKDIR: ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}
       with:
         path: |
-          ~/.cache/pip
           ~/.cache/pypoetry/virtualenvs
           ~/.cache/pypoetry/cache
           ~/.cache/pypoetry/artifacts
           ${{ env.WORKDIR }}/.venv
-        key: py-deps-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles(format('{0}/**/poetry.lock', env.WORKDIR)) }}
+        key: poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles(format('{0}/poetry.lock', env.WORKDIR)) }}
+
+    - run: ${{ inputs.install-command }}
+      working-directory: ${{ inputs.working-directory }}
+      shell: bash

.github/workflows/_lint.yml

@@ -80,40 +80,31 @@ jobs:
           find "$WORKDIR" -name '*.py' -type f -not -newermt "$OLDEST_COMMIT_TIME" -exec touch -c -m -t '200001010000' '{}' '+'
 
           echo "oldest-commit=$OLDEST_COMMIT" >> "$GITHUB_OUTPUT"
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
+      - uses: actions/cache@v3
+        id: cache-pip
+        name: Cache langchain editable pip install - ${{ matrix.python-version }}
+        env:
+          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "4"
+        with:
+          path: |
+            ~/.cache/pip
+          key: pip-editable-langchain-deps-${{ runner.os }}-${{ runner.arch }}-py-${{ matrix.python-version }}
+      - name: Install poetry
+        run: |
+          pipx install "poetry==$POETRY_VERSION"
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        env:
+          SEGMENT_DOWNLOAD_TIMEOUT_MIN: "4"
         with:
           python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: lint-with-extras
-
-      - name: Check Poetry File
-        shell: bash
-        working-directory: ${{ inputs.working-directory }}
-        run: |
-          poetry check
-
-      - name: Check lock file
-        shell: bash
-        working-directory: ${{ inputs.working-directory }}
-        run: |
-          poetry lock --check
-
+          cache: poetry
+          cache-dependency-path: |
+            ${{ env.WORKDIR }}/**/poetry.lock
       - name: Install dependencies
-        # Also installs dev/lint/test/typing dependencies, to ensure we have
-        # type hints for as many of our libraries as possible.
-        # This helps catch errors that require dependencies to be spotted, for example:
-        # https://github.com/langchain-ai/langchain/pull/10249/files#diff-935185cd488d015f026dcd9e19616ff62863e8cde8c0bee70318d3ccbca98341
-        #
-        # If you change this configuration, make sure to change the `cache-key`
-        # in the `poetry_setup` action above to stop using the old cache.
-        # It doesn't matter how you change it, any change will cause a cache-bust.
         working-directory: ${{ inputs.working-directory }}
         run: |
-          poetry install --with dev,lint,test,typing
-
+          poetry install
       - name: Install langchain editable
         working-directory: ${{ inputs.working-directory }}
         if: ${{ inputs.working-directory != 'libs/langchain' }}

.github/workflows/_pydantic_compatibility.yml

@@ -27,19 +27,14 @@ jobs:
     name: Pydantic v1/v2 compatibility - Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
+      - name: Set up Python ${{ matrix.python-version }}
         uses: "./.github/actions/poetry_setup"
         with:
           python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
           working-directory: ${{ inputs.working-directory }}
+          poetry-version: ${{ env.POETRY_VERSION }}
           cache-key: pydantic-cross-compat
-
-      - name: Install dependencies
-        shell: bash
-        run: poetry install
-
+          install-command: poetry install
       - name: Install the opposite major version of pydantic
         # If normal tests use pydantic v1, here we'll use v2, and vice versa.
         shell: bash
@@ -79,15 +74,3 @@ jobs:
       - name: Run pydantic compatibility tests
         shell: bash
         run: make test
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/_release.yml

@@ -31,15 +31,13 @@ jobs:
         working-directory: ${{ inputs.working-directory }}
     steps:
       - uses: actions/checkout@v3
-
-      - name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
+      - name: Install poetry
+        run: pipx install "poetry==$POETRY_VERSION"
+      - name: Set up Python 3.10
+        uses: actions/setup-python@v4
         with:
           python-version: "3.10"
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: release
-
+          cache: "poetry"
       - name: Build project for distribution
         run: poetry build
       - name: Check Version

.github/workflows/_test.yml

@@ -27,31 +27,14 @@ jobs:
     name: Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
+      - name: Set up Python ${{ matrix.python-version }}
         uses: "./.github/actions/poetry_setup"
         with:
           python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
           working-directory: ${{ inputs.working-directory }}
+          poetry-version: ${{ env.POETRY_VERSION }}
           cache-key: core
-
-      - name: Install dependencies
-        shell: bash
-        run: poetry install
-
+          install-command: poetry install
       - name: Run core tests
         shell: bash
         run: make test
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/langchain_ci.yml

@@ -6,8 +6,6 @@ on:
     branches: [ master ]
   pull_request:
     paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
       - '.github/workflows/_lint.yml'
       - '.github/workflows/_test.yml'
       - '.github/workflows/_pydantic_compatibility.yml'
@@ -15,16 +13,6 @@ on:
       - 'libs/langchain/**'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
 env:
   POETRY_VERSION: "1.5.1"
   WORKDIR: "libs/langchain"
@@ -66,32 +54,15 @@ jobs:
     name: Python ${{ matrix.python-version }} extended tests
     steps:
       - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
+      - name: Set up Python ${{ matrix.python-version }}
         uses: "./.github/actions/poetry_setup"
         with:
           python-version: ${{ matrix.python-version }}
+          working-directory: ${{ env.WORKDIR }}
           poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: libs/langchain
           cache-key: extended
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          echo "Running extended tests, installing dependencies with poetry..."
-          poetry install -E extended_testing
-
+          install-command: |
+            echo "Running extended tests, installing dependencies with poetry..."
+            poetry install -E extended_testing
       - name: Run extended tests
         run: make extended_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/langchain_experimental_ci.yml

@@ -6,8 +6,6 @@ on:
     branches: [ master ]
   pull_request:
     paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
       - '.github/workflows/_lint.yml'
       - '.github/workflows/_test.yml'
       - '.github/workflows/langchain_experimental_ci.yml'
@@ -15,20 +13,6 @@ on:
       - 'libs/experimental/**'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-env:
-  POETRY_VERSION: "1.5.1"
-  WORKDIR: "libs/experimental"
-
 jobs:
   lint:
     uses:
@@ -36,94 +20,9 @@ jobs:
     with:
       working-directory: libs/experimental
     secrets: inherit
-
   test:
     uses:
       ./.github/workflows/_test.yml
     with:
       working-directory: libs/experimental
     secrets: inherit
-
-  # It's possible that langchain-experimental works fine with the latest *published* langchain,
-  # but is broken with the langchain on `master`.
-  #
-  # We want to catch situations like that *before* releasing a new langchain, hence this test.
-  test-with-latest-langchain:
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        working-directory: ${{ env.WORKDIR }}
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: test with unpublished langchain - Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ env.WORKDIR }}
-          cache-key: unpublished-langchain
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          echo "Running tests with unpublished langchain, installing dependencies with poetry..."
-          poetry install
-
-          echo "Editably installing langchain outside of poetry, to avoid messing up lockfile..."
-          poetry run pip install -e ../langchain
-
-      - name: Run tests
-        run: make test
-  extended-tests:
-    runs-on: ubuntu-latest
-    defaults:
-      run:
-        working-directory: ${{ env.WORKDIR }}
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }} extended tests
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: libs/experimental
-          cache-key: extended
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          echo "Running extended tests, installing dependencies with poetry..."
-          poetry install -E extended_testing
-
-      - name: Run extended tests
-        run: make extended_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/scheduled_test.yml

@@ -25,37 +25,18 @@ jobs:
     name: Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v3
-
       - name: Set up Python ${{ matrix.python-version }}
         uses: "./.github/actions/poetry_setup"
         with:
           python-version: ${{ matrix.python-version }}
           poetry-version: ${{ env.POETRY_VERSION }}
           working-directory: libs/langchain
-          cache-key: scheduled
-
-      - name: Install dependencies
-        working-directory: libs/langchain
-        shell: bash
-        run: |
-          echo "Running scheduled tests, installing dependencies with poetry..."
-          poetry install --with=test_integration
-
+          install-command: |
+            echo "Running scheduled tests, installing dependencies with poetry..."
+            poetry install --with=test_integration
       - name: Run tests
-        shell: bash
         env:
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
         run: |
           make scheduled_tests
-
-      - name: Ensure the tests did not create any additional files
         shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

docs/api_reference/conf.py

@@ -156,7 +156,7 @@ html_context = {
 html_static_path = ["_static"]
 
 # These paths are either relative to html_static_path
-# or fully qualified paths (e.g. https://...)
+# or fully qualified paths (eg. https://...)
 html_css_files = [
     "css/custom.css",
 ]

docs/api_reference/create_api_rst.py

@@ -228,7 +228,7 @@ Classes
     :toctree: {module}
 """
 
-            for class_ in sorted(classes, key=lambda c: c["qualified_name"]):
+            for class_ in classes:
                 if not class_["is_public"]:
                     continue
 

docs/api_reference/guide_imports.json

@@ -317,7 +317,7 @@
         "Chatbots": "https://python.langchain.com/docs/use_cases/chatbots",
         "Summarization": "https://python.langchain.com/docs/use_cases/summarization",
         "Extraction": "https://python.langchain.com/docs/use_cases/extraction",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "Tagging": "https://python.langchain.com/docs/use_cases/tagging",
         "Code Understanding": "https://python.langchain.com/docs/use_cases/code_understanding",
         "AutoGPT": "https://python.langchain.com/docs/use_cases/autonomous_agents/autogpt",
@@ -338,11 +338,10 @@
         "Neptune Open Cypher QA Chain": "https://python.langchain.com/docs/use_cases/more/graph/neptune_cypher_qa",
         "NebulaGraphQAChain": "https://python.langchain.com/docs/use_cases/more/graph/graph_nebula_qa",
         "KuzuQAChain": "https://python.langchain.com/docs/use_cases/more/graph/graph_kuzu_qa",
-        "FalkorDBQAChain": "https://python.langchain.com/docs/use_cases/more/graph/graph_falkordb_qa",
         "HugeGraph QA Chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_hugegraph_qa",
         "GraphSparqlQAChain": "https://python.langchain.com/docs/use_cases/more/graph/graph_sparql_qa",
         "ArangoDB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_arangodb_qa",
-        "Neo4j DB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_cypher_qa",
+        "Graph DB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_cypher_qa",
         "How to use a SmartLLMChain": "https://python.langchain.com/docs/use_cases/more/self_check/smart_llm",
         "Multi-Agent Simulated Environment: Petting Zoo": "https://python.langchain.com/docs/use_cases/agent_simulations/petting_zoo",
         "Multi-agent decentralized speaker selection": "https://python.langchain.com/docs/use_cases/agent_simulations/multiagent_bidding",
@@ -400,7 +399,7 @@
         "Summarization": "https://python.langchain.com/docs/use_cases/summarization",
         "Extraction": "https://python.langchain.com/docs/use_cases/extraction",
         "Interacting with APIs": "https://python.langchain.com/docs/use_cases/apis",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "QA over Documents": "https://python.langchain.com/docs/use_cases/question_answering/index",
         "Retrieve from vector stores directly": "https://python.langchain.com/docs/use_cases/question_answering/how_to/vector_db_text_generation",
         "Improve document indexing with HyDE": "https://python.langchain.com/docs/use_cases/question_answering/how_to/hyde",
@@ -641,7 +640,7 @@
         "Chatbots": "https://python.langchain.com/docs/use_cases/chatbots",
         "Extraction": "https://python.langchain.com/docs/use_cases/extraction",
         "Interacting with APIs": "https://python.langchain.com/docs/use_cases/apis",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "HuggingGPT": "https://python.langchain.com/docs/use_cases/autonomous_agents/hugginggpt",
         "Perform context-aware text splitting": "https://python.langchain.com/docs/use_cases/question_answering/how_to/document-context-aware-QA",
         "Retrieve from vector stores directly": "https://python.langchain.com/docs/use_cases/question_answering/how_to/vector_db_text_generation",
@@ -1009,7 +1008,7 @@
         "LangSmith Walkthrough": "https://python.langchain.com/docs/guides/langsmith/walkthrough",
         "Comparing Chain Outputs": "https://python.langchain.com/docs/guides/evaluation/examples/comparisons",
         "Agent Trajectory": "https://python.langchain.com/docs/guides/evaluation/trajectory/trajectory_eval",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "Multi-modal outputs: Image & Text": "https://python.langchain.com/docs/use_cases/multi_modal/image_agent",
         "Agent Debates with Tools": "https://python.langchain.com/docs/use_cases/agent_simulations/two_agent_debate_tools",
         "Multiple callback handlers": "https://python.langchain.com/docs/modules/callbacks/multiple_callbacks",
@@ -1268,7 +1267,7 @@
         "SQL Database Agent": "https://python.langchain.com/docs/integrations/toolkits/sql_database",
         "JSON Agent": "https://python.langchain.com/docs/integrations/toolkits/json",
         "NIBittensorLLM": "https://python.langchain.com/docs/integrations/llms/bittensor",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "BabyAGI with Tools": "https://python.langchain.com/docs/use_cases/agents/baby_agi_with_agent",
         "Conversational Retrieval Agent": "https://python.langchain.com/docs/use_cases/question_answering/how_to/conversational_retrieval_agents",
         "Plug-and-Plai": "https://python.langchain.com/docs/use_cases/agents/custom_agent_with_plugin_retrieval_using_plugnplai",
@@ -1832,12 +1831,12 @@
     "create_sql_agent": {
         "CnosDB": "https://python.langchain.com/docs/integrations/providers/cnosdb",
         "SQL Database Agent": "https://python.langchain.com/docs/integrations/toolkits/sql_database",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql"
+        "SQL": "https://python.langchain.com/docs/use_cases/sql"
     },
     "SQLDatabaseToolkit": {
         "CnosDB": "https://python.langchain.com/docs/integrations/providers/cnosdb",
         "SQL Database Agent": "https://python.langchain.com/docs/integrations/toolkits/sql_database",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "Use ToolKits with OpenAI Functions": "https://python.langchain.com/docs/modules/agents/how_to/use_toolkits_with_openai_functions"
     },
     "SageMakerCallbackHandler": {
@@ -1899,7 +1898,7 @@
         "Rebuff": "https://python.langchain.com/docs/integrations/providers/rebuff",
         "SQL Database Agent": "https://python.langchain.com/docs/integrations/toolkits/sql_database",
         "Cookbook": "https://python.langchain.com/docs/guides/expression_language/cookbook",
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "Multiple Retrieval Sources": "https://python.langchain.com/docs/use_cases/question_answering/how_to/multiple_retrieval"
     },
     "Weaviate": {
@@ -2072,8 +2071,8 @@
         "PromptLayer": "https://python.langchain.com/docs/integrations/providers/promptlayer",
         "PromptLayer OpenAI": "https://python.langchain.com/docs/integrations/llms/promptlayer_openai"
     },
-    "Activeloop DeepLake": {
-        "Deep Lake": "https://python.langchain.com/docs/integrations/providers/activeloop_deeplake",
+    "DeepLake": {
+        "Deep Lake": "https://python.langchain.com/docs/integrations/providers/deeplake",
         "Activeloop's Deep Lake": "https://python.langchain.com/docs/integrations/vectorstores/activeloop_deeplake",
         "Analysis of Twitter the-algorithm source code with LangChain, GPT4 and Activeloop's Deep Lake": "https://python.langchain.com/docs/use_cases/question_answering/how_to/code/twitter-the-algorithm-analysis-deeplake",
         "Use LangChain, GPT and Activeloop's Deep Lake to work with code base": "https://python.langchain.com/docs/use_cases/question_answering/how_to/code/code-analysis-deeplake",
@@ -3035,11 +3034,11 @@
         "Interacting with APIs": "https://python.langchain.com/docs/use_cases/apis"
     },
     "create_sql_query_chain": {
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql",
+        "SQL": "https://python.langchain.com/docs/use_cases/sql",
         "Multiple Retrieval Sources": "https://python.langchain.com/docs/use_cases/question_answering/how_to/multiple_retrieval"
     },
     "ElasticsearchDatabaseChain": {
-        "SQL": "https://python.langchain.com/docs/use_cases/qa_structured/sql"
+        "SQL": "https://python.langchain.com/docs/use_cases/sql"
     },
     "FileChatMessageHistory": {
         "AutoGPT": "https://python.langchain.com/docs/use_cases/autonomous_agents/autogpt"
@@ -3175,12 +3174,6 @@
     "KuzuQAChain": {
         "KuzuQAChain": "https://python.langchain.com/docs/use_cases/more/graph/graph_kuzu_qa"
     },
-    "FalkorDBGraph": {
-        "KuzuQAChain": "https://python.langchain.com/docs/use_cases/more/graph/graph_falkordb_qa"
-    },
-    "FalkorDBQAChain": {
-        "FalkorDB QA Chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_falkordb_qa"
-    },
     "HugeGraphQAChain": {
         "HugeGraph QA Chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_hugegraph_qa"
     },
@@ -3209,10 +3202,10 @@
         "Graph QA": "https://python.langchain.com/docs/use_cases/more/graph/graph_qa"
     },
     "GraphCypherQAChain": {
-        "Neo4j DB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_cypher_qa"
+        "Graph DB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_cypher_qa"
     },
     "Neo4jGraph": {
-        "Neo4j DB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_cypher_qa"
+        "Graph DB QA chain": "https://python.langchain.com/docs/use_cases/more/graph/graph_cypher_qa"
     },
     "LLMBashChain": {
         "Bash chain": "https://python.langchain.com/docs/use_cases/more/code_writing/llm_bash"

docs/api_reference/templates/redirects.html

@@ -5,10 +5,9 @@
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <meta http-equiv="Refresh" content="0; url={{ redirect }}" />
-    <meta name="robots" content="follow, index">
-    <meta name="Description" content="Python API reference for LangChain.">
+    <meta name="Description" content="scikit-learn: machine learning in Python">
     <link rel="canonical" href="{{ redirect }}" />
-    <title>LangChain Python API Reference Documentation.</title>
+    <title>scikit-learn: machine learning in Python</title>
   </head>
   <body>
     <p>You will be automatically redirected to the <a href="{{ redirect }}">new location of this page</a>.</p>

docs/docs_skeleton/docs/community.md

@@ -47,8 +47,8 @@ If you’re working on something you’re proud of, and think the LangChain comm
 
 Here’s where our team hangs out, talks shop, spotlights cool work, and shares what we’re up to. We’d love to see you there too.
 
-- **[Twitter](https://twitter.com/LangChainAI):** we post about what we’re working on and what cool things we’re seeing in the space. If you tag @langchainai in your post, we’ll almost certainly see it, and can show you some love!
-- **[Discord](https://discord.gg/6adMQxSpJS):** connect with >30k developers who are building with LangChain
+- **[Twitter](https://twitter.com/LangChainAI):** we post about what we’re working on and what cool things we’re seeing in the space. If you tag @langchainai in your post, we’ll almost certainly see it, and can snow you some love!
+- **[Discord](https://discord.gg/6adMQxSpJS):** connect with with >30k developers who are building with LangChain
 - **[GitHub](https://github.com/langchain-ai/langchain):** open pull requests, contribute to a discussion, and/or contribute
 - **[Subscribe to our bi-weekly Release Notes](https://6w1pwbss0py.typeform.com/to/KjZB1auB):** a twice/month email roundup of the coolest things going on in our orbit
 - **Slack:** if you’re building an application in production at your company, we’d love to get into a Slack channel together. Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) and we’ll get in touch about setting one up.

docs/docs_skeleton/docs/expression_language/index.mdx

@@ -1,14 +0,0 @@
----
-sidebar_class_name: hidden
----
-
-# LangChain Expression Language (LCEL)
-
-LangChain Expression Language or LCEL is a declarative way to easily compose chains together.
-Any chain constructed this way will automatically have full sync, async, and streaming support.
-
-#### [Interface](/docs/expression_language/interface)
-The base interface shared by all LCEL objects
-
-#### [Cookbook](/docs/expression_language/cookbook)
-Examples of common LCEL usage patterns

docs/docs_skeleton/docs/get_started/introduction.mdx

@@ -42,22 +42,23 @@ Log and stream intermediate steps of any chain
 ## Examples, ecosystem, and resources
 ### [Use cases](/docs/use_cases/)
 Walkthroughs and best-practices for common end-to-end use cases, like:
-- [Chatbots](/docs/use_cases/chatbots)
+- [Chatbots](/docs/use_cases/chatbots/)
 - [Answering questions using sources](/docs/use_cases/question_answering/)
-- [Analyzing structured data](/docs/use_cases/sql)
+- [Analyzing structured data](/docs/use_cases/tabular.html)
 - and much more...
 
 ### [Guides](/docs/guides/)
 Learn best practices for developing with LangChain.
 
 ### [Ecosystem](/docs/ecosystem/)
-LangChain is part of a rich ecosystem of tools that integrate with our framework and build on top of it. Check out our growing list of [integrations](/docs/integrations/) and [dependent repos](/docs/additional_resources/dependents).
+LangChain is part of a rich ecosystem of tools that integrate with our framework and build on top of it. Check out our growing list of [integrations](/docs/integrations/) and [dependent repos](/docs/ecosystem/dependents).
 
 ### [Additional resources](/docs/additional_resources/)
 Our community is full of prolific developers, creative builders, and fantastic teachers. Check out [YouTube tutorials](/docs/additional_resources/youtube.html) for great tutorials from folks in the community, and [Gallery](https://github.com/kyrolabs/awesome-langchain) for a list of awesome LangChain projects, compiled by the folks at [KyroLabs](https://kyrolabs.com).
 
-### [Community](/docs/community)
-Head to the [Community navigator](/docs/community) to find places to ask questions, share feedback, meet other developers, and dream about the future of LLM’s.
+<h3><span style={{color:"#2e8555"}}> Support </span></h3>
+
+Join us on [GitHub](https://github.com/hwchase17/langchain) or [Discord](https://discord.gg/6adMQxSpJS) to ask questions, share feedback, meet other developers building with LangChain, and dream about the future of LLM’s.
 
 ## API reference
 

docs/docs_skeleton/docs/get_started/quickstart.mdx

@@ -59,8 +59,8 @@ LangChain provides several objects to easily distinguish between different roles
 If none of those roles sound right, there is also a `ChatMessage` class where you can specify the role manually.
 For more information on how to use these different messages most effectively, see our prompting guide.
 
-LangChain provides a standard interface for both, but it's useful to understand this difference in order to construct prompts for a given language model.
-The standard interface that LangChain provides has two methods:
+LangChain exposes a standard interface for both, but it's useful to understand this difference in order to construct prompts for a given language model.
+The standard interface that LangChain exposes has two methods:
 - `predict`: Takes in a string, returns a string
 - `predict_messages`: Takes in a list of messages, returns a message.
 
@@ -107,7 +107,7 @@ import PromptTemplateChatModel from "@snippets/get_started/quickstart/prompt_tem
 <PromptTemplateLLM/>
 
 However, the advantages of using these over raw string formatting are several.
-You can "partial" out variables - e.g. you can format only some of the variables at a time.
+You can "partial" out variables - eg you can format only some of the variables at a time.
 You can compose them together, easily combining different templates into a single prompt.
 For explanations of these functionalities, see the [section on prompts](/docs/modules/model_io/prompts) for more detail.
 
@@ -121,12 +121,12 @@ Let's take a look at this below:
 
 ChatPromptTemplates can also include other things besides ChatMessageTemplates - see the [section on prompts](/docs/modules/model_io/prompts) for more detail.
 
-## Output parsers
+## Output Parsers
 
 OutputParsers convert the raw output of an LLM into a format that can be used downstream.
 There are few main type of OutputParsers, including:
 
-- Convert text from LLM -> structured information (e.g. JSON)
+- Convert text from LLM -> structured information (eg JSON)
 - Convert a ChatMessage into just a string
 - Convert the extra information returned from a call besides the message (like OpenAI function invocation) into a string.
 
@@ -149,7 +149,7 @@ import LLMChain from "@snippets/get_started/quickstart/llm_chain.mdx"
 
 <LLMChain/>
 
-## Next steps
+## Next Steps
 
 This is it!
 We've now gone over how to create the core building block of LangChain applications - the LLMChains.

docs/docs_skeleton/docs/guides/evaluation/comparison/index.mdx

@@ -3,7 +3,7 @@ sidebar_position: 3
 ---
 # Comparison Evaluators
 
-Comparison evaluators in LangChain help measure two different chains or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.
+Comparison evaluators in LangChain help measure two different chain or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.
 
 These evaluators inherit from the `PairwiseStringEvaluator` class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.
 
@@ -16,7 +16,7 @@ Here's a summary of the key methods and properties of a comparison evaluator:
 - `requires_input`: This property indicates whether this evaluator requires an input string.
 - `requires_reference`: This property specifies whether this evaluator requires a reference label.
 
-Detailed information about creating custom evaluators and the available built-in comparison evaluators is provided in the following sections.
+Detailed information about creating custom evaluators and the available built-in comparison evaluators are provided in the following sections.
 
 import DocCardList from "@theme/DocCardList";
 

docs/docs_skeleton/docs/guides/evaluation/index.mdx

@@ -1,3 +1,7 @@
+---
+sidebar_position: 6
+---
+
 import DocCardList from "@theme/DocCardList";
 
 # Evaluation

docs/docs_skeleton/docs/guides/expression_language/index.mdx

@@ -0,0 +1,9 @@
+# LangChain Expression Language
+
+import DocCardList from "@theme/DocCardList";
+
+LangChain Expression Language is a declarative way to easily compose chains together.
+Any chain constructed this way will automatically have full sync, async, and streaming support.
+See guides below for how to interact with chains constructed this way as well as cookbook examples.
+
+<DocCardList />

docs/docs_skeleton/docs/guides/langsmith/index.md

@@ -2,21 +2,11 @@
 
 import DocCardList from "@theme/DocCardList";
 
-[LangSmith](https://smith.langchain.com) helps you trace and evaluate your language model applications and intelligent agents to help you
+LangSmith helps you trace and evaluate your language model applications and intelligent agents to help you
 move from prototype to production.
 
 Check out the [interactive walkthrough](/docs/guides/langsmith/walkthrough) below to get started.
 
-For more information, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/).
-
-For tutorials and other end-to-end examples demonstrating ways to integrate LangSmith in your workflow,
-check out the [LangSmith Cookbook](https://github.com/langchain-ai/langsmith-cookbook). Some of the guides therein include:
-
-- Leveraging user feedback in your JS application ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/feedback-examples/nextjs/README.md)).
-- Building an automated feedback pipeline ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/feedback-examples/algorithmic-feedback/algorithmic_feedback.ipynb)).
-- How to evaluate and audit your RAG workflows ([link](https://github.com/langchain-ai/langsmith-cookbook/tree/main/testing-examples/qa-correctness)).
-- How to fine-tune a LLM on real usage data ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb)).
-- How to use the [LangChain Hub](https://smith.langchain.com/hub) to version your prompts ([link](https://github.com/langchain-ai/langsmith-cookbook/blob/main/hub-examples/retrieval-qa-chain/retrieval-qa.ipynb))
-
+For more information, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/)
 
 <DocCardList />

docs/docs_skeleton/docs/guides/safety/index.mdx

@@ -1,8 +1,6 @@
-# Moderation
+# Preventing harmful outputs
 
 One of the key concerns with using LLMs is that they may generate harmful or unethical text. This is an area of active research in the field. Here we present some built-in chains inspired by this research, which are intended to make the outputs of LLMs safer.
 
-- [Moderation chain](/docs/guides/safety/moderation): Explicitly check if any output text is harmful and flag it.
-- [Constitutional chain](/docs/guides/safety/constitutional_chain): Prompt the model with a set of principles which should guide it's behavior.
-- [Logical Fallacy chain](/docs/guides/safety/logical_fallacy_chain): Checks the model output against logical fallacies to correct any deviation.
-- [Amazon Comprehend moderation chain](/docs/guides/safety/amazon_comprehend_chain): Use [Amazon Comprehend](https://aws.amazon.com/comprehend/) to detect and handle PII and toxicity.
+- [Moderation chain](/docs/use_cases/safety/moderation): Explicitly check if any output text is harmful and flag it.
+- [Constitutional chain](/docs/use_cases/safety/constitutional_chain): Prompt the model with a set of principles which should guide it's behavior.

docs/docs_skeleton/docs/guides/safety/logical_fallacy_chain.mdx

@@ -1,85 +0,0 @@
-# Removing logical fallacies from model output
-Logical fallacies are flawed reasoning or false arguments that can undermine the validity of a model's outputs. Examples include circular reasoning, false
-dichotomies, ad hominem attacks, etc.  Machine learning models are optimized to perform well on specific metrics like accuracy, perplexity, or loss. However, 
-optimizing for metrics alone does not guarantee logically sound reasoning.
-
-Language models can learn to exploit flaws in reasoning to generate plausible-sounding but logically invalid arguments.  When models rely on fallacies, their outputs become unreliable and untrustworthy, even if they achieve high scores on metrics. Users cannot depend on such outputs. Propagating logical fallacies can spread misinformation, confuse users, and lead to harmful real-world consequences when models are deployed in products or services.
-
-Monitoring and testing specifically for logical flaws is challenging unlike other quality issues. It requires reasoning about arguments rather than pattern matching.
-
-Therefore, it is crucial that model developers proactively address logical fallacies after optimizing metrics. Specialized techniques like causal modeling, robustness testing, and bias mitigation can help avoid flawed reasoning.  Overall, allowing logical flaws to persist makes models less safe and ethical. Eliminating fallacies ensures model outputs remain logically valid and aligned with human reasoning. This maintains user trust and mitigates risks.
-
-
-
-```python
-# Imports
-from langchain.llms import OpenAI
-from langchain.prompts import PromptTemplate
-from langchain.chains.llm import LLMChain
-from langchain_experimental.fallacy_removal.base import FallacyChain
-```
-
-```python
-# Example of a model output being returned with a logical fallacy
-misleading_prompt = PromptTemplate(
-    template="""You have to respond by using only logical fallacies inherent in your answer explanations.
-
-Question: {question}
-
-Bad answer:""",
-    input_variables=["question"],
-)
-
-llm = OpenAI(temperature=0)
-
-misleading_chain = LLMChain(llm=llm, prompt=misleading_prompt)
-
-misleading_chain.run(question="How do I know the earth is round?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    'The earth is round because my professor said it is, and everyone believes my professor'
-```
-
-</CodeOutputBlock>
-
-
-```python
-fallacies = FallacyChain.get_fallacies(["correction"])
-fallacy_chain = FallacyChain.from_llm(
-    chain=misleading_chain,
-    logical_fallacies=fallacies,
-    llm=llm,
-    verbose=True,
-)
-
-fallacy_chain.run(question="How do I know the earth is round?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new FallacyChain chain...
-    Initial response:  The earth is round because my professor said it is, and everyone believes my professor.
-
-    Applying correction...
-
-    Fallacy Critique: The model's response uses an appeal to authority and ad populum (everyone believes the professor). Fallacy Critique Needed.
-
-    Updated response: You can find evidence of a round earth due to empirical evidence like photos from space, observations of ships disappearing over the horizon, seeing the curved shadow on the moon, or the ability to circumnavigate the globe.
-
-
-    > Finished chain.
-
-
-
-
-
-    'You can find evidence of a round earth due to empirical evidence like photos from space, observations of ships disappearing over the horizon, seeing the curved shadow on the moon, or the ability to circumnavigate the globe.'
-```
-
-</CodeOutputBlock>

docs/docs_skeleton/docs/modules/agents/agent_types/index.mdx

@@ -37,11 +37,11 @@ This agent is designed to be used in conversational settings.
 The prompt is designed to make the agent helpful and conversational.
 It uses the ReAct framework to decide which tool to use, and uses memory to remember the previous conversation interactions.
 
-### [Self-ask with search](/docs/modules/agents/agent_types/self_ask_with_search.html)
+### [Self ask with search](/docs/modules/agents/agent_types/self_ask_with_search.html)
 
 This agent utilizes a single tool that should be named `Intermediate Answer`.
 This tool should be able to lookup factual answers to questions. This agent
-is equivalent to the original [self-ask with search paper](https://ofir.io/self-ask.pdf),
+is equivalent to the original [self ask with search paper](https://ofir.io/self-ask.pdf),
 where a Google search API was provided as the tool.
 
 ### [ReAct document store](/docs/modules/agents/agent_types/react_docstore.html)
@@ -54,4 +54,4 @@ This agent is equivalent to the
 original [ReAct paper](https://arxiv.org/pdf/2210.03629.pdf), specifically the Wikipedia example.
 
 ## [Plan-and-execute agents](/docs/modules/agents/agent_types/plan_and_execute.html)
-Plan-and-execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
+Plan and execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).

docs/docs_skeleton/docs/modules/agents/agent_types/plan_and_execute.mdx

@@ -1,6 +1,6 @@
-# Plan-and-execute
+# Plan and execute
 
-Plan-and-execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
+Plan and execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
 
 The planning is almost always done by an LLM.
 

docs/docs_skeleton/docs/modules/agents/how_to/custom_llm_agent.mdx

@@ -1,13 +1,13 @@
-# Custom LLM agent
+# Custom LLM Agent
 
 This notebook goes through how to create your own custom LLM agent.
 
 An LLM agent consists of three parts:
 
-- `PromptTemplate`: This is the prompt template that can be used to instruct the language model on what to do
+- PromptTemplate: This is the prompt template that can be used to instruct the language model on what to do
 - LLM: This is the language model that powers the agent
 - `stop` sequence: Instructs the LLM to stop generating as soon as this string is found
-- `OutputParser`: This determines how to parse the LLM output into an `AgentAction` or `AgentFinish` object
+- OutputParser: This determines how to parse the LLMOutput into an AgentAction or AgentFinish object
 
 import Example from "@snippets/modules/agents/how_to/custom_llm_agent.mdx"
 

docs/docs_skeleton/docs/modules/agents/how_to/custom_llm_chat_agent.mdx

@@ -4,10 +4,10 @@ This notebook goes through how to create your own custom agent based on a chat m
 
 An LLM chat agent consists of three parts:
 
-- `PromptTemplate`: This is the prompt template that can be used to instruct the language model on what to do
-- `ChatModel`: This is the language model that powers the agent
+- PromptTemplate: This is the prompt template that can be used to instruct the language model on what to do
+- ChatModel: This is the language model that powers the agent
 - `stop` sequence: Instructs the LLM to stop generating as soon as this string is found
-- `OutputParser`: This determines how to parse the LLM output into an `AgentAction` or `AgentFinish` object
+- OutputParser: This determines how to parse the LLMOutput into an AgentAction or AgentFinish object
 
 import Example from "@snippets/modules/agents/how_to/custom_llm_chat_agent.mdx"
 

docs/docs_skeleton/docs/modules/chains/document/index.mdx

@@ -3,7 +3,7 @@ sidebar_position: 2
 ---
 # Documents
 
-These are the core chains for working with documents. They are useful for summarizing documents, answering questions over documents, extracting information from documents, and more.
+These are the core chains for working with Documents. They are useful for summarizing documents, answering questions over documents, extracting information from documents, and more.
 
 These chains all implement a common interface:
 

docs/docs_skeleton/docs/modules/chains/document/refine.mdx

@@ -3,10 +3,10 @@ sidebar_position: 1
 ---
 # Refine
 
-The Refine documents chain constructs a response by looping over the input documents and iteratively updating its answer. For each document, it passes all non-document inputs, the current document, and the latest intermediate answer to an LLM chain to get a new answer.
+The refine documents chain constructs a response by looping over the input documents and iteratively updating its answer. For each document, it passes all non-document inputs, the current document, and the latest intermediate answer to an LLM chain to get a new answer.
 
 Since the Refine chain only passes a single document to the LLM at a time, it is well-suited for tasks that require analyzing more documents than can fit in the model's context.
 The obvious tradeoff is that this chain will make far more LLM calls than, for example, the Stuff documents chain.
 There are also certain tasks which are difficult to accomplish iteratively. For example, the Refine chain can perform poorly when documents frequently cross-reference one another or when a task requires detailed information from many documents.
 
-![refine_diagram](/img/refine.jpg)
+![refine_diagram](/img/refine.jpg)

docs/docs_skeleton/docs/modules/chains/foundational/llm_chain.mdx

@@ -1,11 +1,11 @@
 # LLM
 
-An `LLMChain` is a simple chain that adds some functionality around language models. It is used widely throughout LangChain, including in other chains and agents.
+An LLMChain is a simple chain that adds some functionality around language models. It is used widely throughout LangChain, including in other chains and agents.
 
-An `LLMChain` consists of a `PromptTemplate` and a language model (either an LLM or chat model). It formats the prompt template using the input key values provided (and also memory key values, if available), passes the formatted string to LLM and returns the LLM output.
+An LLMChain consists of a PromptTemplate and a language model (either an LLM or chat model). It formats the prompt template using the input key values provided (and also memory key values, if available), passes the formatted string to LLM and returns the LLM output.
 
 ## Get started
 
 import Example from "@snippets/modules/chains/foundational/llm_chain.mdx"
 
-<Example/>
+<Example/>

docs/docs_skeleton/docs/modules/chains/foundational/sequential_chains.mdx

@@ -4,7 +4,7 @@
 
 The next step after calling a language model is make a series of calls to a language model. This is particularly useful when you want to take the output from one call and use it as the input to another.
 
-In this notebook we will walk through some examples for how to do this, using sequential chains. Sequential chains allow you to connect multiple chains and compose them into pipelines that execute some specific scenario. There are two types of sequential chains:
+In this notebook we will walk through some examples for how to do this, using sequential chains. Sequential chains allow you to connect multiple chains and compose them into pipelines that execute some specific scenario.. There are two types of sequential chains:
 
 - `SimpleSequentialChain`: The simplest form of sequential chains, where each step has a singular input/output, and the output of one step is the input to the next.
 - `SequentialChain`: A more general form of sequential chains, allowing for multiple inputs/outputs.

docs/docs_skeleton/docs/modules/chains/index.mdx

@@ -30,4 +30,4 @@ Chains allow us to combine multiple components together to create a single, cohe
 
 import GetStarted from "@snippets/modules/chains/get_started.mdx"
 
-<GetStarted/>
+<GetStarted/>

docs/docs_skeleton/docs/modules/data_connection/document_loaders/index.mdx

@@ -11,7 +11,7 @@ Use document loaders to load data from a source as `Document`'s. A `Document` is
 and associated metadata. For example, there are document loaders for loading a simple `.txt` file, for loading the text
 contents of any web page, or even for loading a transcript of a YouTube video.
 
-Document loaders provide a "load" method for loading data as documents from a configured source. They optionally
+Document loaders expose a "load" method for loading data as documents from a configured source. They optionally
 implement a "lazy load" as well for lazily loading data into memory.
 
 ## Get started

docs/docs_skeleton/docs/modules/data_connection/document_transformers/text_splitters/character_text_splitter.mdx

@@ -2,8 +2,8 @@
 
 This is the simplest method. This splits based on characters (by default "\n\n") and measure chunk length by number of characters.
 
-1. How the text is split: by single character.
-2. How the chunk size is measured: by number of characters.
+1. How the text is split: by single character
+2. How the chunk size is measured: by number of characters
 
 import Example from "@snippets/modules/data_connection/document_transformers/text_splitters/character_text_splitter.mdx"
 

docs/docs_skeleton/docs/modules/data_connection/document_transformers/text_splitters/code_splitter.mdx

@@ -1,6 +1,6 @@
 # Split code
 
-CodeTextSplitter allows you to split your code with multiple languages supported. Import enum `Language` and specify the language. 
+CodeTextSplitter allows you to split your code with multiple language support. Import enum `Language` and specify the language. 
 
 import Example from "@snippets/modules/data_connection/document_transformers/text_splitters/code_splitter.mdx"
 

docs/docs_skeleton/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter.mdx

@@ -2,8 +2,8 @@
 
 This text splitter is the recommended one for generic text. It is parameterized by a list of characters. It tries to split on them in order until the chunks are small enough. The default list is `["\n\n", "\n", " ", ""]`. This has the effect of trying to keep all paragraphs (and then sentences, and then words) together as long as possible, as those would generically seem to be the strongest semantically related pieces of text.
 
-1. How the text is split: by list of characters.
-2. How the chunk size is measured: by number of characters.
+1. How the text is split: by list of characters
+2. How the chunk size is measured: by number of characters
 
 import Example from "@snippets/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter.mdx"
 

docs/docs_skeleton/docs/modules/data_connection/index.mdx

@@ -18,9 +18,9 @@ This encompasses several key modules.
 **[Document loaders](/docs/modules/data_connection/document_loaders/)**
 
 Load documents from many different sources.
-LangChain provides over 100 different document loaders as well as integrations with other major providers in the space,
+LangChain provides over a 100 different document loaders as well as integrations with other major providers in the space,
 like AirByte and Unstructured.
-We provide integrations to load all types of documents (HTML, PDF, code) from all types of locations (private s3 buckets, public websites).
+We provide integrations to load all types of documents (html, PDF, code) from all types of locations (private s3 buckets, public websites).
 
 **[Document transformers](/docs/modules/data_connection/document_transformers/)**
 
@@ -32,18 +32,18 @@ LangChain provides several different algorithms for doing this, as well as logic
 **[Text embedding models](/docs/modules/data_connection/text_embedding/)**
 
 Another key part of retrieval has become creating embeddings for documents.
-Embeddings capture the semantic meaning of the text, allowing you to quickly and
+Embeddings capture the semantic meaning of text, allowing you to quickly and
 efficiently find other pieces of text that are similar.
 LangChain provides integrations with over 25 different embedding providers and methods,
 from open-source to proprietary API,
 allowing you to choose the one best suited for your needs.
-LangChain provides a standard interface, allowing you to easily swap between models.
+LangChain exposes a standard interface, allowing you to easily swap between models.
 
 **[Vector stores](/docs/modules/data_connection/vectorstores/)**
 
 With the rise of embeddings, there has emerged a need for databases to support efficient storage and searching of these embeddings.
 LangChain provides integrations with over 50 different vectorstores, from open-source local ones to cloud-hosted proprietary ones,
-allowing you to choose the one best suited for your needs.
+allowing you choose the one best suited for your needs.
 LangChain exposes a standard interface, allowing you to easily swap between vector stores.
 
 **[Retrievers](/docs/modules/data_connection/retrievers/)**
@@ -55,7 +55,7 @@ However, we have also added a collection of algorithms on top of this to increas
 These include:
 
 - [Parent Document Retriever](/docs/modules/data_connection/retrievers/parent_document_retriever): This allows you to create multiple embeddings per parent document, allowing you to look up smaller chunks but return larger context.
-- [Self Query Retriever](/docs/modules/data_connection/retrievers/self_query): User questions often contain a reference to something that isn't just semantic but rather expresses some logic that can best be represented as a metadata filter. Self-query allows you to parse out the *semantic* part of a query from other *metadata filters* present in the query.
+- [Self Query Retriever](/docs/modules/data_connection/retrievers/self_query): User questions often contain reference to something that isn't just semantic, but rather expresses some logic that can best be represented as a metadata filter. Self-query allows you to parse out the *semantic* part of a query from other *metadata filters* present in the query
 - [Ensemble Retriever](/docs/modules/data_connection/retrievers/ensemble): Sometimes you may want to retrieve documents from multiple different sources, or using multiple different algorithms. The ensemble retriever allows you to easily do this.
 - And more!
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/contextual_compression/index.mdx

@@ -5,10 +5,10 @@ One challenge with retrieval is that usually you don't know the specific queries
 Contextual compression is meant to fix this. The idea is simple: instead of immediately returning retrieved documents as-is, you can compress them using the context of the given query, so that only the relevant information is returned. “Compressing” here refers to both compressing the contents of an individual document and filtering out documents wholesale.
 
 To use the Contextual Compression Retriever, you'll need:
-- a base retriever
+- a base Retriever
 - a Document Compressor
 
-The Contextual Compression Retriever passes queries to the base retriever, takes the initial documents and passes them through the Document Compressor. The Document Compressor takes a list of documents and shortens it by reducing the contents of documents or dropping documents altogether.
+The Contextual Compression Retriever passes queries to the base Retriever, takes the initial documents and passes them through the Document Compressor. The Document Compressor takes a list of Documents and shortens it by reducing the contents of Documents or dropping Documents altogether.
 
 ![](https://drive.google.com/uc?id=1CtNgWODXZudxAWSRiWgSGEoTNrUFT98v)
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/index.mdx

@@ -8,7 +8,7 @@ Head to [Integrations](/docs/integrations/retrievers/) for documentation on buil
 :::
 
 A retriever is an interface that returns documents given an unstructured query. It is more general than a vector store.
-A retriever does not need to be able to store documents, only to return (or retrieve) them. Vector stores can be used
+A retriever does not need to be able to store documents, only to return (or retrieve) it. Vector stores can be used
 as the backbone of a retriever, but there are other types of retrievers as well.
 
 ## Get started

docs/docs_skeleton/docs/modules/data_connection/retrievers/self_query/index.mdx

@@ -1,6 +1,6 @@
 # Self-querying
 
-A self-querying retriever is one that, as the name suggests, has the ability to query itself. Specifically, given any natural language query, the retriever uses a query-constructing LLM chain to write a structured query and then applies that structured query to its underlying VectorStore. This allows the retriever to not only use the user-input query for semantic similarity comparison with the contents of stored documents but to also extract filters from the user query on the metadata of stored documents and to execute those filters.
+A self-querying retriever is one that, as the name suggests, has the ability to query itself. Specifically, given any natural language query, the retriever uses a query-constructing LLM chain to write a structured query and then applies that structured query to it's underlying VectorStore. This allows the retriever to not only use the user-input query for semantic similarity comparison with the contents of stored documented, but to also extract filters from the user query on the metadata of stored documents and to execute those filters.
 
 ![](https://drive.google.com/uc?id=1OQUN-0MJcDUxmPXofgS7MqReEs720pqS)
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/time_weighted_vectorstore.mdx

@@ -8,7 +8,7 @@ The algorithm for scoring them is:
 semantic_similarity + (1.0 - decay_rate) ^ hours_passed
 ```
 
-Notably, `hours_passed` refers to the hours passed since the object in the retriever **was last accessed**, not since it was created. This means that frequently accessed objects remain "fresh".
+Notably, `hours_passed` refers to the hours passed since the object in the retriever **was last accessed**, not since it was created. This means that frequently accessed objects remain "fresh."
 
 import Example from "@snippets/modules/data_connection/retrievers/how_to/time_weighted_vectorstore.mdx"
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/vectorstore.mdx

@@ -1,9 +1,9 @@
 # Vector store-backed retriever
 
-A vector store retriever is a retriever that uses a vector store to retrieve documents. It is a lightweight wrapper around the vector store class to make it conform to the retriever interface.
+A vector store retriever is a retriever that uses a vector store to retrieve documents. It is a lightweight wrapper around the Vector Store class to make it conform to the Retriever interface.
 It uses the search methods implemented by a vector store, like similarity search and MMR, to query the texts in the vector store.
 
-Once you construct a vector store, it's very easy to construct a retriever. Let's walk through an example.
+Once you construct a Vector store, it's very easy to construct a retriever. Let's walk through an example.
 
 import Example from "@snippets/modules/data_connection/retrievers/how_to/vectorstore.mdx"
 

docs/docs_skeleton/docs/modules/data_connection/text_embedding/index.mdx

@@ -11,7 +11,7 @@ The Embeddings class is a class designed for interfacing with text embedding mod
 
 Embeddings create a vector representation of a piece of text. This is useful because it means we can think about text in the vector space, and do things like semantic search where we look for pieces of text that are most similar in the vector space.
 
-The base Embeddings class in LangChain provides two methods: one for embedding documents and one for embedding a query. The former takes as input multiple texts, while the latter takes a single text. The reason for having these as two separate methods is that some embedding providers have different embedding methods for documents (to be searched over) vs queries (the search query itself).
+The base Embeddings class in LangChain exposes two methods: one for embedding documents and one for embedding a query. The former takes as input multiple texts, while the latter takes a single text. The reason for having these as two separate methods is that some embedding providers have different embedding methods for documents (to be searched over) vs queries (the search query itself).
 
 ## Get started
 

docs/docs_skeleton/docs/modules/data_connection/vectorstores/index.mdx

@@ -16,7 +16,7 @@ for you.
 
 ## Get started
 
-This walkthrough showcases basic functionality related to vector stores. A key part of working with vector stores is creating the vector to put in them, which is usually created via embeddings. Therefore, it is recommended that you familiarize yourself with the [text embedding model](/docs/modules/data_connection/text_embedding/) interfaces before diving into this.
+This walkthrough showcases basic functionality related to VectorStores. A key part of working with vector stores is creating the vector to put in them, which is usually created via embeddings. Therefore, it is recommended that you familiarize yourself with the [text embedding model](/docs/modules/data_connection/text_embedding/) interfaces before diving into this.
 
 import GetStarted from "@snippets/modules/data_connection/vectorstores/get_started.mdx"
 

docs/docs_skeleton/docs/modules/memory/chat_messages/index.mdx

@@ -8,10 +8,10 @@ Head to [Integrations](/docs/integrations/memory/) for documentation on built-in
 :::
 
 One of the core utility classes underpinning most (if not all) memory modules is the `ChatMessageHistory` class.
-This is a super lightweight wrapper which provides convenience methods for saving HumanMessages, AIMessages, and then fetching them all.
+This is a super lightweight wrapper which exposes convenience methods for saving Human messages, AI messages, and then fetching them all.
 
 You may want to use this class directly if you are managing memory outside of a chain.
 
 import GetStarted from "@snippets/modules/memory/chat_messages/get_started.mdx"
 
-<GetStarted/>
+<GetStarted/>

docs/docs_skeleton/docs/modules/memory/index.mdx

@@ -32,7 +32,7 @@ Even if these are not all used directly, they need to be stored in some form.
 One of the key parts of the LangChain memory module is a series of integrations for storing these chat messages,
 from in-memory lists to persistent databases.
 
-- [Chat message storage](/docs/modules/memory/chat_messages/): How to work with Chat Messages, and the various integrations offered.
+- [Chat message storage](/docs/modules/memory/chat_messages/): How to work with Chat Messages, and the various integrations offered
 
 ### Querying: Data structures and algorithms on top of chat messages
 Keeping a list of chat messages is fairly straight-forward.

docs/docs_skeleton/docs/modules/memory/types/buffer.mdx

@@ -1,6 +1,6 @@
-# Conversation Buffer
+# Conversation buffer memory
 
-This notebook shows how to use `ConversationBufferMemory`. This memory allows for storing messages and then extracts the messages in a variable.
+This notebook shows how to use `ConversationBufferMemory`. This memory allows for storing of messages and then extracts the messages in a variable.
 
 We can first extract it as a string.
 

docs/docs_skeleton/docs/modules/memory/types/buffer_window.mdx

@@ -1,6 +1,6 @@
-# Conversation Buffer Window
+# Conversation buffer window memory
 
-`ConversationBufferWindowMemory` keeps a list of the interactions of the conversation over time. It only uses the last K interactions. This can be useful for keeping a sliding window of the most recent interactions, so the buffer does not get too large.
+`ConversationBufferWindowMemory` keeps a list of the interactions of the conversation over time. It only uses the last K interactions. This can be useful for keeping a sliding window of the most recent interactions, so the buffer does not get too large
 
 Let's first explore the basic functionality of this type of memory.
 

docs/docs_skeleton/docs/modules/memory/types/entity_summary_memory.mdx

@@ -1,6 +1,6 @@
-# Entity
+# Entity memory
 
-Entity memory remembers given facts about specific entities in a conversation. It extracts information on entities (using an LLM) and builds up its knowledge about that entity over time (also using an LLM).
+Entity Memory remembers given facts about specific entities in a conversation. It extracts information on entities (using an LLM) and builds up its knowledge about that entity over time (also using an LLM).
 
 Let's first walk through using this functionality.
 

docs/docs_skeleton/docs/modules/memory/types/index.mdx

@@ -1,8 +1,8 @@
 ---
 sidebar_position: 2
 ---
-# Memory types
+# Memory Types
 
 There are many different types of memory.
-Each has their own parameters, their own return types, and is useful in different scenarios.
+Each have their own parameters, their own return types, and are useful in different scenarios.
 Please see their individual page for more detail on each one.

docs/docs_skeleton/docs/modules/memory/types/summary.mdx

@@ -1,4 +1,4 @@
-# Conversation Summary
+# Conversation summary memory
 Now let's take a look at using a slightly more complex type of memory - `ConversationSummaryMemory`. This type of memory creates a summary of the conversation over time. This can be useful for condensing information from the conversation over time.
 Conversation summary memory summarizes the conversation as it happens and stores the current summary in memory. This memory can then be used to inject the summary of the conversation so far into a prompt/chain. This memory is most useful for longer conversations, where keeping the past message history in the prompt verbatim would take up too many tokens.
 

docs/docs_skeleton/docs/modules/memory/types/vectorstore_retriever_memory.mdx

@@ -1,6 +1,6 @@
-# Backed by a Vector Store
+# Vector store-backed memory
 
-`VectorStoreRetrieverMemory` stores memories in a vector store and queries the top-K most "salient" docs every time it is called.
+`VectorStoreRetrieverMemory` stores memories in a VectorDB and queries the top-K most "salient" docs every time it is called.
 
 This differs from most of the other Memory classes in that it doesn't explicitly track the order of interactions.
 

docs/docs_skeleton/docs/modules/model_io/models/chat/chat_model_caching.mdx

@@ -1,5 +1,5 @@
 # Caching
-LangChain provides an optional caching layer for chat models. This is useful for two reasons:
+LangChain provides an optional caching layer for Chat Models. This is useful for two reasons:
 
 It can save you money by reducing the number of API calls you make to the LLM provider, if you're often requesting the same completion multiple times.
 It can speed up your application by reducing the number of API calls you make to the LLM provider.

docs/docs_skeleton/docs/modules/model_io/models/chat/index.mdx

@@ -8,8 +8,8 @@ Head to [Integrations](/docs/integrations/chat/) for documentation on built-in i
 :::
 
 Chat models are a variation on language models.
-While chat models use language models under the hood, the interface they use is a bit different.
-Rather than using a "text in, text out" API, they use an interface where "chat messages" are the inputs and outputs.
+While chat models use language models under the hood, the interface they expose is a bit different.
+Rather than expose a "text in, text out" API, they expose an interface where "chat messages" are the inputs and outputs.
 
 Chat model APIs are fairly new, so we are still figuring out the correct abstractions.
 

docs/docs_skeleton/docs/modules/model_io/models/chat/prompts.mdx

@@ -1,6 +1,6 @@
 # Prompts
 
-Prompts for chat models are built around messages, instead of just plain text.
+Prompts for Chat models are built around messages, instead of just plain text.
 
 import Prompts from "@snippets/modules/model_io/models/chat/how_to/prompts.mdx"
 

docs/docs_skeleton/docs/modules/model_io/models/chat/streaming.mdx

@@ -1,6 +1,6 @@
 # Streaming
 
-Some chat models provide a streaming response. This means that instead of waiting for the entire response to be returned, you can start processing it as soon as it's available. This is useful if you want to display the response to the user as it's being generated, or if you want to process the response as it's being generated.
+Some Chat models provide a streaming response. This means that instead of waiting for the entire response to be returned, you can start processing it as soon as it's available. This is useful if you want to display the response to the user as it's being generated, or if you want to process the response as it's being generated.
 
 import StreamingChatModel from "@snippets/modules/model_io/models/chat/how_to/streaming.mdx"
 

docs/docs_skeleton/docs/modules/model_io/models/index.mdx

@@ -8,16 +8,16 @@ LangChain provides interfaces and integrations for two types of models:
 - [LLMs](/docs/modules/model_io/models/llms/): Models that take a text string as input and return a text string
 - [Chat models](/docs/modules/model_io/models/chat/): Models that are backed by a language model but take a list of Chat Messages as input and return a Chat Message
 
-## LLMs vs chat models
+## LLMs vs Chat Models
 
-LLMs and chat models are subtly but importantly different. LLMs in LangChain refer to pure text completion models.
+LLMs and Chat Models are subtly but importantly different. LLMs in LangChain refer to pure text completion models.
 The APIs they wrap take a string prompt as input and output a string completion. OpenAI's GPT-3 is implemented as an LLM.
 Chat models are often backed by LLMs but tuned specifically for having conversations.
-And, crucially, their provider APIs use a different interface than pure text completion models. Instead of a single string,
+And, crucially, their provider APIs expose a different interface than pure text completion models. Instead of a single string,
 they take a list of chat messages as input. Usually these messages are labeled with the speaker (usually one of "System",
-"AI", and "Human"). And they return an AI chat message as output. GPT-4 and Anthropic's Claude are both implemented as chat models.
+"AI", and "Human"). And they return a ("AI") chat message as output. GPT-4 and Anthropic's Claude are both implemented as Chat Models.
 
-To make it possible to swap LLMs and chat models, both implement the Base Language Model interface. This includes common
+To make it possible to swap LLMs and Chat Models, both implement the Base Language Model interface. This exposes common
 methods "predict", which takes a string and returns a string, and "predict messages", which takes messages and returns a message.
-If you are using a specific model it's recommended you use the methods specific to that model class (i.e., "predict" for LLMs and "predict messages" for chat models),
+If you are using a specific model it's recommended you use the methods specific to that model class (i.e., "predict" for LLMs and "predict messages" for Chat Models),
 but if you're creating an application that should work with different types of models the shared interface can be helpful.

docs/docs_skeleton/docs/modules/model_io/prompts/prompt_templates/few_shot_examples.mdx

@@ -1,6 +1,6 @@
 # Few-shot prompt templates
 
-In this tutorial, we'll learn how to create a prompt template that uses few-shot examples. A few-shot prompt template can be constructed from either a set of examples, or from an Example Selector object.
+In this tutorial, we'll learn how to create a prompt template that uses few shot examples. A few shot prompt template can be constructed from either a set of examples, or from an Example Selector object.
 
 import Example from "@snippets/modules/model_io/prompts/prompt_templates/few_shot_examples.mdx"
 

docs/docs_skeleton/docs/modules/model_io/prompts/prompt_templates/index.mdx

@@ -6,7 +6,7 @@ sidebar_position: 0
 
 Prompt templates are pre-defined recipes for generating prompts for language models.
 
-A template may include instructions, few-shot examples, and specific context and
+A template may include instructions, few shot examples, and specific context and
 questions appropriate for a given task.
 
 LangChain provides tooling to create and work with prompt templates.

docs/docs_skeleton/docs/modules/model_io/prompts/prompt_templates/partial.mdx

@@ -1,6 +1,6 @@
 # Partial prompt templates
 
-Like other methods, it can make sense to "partial" a prompt template - e.g. pass in a subset of the required values, as to create a new prompt template which expects only the remaining subset of values.
+Like other methods, it can make sense to "partial" a prompt template - eg pass in a subset of the required values, as to create a new prompt template which expects only the remaining subset of values.
 
 LangChain supports this in two ways:
 1. Partial formatting with string values.

docs/docs_skeleton/docs/modules/model_io/prompts/prompt_templates/prompt_composition.mdx

@@ -2,8 +2,8 @@
 
 This notebook goes over how to compose multiple prompts together. This can be useful when you want to reuse parts of prompts. This can be done with a PipelinePrompt. A PipelinePrompt consists of two main parts:
 
-- Final prompt: The final prompt that is returned
-- Pipeline prompts: A list of tuples, consisting of a string name and a prompt template. Each prompt template will be formatted and then passed to future prompt templates as a variable with the same name.
+- Final prompt: This is the final prompt that is returned
+- Pipeline prompts: This is a list of tuples, consisting of a string name and a prompt template. Each prompt template will be formatted and then passed to future prompt templates as a variable with the same name.
 
 import Example from "@snippets/modules/model_io/prompts/prompt_templates/prompt_composition.mdx"
 

docs/docs_skeleton/docs/use_cases/question_answering/_category_.yml

@@ -1,2 +0,0 @@
-position: 0
-collapsed: false

docs/docs_skeleton/docs/use_cases/question_answering/how_to/chat_vector_db.mdx

@@ -5,7 +5,7 @@ sidebar_position: 2
 # Store and reference chat history
 The ConversationalRetrievalQA chain builds on RetrievalQAChain to provide a chat history component.
 
-It first combines the chat history (either explicitly passed in or retrieved from the provided memory) and the question into a standalone question, then looks up relevant documents from the retriever, and finally passes those documents and the question to a question-answering chain to return a response.
+It first combines the chat history (either explicitly passed in or retrieved from the provided memory) and the question into a standalone question, then looks up relevant documents from the retriever, and finally passes those documents and the question to a question answering chain to return a response.
 
 To create one, you will need a retriever. In the below example, we will create one from a vector store, which can be created from embeddings.
 

docs/docs_skeleton/docs/use_cases/web_scraping/index.mdx

@@ -0,0 +1,9 @@
+---
+sidebar_position: 3
+---
+
+# Web Scraping
+
+Web scraping has historically been a challenging endeavor due to the ever-changing nature of website structures, making it tedious for developers to maintain their scraping scripts. Traditional methods often rely on specific HTML tags and patterns which, when altered, can disrupt data extraction processes.
+
+Enter the LLM-based method for parsing HTML: By leveraging the capabilities of LLMs, and especially OpenAI Functions in LangChain's extraction chain, developers can instruct the model to extract only the desired data in a specified format. This method not only streamlines the extraction process but also significantly reduces the time spent on manual debugging and script modifications. Its adaptability means that even if websites undergo significant design changes, the extraction remains consistent and robust. This level of resilience translates to reduced maintenance efforts, cost savings, and ensures a higher quality of extracted data. Compared to its predecessors, LLM-based approach wins out the web scraping domain by transforming a historically cumbersome task into a more automated and efficient process.

docs/docs_skeleton/sidebars.js

@@ -44,16 +44,6 @@ module.exports = {
         id: "modules/index"
       },
     },
-    {
-      type: "category",
-      label: "LangChain Expression Language",
-      collapsed: true,
-      items: [{ type: "autogenerated", dirName: "expression_language" } ],
-      link: {
-        type: 'doc',
-        id: "expression_language/index"
-      },
-    },
     {
       type: "category",
       label: "Guides",
@@ -62,7 +52,17 @@ module.exports = {
       link: {
         type: 'generated-index',
         description: 'Design guides for key parts of the development process',
-        slug: "guides",
+      slug: "guides",
+      },
+    },
+    {
+      type: "category",
+      label: "Ecosystem",
+      collapsed: true,
+      items: [{ type: "autogenerated", dirName: "ecosystem" }],
+      link: {
+        type: 'generated-index',
+      slug: "ecosystem",
       },
     },
     {
@@ -72,7 +72,7 @@ module.exports = {
       items: [{ type: "autogenerated", dirName: "additional_resources" }, { type: "link", label: "Gallery", href: "https://github.com/kyrolabs/awesome-langchain" }],
       link: {
         type: 'generated-index',
-        slug: "additional_resources",
+      slug: "additional_resources",
       },
     },
     'community'

docs/docs_skeleton/src/theme/CodeBlock/index.js

@@ -24,7 +24,8 @@ function Imports({ imports }) {
           <li key={imported}>
             <a href={docs}>
               <span>{imported}</span>
-            </a>
+            </a>{" "}
+            from <code>{source}</code>
           </li>
         ))}
       </ul>

docs/docs_skeleton/vercel.json

@@ -1,9 +1,5 @@
 {
   "redirects": [
-    {
-      "source": "/docs/modules/data_connection/caching_embeddings(/?)",
-      "destination": "/docs/modules/data_connection/text_embedding/caching_embeddings"
-    },
     {
       "source": "/en/latest/additional_resources/youtube.html",
       "destination": "/docs/additional_resources/youtube"
@@ -170,7 +166,7 @@
     },
     {
       "source": "/docs/integrations/deeplake",
-      "destination": "/docs/integrations/providers/activeloop_deeplake"
+      "destination": "/docs/integrations/providers/deeplake"
     },
     {
       "source": "/docs/integrations/diffbot",
@@ -2952,46 +2948,6 @@
       "source": "/docs/modules/model_io/models/llms/integrations/writer",
       "destination": "/docs/integrations/llms/writer"
     },
-    {
-      "source": "/docs/integrations/llms/amazon_api_gateway_example",
-      "destination": "/docs/integrations/llms/amazon_api_gateway"
-    },
-    {
-      "source": "/docs/integrations/llms/azureml_endpoint_example",
-      "destination": "/docs/integrations/llms/azure_ml"
-    },
-    {
-      "source": "/docs/integrations/llms/azure_openai_example",
-      "destination": "/docs/integrations/llms/azure_openai"
-    },
-    {
-      "source": "/docs/integrations/llms/cerebriumai_example",
-      "destination": "/docs/integrations/llms/cerebriumai"
-    },
-    {
-      "source": "/docs/integrations/llms/deepinfra_example",
-      "destination": "/docs/integrations/llms/deepinfra"
-    },
-    {
-      "source": "/docs/integrations/llms/Fireworks",
-      "destination": "/docs/integrations/llms/fireworks"
-    },
-    {
-      "source": "/docs/integrations/llms/forefrontai_example",
-      "destination": "/docs/integrations/llms/forefrontai"
-    },
-    {
-      "source": "/docs/integrations/llms/gooseai_example",
-      "destination": "/docs/integrations/llms/gooseai"
-    },
-    {
-      "source": "/docs/integrations/llms/petals_example",
-      "destination": "/docs/integrations/llms/petals"
-    },
-    {
-      "source": "/docs/integrations/llms/pipelineai_example",
-      "destination": "/docs/integrations/llms/pipelineai"
-    },
     {
       "source": "/en/latest/modules/prompts.html",
       "destination": "/docs/modules/model_io/prompts"
@@ -3178,11 +3134,7 @@
     },
     {
       "source": "/en/latest/use_cases/tabular.html",
-      "destination": "/docs/use_cases/qa_structured"
-    },
-    {
-      "source": "/docs/use_cases/sql(/?)",
-      "destination": "/docs/use_cases/qa_structured/sql"
+      "destination": "/docs/use_cases/tabular"
     },
     {
       "source": "/en/latest/youtube.html",
@@ -3374,7 +3326,7 @@
     },
     {
       "source": "/docs/modules/chains/popular/sqlite",
-      "destination": "/docs/use_cases/qa_structured/sql"
+      "destination": "/docs/use_cases/tabular/sqlite"
     },
     {
       "source": "/docs/modules/chains/popular/openai_functions",
@@ -3480,14 +3432,6 @@
       "source": "/docs/modules/chains/additional/graph_kuzu_qa",
       "destination": "/docs/use_cases/more/graph/graph_kuzu_qa"
     },
-    {
-      "source": "/docs/use_cases/graph/graph_falkordb_qa",
-      "destination": "/docs/use_cases/more/graph/graph_falkordb_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_falkordb_qa",
-      "destination": "/docs/use_cases/more/graph/graph_falkordb_qa"
-    },
     {
       "source": "/docs/use_cases/graph/graph_nebula_qa",
       "destination": "/docs/use_cases/more/graph/graph_nebula_qa"
@@ -3586,7 +3530,7 @@
     },
     {
       "source": "/docs/modules/chains/additional/elasticsearch_database",
-      "destination": "/docs/use_cases/qa_structured/integrations/elasticsearch"
+      "destination": "/docs/use_cases/tabular/elasticsearch_database"
     },
     {
       "source": "/docs/modules/chains/additional/tagging",
@@ -3599,18 +3543,6 @@
     {
       "source": "/en/latest/integrations/:path*",
       "destination": "/docs/integrations/providers/:path*"
-    },
-    {
-      "source": "/docs/guides/expression_language(/?)",
-      "destination": "/docs/expression_language/"
-    },
-    {
-      "source": "/docs/guides/expression_language/:path*",
-      "destination": "/docs/expression_language/:path*"
-    },
-    {
-      "source": "/docs/ecosystem/dependents",
-      "destination": "/docs/additional_resources/dependents"
     }
   ]
 }

docs/extras/_templates/integration.mdx

@@ -47,7 +47,7 @@ from langchain.embeddings import integration_class_REPLACE_ME
 ```
 
 
-## Chat models
+## Chat Models
 
 See a [usage example](/docs/integrations/chat/INCLUDE_REAL_NAME)
 

docs/extras/additional_resources/youtube.mdx

@@ -1,6 +1,6 @@
 # YouTube videos
 
-⛓ icon marks a new addition [last update 2023-09-05]
+⛓ icon marks a new addition [last update 2023-06-20]
 
 ### [Official LangChain YouTube channel](https://www.youtube.com/@LangChain)
 
@@ -86,20 +86,20 @@
 - [`Llama Index`: Chat with Documentation using URL Loader](https://youtu.be/XJRoDEctAwA) by [Merk](https://www.youtube.com/@merksworld)
 - [Using OpenAI, LangChain, and `Gradio` to Build Custom GenAI Applications](https://youtu.be/1MsmqMg3yUc) by [David Hundley](https://www.youtube.com/@dkhundley)
 - [LangChain, Chroma DB, OpenAI Beginner Guide | ChatGPT with your PDF](https://youtu.be/FuqdVNB_8c0)
-- [Build AI chatbot with custom knowledge base using OpenAI API and GPT Index](https://youtu.be/vDZAZuaXf48) by [Irina Nik](https://www.youtube.com/@irina_nik)
-- [Build Your Own Auto-GPT Apps with LangChain (Python Tutorial)](https://youtu.be/NYSWn1ipbgg) by [Dave Ebbelaar](https://www.youtube.com/@daveebbelaar)
-- [Chat with Multiple `PDFs` | LangChain App Tutorial in Python (Free LLMs and Embeddings)](https://youtu.be/dXxQ0LR-3Hg) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
-- [Chat with a `CSV` | `LangChain Agents` Tutorial (Beginners)](https://youtu.be/tjeti5vXWOU) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
-- [Create Your Own ChatGPT with `PDF` Data in 5 Minutes (LangChain Tutorial)](https://youtu.be/au2WVVGUvc8) by [Liam Ottley](https://www.youtube.com/@LiamOttley)
-- [Using ChatGPT with YOUR OWN Data. This is magical. (LangChain OpenAI API)](https://youtu.be/9AXP7tCI9PI) by [TechLead](https://www.youtube.com/@TechLead)
-- [Build a Custom Chatbot with OpenAI: `GPT-Index` & LangChain | Step-by-Step Tutorial](https://youtu.be/FIDv6nc4CgU) by [Fabrikod](https://www.youtube.com/@fabrikod)
-- [`Flowise` is an open source no-code UI visual tool to build 🦜🔗LangChain applications](https://youtu.be/CovAPtQPU0k) by [Cobus Greyling](https://www.youtube.com/@CobusGreylingZA)
-- [LangChain & GPT 4 For Data Analysis: The `Pandas` Dataframe Agent](https://youtu.be/rFQ5Kmkd4jc) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
-- [`GirlfriendGPT` - AI girlfriend with LangChain](https://youtu.be/LiN3D1QZGQw) by [Toolfinder AI](https://www.youtube.com/@toolfinderai)
-- [`PrivateGPT`: Chat to your FILES OFFLINE and FREE [Installation and Tutorial]](https://youtu.be/G7iLllmx4qc) by [Prompt Engineering](https://www.youtube.com/@engineerprompt)
-- [How to build with Langchain 10x easier | ⛓️ LangFlow & `Flowise`](https://youtu.be/Ya1oGL7ZTvU) by [AI Jason](https://www.youtube.com/@AIJasonZ)
-- [Getting Started With LangChain In 20 Minutes- Build Celebrity Search Application](https://youtu.be/_FpT1cwcSLg) by [Krish Naik](https://www.youtube.com/@krishnaik06)
-- ⛓ [LangChain HowTo and Guides YouTube playlist](https://www.youtube.com/playlist?list=PL8motc6AQftk1Bs42EW45kwYbyJ4jOdiZ) by [Sam Witteveen](https://www.youtube.com/@samwitteveenai/)
+- ⛓ [Build AI chatbot with custom knowledge base using OpenAI API and GPT Index](https://youtu.be/vDZAZuaXf48) by [Irina Nik](https://www.youtube.com/@irina_nik)
+- ⛓ [Build Your Own Auto-GPT Apps with LangChain (Python Tutorial)](https://youtu.be/NYSWn1ipbgg) by [Dave Ebbelaar](https://www.youtube.com/@daveebbelaar)
+- ⛓ [Chat with Multiple `PDFs` | LangChain App Tutorial in Python (Free LLMs and Embeddings)](https://youtu.be/dXxQ0LR-3Hg) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
+- ⛓ [Chat with a `CSV` | `LangChain Agents` Tutorial (Beginners)](https://youtu.be/tjeti5vXWOU) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
+- ⛓ [Create Your Own ChatGPT with `PDF` Data in 5 Minutes (LangChain Tutorial)](https://youtu.be/au2WVVGUvc8) by [Liam Ottley](https://www.youtube.com/@LiamOttley)
+- ⛓ [Using ChatGPT with YOUR OWN Data. This is magical. (LangChain OpenAI API)](https://youtu.be/9AXP7tCI9PI) by [TechLead](https://www.youtube.com/@TechLead)
+- ⛓ [Build a Custom Chatbot with OpenAI: `GPT-Index` & LangChain | Step-by-Step Tutorial](https://youtu.be/FIDv6nc4CgU) by [Fabrikod](https://www.youtube.com/@fabrikod)
+- ⛓ [`Flowise` is an open source no-code UI visual tool to build 🦜🔗LangChain applications](https://youtu.be/CovAPtQPU0k) by [Cobus Greyling](https://www.youtube.com/@CobusGreylingZA)
+- ⛓ [LangChain & GPT 4 For Data Analysis: The `Pandas` Dataframe Agent](https://youtu.be/rFQ5Kmkd4jc) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
+- ⛓ [`GirlfriendGPT` - AI girlfriend with LangChain](https://youtu.be/LiN3D1QZGQw) by [Toolfinder AI](https://www.youtube.com/@toolfinderai)
+- ⛓ [`PrivateGPT`: Chat to your FILES OFFLINE and FREE [Installation and Tutorial]](https://youtu.be/G7iLllmx4qc) by [Prompt Engineering](https://www.youtube.com/@engineerprompt)
+- ⛓ [How to build with Langchain 10x easier | ⛓️ LangFlow & `Flowise`](https://youtu.be/Ya1oGL7ZTvU) by [AI Jason](https://www.youtube.com/@AIJasonZ)
+- ⛓ [Getting Started With LangChain In 20 Minutes- Build Celebrity Search Application](https://youtu.be/_FpT1cwcSLg) by [Krish Naik](https://www.youtube.com/@krishnaik06)
+
 
 
 ### [Prompt Engineering and LangChain](https://www.youtube.com/watch?v=muXbPpG_ys4&list=PLEJK-H61Xlwzm5FYLDdKt_6yibO33zoMW) by [Venelin Valkov](https://www.youtube.com/@venelin_valkov)

docs/extras/ecosystem/dependents.mdx

@@ -51,7 +51,6 @@ Dependents stats for `langchain-ai/langchain`
 |[e2b-dev/e2b](https://github.com/e2b-dev/e2b) | 5365 |
 |[mage-ai/mage-ai](https://github.com/mage-ai/mage-ai) | 5352 |
 |[wenda-LLM/wenda](https://github.com/wenda-LLM/wenda) | 5192 |
-|[liaokongVFX/LangChain-Chinese-Getting-Started-Guide](https://github.com/liaokongVFX/LangChain-Chinese-Getting-Started-Guide) | 5129 |
 |[zilliztech/GPTCache](https://github.com/zilliztech/GPTCache) | 4993 |
 |[GreyDGL/PentestGPT](https://github.com/GreyDGL/PentestGPT) | 4831 |
 |[zauberzeug/nicegui](https://github.com/zauberzeug/nicegui) | 4824 |

docs/extras/guides/debugging.md

@@ -2,13 +2,13 @@
 
 If you're building with LLMs, at some point something will break, and you'll need to debug. A model call will fail, or the model output will be misformatted, or there will be some nested model calls and it won't be clear where along the way an incorrect output was created.
 
-Here are a few different tools and functionalities to aid in debugging.
+Here's a few different tools and functionalities to aid in debugging.
 
 
 
 ## Tracing
 
-Platforms with tracing capabilities like [LangSmith](/docs/guides/langsmith/) and [WandB](/docs/integrations/providers/wandb_tracing) are the most comprehensive solutions for debugging. These platforms make it easy to not only log and visualize LLM apps, but also to actively debug, test and refine them.
+Platforms with tracing capabilities like [LangSmith](/docs/guides/langsmith/) and [WandB](/docs/ecosystem/integrations/agent_with_wandb_tracing) are the most comprehensive solutions for debugging. These platforms make it easy to not only log and visualize LLM apps, but also to actively debug, test and refine them.
 
 For anyone building production-grade LLM applications, we highly recommend using a platform like this.
 
@@ -18,9 +18,9 @@ For anyone building production-grade LLM applications, we highly recommend using
 
 If you're prototyping in Jupyter Notebooks or running Python scripts, it can be helpful to print out the intermediate steps of a Chain run. 
 
-There are a number of ways to enable printing at varying degrees of verbosity.
+There's a number of ways to enable printing at varying degrees of verbosity.
 
-Let's suppose we have a simple agent, and want to visualize the actions it takes and tool outputs it receives. Without any debugging, here's what we see:
+Let's suppose we have a simple agent and want to visualize the actions it takes and tool outputs it receives. Without any debugging, here's what we see:
 
 
 ```python

docs/extras/guides/deployments/template_repos.mdx

@@ -14,7 +14,7 @@ It also contains instructions for how to deploy this app on the Streamlit platfo
 
 ## [Gradio (on Hugging Face)](https://github.com/hwchase17/langchain-gradio-template)
 
-This repo serves as a template for how to deploy a LangChain with Gradio.
+This repo serves as a template for how deploy a LangChain with Gradio.
 It implements a chatbot interface, with a "Bring-Your-Own-Token" approach (nice for not wracking up big bills).
 It also contains instructions for how to deploy this app on the Hugging Face platform.
 This is heavily influenced by James Weaver's [excellent examples](https://huggingface.co/JavaFXpert).
@@ -27,7 +27,7 @@ Chainlit [doc](https://docs.chainlit.io/langchain) on the integration with LangC
 
 ## [Beam](https://github.com/slai-labs/get-beam/tree/main/examples/langchain-question-answering)
 
-This repo serves as a template for how to deploy a LangChain with [Beam](https://beam.cloud).
+This repo serves as a template for how deploy a LangChain with [Beam](https://beam.cloud).
 
 It implements a Question Answering app and contains instructions for deploying the app as a serverless REST API.
 
@@ -49,7 +49,7 @@ A minimal example of how to deploy LangChain to [Fly.io](https://fly.io/) using
 
 ## [Digitalocean App Platform](https://github.com/homanp/digitalocean-langchain)
 
-A minimal example of how to deploy LangChain to DigitalOcean App Platform.
+A minimal example on how to deploy LangChain to DigitalOcean App Platform.
 
 ## [CI/CD Google Cloud Build + Dockerfile + Serverless Google Cloud Run](https://github.com/g-emarco/github-assistant)
 
@@ -57,7 +57,7 @@ Boilerplate LangChain project on how to deploy to Google Cloud Run using Docker
 
 ## [Google Cloud Run](https://github.com/homanp/gcp-langchain)
 
-A minimal example of how to deploy LangChain to Google Cloud Run.
+A minimal example on how to deploy LangChain to Google Cloud Run.
 
 ## [SteamShip](https://github.com/steamship-core/steamship-langchain/)
 
@@ -82,4 +82,4 @@ These templates serve as examples of how to build, deploy, and share LangChain a
 
 ## [AzureML Online Endpoint](https://github.com/Azure/azureml-examples/blob/main/sdk/python/endpoints/online/llm/langchain/1_langchain_basic_deploy.ipynb)
 
-A minimal example of how to deploy LangChain to an Azure Machine Learning Online Endpoint. 
+A minimal example of how to deploy LangChain to an Azure Machine Learning Online Endpoint. 

docs/extras/guides/expression_language/interface.ipynb

@@ -62,7 +62,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 6,
    "id": "d1850a1f",
    "metadata": {},
    "outputs": [],
@@ -72,7 +72,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 7,
    "id": "56d0669f",
    "metadata": {},
    "outputs": [],
@@ -170,36 +170,6 @@
     "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "2434ab15",
-   "metadata": {},
-   "source": [
-    "You can set the number of concurrent requests by using the `max_concurrency` parameter"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "a08522f6",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False),\n",
-       " AIMessage(content=\"Why don't cats play poker in the wild?\\n\\nToo many cheetahs!\", additional_kwargs={}, example=False)]"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}], config={\"max_concurrency\": 5})"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "b960cbfe",

docs/extras/guides/local_llms.ipynb

@@ -5,7 +5,7 @@
    "id": "b8982428",
    "metadata": {},
    "source": [
-    "# Run LLMs locally\n",
+    "# Private, local, open source LLMs\n",
     "\n",
     "## Use case\n",
     "\n",
@@ -146,7 +146,7 @@
    "source": [
     "## Environment\n",
     "\n",
-    "Inference speed is a challenge when running models locally (see above).\n",
+    "Inference speed is a chllenge when running models locally (see above).\n",
     "\n",
     "To minimize latency, it is desiable to run models locally on GPU, which ships with many consumer laptops [e.g., Apple devices](https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/).\n",
     "\n",
@@ -264,19 +264,88 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dirclear"
+    "pip install llama-cpp-python"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "a88bf0c8-e989-4bcd-bcb7-4d7757e684f2",
+   "execution_count": 43,
+   "id": "9d5f94b5",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "objc[10142]: Class GGMLMetalClass is implemented in both /Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/gpt4all/llmodel_DO_NOT_MODIFY/build/libreplit-mainline-metal.dylib (0x2a0c4c208) and /Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib (0x2c28bc208). One of the two will be used. Which one is undefined.\n",
+      "llama.cpp: loading model from /Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\n",
+      "llama_model_load_internal: format     = ggjt v3 (latest)\n",
+      "llama_model_load_internal: n_vocab    = 32000\n",
+      "llama_model_load_internal: n_ctx      = 2048\n",
+      "llama_model_load_internal: n_embd     = 5120\n",
+      "llama_model_load_internal: n_mult     = 256\n",
+      "llama_model_load_internal: n_head     = 40\n",
+      "llama_model_load_internal: n_layer    = 40\n",
+      "llama_model_load_internal: n_rot      = 128\n",
+      "llama_model_load_internal: freq_base  = 10000.0\n",
+      "llama_model_load_internal: freq_scale = 1\n",
+      "llama_model_load_internal: ftype      = 2 (mostly Q4_0)\n",
+      "llama_model_load_internal: n_ff       = 13824\n",
+      "llama_model_load_internal: model size = 13B\n",
+      "llama_model_load_internal: ggml ctx size =    0.09 MB\n",
+      "llama_model_load_internal: mem required  = 8953.71 MB (+ 1608.00 MB per state)\n",
+      "llama_new_context_with_model: kv self size  = 1600.00 MB\n",
+      "ggml_metal_init: allocating\n",
+      "ggml_metal_init: using MPS\n",
+      "ggml_metal_init: loading '/Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/ggml-metal.metal'\n",
+      "ggml_metal_init: loaded kernel_add                            0x47774af60\n",
+      "ggml_metal_init: loaded kernel_mul                            0x47774bc00\n",
+      "ggml_metal_init: loaded kernel_mul_row                        0x47774c230\n",
+      "ggml_metal_init: loaded kernel_scale                          0x47774c890\n",
+      "ggml_metal_init: loaded kernel_silu                           0x47774cef0\n",
+      "ggml_metal_init: loaded kernel_relu                           0x10e33e500\n",
+      "ggml_metal_init: loaded kernel_gelu                           0x47774b2f0\n",
+      "ggml_metal_init: loaded kernel_soft_max                       0x47771a580\n",
+      "ggml_metal_init: loaded kernel_diag_mask_inf                  0x47774dab0\n",
+      "ggml_metal_init: loaded kernel_get_rows_f16                   0x47774e110\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_0                  0x47774e7d0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_1                  0x13efd7170\n",
+      "ggml_metal_init: loaded kernel_get_rows_q2_K                  0x13efd73d0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q3_K                  0x13efd7630\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_K                  0x13efd7890\n",
+      "ggml_metal_init: loaded kernel_get_rows_q5_K                  0x4744c9740\n",
+      "ggml_metal_init: loaded kernel_get_rows_q6_K                  0x4744ca6b0\n",
+      "ggml_metal_init: loaded kernel_rms_norm                       0x4744cb250\n",
+      "ggml_metal_init: loaded kernel_norm                           0x4744cb970\n",
+      "ggml_metal_init: loaded kernel_mul_mat_f16_f32                0x10e33f700\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_0_f32               0x10e33fcd0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_1_f32               0x4744cc2d0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q2_K_f32               0x4744cc6f0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q3_K_f32               0x4744cd6b0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_K_f32               0x4744cde20\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q5_K_f32               0x10e33ff30\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q6_K_f32               0x10e340190\n",
+      "ggml_metal_init: loaded kernel_rope                           0x10e3403f0\n",
+      "ggml_metal_init: loaded kernel_alibi_f32                      0x10e340de0\n",
+      "ggml_metal_init: loaded kernel_cpy_f32_f16                    0x10e3416d0\n",
+      "ggml_metal_init: loaded kernel_cpy_f32_f32                    0x10e342080\n",
+      "ggml_metal_init: loaded kernel_cpy_f16_f16                    0x10e342ca0\n",
+      "ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB\n",
+      "ggml_metal_init: hasUnifiedMemory             = true\n",
+      "ggml_metal_init: maxTransferRate              = built-in GPU\n",
+      "ggml_metal_add_buffer: allocated 'data            ' buffer, size =  6984.06 MB, ( 6986.19 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'eval            ' buffer, size =  1032.00 MB, ( 8018.19 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'kv              ' buffer, size =  1602.00 MB, ( 9620.19 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'scr0            ' buffer, size =   426.00 MB, (10046.19 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'scr1            ' buffer, size =   512.00 MB, (10558.19 / 21845.34)\n",
+      "AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | \n"
+     ]
+    }
+   ],
    "source": [
     "from langchain.llms import LlamaCpp\n",
     "llm = LlamaCpp(\n",
-    "    model_path=\"/Users/rlm/Desktop/Code/llama.cpp/models/openorca-platypus2-13b.gguf.q4_0.bin\",\n",
+    "    model_path=\"/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\",\n",
     "    n_gpu_layers=1,\n",
     "    n_batch=512,\n",
     "    n_ctx=2048,\n",
@@ -379,10 +448,87 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "915ecd4c-8f6b-4de3-a787-b64cb7c682b4",
+   "execution_count": 46,
+   "id": "b55a2147",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Found model file at  /Users/rlm/Desktop/Code/gpt4all/models/nous-hermes-13b.ggmlv3.q4_0.bin\n",
+      "llama_new_context_with_model: max tensor size =    87.89 MB\n",
+      "llama_new_context_with_model: max tensor size =    87.89 MB\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "llama.cpp: using Metal\n",
+      "llama.cpp: loading model from /Users/rlm/Desktop/Code/gpt4all/models/nous-hermes-13b.ggmlv3.q4_0.bin\n",
+      "llama_model_load_internal: format     = ggjt v3 (latest)\n",
+      "llama_model_load_internal: n_vocab    = 32001\n",
+      "llama_model_load_internal: n_ctx      = 2048\n",
+      "llama_model_load_internal: n_embd     = 5120\n",
+      "llama_model_load_internal: n_mult     = 256\n",
+      "llama_model_load_internal: n_head     = 40\n",
+      "llama_model_load_internal: n_layer    = 40\n",
+      "llama_model_load_internal: n_rot      = 128\n",
+      "llama_model_load_internal: ftype      = 2 (mostly Q4_0)\n",
+      "llama_model_load_internal: n_ff       = 13824\n",
+      "llama_model_load_internal: n_parts    = 1\n",
+      "llama_model_load_internal: model size = 13B\n",
+      "llama_model_load_internal: ggml ctx size =    0.09 MB\n",
+      "llama_model_load_internal: mem required  = 9031.71 MB (+ 1608.00 MB per state)\n",
+      "llama_new_context_with_model: kv self size  = 1600.00 MB\n",
+      "ggml_metal_init: allocating\n",
+      "ggml_metal_init: using MPS\n",
+      "ggml_metal_init: loading '/Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/gpt4all/llmodel_DO_NOT_MODIFY/build/ggml-metal.metal'\n",
+      "ggml_metal_init: loaded kernel_add                            0x37944d850\n",
+      "ggml_metal_init: loaded kernel_mul                            0x37944f350\n",
+      "ggml_metal_init: loaded kernel_mul_row                        0x37944fdd0\n",
+      "ggml_metal_init: loaded kernel_scale                          0x3794505a0\n",
+      "ggml_metal_init: loaded kernel_silu                           0x379450800\n",
+      "ggml_metal_init: loaded kernel_relu                           0x379450a60\n",
+      "ggml_metal_init: loaded kernel_gelu                           0x379450cc0\n",
+      "ggml_metal_init: loaded kernel_soft_max                       0x379450ff0\n",
+      "ggml_metal_init: loaded kernel_diag_mask_inf                  0x379451250\n",
+      "ggml_metal_init: loaded kernel_get_rows_f16                   0x3794514b0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_0                  0x379451710\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_1                  0x379451970\n",
+      "ggml_metal_init: loaded kernel_get_rows_q2_k                  0x379451bd0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q3_k                  0x379451e30\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_k                  0x379452090\n",
+      "ggml_metal_init: loaded kernel_get_rows_q5_k                  0x3794522f0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q6_k                  0x379452550\n",
+      "ggml_metal_init: loaded kernel_rms_norm                       0x3794527b0\n",
+      "ggml_metal_init: loaded kernel_norm                           0x379452a10\n",
+      "ggml_metal_init: loaded kernel_mul_mat_f16_f32                0x379452c70\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_0_f32               0x379452ed0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_1_f32               0x379453130\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q2_k_f32               0x379453390\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q3_k_f32               0x3794535f0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_k_f32               0x379453850\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q5_k_f32               0x379453ab0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q6_k_f32               0x379453d10\n",
+      "ggml_metal_init: loaded kernel_rope                           0x379453f70\n",
+      "ggml_metal_init: loaded kernel_alibi_f32                      0x3794541d0\n",
+      "ggml_metal_init: loaded kernel_cpy_f32_f16                    0x379454430\n",
+      "ggml_metal_init: loaded kernel_cpy_f32_f32                    0x379454690\n",
+      "ggml_metal_init: loaded kernel_cpy_f16_f16                    0x3794548f0\n",
+      "ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB\n",
+      "ggml_metal_init: hasUnifiedMemory             = true\n",
+      "ggml_metal_init: maxTransferRate              = built-in GPU\n",
+      "ggml_metal_add_buffer: allocated 'data            ' buffer, size =  6984.06 MB, (17542.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'eval            ' buffer, size =  1024.00 MB, (18566.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'kv              ' buffer, size =  1602.00 MB, (20168.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'scr0            ' buffer, size =   512.00 MB, (20680.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'scr1            ' buffer, size =   512.00 MB, (21192.94 / 21845.34)\n",
+      "ggml_metal_free: deallocating\n"
+     ]
+    }
+   ],
    "source": [
     "from langchain.llms import GPT4All\n",
     "llm = GPT4All(model=\"/Users/rlm/Desktop/Code/gpt4all/models/nous-hermes-13b.ggmlv3.q4_0.bin\")"
@@ -418,21 +564,89 @@
     "\n",
     "Some LLMs will benefit from specific prompts.\n",
     "\n",
-    "For example, LLaMA will use [special tokens](https://twitter.com/RLanceMartin/status/1681879318493003776?s=20).\n",
+    "For example, llama2 can use [special tokens](https://twitter.com/RLanceMartin/status/1681879318493003776?s=20).\n",
     "\n",
     "We can use `ConditionalPromptSelector` to set prompt based on the model type."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "16759b7c-7903-4269-b7b4-f83b313d8091",
+   "execution_count": 57,
+   "id": "d082b10a",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "llama.cpp: loading model from /Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\n",
+      "llama_model_load_internal: format     = ggjt v3 (latest)\n",
+      "llama_model_load_internal: n_vocab    = 32000\n",
+      "llama_model_load_internal: n_ctx      = 2048\n",
+      "llama_model_load_internal: n_embd     = 5120\n",
+      "llama_model_load_internal: n_mult     = 256\n",
+      "llama_model_load_internal: n_head     = 40\n",
+      "llama_model_load_internal: n_layer    = 40\n",
+      "llama_model_load_internal: n_rot      = 128\n",
+      "llama_model_load_internal: freq_base  = 10000.0\n",
+      "llama_model_load_internal: freq_scale = 1\n",
+      "llama_model_load_internal: ftype      = 2 (mostly Q4_0)\n",
+      "llama_model_load_internal: n_ff       = 13824\n",
+      "llama_model_load_internal: model size = 13B\n",
+      "llama_model_load_internal: ggml ctx size =    0.09 MB\n",
+      "llama_model_load_internal: mem required  = 8953.71 MB (+ 1608.00 MB per state)\n",
+      "llama_new_context_with_model: kv self size  = 1600.00 MB\n",
+      "ggml_metal_init: allocating\n",
+      "ggml_metal_init: using MPS\n",
+      "ggml_metal_init: loading '/Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/ggml-metal.metal'\n",
+      "ggml_metal_init: loaded kernel_add                            0x4744d09d0\n",
+      "ggml_metal_init: loaded kernel_mul                            0x3781cb3d0\n",
+      "ggml_metal_init: loaded kernel_mul_row                        0x37813bb60\n",
+      "ggml_metal_init: loaded kernel_scale                          0x474481080\n",
+      "ggml_metal_init: loaded kernel_silu                           0x4744d29f0\n",
+      "ggml_metal_init: loaded kernel_relu                           0x3781254c0\n",
+      "ggml_metal_init: loaded kernel_gelu                           0x47447f280\n",
+      "ggml_metal_init: loaded kernel_soft_max                       0x4744cf470\n",
+      "ggml_metal_init: loaded kernel_diag_mask_inf                  0x4744cf6d0\n",
+      "ggml_metal_init: loaded kernel_get_rows_f16                   0x4744cf930\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_0                  0x4744cfb90\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_1                  0x4744cfdf0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q2_K                  0x4744d0050\n",
+      "ggml_metal_init: loaded kernel_get_rows_q3_K                  0x4744ce980\n",
+      "ggml_metal_init: loaded kernel_get_rows_q4_K                  0x4744cebe0\n",
+      "ggml_metal_init: loaded kernel_get_rows_q5_K                  0x4744cee40\n",
+      "ggml_metal_init: loaded kernel_get_rows_q6_K                  0x4744cf0a0\n",
+      "ggml_metal_init: loaded kernel_rms_norm                       0x474482450\n",
+      "ggml_metal_init: loaded kernel_norm                           0x4744826b0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_f16_f32                0x474482910\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_0_f32               0x474482b70\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_1_f32               0x474482dd0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q2_K_f32               0x474483030\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q3_K_f32               0x474483290\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q4_K_f32               0x4744834f0\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q5_K_f32               0x474483750\n",
+      "ggml_metal_init: loaded kernel_mul_mat_q6_K_f32               0x4744839b0\n",
+      "ggml_metal_init: loaded kernel_rope                           0x474483c10\n",
+      "ggml_metal_init: loaded kernel_alibi_f32                      0x474483e70\n",
+      "ggml_metal_init: loaded kernel_cpy_f32_f16                    0x4744840d0\n",
+      "ggml_metal_init: loaded kernel_cpy_f32_f32                    0x474484330\n",
+      "ggml_metal_init: loaded kernel_cpy_f16_f16                    0x474484590\n",
+      "ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB\n",
+      "ggml_metal_init: hasUnifiedMemory             = true\n",
+      "ggml_metal_init: maxTransferRate              = built-in GPU\n",
+      "ggml_metal_add_buffer: allocated 'data            ' buffer, size =  6984.06 MB, ( 6986.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'eval            ' buffer, size =  1032.00 MB, ( 8018.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'kv              ' buffer, size =  1602.00 MB, ( 9620.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'scr0            ' buffer, size =   426.00 MB, (10046.94 / 21845.34)\n",
+      "ggml_metal_add_buffer: allocated 'scr1            ' buffer, size =   512.00 MB, (10558.94 / 21845.34)\n",
+      "AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | \n"
+     ]
+    }
+   ],
    "source": [
     "# Set our LLM\n",
     "llm = LlamaCpp(\n",
-    "    model_path=\"/Users/rlm/Desktop/Code/llama.cpp/models/openorca-platypus2-13b.gguf.q4_0.bin\",\n",
+    "    model_path=\"/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\",\n",
     "    n_gpu_layers=1,\n",
     "    n_batch=512,\n",
     "    n_ctx=2048,\n",
@@ -447,7 +661,7 @@
    "id": "66656084",
    "metadata": {},
    "source": [
-    "Set the associated prompt based upon the model version."
+    "Set the associated prompt."
    ]
   },
   {
@@ -545,18 +759,6 @@
     "llm_chain.run({\"question\":question})"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "6e0d37e7-f1d9-4848-bf2c-c22392ee141f",
-   "metadata": {},
-   "source": [
-    "We also can use the LangChain Prompt Hub to fetch and / or store prompts that are model specific.\n",
-    "\n",
-    "This will work with your [LangSmith API key](https://docs.smith.langchain.com/).\n",
-    "\n",
-    "For example, [here](https://smith.langchain.com/hub/rlm/rag-prompt-llama) is a prompt for RAG with LLaMA-specific tokens."
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "6ba66260",
@@ -568,12 +770,16 @@
     "\n",
     "For example, here is a guide to [RAG](docs/use_cases/question_answering/how_to/local_retrieval_qa) with local LLMs.\n",
     "\n",
-    "In general, use cases for local LLMs can be driven by at least two factors:\n",
+    "In general, use cases for local model can be driven by at least two factors:\n",
     "\n",
     "* `Privacy`: private data (e.g., journals, etc) that a user does not want to share \n",
     "* `Cost`: text preprocessing (extraction/tagging), summarization, and agent simulations are token-use-intensive tasks\n",
     "\n",
-    "In addition, [here](https://blog.langchain.dev/using-langsmith-to-support-fine-tuning-of-open-source-llms/) is an overview on fine-tuning, which can utilize open source LLMs."
+    "There are a few approach to support specific use-cases: \n",
+    "\n",
+    "* Fine-tuning (e.g., [gpt-llm-trainer](https://github.com/mshumer/gpt-llm-trainer), [Anyscale](https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications)) \n",
+    "* [Function-calling](https://github.com/MeetKai/functionary/tree/main) for use-cases like extraction or tagging\n",
+    "\n"
    ]
   }
  ],
@@ -593,7 +799,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.9.16"
   }
  },
  "nbformat": 4,

docs/extras/guides/privacy/_category_.yml

@@ -1 +0,0 @@
-label: 'Privacy'

docs/extras/guides/privacy/presidio_data_anonymization.ipynb

@@ -1,469 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Data anonymization with Microsoft Presidio\n",
-    "\n",
-    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/extras/guides/privacy/presidio_data_anonymization.ipynb)\n",
-    "\n",
-    "## Use case\n",
-    "\n",
-    "Data anonymization is crucial before passing information to a language model like GPT-4 because it helps protect privacy and maintain confidentiality. If data is not anonymized, sensitive information such as names, addresses, contact numbers, or other identifiers linked to specific individuals could potentially be learned and misused. Hence, by obscuring or removing this personally identifiable information (PII), data can be used freely without compromising individuals' privacy rights or breaching data protection laws and regulations.\n",
-    "\n",
-    "## Overview\n",
-    "\n",
-    "Anonynization consists of two steps:\n",
-    "\n",
-    "1. **Identification:** Identify all data fields that contain personally identifiable information (PII).\n",
-    "2. **Replacement**: Replace all PIIs with pseudo values or codes that do not reveal any personal information about the individual but can be used for reference. We're not using regular encryption, because the language model won't be able to understand the meaning or context of the encrypted data.\n",
-    "\n",
-    "We use *Microsoft Presidio* together with *Faker* framework for anonymization purposes because of the wide range of functionalities they provide. The full implementation is available in `PresidioAnonymizer`.\n",
-    "\n",
-    "## Quickstart\n",
-    "\n",
-    "Below you will find the use case on how to leverage anonymization in LangChain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install necessary packages\n",
-    "# ! pip install langchain langchain-experimental openai presidio-analyzer presidio-anonymizer spacy Faker\n",
-    "# ! python -m spacy download en_core_web_lg"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "Let's see how PII anonymization works using a sample sentence:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My name is Laura Ruiz, call me at +1-412-982-8374x13414 or email me at javierwatkins@example.net'"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain_experimental.data_anonymizer import PresidioAnonymizer\n",
-    "\n",
-    "anonymizer = PresidioAnonymizer()\n",
-    "\n",
-    "anonymizer.anonymize(\n",
-    "    \"My name is Slim Shady, call me at 313-666-7440 or email me at real.slim.shady@gmail.com\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Using with LangChain Expression Language\n",
-    "\n",
-    "With LCEL we can easily chain together anonymization with the rest of our application."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Set env var OPENAI_API_KEY or load from a .env file:\n",
-    "# import dotenv\n",
-    "\n",
-    "# dotenv.load_dotenv()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "text = f\"\"\"Slim Shady recently lost his wallet. \n",
-    "Inside is some cash and his credit card with the number 4916 0387 9536 0861. \n",
-    "If you would find it, please call at 313-666-7440 or write an email here: real.slim.shady@gmail.com.\"\"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Dear Sir/Madam,\n",
-      "\n",
-      "We regret to inform you that Richard Fields has recently misplaced his wallet, which contains a sum of cash and his credit card bearing the number 30479847307774. \n",
-      "\n",
-      "Should you happen to come across it, we kindly request that you contact us immediately at 6439182672 or via email at frank45@example.com.\n",
-      "\n",
-      "Thank you for your attention to this matter.\n",
-      "\n",
-      "Yours faithfully,\n",
-      "\n",
-      "[Your Name]\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.prompts.prompt import PromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "anonymizer = PresidioAnonymizer()\n",
-    "\n",
-    "template = \"\"\"Rewrite this text into an official, short email:\n",
-    "\n",
-    "{anonymized_text}\"\"\"\n",
-    "prompt = PromptTemplate.from_template(template)\n",
-    "llm = ChatOpenAI(temperature=0)\n",
-    "\n",
-    "chain = {\"anonymized_text\": anonymizer.anonymize} | prompt | llm\n",
-    "response = chain.invoke(text)\n",
-    "print(response.content)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Customization\n",
-    "We can specify ``analyzed_fields`` to only anonymize particular types of data."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My name is Adrian Fleming, call me at 313-666-7440 or email me at real.slim.shady@gmail.com'"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer = PresidioAnonymizer(analyzed_fields=[\"PERSON\"])\n",
-    "\n",
-    "anonymizer.anonymize(\n",
-    "    \"My name is Slim Shady, call me at 313-666-7440 or email me at real.slim.shady@gmail.com\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "As can be observed, the name was correctly identified and replaced with another. The `analyzed_fields` attribute is responsible for what values are to be detected and substituted. We can add *PHONE_NUMBER* to the list:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My name is Justin Miller, call me at 761-824-1889 or email me at real.slim.shady@gmail.com'"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer = PresidioAnonymizer(analyzed_fields=[\"PERSON\", \"PHONE_NUMBER\"])\n",
-    "anonymizer.anonymize(\n",
-    "    \"My name is Slim Shady, call me at 313-666-7440 or email me at real.slim.shady@gmail.com\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "If no analyzed_fields are specified, by default the anonymizer will detect all supported formats. Below is the full list of them:\n",
-    "\n",
-    "`['PERSON', 'EMAIL_ADDRESS', 'PHONE_NUMBER', 'IBAN_CODE', 'CREDIT_CARD', 'CRYPTO', 'IP_ADDRESS', 'LOCATION', 'DATE_TIME', 'NRP', 'MEDICAL_LICENSE', 'URL', 'US_BANK_NUMBER', 'US_DRIVER_LICENSE', 'US_ITIN', 'US_PASSPORT', 'US_SSN']`\n",
-    "\n",
-    "**Disclaimer:** We suggest carefully defining the private data to be detected - Presidio doesn't work perfectly and it sometimes makes mistakes, so it's better to have more control over the data."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My name is Dr. Jennifer Baker, call me at (508)839-9329x232 or email me at ehamilton@example.com'"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer = PresidioAnonymizer()\n",
-    "anonymizer.anonymize(\n",
-    "    \"My name is Slim Shady, call me at 313-666-7440 or email me at real.slim.shady@gmail.com\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "It may be that the above list of detected fields is not sufficient. For example, the already available *PHONE_NUMBER* field does not support polish phone numbers and confuses it with another field:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My polish phone number is NRGN41434238921378'"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer = PresidioAnonymizer()\n",
-    "anonymizer.anonymize(\"My polish phone number is 666555444\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "You can then write your own recognizers and add them to the pool of those present. How exactly to create recognizers is described in the [Presidio documentation](https://microsoft.github.io/presidio/samples/python/customizing_presidio_analyzer/)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Define the regex pattern in a Presidio `Pattern` object:\n",
-    "from presidio_analyzer import Pattern, PatternRecognizer\n",
-    "\n",
-    "\n",
-    "polish_phone_numbers_pattern = Pattern(\n",
-    "    name=\"polish_phone_numbers_pattern\",\n",
-    "    regex=\"(?<!\\w)(\\(?(\\+|00)?48\\)?)?[ -]?\\d{3}[ -]?\\d{3}[ -]?\\d{3}(?!\\w)\",\n",
-    "    score=1,\n",
-    ")\n",
-    "\n",
-    "# Define the recognizer with one or more patterns\n",
-    "polish_phone_numbers_recognizer = PatternRecognizer(\n",
-    "    supported_entity=\"POLISH_PHONE_NUMBER\", patterns=[polish_phone_numbers_pattern]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "Now, we can add recognizer by calling `add_recognizer` method on the anonymizer:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "anonymizer.add_recognizer(polish_phone_numbers_recognizer)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "And voilà! With the added pattern-based recognizer, the anonymizer now handles polish phone numbers."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "My polish phone number is <POLISH_PHONE_NUMBER>\n",
-      "My polish phone number is <POLISH_PHONE_NUMBER>\n",
-      "My polish phone number is <POLISH_PHONE_NUMBER>\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(anonymizer.anonymize(\"My polish phone number is 666555444\"))\n",
-    "print(anonymizer.anonymize(\"My polish phone number is 666 555 444\"))\n",
-    "print(anonymizer.anonymize(\"My polish phone number is +48 666 555 444\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "The problem is - even though we recognize polish phone numbers now, we don't have a method (operator) that would tell how to substitute a given field - because of this, in the outpit we only provide string `<POLISH_PHONE_NUMBER>` We need to create a method to replace it correctly: "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'511 622 683'"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from faker import Faker\n",
-    "\n",
-    "fake = Faker(locale=\"pl_PL\")\n",
-    "\n",
-    "\n",
-    "def fake_polish_phone_number(_=None):\n",
-    "    return fake.phone_number()\n",
-    "\n",
-    "\n",
-    "fake_polish_phone_number()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\\\n",
-    "We used Faker to create pseudo data. Now we can create an operator and add it to the anonymizer. For complete information about operators and their creation, see the Presidio documentation for [simple](https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/) and [custom](https://microsoft.github.io/presidio/tutorial/11_custom_anonymization/) anonymization."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from presidio_anonymizer.entities import OperatorConfig\n",
-    "\n",
-    "new_operators = {\n",
-    "    \"POLISH_PHONE_NUMBER\": OperatorConfig(\n",
-    "        \"custom\", {\"lambda\": fake_polish_phone_number}\n",
-    "    )\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "anonymizer.add_operators(new_operators)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My polish phone number is +48 734 630 977'"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer.anonymize(\"My polish phone number is 666555444\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Future works\n",
-    "\n",
-    "- **deanonymization** - add the ability to reverse anonymization. For example, the workflow could look like this: `anonymize -> LLMChain -> deanonymize`. By doing this, we will retain anonymity in requests to, for example, OpenAI, and then be able restore the original data.\n",
-    "- **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/extras/guides/privacy/presidio_reversible_anonymization.ipynb

@@ -1,461 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Reversible data anonymization with Microsoft Presidio\n",
-    "\n",
-    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/extras/guides/privacy/presidio_reversible_anonymization.ipynb)\n",
-    "\n",
-    "\n",
-    "## Use case\n",
-    "\n",
-    "We have already written about the importance of anonymizing sensitive data in the previous section. **Reversible Anonymization** is an equally essential technology while sharing information with language models, as it balances data protection with data usability. This technique involves masking sensitive personally identifiable information (PII), yet it can be reversed and original data can be restored when authorized users need it. Its main advantage lies in the fact that while it conceals individual identities to prevent misuse, it also allows the concealed data to be accurately unmasked should it be necessary for legal or compliance purposes. \n",
-    "\n",
-    "## Overview\n",
-    "\n",
-    "We implemented the `PresidioReversibleAnonymizer`, which consists of two parts:\n",
-    "\n",
-    "1. anonymization - it works the same way as `PresidioAnonymizer`, plus the object itself stores a mapping of made-up values to original ones, for example:\n",
-    "```\n",
-    "    {\n",
-    "        \"PERSON\": {\n",
-    "            \"<anonymized>\": \"<original>\",\n",
-    "            \"John Doe\": \"Slim Shady\"\n",
-    "        },\n",
-    "        \"PHONE_NUMBER\": {\n",
-    "            \"111-111-1111\": \"555-555-5555\"\n",
-    "        }\n",
-    "        ...\n",
-    "    }\n",
-    "```\n",
-    "\n",
-    "2. deanonymization - using the mapping described above, it matches fake data with original data and then substitutes it.\n",
-    "\n",
-    "Between anonymization and deanonymization user can perform different operations, for example, passing the output to LLM.\n",
-    "\n",
-    "## Quickstart\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install necessary packages\n",
-    "# ! pip install langchain langchain-experimental openai presidio-analyzer presidio-anonymizer spacy Faker\n",
-    "# ! python -m spacy download en_core_web_lg"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "`PresidioReversibleAnonymizer` is not significantly different from its predecessor (`PresidioAnonymizer`) in terms of anonymization:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'My name is Maria Lynch, call me at 7344131647 or email me at jamesmichael@example.com. By the way, my card number is: 4838637940262'"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain_experimental.data_anonymizer import PresidioReversibleAnonymizer\n",
-    "\n",
-    "anonymizer = PresidioReversibleAnonymizer(\n",
-    "    analyzed_fields=[\"PERSON\", \"PHONE_NUMBER\", \"EMAIL_ADDRESS\", \"CREDIT_CARD\"],\n",
-    "    # Faker seed is used here to make sure the same fake data is generated for the test purposes\n",
-    "    # In production, it is recommended to remove the faker_seed parameter (it will default to None)\n",
-    "    faker_seed=42,\n",
-    ")\n",
-    "\n",
-    "anonymizer.anonymize(\n",
-    "    \"My name is Slim Shady, call me at 313-666-7440 or email me at real.slim.shady@gmail.com. \"\n",
-    "    \"By the way, my card number is: 4916 0387 9536 0861\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "This is what the full string we want to deanonymize looks like:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Maria Lynch recently lost his wallet. \n",
-      "Inside is some cash and his credit card with the number 4838637940262. \n",
-      "If you would find it, please call at 7344131647 or write an email here: jamesmichael@example.com.\n",
-      "Maria Lynch would be very grateful!\n"
-     ]
-    }
-   ],
-   "source": [
-    "# We know this data, as we set the faker_seed parameter\n",
-    "fake_name = \"Maria Lynch\"\n",
-    "fake_phone = \"7344131647\"\n",
-    "fake_email = \"jamesmichael@example.com\"\n",
-    "fake_credit_card = \"4838637940262\"\n",
-    "\n",
-    "anonymized_text = f\"\"\"{fake_name} recently lost his wallet. \n",
-    "Inside is some cash and his credit card with the number {fake_credit_card}. \n",
-    "If you would find it, please call at {fake_phone} or write an email here: {fake_email}.\n",
-    "{fake_name} would be very grateful!\"\"\"\n",
-    "\n",
-    "print(anonymized_text)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "And now, using the `deanonymize` method, we can reverse the process:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Slim Shady recently lost his wallet. \n",
-      "Inside is some cash and his credit card with the number 4916 0387 9536 0861. \n",
-      "If you would find it, please call at 313-666-7440 or write an email here: real.slim.shady@gmail.com.\n",
-      "Slim Shady would be very grateful!\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(anonymizer.deanonymize(anonymized_text))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Using with LangChain Expression Language\n",
-    "\n",
-    "With LCEL we can easily chain together anonymization and deanonymization with the rest of our application. This is an example of using the anonymization mechanism with a query to LLM (without deanonymization for now):"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "text = f\"\"\"Slim Shady recently lost his wallet. \n",
-    "Inside is some cash and his credit card with the number 4916 0387 9536 0861. \n",
-    "If you would find it, please call at 313-666-7440 or write an email here: real.slim.shady@gmail.com.\"\"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Dear Sir/Madam,\n",
-      "\n",
-      "We regret to inform you that Mr. Dana Rhodes has reported the loss of his wallet. The wallet contains a sum of cash and his credit card, bearing the number 4397528473885757. \n",
-      "\n",
-      "If you happen to come across the aforementioned wallet, we kindly request that you contact us immediately at 258-481-7074x714 or via email at laurengoodman@example.com.\n",
-      "\n",
-      "Your prompt assistance in this matter would be greatly appreciated.\n",
-      "\n",
-      "Yours faithfully,\n",
-      "\n",
-      "[Your Name]\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.prompts.prompt import PromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "anonymizer = PresidioReversibleAnonymizer()\n",
-    "\n",
-    "template = \"\"\"Rewrite this text into an official, short email:\n",
-    "\n",
-    "{anonymized_text}\"\"\"\n",
-    "prompt = PromptTemplate.from_template(template)\n",
-    "llm = ChatOpenAI(temperature=0)\n",
-    "\n",
-    "chain = {\"anonymized_text\": anonymizer.anonymize} | prompt | llm\n",
-    "response = chain.invoke(text)\n",
-    "print(response.content)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Now, let's add **deanonymization step** to our sequence:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Dear Sir/Madam,\n",
-      "\n",
-      "We regret to inform you that Mr. Slim Shady has recently misplaced his wallet. The wallet contains a sum of cash and his credit card, bearing the number 4916 0387 9536 0861. \n",
-      "\n",
-      "If by any chance you come across the lost wallet, kindly contact us immediately at 313-666-7440 or send an email to real.slim.shady@gmail.com.\n",
-      "\n",
-      "Your prompt assistance in this matter would be greatly appreciated.\n",
-      "\n",
-      "Yours faithfully,\n",
-      "\n",
-      "[Your Name]\n"
-     ]
-    }
-   ],
-   "source": [
-    "chain = chain | (lambda ai_message: anonymizer.deanonymize(ai_message.content))\n",
-    "response = chain.invoke(text)\n",
-    "print(response)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Anonymized data was given to the model itself, and therefore it was protected from being leaked to the outside world. Then, the model's response was processed, and the factual value was replaced with the real one."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Extra knowledge"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "`PresidioReversibleAnonymizer` stores the mapping of the fake values to the original values in the `deanonymizer_mapping` parameter, where key is fake PII and value is the original one: "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'PERSON': {'Maria Lynch': 'Slim Shady'},\n",
-       " 'PHONE_NUMBER': {'7344131647': '313-666-7440'},\n",
-       " 'EMAIL_ADDRESS': {'jamesmichael@example.com': 'real.slim.shady@gmail.com'},\n",
-       " 'CREDIT_CARD': {'4838637940262': '4916 0387 9536 0861'}}"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain_experimental.data_anonymizer import PresidioReversibleAnonymizer\n",
-    "\n",
-    "anonymizer = PresidioReversibleAnonymizer(\n",
-    "    analyzed_fields=[\"PERSON\", \"PHONE_NUMBER\", \"EMAIL_ADDRESS\", \"CREDIT_CARD\"],\n",
-    "    # Faker seed is used here to make sure the same fake data is generated for the test purposes\n",
-    "    # In production, it is recommended to remove the faker_seed parameter (it will default to None)\n",
-    "    faker_seed=42,\n",
-    ")\n",
-    "\n",
-    "anonymizer.anonymize(\n",
-    "    \"My name is Slim Shady, call me at 313-666-7440 or email me at real.slim.shady@gmail.com. \"\n",
-    "    \"By the way, my card number is: 4916 0387 9536 0861\"\n",
-    ")\n",
-    "\n",
-    "anonymizer.deanonymizer_mapping"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Anonymizing more texts will result in new mapping entries:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Do you have his VISA card number? Yep, it's 3537672423884966. I'm William Bowman by the way.\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'PERSON': {'Maria Lynch': 'Slim Shady', 'William Bowman': 'John Doe'},\n",
-       " 'PHONE_NUMBER': {'7344131647': '313-666-7440'},\n",
-       " 'EMAIL_ADDRESS': {'jamesmichael@example.com': 'real.slim.shady@gmail.com'},\n",
-       " 'CREDIT_CARD': {'4838637940262': '4916 0387 9536 0861',\n",
-       "  '3537672423884966': '4001 9192 5753 7193'}}"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "print(\n",
-    "    anonymizer.anonymize(\n",
-    "        \"Do you have his VISA card number? Yep, it's 4001 9192 5753 7193. I'm John Doe by the way.\"\n",
-    "    )\n",
-    ")\n",
-    "\n",
-    "anonymizer.deanonymizer_mapping"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can save the mapping itself to a file for future use: "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# We can save the deanonymizer mapping as a JSON or YAML file\n",
-    "\n",
-    "anonymizer.save_deanonymizer_mapping(\"deanonymizer_mapping.json\")\n",
-    "# anonymizer.save_deanonymizer_mapping(\"deanonymizer_mapping.yaml\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "And then, load it in another `PresidioReversibleAnonymizer` instance:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{}"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer = PresidioReversibleAnonymizer()\n",
-    "\n",
-    "anonymizer.deanonymizer_mapping"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'PERSON': {'Maria Lynch': 'Slim Shady', 'William Bowman': 'John Doe'},\n",
-       " 'PHONE_NUMBER': {'7344131647': '313-666-7440'},\n",
-       " 'EMAIL_ADDRESS': {'jamesmichael@example.com': 'real.slim.shady@gmail.com'},\n",
-       " 'CREDIT_CARD': {'4838637940262': '4916 0387 9536 0861',\n",
-       "  '3537672423884966': '4001 9192 5753 7193'}}"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "anonymizer.load_deanonymizer_mapping(\"deanonymizer_mapping.json\")\n",
-    "\n",
-    "anonymizer.deanonymizer_mapping"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Future works\n",
-    "\n",
-    "- **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object.\n",
-    "- **better matching and substitution of fake values for real ones** - currently the strategy is based on matching full strings and then substituting them. Due to the indeterminism of language models, it may happen that the value in the answer is slightly changed (e.g. *John Doe* -> *John* or *Main St, New York* -> *New York*) and such a substitution is then no longer possible. Therefore, it is worth adjusting the matching for your needs."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/extras/guides/pydantic_compatibility.md

@@ -1,10 +1,10 @@
-# Pydantic compatibility
+# Pydantic Compatibility
 
 - Pydantic v2 was released in June, 2023 (https://docs.pydantic.dev/2.0/blog/pydantic-v2-final/)
 - v2 contains has a number of breaking changes (https://docs.pydantic.dev/2.0/migration/)
 - Pydantic v2 and v1 are under the same package name, so both versions cannot be installed at the same time
 
-## LangChain Pydantic migration plan
+## LangChain Pydantic Migration Plan
 
 As of `langchain>=0.0.267`, LangChain will allow users to install either Pydantic V1 or V2. 
    * Internally LangChain will continue to [use V1](https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features).

docs/extras/integrations/callbacks/context.ipynb

@@ -93,7 +93,7 @@
    "metadata": {},
    "source": [
     "## Usage\n",
-    "### Using the Context callback within a chat model\n",
+    "### Using the Context callback within a Chat Model\n",
     "\n",
     "The Context callback handler can be used to directly record transcripts between users and AI assistants.\n",
     "\n",

docs/extras/integrations/callbacks/llmonitor.md

@@ -1,63 +0,0 @@
-# LLMonitor
-
-[LLMonitor](https://llmonitor.com) is an open-source observability platform that provides cost tracking, user tracking and powerful agent tracing.
-
-<video controls width='100%' >
-  <source src='https://llmonitor.com/videos/demo-annotated.mp4'/>
-</video>
-
-## Setup
-Create an account on [llmonitor.com](https://llmonitor.com), create an `App`, and then copy the associated `tracking id`.
-Once you have it, set it as an environment variable by running:
-```bash
-export LLMONITOR_APP_ID="..."
-```
-
-If you'd prefer not to set an environment variable, you can pass the key directly when initializing the callback handler:
-```python
-from langchain.callbacks import LLMonitorCallbackHandler
-
-handler = LLMonitorCallbackHandler(app_id="...")
-```
-
-## Usage with LLM/Chat models
-```python
-from langchain.llms import OpenAI
-from langchain.chat_models import ChatOpenAI
-from langchain.callbacks import LLMonitorCallbackHandler
-
-handler = LLMonitorCallbackHandler(app_id="...")
-
-llm = OpenAI(
-    callbacks=[handler],
-)
-
-chat = ChatOpenAI(
-    callbacks=[handler],
-    metadata={"userId": "123"},  # you can assign user ids to models in the metadata
-)
-```
-
-
-## Usage with agents
-```python
-from langchain.agents import load_tools, initialize_agent, AgentType
-from langchain.llms import OpenAI
-from langchain.callbacks import LLMonitorCallbackHandler
-
-handler = LLMonitorCallbackHandler(app_id="...")
-
-llm = OpenAI(temperature=0)
-tools = load_tools(["serpapi", "llm-math"], llm=llm)
-agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)
-agent.run(
-    "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?",
-    callbacks=[handler],
-    metadata={
-        "agentName": "Leo DiCaprio's girlfriend",  # you can assign a custom agent in the metadata
-    },
-)
-```
-
-## Support
-For any question or issue with integration you can reach out to the LLMonitor team on [Discord](http://discord.com/invite/8PafSG58kK) or via [email](mailto:vince@llmonitor.com).

docs/extras/integrations/callbacks/promptlayer.ipynb

@@ -11,7 +11,7 @@
     "\n",
     "[PromptLayer](https://promptlayer.com) is a an LLM observability platform that lets you visualize requests, version prompts, and track usage. In this guide we will go over how to setup the `PromptLayerCallbackHandler`. \n",
     "\n",
-    "While PromptLayer does have LLMs that integrate directly with LangChain (e.g. [`PromptLayerOpenAI`](https://python.langchain.com/docs/integrations/llms/promptlayer_openai)), this callback is the recommended way to integrate PromptLayer with LangChain.\n",
+    "While PromptLayer does have LLMs that integrate directly with LangChain (eg [`PromptLayerOpenAI`](https://python.langchain.com/docs/integrations/llms/promptlayer_openai)), this callback is the recommended way to integrate PromptLayer with LangChain.\n",
     "\n",
     "See [our docs](https://docs.promptlayer.com/languages/langchain) for more information."
    ]

docs/extras/integrations/chat/bedrock.ipynb

@@ -1,106 +0,0 @@
-{
-    "cells": [
-        {
-            "cell_type": "markdown",
-            "id": "bf733a38-db84-4363-89e2-de6735c37230",
-            "metadata": {},
-            "source": [
-                "# Bedrock Chat\n",
-                "\n",
-                "[Amazon Bedrock](https://aws.amazon.com/bedrock/) is a fully managed service that makes FMs from leading AI startups and Amazon available via an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "id": "d51edc81",
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "%pip install boto3"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 1,
-            "id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "from langchain.chat_models import BedrockChat\n",
-                "from langchain.schema import HumanMessage"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 2,
-            "id": "70cf04e8-423a-4ff6-8b09-f11fb711c817",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [],
-            "source": [
-                "chat = BedrockChat(model_id=\"anthropic.claude-v2\", model_kwargs={\"temperature\":0.1})"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": 3,
-            "id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
-            "metadata": {
-                "tags": []
-            },
-            "outputs": [
-                {
-                    "data": {
-                        "text/plain": [
-                            "AIMessage(content=\" Voici la traduction en français : J'adore programmer.\", additional_kwargs={}, example=False)"
-                        ]
-                    },
-                    "execution_count": 3,
-                    "metadata": {},
-                    "output_type": "execute_result"
-                }
-            ],
-            "source": [
-                "messages = [\n",
-                "    HumanMessage(\n",
-                "        content=\"Translate this sentence from English to French. I love programming.\"\n",
-                "    )\n",
-                "]\n",
-                "chat(messages)"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "id": "c253883f",
-            "metadata": {},
-            "outputs": [],
-            "source": []
-        }
-    ],
-    "metadata": {
-        "kernelspec": {
-            "display_name": "Python 3 (ipykernel)",
-            "language": "python",
-            "name": "python3"
-        },
-        "language_info": {
-            "codemirror_mode": {
-                "name": "ipython",
-                "version": 3
-            },
-            "file_extension": ".py",
-            "mimetype": "text/x-python",
-            "name": "python",
-            "nbconvert_exporter": "python",
-            "pygments_lexer": "ipython3",
-            "version": "3.11.4"
-        }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/extras/integrations/chat/ollama.ipynb

@@ -1,382 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Ollama\n",
-    "\n",
-    "[Ollama](https://ollama.ai/) allows you to run open-source large language models, such as LLaMA2, locally.\n",
-    "\n",
-    "Ollama bundles model weights, configuration, and data into a single package, defined by a Modelfile. \n",
-    "\n",
-    "It optimizes setup and configuration details, including GPU usage.\n",
-    "\n",
-    "For a complete list of supported models and model variants, see the [Ollama model library](https://ollama.ai/library).\n",
-    "\n",
-    "## Setup\n",
-    "\n",
-    "First, follow [these instructions](https://github.com/jmorganca/ollama) to set up and run a local Ollama instance:\n",
-    "\n",
-    "* [Download](https://ollama.ai/download)\n",
-    "* Fetch a model via `ollama pull <model family>`\n",
-    "* e.g., for `Llama-7b`: `ollama pull llama2`\n",
-    "* This will download the most basic version of the model (e.g., minimum # parameters and 4-bit quantization)\n",
-    "* On Mac, it will download to:\n",
-    "\n",
-    "`~/.ollama/models/manifests/registry.ollama.ai/library/<model family>/latest`\n",
-    "\n",
-    "* And we can specify a particular version, e.g., for `ollama pull vicuna:13b-v1.5-16k-q4_0`\n",
-    "* The file is here with the model version in place of `latest`\n",
-    "\n",
-    "`~/.ollama/models/manifests/registry.ollama.ai/library/vicuna/13b-v1.5-16k-q4_0`\n",
-    "\n",
-    "You can easily access models in a few ways:\n",
-    "\n",
-    "1/ if the app is running:\n",
-    "* All of your local models are automatically served on `localhost:11434`\n",
-    "* Select your model when setting `llm = Ollama(..., model=\"<model family>:<version>\")`\n",
-    "* If you set `llm = Ollama(..., model=\"<model family\")` withoout a version it will simply look for `latest`\n",
-    "\n",
-    "2/ if building from source or just running the binary: \n",
-    "* Then you must run `ollama serve`\n",
-    "* All of your local models are automatically served on `localhost:11434`\n",
-    "* Then, select as shown above\n",
-    "\n",
-    "\n",
-    "## Usage\n",
-    "\n",
-    "You can see a full list of supported parameters on the [API reference page](https://api.python.langchain.com/en/latest/llms/langchain.llms.ollama.Ollama.html).\n",
-    "\n",
-    "If you are using a LLaMA `chat` model (e.g., `ollama pull llama2:7b-chat`) then you can use the `ChatOllama` interface.\n",
-    "\n",
-    "This includes [special tokens](https://huggingface.co/blog/llama2#how-to-prompt-llama-2) for system message and user input."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOllama\n",
-    "from langchain.callbacks.manager import CallbackManager\n",
-    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler                                  \n",
-    "chat_model = ChatOllama(model=\"llama2:7b-chat\", \n",
-    "                        callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "With `StreamingStdOutCallbackHandler`, you will see tokens streamed."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      " Artificial intelligence (AI) has a rich and varied history that spans several decades. Hinweis: The following is a brief overview of the major milestones in the history of AI, but it is by no means exhaustive.\n",
-      "\n",
-      "1. Early Beginnings (1950s-1960s): The term \"Artificial Intelligence\" was coined in 1956 by computer scientist John McCarthy. However, the concept of creating machines that can think and learn like humans dates back to ancient times. In the 1950s and 1960s, researchers began exploring the possibilities of AI using simple algorithms and machine learning techniques.\n",
-      "2. Rule-Based Systems (1970s-1980s): In the 1970s and 1980s, AI research focused on developing rule-based systems, which use predefined rules to reason and make decisions. This led to the development of expert systems, which were designed to mimic the decision-making abilities of human experts in specific domains.\n",
-      "3. Machine Learning (1980s-1990s): The 1980s saw a shift towards machine learning, which enables machines to learn from data without being explicitly programmed. This led to the development of algorithms such as decision trees, neural networks, and support vector machines.\n",
-      "4. Deep Learning (2000s-present): In the early 2000s, deep learning emerged as a subfield of machine learning, focusing on neural networks with multiple layers. These networks can learn complex representations of data, leading to breakthroughs in image and speech recognition, natural language processing, and other areas.\n",
-      "5. Natural Language Processing (NLP) (1980s-present): NLP has been an active area of research since the 1980s, with a focus on developing algorithms that can understand and generate human language. This has led to applications such as chatbots, voice assistants, and language translation systems.\n",
-      "6. Robotics (1970s-present): The development of robotics has been closely tied to AI research, with a focus on creating machines that can perform tasks that typically require human intelligence, such as manipulation and locomotion.\n",
-      "7. Computer Vision (1980s-present): Computer vision has been an active area of research since the 1980s, with a focus on enabling machines to interpret and understand visual data from the world around us. This has led to applications such as image recognition, object detection, and autonomous driving.\n",
-      "8. Ethics and Society (1990s-present): As AI technology has become more advanced and integrated into various aspects of society, there has been a growing concern about the ethical implications of AI. This includes issues related to privacy, bias, and job displacement.\n",
-      "9. Reinforcement Learning (2000s-present): Reinforcement learning is a subfield of machine learning that involves training machines to make decisions based on feedback from their environment. This has led to breakthroughs in areas such as game playing, robotics, and autonomous driving.\n",
-      "10. Generative Models (2010s-present): Generative models are a class of AI algorithms that can generate new data that is similar to a given dataset. This has led to applications such as image synthesis, music generation, and language creation.\n",
-      "\n",
-      "These are just a few of the many developments in the history of AI. As the field continues to evolve, we can expect even more exciting breakthroughs and innovations in the years to come."
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Artificial intelligence (AI) has a rich and varied history that spans several decades. Hinweis: The following is a brief overview of the major milestones in the history of AI, but it is by no means exhaustive.\\n\\n1. Early Beginnings (1950s-1960s): The term \"Artificial Intelligence\" was coined in 1956 by computer scientist John McCarthy. However, the concept of creating machines that can think and learn like humans dates back to ancient times. In the 1950s and 1960s, researchers began exploring the possibilities of AI using simple algorithms and machine learning techniques.\\n2. Rule-Based Systems (1970s-1980s): In the 1970s and 1980s, AI research focused on developing rule-based systems, which use predefined rules to reason and make decisions. This led to the development of expert systems, which were designed to mimic the decision-making abilities of human experts in specific domains.\\n3. Machine Learning (1980s-1990s): The 1980s saw a shift towards machine learning, which enables machines to learn from data without being explicitly programmed. This led to the development of algorithms such as decision trees, neural networks, and support vector machines.\\n4. Deep Learning (2000s-present): In the early 2000s, deep learning emerged as a subfield of machine learning, focusing on neural networks with multiple layers. These networks can learn complex representations of data, leading to breakthroughs in image and speech recognition, natural language processing, and other areas.\\n5. Natural Language Processing (NLP) (1980s-present): NLP has been an active area of research since the 1980s, with a focus on developing algorithms that can understand and generate human language. This has led to applications such as chatbots, voice assistants, and language translation systems.\\n6. Robotics (1970s-present): The development of robotics has been closely tied to AI research, with a focus on creating machines that can perform tasks that typically require human intelligence, such as manipulation and locomotion.\\n7. Computer Vision (1980s-present): Computer vision has been an active area of research since the 1980s, with a focus on enabling machines to interpret and understand visual data from the world around us. This has led to applications such as image recognition, object detection, and autonomous driving.\\n8. Ethics and Society (1990s-present): As AI technology has become more advanced and integrated into various aspects of society, there has been a growing concern about the ethical implications of AI. This includes issues related to privacy, bias, and job displacement.\\n9. Reinforcement Learning (2000s-present): Reinforcement learning is a subfield of machine learning that involves training machines to make decisions based on feedback from their environment. This has led to breakthroughs in areas such as game playing, robotics, and autonomous driving.\\n10. Generative Models (2010s-present): Generative models are a class of AI algorithms that can generate new data that is similar to a given dataset. This has led to applications such as image synthesis, music generation, and language creation.\\n\\nThese are just a few of the many developments in the history of AI. As the field continues to evolve, we can expect even more exciting breakthroughs and innovations in the years to come.', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.schema import HumanMessage\n",
-    "\n",
-    "messages = [\n",
-    "    HumanMessage(content=\"Tell me about the history of AI\")\n",
-    "]\n",
-    "chat_model(messages)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## RAG\n",
-    "\n",
-    "We can use Olama with RAG, [just as shown here](https://python.langchain.com/docs/use_cases/question_answering/how_to/local_retrieval_qa).\n",
-    "\n",
-    "Let's use the 13b model:\n",
-    "\n",
-    "```\n",
-    "ollama pull llama2:13b\n",
-    "```\n",
-    "\n",
-    "Or, the 13b-chat model:\n",
-    "\n",
-    "```\n",
-    "ollama pull llama2:13b-chat\n",
-    "```\n",
-    "\n",
-    "Let's also use local embeddings from `GPT4AllEmbeddings` and `Chroma`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "! pip install gpt4all chromadb"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import WebBaseLoader\n",
-    "loader = WebBaseLoader(\"https://lilianweng.github.io/posts/2023-06-23-agent/\")\n",
-    "data = loader.load()\n",
-    "\n",
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
-    "all_splits = text_splitter.split_documents(data)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Found model file at  /Users/rlm/.cache/gpt4all/ggml-all-MiniLM-L6-v2-f16.bin\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.vectorstores import Chroma\n",
-    "from langchain.embeddings import GPT4AllEmbeddings\n",
-    "\n",
-    "vectorstore = Chroma.from_documents(documents=all_splits, embedding=GPT4AllEmbeddings())"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "4"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "question = \"What are the approaches to Task Decomposition?\"\n",
-    "docs = vectorstore.similarity_search(question)\n",
-    "len(docs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain import PromptTemplate\n",
-    "\n",
-    "# Prompt\n",
-    "template = \"\"\"[INST] <<SYS>> Use the following pieces of context to answer the question at the end. \n",
-    "If you don't know the answer, just say that you don't know, don't try to make up an answer. \n",
-    "Use three sentences maximum and keep the answer as concise as possible. <</SYS>>\n",
-    "{context}\n",
-    "Question: {question}\n",
-    "Helpful Answer:[/INST]\"\"\"\n",
-    "QA_CHAIN_PROMPT = PromptTemplate(\n",
-    "    input_variables=[\"context\", \"question\"],\n",
-    "    template=template,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Chat model\n",
-    "from langchain.chat_models import ChatOllama\n",
-    "from langchain.callbacks.manager import CallbackManager\n",
-    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
-    "chat_model = ChatOllama(model=\"llama2:13b-chat\",\n",
-    "                        verbose=True,\n",
-    "                        callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# QA chain\n",
-    "from langchain.chains import RetrievalQA\n",
-    "qa_chain = RetrievalQA.from_chain_type(\n",
-    "    chat_model,\n",
-    "    retriever=vectorstore.as_retriever(),\n",
-    "    chain_type_kwargs={\"prompt\": QA_CHAIN_PROMPT},\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      " Based on the provided context, there are three approaches to task decomposition for AI agents:\n",
-      "\n",
-      "1. LLM with simple prompting, such as \"Steps for XYZ.\" or \"What are the subgoals for achieving XYZ?\"\n",
-      "2. Task-specific instructions, such as \"Write a story outline\" for writing a novel.\n",
-      "3. Human inputs."
-     ]
-    }
-   ],
-   "source": [
-    "question = \"What are the various approaches to Task Decomposition for AI Agents?\"\n",
-    "result = qa_chain({\"query\": question})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "You can also get logging for tokens."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      " Based on the given context, here is the answer to the question \"What are the approaches to Task Decomposition?\"\n",
-      "\n",
-      "There are three approaches to task decomposition:\n",
-      "\n",
-      "1. LLM with simple prompting, such as \"Steps for XYZ.\" or \"What are the subgoals for achieving XYZ?\"\n",
-      "2. Using task-specific instructions, like \"Write a story outline\" for writing a novel.\n",
-      "3. With human inputs.{'model': 'llama2:13b-chat', 'created_at': '2023-08-23T15:37:51.469127Z', 'done': True, 'context': [1, 29871, 1, 29961, 25580, 29962, 518, 25580, 29962, 518, 25580, 29962, 3532, 14816, 29903, 6778, 4803, 278, 1494, 12785, 310, 3030, 304, 1234, 278, 1139, 472, 278, 1095, 29889, 29871, 13, 3644, 366, 1016, 29915, 29873, 1073, 278, 1234, 29892, 925, 1827, 393, 366, 1016, 29915, 29873, 1073, 29892, 1016, 29915, 29873, 1018, 304, 1207, 701, 385, 1234, 29889, 29871, 13, 11403, 2211, 25260, 7472, 322, 3013, 278, 1234, 408, 3022, 895, 408, 1950, 29889, 529, 829, 14816, 29903, 6778, 13, 5398, 26227, 508, 367, 2309, 313, 29896, 29897, 491, 365, 26369, 411, 2560, 9508, 292, 763, 376, 7789, 567, 363, 1060, 29979, 29999, 7790, 29876, 29896, 19602, 376, 5618, 526, 278, 1014, 1484, 1338, 363, 3657, 15387, 1060, 29979, 29999, 29973, 613, 313, 29906, 29897, 491, 773, 3414, 29899, 14940, 11994, 29936, 321, 29889, 29887, 29889, 376, 6113, 263, 5828, 27887, 1213, 363, 5007, 263, 9554, 29892, 470, 313, 29941, 29897, 411, 5199, 10970, 29889, 13, 13, 5398, 26227, 508, 367, 2309, 313, 29896, 29897, 491, 365, 26369, 411, 2560, 9508, 292, 763, 376, 7789, 567, 363, 1060, 29979, 29999, 7790, 29876, 29896, 19602, 376, 5618, 526, 278, 1014, 1484, 1338, 363, 3657, 15387, 1060, 29979, 29999, 29973, 613, 313, 29906, 29897, 491, 773, 3414, 29899, 14940, 11994, 29936, 321, 29889, 29887, 29889, 376, 6113, 263, 5828, 27887, 1213, 363, 5007, 263, 9554, 29892, 470, 313, 29941, 29897, 411, 5199, 10970, 29889, 13, 13, 1451, 16047, 267, 297, 1472, 29899, 8489, 18987, 322, 3414, 26227, 29901, 1858, 9450, 975, 263, 3309, 29891, 4955, 322, 17583, 3902, 8253, 278, 1650, 2913, 3933, 18066, 292, 29889, 365, 26369, 29879, 21117, 304, 10365, 13900, 746, 20050, 411, 15668, 4436, 29892, 3907, 963, 3109, 16424, 9401, 304, 25618, 1058, 5110, 515, 14260, 322, 1059, 29889, 13, 13, 1451, 16047, 267, 297, 1472, 29899, 8489, 18987, 322, 3414, 26227, 29901, 1858, 9450, 975, 263, 3309, 29891, 4955, 322, 17583, 3902, 8253, 278, 1650, 2913, 3933, 18066, 292, 29889, 365, 26369, 29879, 21117, 304, 10365, 13900, 746, 20050, 411, 15668, 4436, 29892, 3907, 963, 3109, 16424, 9401, 304, 25618, 1058, 5110, 515, 14260, 322, 1059, 29889, 13, 16492, 29901, 1724, 526, 278, 13501, 304, 9330, 897, 510, 3283, 29973, 13, 29648, 1319, 673, 10834, 29914, 25580, 29962, 518, 29914, 25580, 29962, 518, 29914, 25580, 29962, 29871, 16564, 373, 278, 2183, 3030, 29892, 1244, 338, 278, 1234, 304, 278, 1139, 376, 5618, 526, 278, 13501, 304, 9330, 897, 510, 3283, 3026, 13, 13, 8439, 526, 2211, 13501, 304, 3414, 26227, 29901, 13, 13, 29896, 29889, 365, 26369, 411, 2560, 9508, 292, 29892, 1316, 408, 376, 7789, 567, 363, 1060, 29979, 29999, 1213, 470, 376, 5618, 526, 278, 1014, 1484, 1338, 363, 3657, 15387, 1060, 29979, 29999, 3026, 13, 29906, 29889, 5293, 3414, 29899, 14940, 11994, 29892, 763, 376, 6113, 263, 5828, 27887, 29908, 363, 5007, 263, 9554, 29889, 13, 29941, 29889, 2973, 5199, 10970, 29889, 2], 'total_duration': 9514823750, 'load_duration': 795542, 'sample_count': 99, 'sample_duration': 68732000, 'prompt_eval_count': 146, 'prompt_eval_duration': 6206275000, 'eval_count': 98, 'eval_duration': 3229641000}\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.schema import LLMResult\n",
-    "from langchain.callbacks.base import BaseCallbackHandler\n",
-    "\n",
-    "class GenerationStatisticsCallback(BaseCallbackHandler):\n",
-    "    def on_llm_end(self, response: LLMResult, **kwargs) -> None:\n",
-    "        print(response.generations[0][0].generation_info)\n",
-    "        \n",
-    "callback_manager = CallbackManager([StreamingStdOutCallbackHandler(), GenerationStatisticsCallback()])\n",
-    "\n",
-    "chat_model = ChatOllama(model=\"llama2:13b-chat\",\n",
-    "                        verbose=True,\n",
-    "                        callback_manager=callback_manager)\n",
-    "\n",
-    "qa_chain = RetrievalQA.from_chain_type(\n",
-    "    chat_model,\n",
-    "    retriever=vectorstore.as_retriever(),\n",
-    "    chain_type_kwargs={\"prompt\": QA_CHAIN_PROMPT},\n",
-    ")\n",
-    "\n",
-    "question = \"What are the approaches to Task Decomposition?\"\n",
-    "result = qa_chain({\"query\": question})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "`eval_count` / (`eval_duration`/10e9)  gets `tok / s`"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "30.343929867127645"
-      ]
-     },
-     "execution_count": 17,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "98 / (3229641000/1000/1000/1000)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.16"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/chat/openai.ipynb

@@ -142,40 +142,13 @@
     ")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "57e27714",
-   "metadata": {},
-   "source": [
-    "## Fine-tuning\n",
-    "\n",
-    "You can call fine-tuned OpenAI models by passing in your corresponding `modelName` parameter.\n",
-    "\n",
-    "This generally takes the form of `ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}`. For example:"
-   ]
-  },
   {
    "cell_type": "code",
-   "execution_count": 6,
-   "id": "33c4a8b0",
+   "execution_count": null,
+   "id": "c095285d",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"J'adore la programmation.\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "fine_tuned_model = ChatOpenAI(temperature=0, model_name=\"ft:gpt-3.5-turbo-0613:langchain::7qTVM5AR\")\n",
-    "\n",
-    "fine_tuned_model(messages)"
-   ]
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -194,7 +167,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.9.7"
   }
  },
  "nbformat": 4,

docs/extras/integrations/chat_loaders/discord.ipynb

@@ -1,325 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "c4ff9336-1cf3-459e-bd70-d1314c1da6a0",
-   "metadata": {},
-   "source": [
-    "# Discord\n",
-    "\n",
-    "This notebook shows how to create your own chat loader that works on copy-pasted messages (from dms) to a list of LangChain messages.\n",
-    "\n",
-    "The process has four steps:\n",
-    "1. Create the chat .txt file by copying chats from the Discord app and pasting them in a file on your local computer\n",
-    "2. Copy the chat loader definition from below to a local file.\n",
-    "3. Initialize the `DiscordChatLoader` with the file path pointed to the text file.\n",
-    "4. Call `loader.load()` (or `loader.lazy_load()`) to perform the conversion.\n",
-    "\n",
-    "## 1. Creat message dump\n",
-    "\n",
-    "Currently (2023/08/23) this loader only supports .txt files in the format generated by copying messages in the app to your clipboard and pasting in a file. Below is an example."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "e4ccfdfa-6869-4d67-90a0-ab99f01b7553",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Overwriting discord_chats.txt\n"
-     ]
-    }
-   ],
-   "source": [
-    "%%writefile discord_chats.txt\n",
-    "talkingtower — 08/15/2023 11:10 AM\n",
-    "Love music! Do you like jazz?\n",
-    "reporterbob — 08/15/2023 9:27 PM\n",
-    "Yes! Jazz is fantastic. Ever heard this one?\n",
-    "Website\n",
-    "Listen to classic jazz track...\n",
-    "\n",
-    "talkingtower — Yesterday at 5:03 AM\n",
-    "Indeed! Great choice. 🎷\n",
-    "reporterbob — Yesterday at 5:23 AM\n",
-    "Thanks! How about some virtual sightseeing?\n",
-    "Website\n",
-    "Virtual tour of famous landmarks...\n",
-    "\n",
-    "talkingtower — Today at 2:38 PM\n",
-    "Sounds fun! Let's explore.\n",
-    "reporterbob — Today at 2:56 PM\n",
-    "Enjoy the tour! See you around.\n",
-    "talkingtower — Today at 3:00 PM\n",
-    "Thank you! Goodbye! 👋\n",
-    "reporterbob — Today at 3:02 PM\n",
-    "Farewell! Happy exploring."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "359565a7-dad3-403c-a73c-6414b1295127",
-   "metadata": {},
-   "source": [
-    "## 2. Define chat loader\n",
-    "\n",
-    "LangChain currently does not support "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "a429e0c4-4d7d-45f8-bbbb-c7fc5229f6af",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import logging\n",
-    "import re\n",
-    "from typing import Iterator, List\n",
-    "\n",
-    "from langchain import schema\n",
-    "from langchain.chat_loaders import base as chat_loaders\n",
-    "\n",
-    "logger = logging.getLogger()\n",
-    "\n",
-    "\n",
-    "class DiscordChatLoader(chat_loaders.BaseChatLoader):\n",
-    "    \n",
-    "    def __init__(self, path: str):\n",
-    "        \"\"\"\n",
-    "        Initialize the Discord chat loader.\n",
-    "\n",
-    "        Args:\n",
-    "            path: Path to the exported Discord chat text file.\n",
-    "        \"\"\"\n",
-    "        self.path = path\n",
-    "        self._message_line_regex = re.compile(\n",
-    "            r\"(.+?) — (\\w{3,9} \\d{1,2}(?:st|nd|rd|th)?(?:, \\d{4})? \\d{1,2}:\\d{2} (?:AM|PM)|Today at \\d{1,2}:\\d{2} (?:AM|PM)|Yesterday at \\d{1,2}:\\d{2} (?:AM|PM))\",  # noqa\n",
-    "            flags=re.DOTALL,\n",
-    "        )\n",
-    "\n",
-    "    def _load_single_chat_session_from_txt(\n",
-    "        self, file_path: str\n",
-    "    ) -> chat_loaders.ChatSession:\n",
-    "        \"\"\"\n",
-    "        Load a single chat session from a text file.\n",
-    "\n",
-    "        Args:\n",
-    "            file_path: Path to the text file containing the chat messages.\n",
-    "\n",
-    "        Returns:\n",
-    "            A `ChatSession` object containing the loaded chat messages.\n",
-    "        \"\"\"\n",
-    "        with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
-    "            lines = file.readlines()\n",
-    "\n",
-    "        results: List[schema.BaseMessage] = []\n",
-    "        current_sender = None\n",
-    "        current_timestamp = None\n",
-    "        current_content = []\n",
-    "        for line in lines:\n",
-    "            if re.match(\n",
-    "                r\".+? — (\\d{2}/\\d{2}/\\d{4} \\d{1,2}:\\d{2} (?:AM|PM)|Today at \\d{1,2}:\\d{2} (?:AM|PM)|Yesterday at \\d{1,2}:\\d{2} (?:AM|PM))\",  # noqa\n",
-    "                line,\n",
-    "            ):\n",
-    "                if current_sender and current_content:\n",
-    "                    results.append(\n",
-    "                        schema.HumanMessage(\n",
-    "                            content=\"\".join(current_content).strip(),\n",
-    "                            additional_kwargs={\n",
-    "                                \"sender\": current_sender,\n",
-    "                                \"events\": [{\"message_time\": current_timestamp}],\n",
-    "                            },\n",
-    "                        )\n",
-    "                    )\n",
-    "                current_sender, current_timestamp = line.split(\" — \")[:2]\n",
-    "                current_content = [\n",
-    "                    line[len(current_sender) + len(current_timestamp) + 4 :].strip()\n",
-    "                ]\n",
-    "            elif re.match(r\"\\[\\d{1,2}:\\d{2} (?:AM|PM)\\]\", line.strip()):\n",
-    "                results.append(\n",
-    "                    schema.HumanMessage(\n",
-    "                        content=\"\".join(current_content).strip(),\n",
-    "                        additional_kwargs={\n",
-    "                            \"sender\": current_sender,\n",
-    "                            \"events\": [{\"message_time\": current_timestamp}],\n",
-    "                        },\n",
-    "                    )\n",
-    "                )\n",
-    "                current_timestamp = line.strip()[1:-1]\n",
-    "                current_content = []\n",
-    "            else:\n",
-    "                current_content.append(\"\\n\" + line.strip())\n",
-    "\n",
-    "        if current_sender and current_content:\n",
-    "            results.append(\n",
-    "                schema.HumanMessage(\n",
-    "                    content=\"\".join(current_content).strip(),\n",
-    "                    additional_kwargs={\n",
-    "                        \"sender\": current_sender,\n",
-    "                        \"events\": [{\"message_time\": current_timestamp}],\n",
-    "                    },\n",
-    "                )\n",
-    "            )\n",
-    "\n",
-    "        return chat_loaders.ChatSession(messages=results)\n",
-    "\n",
-    "    def lazy_load(self) -> Iterator[chat_loaders.ChatSession]:\n",
-    "        \"\"\"\n",
-    "        Lazy load the messages from the chat file and yield them in the required format.\n",
-    "\n",
-    "        Yields:\n",
-    "            A `ChatSession` object containing the loaded chat messages.\n",
-    "        \"\"\"\n",
-    "        yield self._load_single_chat_session_from_txt(self.path)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c8240393-48be-44d2-b0d6-52c215cd8ac2",
-   "metadata": {},
-   "source": [
-    "## 2. Create loader\n",
-    "\n",
-    "We will point to the file we just wrote to disk."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "1268de40-b0e5-445d-9cd8-54856cd0293a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = DiscordChatLoader(\n",
-    "    path=\"./discord_chats.txt\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4928df4b-ae31-48a7-bd76-be3ecee1f3e0",
-   "metadata": {},
-   "source": [
-    "## 3. Load Messages\n",
-    "\n",
-    "Assuming the format is correct, the loader will convert the chats to langchain messages."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "c8a0836d-4a22-4790-bfe9-97f2145bb0d6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import List\n",
-    "from langchain.chat_loaders.base import ChatSession\n",
-    "from langchain.chat_loaders.utils import (\n",
-    "    map_ai_messages,\n",
-    "    merge_chat_runs,\n",
-    ")\n",
-    "\n",
-    "raw_messages = loader.lazy_load()\n",
-    "# Merge consecutive messages from the same sender into a single message\n",
-    "merged_messages = merge_chat_runs(raw_messages)\n",
-    "# Convert messages from \"talkingtower\" to AI messages\n",
-    "messages: List[ChatSession] = list(map_ai_messages(merged_messages, sender=\"talkingtower\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "1913963b-c44e-4f7a-aba7-0423c9b8bd59",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[{'messages': [AIMessage(content='Love music! Do you like jazz?', additional_kwargs={'sender': 'talkingtower', 'events': [{'message_time': '08/15/2023 11:10 AM\\n'}]}, example=False),\n",
-       "   HumanMessage(content='Yes! Jazz is fantastic. Ever heard this one?\\nWebsite\\nListen to classic jazz track...', additional_kwargs={'sender': 'reporterbob', 'events': [{'message_time': '08/15/2023 9:27 PM\\n'}]}, example=False),\n",
-       "   AIMessage(content='Indeed! Great choice. 🎷', additional_kwargs={'sender': 'talkingtower', 'events': [{'message_time': 'Yesterday at 5:03 AM\\n'}]}, example=False),\n",
-       "   HumanMessage(content='Thanks! How about some virtual sightseeing?\\nWebsite\\nVirtual tour of famous landmarks...', additional_kwargs={'sender': 'reporterbob', 'events': [{'message_time': 'Yesterday at 5:23 AM\\n'}]}, example=False),\n",
-       "   AIMessage(content=\"Sounds fun! Let's explore.\", additional_kwargs={'sender': 'talkingtower', 'events': [{'message_time': 'Today at 2:38 PM\\n'}]}, example=False),\n",
-       "   HumanMessage(content='Enjoy the tour! See you around.', additional_kwargs={'sender': 'reporterbob', 'events': [{'message_time': 'Today at 2:56 PM\\n'}]}, example=False),\n",
-       "   AIMessage(content='Thank you! Goodbye! 👋', additional_kwargs={'sender': 'talkingtower', 'events': [{'message_time': 'Today at 3:00 PM\\n'}]}, example=False),\n",
-       "   HumanMessage(content='Farewell! Happy exploring.', additional_kwargs={'sender': 'reporterbob', 'events': [{'message_time': 'Today at 3:02 PM\\n'}]}, example=False)]}]"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "messages"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8595a518-5c89-44aa-94a7-ca51e7e2a5fa",
-   "metadata": {},
-   "source": [
-    "### Next Steps\n",
-    "\n",
-    "You can then use these messages how you see fit, such as finetuning a model, few-shot example selection, or directly make predictions for the next message  "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "08ff0a1e-fca0-4da3-aacd-d7401f99d946",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Thank you! Have a wonderful day! 🌟"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "llm = ChatOpenAI()\n",
-    "\n",
-    "for chunk in llm.stream(messages[0]['messages']):\n",
-    "    print(chunk.content, end=\"\", flush=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "50a5251f-074a-4a3c-a2b0-b1de85e0ac6a",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/chat_loaders/facebook.ipynb

@@ -1,579 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "e4bd269b",
-   "metadata": {},
-   "source": [
-    "# Facebook Messenger\n",
-    "\n",
-    "This notebook shows how to load data from Facebook in a format you can finetune on. The overall steps are:\n",
-    "\n",
-    "1. Download your messenger data to disk.\n",
-    "2. Create the Chat Loader and call `loader.load()` (or `loader.lazy_load()`) to perform the conversion.\n",
-    "3. Optionally use `merge_chat_runs` to combine message from the same sender in sequence, and/or `map_ai_messages` to convert messages from the specified sender to the \"AIMessage\" class. Once you've done this, call `convert_messages_for_finetuning` to prepare your data for fine-tuning.\n",
-    "\n",
-    "\n",
-    "Once this has been done, you can fine-tune your model. To do so you would complete the following steps:\n",
-    "\n",
-    "4. Upload your messages to OpenAI and run a fine-tuning job.\n",
-    "6. Use the resulting model in your LangChain app!\n",
-    "\n",
-    "\n",
-    "Let's begin.\n",
-    "\n",
-    "\n",
-    "## 1. Download Data\n",
-    "\n",
-    "To download your own messenger data, following instructions [here](https://www.zapptales.com/en/download-facebook-messenger-chat-history-how-to/). IMPORTANT - make sure to download them in JSON format (not HTML).\n",
-    "\n",
-    "We are hosting an example dump at [this google drive link](https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing) that we will use in this walkthrough."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "647f2158-a42e-4634-b283-b8492caf542a",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "File file.zip downloaded.\n",
-      "File file.zip has been unzipped.\n"
-     ]
-    }
-   ],
-   "source": [
-    "# This uses some example data\n",
-    "import requests\n",
-    "import zipfile\n",
-    "\n",
-    "def download_and_unzip(url: str, output_path: str = 'file.zip') -> None:\n",
-    "    file_id = url.split('/')[-2]\n",
-    "    download_url = f'https://drive.google.com/uc?export=download&id={file_id}'\n",
-    "\n",
-    "    response = requests.get(download_url)\n",
-    "    if response.status_code != 200:\n",
-    "        print('Failed to download the file.')\n",
-    "        return\n",
-    "\n",
-    "    with open(output_path, 'wb') as file:\n",
-    "        file.write(response.content)\n",
-    "        print(f'File {output_path} downloaded.')\n",
-    "\n",
-    "    with zipfile.ZipFile(output_path, 'r') as zip_ref:\n",
-    "        zip_ref.extractall()\n",
-    "        print(f'File {output_path} has been unzipped.')\n",
-    "\n",
-    "# URL of the file to download\n",
-    "url = 'https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing'\n",
-    "\n",
-    "# Download and unzip\n",
-    "download_and_unzip(url)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "48ef8bb1-fc28-453c-835a-94a552f05a91",
-   "metadata": {},
-   "source": [
-    "## 2. Create Chat Loader\n",
-    "\n",
-    "We have 2 different `FacebookMessengerChatLoader` classes, one for an entire directory of chats, and one to load individual files. We"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "a0869bc6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "directory_path = \"./hogwarts\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "0460bf25",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_loaders.facebook_messenger import (\n",
-    "    SingleFileFacebookMessengerChatLoader,\n",
-    "    FolderFacebookMessengerChatLoader,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "f61ee277",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = SingleFileFacebookMessengerChatLoader(\n",
-    "    path=\"./hogwarts/inbox/HermioneGranger/messages_Hermione_Granger.json\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "ec466ad7",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[HumanMessage(content=\"Hi Hermione! How's your summer going so far?\", additional_kwargs={'sender': 'Harry Potter'}, example=False),\n",
-       " HumanMessage(content=\"Harry! Lovely to hear from you. My summer is going well, though I do miss everyone. I'm spending most of my time going through my books and researching fascinating new topics. How about you?\", additional_kwargs={'sender': 'Hermione Granger'}, example=False),\n",
-       " HumanMessage(content=\"I miss you all too. The Dursleys are being their usual unpleasant selves but I'm getting by. At least I can practice some spells in my room without them knowing. Let me know if you find anything good in your researching!\", additional_kwargs={'sender': 'Harry Potter'}, example=False)]"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat_session = loader.load()[0]\n",
-    "chat_session[\"messages\"][:3]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "8a3ee473",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = FolderFacebookMessengerChatLoader(\n",
-    "    path=\"./hogwarts\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "9f41e122",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "9"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat_sessions = loader.load()\n",
-    "len(chat_sessions)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d4aa3580-adc1-4b48-9bba-0e8e8d9f44ce",
-   "metadata": {},
-   "source": [
-    "## 3. Prepare for fine-tuning\n",
-    "\n",
-    "Calling `load()` returns all the chat messages we could extract as human messages. When conversing with chat bots, conversations typically follow a more strict alternating dialogue pattern relative to real conversations. \n",
-    "\n",
-    "You can choose to merge message \"runs\" (consecutive messages from the same sender) and select a sender to represent the \"AI\". The fine-tuned LLM will learn to generate these AI messages."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "5a78030d-b757-4bbe-8a6c-841056f46df7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_loaders.utils import (\n",
-    "    merge_chat_runs,\n",
-    "    map_ai_messages,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "ff35b028-78bf-4c5b-9ec6-939fe67de7f7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "merged_sessions = merge_chat_runs(chat_sessions)\n",
-    "alternating_sessions = list(map_ai_messages(merged_sessions, \"Harry Potter\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "4b11906e-a496-4d01-9f0d-1938c14147bf",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Professor Snape, I was hoping I could speak with you for a moment about something that's been concerning me lately.\", additional_kwargs={'sender': 'Harry Potter'}, example=False),\n",
-       " HumanMessage(content=\"What is it, Potter? I'm quite busy at the moment.\", additional_kwargs={'sender': 'Severus Snape'}, example=False),\n",
-       " AIMessage(content=\"I apologize for the interruption, sir. I'll be brief. I've noticed some strange activity around the school grounds at night. I saw a cloaked figure lurking near the Forbidden Forest last night. I'm worried someone may be plotting something sinister.\", additional_kwargs={'sender': 'Harry Potter'}, example=False)]"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Now all of Harry Potter's messages will take the AI message class\n",
-    "# which maps to the 'assistant' role in OpenAI's training format\n",
-    "alternating_sessions[0]['messages'][:3]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d985478d-062e-47b9-ae9a-102f59be07c0",
-   "metadata": {},
-   "source": [
-    "#### Now we can convert to OpenAI format dictionaries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "21372331",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.adapters.openai import convert_messages_for_finetuning"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 38,
-   "id": "92c5ae7a",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Prepared 9 dialogues for training\n"
-     ]
-    }
-   ],
-   "source": [
-    "training_data = convert_messages_for_finetuning(alternating_sessions)\n",
-    "print(f\"Prepared {len(training_data)} dialogues for training\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 33,
-   "id": "dfcbd181",
-   "metadata": {
-    "scrolled": true
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[{'role': 'assistant',\n",
-       "  'content': \"Professor Snape, I was hoping I could speak with you for a moment about something that's been concerning me lately.\"},\n",
-       " {'role': 'user',\n",
-       "  'content': \"What is it, Potter? I'm quite busy at the moment.\"},\n",
-       " {'role': 'assistant',\n",
-       "  'content': \"I apologize for the interruption, sir. I'll be brief. I've noticed some strange activity around the school grounds at night. I saw a cloaked figure lurking near the Forbidden Forest last night. I'm worried someone may be plotting something sinister.\"}]"
-      ]
-     },
-     "execution_count": 33,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "training_data[0][:3]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f1a9fd64-4f9f-42d3-b5dc-2a340e51e9e7",
-   "metadata": {},
-   "source": [
-    "OpenAI currently requires at least 10 training examples for a fine-tuning job, though they recommend between 50-100 for most tasks. Since we only have 9 chat sessions, we can subdivide them (optionally with some overlap) so that each training example is comprised of a portion of a whole conversation.\n",
-    "\n",
-    "Facebook chat sessions (1 per person) often span multiple days and conversations,\n",
-    "so the long-range dependencies may not be that important to model anyhow."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "13cd290a-b1e9-4686-bb5e-d99de8b8612b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "100"
-      ]
-     },
-     "execution_count": 42,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Our chat is alternating, we will make each datapoint a group of 8 messages,\n",
-    "# with 2 messages overlapping\n",
-    "chunk_size = 8\n",
-    "overlap = 2\n",
-    "\n",
-    "training_examples = [\n",
-    "    conversation_messages[i: i + chunk_size] \n",
-    "    for conversation_messages in training_data\n",
-    "    for i in range(\n",
-    "        0, len(conversation_messages) - chunk_size + 1, \n",
-    "        chunk_size - overlap)\n",
-    "]\n",
-    "\n",
-    "len(training_examples)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cc8baf41-ff07-4492-96bd-b2472ee7cef9",
-   "metadata": {},
-   "source": [
-    "## 4. Fine-tune the model\n",
-    "\n",
-    "It's time to fine-tune the model. Make sure you have `openai` installed\n",
-    "and have set your `OPENAI_API_KEY` appropriately"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 43,
-   "id": "95ce3f63-3c80-44b2-9060-534ad74e16fa",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# %pip install -U openai --quiet"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 58,
-   "id": "ab9e28eb",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "File file-zCyNBeg4snpbBL7VkvsuhCz8 ready afer 30.55 seconds.\n"
-     ]
-    }
-   ],
-   "source": [
-    "import json\n",
-    "from io import BytesIO\n",
-    "import time\n",
-    "\n",
-    "import openai\n",
-    "\n",
-    "# We will write the jsonl file in memory\n",
-    "my_file = BytesIO()\n",
-    "for m in training_examples:\n",
-    "    my_file.write((json.dumps({\"messages\": m}) + \"\\n\").encode('utf-8'))\n",
-    "\n",
-    "my_file.seek(0)\n",
-    "training_file = openai.File.create(\n",
-    "  file=my_file,\n",
-    "  purpose='fine-tune'\n",
-    ")\n",
-    "\n",
-    "# OpenAI audits each training file for compliance reasons.\n",
-    "# This make take a few minutes\n",
-    "status = openai.File.retrieve(training_file.id).status\n",
-    "start_time = time.time()\n",
-    "while status != \"processed\":\n",
-    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
-    "    time.sleep(5)\n",
-    "    status = openai.File.retrieve(training_file.id).status\n",
-    "print(f\"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "759a7f51-fde9-4b75-aaa9-e600e6537bd1",
-   "metadata": {},
-   "source": [
-    "With the file ready, it's time to kick off a training job."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 59,
-   "id": "3f451425",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "job = openai.FineTuningJob.create(\n",
-    "    training_file=training_file.id,\n",
-    "    model=\"gpt-3.5-turbo\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "489b23ef-5e14-42a9-bafb-44220ec6960b",
-   "metadata": {},
-   "source": [
-    "Grab a cup of tea while your model is being prepared. This may take some time!"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 60,
-   "id": "bac1637a-c087-4523-ade1-c47f9bf4c6f4",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Status=[running]... 908.87s\r"
-     ]
-    }
-   ],
-   "source": [
-    "status = openai.FineTuningJob.retrieve(job.id).status\n",
-    "start_time = time.time()\n",
-    "while status != \"succeeded\":\n",
-    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
-    "    time.sleep(5)\n",
-    "    job = openai.FineTuningJob.retrieve(job.id)\n",
-    "    status = job.status"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 66,
-   "id": "535895e1-bc69-40e5-82ed-e24ed2baeeee",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "ft:gpt-3.5-turbo-0613:personal::7rDwkaOq\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(job.fine_tuned_model)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "502ff73b-f9e9-49ce-ba45-401811e57946",
-   "metadata": {},
-   "source": [
-    "## 5. Use in LangChain\n",
-    "\n",
-    "You can use the resulting model ID directly the `ChatOpenAI` model class."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 67,
-   "id": "3925d60d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "model = ChatOpenAI(\n",
-    "    model=job.fine_tuned_model,\n",
-    "    temperature=1,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 69,
-   "id": "7190cf2e-ab34-4ceb-bdad-45f24f069c29",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"human\", \"{input}\"),\n",
-    "    ]\n",
-    ")\n",
-    "\n",
-    "chain = prompt | model | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 72,
-   "id": "f02057e9-f914-40b1-9c9d-9432ff594b98",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "The usual - Potions, Transfiguration, Defense Against the Dark Arts. What about you?"
-     ]
-    }
-   ],
-   "source": [
-    "for tok in chain.stream({\"input\": \"What classes are you taking?\"}):\n",
-    "    print(tok, end=\"\", flush=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "35331503-3cc6-4d64-955e-64afe6b5fef3",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/chat_loaders/gmail.ipynb

@@ -1,179 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "b3d1705d",
-   "metadata": {},
-   "source": [
-    "# GMail\n",
-    "\n",
-    "This loader goes over how to load data from GMail. There are many ways you could want to load data from GMail. This loader is currently fairly opionated in how to do so. The way it does it is it first looks for all messages that you have sent. It then looks for messages where you are responding to a previous email. It then fetches that previous email, and creates a training example of that email, followed by your email.\n",
-    "\n",
-    "Note that there are clear limitations here. For example, all examples created are only looking at the previous email for context.\n",
-    "\n",
-    "To use:\n",
-    "\n",
-    "- Set up a Google Developer Account: Go to the Google Developer Console, create a project, and enable the Gmail API for that project. This will give you a credentials.json file that you'll need later.\n",
-    "\n",
-    "- Install the Google Client Library: Run the following command to install the Google Client Library:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "84578039",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install --upgrade google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "be18f796",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os.path\n",
-    "import base64\n",
-    "import json\n",
-    "import re\n",
-    "import time\n",
-    "from google.auth.transport.requests import Request\n",
-    "from google.oauth2.credentials import Credentials\n",
-    "from google_auth_oauthlib.flow import InstalledAppFlow\n",
-    "from googleapiclient.discovery import build\n",
-    "import logging\n",
-    "import requests\n",
-    "\n",
-    "SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']\n",
-    "\n",
-    "\n",
-    "creds = None\n",
-    "# The file token.json stores the user's access and refresh tokens, and is\n",
-    "# created automatically when the authorization flow completes for the first\n",
-    "# time.\n",
-    "if os.path.exists('email_token.json'):\n",
-    "    creds = Credentials.from_authorized_user_file('email_token.json', SCOPES)\n",
-    "# If there are no (valid) credentials available, let the user log in.\n",
-    "if not creds or not creds.valid:\n",
-    "    if creds and creds.expired and creds.refresh_token:\n",
-    "        creds.refresh(Request())\n",
-    "    else:\n",
-    "        flow = InstalledAppFlow.from_client_secrets_file(               \n",
-    "            # your creds file here. Please create json file as here https://cloud.google.com/docs/authentication/getting-started\n",
-    "            'creds.json', SCOPES)\n",
-    "        creds = flow.run_local_server(port=0)\n",
-    "    # Save the credentials for the next run\n",
-    "    with open('email_token.json', 'w') as token:\n",
-    "        token.write(creds.to_json())"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "a2793ba0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_loaders.gmail import GMailLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "2154597f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = GMailLoader(creds=creds, n=3)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "0b7d11bd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "data = loader.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "74764bc7",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "2"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Sometimes there can be errors which we silently ignore\n",
-    "len(data)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "d9360a85",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_loaders.utils import (\n",
-    "    map_ai_messages,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "a9646f7a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# This makes messages sent by hchase@langchain.com the AI Messages\n",
-    "# This means you will train an LLM to predict as if it's responding as hchase\n",
-    "training_data = list(map_ai_messages(data, sender=\"Harrison Chase <hchase@langchain.com>\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d1a182f0",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/chat_loaders/imessage.ipynb

@@ -1,420 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "01fcfa2f-33a9-48f3-835a-b1956c394d6b",
-   "metadata": {},
-   "source": [
-    "# iMessage\n",
-    "\n",
-    "This notebook shows how to use the iMessage chat loader. This class helps convert iMessage conversations to LangChain chat messages.\n",
-    "\n",
-    "On MacOS, iMessage stores conversations in a sqlite database at `~/Library/Messages/chat.db` (at least for macOS Ventura 13.4). \n",
-    "The `IMessageChatLoader` loads from this database file. \n",
-    "\n",
-    "1. Create the `IMessageChatLoader` with the file path pointed to `chat.db` database you'd like to process.\n",
-    "2. Call `loader.load()` (or `loader.lazy_load()`) to perform the conversion. Optionally use `merge_chat_runs` to combine message from the same sender in sequence, and/or `map_ai_messages` to convert messages from the specified sender to the \"AIMessage\" class.\n",
-    "\n",
-    "## 1. Access Chat DB\n",
-    "\n",
-    "It's likely that your terminal is denied access to `~/Library/Messages`. To use this class, you can copy the DB to an accessible directory (e.g., Documents) and load from there. Alternatively (and not recommended), you can grant full disk access for your terminal emulator in System Settings > Securityand Privacy > Full Disk Access.\n",
-    "\n",
-    "We have created an example database you can use at [this linked drive file](https://drive.google.com/file/d/1NebNKqTA2NXApCmeH6mu0unJD2tANZzo/view?usp=sharing)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "036ce7e0-a38f-4cbe-89a6-a205ae7c23be",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "File chat.db downloaded.\n"
-     ]
-    }
-   ],
-   "source": [
-    "# This uses some example data\n",
-    "import requests\n",
-    "\n",
-    "def download_drive_file(url: str, output_path: str = 'chat.db') -> None:\n",
-    "    file_id = url.split('/')[-2]\n",
-    "    download_url = f'https://drive.google.com/uc?export=download&id={file_id}'\n",
-    "\n",
-    "    response = requests.get(download_url)\n",
-    "    if response.status_code != 200:\n",
-    "        print('Failed to download the file.')\n",
-    "        return\n",
-    "\n",
-    "    with open(output_path, 'wb') as file:\n",
-    "        file.write(response.content)\n",
-    "        print(f'File {output_path} downloaded.')\n",
-    "\n",
-    "url = 'https://drive.google.com/file/d/1NebNKqTA2NXApCmeH6mu0unJD2tANZzo/view?usp=sharing'\n",
-    "\n",
-    "# Download file to chat.db\n",
-    "download_drive_file(url)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cf60f703-76f1-4602-a723-02c59535c1af",
-   "metadata": {},
-   "source": [
-    "## 2. Create the Chat Loader\n",
-    "\n",
-    "Provide the loader with the file path to the zip directory. You can optionally specify the user id that maps to an ai message as well an configure whether to merge message runs."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "4b8b432a-d2bc-49e1-b35f-761730a8fd6d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_loaders.imessage import IMessageChatLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "8ec6661b-0aca-48ae-9e2b-6412856c287b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = IMessageChatLoader(\n",
-    "    path=\"./chat.db\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8805a7c5-84b4-49f5-8989-0022f2054ace",
-   "metadata": {},
-   "source": [
-    "## 3. Load messages\n",
-    "\n",
-    "The `load()` (or `lazy_load`) methods return a list of \"ChatSessions\" that currently just contain a list of messages per loaded conversation. All messages are mapped to \"HumanMessage\" objects to start. \n",
-    "\n",
-    "You can optionally choose to merge message \"runs\" (consecutive messages from the same sender) and select a sender to represent the \"AI\". The fine-tuned LLM will learn to generate these AI messages."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "fcd69b3e-020d-4a15-8a0d-61c2d34e1ee1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import List\n",
-    "from langchain.chat_loaders.base import ChatSession\n",
-    "from langchain.chat_loaders.utils import (\n",
-    "    map_ai_messages,\n",
-    "    merge_chat_runs,\n",
-    ")\n",
-    "\n",
-    "raw_messages = loader.lazy_load()\n",
-    "# Merge consecutive messages from the same sender into a single message\n",
-    "merged_messages = merge_chat_runs(raw_messages)\n",
-    "# Convert messages from \"Tortoise\" to AI messages. Do you have a guess who these conversations are between?\n",
-    "chat_sessions: List[ChatSession] = list(map_ai_messages(merged_messages, sender=\"Tortoise\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "370b8c26-c7a8-434c-a225-45c20ff14a03",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Slow and steady, that's my motto.\", additional_kwargs={'message_time': 1693182723, 'sender': 'Tortoise'}, example=False),\n",
-       " HumanMessage(content='Speed is key!', additional_kwargs={'message_time': 1693182753, 'sender': 'Hare'}, example=False),\n",
-       " AIMessage(content='A balanced approach is more reliable.', additional_kwargs={'message_time': 1693182783, 'sender': 'Tortoise'}, example=False)]"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Now all of the Tortoise's messages will take the AI message class\n",
-    "# which maps to the 'assistant' role in OpenAI's training format\n",
-    "alternating_sessions[0]['messages'][:3]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "05208f9d-3193-4a8d-86a5-13df2c8197e5",
-   "metadata": {},
-   "source": [
-    "## 3. Prepare for fine-tuning\n",
-    "\n",
-    "Now it's time to convert our chat  messages to OpenAI dictionaries. We can use the `convert_messages_for_finetuning` utility to do so."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "8834861f-f37f-4c08-96c6-917269bf09b8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.adapters.openai import convert_messages_for_finetuning"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "ce7ab0f9-6e6a-4a1c-8b86-c635251d437e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Prepared 10 dialogues for training\n"
-     ]
-    }
-   ],
-   "source": [
-    "training_data = convert_messages_for_finetuning(alternating_sessions)\n",
-    "print(f\"Prepared {len(training_data)} dialogues for training\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b494d64c-8056-42ae-b4c1-a9cfabc002ea",
-   "metadata": {},
-   "source": [
-    "## 4. Fine-tune the model\n",
-    "\n",
-    "It's time to fine-tune the model. Make sure you have `openai` installed\n",
-    "and have set your `OPENAI_API_KEY` appropriately"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "b4b60daa-b899-4291-a09a-412ce9c218fc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# %pip install -U openai --quiet"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "2cca6c95-c0d6-4826-b4fa-1c403f217f93",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "File file-zHIgf4r8LltZG3RFpkGd4Sjf ready after 10.19 seconds.\n"
-     ]
-    }
-   ],
-   "source": [
-    "import json\n",
-    "from io import BytesIO\n",
-    "import time\n",
-    "\n",
-    "import openai\n",
-    "\n",
-    "# We will write the jsonl file in memory\n",
-    "my_file = BytesIO()\n",
-    "for m in training_data:\n",
-    "    my_file.write((json.dumps({\"messages\": m}) + \"\\n\").encode('utf-8'))\n",
-    "\n",
-    "my_file.seek(0)\n",
-    "training_file = openai.File.create(\n",
-    "  file=my_file,\n",
-    "  purpose='fine-tune'\n",
-    ")\n",
-    "\n",
-    "# OpenAI audits each training file for compliance reasons.\n",
-    "# This make take a few minutes\n",
-    "status = openai.File.retrieve(training_file.id).status\n",
-    "start_time = time.time()\n",
-    "while status != \"processed\":\n",
-    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
-    "    time.sleep(5)\n",
-    "    status = openai.File.retrieve(training_file.id).status\n",
-    "print(f\"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "60ee0476-3113-4dc8-a886-bce878c60b07",
-   "metadata": {},
-   "source": [
-    "With the file ready, it's time to kick off a training job."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "c376ddca-5b4f-4e5a-bf4e-6beeb467eacc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "job = openai.FineTuningJob.create(\n",
-    "    training_file=training_file.id,\n",
-    "    model=\"gpt-3.5-turbo\",\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "09344c60-0bee-4989-b8d1-4a8821553cc3",
-   "metadata": {},
-   "source": [
-    "Grab a cup of tea while your model is being prepared. This may take some time!"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "22eae900-04ca-456b-ba51-1dfff1f8e0e1",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Status=[running]... 524.95s\r"
-     ]
-    }
-   ],
-   "source": [
-    "status = openai.FineTuningJob.retrieve(job.id).status\n",
-    "start_time = time.time()\n",
-    "while status != \"succeeded\":\n",
-    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
-    "    time.sleep(5)\n",
-    "    job = openai.FineTuningJob.retrieve(job.id)\n",
-    "    status = job.status"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "39e72616-a7d9-44b8-a4eb-506611d119f4",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "ft:gpt-3.5-turbo-0613:personal::7sKoRdlz\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(job.fine_tuned_model)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0d717749-b1b6-451f-b3c5-3286b82d45b9",
-   "metadata": {},
-   "source": [
-    "## 5. Use in LangChain\n",
-    "\n",
-    "You can use the resulting model ID directly the `ChatOpenAI` model class."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 22,
-   "id": "1579dfca-95c6-47b7-8549-1195b9dce5b0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "model = ChatOpenAI(\n",
-    "    model=job.fine_tuned_model,\n",
-    "    temperature=1,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 39,
-   "id": "6f53d1b1-dcbf-4976-a61a-17f74c6f1b0a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"system\", \"You are speaking to hare.\"),\n",
-    "        (\"human\", \"{input}\"),\n",
-    "    ]\n",
-    ")\n",
-    "\n",
-    "chain = prompt | model | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 41,
-   "id": "6619c9bc-54ea-4136-bd9a-44557f7da724",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "A symbol of interconnectedness."
-     ]
-    }
-   ],
-   "source": [
-    "for tok in chain.stream({\"input\": \"What's the golden thread?\"}):\n",
-    "    print(tok, end=\"\", flush=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "88e0d1a1-48a9-4d9d-9f4e-010cbbb65af8",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/chat_loaders/index.mdx

@@ -1,188 +0,0 @@
----
-sidebar_position: 0
----
-
-# Chat loaders
-
-Like document loaders, chat loaders are utilities designed to help load conversations from popular communication platforms such as Facebook, Slack, Discord, etc. These are loaded into memory as LangChain chat message objects. Such utilities facilitate tasks such as fine-tuning a language model to match your personal style or voice. 
-
-This brief guide will illustrate the process using [OpenAI's fine-tuning API](https://platform.openai.com/docs/guides/fine-tuning) comprised of six steps:
-
-1. Export your Facebook Messenger chat data in a compatible format for your intended chat loader.
-2. Load the chat data into memory as LangChain chat message objects. (_this is what is covered in each integration notebook in this section of the documentation_).
-    - Assign a person to the "AI" role and optionally filter, group, and merge messages.
-3. Export these acquired messages in a format expected by the fine-tuning API.
-4. Upload this data to OpenAI.
-5. Fine-tune your model.
-6. Implement the fine-tuned model in LangChain.
-
-This guide is not wholly comprehensive but is designed to take you through the fundamentals of going from raw data to fine-tuned model.
-
-We will demonstrate the procedure through an example of fine-tuning a `gpt-3.5-turbo` model on Facebook Messenger data. 
-
-### 1. Export your chat data
-
-To export your Facebook messenger data, you can follow the [instructions here](https://www.zapptales.com/en/download-facebook-messenger-chat-history-how-to/). 
-
-:::important JSON format
-You must select "JSON format" (instead of HTML) when exporting your data to be compatible with the current loader.
-:::
-
-OpenAI requires at least 10 examples to fine-tune your model, but they recommend between 50-100 for more optimal results.
-You can use the example data stored at [this google drive link](https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing) to test the process.
-
-### 2. Load the chat
-
-Once you've obtained your chat data, you can load it into memory as LangChain chat message objects. Here’s an example of loading data using the Python code:
-
-```python
-from langchain.chat_loaders.facebook_messenger import FolderFacebookMessengerChatLoader
-
-loader = FolderFacebookMessengerChatLoader(
-    path="./facebook_messenger_chats",
-)
-
-chat_sessions = loader.load()
-```
-
-In this snippet, we point the loader to a directory of Facebook chat dumps which are then loaded as multiple "sessions" of messages, one session per conversation file.
-
-Once you've loaded the messages, you should decide which person you want to fine-tune the model to (usually yourself). You can also decide to merge consecutive messages from the same sender into a single chat message.
-For both of these tasks, you can use the chat_loaders utilities to do so:
-
-```
-from langchain.chat_loaders.utils import (
-    merge_chat_runs,
-    map_ai_messages,
-)
-
-merged_sessions = merge_chat_runs(chat_sessions)
-alternating_sessions = list(map_ai_messages(merged_sessions, "My Name"))
-```
-
-### 3. Export messages to OpenAI format
-
-Convert the chat messages to dictionaries using the `convert_messages_for_finetuning` function. Then, group the data into chunks for better context modeling and overlap management.
-
-```python
-from langchain.adapters.openai import convert_messages_for_finetuning
-
-openai_messages = convert_messages_for_finetuning(chat_sessions)
-```
-
-At this point, the data is ready for upload to OpenAI. You can choose to split up conversations into smaller chunks for training if you
-do not have enough conversations to train on. Feel free to play around with different chunk sizes or with adding system messages to the fine-tuning data.
-
-```python
-chunk_size = 8
-overlap = 2
-
-message_groups = [
-    conversation_messages[i: i + chunk_size] 
-    for conversation_messages in openai_messages
-    for i in range(
-        0, len(conversation_messages) - chunk_size + 1, 
-        chunk_size - overlap)
-]
-
-len(message_groups)
-# 9
-```
-
-### 4. Upload the data to OpenAI
-
-Ensure you have set your OpenAI API key by following these [instructions](https://platform.openai.com/account/api-keys), then upload the training file.
-An audit is performed to ensure data compliance, so you may have to wait a few minutes for the dataset to become ready for use.
-
-```python
-import time
-import json
-import io
-
-import openai
-
-my_file = io.BytesIO()
-for group in message_groups:
-    my_file.write((json.dumps({"messages": group}) + "\n").encode('utf-8'))
-
-my_file.seek(0)
-training_file = openai.File.create(
-  file=my_file,
-  purpose='fine-tune'
-)
-
-# Wait while the file is processed
-status = openai.File.retrieve(training_file.id).status
-start_time = time.time()
-while status != "processed":
-    print(f"Status=[{status}]... {time.time() - start_time:.2f}s", end="\r", flush=True)
-    time.sleep(5)
-    status = openai.File.retrieve(training_file.id).status
-print(f"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.")
-```
-
-Once this is done, you can proceed to the model training!
-
-### 5. Fine-tune the model
-
-Start the fine-tuning job with your chosen base model.
-
-```python
-job = openai.FineTuningJob.create(
-    training_file=training_file.id,
-    model="gpt-3.5-turbo",
-)
-```
-
-This might take a while. Check the status with `openai.FineTuningJob.retrieve(job.id).status` and wait for it to report `succeeded`.
-
-```python
-# It may take 10-20+ minutes to complete training.
-status = openai.FineTuningJob.retrieve(job.id).status
-start_time = time.time()
-while status != "succeeded":
-    print(f"Status=[{status}]... {time.time() - start_time:.2f}s", end="\r", flush=True)
-    time.sleep(5)
-    job = openai.FineTuningJob.retrieve(job.id)
-    status = job.status
-```
-
-### 6. Use the model in LangChain
-
-You're almost there! Use the fine-tuned model in LangChain.
-
-```python
-from langchain import chat_models
-
-model_name = job.fine_tuned_model
-# Example: ft:gpt-3.5-turbo-0613:personal::5mty86jblapsed
-model = chat_models.ChatOpenAI(model=model_name)
-```
-
-```python
-from langchain.prompts import ChatPromptTemplate
-from langchain.schema.output_parser import StrOutputParser 
-
-prompt = ChatPromptTemplate.from_messages(
-    [
-        ("human", "{input}"),
-    ]
-)
-
-chain = prompt | model | StrOutputParser()
-
-for tok in chain.stream({"input": "What classes are you taking?"}):
-    print(tok, end="", flush=True)
-
-# The usual - Potions, Transfiguration, Defense Against the Dark Arts. What about you?
-```
-
-And that's it! You've successfully fine-tuned a model and used it in LangChain.
-
-## Supported Chat Loaders
-
-LangChain currently supports the following chat loaders. Feel free to contribute more!
-
-import DocCardList from "@theme/DocCardList";
-
-<DocCardList />