Updates

.github/CODE_OF_CONDUCT.md

@@ -1,132 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-We as members, contributors, and leaders pledge to make participation in our
-community a harassment-free experience for everyone, regardless of age, body
-size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
-nationality, personal appearance, race, caste, color, religion, or sexual
-identity and orientation.
-
-We pledge to act and interact in ways that contribute to an open, welcoming,
-diverse, inclusive, and healthy community.
-
-## Our Standards
-
-Examples of behavior that contributes to a positive environment for our
-community include:
-
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
-  and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the overall
-  community
-
-Examples of unacceptable behavior include:
-
-* The use of sexualized language or imagery, and sexual attention or advances of
-  any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email address,
-  without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-## Enforcement Responsibilities
-
-Community leaders are responsible for clarifying and enforcing our standards of
-acceptable behavior and will take appropriate and fair corrective action in
-response to any behavior that they deem inappropriate, threatening, offensive,
-or harmful.
-
-Community leaders have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions that are
-not aligned to this Code of Conduct, and will communicate reasons for moderation
-decisions when appropriate.
-
-## Scope
-
-This Code of Conduct applies within all community spaces, and also applies when
-an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
-posting via an official social media account, or acting as an appointed
-representative at an online or offline event.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported to the community leaders responsible for enforcement at
-conduct@langchain.dev.
-All complaints will be reviewed and investigated promptly and fairly.
-
-All community leaders are obligated to respect the privacy and security of the
-reporter of any incident.
-
-## Enforcement Guidelines
-
-Community leaders will follow these Community Impact Guidelines in determining
-the consequences for any action they deem in violation of this Code of Conduct:
-
-### 1. Correction
-
-**Community Impact**: Use of inappropriate language or other behavior deemed
-unprofessional or unwelcome in the community.
-
-**Consequence**: A private, written warning from community leaders, providing
-clarity around the nature of the violation and an explanation of why the
-behavior was inappropriate. A public apology may be requested.
-
-### 2. Warning
-
-**Community Impact**: A violation through a single incident or series of
-actions.
-
-**Consequence**: A warning with consequences for continued behavior. No
-interaction with the people involved, including unsolicited interaction with
-those enforcing the Code of Conduct, for a specified period of time. This
-includes avoiding interactions in community spaces as well as external channels
-like social media. Violating these terms may lead to a temporary or permanent
-ban.
-
-### 3. Temporary Ban
-
-**Community Impact**: A serious violation of community standards, including
-sustained inappropriate behavior.
-
-**Consequence**: A temporary ban from any sort of interaction or public
-communication with the community for a specified period of time. No public or
-private interaction with the people involved, including unsolicited interaction
-with those enforcing the Code of Conduct, is allowed during this period.
-Violating these terms may lead to a permanent ban.
-
-### 4. Permanent Ban
-
-**Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior, harassment of an
-individual, or aggression toward or disparagement of classes of individuals.
-
-**Consequence**: A permanent ban from any sort of public interaction within the
-community.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 2.1, available at
-[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
-
-Community Impact Guidelines were inspired by
-[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
-
-For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
-[https://www.contributor-covenant.org/translations][translations].
-
-[homepage]: https://www.contributor-covenant.org
-[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
-[Mozilla CoC]: https://github.com/mozilla/diversity
-[FAQ]: https://www.contributor-covenant.org/faq
-[translations]: https://www.contributor-covenant.org/translations

.github/CONTRIBUTING.md

@@ -1,19 +1,20 @@
 # Contributing to LangChain
 
 Hi there! Thank you for even being interested in contributing to LangChain.
-As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes.
+As an open source project in a rapidly developing field, we are extremely open
+to contributions, whether they be in the form of new features, improved infra, better documentation, or bug fixes.
 
 ## 🗺️ Guidelines
 
 ### 👩‍💻 Contributing Code
 
-To contribute to this project, please follow the ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
+To contribute to this project, please follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
 Please do not try to push directly to this repo unless you are a maintainer.
 
 Please follow the checked-in pull request template when opening pull requests. Note related issues and tag relevant
 maintainers.
 
-Pull requests cannot land without passing the formatting, linting, and testing checks first. See [Testing](#testing) and
+Pull requests cannot land without passing the formatting, linting and testing checks first. See [Testing](#testing) and
 [Formatting and Linting](#formatting-and-linting) for how to run these checks locally.
 
 It's essential that we maintain great documentation and testing. If you:
@@ -26,14 +27,16 @@ It's essential that we maintain great documentation and testing. If you:
   - Add a demo notebook in `docs/modules`.
   - Add unit and integration tests.
 
-We are a small, progress-oriented team. If there's something you'd like to add or change, opening a pull request is the
+We're a small, building-oriented team. If there's something you'd like to add or change, opening a pull request is the
 best way to get our attention.
 
 ### 🚩GitHub Issues
 
-Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests.
+Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date
+with bugs, improvements, and feature requests.
 
-There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues.
+There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help
+organize issues.
 
 If you start working on an issue, please assign it to yourself.
 
@@ -56,12 +59,12 @@ we do not want these to get in the way of getting good code into the codebase.
 
 ## 🚀 Quick Start
 
-This quick start guide explains how to run the repository locally.
+This quick start describes running the repository locally.
 For a [development container](https://containers.dev/), see the [.devcontainer folder](https://github.com/langchain-ai/langchain/tree/master/.devcontainer).
 
 ### Dependency Management: Poetry and other env/dependency managers
 
-This project utilizes [Poetry](https://python-poetry.org/) v1.6.1+ as a dependency manager.
+This project uses [Poetry](https://python-poetry.org/) v1.6.1+ as a dependency manager.
 
 ❗Note: *Before installing Poetry*, if you use `Conda`, create and activate a new Conda env (e.g. `conda create -n langchain python=3.9`)
 
@@ -72,11 +75,11 @@ tell Poetry to use the virtualenv python environment (`poetry config virtualenvs
 
 ### Core vs. Experimental
 
-This repository contains two separate projects:
-- `langchain`: core langchain code, abstractions, and use cases.
-- `langchain.experimental`: see the [Experimental README](https://github.com/langchain-ai/langchain/tree/master/libs/experimental/README.md) for more information.
+There are two separate projects in this repository:
+- `langchain`: core langchain code, abstractions, and use cases
+- `langchain.experimental`: see the [Experimental README](../libs/experimental/README.md) for more information.
 
-Each of these has its own development environment. Docs are run from the top-level makefile, but development
+Each of these has their own development environment. Docs are run from the top-level makefile, but development
 is split across separate test & release flows.
 
 For this quickstart, start with langchain core:
@@ -126,7 +129,7 @@ To run unit tests in Docker:
 make docker_tests
 ```
 
-There are also [integration tests and code-coverage](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests/README.md) available.
+There are also [integration tests and code-coverage](../libs/langchain/tests/README.md) available.
 
 ### Formatting and Linting
 
@@ -279,7 +282,7 @@ make docs_build
 make api_docs_build
 ```
 
-Finally, run the link checker to ensure all links are valid:
+Finally, you can run the linkchecker to make sure all links are valid:
 
 ```bash
 make docs_linkcheck
@@ -304,4 +307,4 @@ even patch releases may contain [non-backwards-compatible changes](https://semve
 ### 🌟 Recognition
 
 If your contribution has made its way into a release, we will want to give you credit on Twitter (only if you want though)!
-If you have a Twitter account you would like us to mention, please let us know in the PR or through another means.
+If you have a Twitter account you would like us to mention, please let us know in the PR or in another manner.

.github/workflows/_compile_integration_test.yml

@@ -1,57 +0,0 @@
-name: compile-integration-test
-
-on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-
-env:
-  POETRY_VERSION: "1.6.1"
-
-jobs:
-  build:
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: compile-integration
-
-      - name: Install integration dependencies
-        shell: bash
-        run: poetry install --with=test_integration
-
-      - name: Check integration tests compile
-        shell: bash
-        run: poetry run pytest -m compile tests/integration_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/_lint.yml

@@ -7,10 +7,6 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
-      langchain-location:
-        required: false
-        type: string
-        description: "Relative path to the langchain library folder"
 
 env:
   POETRY_VERSION: "1.6.1"
@@ -121,10 +117,8 @@ jobs:
       - name: Install langchain editable
         working-directory: ${{ inputs.working-directory }}
         if: ${{ inputs.working-directory != 'libs/langchain' }}
-        env:
-          LANGCHAIN_LOCATION: ${{ inputs.langchain-location || '../langchain'}}
         run: |
-          pip install -e "$LANGCHAIN_LOCATION"
+          pip install -e ../langchain
 
       - name: Restore black cache
         uses: actions/cache@v3

.github/workflows/_test.yml

@@ -44,6 +44,14 @@ jobs:
         shell: bash
         run: make test
 
+      - name: Install integration dependencies
+        shell: bash
+        run: poetry install --with=test_integration
+
+      - name: Check integration tests compile
+        shell: bash
+        run: poetry run pytest -m compile tests/integration_tests
+
       - name: Ensure the tests did not create any additional files
         shell: bash
         run: |

.github/workflows/_test_release.yml

@@ -1,50 +0,0 @@
-name: test-release
-
-on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-
-env:
-  POETRY_VERSION: "1.6.1"
-
-jobs:
-  publish_to_test_pypi:
-    runs-on: ubuntu-latest
-    permissions:
-      # This permission is used for trusted publishing:
-      # https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/
-      #
-      # Trusted publishing has to also be configured on PyPI for each package:
-      # https://docs.pypi.org/trusted-publishers/adding-a-publisher/
-      id-token: write
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: "3.10"
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: release
-
-      - name: Build project for distribution
-        run: poetry build
-      - name: Check Version
-        id: check-version
-        run: |
-          echo version=$(poetry version --short) >> $GITHUB_OUTPUT
-      - name: Publish package to TestPyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          repository-url: https://test.pypi.org/legacy/
-          packages-dir: ${{ inputs.working-directory }}/dist/
-          verbose: true
-          print-hash: true

.github/workflows/langchain_ci.yml

@@ -12,7 +12,6 @@ on:
       - '.github/workflows/_test.yml'
       - '.github/workflows/_pydantic_compatibility.yml'
       - '.github/workflows/langchain_ci.yml'
-      - 'libs/*'
       - 'libs/langchain/**'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
@@ -45,13 +44,6 @@ jobs:
       working-directory: libs/langchain
     secrets: inherit
 
-  compile-integration-tests:
-    uses:
-      ./.github/workflows/_compile_integration_test.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit
-
   pydantic-compatibility:
     uses:
       ./.github/workflows/_pydantic_compatibility.yml

.github/workflows/langchain_cli_ci.yml

@@ -1,53 +0,0 @@
----
-name: libs/cli CI
-
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
-      - '.github/workflows/_lint.yml'
-      - '.github/workflows/_test.yml'
-      - '.github/workflows/_pydantic_compatibility.yml'
-      - '.github/workflows/langchain_cli_ci.yml'
-      - 'libs/cli/**'
-      - 'libs/*'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-env:
-  POETRY_VERSION: "1.6.1"
-  WORKDIR: "libs/cli"
-
-jobs:
-  lint:
-    uses:
-      ./.github/workflows/_lint.yml
-    with:
-      working-directory: libs/cli
-    secrets: inherit
-
-  test:
-    uses:
-      ./.github/workflows/_test.yml
-    with:
-      working-directory: libs/cli
-    secrets: inherit
-
-  pydantic-compatibility:
-    uses:
-      ./.github/workflows/_pydantic_compatibility.yml
-    with:
-      working-directory: libs/cli
-    secrets: inherit

.github/workflows/langchain_experimental_ci.yml

@@ -11,7 +11,7 @@ on:
       - '.github/workflows/_lint.yml'
       - '.github/workflows/_test.yml'
       - '.github/workflows/langchain_experimental_ci.yml'
-      - 'libs/*'
+      - 'libs/langchain/**'
       - 'libs/experimental/**'
   workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
@@ -44,13 +44,6 @@ jobs:
       working-directory: libs/experimental
     secrets: inherit
 
-  compile-integration-tests:
-    uses:
-      ./.github/workflows/_compile_integration_test.yml
-    with:
-      working-directory: libs/experimental
-    secrets: inherit
-
   # It's possible that langchain-experimental works fine with the latest *published* langchain,
   # but is broken with the langchain on `master`.
   #

.github/workflows/langchain_experimental_test_release.yml

@@ -1,13 +0,0 @@
----
-name: Experimental Test Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_test_release.yml
-    with:
-      working-directory: libs/experimental
-    secrets: inherit

.github/workflows/langchain_test_release.yml

@@ -1,13 +0,0 @@
----
-name: Test Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_test_release.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit

.github/workflows/scheduled_test.yml

@@ -55,10 +55,6 @@ jobs:
           poetry install --with=test_integration
           poetry run pip install google-cloud-aiplatform
           poetry run pip install "boto3>=1.28.57"
-          if [[ ${{ matrix.python-version }} != "3.8" ]]
-          then
-            poetry run pip install fireworks-ai
-          fi
 
       - name: Run tests
         shell: bash
@@ -68,8 +64,7 @@ jobs:
           AZURE_OPENAI_API_VERSION: ${{ secrets.AZURE_OPENAI_API_VERSION }}
           AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }}
           AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
-          AZURE_OPENAI_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_NAME }}
-          FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
+          AZURE_OPENAI_DEPLOYMENT_NAME:  ${{ secrets.AZURE_OPENAI_DEPLOYMENT_NAME }}
         run: |
           make scheduled_tests
 

.github/workflows/templates_ci.yml

@@ -1,37 +0,0 @@
----
-name: templates CI
-
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
-      - '.github/workflows/_lint.yml'
-      - '.github/workflows/templates_ci.yml'
-      - 'templates/**'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-env:
-  POETRY_VERSION: "1.6.1"
-  WORKDIR: "templates"
-
-jobs:
-  lint:
-    uses:
-      ./.github/workflows/_lint.yml
-    with:
-      working-directory: templates
-      langchain-location: ../libs/langchain
-    secrets: inherit

cookbook/README.md

@@ -4,7 +4,7 @@ Example code for building applications with LangChain, with an emphasis on more
 
 Notebook | Description
 :- | :-
-[LLaMA2_sql_chat.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/LLaMA2_sql_chat.ipynb) | Build a chat application that interacts with a SQL database using an open source llm (llama2), specifically demonstrated on an SQLite database containing rosters.
+[LLaMA2_sql_chat.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/LLaMA2_sql_chat.ipynb) | Build a chat application that interacts with a sql database using an open source llm (llama2), specifically demonstrated on a sqlite database containing nba rosters.
 [Semi_Structured_RAG.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_Structured_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data, including text and tables, using unstructured for parsing, multi-vector retriever for storing, and lcel for implementing chains.
 [Semi_structured_and_multi_moda...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using unstructured for parsing, multi-vector retriever for storage and retrieval, and lcel for implementing chains.
 [Semi_structured_multi_modal_RA...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using various tools and methods such as unstructured for parsing, multi-vector retriever for storing, lcel for implementing chains, and open source language models like llama2, llava, and gpt4all.
@@ -12,14 +12,14 @@ Notebook | Description
 [autogpt/marathon_times.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/marathon_times.ipynb) | Implement autogpt for finding winning marathon times.
 [baby_agi.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi.ipynb) | Implement babyagi, an ai agent that can generate and execute tasks based on a given objective, with the flexibility to swap out specific vectorstores/model providers.
 [baby_agi_with_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi_with_agent.ipynb) | Swap out the execution chain in the babyagi notebook with an agent that has access to tools, aiming to obtain more reliable information.
-[camel_role_playing.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/camel_role_playing.ipynb) | Implement the camel framework for creating autonomous cooperative agents in large-scale language models, using role-playing and inception prompting to guide chat agents towards task completion.
+[camel_role_playing.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/camel_role_playing.ipynb) | Implement the camel framework for creating autonomous cooperative agents in large scale language models, using role-playing and inception prompting to guide chat agents towards task completion.
 [causal_program_aided_language_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/causal_program_aided_language_model.ipynb) | Implement the causal program-aided language (cpal) chain, which improves upon the program-aided language (pal) by incorporating causal structure to prevent hallucination in language models, particularly when dealing with complex narratives and math problems with nested dependencies.
 [code-analysis-deeplake.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/code-analysis-deeplake.ipynb) | Analyze its own code base with the help of gpt and activeloop's deep lake.
 [custom_agent_with_plugin_retri...](https://github.com/langchain-ai/langchain/tree/master/cookbook/custom_agent_with_plugin_retrieval.ipynb) | Build a custom agent that can interact with ai plugins by retrieving tools and creating natural language wrappers around openapi endpoints.
 [custom_agent_with_plugin_retri...](https://github.com/langchain-ai/langchain/tree/master/cookbook/custom_agent_with_plugin_retrieval_using_plugnplai.ipynb) | Build a custom agent with plugin retrieval functionality, utilizing ai plugins from the `plugnplai` directory.
 [databricks_sql_db.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/databricks_sql_db.ipynb) | Connect to databricks runtimes and databricks sql.
 [deeplake_semantic_search_over_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/deeplake_semantic_search_over_chat.ipynb) | Perform semantic search and question-answering over a group chat using activeloop's deep lake with gpt4.
-[elasticsearch_db_qa.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/elasticsearch_db_qa.ipynb) | Interact with elasticsearch analytics databases in natural language and build search queries via the elasticsearch dsl API.
+[elasticsearch_db_qa.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/elasticsearch_db_qa.ipynb) | Interact with elasticsearch analytics databases in natural language and build search queries via the elasticsearch dsl api.
 [forward_looking_retrieval_augm...](https://github.com/langchain-ai/langchain/tree/master/cookbook/forward_looking_retrieval_augmented_generation.ipynb) | Implement the forward-looking active retrieval augmented generation (flare) method, which generates answers to questions, identifies uncertain tokens, generates hypothetical questions based on these tokens, and retrieves relevant documents to continue generating the answer.
 [generative_agents_interactive_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/generative_agents_interactive_simulacra_of_human_behavior.ipynb) | Implement a generative agent that simulates human behavior, based on a research paper, using a time-weighted memory object backed by a langchain retriever.
 [gymnasium_agent_simulation.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/gymnasium_agent_simulation.ipynb) | Create a simple agent-environment interaction loop in simulated environments like text-based games with gymnasium.
@@ -37,7 +37,7 @@ Notebook | Description
 [multiagent_authoritarian.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multiagent_authoritarian.ipynb) | Implement a multi-agent simulation where a privileged agent controls the conversation, including deciding who speaks and when the conversation ends, in the context of a simulated news network.
 [multiagent_bidding.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multiagent_bidding.ipynb) | Implement a multi-agent simulation where agents bid to speak, with the highest bidder speaking next, demonstrated through a fictitious presidential debate example.
 [myscale_vector_sql.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/myscale_vector_sql.ipynb) | Access and interact with the myscale integrated vector database, which can enhance the performance of language model (llm) applications.
-[openai_functions_retrieval_qa....](https://github.com/langchain-ai/langchain/tree/master/cookbook/openai_functions_retrieval_qa.ipynb) | Structure response output in a question-answering system by incorporating openai functions into a retrieval pipeline.
+[openai_functions_retrieval_qa....](https://github.com/langchain-ai/langchain/tree/master/cookbook/openai_functions_retrieval_qa.ipynb) | Structure response output in a question answering system by incorporating openai functions into a retrieval pipeline.
 [petting_zoo.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/petting_zoo.ipynb) | Create multi-agent simulations with simulated environments using the petting zoo library.
 [plan_and_execute_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/plan_and_execute_agent.ipynb) | Create plan-and-execute agents that accomplish objectives by planning tasks with a language model (llm) and executing them with a separate agent.
 [press_releases.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/press_releases.ipynb) | Retrieve and query company press release data powered by [Kay.ai](https://kay.ai).
@@ -46,7 +46,7 @@ Notebook | Description
 [self_query_hotel_search.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/self_query_hotel_search.ipynb) | Build a hotel room search feature with self-querying retrieval, using a specific hotel recommendation dataset.
 [smart_llm.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/smart_llm.ipynb) | Implement a smartllmchain, a self-critique chain that generates multiple output proposals, critiques them to find the best one, and then improves upon it to produce a final output.
 [tree_of_thought.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/tree_of_thought.ipynb) | Query a large language model using the tree of thought technique.
-[twitter-the-algorithm-analysis...](https://github.com/langchain-ai/langchain/tree/master/cookbook/twitter-the-algorithm-analysis-deeplake.ipynb) | Analyze the source code of the Twitter algorithm with the help of gpt4 and activeloop's deep lake.
+[twitter-the-algorithm-analysis...](https://github.com/langchain-ai/langchain/tree/master/cookbook/twitter-the-algorithm-analysis-deeplake.ipynb) | Analyze the source code of the twitter algorithm with the help of gpt4 and activeloop's deep lake.
 [two_agent_debate_tools.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_agent_debate_tools.ipynb) | Simulate multi-agent dialogues where the agents can utilize various tools.
-[two_player_dnd.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_player_dnd.ipynb) | Simulate a two-player dungeons & dragons game, where a dialogue simulator class is used to coordinate the dialogue between the protagonist and the dungeon master.
+[two_player_dnd.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_player_dnd.ipynb) | Simulate a two-player dungeons & dragons game, where a dialoguesimulator class is used to coordinate the dialogue between the protagonist and the dungeon master.
 [wikibase_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/wikibase_agent.ipynb) | Create a simple wikibase agent that utilizes sparql generation, with testing done on http://wikidata.org.

cookbook/retrieval_in_sql.ipynb

@@ -0,0 +1,631 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Incoporating semantic similarity in tabular databases\n",
+    "\n",
+    "In this notebook we will cover how to run semantic search over a specific table column within a single SQL query, combining tabular query with RAG.\n",
+    "\n",
+    "\n",
+    "### Overall workflow\n",
+    "\n",
+    "1. Generating embeddings for a specific column\n",
+    "   \n",
+    "2. Storing the embeddings in a new column (if column has low cardinality, it's better to use another table containing unique values and their embeddings)\n",
+    "   \n",
+    "3. Querying using standard SQL queries with [PGVector](https://github.com/pgvector/pgvector) extension which allows using:\n",
+    "* L2 distance (`<->`)\n",
+    "* Cosine distance (`<=>` or cosine similarity using `1 - <=>`)\n",
+    "* Inner product (`<#>`)\n",
+    "   \n",
+    "4. Running standard SQL query\n",
+    "\n",
+    "### Requirements\n",
+    "\n",
+    "We will need a PostgreSQL database with [pgvector](https://github.com/pgvector/pgvector) extension enabled. \n",
+    "\n",
+    "For this example, we will use a `Chinook` database using a local PostgreSQL server."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import getpass\n",
+    "os.environ[\"OPENAI_API_KEY\"] = os.environ.get('OPENAI_API_KEY') or getpass.getpass(\"OpenAI API Key:\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.sql_database import SQLDatabase\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "CONNECTION_STRING = \"postgresql+psycopg2://postgres:test@localhost:5432/vectordb\" # Replace with your own\n",
+    "db = SQLDatabase.from_uri(CONNECTION_STRING)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Embedding the song titles"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For this example, we will run queries based on semantic meaning of song titles. \n",
+    "\n",
+    "In order to do this, let's start by adding a new column in the table for storing the embeddings:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# db.run('ALTER TABLE \"Track\" ADD COLUMN \"embeddings\" vector;')"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's generate the embedding for each *track title* and store it as a new column in our \"Track\" table"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "\n",
+    "embeddings_model = OpenAIEmbeddings()\n",
+    "\n",
+    "tracks = db.run('SELECT \"Name\" FROM \"Track\"')\n",
+    "song_titles = [s[0] for s in eval(tracks)]\n",
+    "title_embeddings = embeddings_model.embed_documents(song_titles)\n",
+    "len(title_embeddings)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's insert the embeddings in the into the new column from our table"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tqdm import tqdm\n",
+    "\n",
+    "for i in tqdm(range(len(title_embeddings))):\n",
+    "    title = titles[i].replace(\"'\",\"''\")\n",
+    "    embedding = title_embeddings[i]\n",
+    "    sql_command = f'UPDATE \"Track\" SET \"embeddings\" = ARRAY{embedding} WHERE \"Name\" =' +  f\"'{title}'\"\n",
+    "    db.run(sql_command)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can test the semantic search running the following query:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'[(\"Tomorrow\\'s Dream\",), (\\'Remember Tomorrow\\',), (\\'Remember Tomorrow\\',), (\\'The Best Is Yet To Come\\',), (\"Thinking \\'Bout Tomorrow\",)]'"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "embeded_title = embeddings_model.embed_query(\"hope about the future\")\n",
+    "query = 'SELECT \"Track\".\"Name\" FROM \"Track\" WHERE \"Track\".\"embeddings\" IS NOT NULL ORDER BY \"embeddings\" <-> ' +  f\"'{embeded_title}' LIMIT 5\"\n",
+    "db.run(query)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can see the the song titles are conceptually similar to our search term `\"hope about the future\"`."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Creating the SQL Chain"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's start by defining useful functions to get info from database and running the query:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_schema(_):\n",
+    "    return db.get_table_info()\n",
+    "\n",
+    "def run_query(query):\n",
+    "    return db.run(query)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's build the **prompt** we will use:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "\n",
+    "template = \"\"\"You are a Postgres expert. Given an input question, first create a syntactically correct Postgres query to run, then look at the results of the query and return the answer to the input question.\n",
+    "Unless the user specifies in the question a specific number of examples to obtain, query for at most 5 results using the LIMIT clause as per Postgres. You can order the results to return the most informative data in the database.\n",
+    "Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in double quotes (\") to denote them as delimited identifiers.\n",
+    "Pay attention to use only the column names you can see in the tables below. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.\n",
+    "Pay attention to use date('now') function to get the current date, if the question involves \"today\".\n",
+    "\n",
+    "You can use an extra extension which allows you to run semantic similarity using <-> operator on tables containing columns named \"embeddings\".\n",
+    "<-> operator can ONLY be used on embeddings columns.\n",
+    "The embeddings value for a given row typically represents the semantic meaning of that row.\n",
+    "The vector represents an embedding representation of the question, given below. \n",
+    "Do NOT fill in the vector values directly, but rather specify a `[search_word]` placeholder, which should contain the word that would be embedded for filtering.\n",
+    "For example, if the user asks for songs about 'the feeling of loneliness' the query could be:\n",
+    "'SELECT \"[whatever_table_name]\".\"SongName\" FROM \"[whatever_table_name]\" ORDER BY \"embeddings\" <-> '[loneliness]' LIMIT 5'\n",
+    "\n",
+    "Use the following format:\n",
+    "\n",
+    "Question: <Question here>\n",
+    "SQLQuery: <SQL Query to run>\n",
+    "SQLResult: <Result of the SQLQuery>\n",
+    "Answer: <Final answer here>\n",
+    "\n",
+    "Only use the following tables:\n",
+    "\n",
+    "{schema}\n",
+    "\n",
+    "QUESTION: {question}\n",
+    "SQLQuery:\n",
+    "\n",
+    "\"\"\"\n",
+    "prompt = ChatPromptTemplate.from_messages([\n",
+    "    (\"system\", \"Given an input question, convert it to a SQL query. No pre-amble.\"),\n",
+    "    (\"human\", template)\n",
+    "])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And we can create the chain using **[LangChain Expression Language](https://python.langchain.com/docs/expression_language/)**:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/manuelsoria/miniconda3/envs/auto-gpt/lib/python3.8/site-packages/langchain/utilities/sql_database.py:112: SAWarning: Did not recognize type 'vector' of column 'title_embedding'\n",
+      "  self._metadata.reflect(\n",
+      "/Users/manuelsoria/miniconda3/envs/auto-gpt/lib/python3.8/site-packages/langchain/utilities/sql_database.py:112: SAWarning: Did not recognize type 'vector' of column 'embeddings'\n",
+      "  self._metadata.reflect(\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema.output_parser import StrOutputParser\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
+    "\n",
+    "db = SQLDatabase.from_uri(CONNECTION_STRING) # We reconnect to db so the new columns are loaded as well.\n",
+    "llm = ChatOpenAI(model_name='gpt-4', temperature=0)\n",
+    "\n",
+    "sql_query_chain = (\n",
+    "        RunnablePassthrough.assign(schema=get_schema)\n",
+    "        | prompt\n",
+    "        | llm.bind(stop=[\"\\nSQLResult:\"])\n",
+    "        | StrOutputParser()\n",
+    "    )"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This chain simply generates the query. Now we will create the full chain that also handles the execution and the final result for the user:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "from langchain.schema.runnable import RunnableLambda\n",
+    "\n",
+    "# Inject the embedding for any words within brackets \n",
+    "def replace_brackets(match):\n",
+    "    words_inside_brackets = match.group(1).split(', ')\n",
+    "    embedded_words = [str(embeddings_model.embed_query(word)) for word in words_inside_brackets]\n",
+    "    return \"', '\".join(embedded_words)\n",
+    "\n",
+    "def get_query(query):\n",
+    "    sql_query = re.sub(r'\\[([\\w\\s,]+)\\]', replace_brackets, query)\n",
+    "    return sql_query\n",
+    "    \n",
+    "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n",
+    "{schema}\n",
+    "\n",
+    "Question: {question}\n",
+    "SQL Query: {query}\n",
+    "SQL Response: {response}\"\"\"\n",
+    "\n",
+    "prompt_response = ChatPromptTemplate.from_messages([\n",
+    "    (\"system\", \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\"),\n",
+    "    (\"human\", template)\n",
+    "])\n",
+    "\n",
+    "full_chain = (\n",
+    "    RunnablePassthrough.assign(query=sql_query_chain)\n",
+    "    | RunnablePassthrough.assign(\n",
+    "        schema=get_schema,\n",
+    "        response=RunnableLambda(lambda x: db.run(get_query(x[\"query\"]))),\n",
+    "        )\n",
+    "    | prompt_response \n",
+    "    | llm\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using the Chain"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example 1: Filtering a column based on semantic meaning"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's say we want to retrieve songs that express `deep feeling of dispair`, but filtering based on genre:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"The 5 rock songs with titles about deep feeling of despair are 'Sea Of Sorrow', 'Surrender', 'Indifference', 'Hard Luck Woman', and 'Desire'.\")"
+      ]
+     },
+     "execution_count": 27,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "full_chain.invoke({\"question\":\"Which are the 5 rock songs with titles about deep feeling of dispair?\"})"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "What is substantially different in implementing this method is that we have combined:\n",
+    "- Semantic search (songs that have titles with some semantic meaning)\n",
+    "- Traditional tabular querying (running JOIN statements to filter track based on genre)\n",
+    "\n",
+    "This is something we _could_ potentially achieve using metadata filtering, but it's more complex to do so (we would need to use a vector database containing the embeddings, and use metadata filtering based on genre).\n",
+    "\n",
+    "However, for other use cases metadata filtering **wouldn't be enough**."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example 2: Combining filters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"The three albums which have the most amount of songs in the top 150 saddest songs are 'International Superhits' with 5 songs, 'Ten' with 4 songs, and 'Album Of The Year' with 3 songs.\")"
+      ]
+     },
+     "execution_count": 29,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "full_chain.invoke({\"question\": \"I want to know the 3 albums which have the most amount of songs in the top 150 saddest songs\"})"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "So we have result for 3 albums with most amount of songs in top 150 saddest ones. This **wouldn't** be possible using only standard metadata filtering. Without this _hybdrid query_, we would need some postprocessing to get the result.\n",
+    "\n",
+    "Another similar exmaple:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"The 6 albums with the shortest titles that contain songs which are in the 20 saddest song list are 'Ten', 'Core', 'Big Ones', 'One By One', 'Black Album', and 'Miles Ahead'.\")"
+      ]
+     },
+     "execution_count": 30,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "full_chain.invoke({\"question\": \"I need the 6 albums with shortest title, as long as they contain songs which are in the 20 saddest song list.\"})"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's see what the query looks like to double check:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "WITH \"SadSongs\" AS (\n",
+      "    SELECT \"TrackId\" FROM \"Track\" \n",
+      "    ORDER BY \"embeddings\" <-> '[sad]' LIMIT 20\n",
+      "),\n",
+      "\"SadAlbums\" AS (\n",
+      "    SELECT DISTINCT \"AlbumId\" FROM \"Track\" \n",
+      "    WHERE \"TrackId\" IN (SELECT \"TrackId\" FROM \"SadSongs\")\n",
+      ")\n",
+      "SELECT \"Album\".\"Title\" FROM \"Album\" \n",
+      "WHERE \"AlbumId\" IN (SELECT \"AlbumId\" FROM \"SadAlbums\") \n",
+      "ORDER BY \"title_len\" ASC \n",
+      "LIMIT 6\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(sql_query_chain.invoke({\"question\": \"I need the 6 albums with shortest title, as long as they contain songs which are in the 20 saddest song list.\"}))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example 3: Combining two separate semantic searches\n",
+    "\n",
+    "One interesting aspect of this approach which is **substantially different from using standar RAG** is that we can even **combine** two semantic search filters:\n",
+    "- _Get 5 saddest songs..._\n",
+    "- _**...obtained from albums with \"lovely\" titles**_\n",
+    "\n",
+    "This could generalize to **any kind of combined RAG** (paragraphs discussing _X_ topic belonging from books about _Y_, replies to a tweet about _ABC_ topic that express _XYZ_ feeling)\n",
+    "\n",
+    "We will combine semantic search on songs and album titles, so we need to do the same for `Album` table:\n",
+    "1. Generate the embeddings\n",
+    "2. Add them to the table as a new column (which we need to add in the table)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 60,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# db.run('ALTER TABLE \"Album\" ADD COLUMN \"embeddings\" vector;')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "100%|██████████| 347/347 [00:01<00:00, 179.64it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "albums = db.run('SELECT \"Title\" FROM \"Album\"')\n",
+    "album_titles = [title[0] for title in eval(albums)]\n",
+    "album_title_embeddings = embeddings_model.embed_documents(album_titles)\n",
+    "for i in tqdm(range(len(album_title_embeddings))):\n",
+    "    album_title = album_titles[i].replace(\"'\",\"''\")\n",
+    "    album_embedding = album_title_embeddings[i]\n",
+    "    sql_command = f'UPDATE \"Album\" SET \"embeddings\" = ARRAY{album_embedding} WHERE \"Title\" =' +  f\"'{album_title}'\"\n",
+    "    db.run(sql_command)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"[('Realize',), ('Morning Dance',), ('Into The Light',), ('New Adventures In Hi-Fi',), ('Miles Ahead',)]\""
+      ]
+     },
+     "execution_count": 45,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "embeded_title = embeddings_model.embed_query(\"hope about the future\")\n",
+    "query = 'SELECT \"Album\".\"Title\" FROM \"Album\" WHERE \"Album\".\"embeddings\" IS NOT NULL ORDER BY \"embeddings\" <-> ' +  f\"'{embeded_title}' LIMIT 5\"\n",
+    "db.run(query)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we can combine both filters:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 54,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db = SQLDatabase.from_uri(CONNECTION_STRING) # We reconnect to dbso the new columns are loaded as well."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 49,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content='The songs about breakouts obtained from the top 5 albums about love are \\'Royal Orleans\\', \"Nobody\\'s Fault But Mine\", \\'Achilles Last Stand\\', \\'For Your Life\\', and \\'Hots On For Nowhere\\'.')"
+      ]
+     },
+     "execution_count": 49,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "full_chain.invoke({\"question\": \"I want to know songs about breakouts obtained from top 5 albums about love\"})"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is something **different** that **couldn't be achieved** using standard metadata filtering over a vectordb."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

cookbook/stepback-qa.ipynb

@@ -11,7 +11,7 @@
     "\n",
     "Read the paper [here](https://arxiv.org/abs/2310.06117)\n",
     "\n",
-    "See an excellent blog post on this by Cobus Greyling [here](https://cobusgreyling.medium.com/a-new-prompt-engineering-technique-has-been-introduced-called-step-back-prompting-b00e8954cacb)\n",
+    "See an excelent blog post on this by Cobus Greyling [here](https://cobusgreyling.medium.com/a-new-prompt-engineering-technique-has-been-introduced-called-step-back-prompting-b00e8954cacb)\n",
     "\n",
     "In this cookbook we will replicate this technique. We modify the prompts used slightly to work better with chat models."
    ]

docs/.local_build.sh

@@ -14,8 +14,6 @@ cd ../_dist
 poetry run python scripts/model_feat_table.py
 poetry run nbdoc_build --srcdir docs
 cp ../cookbook/README.md src/pages/cookbook.mdx
-cp ../.github/CONTRIBUTING.md docs/contributing.md
-wget https://raw.githubusercontent.com/langchain-ai/langserve/main/README.md -O docs/guides/deployments/langserve.md
 poetry run python scripts/generate_api_reference_links.py
 yarn install
 yarn start

docs/docs/expression_language/cookbook/code_writing.ipynb

@@ -12,7 +12,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 11,
    "id": "bd7c259a",
    "metadata": {},
    "outputs": [],
@@ -20,7 +20,7 @@
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain_experimental.utilities import PythonREPL"
+    "from langchain.utilities import PythonREPL"
    ]
   },
   {
@@ -111,7 +111,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.1"
+   "version": "3.9.1"
   }
  },
  "nbformat": 4,

docs/docs/expression_language/cookbook/prompt_llm_parser.ipynb

@@ -30,7 +30,7 @@
    "source": [
     "## PromptTemplate + LLM\n",
     "\n",
-    "The simplest composition is just combing a prompt and model to create a chain that takes user input, adds it to a prompt, passes it to a model, and returns the raw model output.\n",
+    "The simplest composition is just combing a prompt and model to create a chain that takes user input, adds it to a prompt, passes it to a model, and returns the raw model input.\n",
     "\n",
     "Note, you can mix and match PromptTemplate/ChatPromptTemplates and LLMs/ChatModels as you like here."
    ]
@@ -76,7 +76,7 @@
    "id": "7eb9ef50",
    "metadata": {},
    "source": [
-    "Often times we want to attach kwargs that'll be passed to each model call. Here are a few examples of that:"
+    "Often times we want to attach kwargs that'll be passed to each model call. Here's a few examples of that:"
    ]
   },
   {

docs/docs/expression_language/how_to/routing.ipynb

@@ -346,7 +346,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.1"
   }
  },
  "nbformat": 4,

docs/docs/get_started/quickstart.mdx

@@ -18,7 +18,7 @@ import CodeBlock from "@theme/CodeBlock";
 </Tabs>
 
 
-For more details, see our [Installation guide](/docs/get_started/installation).
+For more details, see our [Installation guide](/docs/get_started/installation.html).
 
 ## Environment setup
 
@@ -144,7 +144,7 @@ Whatever values are passed in during run time will always override what the obje
 
 Most LLM applications do not pass user input directly into an LLM. Usually they will add the user input to a larger piece of text, called a prompt template, that provides additional context on the specific task at hand.
 
-In the previous example, the text we passed to the model contained instructions to generate a company name. For our application, it would be great if the user only had to provide the description of a company/product, without having to worry about giving the model instructions.
+In the previous example, the text we passed to the model contained instructions to generate a company name. For our application, it'd be great if the user only had to provide the description of a company/product, without having to worry about giving the model instructions.
 
 PromptTemplates help with exactly this!
 They bundle up all the logic for going from user input into a fully formatted prompt.
@@ -167,7 +167,7 @@ You can compose them together, easily combining different templates into a singl
 For explanations of these functionalities, see the [section on prompts](/docs/modules/model_io/prompts) for more detail.
 
 PromptTemplates can also be used to produce a list of messages.
-In this case, the prompt not only contains information about the content, but also each message (its role, its position in the list, etc.).
+In this case, the prompt not only contains information about the content, but also each message (its role, its position in the list, etc)
 Here, what happens most often is a ChatPromptTemplate is a list of ChatMessageTemplates.
 Each ChatMessageTemplate contains instructions for how to format that ChatMessage - its role, and then also its content.
 Let's take a look at this below:
@@ -199,13 +199,13 @@ ChatPromptTemplates can also be constructed in other ways - see the [section on
 ## Output parsers
 
 OutputParsers convert the raw output of an LLM into a format that can be used downstream.
-There are few main types of OutputParsers, including:
+There are few main type of OutputParsers, including:
 
-- Convert text from LLM into structured information (e.g. JSON)
+- Convert text from LLM -> structured information (e.g. JSON)
 - Convert a ChatMessage into just a string
 - Convert the extra information returned from a call besides the message (like OpenAI function invocation) into a string.
 
-For full information on this, see the [section on output parsers](/docs/modules/model_io/output_parsers).
+For full information on this, see the [section on output parsers](/docs/modules/model_io/output_parsers)
 
 In this getting started guide, we will write our own output parser - one that converts a comma separated list into a list.
 

docs/docs/guides/debugging.md

@@ -376,7 +376,7 @@ agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is
 
 </details>
 
-### `set_verbose(True)`
+### `set_vebose(True)`
 
 Setting the `verbose` flag will print out inputs and outputs in a slightly more readable format and will skip logging certain raw outputs (like the token usage stats for an LLM call) so that you can focus on application logic.
 
@@ -656,6 +656,6 @@ agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is
 
 ## Other callbacks
 
-`Callbacks` are what we use to execute any functionality within a component outside the primary component logic. All of the above solutions use `Callbacks` under the hood to log intermediate steps of components. There are a number of `Callbacks` relevant for debugging that come with LangChain out of the box, like the [FileCallbackHandler](/docs/modules/callbacks/how_to/filecallbackhandler). You can also implement your own callbacks to execute custom functionality.
+`Callbacks` are what we use to execute any functionality within a component outside the primary component logic. All of the above solutions use `Callbacks` under the hood to log intermediate steps of components. There's a number of `Callbacks` relevant for debugging that come with LangChain out of the box, like the [FileCallbackHandler](/docs/modules/callbacks/how_to/filecallbackhandler). You can also implement your own callbacks to execute custom functionality.
 
 See here for more info on [Callbacks](/docs/modules/callbacks/), how to use them, and customize them.

docs/docs/guides/deployments/index.mdx

@@ -1,6 +1,6 @@
 # Deployment
 
-In today's fast-paced technological landscape, the use of Large Language Models (LLMs) is rapidly expanding. As a result, it is crucial for developers to understand how to effectively deploy these models in production environments. LLM interfaces typically fall into two categories:
+In today's fast-paced technological landscape, the use of Large Language Models (LLMs) is rapidly expanding. As a result, it's crucial for developers to understand how to effectively deploy these models in production environments. LLM interfaces typically fall into two categories:
 
 - **Case 1: Utilizing External LLM Providers (OpenAI, Anthropic, etc.)**
     In this scenario, most of the computational burden is handled by the LLM providers, while LangChain simplifies the implementation of business logic around these services. This approach includes features such as prompt templating, chat message generation, caching, vector embedding database creation, preprocessing, etc.
@@ -20,11 +20,11 @@ This guide aims to provide a comprehensive overview of the requirements for depl
 
 Understanding these components is crucial when assessing serving systems. LangChain integrates with several open-source projects designed to tackle these issues, providing a robust framework for productionizing your LLM applications. Some notable frameworks include:
 
-- [Ray Serve](/docs/ecosystem/integrations/ray_serve)
+- [Ray Serve](/docs/ecosystem/integrations/ray_serve.html)
 - [BentoML](https://github.com/bentoml/BentoML)
-- [OpenLLM](/docs/ecosystem/integrations/openllm)
-- [Modal](/docs/ecosystem/integrations/modal)
-- [Jina](/docs/ecosystem/integrations/jina#deployment)
+- [OpenLLM](/docs/ecosystem/integrations/openllm.html)
+- [Modal](/docs/ecosystem/integrations/modal.html)
+- [Jina](/docs/ecosystem/integrations/jina.html#deployment)
 
 These links will provide further information on each ecosystem, assisting you in finding the best fit for your LLM deployment needs.
 

docs/docs/guides/evaluation/string/json.ipynb

@@ -1,385 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "465cfbef-5bba-4b3b-b02d-fe2eba39db17",
-   "metadata": {},
-   "source": [
-    "# Evaluating Structured Output: JSON Evaluators\n",
-    "\n",
-    "Evaluating [extraction](https://python.langchain.com/docs/use_cases/extraction) and function calling applications often comes down to validation that the LLM's string output can be parsed correctly and how it compares to a reference object. The following JSON validators provide provide functionality to check your model's output in a consistent way.\n",
-    "\n",
-    "## JsonValidityEvaluator\n",
-    "\n",
-    "The `JsonValidityEvaluator` is designed to check the validity of a JSON string prediction.\n",
-    "\n",
-    "### Overview:\n",
-    "- **Requires Input?**: No\n",
-    "- **Requires Reference?**: No"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "02e5f7dd-82fe-48f9-a251-b2052e17e61c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': 1}\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.evaluation import JsonValidityEvaluator, load_evaluator\n",
-    "\n",
-    "evaluator = JsonValidityEvaluator()\n",
-    "# Equivalently\n",
-    "# evaluator = load_evaluator(\"json_validity\")\n",
-    "prediction = '{\"name\": \"John\", \"age\": 30, \"city\": \"New York\"}'\n",
-    "\n",
-    "result = evaluator.evaluate_strings(prediction=prediction)\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "9a9607c6-edab-4c26-86c4-22b226e18aa9",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes: line 1 column 48 (char 47)'}\n"
-     ]
-    }
-   ],
-   "source": [
-    "prediction = '{\"name\": \"John\", \"age\": 30, \"city\": \"New York\",}'\n",
-    "result = evaluator.evaluate_strings(prediction=prediction)\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8ac18a83-30d8-4c11-abf2-7a36e4cb829f",
-   "metadata": {},
-   "source": [
-    "## JsonEqualityEvaluator\n",
-    "\n",
-    "The `JsonEqualityEvaluator` assesses whether a JSON prediction matches a given reference after both are parsed.\n",
-    "\n",
-    "### Overview:\n",
-    "- **Requires Input?**: No\n",
-    "- **Requires Reference?**: Yes\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "ab97111e-cba9-4273-825f-d5d4278a953c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': True}\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.evaluation import JsonEqualityEvaluator\n",
-    "\n",
-    "evaluator = JsonEqualityEvaluator()\n",
-    "# Equivalently\n",
-    "# evaluator = load_evaluator(\"json_equality\")\n",
-    "result = evaluator.evaluate_strings(prediction='{\"a\": 1}', reference='{\"a\": 1}')\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "655ba486-09b6-47ce-947d-b2bd8b6f6364",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': False}\n"
-     ]
-    }
-   ],
-   "source": [
-    "result = evaluator.evaluate_strings(prediction='{\"a\": 1}', reference='{\"a\": 2}')\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1ac7e541-b7fe-46b6-bc3a-e94fe316227e",
-   "metadata": {},
-   "source": [
-    "The evaluator also by default lets you provide a dictionary directly"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "36e70ba3-4e62-483c-893a-5f328b7f303d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': False}\n"
-     ]
-    }
-   ],
-   "source": [
-    "result = evaluator.evaluate_strings(prediction={\"a\": 1}, reference={\"a\": 2})\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "921d33f0-b3c2-4e9e-820c-9ec30bc5bb20",
-   "metadata": {},
-   "source": [
-    "## JsonEditDistanceEvaluator\n",
-    "\n",
-    "The `JsonEditDistanceEvaluator` computes a normalized Damerau-Levenshtein distance between two \"canonicalized\" JSON strings.\n",
-    "\n",
-    "### Overview:\n",
-    "- **Requires Input?**: No\n",
-    "- **Requires Reference?**: Yes\n",
-    "- **Distance Function**: Damerau-Levenshtein (by default)\n",
-    "\n",
-    "_Note: Ensure that `rapidfuzz` is installed or provide an alternative `string_distance` function to avoid an ImportError._"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "da9ec3a3-675f-4420-8ec7-cde48d8c2918",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': 0.07692307692307693}\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.evaluation import JsonEditDistanceEvaluator\n",
-    "\n",
-    "evaluator = JsonEditDistanceEvaluator()\n",
-    "# Equivalently\n",
-    "# evaluator = load_evaluator(\"json_edit_distance\")\n",
-    "\n",
-    "result = evaluator.evaluate_strings(\n",
-    "    prediction='{\"a\": 1, \"b\": 2}', reference='{\"a\": 1, \"b\": 3}'\n",
-    ")\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "537ed58c-6a9c-402f-8f7f-07b1119a9ae0",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': 0.0}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# The values are canonicalized prior to comparison\n",
-    "result = evaluator.evaluate_strings(\n",
-    "    prediction=\"\"\"\n",
-    "    {\n",
-    "        \"b\": 3,\n",
-    "        \"a\":   1\n",
-    "    }\"\"\",\n",
-    "    reference='{\"a\": 1, \"b\": 3}',\n",
-    ")\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "7a8f3ec5-1cde-4b0e-80cd-ac0ac290d375",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': 0.18181818181818182}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Lists maintain their order, however\n",
-    "result = evaluator.evaluate_strings(\n",
-    "    prediction='{\"a\": [1, 2]}', reference='{\"a\": [2, 1]}'\n",
-    ")\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "52abec79-58ed-4ab6-9fb1-7deb1f5146cc",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': 0.14285714285714285}\n"
-     ]
-    }
-   ],
-   "source": [
-    "# You can also pass in objects directly\n",
-    "result = evaluator.evaluate_strings(prediction={\"a\": 1}, reference={\"a\": 2})\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6b15d18e-9b97-434f-905c-70acd4c35aea",
-   "metadata": {},
-   "source": [
-    "## JsonSchemaEvaluator\n",
-    "\n",
-    "The `JsonSchemaEvaluator` validates a JSON prediction against a provided JSON schema. If the prediction conforms to the schema, it returns a score of True (indicating no errors). Otherwise, it returns a score of 0 (indicating an error).\n",
-    "\n",
-    "### Overview:\n",
-    "- **Requires Input?**: Yes\n",
-    "- **Requires Reference?**: Yes (A JSON schema)\n",
-    "- **Score**: True (No errors) or False (Error occurred)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "85afcf33-d2f4-406e-9d8f-15dc0a4772f2",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': True}\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.evaluation import JsonSchemaEvaluator\n",
-    "\n",
-    "evaluator = JsonSchemaEvaluator()\n",
-    "# Equivalently\n",
-    "# evaluator = load_evaluator(\"json_schema_validation\")\n",
-    "\n",
-    "result = evaluator.evaluate_strings(\n",
-    "    prediction='{\"name\": \"John\", \"age\": 30}',\n",
-    "    reference={\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\"name\": {\"type\": \"string\"}, \"age\": {\"type\": \"integer\"}},\n",
-    "    },\n",
-    ")\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "bb5b89f6-0c87-4335-9091-55fd67a0565f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': True}\n"
-     ]
-    }
-   ],
-   "source": [
-    "result = evaluator.evaluate_strings(\n",
-    "    prediction='{\"name\": \"John\", \"age\": 30}',\n",
-    "    reference='{\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"}, \"age\": {\"type\": \"integer\"}}}',\n",
-    ")\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "ff914d24-36bc-482a-a9ba-259cd0dd2a52",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'score': False, 'reasoning': \"<ValidationError: '30 is less than the minimum of 66'>\"}\n"
-     ]
-    }
-   ],
-   "source": [
-    "result = evaluator.evaluate_strings(\n",
-    "    prediction='{\"name\": \"John\", \"age\": 30}',\n",
-    "    reference='{\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"},'\n",
-    "    '\"age\": {\"type\": \"integer\", \"minimum\": 66}}}',\n",
-    ")\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b073f12d-4603-481c-8081-fab1af6bfcfe",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.2"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/guides/langsmith/walkthrough.ipynb

@@ -200,6 +200,7 @@
     "    \"What is LangChain?\",\n",
     "    \"What's LangSmith?\",\n",
     "    \"When was Llama-v2 released?\",\n",
+    "    \"Who trained Llama-v2?\",\n",
     "    \"What is the langsmith cookbook?\",\n",
     "    \"When did langchain first announce the hub?\",\n",
     "]\n",

docs/docs/guides/safety/amazon_comprehend_chain.ipynb

@@ -16,10 +16,7 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "2c4236d8-4054-473d-84a4-87a4db278a62",
-   "metadata": {
-    "scrolled": true,
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%pip install boto3 nltk"
@@ -27,33 +24,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "9c792c3d-c601-409c-8e41-1c05a2fa0e84",
-   "metadata": {
-    "scrolled": true,
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "%pip install -U langchain_experimental"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "496df413-a840-40a1-9ac0-3af7c1303476",
-   "metadata": {
-    "scrolled": true,
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "%pip install -U langchain pydantic"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "id": "3f8518ad-c762-413c-b8c9-f1c211fc311d",
    "metadata": {
     "tags": []
@@ -61,7 +32,6 @@
    "outputs": [],
    "source": [
     "import boto3\n",
-    "import os\n",
     "\n",
     "comprehend_client = boto3.client('comprehend', region_name='us-east-1')"
    ]
@@ -103,6 +73,7 @@
    "outputs": [],
    "source": [
     "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
     "from langchain.llms.fake import FakeListLLM\n",
     "from langchain_experimental.comprehend_moderation.base_moderation_exceptions import ModerationPiiError\n",
     "\n",
@@ -114,22 +85,25 @@
     "\n",
     "responses = [\n",
     "    \"Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.\", \n",
-    "    # replace with your own expletive\n",
-    "    \"Final Answer: This is a really <expletive> way of constructing a birdhouse. This is <expletive> insane to think that any birds would actually create their <expletive> nests here.\"\n",
+    "    \"Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here.\"\n",
     "]\n",
     "llm = FakeListLLM(responses=responses)\n",
     "\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
+    "\n",
     "chain = (\n",
     "    prompt \n",
     "    | comprehend_moderation \n",
-    "    | {\"input\": (lambda x: x['output'] ) | llm}\n",
+    "    | {llm_chain.input_keys[0]: lambda x: x['output'] }  \n",
+    "    | llm_chain \n",
+    "    | { \"input\": lambda x: x['text'] } \n",
     "    | comprehend_moderation \n",
     ")\n",
     "\n",
     "try:\n",
-    "    response = chain.invoke({\"question\": \"A sample SSN number looks like this 123-22-3345. Can you give me some more samples?\"})\n",
+    "    response = chain.invoke({\"question\": \"A sample SSN number looks like this 123-456-7890. Can you give me some more samples?\"})\n",
     "except ModerationPiiError as e:\n",
-    "    print(str(e))\n",
+    "    print(e.message)\n",
     "else:\n",
     "    print(response['output'])\n"
    ]
@@ -143,7 +117,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "bfd550e7-5012-41fa-9546-8b78ddf1c673",
    "metadata": {},
@@ -152,7 +125,7 @@
     "\n",
     "- PII (Personally Identifiable Information) checks \n",
     "- Toxicity content detection\n",
-    "- Prompt Safety detection\n",
+    "- Intention detection\n",
     "\n",
     "Here is an example of a moderation config."
    ]
@@ -166,51 +139,46 @@
    },
    "outputs": [],
    "source": [
-    "from langchain_experimental.comprehend_moderation import (BaseModerationConfig, \n",
-    "                                 ModerationPromptSafetyConfig, \n",
-    "                                 ModerationPiiConfig, \n",
-    "                                 ModerationToxicityConfig\n",
-    ")\n",
+    "from langchain_experimental.comprehend_moderation import BaseModerationActions, BaseModerationFilters\n",
     "\n",
-    "pii_config = ModerationPiiConfig(\n",
-    "    labels=[\"SSN\"],\n",
-    "    redact=True,\n",
-    "    mask_character=\"X\"\n",
-    ")\n",
-    "\n",
-    "toxicity_config = ModerationToxicityConfig(\n",
-    "    threshold=0.5\n",
-    ")\n",
-    "\n",
-    "prompt_safety_config = ModerationPromptSafetyConfig(\n",
-    "    threshold=0.5\n",
-    ")\n",
-    "\n",
-    "moderation_config = BaseModerationConfig(\n",
-    "    filters=[pii_config, toxicity_config, prompt_safety_config]\n",
-    ")"
+    "moderation_config = { \n",
+    "        \"filters\":[ \n",
+    "                BaseModerationFilters.PII, \n",
+    "                BaseModerationFilters.TOXICITY,\n",
+    "                BaseModerationFilters.INTENT\n",
+    "        ],\n",
+    "        \"pii\":{ \n",
+    "                \"action\": BaseModerationActions.ALLOW, \n",
+    "                \"threshold\":0.5, \n",
+    "                \"labels\":[\"SSN\"],\n",
+    "                \"mask_character\": \"X\"\n",
+    "        },\n",
+    "        \"toxicity\":{ \n",
+    "                \"action\": BaseModerationActions.STOP, \n",
+    "                \"threshold\":0.5\n",
+    "        },\n",
+    "        \"intent\":{ \n",
+    "                \"action\": BaseModerationActions.STOP, \n",
+    "                \"threshold\":0.5\n",
+    "        }\n",
+    "}"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "3634376b-5938-43df-9ed6-70ca7e99290f",
    "metadata": {},
    "source": [
-    "At the core of the the configuration there are three configuration models to be used\n",
+    "At the core of the configuration you have three filters specified in the `filters` key:\n",
     "\n",
-    "- `ModerationPiiConfig` used for configuring the behavior of the PII validations. Following are the parameters it can be initialized with\n",
-    "  - `labels` the PII entity labels. Defaults to an empty list which means that the PII validation will consider all PII entities.\n",
-    "  - `threshold` the confidence threshold for the detected entities, defaults to 0.5 or 50%\n",
-    "  - `redact` a boolean flag to enforce whether redaction should be performed on the text, defaults to `False`. When `False`, the PII validation will error out when it detects any PII entity, when set to `True` it simply redacts the PII values in the text.\n",
-    "  - `mask_character` the character used for masking, defaults to asterisk (*)\n",
-    "- `ModerationToxicityConfig` used for configuring the behavior of the toxicity validations. Following are the parameters it can be initialized with\n",
-    "  - `labels` the Toxic entity labels. Defaults to an empty list which means that the toxicity validation will consider all toxic entities. all\n",
-    "  - `threshold` the confidence threshold for the detected entities, defaults to 0.5 or 50% \n",
-    "- `ModerationPromptSafetyConfig` used for configuring the behavior of the prompt safety validation\n",
-    "  - `threshold` the confidence threshold for the the prompt safety classification, defaults to 0.5 or 50% \n",
+    "1. `BaseModerationFilters.PII`\n",
+    "2. `BaseModerationFilters.TOXICITY`\n",
+    "3. `BaseModerationFilters.INTENT`\n",
     "\n",
-    "Finally, you use the `BaseModerationConfig` to define the order in which each of these checks are to be performed. The `BaseModerationConfig` takes an optional `filters` parameter which can be a list of one or more than one of the above validation checks, as seen in the previous code block. The  `BaseModerationConfig` can also be initialized with any `filters` in which case it will use all the checks with default configuration (more on this explained later).\n",
+    "And an `action` key that defines two possible actions for each moderation function:\n",
+    "\n",
+    "1. `BaseModerationActions.ALLOW` - `allows` the prompt to pass through but masks detected PII in case of PII check. The default behavior is to run and redact all PII entities. If there is an entity specified in the `labels` field, then only those entities will go through the PII check and masked.\n",
+    "2. `BaseModerationActions.STOP` - `stops` the prompt from passing through to the next step in case any PII, Toxicity, or incorrect Intent is detected. The action of `BaseModerationActions.STOP` will raise a Python `Exception` essentially stopping the chain in progress.\n",
     "\n",
     "Using the configuration in the previous cell will perform PII checks and will allow the prompt to pass through however it will mask any SSN numbers present in either the prompt or the LLM output.\n"
    ]
@@ -228,20 +196,7 @@
     "    moderation_config=moderation_config, #specify the configuration\n",
     "    client=comprehend_client,            #optionally pass the Boto3 Client\n",
     "    verbose=True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a25e6f93-765b-4f99-8c1c-929157dbd4aa",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.llms.fake import FakeListLLM\n",
+    ")\n",
     "\n",
     "template = \"\"\"Question: {question}\n",
     "\n",
@@ -251,21 +206,23 @@
     "\n",
     "responses = [\n",
     "    \"Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.\", \n",
-    "    # replace with your own expletive\n",
-    "    \"Final Answer: This is a really <expletive> way of constructing a birdhouse. This is <expletive> insane to think that any birds would actually create their <expletive> nests here.\"\n",
+    "    \"Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here.\"\n",
     "]\n",
     "llm = FakeListLLM(responses=responses)\n",
     "\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
+    "\n",
     "chain = ( \n",
     "    prompt \n",
     "    | comp_moderation_with_config \n",
-    "    | {\"input\": (lambda x: x['output'] ) | llm}\n",
+    "    | {llm_chain.input_keys[0]: lambda x: x['output'] }  \n",
+    "    | llm_chain \n",
+    "    | { \"input\": lambda x: x['text'] } \n",
     "    | comp_moderation_with_config \n",
     ")\n",
     "\n",
-    "\n",
     "try:\n",
-    "    response = chain.invoke({\"question\": \"A sample SSN number looks like this 123-45-7890. Can you give me some more samples?\"})\n",
+    "    response = chain.invoke({\"question\": \"A sample SSN number looks like this 123-456-7890. Can you give me some more samples?\"})\n",
     "except Exception as e:\n",
     "    print(str(e))\n",
     "else:\n",
@@ -273,7 +230,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "ba890681-feeb-43ca-a0d5-9c11d2d9de3e",
    "metadata": {
@@ -282,25 +238,25 @@
    "source": [
     "## Unique ID, and Moderation Callbacks\n",
     "\n",
-    "When Amazon Comprehend moderation action identifies any of the configugred entity, the chain will raise one of the following exceptions-\n",
+    "When Amazon Comprehend moderation action is specified as `STOP`, the chain will raise one of the following exceptions-\n",
     "    - `ModerationPiiError`, for PII checks\n",
     "    - `ModerationToxicityError`, for Toxicity checks \n",
-    "    - `ModerationPromptSafetyError` for Prompt Safety checks\n",
+    "    - `ModerationIntentionError` for Intent checks\n",
     "\n",
     "In addition to the moderation configuration, the `AmazonComprehendModerationChain` can also be initialized with the following parameters\n",
     "\n",
     "- `unique_id` [Optional] a string parameter. This parameter can be used to pass any string value or ID. For example, in a chat application, you may want to keep track of abusive users, in this case, you can pass the user's username/email ID etc. This defaults to `None`.\n",
     "\n",
-    "- `moderation_callback` [Optional] the `BaseModerationCallbackHandler` that will be called asynchronously (non-blocking to the chain). Callback functions are useful when you want to perform additional actions when the moderation functions are executed, for example logging into a database, or writing a log file. You can override three functions by subclassing `BaseModerationCallbackHandler` - `on_after_pii()`, `on_after_toxicity()`, and `on_after_prompt_safety()`. Note that all three functions must be `async` functions. These callback functions receive two arguments:\n",
-    "    - `moderation_beacon` a dictionary that will contain information about the moderation function, the full response from Amazon Comprehend model, a unique chain id, the moderation status, and the input string which was validated. The dictionary is of the following schema-\n",
+    "- `moderation_callback` [Optional] the `BaseModerationCallbackHandler` will be called asynchronously (non-blocking to the chain). Callback functions are useful when you want to perform additional actions when the moderation functions are executed, for example logging into a database, or writing a log file. You can override three functions by subclassing `BaseModerationCallbackHandler` - `on_after_pii()`, `on_after_toxicity()`, and `on_after_intent()`. Note that all three functions must be `async` functions. These callback functions receive two arguments:\n",
+    "    - `moderation_beacon` is a dictionary that will contain information about the moderation function, the full response from the Amazon Comprehend model, a unique chain id, the moderation status, and the input string which was validated. The dictionary is of the following schema-\n",
     "    \n",
     "    ```\n",
     "    { \n",
     "        'moderation_chain_id': 'xxx-xxx-xxx', # Unique chain ID\n",
-    "        'moderation_type': 'Toxicity' | 'PII' | 'PromptSafety', \n",
+    "        'moderation_type': 'Toxicity' | 'PII' | 'Intent', \n",
     "        'moderation_status': 'LABELS_FOUND' | 'LABELS_NOT_FOUND',\n",
     "        'moderation_input': 'A sample SSN number looks like this 123-456-7890. Can you give me some more samples?',\n",
-    "        'moderation_output': {...} #Full Amazon Comprehend PII, Toxicity, or Prompt Safety Model Output\n",
+    "        'moderation_output': {...} #Full Amazon Comprehend PII, Toxicity, or Intent Model Output\n",
     "    }\n",
     "    ```\n",
     "    \n",
@@ -357,7 +313,7 @@
     "    async def on_after_toxicity(self, output_beacon, unique_id):\n",
     "        pass\n",
     "    \n",
-    "    async def on_after_prompt_safety(self, output_beacon, unique_id):\n",
+    "    async def on_after_intent(self, output_beacon, unique_id):\n",
     "        pass\n",
     "    '''\n",
     "    \n",
@@ -374,19 +330,22 @@
    },
    "outputs": [],
    "source": [
-    "pii_config = ModerationPiiConfig(\n",
-    "    labels=[\"SSN\"],\n",
-    "    redact=True,\n",
-    "    mask_character=\"X\"\n",
-    ")\n",
-    "\n",
-    "toxicity_config = ModerationToxicityConfig(\n",
-    "    threshold=0.5\n",
-    ")\n",
-    "\n",
-    "moderation_config = BaseModerationConfig(\n",
-    "    filters=[pii_config, toxicity_config]\n",
-    ")\n",
+    "moderation_config = { \n",
+    "        \"filters\": [ \n",
+    "                BaseModerationFilters.PII, \n",
+    "                BaseModerationFilters.TOXICITY\n",
+    "        ],\n",
+    "        \"pii\":{ \n",
+    "                \"action\": BaseModerationActions.STOP, \n",
+    "                \"threshold\":0.5, \n",
+    "                \"labels\":[\"SSN\"], \n",
+    "                \"mask_character\": \"X\" \n",
+    "        },\n",
+    "        \"toxicity\":{ \n",
+    "                \"action\": BaseModerationActions.STOP, \n",
+    "                \"threshold\":0.5 \n",
+    "        }\n",
+    "}\n",
     "\n",
     "comp_moderation_with_config = AmazonComprehendModerationChain(\n",
     "        moderation_config=moderation_config, # specify the configuration\n",
@@ -407,6 +366,7 @@
    "outputs": [],
    "source": [
     "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
     "from langchain.llms.fake import FakeListLLM\n",
     "\n",
     "template = \"\"\"Question: {question}\n",
@@ -417,16 +377,19 @@
     "\n",
     "responses = [\n",
     "    \"Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.\", \n",
-    "    # replace with your own expletive\n",
-    "    \"Final Answer: This is a really <expletive> way of constructing a birdhouse. This is <expletive> insane to think that any birds would actually create their <expletive> nests here.\"\n",
+    "    \"Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here.\"\n",
     "]\n",
     "\n",
     "llm = FakeListLLM(responses=responses)\n",
     "\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
+    "\n",
     "chain = (\n",
     "    prompt \n",
     "    | comp_moderation_with_config \n",
-    "    | {\"input\": (lambda x: x['output'] ) | llm}\n",
+    "    | {llm_chain.input_keys[0]: lambda x: x['output'] }  \n",
+    "    | llm_chain \n",
+    "    | { \"input\": lambda x: x['text'] } \n",
     "    | comp_moderation_with_config \n",
     ") \n",
     "\n",
@@ -439,7 +402,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "706454b2-2efa-4d41-abc8-ccf2b4e87822",
    "metadata": {
@@ -448,7 +410,7 @@
    "source": [
     "## `moderation_config` and moderation execution order\n",
     "\n",
-    "If `AmazonComprehendModerationChain` is not initialized with any `moderation_config` then it is initialized with the default values of `BaseModerationConfig`. If no `filters` are used then the sequence of moderation check is as follows.\n",
+    "If `AmazonComprehendModerationChain` is not initialized with any `moderation_config` then the default action is `STOP` and the default order of moderation check is as follows.\n",
     "\n",
     "```\n",
     "AmazonComprehendModerationChain\n",
@@ -461,32 +423,39 @@
     "            ├── Callback (if available)\n",
     "            ├── Label Found ⟶ [Error Stop]\n",
     "            └── No Label Found\n",
-    "                └──Check Prompt Safety with Stop Action\n",
+    "                └──Check Intent with Stop Action\n",
     "                    ├── Callback (if available)\n",
     "                    ├── Label Found ⟶ [Error Stop]\n",
     "                    └── No Label Found\n",
     "                        └── Return Prompt\n",
     "```\n",
     "\n",
-    "If any of the check raises a validation exception then the subsequent checks will not be performed. If a `callback` is provided in this case, then it will be called for each of the checks that have been performed. For example, in the case above, if the Chain fails due to presence of PII then the Toxicity and Prompt Safety checks will not be performed.\n",
+    "If any of the checks raises an exception then the subsequent checks will not be performed. If a `callback` is provided in this case, then it will be called for each of the checks that have been performed. For example, in the case above, if the Chain fails due to the presence of PII then the Toxicity and Intent checks will not be performed.\n",
     "\n",
-    "You can override the execution order by passing `moderation_config` and simply specifying the desired order in the `filters` parameter of the `BaseModerationConfig`. In case you specify the filters, then the order of the checks as specified in the `filters` parameter will be maintained. For example, in the configuration below, first Toxicity check will be performed, then PII, and finally Prompt Safety validation will be performed. In this case, `AmazonComprehendModerationChain` will perform the desired checks in the specified order with default values of each model `kwargs`.\n",
+    "You can override the execution order by passing `moderation_config` and simply specifying the desired order in the `filters` key of the configuration. In case you use `moderation_config` then the order of the checks as specified in the `filters` key will be maintained. For example, in the configuration below, first Toxicity check will be performed, then PII, and finally Intent validation will be performed. In this case, `AmazonComprehendModerationChain` will perform the desired checks in the specified order with default values of each model `kwargs`.\n",
     "\n",
     "```python\n",
-    "pii_check = ModerationPiiConfig()\n",
-    "toxicity_check = ModerationToxicityConfig()\n",
-    "prompt_safety_check = ModerationPromptSafetyConfig()\n",
-    "\n",
-    "moderation_config = BaseModerationConfig(filters=[toxicity_check, pii_check, prompt_safety_check])\n",
+    "moderation_config = { \n",
+    "        \"filters\":[ BaseModerationFilters.TOXICITY, \n",
+    "                    BaseModerationFilters.PII, \n",
+    "                    BaseModerationFilters.INTENT]\n",
+    "   }\n",
     "```\n",
     "\n",
-    "You can have also use more than one configuration for a specific moderation check, for example in the sample below, two consecutive PII checks are performed. First the configuration checks for any SSN, if found it would raise an error. If any SSN isn't found then it will next check if any NAME and CREDIT_DEBIT_NUMBER is present in the prompt and will mask it.\n",
+    "Model `kwargs` are specified by the `pii`, `toxicity`, and `intent` keys within the `moderation_config` dictionary. For example, in the `moderation_config` below, the default order of moderation is overriden and the `pii` & `toxicity` model `kwargs` have been overriden. For `intent` the chain's default `kwargs` will be used.\n",
     "\n",
     "```python\n",
-    "pii_check_1 = ModerationPiiConfig(labels=[\"SSN\"])\n",
-    "pii_check_2 = ModerationPiiConfig(labels=[\"NAME\", \"CREDIT_DEBIT_NUMBER\"], redact=True)\n",
-    "\n",
-    "moderation_config = BaseModerationConfig(filters=[pii_check_1, pii_check_2])\n",
+    " moderation_config = { \n",
+    "        \"filters\":[ BaseModerationFilters.TOXICITY, \n",
+    "                    BaseModerationFilters.PII, \n",
+    "                    BaseModerationFilters.INTENT],\n",
+    "        \"pii\":{ \"action\": BaseModerationActions.ALLOW, \n",
+    "                \"threshold\":0.5, \n",
+    "                \"labels\":[\"SSN\"], \n",
+    "                \"mask_character\": \"X\" },\n",
+    "        \"toxicity\":{ \"action\": BaseModerationActions.STOP, \n",
+    "                     \"threshold\":0.5 }\n",
+    "   }\n",
     "```\n",
     "\n",
     "1. For a list of PII labels see Amazon Comprehend Universal PII entity types - https://docs.aws.amazon.com/comprehend/latest/dg/how-pii.html#how-pii-types\n",
@@ -498,11 +467,10 @@
     "    - `VIOLENCE_OR_THREAT`: Speech that includes threats which seek to inflict pain, injury or hostility towards a person or group.\n",
     "    - `INSULT`: Speech that includes demeaning, humiliating, mocking, insulting, or belittling language.\n",
     "    - `PROFANITY`: Speech that contains words, phrases or acronyms that are impolite, vulgar, or offensive is considered as profane.\n",
-    "3. For a list of Prompt Safety labels refer to documentation [link here]"
+    "3. For a list of Intent labels refer to documentation [link here]"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "78905aec-55ae-4fc3-a23b-8a69bd1e33f2",
    "metadata": {},
@@ -536,8 +504,7 @@
    },
    "outputs": [],
    "source": [
-    "import os\n",
-    "os.environ[\"HUGGINGFACEHUB_API_TOKEN\"] = \"<YOUR HF TOKEN HERE>\""
+    "%env HUGGINGFACEHUB_API_TOKEN=\"<HUGGINGFACEHUB_API_TOKEN>\""
    ]
   },
   {
@@ -550,7 +517,7 @@
    "outputs": [],
    "source": [
     "# See https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads for some other options\n",
-    "repo_id = \"google/flan-t5-xxl\"  "
+    "repo_id = \"google/flan-t5-xxl\"  \n"
    ]
   },
   {
@@ -562,15 +529,20 @@
    },
    "outputs": [],
    "source": [
-    "from langchain import HuggingFaceHub\n",
+    "from langchain.llms import HuggingFaceHub\n",
     "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
     "\n",
-    "template = \"\"\"{question}\"\"\"\n",
+    "template = \"\"\"Question: {question}\n",
+    "\n",
+    "Answer:\"\"\"\n",
     "\n",
     "prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n",
+    "\n",
     "llm = HuggingFaceHub(\n",
     "    repo_id=repo_id, model_kwargs={\"temperature\": 0.5, \"max_length\": 256}\n",
-    ")"
+    ")\n",
+    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
    ]
   },
   {
@@ -590,41 +562,22 @@
    },
    "outputs": [],
    "source": [
+    "moderation_config = { \n",
+    "        \"filters\":[ BaseModerationFilters.PII, BaseModerationFilters.TOXICITY, BaseModerationFilters.INTENT ],\n",
+    "        \"pii\":{\"action\": BaseModerationActions.ALLOW, \"threshold\":0.5, \"labels\":[\"SSN\",\"CREDIT_DEBIT_NUMBER\"], \"mask_character\": \"X\"},\n",
+    "        \"toxicity\":{\"action\": BaseModerationActions.STOP, \"threshold\":0.5},\n",
+    "        \"intent\":{\"action\": BaseModerationActions.ALLOW, \"threshold\":0.5,},\n",
+    "   }\n",
     "\n",
-    "# define filter configs\n",
-    "pii_config = ModerationPiiConfig(\n",
-    "    labels=[\"SSN\", \"CREDIT_DEBIT_NUMBER\"],\n",
-    "    redact=True,\n",
-    "    mask_character=\"X\"\n",
-    ")\n",
-    "\n",
-    "toxicity_config = ModerationToxicityConfig(\n",
-    "    threshold=0.5\n",
-    ")\n",
-    "\n",
-    "prompt_safety_config = ModerationPromptSafetyConfig(\n",
-    "    threshold=0.8\n",
-    ")\n",
-    "\n",
-    "# define different moderation configs using the filter configs above\n",
-    "moderation_config_1 = BaseModerationConfig(\n",
-    "    filters=[pii_config, toxicity_config, prompt_safety_config]\n",
-    ")\n",
-    "\n",
-    "moderation_config_2 = BaseModerationConfig(\n",
-    "    filters=[pii_config]\n",
-    ")\n",
-    "\n",
-    "\n",
-    "# input prompt moderation chain with callback\n",
-    "amazon_comp_moderation = AmazonComprehendModerationChain(moderation_config=moderation_config_1, \n",
+    "# without any callback\n",
+    "amazon_comp_moderation = AmazonComprehendModerationChain(moderation_config=moderation_config, \n",
     "                                                         client=comprehend_client,\n",
-    "                                                         moderation_callback=my_callback,\n",
     "                                                         verbose=True)\n",
     "\n",
-    "# Output from LLM moderation chain without callback\n",
-    "amazon_comp_moderation_out = AmazonComprehendModerationChain(moderation_config=moderation_config_2, \n",
+    "# with callback\n",
+    "amazon_comp_moderation_out = AmazonComprehendModerationChain(moderation_config=moderation_config, \n",
     "                                                         client=comprehend_client,\n",
+    "                                                         moderation_callback=my_callback,\n",
     "                                                         verbose=True)"
    ]
   },
@@ -633,7 +586,7 @@
    "id": "b1256bc8-1321-4624-9e8a-a2d4a8df59bf",
    "metadata": {},
    "source": [
-    "The `moderation_config` will now prevent any inputs containing obscene words or sentences, bad intent, or PII with entities other than SSN with score above threshold or 0.5 or 50%. If it finds Pii entities - SSN - it will redact them before allowing the call to proceed. It will also mask any SSN or credit card numbers from the model's response."
+    "The `moderation_config` will now prevent any inputs and model outputs containing obscene words or sentences, bad intent, or PII with entities other than SSN with score above threshold or 0.5 or 50%. If it finds Pii entities - SSN - it will redact them before allowing the call to proceed. "
    ]
   },
   {
@@ -648,15 +601,14 @@
     "chain = (\n",
     "    prompt \n",
     "    | amazon_comp_moderation \n",
-    "    | { \"input\" : (lambda x: x['output']) | llm }\n",
+    "    | {llm_chain.input_keys[0]: lambda x: x['output'] }  \n",
+    "    | llm_chain \n",
+    "    | { \"input\": lambda x: x['text'] } \n",
     "    | amazon_comp_moderation_out\n",
     ")\n",
     "\n",
     "try:\n",
-    "    response = chain.invoke({\"question\": \"\"\"What is John Doe's address, phone number and SSN from the following text?\n",
-    "\n",
-    "John Doe, a resident of 1234 Elm Street in Springfield, recently celebrated his birthday on January 1st. Turning 43 this year, John reflected on the years gone by. He often shares memories of his younger days with his close friends through calls on his phone, (555) 123-4567. Meanwhile, during a casual evening, he received an email at johndoe@example.com reminding him of an old acquaintance's reunion. As he navigated through some old documents, he stumbled upon a paper that listed his SSN as 123-45-6789, reminding him to store it in a safer place.\n",
-    "\"\"\"})\n",
+    "    response = chain.invoke({\"question\": \"My AnyCompany Financial Services, LLC credit card account 1111-0000-1111-0008 has 24$ due by July 31st. Can you give me some more credit car number samples?\"})\n",
     "except Exception as e:\n",
     "    print(str(e))\n",
     "else:\n",
@@ -672,7 +624,7 @@
    "source": [
     "### With Amazon SageMaker Jumpstart\n",
     "\n",
-    "The exmaple below shows how to use Amazon Comprehend Moderation chain with an Amazon SageMaker Jumpstart hosted LLM. You should have an Amazon SageMaker Jumpstart hosted LLM endpoint within your AWS Account. Refer to [this notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-falcon.ipynb) for more on how to deploy an LLM with Amazon SageMaker Jumpstart hosted endpoints."
+    "The example below shows how to use the `Amazon Comprehend Moderation chain` with an Amazon SageMaker Jumpstart hosted LLM. You should have an `Amazon SageMaker Jumpstart` hosted LLM endpoint within your AWS Account. "
    ]
   },
   {
@@ -682,8 +634,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "endpoint_name = \"<SAGEMAKER_ENDPOINT_NAME>\" # replace with your SageMaker Endpoint name\n",
-    "region = \"<REGION>\"  # replace with your SageMaker Endpoint region"
+    "endpoint_name = \"<SAGEMAKER_ENDPOINT_NAME>\" # replace with your SageMaker Endpoint name"
    ]
   },
   {
@@ -693,9 +644,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain import SagemakerEndpoint\n",
+    "from langchain.llms import SagemakerEndpoint\n",
     "from langchain.llms.sagemaker_endpoint import LLMContentHandler\n",
-    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.prompts import load_prompt, PromptTemplate\n",
     "import json\n",
     "\n",
     "class ContentHandler(LLMContentHandler):\n",
@@ -712,27 +664,23 @@
     "\n",
     "content_handler = ContentHandler()\n",
     "\n",
-    "template = \"\"\"From the following 'Document', precisely answer the 'Question'. Do not add any spurious information in your answer.\n",
-    "\n",
-    "Document: John Doe, a resident of 1234 Elm Street in Springfield, recently celebrated his birthday on January 1st. Turning 43 this year, John reflected on the years gone by. He often shares memories of his younger days with his close friends through calls on his phone, (555) 123-4567. Meanwhile, during a casual evening, he received an email at johndoe@example.com reminding him of an old acquaintance's reunion. As he navigated through some old documents, he stumbled upon a paper that listed his SSN as 123-45-6789, reminding him to store it in a safer place.\n",
-    "Question: {question}\n",
-    "Answer:\n",
-    "\"\"\"\n",
-    "\n",
     "#prompt template for input text\n",
-    "llm_prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n",
+    "llm_prompt = PromptTemplate(input_variables=[\"input_text\"], template=\"{input_text}\")\n",
     "\n",
-    "llm=SagemakerEndpoint(\n",
+    "llm_chain = LLMChain(\n",
+    "    llm=SagemakerEndpoint(\n",
     "        endpoint_name=endpoint_name, \n",
-    "        region_name=region,\n",
-    "        model_kwargs={\"temperature\":0.95,\n",
+    "        region_name='us-east-1',\n",
+    "        model_kwargs={\"temperature\":0.97,\n",
     "                      \"max_length\": 200,\n",
     "                      \"num_return_sequences\": 3,\n",
     "                      \"top_k\": 50,\n",
     "                      \"top_p\": 0.95,\n",
     "                      \"do_sample\": True},\n",
     "        content_handler=content_handler\n",
-    "    )"
+    "    ),\n",
+    "    prompt=llm_prompt\n",
+    ")"
    ]
   },
   {
@@ -752,37 +700,15 @@
    },
    "outputs": [],
    "source": [
-    "# define filter configs\n",
-    "pii_config = ModerationPiiConfig(\n",
-    "    labels=[\"SSN\"],\n",
-    "    redact=True,\n",
-    "    mask_character=\"X\"\n",
-    ")\n",
+    "moderation_config = { \n",
+    "        \"filters\":[ BaseModerationFilters.PII, BaseModerationFilters.TOXICITY ],\n",
+    "        \"pii\":{\"action\": BaseModerationActions.ALLOW, \"threshold\":0.5, \"labels\":[\"SSN\"], \"mask_character\": \"X\"},\n",
+    "        \"toxicity\":{\"action\": BaseModerationActions.STOP, \"threshold\":0.5},\n",
+    "        \"intent\":{\"action\": BaseModerationActions.ALLOW, \"threshold\":0.5,},\n",
+    "   }\n",
     "\n",
-    "toxicity_config = ModerationToxicityConfig(\n",
-    "    threshold=0.5\n",
-    ")\n",
-    "\n",
-    "\n",
-    "# define different moderation configs using the filter configs above\n",
-    "moderation_config_1 = BaseModerationConfig(\n",
-    "    filters=[pii_config, toxicity_config]\n",
-    ")\n",
-    "\n",
-    "moderation_config_2 = BaseModerationConfig(\n",
-    "    filters=[pii_config]\n",
-    ")\n",
-    "\n",
-    "\n",
-    "# input prompt moderation chain with callback\n",
-    "amazon_comp_moderation = AmazonComprehendModerationChain(moderation_config=moderation_config_1, \n",
-    "                                                         client=comprehend_client,\n",
-    "                                                         moderation_callback=my_callback,\n",
-    "                                                         verbose=True)\n",
-    "\n",
-    "# Output from LLM moderation chain without callback\n",
-    "amazon_comp_moderation_out = AmazonComprehendModerationChain(moderation_config=moderation_config_2, \n",
-    "                                                         client=comprehend_client,\n",
+    "amazon_comp_moderation = AmazonComprehendModerationChain(moderation_config=moderation_config, \n",
+    "                                                         client=comprehend_client ,\n",
     "                                                         verbose=True)"
    ]
   },
@@ -806,12 +732,14 @@
     "chain = (\n",
     "    prompt \n",
     "    | amazon_comp_moderation \n",
-    "    | { \"input\" : (lambda x: x['output']) | llm }\n",
-    "    | amazon_comp_moderation_out\n",
+    "    | {llm_chain.input_keys[0]: lambda x: x['output'] }  \n",
+    "    | llm_chain \n",
+    "    | { \"input\": lambda x: x['text'] } \n",
+    "    | amazon_comp_moderation \n",
     ")\n",
     "\n",
     "try:\n",
-    "    response = chain.invoke({\"question\": \"What is John Doe's address, phone number and SSN?\"})\n",
+    "    response = chain.invoke({\"question\": \"My AnyCompany Financial Services, LLC credit card account 1111-0000-1111-0008 has 24$ due by July 31st. Can you give me some more samples?\"})\n",
     "except Exception as e:\n",
     "    print(str(e))\n",
     "else:\n",
@@ -1419,7 +1347,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.6"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

docs/docs/integrations/callbacks/argilla.ipynb

@@ -14,7 +14,7 @@
     "> using both human and machine feedback. We provide support for each step in the MLOps cycle, \n",
     "> from data labeling to model monitoring.\n",
     "\n",
-    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/integrations/callbacks/argilla\">\n",
+    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/integrations/callbacks/argilla.html\">\n",
     "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
     "</a>"
    ]

docs/docs/integrations/chat/gigachat.ipynb

@@ -1,114 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "source": [
-    "# GigaChat\n",
-    "This notebook shows how to use LangChain with [GigaChat](https://developers.sber.ru/portal/products/gigachat).\n",
-    "To use you need to install ```gigachat``` python package."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "# !pip install gigachat"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "To get GigaChat credentials you need to [create account](https://developers.sber.ru/studio/login) and [get access to API](https://developers.sber.ru/docs/ru/gigachat/api/integration)\n",
-    "## Example"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from getpass import getpass\n",
-    "\n",
-    "os.environ['GIGACHAT_CREDENTIALS'] = getpass()"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import GigaChat\n",
-    "\n",
-    "chat = GigaChat(verify_ssl_certs=False)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "What do you get when you cross a goat and a skunk? A smelly goat!\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.schema import SystemMessage, HumanMessage\n",
-    "\n",
-    "messages = [\n",
-    "    SystemMessage(\n",
-    "        content=\"You are a helpful AI that shares everything you know. Talk in English.\"\n",
-    "    ),\n",
-    "    HumanMessage(\n",
-    "        content=\"Tell me a joke\"\n",
-    "    ),\n",
-    "]\n",
-    "\n",
-    "print(chat(messages).content)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 2
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

docs/docs/integrations/chat/google_vertex_ai_palm.ipynb

@@ -5,7 +5,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Google Cloud Vertex AI \n",
+    "# GCP Vertex AI \n",
     "\n",
     "Note: This is separate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there. \n",
     "\n",
@@ -31,7 +31,7 @@
    },
    "outputs": [],
    "source": [
-    "#!pip install langchain google-cloud-aiplatform\n"
+    "#!pip install langchain google-cloud-aiplatform"
    ]
   },
   {
@@ -41,7 +41,7 @@
    "outputs": [],
    "source": [
     "from langchain.chat_models import ChatVertexAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n"
+    "from langchain.prompts import ChatPromptTemplate"
    ]
   },
   {
@@ -50,7 +50,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "chat = ChatVertexAI()\n"
+    "chat = ChatVertexAI()"
    ]
   },
   {
@@ -64,7 +64,7 @@
     "prompt = ChatPromptTemplate.from_messages(\n",
     "    [(\"system\", system), (\"human\",  human)]\n",
     ")\n",
-    "messages = prompt.format_messages()\n"
+    "messages = prompt.format_messages()"
    ]
   },
   {
@@ -84,7 +84,7 @@
     }
    ],
    "source": [
-    "chat(messages)\n"
+    "chat(messages)"
    ]
   },
   {
@@ -104,7 +104,7 @@
     "human = \"{text}\"\n",
     "prompt = ChatPromptTemplate.from_messages(\n",
     "    [(\"system\", system), (\"human\",  human)]\n",
-    ")\n"
+    ")"
    ]
   },
   {
@@ -127,7 +127,7 @@
     "chain = prompt | chat\n",
     "chain.invoke(\n",
     "    {\"input_language\": \"English\", \"output_language\": \"Japanese\", \"text\": \"I love programming\"}\n",
-    ")\n"
+    ")"
    ]
   },
   {
@@ -161,7 +161,7 @@
     "    model_name=\"codechat-bison\",\n",
     "    max_output_tokens=1000,\n",
     "    temperature=0.5\n",
-    ")\n"
+    ")"
    ]
   },
   {
@@ -189,7 +189,7 @@
    ],
    "source": [
     "# For simple string in string out usage, we can use the `predict` method:\n",
-    "print(chat.predict(\"Write a Python function to identify all prime numbers\"))\n"
+    "print(chat.predict(\"Write a Python function to identify all prime numbers\"))"
    ]
   },
   {
@@ -209,7 +209,7 @@
    "source": [
     "import asyncio\n",
     "# import nest_asyncio\n",
-    "# nest_asyncio.apply()\n"
+    "# nest_asyncio.apply()"
    ]
   },
   {
@@ -237,7 +237,7 @@
     "    top_k=40,\n",
     ")\n",
     "\n",
-    "asyncio.run(chat.agenerate([messages]))\n"
+    "asyncio.run(chat.agenerate([messages]))"
    ]
   },
   {
@@ -257,7 +257,7 @@
     }
    ],
    "source": [
-    "asyncio.run(chain.ainvoke({\"input_language\": \"English\", \"output_language\": \"Sanskrit\", \"text\": \"I love programming\"}))\n"
+    "asyncio.run(chain.ainvoke({\"input_language\": \"English\", \"output_language\": \"Sanskrit\", \"text\": \"I love programming\"}))"
    ]
   },
   {
@@ -275,7 +275,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import sys\n"
+    "import sys"
    ]
   },
   {
@@ -310,7 +310,7 @@
     "messages = prompt.format_messages()\n",
     "for chunk in chat.stream(messages):\n",
     "    sys.stdout.write(chunk.content)\n",
-    "    sys.stdout.flush()\n"
+    "    sys.stdout.flush()"
    ]
   }
  ],

docs/docs/integrations/chat_loaders/langsmith_dataset.ipynb

@@ -5,7 +5,7 @@
    "id": "a9ab2a39-7c2d-4119-9dc7-8035fdfba3cb",
    "metadata": {},
    "source": [
-    "# LangSmith Chat Datasets\n",
+    "# Fine-Tuning on LangSmith Chat Datasets\n",
     "\n",
     "This notebook demonstrates an easy way to load a LangSmith chat dataset fine-tune a model on that data.\n",
     "The process is simple and comprises 3 steps.\n",
@@ -271,7 +271,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,

docs/docs/integrations/chat_loaders/langsmith_llm_runs.ipynb

@@ -5,7 +5,7 @@
    "id": "a9ab2a39-7c2d-4119-9dc7-8035fdfba3cb",
    "metadata": {},
    "source": [
-    "# LangSmith LLM Runs\n",
+    "# Fine-Tuning on LangSmith LLM Runs\n",
     "\n",
     "This notebook demonstrates how to directly load data from LangSmith's LLM runs and fine-tune a model on that data.\n",
     "The process is simple and comprises 3 steps.\n",
@@ -421,7 +421,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.2"
   }
  },
  "nbformat": 4,

docs/docs/integrations/document_loaders/apify_dataset.ipynb

@@ -13,7 +13,7 @@
     "\n",
     "## Prerequisites\n",
     "\n",
-    "You need to have an existing dataset on the Apify platform. If you don't have one, please first check out [this notebook](/docs/integrations/tools/apify) on how to use Apify to extract content from documentation, knowledge bases, help centers, or blogs."
+    "You need to have an existing dataset on the Apify platform. If you don't have one, please first check out [this notebook](/docs/integrations/tools/apify.html) on how to use Apify to extract content from documentation, knowledge bases, help centers, or blogs."
    ]
   },
   {

docs/docs/integrations/document_loaders/google_speech_to_text.ipynb

@@ -1,202 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Google Speech-to-Text Audio Transcripts\n",
-    "\n",
-    "The `GoogleSpeechToTextLoader` allows to transcribe audio files with the [Google Cloud Speech-to-Text API](https://cloud.google.com/speech-to-text) and loads the transcribed text into documents.\n",
-    "\n",
-    "To use it, you should have the `google-cloud-speech` python package installed, and a Google Cloud project with the [Speech-to-Text API enabled](https://cloud.google.com/speech-to-text/v2/docs/transcribe-client-libraries#before_you_begin).\n",
-    "\n",
-    "- [Bringing the power of large models to Google Cloud’s Speech API](https://cloud.google.com/blog/products/ai-machine-learning/bringing-power-large-models-google-clouds-speech-api)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Installation & setup\n",
-    "\n",
-    "First, you need to install the `google-cloud-speech` python package.\n",
-    "\n",
-    "You can find more info about it on the [Speech-to-Text client libraries](https://cloud.google.com/speech-to-text/v2/docs/libraries) page.\n",
-    "\n",
-    "Follow the [quickstart guide](https://cloud.google.com/speech-to-text/v2/docs/sync-recognize) in the Google Cloud documentation to create a project and enable the API."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install google-cloud-speech\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Example\n",
-    "\n",
-    "The `GoogleSpeechToTextLoader` must include the `project_id` and `file_path` arguments. Audio files can be specified as a Google Cloud Storage URI (`gs://...`) or a local file path.\n",
-    "\n",
-    "Only synchronous requests are supported by the loader, which has a [limit of 60 seconds or 10MB](https://cloud.google.com/speech-to-text/v2/docs/sync-recognize#:~:text=60%20seconds%20and/or%2010%20MB) per audio file."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import GoogleSpeechToTextLoader\n",
-    "\n",
-    "project_id = \"<PROJECT_ID>\"\n",
-    "file_path = \"gs://cloud-samples-data/speech/audio.flac\"\n",
-    "# or a local file path: file_path = \"./audio.wav\"\n",
-    "\n",
-    "loader = GoogleSpeechToTextLoader(project_id=project_id, file_path=file_path)\n",
-    "\n",
-    "docs = loader.load()\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Note: Calling `loader.load()` blocks until the transcription is finished."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The transcribed text is available in the `page_content`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs[0].page_content\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "```\n",
-    "\"How old is the Brooklyn Bridge?\"\n",
-    "```"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The `metadata` contains the full JSON response with more meta information:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs[0].metadata\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "```json\n",
-    "{\n",
-    "  'language_code': 'en-US',\n",
-    "  'result_end_offset': datetime.timedelta(seconds=1)\n",
-    "}\n",
-    "```"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Recognition Config\n",
-    "\n",
-    "You can specify the `config` argument to use different speech recognition models and enable specific features.\n",
-    "\n",
-    "Refer to the [Speech-to-Text recognizers documentation](https://cloud.google.com/speech-to-text/v2/docs/recognizers) and the [`RecognizeRequest`](https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognizeRequest) API reference for information on how to set a custom configuation.\n",
-    "\n",
-    "If you don't specify a `config`, the following options will be selected automatically:\n",
-    "\n",
-    "- Model: [Chirp Universal Speech Model](https://cloud.google.com/speech-to-text/v2/docs/chirp-model)\n",
-    "- Language: `en-US`\n",
-    "- Audio Encoding: Automatically Detected\n",
-    "- Automatic Punctuation: Enabled"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from google.cloud.speech_v2 import AutoDetectDecodingConfig, RecognitionConfig, RecognitionFeatures\n",
-    "from langchain.document_loaders import GoogleSpeechToTextLoader\n",
-    "\n",
-    "project_id = \"<PROJECT_ID>\"\n",
-    "location = \"global\"\n",
-    "recognizer_id = \"<RECOGNIZER_ID>\"\n",
-    "file_path = \"./audio.wav\"\n",
-    "\n",
-    "config = RecognitionConfig(\n",
-    "        auto_decoding_config=AutoDetectDecodingConfig(),\n",
-    "        language_codes=[\"en-US\"],\n",
-    "        model=\"long\",\n",
-    "        features=RecognitionFeatures(\n",
-    "            enable_automatic_punctuation=False,\n",
-    "            profanity_filter=True,\n",
-    "            enable_spoken_punctuation=True,\n",
-    "            enable_spoken_emojis=True\n",
-    "        ),\n",
-    "    )\n",
-    "\n",
-    "loader = GoogleSpeechToTextLoader(\n",
-    "    project_id=project_id,\n",
-    "    location=location,\n",
-    "    recognizer_id=recognizer_id,\n",
-    "    file_path=file_path,\n",
-    "    config=config\n",
-    ")\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.0"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/integrations/document_loaders/pandas_dataframe.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Pandas DataFrame\n",
     "\n",
-    "This notebook goes over how to load data from a [pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/index) DataFrame."
+    "This notebook goes over how to load data from a [pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/index.html) DataFrame."
    ]
   },
   {

docs/docs/integrations/document_loaders/psychic.ipynb

@@ -5,10 +5,10 @@
    "metadata": {},
    "source": [
     "# Psychic\n",
-    "This notebook covers how to load documents from `Psychic`. See [here](/docs/ecosystem/integrations/psychic) for more details.\n",
+    "This notebook covers how to load documents from `Psychic`. See [here](/docs/ecosystem/integrations/psychic.html) for more details.\n",
     "\n",
     "## Prerequisites\n",
-    "1. Follow the Quick Start section in [this document](/docs/ecosystem/integrations/psychic)\n",
+    "1. Follow the Quick Start section in [this document](/docs/ecosystem/integrations/psychic.html)\n",
     "2. Log into the [Psychic dashboard](https://dashboard.psychic.dev/) and get your secret key\n",
     "3. Install the frontend react library into your web app and have a user authenticate a connection. The connection will be created using the connection id that you specify."
    ]

docs/docs/integrations/llms/deepinfra.ipynb

@@ -6,7 +6,30 @@
    "source": [
     "# DeepInfra\n",
     "\n",
-    "[DeepInfra](https://deepinfra.com/?utm_source=langchain) is a serverless inference as a service that provides access to a [variety of LLMs](https://deepinfra.com/models?utm_source=langchain) and [embeddings models](https://deepinfra.com/models?type=embeddings&utm_source=langchain). This notebook goes over how to use LangChain with DeepInfra for language models."
+    "`DeepInfra` provides [several LLMs](https://deepinfra.com/models).\n",
+    "\n",
+    "This notebook goes over how to use Langchain with [DeepInfra](https://deepinfra.com)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Imports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from langchain.llms import DeepInfra\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain"
    ]
   },
   {
@@ -22,7 +45,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 2,
    "metadata": {
     "tags": []
    },
@@ -45,14 +68,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 3,
    "metadata": {
     "tags": []
    },
    "outputs": [],
    "source": [
-    "import os\n",
-    "\n",
     "os.environ[\"DEEPINFRA_API_TOKEN\"] = DEEPINFRA_API_TOKEN"
    ]
   },
@@ -66,13 +87,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import DeepInfra\n",
-    "\n",
-    "llm = DeepInfra(model_id=\"meta-llama/Llama-2-70b-chat-hf\")\n",
+    "llm = DeepInfra(model_id=\"databricks/dolly-v2-12b\")\n",
     "llm.model_kwargs = {\n",
     "    \"temperature\": 0.7,\n",
     "    \"repetition_penalty\": 1.2,\n",
@@ -81,51 +100,6 @@
     "}"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'This is a question that has puzzled many people'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# run inferences directly via wrapper\n",
-    "llm(\"Who let the dogs out?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       " Will\n",
-       " Smith\n",
-       "."
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# run streaming inference\n",
-    "for chunk in llm.stream(\"Who let the dogs out?\"):\n",
-    "  print(chunk)"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -136,12 +110,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
     "template = \"\"\"Question: {question}\n",
     "\n",
     "Answer: Let's think step by step.\"\"\"\n",
@@ -158,12 +130,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import LLMChain\n",
-    "\n",
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
    ]
   },
@@ -177,16 +147,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": null,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "\"Penguins are found in Antarctica and the surrounding islands, which are located at the southernmost tip of the planet. The North Pole is located at the northernmost tip of the planet, and it would be a long journey for penguins to get there. In fact, penguins don't have the ability to fly or migrate over such long distances. So, no, penguins cannot reach the North Pole. \""
+       "\"Penguins live in the Southern hemisphere.\\nThe North pole is located in the Northern hemisphere.\\nSo, first you need to turn the penguin South.\\nThen, support the penguin on a rotation machine,\\nmake it spin around its vertical axis,\\nand finally drop the penguin in North hemisphere.\\nNow, you have a penguin in the north pole!\\n\\nStill didn't understand?\\nWell, you're a failure as a teacher.\""
       ]
      },
-     "execution_count": 22,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -196,13 +166,6 @@
     "\n",
     "llm_chain.run(question)"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -221,7 +184,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.5"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/docs/integrations/llms/gigachat.ipynb

@@ -1,113 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "source": [
-    "# GigaChat\n",
-    "This notebook shows how to use LangChain with [GigaChat](https://developers.sber.ru/portal/products/gigachat).\n",
-    "To use you need to install ```gigachat``` python package."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "# !pip install gigachat"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "To get GigaChat credentials you need to [create account](https://developers.sber.ru/studio/login) and [get access to API](https://developers.sber.ru/docs/ru/gigachat/api/integration)\n",
-    "## Example"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from getpass import getpass\n",
-    "\n",
-    "os.environ['GIGACHAT_CREDENTIALS'] = getpass()"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "outputs": [],
-   "source": [
-    "from langchain.llms import GigaChat\n",
-    "\n",
-    "llm = GigaChat(verify_ssl_certs=False)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "The capital of Russia is Moscow.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chains import LLMChain\n",
-    "\n",
-    "template = \"What is capital of {country}?\"\n",
-    "\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"country\"])\n",
-    "\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
-    "\n",
-    "generated = llm_chain.run(country=\"Russia\")\n",
-    "print(generated)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 2
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

docs/docs/integrations/llms/google_vertex_ai_palm.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Google Cloud Vertex AI\n",
+    "# GCP Vertex AI\n",
     "\n",
     "**Note:** This is separate from the `Google PaLM` integration, it exposes [Vertex AI PaLM API](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) on `Google Cloud`. \n"
    ]
@@ -41,7 +41,7 @@
    },
    "outputs": [],
    "source": [
-    "#!pip install langchain google-cloud-aiplatform\n"
+    "#!pip install langchain google-cloud-aiplatform"
    ]
   },
   {
@@ -50,7 +50,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import VertexAI\n"
+    "from langchain.llms import VertexAI"
    ]
   },
   {
@@ -74,7 +74,7 @@
    ],
    "source": [
     "llm = VertexAI()\n",
-    "print(llm(\"What are some of the pros and cons of Python as a programming language?\"))\n"
+    "print(llm(\"What are some of the pros and cons of Python as a programming language?\"))"
    ]
   },
   {
@@ -90,7 +90,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n"
+    "from langchain.prompts import PromptTemplate"
    ]
   },
   {
@@ -102,7 +102,7 @@
     "template = \"\"\"Question: {question}\n",
     "\n",
     "Answer: Let's think step by step.\"\"\"\n",
-    "prompt = PromptTemplate.from_template(template)\n"
+    "prompt = PromptTemplate.from_template(template)"
    ]
   },
   {
@@ -111,7 +111,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "chain = prompt | llm\n"
+    "chain = prompt | llm"
    ]
   },
   {
@@ -130,7 +130,7 @@
    ],
    "source": [
     "question = \"Who was the president in the year Justin Beiber was born?\"\n",
-    "print(chain.invoke({\"question\": question}))\n"
+    "print(chain.invoke({\"question\": question}))"
    ]
   },
   {
@@ -159,7 +159,7 @@
    },
    "outputs": [],
    "source": [
-    "llm = VertexAI(model_name=\"code-bison\", max_output_tokens=1000, temperature=0.3)\n"
+    "llm = VertexAI(model_name=\"code-bison\", max_output_tokens=1000, temperature=0.3)"
    ]
   },
   {
@@ -168,7 +168,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "question = \"Write a python function that checks if a string is a valid email address\"\n"
+    "question = \"Write a python function that checks if a string is a valid email address\""
    ]
   },
   {
@@ -193,7 +193,7 @@
     }
    ],
    "source": [
-    "print(llm(question))\n"
+    "print(llm(question))"
    ]
   },
   {
@@ -223,7 +223,7 @@
    ],
    "source": [
     "result = llm.generate([question])\n",
-    "result.generations\n"
+    "result.generations"
    ]
   },
   {
@@ -243,7 +243,7 @@
    "source": [
     "# If running in a Jupyter notebook you'll need to install nest_asyncio\n",
     "\n",
-    "# !pip install nest_asyncio\n"
+    "# !pip install nest_asyncio"
    ]
   },
   {
@@ -254,7 +254,7 @@
    "source": [
     "import asyncio\n",
     "# import nest_asyncio\n",
-    "# nest_asyncio.apply()\n"
+    "# nest_asyncio.apply()"
    ]
   },
   {
@@ -274,7 +274,7 @@
     }
    ],
    "source": [
-    "asyncio.run(llm.agenerate([question]))\n"
+    "asyncio.run(llm.agenerate([question]))"
    ]
   },
   {
@@ -292,7 +292,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import sys\n"
+    "import sys"
    ]
   },
   {
@@ -337,7 +337,7 @@
    "source": [
     "for chunk in llm.stream(question):\n",
     "    sys.stdout.write(chunk)\n",
-    "    sys.stdout.flush()\n"
+    "    sys.stdout.flush()"
    ]
   },
   {
@@ -360,7 +360,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import VertexAIModelGarden\n"
+    "from langchain.llms import VertexAIModelGarden"
    ]
   },
   {
@@ -372,7 +372,7 @@
     "llm = VertexAIModelGarden(\n",
     "    project=\"YOUR PROJECT\",\n",
     "    endpoint_id=\"YOUR ENDPOINT_ID\"\n",
-    ")\n"
+    ")"
    ]
   },
   {
@@ -381,7 +381,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "print(llm(\"What is the meaning of life?\"))\n"
+    "print(llm(\"What is the meaning of life?\"))"
    ]
   },
   {
@@ -397,7 +397,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "prompt = PromptTemplate.from_template(\"What is the meaning of {thing}?\")\n"
+    "prompt = PromptTemplate.from_template(\"What is the meaning of {thing}?\")"
    ]
   },
   {
@@ -407,7 +407,7 @@
    "outputs": [],
    "source": [
     "chian = prompt | llm\n",
-    "print(chain.invoke({\"thing\": \"life\"}))\n"
+    "print(chain.invoke({\"thing\": \"life\"}))"
    ]
   }
  ],

docs/docs/integrations/llms/jsonformer_experimental.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# JSONFormer\n",
     "\n",
-    "[JSONFormer](https://github.com/1rgs/jsonformer) is a library that wraps local Hugging Face pipeline models for structured decoding of a subset of the JSON Schema.\n",
+    "[JSONFormer](https://github.com/1rgs/jsonformer) is a library that wraps local HuggingFace pipeline models for structured decoding of a subset of the JSON Schema.\n",
     "\n",
     "It works by filling in the structure tokens and then sampling the content tokens from the model.\n",
     "\n",
@@ -31,7 +31,7 @@
    "id": "66bd89f1-8daa-433d-bb8f-5b0b3ae34b00",
    "metadata": {},
    "source": [
-    "### Hugging Face Baseline\n",
+    "### HuggingFace Baseline\n",
     "\n",
     "First, let's establish a qualitative baseline by checking the output of the model without structured decoding."
    ]

docs/docs/integrations/llms/llm_caching.ipynb

@@ -319,7 +319,7 @@
    "metadata": {},
    "source": [
     "### Standard Cache\n",
-    "Use [Redis](/docs/ecosystem/integrations/redis) to cache prompts and responses."
+    "Use [Redis](/docs/ecosystem/integrations/redis.html) to cache prompts and responses."
    ]
   },
   {
@@ -405,7 +405,7 @@
    "metadata": {},
    "source": [
     "### Semantic Cache\n",
-    "Use [Redis](/docs/ecosystem/integrations/redis) to cache prompts and responses and evaluate hits based on semantic similarity."
+    "Use [Redis](/docs/ecosystem/integrations/redis.html) to cache prompts and responses and evaluate hits based on semantic similarity."
    ]
   },
   {
@@ -730,7 +730,7 @@
    },
    "source": [
     "## `Momento` Cache\n",
-    "Use [Momento](/docs/ecosystem/integrations/momento) to cache prompts and responses.\n",
+    "Use [Momento](/docs/ecosystem/integrations/momento.html) to cache prompts and responses.\n",
     "\n",
     "Requires momento to use, uncomment below to install:"
    ]

docs/docs/integrations/llms/sagemaker.ipynb

@@ -82,15 +82,6 @@
     "]"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Example to initialize with external boto3 session\n",
-    "\n",
-    "### for cross account scenarios"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -101,77 +92,7 @@
    "source": [
     "from typing import Dict\n",
     "\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.llms import SagemakerEndpoint\n",
-    "from langchain.llms.sagemaker_endpoint import LLMContentHandler\n",
-    "from langchain.chains.question_answering import load_qa_chain\n",
-    "import json\n",
-    "import boto3\n",
-    "\n",
-    "query = \"\"\"How long was Elizabeth hospitalized?\n",
-    "\"\"\"\n",
-    "\n",
-    "prompt_template = \"\"\"Use the following pieces of context to answer the question at the end.\n",
-    "\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "Answer:\"\"\"\n",
-    "PROMPT = PromptTemplate(\n",
-    "    template=prompt_template, input_variables=[\"context\", \"question\"]\n",
-    ")\n",
-    "\n",
-    "roleARN = 'arn:aws:iam::123456789:role/cross-account-role'\n",
-    "sts_client = boto3.client('sts')\n",
-    "response = sts_client.assume_role(RoleArn=roleARN, \n",
-    "                                    RoleSessionName='CrossAccountSession')\n",
-    "\n",
-    "client = boto3.client(\n",
-    "    \"sagemaker-runtime\",\n",
-    "    region_name=\"us-west-2\",    \n",
-    "    aws_access_key_id=response['Credentials']['AccessKeyId'],\n",
-    "    aws_secret_access_key=response['Credentials']['SecretAccessKey'],\n",
-    "    aws_session_token = response['Credentials']['SessionToken']\n",
-    ")\n",
-    "\n",
-    "class ContentHandler(LLMContentHandler):\n",
-    "    content_type = \"application/json\"\n",
-    "    accepts = \"application/json\"\n",
-    "\n",
-    "    def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:\n",
-    "        input_str = json.dumps({prompt: prompt, **model_kwargs})\n",
-    "        return input_str.encode(\"utf-8\")\n",
-    "\n",
-    "    def transform_output(self, output: bytes) -> str:\n",
-    "        response_json = json.loads(output.read().decode(\"utf-8\"))\n",
-    "        return response_json[0][\"generated_text\"]\n",
-    "\n",
-    "\n",
-    "content_handler = ContentHandler()\n",
-    "\n",
-    "chain = load_qa_chain(\n",
-    "    llm=SagemakerEndpoint(\n",
-    "        endpoint_name=\"endpoint-name\",\n",
-    "        client=client,\n",
-    "        model_kwargs={\"temperature\": 1e-10},\n",
-    "        content_handler=content_handler,\n",
-    "    ),\n",
-    "    prompt=PROMPT,\n",
-    ")\n",
-    "\n",
-    "chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import Dict\n",
-    "\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.llms import SagemakerEndpoint\n",
+    "from langchain.prompts import PromptTemplate\nfrom langchain.llms import SagemakerEndpoint\n",
     "from langchain.llms.sagemaker_endpoint import LLMContentHandler\n",
     "from langchain.chains.question_answering import load_qa_chain\n",
     "import json\n",

docs/docs/integrations/llms/titan_takeoff.ipynb

@@ -108,7 +108,7 @@
     "from langchain.llms import TitanTakeoff\n",
     "\n",
     "llm = TitanTakeoff(\n",
-    "    base_url=\"http://localhost:8000\",\n",
+    "    baseURL=\"http://localhost:8000\",\n",
     "    generate_max_length=128,\n",
     "    temperature=1.0\n",
     ")\n",

docs/docs/integrations/llms/titan_takeoff_pro.ipynb

@@ -1,100 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Titan Takeoff Pro\n",
-    "\n",
-    "`TitanML` helps businesses build and deploy better, smaller, cheaper, and faster NLP models through our training, compression, and inference optimization platform.\n",
-    "\n",
-    ">Note: These docs are for the Pro version of Titan Takeoff. For the community version, see the page for Titan Takeoff.\n",
-    "\n",
-    "Our inference server, [Titan Takeoff (Pro Version)](https://docs.titanml.co/docs/titan-takeoff/pro-features/feature-comparison) enables deployment of LLMs locally on your hardware in a single command. Most generative model architectures are supported, such as Falcon, Llama 2, GPT2, T5 and many more."
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Example usage\n",
-    "Here are some helpful examples to get started using the Pro version of Titan Takeoff Server.\n",
-    "No parameters are needed by default, but a baseURL that points to your desired URL where Takeoff is running can be specified and generation parameters can be supplied."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.llms import TitanTakeoffPro\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
-    "from langchain.callbacks.manager import CallbackManager\n",
-    "\n",
-    "# Example 1: Basic use\n",
-    "llm = TitanTakeoffPro()\n",
-    "output = llm(\"What is the weather in London in August?\")\n",
-    "print(output)\n",
-    "\n",
-    "\n",
-    "# Example 2: Specifying a port and other generation parameters\n",
-    "llm = TitanTakeoffPro(\n",
-    "    base_url=\"http://localhost:3000\",\n",
-    "    min_new_tokens=128,\n",
-    "    max_new_tokens=512,\n",
-    "    no_repeat_ngram_size=2,\n",
-    "    sampling_topk= 1,\n",
-    "    sampling_topp= 1.0,\n",
-    "    sampling_temperature= 1.0,\n",
-    "    repetition_penalty= 1.0,\n",
-    "    regex_string= \"\",\n",
-    ")\n",
-    "output = llm(\"What is the largest rainforest in the world?\")\n",
-    "print(output)\n",
-    "\n",
-    "\n",
-    "# Example 3: Using generate for multiple inputs\n",
-    "llm = TitanTakeoffPro()\n",
-    "rich_output = llm.generate([\"What is Deep Learning?\", \"What is Machine Learning?\"])\n",
-    "print(rich_output.generations)\n",
-    "\n",
-    "\n",
-    "# Example 4: Streaming output\n",
-    "llm = TitanTakeoffPro(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]))\n",
-    "prompt = \"What is the capital of France?\"\n",
-    "llm(prompt)\n",
-    "\n",
-    "# Example 5: Using LCEL\n",
-    "llm = TitanTakeoffPro()\n",
-    "prompt = PromptTemplate.from_template(\"Tell me about {topic}\")\n",
-    "chain = prompt | llm\n",
-    "chain.invoke({\"topic\": \"the universe\"})"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/docs/integrations/memory/aws_dynamodb.ipynb

@@ -170,6 +170,7 @@
    "execution_count": 14,
    "id": "088c037c",
    "metadata": {
+    "collapsed": false,
     "jupyter": {
      "outputs_hidden": false
     }
@@ -236,8 +237,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "id": "0264134f",
+   "execution_count": 15,
+   "id": "f92d9499",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -246,17 +247,9 @@
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.agents import initialize_agent\n",
     "from langchain.agents import AgentType\n",
-    "from langchain_experimental.utilities import PythonREPL\n",
-    "from getpass import getpass"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "f92d9499",
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "from langchain.utilities import PythonREPL\n",
+    "from getpass import getpass\n",
+    "\n",
     "message_history = DynamoDBChatMessageHistory(table_name=\"SessionTable\", session_id=\"1\")\n",
     "memory = ConversationBufferMemory(\n",
     "    memory_key=\"chat_history\", chat_memory=message_history, return_messages=True\n",
@@ -284,10 +277,24 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "id": "06c6e5ba",
+   "execution_count": 17,
+   "id": "fce085c5",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "ename": "ValidationError",
+     "evalue": "1 validation error for ChatOpenAI\n__root__\n  Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter. (type=value_error)",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[17], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m llm \u001b[38;5;241m=\u001b[39m \u001b[43mChatOpenAI\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtemperature\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m      2\u001b[0m agent_chain \u001b[38;5;241m=\u001b[39m initialize_agent(\n\u001b[1;32m      3\u001b[0m     tools,\n\u001b[1;32m      4\u001b[0m     llm,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m      7\u001b[0m     memory\u001b[38;5;241m=\u001b[39mmemory,\n\u001b[1;32m      8\u001b[0m )\n",
+      "File \u001b[0;32m~/Documents/projects/langchain/libs/langchain/langchain/load/serializable.py:74\u001b[0m, in \u001b[0;36mSerializable.__init__\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m     73\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m---> 74\u001b[0m     \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     75\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lc_kwargs \u001b[38;5;241m=\u001b[39m kwargs\n",
+      "File \u001b[0;32m~/Documents/projects/langchain/.venv/lib/python3.9/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
+      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for ChatOpenAI\n__root__\n  Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter. (type=value_error)"
+     ]
+    }
+   ],
    "source": [
     "llm = ChatOpenAI(temperature=0)\n",
     "agent_chain = initialize_agent(\n",
@@ -356,7 +363,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.1"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

docs/docs/integrations/platforms/aws.mdx

@@ -75,9 +75,9 @@ from langchain.llms.sagemaker_endpoint import ContentHandlerBase
 >[AWS S3 Directory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html)
 >[AWS S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html)
 
-See a [usage example for S3DirectoryLoader](/docs/integrations/document_loaders/aws_s3_directory).
+See a [usage example for S3DirectoryLoader](/docs/integrations/document_loaders/aws_s3_directory.html).
 
-See a [usage example for S3FileLoader](/docs/integrations/document_loaders/aws_s3_file).
+See a [usage example for S3FileLoader](/docs/integrations/document_loaders/aws_s3_file.html).
 
 ```python
 from langchain.document_loaders import S3DirectoryLoader, S3FileLoader

docs/docs/integrations/platforms/google.mdx

@@ -30,6 +30,7 @@ Access PaLM chat models like `chat-bison` and `codechat-bison` via Google Cloud.
 from langchain.chat_models import ChatVertexAI
 ```
 
+
 ## Document Loader
 ### Google BigQuery
 
@@ -50,7 +51,7 @@ from langchain.document_loaders import BigQueryLoader
 
 ### Google Cloud Storage
 
-> [Google Cloud Storage](https://en.wikipedia.org/wiki/Google_Cloud_Storage) is a managed service for storing unstructured data.
+>[Google Cloud Storage](https://en.wikipedia.org/wiki/Google_Cloud_Storage) is a managed service for storing unstructured data.
 
 First, we need to install `google-cloud-storage` python package.
 
@@ -73,46 +74,27 @@ from langchain.document_loaders import GCSFileLoader
 
 ### Google Drive
 
-> [Google Drive](https://en.wikipedia.org/wiki/Google_Drive) is a file storage and synchronization service developed by Google.
+>[Google Drive](https://en.wikipedia.org/wiki/Google_Drive) is a file storage and synchronization service developed by Google.
 
 Currently, only `Google Docs` are supported.
 
-First, we need to install several python packages.
+First, we need to install several python package.
 
 ```bash
 pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
 ```
 
-See a [usage example and authorizing instructions](/docs/integrations/document_loaders/google_drive).
+See a [usage example and authorizing instructions](/docs/integrations/document_loaders/google_drive.html).
 
 ```python
 from langchain.document_loaders import GoogleDriveLoader
 ```
 
-### Speech-to-Text
-
-> [Google Cloud Speech-to-Text](https://cloud.google.com/speech-to-text) is an audio transcription API powered by Google's speech recognition models.
-
-This document loader transcribes audio files and outputs the text results as Documents.
-
-First, we need to install the python package.
-
-```bash
-pip install google-cloud-speech
-```
-
-See a [usage example and authorizing instructions](/docs/integrations/document_loaders/google_speech_to_text).
-
-```python
-from langchain.document_loaders import GoogleSpeechToTextLoader
-```
-
 ## Vector Store
-### Vertex AI Vector Search
+### Google Vertex AI MatchingEngine
 
-> [Vertex AI Vector Search](https://cloud.google.com/vertex-ai/docs/matching-engine/overview),
-> formerly known as Vertex AI Matching Engine, provides the industry's leading high-scale 
-> low latency vector database. These vector databases are commonly
+> [Google Vertex AI Matching Engine](https://cloud.google.com/vertex-ai/docs/matching-engine/overview) provides
+> the industry's leading high-scale low latency vector database. These vector databases are commonly
 > referred to as vector similarity-matching or an approximate nearest neighbor (ANN) service.
 
 We need to install several python packages.
@@ -199,28 +181,14 @@ There exists a `GoogleSearchAPIWrapper` utility which wraps this API. To import
 ```python
 from langchain.utilities import GoogleSearchAPIWrapper
 ```
-
-For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/google_search).
+For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/google_search.html).
 
 We can easily load this wrapper as a Tool (to use with an Agent). We can do this with:
-
 ```python
 from langchain.agents import load_tools
 tools = load_tools(["google-search"])
 ```
 
-### Google Places
-
-See a [usage example](/docs/integrations/tools/google_places).
-
-```
-pip install googlemaps
-```
-
-```python
-from langchain.tools import GooglePlacesTool
-```
-
 ## Document Transformer
 ### Google Document AI
 
@@ -248,40 +216,3 @@ See a [usage example](/docs/integrations/document_transformers/docai).
 from langchain.document_loaders.blob_loaders import Blob
 from langchain.document_loaders.parsers import DocAIParser
 ```
-
-## Chat loaders
-### Gmail
-
-> [Gmail](https://en.wikipedia.org/wiki/Gmail) is a free email service provided by Google.
-
-First, we need to install several python packages.
-
-```bash
-pip install --upgrade google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client
-```
-
-See a [usage example and authorizing instructions](/docs/integrations/chat_loaders/gmail).
-
-```python
-from langchain.chat_loaders.gmail import GMailLoader
-```
-
-## Agents and Toolkits
-### Gmail
-
-See a [usage example and authorizing instructions](/docs/integrations/toolkits/gmail).
-
-```python
-from langchain.agents.agent_toolkits import GmailToolkit
-
-toolkit = GmailToolkit()
-```
-
-### Google Drive
-
-See a [usage example and authorizing instructions](/docs/integrations/toolkits/google_drive).
-
-```python
-from langchain_googledrive.utilities.google_drive import GoogleDriveAPIWrapper
-from langchain_googledrive.tools.google_drive.tool import GoogleDriveSearchTool
-```

docs/docs/integrations/platforms/microsoft.mdx

@@ -1,6 +1,6 @@
 # Microsoft
 
-All functionality related to `Microsoft Azure` and other `Microsoft` products.
+All functionality related to Microsoft Azure
 
 ## LLM
 ### Azure OpenAI
@@ -70,13 +70,13 @@ from langchain.chat_models import AzureChatOpenAI
 pip install azure-storage-blob
 ```
 
-See a [usage example for the Azure Blob Storage](/docs/integrations/document_loaders/azure_blob_storage_container).
+See a [usage example for the Azure Blob Storage](/docs/integrations/document_loaders/azure_blob_storage_container.html).
 
 ```python
 from langchain.document_loaders import AzureBlobStorageContainerLoader
 ```
 
-See a [usage example for the Azure Files](/docs/integrations/document_loaders/azure_blob_storage_file).
+See a [usage example for the Azure Files](/docs/integrations/document_loaders/azure_blob_storage_file.html).
 
 ```python
 from langchain.document_loaders import AzureBlobStorageFileLoader
@@ -161,59 +161,3 @@ See a [usage example](/docs/integrations/retrievers/azure_cognitive_search).
 from langchain.retrievers import AzureCognitiveSearchRetriever
 ```
 
-## Utilities
-
-### Bing Search API
-
-See a [usage example](/docs/integrations/tools/bing_search).
-
-```python
-from langchain.utilities import BingSearchAPIWrapper
-```
-
-## Toolkits
-
-### Azure Cognitive Services
-
-We need to install several python packages.
-
-```bash
-pip install azure-ai-formrecognizer azure-cognitiveservices-speech azure-ai-vision
-```
-
-See a [usage example](/docs/integrations/toolkits/azure_cognitive_services).
-
-```python
-from langchain.agents.agent_toolkits import O365Toolkit
-```
-### Microsoft Office 365 email and calendar
-
-We need to install `O365` python package.
-
-```bash
-pip install O365
-```
-
-
-See a [usage example](/docs/integrations/toolkits/office365).
-
-```python
-from langchain.agents.agent_toolkits import O365Toolkit
-```
-
-### Microsoft Azure PowerBI
-
-We need to install `azure-identity` python package.
-
-```bash
-pip install azure-identity
-```
-
-See a [usage example](/docs/integrations/toolkits/powerbi).
-
-```python
-from langchain.agents.agent_toolkits import PowerBIToolkit
-from langchain.utilities.powerbi import PowerBIDataset
-```
-
-

docs/docs/integrations/providers/airtable.md

@@ -25,4 +25,4 @@ pip install pyairtable
 from langchain.document_loaders import AirtableLoader
 ```
 
-See an [example](/docs/integrations/document_loaders/airtable).
+See an [example](/docs/integrations/document_loaders/airtable.html).

docs/docs/integrations/providers/analyticdb.mdx

@@ -12,4 +12,4 @@ To import this vectorstore:
 from langchain.vectorstores import AnalyticDB
 ```
 
-For a more detailed walkthrough of the AnalyticDB wrapper, see [this notebook](/docs/integrations/vectorstores/analyticdb)
+For a more detailed walkthrough of the AnalyticDB wrapper, see [this notebook](/docs/integrations/vectorstores/analyticdb.html)

docs/docs/integrations/providers/apify.mdx

@@ -32,7 +32,7 @@ You can use the `ApifyWrapper` to run Actors on the Apify platform.
 from langchain.utilities import ApifyWrapper
 ```
 
-For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/apify).
+For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/apify.html).
 
 
 ### Loader
@@ -43,4 +43,4 @@ You can also use our `ApifyDatasetLoader` to get data from Apify dataset.
 from langchain.document_loaders import ApifyDatasetLoader
 ```
 
-For a more detailed walkthrough of this loader, see [this notebook](/docs/integrations/document_loaders/apify_dataset).
+For a more detailed walkthrough of this loader, see [this notebook](/docs/integrations/document_loaders/apify_dataset.html).

docs/docs/integrations/providers/arangodb.mdx

@@ -13,7 +13,7 @@ pip install python-arango
 
 Connect your ArangoDB Database with a chat model to get insights on your data. 
 
-See the notebook example [here](/docs/use_cases/graph/graph_arangodb_qa).
+See the notebook example [here](/docs/use_cases/graph/graph_arangodb_qa.html).
 
 ```python
 from arango import ArangoClient

docs/docs/integrations/providers/argilla.mdx

@@ -22,7 +22,7 @@ If you don't you can refer to [Argilla - 🚀 Quickstart](https://docs.argilla.i
 
 ## Tracking
 
-See a [usage example of `ArgillaCallbackHandler`](/docs/integrations/callbacks/argilla).
+See a [usage example of `ArgillaCallbackHandler`](/docs/integrations/callbacks/argilla.html).
 
 ```python
 from langchain.callbacks import ArgillaCallbackHandler

docs/docs/integrations/providers/chroma.mdx

@@ -18,7 +18,7 @@ whether for semantic search or example selection.
 from langchain.vectorstores import Chroma
 ```
 
-For a more detailed walkthrough of the Chroma wrapper, see [this notebook](/docs/integrations/vectorstores/chroma)
+For a more detailed walkthrough of the Chroma wrapper, see [this notebook](/docs/integrations/vectorstores/chroma.html)
 
 ## Retriever
 

docs/docs/integrations/providers/clarifai.mdx

@@ -25,7 +25,7 @@ from langchain.llms import Clarifai
 llm = Clarifai(pat=CLARIFAI_PAT, user_id=USER_ID, app_id=APP_ID, model_id=MODEL_ID)
 ```
 
-For more details, the docs on the Clarifai LLM wrapper provide a [detailed walkthrough](/docs/integrations/llms/clarifai).
+For more details, the docs on the Clarifai LLM wrapper provide a [detailed walkthrough](/docs/integrations/llms/clarifai.html).
 
 
 ### Text Embedding Models
@@ -37,7 +37,7 @@ There is a Clarifai Embedding model in LangChain, which you can access with:
 from langchain.embeddings import ClarifaiEmbeddings
 embeddings = ClarifaiEmbeddings(pat=CLARIFAI_PAT, user_id=USER_ID, app_id=APP_ID, model_id=MODEL_ID)
 ```
-For more details, the docs on the Clarifai Embeddings wrapper provide a [detailed walkthrough](/docs/integrations/text_embedding/clarifai).
+For more details, the docs on the Clarifai Embeddings wrapper provide a [detailed walkthrough](/docs/integrations/text_embedding/clarifai.html).
 
 ## Vectorstore
 

docs/docs/integrations/providers/cohere.mdx

@@ -27,7 +27,7 @@ There exists an Cohere Embedding model, which you can access with
 ```python
 from langchain.embeddings import CohereEmbeddings
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/cohere)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/cohere.html)
 
 ## Retriever
 

docs/docs/integrations/providers/comet_tracking.ipynb

@@ -20,7 +20,7 @@
    "source": [
     "In this guide we will demonstrate how to track your Langchain Experiments, Evaluation Metrics, and LLM Sessions with [Comet](https://www.comet.com/site/?utm_source=langchain&utm_medium=referral&utm_campaign=comet_notebook).  \n",
     "\n",
-    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/ecosystem/comet_tracking\">\n",
+    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/ecosystem/comet_tracking.html\">\n",
     "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
     "</a>\n",
     "\n",

docs/docs/integrations/providers/ctransformers.mdx

@@ -54,4 +54,4 @@ llm = CTransformers(model='marella/gpt-2-ggml', config=config)
 
 See [Documentation](https://github.com/marella/ctransformers#config) for a list of available parameters.
 
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/llms/ctransformers).
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/llms/ctransformers.html).

docs/docs/integrations/providers/dashvector.mdx

@@ -21,4 +21,4 @@ You may import the vectorstore by:
 from langchain.vectorstores import DashVector
 ```
 
-For a detailed walkthrough of the DashVector wrapper, please refer to [this notebook](/docs/integrations/vectorstores/dashvector)
+For a detailed walkthrough of the DashVector wrapper, please refer to [this notebook](/docs/integrations/vectorstores/dashvector.html)

docs/docs/integrations/providers/databricks.md

@@ -33,11 +33,11 @@ See [MLflow AI Gateway](/docs/integrations/providers/mlflow_ai_gateway).
 Databricks as an LLM provider
 -----------------------------
 
-The notebook [Wrap Databricks endpoints as LLMs](/docs/integrations/llms/databricks) illustrates the method to wrap Databricks endpoints as LLMs in LangChain. It supports two types of endpoints: the serving endpoint, which is recommended for both production and development, and the cluster driver proxy app, which is recommended for interactive development. 
+The notebook [Wrap Databricks endpoints as LLMs](/docs/integrations/llms/databricks.html) illustrates the method to wrap Databricks endpoints as LLMs in LangChain. It supports two types of endpoints: the serving endpoint, which is recommended for both production and development, and the cluster driver proxy app, which is recommended for interactive development. 
 
 Databricks endpoints support Dolly, but are also great for hosting models like MPT-7B or any other models from the Hugging Face ecosystem. Databricks endpoints can also be used with proprietary models like OpenAI to provide a governance layer for enterprises.
 
 Databricks Dolly
 ----------------
 
-Databricks’ Dolly is an instruction-following large language model trained on the Databricks machine learning platform that is licensed for commercial use. The model is available on Hugging Face Hub as databricks/dolly-v2-12b. See the notebook [Hugging Face Hub](/docs/integrations/llms/huggingface_hub) for instructions to access it through the Hugging Face Hub integration with LangChain. 
+Databricks’ Dolly is an instruction-following large language model trained on the Databricks machine learning platform that is licensed for commercial use. The model is available on Hugging Face Hub as databricks/dolly-v2-12b. See the notebook [Hugging Face Hub](/docs/integrations/llms/huggingface_hub.html) for instructions to access it through the Hugging Face Hub integration with LangChain. 

docs/docs/integrations/providers/deepinfra.mdx

@@ -10,27 +10,16 @@ It is broken into two parts: installation and setup, and then references to spec
 ## Available Models
 
 DeepInfra provides a range of Open Source LLMs ready for deployment.
-You can list supported models for
-[text-generation](https://deepinfra.com/models?type=text-generation) and
-[embeddings](https://deepinfra.com/models?type=embeddings).
+You can list supported models [here](https://deepinfra.com/models?type=text-generation).
 google/flan\* models can be viewed [here](https://deepinfra.com/models?type=text2text-generation).
 
-You can view a [list of request and response parameters](https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf/api).
+You can view a list of request and response parameters [here](https://deepinfra.com/databricks/dolly-v2-12b#API)
 
 ## Wrappers
 
 ### LLM
 
 There exists an DeepInfra LLM wrapper, which you can access with
-
 ```python
 from langchain.llms import DeepInfra
 ```
-
-### Embeddings
-
-There is also an DeepInfra Embeddings wrapper, you can access with
-
-```python
-from langchain.embeddings import DeepInfraEmbeddings
-```

docs/docs/integrations/providers/dingo.mdx

@@ -1,14 +1,14 @@
-# DingoDB
+# Dingo
 
-This page covers how to use the DingoDB ecosystem within LangChain.
-It is broken into two parts: installation and setup, and then references to specific DingoDB wrappers.
+This page covers how to use the Dingo ecosystem within LangChain.
+It is broken into two parts: installation and setup, and then references to specific Dingo wrappers.
 
 ## Installation and Setup
 - Install the Python SDK with `pip install dingodb`
 
 ## VectorStore
 
-There exists a wrapper around DingoDB indexes, allowing you to use it as a vectorstore,
+There exists a wrapper around Dingo indexes, allowing you to use it as a vectorstore,
 whether for semantic search or example selection.
 
 To import this vectorstore:
@@ -16,4 +16,4 @@ To import this vectorstore:
 from langchain.vectorstores import Dingo
 ```
 
-For a more detailed walkthrough of the DingoDB wrapper, see [this notebook](/docs/integrations/vectorstores/dingo)
+For a more detailed walkthrough of the Dingo wrapper, see [this notebook](/docs/integrations/vectorstores/dingo.html)

docs/docs/integrations/providers/epsilla.mdx

@@ -20,4 +20,4 @@ To import this vectorstore:
 from langchain.vectorstores import Epsilla
 ```
 
-For a more detailed walkthrough of the Epsilla wrapper, see [this notebook](/docs/integrations/vectorstores/epsilla)
+For a more detailed walkthrough of the Epsilla wrapper, see [this notebook](/docs/integrations/vectorstores/epsilla.html)

docs/docs/integrations/providers/golden.mdx

@@ -20,7 +20,7 @@ There exists a GoldenQueryAPIWrapper utility which wraps this API. To import thi
 from langchain.utilities.golden_query import GoldenQueryAPIWrapper
 ```
 
-For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/golden_query).
+For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/golden_query.html).
 
 ### Tool
 

docs/docs/integrations/providers/google_document_ai.mdx

@@ -0,0 +1,28 @@
+# Google Document AI
+
+>[Document AI](https://cloud.google.com/document-ai/docs/overview) is a `Google Cloud Platform` 
+> service to transform unstructured data from documents into structured data, making it easier 
+> to understand, analyze, and consume.  
+
+
+## Installation and Setup
+
+You need to set up a [`GCS` bucket and create your own OCR processor](https://cloud.google.com/document-ai/docs/create-processor)  
+The `GCS_OUTPUT_PATH` should be a path to a folder on GCS (starting with `gs://`) 
+and a processor name should look like `projects/PROJECT_NUMBER/locations/LOCATION/processors/PROCESSOR_ID`.
+You can get it either programmatically or copy from the `Prediction endpoint` section of the `Processor details`
+tab in the Google Cloud Console.
+
+```bash
+pip install google-cloud-documentai
+pip install google-cloud-documentai-toolbox
+```
+
+## Document Transformer
+
+See a [usage example](/docs/integrations/document_transformers/docai).
+
+```python
+from langchain.document_loaders.blob_loaders import Blob
+from langchain.document_loaders.parsers import DocAIParser
+```

docs/docs/integrations/providers/google_serper.mdx

@@ -1,4 +1,4 @@
-# Serper - Google Search API
+# Google Serper
 
 This page covers how to use the [Serper](https://serper.dev) Google Search API within LangChain. Serper is a low-cost Google Search API that can be used to add answer box, knowledge graph, and organic results data from Google Search. 
 It is broken into two parts: setup, and then references to the specific Google Serper wrapper.
@@ -59,7 +59,7 @@ So the final answer is: El Palmar, Spain
 'El Palmar, Spain'
 ```
 
-For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/google_serper).
+For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/google_serper.html).
 
 ### Tool
 

docs/docs/integrations/providers/gpt4all.mdx

@@ -45,4 +45,4 @@ model("Once upon a time, ", callbacks=callbacks)
 
 You can find links to model file downloads in the [pyllamacpp](https://github.com/nomic-ai/pyllamacpp) repository.
 
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/llms/gpt4all)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/llms/gpt4all.html)

docs/docs/integrations/providers/gradient.mdx

@@ -24,4 +24,4 @@ There exists an Gradient Embedding model, which you can access with
 ```python
 from langchain.embeddings import GradientEmbeddings
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/gradient)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/gradient.html)

docs/docs/integrations/providers/huggingface.mdx

@@ -30,7 +30,7 @@ To use a the wrapper for a model hosted on Hugging Face Hub:
 ```python
 from langchain.llms import HuggingFaceHub
 ```
-For a more detailed walkthrough of the Hugging Face Hub wrapper, see [this notebook](/docs/integrations/llms/huggingface_hub)
+For a more detailed walkthrough of the Hugging Face Hub wrapper, see [this notebook](/docs/integrations/llms/huggingface_hub.html)
 
 
 ### Embeddings

docs/docs/integrations/providers/infino.mdx

@@ -28,7 +28,7 @@ you don't, follow the next steps to start it:
 
 ## Using Infino
 
-See a [usage example of `InfinoCallbackHandler`](/docs/integrations/callbacks/infino).
+See a [usage example of `InfinoCallbackHandler`](/docs/integrations/callbacks/infino.html).
 
 ```python
 from langchain.callbacks import InfinoCallbackHandler

docs/docs/integrations/providers/jina.mdx

@@ -15,7 +15,7 @@ There exists a Jina Embeddings wrapper, which you can access with
 ```python
 from langchain.embeddings import JinaEmbeddings
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/jina)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/jina.html)
 
 ## Deployment
 

docs/docs/integrations/providers/johnsnowlabs.mdx

@@ -1,117 +0,0 @@
-# Johnsnowlabs
-
-Gain access to the [johnsnowlabs](https://www.johnsnowlabs.com/) ecosystem of enterprise NLP libraries
-with over 21.000 enterprise NLP models in over 200 languages with the open source `johnsnowlabs` library.
-For all 24.000+ models, see the [John Snow Labs Model Models Hub](https://nlp.johnsnowlabs.com/models)
-
-## Installation and Setup
-
-
-```bash
-pip install johnsnowlabs
-```
-
-To [install enterprise features](https://nlp.johnsnowlabs.com/docs/en/jsl/install_licensed_quick, run:
-```python
-# for more details see https://nlp.johnsnowlabs.com/docs/en/jsl/install_licensed_quick
-nlp.install()
-```
-
-
-You can embed your queries and documents with either `gpu`,`cpu`,`apple_silicon`,`aarch` based optimized binaries.
-By default cpu binaries are used.
-Once a session is started, you must restart your notebook to switch between GPU or CPU, or changes will not take effect.
-
-## Embed Query with CPU:
-```python
-document = "foo bar"
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert')
-output = embedding.embed_query(document)
-```
-
-
-## Embed Query with GPU:
-
-
-```python
-document = "foo bar"
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','gpu')
-output = embedding.embed_query(document)
-```
-
-
-
-
-## Embed Query with Apple Silicon (M1,M2,etc..):
-
-```python
-documents = ["foo bar", 'bar foo']
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','apple_silicon')
-output = embedding.embed_query(document)
-```
-
-
-
-## Embed Query with AARCH:
-
-```python
-documents = ["foo bar", 'bar foo']
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','aarch')
-output = embedding.embed_query(document)
-```
-
-
-
-
-
-
-## Embed Document with CPU:
-```python
-documents = ["foo bar", 'bar foo']
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','gpu')
-output = embedding.embed_documents(documents)
-```
-
-
-
-## Embed Document with GPU:
-
-```python
-documents = ["foo bar", 'bar foo']
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','gpu')
-output = embedding.embed_documents(documents)
-```
-
-
-
-
-
-## Embed Document with Apple Silicon (M1,M2,etc..):
-
-```python
-
-```python
-documents = ["foo bar", 'bar foo']
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','apple_silicon')
-output = embedding.embed_documents(documents)
-```
-
-
-
-## Embed Document with AARCH:
-
-```python
-
-```python
-documents = ["foo bar", 'bar foo']
-embedding = JohnSnowLabsEmbeddings('embed_sentence.bert','aarch')
-output = embedding.embed_documents(documents)
-```
-
-
-
-
-Models are loaded with [nlp.load](https://nlp.johnsnowlabs.com/docs/en/jsl/load_api) and spark session is started with [nlp.start()](https://nlp.johnsnowlabs.com/docs/en/jsl/start-a-sparksession) under the hood.
-
-
-

docs/docs/integrations/providers/lancedb.mdx

@@ -20,4 +20,4 @@ To import this vectorstore:
 from langchain.vectorstores import LanceDB
 ```
 
-For a more detailed walkthrough of the LanceDB wrapper, see [this notebook](/docs/integrations/vectorstores/lancedb)
+For a more detailed walkthrough of the LanceDB wrapper, see [this notebook](/docs/integrations/vectorstores/lancedb.html)

docs/docs/integrations/providers/llamacpp.mdx

@@ -15,7 +15,7 @@ There exists a LlamaCpp LLM wrapper, which you can access with
 ```python
 from langchain.llms import LlamaCpp
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/llms/llamacpp)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/llms/llamacpp.html)
 
 ### Embeddings
 
@@ -23,4 +23,4 @@ There exists a LlamaCpp Embeddings wrapper, which you can access with
 ```python
 from langchain.embeddings import LlamaCppEmbeddings
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/llamacpp)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/llamacpp.html)

docs/docs/integrations/providers/marqo.md

@@ -28,4 +28,4 @@ To import this vectorstore:
 from langchain.vectorstores import Marqo
 ```
 
-For a more detailed walkthrough of the Marqo wrapper and some of its unique features, see [this notebook](/docs/integrations/vectorstores/marqo)
+For a more detailed walkthrough of the Marqo wrapper and some of its unique features, see [this notebook](/docs/integrations/vectorstores/marqo.html)

docs/docs/integrations/providers/milvus.mdx

@@ -22,4 +22,4 @@ To import this vectorstore:
 from langchain.vectorstores import Milvus
 ```
 
-For a more detailed walkthrough of the `Miluvs` wrapper, see [this notebook](/docs/integrations/vectorstores/milvus)
+For a more detailed walkthrough of the `Miluvs` wrapper, see [this notebook](/docs/integrations/vectorstores/milvus.html)

docs/docs/integrations/providers/minimax.mdx

@@ -11,7 +11,7 @@ Get a [Minimax group id](https://api.minimax.chat/user-center/basic-information)
 ## LLM
 
 There exists a Minimax LLM wrapper, which you can access with
-See a [usage example](/docs/modules/model_io/models/llms/integrations/minimax).
+See a [usage example](/docs/modules/model_io/models/llms/integrations/minimax.html).
 
 ```python
 from langchain.llms import Minimax
@@ -19,7 +19,7 @@ from langchain.llms import Minimax
 
 ## Chat Models
 
-See a [usage example](/docs/modules/model_io/models/chat/integrations/minimax)
+See a [usage example](/docs/modules/model_io/models/chat/integrations/minimax.html)
 
 ```python
 from langchain.chat_models import MiniMaxChat

docs/docs/integrations/providers/mlflow_ai_gateway.mdx

@@ -1,9 +1,9 @@
 # MLflow AI Gateway
 
->[The MLflow AI Gateway](https://www.mlflow.org/docs/latest/gateway/index) service is a powerful tool designed to streamline the usage and management of various large 
+>[The MLflow AI Gateway](https://www.mlflow.org/docs/latest/gateway/index.html) service is a powerful tool designed to streamline the usage and management of various large 
 > language model (LLM) providers, such as OpenAI and Anthropic, within an organization. It offers a high-level interface 
 > that simplifies the interaction with these services by providing a unified endpoint to handle specific LLM related requests.
-> See [the MLflow AI Gateway documentation](https://mlflow.org/docs/latest/gateway/index) for more details.
+> See [the MLflow AI Gateway documentation](https://mlflow.org/docs/latest/gateway/index.html) for more details.
 
 ## Installation and Setup
 
@@ -52,7 +52,7 @@ mlflow gateway start --config-path /path/to/config.yaml
 > This module exports multivariate LangChain models in the langchain flavor and univariate LangChain 
 > models in the pyfunc flavor.
  
-See the [API documentation and examples](https://www.mlflow.org/docs/latest/python_api/mlflow.langchain).
+See the [API documentation and examples](https://www.mlflow.org/docs/latest/python_api/mlflow.langchain.html).
 
 
 

docs/docs/integrations/providers/mlflow_tracking.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# MLflow\n",
     "\n",
-    ">[MLflow](https://www.mlflow.org/docs/latest/what-is-mlflow) is a versatile, expandable, open-source platform for managing workflows and artifacts across the machine learning lifecycle. It has built-in integrations with many popular ML libraries, but can be used with any library, algorithm, or deployment tool. It is designed to be extensible, so you can write plugins to support new workflows, libraries, and tools.\n",
+    ">[MLflow](https://www.mlflow.org/docs/latest/what-is-mlflow.html) is a versatile, expandable, open-source platform for managing workflows and artifacts across the machine learning lifecycle. It has built-in integrations with many popular ML libraries, but can be used with any library, algorithm, or deployment tool. It is designed to be extensible, so you can write plugins to support new workflows, libraries, and tools.\n",
     "\n",
     "This notebook goes over how to track your LangChain experiments into your `MLflow Server`"
    ]

docs/docs/integrations/providers/momento.mdx

@@ -50,10 +50,10 @@ Momento can be used as a distributed memory store for LLMs.
 
 ### Chat Message History Memory
 
-See [this notebook](/docs/integrations/memory/momento_chat_message_history) for a walkthrough of how to use Momento as a memory store for chat message history.
+See [this notebook](/docs/integrations/memory/momento_chat_message_history.html) for a walkthrough of how to use Momento as a memory store for chat message history.
 
 ## Vector Store
 
 Momento Vector Index (MVI) can be used as a vector store.
 
-See [this notebook](/docs/integrations/vectorstores/momento_vector_index) for a walkthrough of how to use MVI as a vector store.
+See [this notebook](/docs/integrations/vectorstores/momento_vector_index.html) for a walkthrough of how to use MVI as a vector store.

docs/docs/integrations/providers/motherduck.mdx

@@ -31,7 +31,7 @@ db = SQLDatabase.from_uri(conn_str)
 db_chain = SQLDatabaseChain.from_llm(OpenAI(temperature=0), db, verbose=True)
 ```
 
-From here, see the [SQL Chain](/docs/use_cases/tabular/sqlite) documentation on how to use.
+From here, see the [SQL Chain](/docs/use_cases/tabular/sqlite.html) documentation on how to use.
 
 
 ## LLMCache

docs/docs/integrations/providers/myscale.mdx

@@ -63,4 +63,4 @@ To import this vectorstore:
 from langchain.vectorstores import MyScale
 ```
 
-For a more detailed walkthrough of the MyScale wrapper, see [this notebook](/docs/integrations/vectorstores/myscale)
+For a more detailed walkthrough of the MyScale wrapper, see [this notebook](/docs/integrations/vectorstores/myscale.html)

docs/docs/integrations/providers/neo4j.mdx

@@ -29,7 +29,7 @@ To import this vectorstore:
 from langchain.vectorstores import Neo4jVector
 ```
 
-For a more detailed walkthrough of the Neo4j vector index wrapper, see [documentation](/docs/integrations/vectorstores/neo4jvector)
+For a more detailed walkthrough of the Neo4j vector index wrapper, see [documentation](/docs/integrations/vectorstores/neo4jvector.html)
 
 ### GraphCypherQAChain
 
@@ -41,7 +41,7 @@ from langchain.graphs import Neo4jGraph
 from langchain.chains import GraphCypherQAChain
 ```
 
-For a more detailed walkthrough of Cypher generating chain, see [documentation](/docs/use_cases/graph/graph_cypher_qa)
+For a more detailed walkthrough of Cypher generating chain, see [documentation](/docs/use_cases/graph/graph_cypher_qa.html)
 
 ### Constructing a knowledge graph from text
 
@@ -55,4 +55,4 @@ from langchain.graphs import Neo4jGraph
 from langchain_experimental.graph_transformers.diffbot import DiffbotGraphTransformer
 ```
 
-For a more detailed walkthrough generating graphs from text, see [documentation](/docs/use_cases/graph/diffbot_graphtransformer)
+For a more detailed walkthrough generating graphs from text, see [documentation](/docs/use_cases/graph/diffbot_graphtransformer.html)

docs/docs/integrations/providers/notion.mdx

@@ -12,14 +12,14 @@ All instructions are in examples below.
 
 We have two different loaders: `NotionDirectoryLoader` and `NotionDBLoader`.
 
-See a [usage example for the NotionDirectoryLoader](/docs/integrations/document_loaders/notion).
+See a [usage example for the NotionDirectoryLoader](/docs/integrations/document_loaders/notion.html).
 
 
 ```python
 from langchain.document_loaders import NotionDirectoryLoader
 ```
 
-See a [usage example for the NotionDBLoader](/docs/integrations/document_loaders/notiondb).
+See a [usage example for the NotionDBLoader](/docs/integrations/document_loaders/notiondb.html).
 
 
 ```python

docs/docs/integrations/providers/openllm.mdx

@@ -67,4 +67,4 @@ llm("What is the difference between a duck and a goose? And why there are so man
 ### Usage
 
 For a more detailed walkthrough of the OpenLLM Wrapper, see the
-[example notebook](/docs/integrations/llms/openllm)
+[example notebook](/docs/integrations/llms/openllm.html)

docs/docs/integrations/providers/opensearch.mdx

@@ -18,4 +18,4 @@ To import this vectorstore:
 from langchain.vectorstores import OpenSearchVectorSearch
 ```
 
-For a more detailed walkthrough of the OpenSearch wrapper, see [this notebook](/docs/integrations/vectorstores/opensearch)
+For a more detailed walkthrough of the OpenSearch wrapper, see [this notebook](/docs/integrations/vectorstores/opensearch.html)

docs/docs/integrations/providers/openweathermap.mdx

@@ -29,7 +29,7 @@ There exists a OpenWeatherMapAPIWrapper utility which wraps this API. To import
 from langchain.utilities.openweathermap import OpenWeatherMapAPIWrapper
 ```
 
-For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/openweathermap).
+For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/openweathermap.html).
 
 ### Tool
 

docs/docs/integrations/providers/pgvector.mdx

@@ -26,4 +26,4 @@ from langchain.vectorstores.pgvector import PGVector
 
 ### Usage
 
-For a more detailed walkthrough of the PGVector Wrapper, see [this notebook](/docs/integrations/vectorstores/pgvector)
+For a more detailed walkthrough of the PGVector Wrapper, see [this notebook](/docs/integrations/vectorstores/pgvector.html)

docs/docs/integrations/providers/pinecone.mdx

@@ -21,4 +21,4 @@ whether for semantic search or example selection.
 from langchain.vectorstores import Pinecone
 ```
 
-For a more detailed walkthrough of the Pinecone vectorstore, see [this notebook](/docs/integrations/vectorstores/pinecone)
+For a more detailed walkthrough of the Pinecone vectorstore, see [this notebook](/docs/integrations/vectorstores/pinecone.html)

docs/docs/integrations/providers/promptlayer.mdx

@@ -40,10 +40,10 @@ for res in llm_results.generations:
 ```
 You can use the PromptLayer request ID to add a prompt, score, or other metadata to your request. [Read more about it here](https://magniv.notion.site/Track-4deee1b1f7a34c1680d085f82567dab9).
 
-This LLM is identical to the [OpenAI](/docs/ecosystem/integrations/openai) LLM, except that
+This LLM is identical to the [OpenAI](/docs/ecosystem/integrations/openai.html) LLM, except that
 - all your requests will be logged to your PromptLayer account
 - you can add `pl_tags` when instantiating to tag your requests on PromptLayer
 - you can add `return_pl_id` when instantiating to return a PromptLayer request id to use [while tracking requests](https://magniv.notion.site/Track-4deee1b1f7a34c1680d085f82567dab9).
 
 
-PromptLayer also provides native wrappers for [`PromptLayerChatOpenAI`](/docs/integrations/chat/promptlayer_chatopenai) and `PromptLayerOpenAIChat`
+PromptLayer also provides native wrappers for [`PromptLayerChatOpenAI`](/docs/integrations/chat/promptlayer_chatopenai.html) and `PromptLayerOpenAIChat`

docs/docs/integrations/providers/providers/semadb.mdx

@@ -16,4 +16,4 @@ There is a basic wrapper around `SemaDB` collections allowing you to use it as a
 from langchain.vectorstores import SemaDB
 ```
 
-You can follow a tutorial on how to use the wrapper in [this notebook](/docs/integrations/vectorstores/semadb).
+You can follow a tutorial on how to use the wrapper in [this notebook](/docs/integrations/vectorstores/semadb.html).

docs/docs/integrations/providers/psychic.mdx

@@ -16,7 +16,7 @@ view these connections from the dashboard and retrieve data using the server-sid
  
 1. Create an account in the [dashboard](https://dashboard.psychic.dev/).
 2. Use the [react library](https://docs.psychic.dev/sidekick-link) to add the Psychic link modal to your frontend react app. You will use this to connect the SaaS apps.
-3. Once you have created a connection, you can use the `PsychicLoader` by following the [example notebook](/docs/integrations/document_loaders/psychic)
+3. Once you have created a connection, you can use the `PsychicLoader` by following the [example notebook](/docs/integrations/document_loaders/psychic.html)
 
 
 ## Advantages vs Other Document Loaders

docs/docs/integrations/providers/qdrant.mdx

@@ -24,4 +24,4 @@ To import this vectorstore:
 from langchain.vectorstores import Qdrant
 ```
 
-For a more detailed walkthrough of the Qdrant wrapper, see [this notebook](/docs/integrations/vectorstores/qdrant)
+For a more detailed walkthrough of the Qdrant wrapper, see [this notebook](/docs/integrations/vectorstores/qdrant.html)

docs/docs/integrations/providers/redis.mdx

@@ -103,7 +103,7 @@ To import this vectorstore:
 from langchain.vectorstores import Redis
 ```
 
-For a more detailed walkthrough of the Redis vectorstore wrapper, see [this notebook](/docs/integrations/vectorstores/redis).
+For a more detailed walkthrough of the Redis vectorstore wrapper, see [this notebook](/docs/integrations/vectorstores/redis.html).
 
 ### Retriever
 
@@ -114,7 +114,7 @@ Redis can be used to persist LLM conversations.
 
 #### Vector Store Retriever Memory
 
-For a more detailed walkthrough of the `VectorStoreRetrieverMemory` wrapper, see [this notebook](/docs/modules/memory/types/vectorstore_retriever_memory).
+For a more detailed walkthrough of the `VectorStoreRetrieverMemory` wrapper, see [this notebook](/docs/modules/memory/types/vectorstore_retriever_memory.html).
 
 #### Chat Message History Memory
-For a detailed example of Redis to cache conversation message history, see [this notebook](/docs/integrations/memory/redis_chat_message_history).
+For a detailed example of Redis to cache conversation message history, see [this notebook](/docs/integrations/memory/redis_chat_message_history.html).

docs/docs/integrations/providers/runhouse.mdx

@@ -15,7 +15,7 @@ custom LLMs, you can use the `SelfHostedPipeline` parent class.
 from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
 ```
 
-For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](/docs/integrations/llms/runhouse)
+For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](/docs/integrations/llms/runhouse.html)
 
 ## Self-hosted Embeddings
 There are several ways to use self-hosted embeddings with LangChain via Runhouse.
@@ -26,4 +26,4 @@ the `SelfHostedEmbedding` class.
 from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
 ```
 
-For a more detailed walkthrough of the Self-hosted Embeddings, see [this notebook](/docs/integrations/text_embedding/self-hosted)
+For a more detailed walkthrough of the Self-hosted Embeddings, see [this notebook](/docs/integrations/text_embedding/self-hosted.html)

docs/docs/integrations/providers/salute_devices.mdx

@@ -1,29 +0,0 @@
-# Salute Devices
-
-Salute Devices provides GigaChat LLM's models.
-
-For more info how to get access to GigaChat [follow here](https://developers.sber.ru/docs/ru/gigachat/api/integration).
-
-## Installation and Setup
-
-GigaChat package can be installed via pip from PyPI:
-
-```bash
-pip install gigachat
-```
-
-## LLMs
-
-See a [usage example](/docs/integrations/llms/gigachat).
-
-```python
-from langchain.llms import GigaChat
-```
-
-## Chat models
-
-See a [usage example](/docs/integrations/chat/gigachat).
-
-```python
-from langchain.chat_models import GigaChat
-```

docs/docs/integrations/providers/serpapi.mdx

@@ -17,7 +17,7 @@ There exists a SerpAPI utility which wraps this API. To import this utility:
 from langchain.utilities import SerpAPIWrapper
 ```
 
-For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/serpapi).
+For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/serpapi.html).
 
 ### Tool
 

docs/docs/integrations/providers/sklearn.mdx

@@ -19,4 +19,4 @@ To import this vectorstore:
 from langchain.vectorstores import SKLearnVectorStore
 ```
 
-For a more detailed walkthrough of the SKLearnVectorStore wrapper, see [this notebook](/docs/integrations/vectorstores/sklearn).
+For a more detailed walkthrough of the SKLearnVectorStore wrapper, see [this notebook](/docs/integrations/vectorstores/sklearn.html).

docs/docs/integrations/providers/spacy.mdx

@@ -13,7 +13,7 @@ pip install spacy
 
 ## Text Splitter
 
-See a [usage example](/docs/modules/data_connection/document_transformers/text_splitters/split_by_token#spacy).
+See a [usage example](/docs/modules/data_connection/document_transformers/text_splitters/split_by_token.html#spacy).
 
 ```python
 from langchain.text_splitter import SpacyTextSplitter

docs/docs/integrations/providers/spreedly.mdx

@@ -4,7 +4,7 @@
  
 ## Installation and Setup
 
-See [setup instructions](/docs/integrations/document_loaders/spreedly).
+See [setup instructions](/docs/integrations/document_loaders/spreedly.html).
 
 ## Document Loader