langservehub template cli

.github/CODE_OF_CONDUCT.md

@@ -1,132 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-We as members, contributors, and leaders pledge to make participation in our
-community a harassment-free experience for everyone, regardless of age, body
-size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
-nationality, personal appearance, race, caste, color, religion, or sexual
-identity and orientation.
-
-We pledge to act and interact in ways that contribute to an open, welcoming,
-diverse, inclusive, and healthy community.
-
-## Our Standards
-
-Examples of behavior that contributes to a positive environment for our
-community include:
-
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
-  and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the overall
-  community
-
-Examples of unacceptable behavior include:
-
-* The use of sexualized language or imagery, and sexual attention or advances of
-  any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email address,
-  without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-## Enforcement Responsibilities
-
-Community leaders are responsible for clarifying and enforcing our standards of
-acceptable behavior and will take appropriate and fair corrective action in
-response to any behavior that they deem inappropriate, threatening, offensive,
-or harmful.
-
-Community leaders have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions that are
-not aligned to this Code of Conduct, and will communicate reasons for moderation
-decisions when appropriate.
-
-## Scope
-
-This Code of Conduct applies within all community spaces, and also applies when
-an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
-posting via an official social media account, or acting as an appointed
-representative at an online or offline event.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported to the community leaders responsible for enforcement at
-conduct@langchain.dev.
-All complaints will be reviewed and investigated promptly and fairly.
-
-All community leaders are obligated to respect the privacy and security of the
-reporter of any incident.
-
-## Enforcement Guidelines
-
-Community leaders will follow these Community Impact Guidelines in determining
-the consequences for any action they deem in violation of this Code of Conduct:
-
-### 1. Correction
-
-**Community Impact**: Use of inappropriate language or other behavior deemed
-unprofessional or unwelcome in the community.
-
-**Consequence**: A private, written warning from community leaders, providing
-clarity around the nature of the violation and an explanation of why the
-behavior was inappropriate. A public apology may be requested.
-
-### 2. Warning
-
-**Community Impact**: A violation through a single incident or series of
-actions.
-
-**Consequence**: A warning with consequences for continued behavior. No
-interaction with the people involved, including unsolicited interaction with
-those enforcing the Code of Conduct, for a specified period of time. This
-includes avoiding interactions in community spaces as well as external channels
-like social media. Violating these terms may lead to a temporary or permanent
-ban.
-
-### 3. Temporary Ban
-
-**Community Impact**: A serious violation of community standards, including
-sustained inappropriate behavior.
-
-**Consequence**: A temporary ban from any sort of interaction or public
-communication with the community for a specified period of time. No public or
-private interaction with the people involved, including unsolicited interaction
-with those enforcing the Code of Conduct, is allowed during this period.
-Violating these terms may lead to a permanent ban.
-
-### 4. Permanent Ban
-
-**Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior, harassment of an
-individual, or aggression toward or disparagement of classes of individuals.
-
-**Consequence**: A permanent ban from any sort of public interaction within the
-community.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 2.1, available at
-[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
-
-Community Impact Guidelines were inspired by
-[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
-
-For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
-[https://www.contributor-covenant.org/translations][translations].
-
-[homepage]: https://www.contributor-covenant.org
-[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
-[Mozilla CoC]: https://github.com/mozilla/diversity
-[FAQ]: https://www.contributor-covenant.org/faq
-[translations]: https://www.contributor-covenant.org/translations

.github/CONTRIBUTING.md

@@ -1,7 +1,7 @@
 # Contributing to LangChain
 
 Hi there! Thank you for even being interested in contributing to LangChain.
-As an open-source project in a rapidly developing field, we are extremely open
+As an open source project in a rapidly developing field, we are extremely open
 to contributions, whether they be in the form of new features, improved infra, better documentation, or bug fixes.
 
 ## 🗺️ Guidelines
@@ -14,7 +14,7 @@ Please do not try to push directly to this repo unless you are a maintainer.
 Please follow the checked-in pull request template when opening pull requests. Note related issues and tag relevant
 maintainers.
 
-Pull requests cannot land without passing the formatting, linting, and testing checks first. See [Testing](#testing) and
+Pull requests cannot land without passing the formatting, linting and testing checks first. See [Testing](#testing) and
 [Formatting and Linting](#formatting-and-linting) for how to run these checks locally.
 
 It's essential that we maintain great documentation and testing. If you:
@@ -79,7 +79,7 @@ There are two separate projects in this repository:
 - `langchain`: core langchain code, abstractions, and use cases
 - `langchain.experimental`: see the [Experimental README](../libs/experimental/README.md) for more information.
 
-Each of these has its own development environment. Docs are run from the top-level makefile, but development
+Each of these has their own development environment. Docs are run from the top-level makefile, but development
 is split across separate test & release flows.
 
 For this quickstart, start with langchain core:
@@ -289,13 +289,6 @@ make docs_linkcheck
 make api_docs_linkcheck
 ```
 
-### Verify Documentation changes
-
-After pushing documentation changes to the repository, you can preview and verify that the changes are 
-what you wanted by clicking the `View deployment` or `Visit Preview` buttons on the pull request `Conversation` page. 
-This will take you to a preview of the documentation changes.
-This preview is created by [Vercel](https://vercel.com/docs/getting-started-with-vercel).
-
 ## 🏭 Release Process
 
 As of now, LangChain has an ad hoc release process: releases are cut with high frequency by

.github/workflows/_compile_integration_test.yml

@@ -1,57 +0,0 @@
-name: compile-integration-test
-
-on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-
-env:
-  POETRY_VERSION: "1.6.1"
-
-jobs:
-  build:
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version:
-          - "3.8"
-          - "3.9"
-          - "3.10"
-          - "3.11"
-    name: Python ${{ matrix.python-version }}
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: ${{ matrix.python-version }}
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: compile-integration
-
-      - name: Install integration dependencies
-        shell: bash
-        run: poetry install --with=test_integration
-
-      - name: Check integration tests compile
-        shell: bash
-        run: poetry run pytest -m compile tests/integration_tests
-
-      - name: Ensure the tests did not create any additional files
-        shell: bash
-        run: |
-          set -eu
-
-          STATUS="$(git status)"
-          echo "$STATUS"
-
-          # grep will exit non-zero if the target message isn't found,
-          # and `set -e` above will cause the step to fail.
-          echo "$STATUS" | grep 'nothing to commit, working tree clean'

.github/workflows/_lint.yml

@@ -34,7 +34,7 @@ jobs:
           - "3.8"
           - "3.11"
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
         with:
           # Fetch the last FETCH_DEPTH commits, so the mtime-changing script
           # can accurately set the mtimes of files modified in the last FETCH_DEPTH commits.

.github/workflows/_pydantic_compatibility.yml

@@ -26,7 +26,7 @@ jobs:
           - "3.11"
     name: Pydantic v1/v2 compatibility - Python ${{ matrix.python-version }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
         uses: "./.github/actions/poetry_setup"

.github/workflows/_release.yml

@@ -30,7 +30,7 @@ jobs:
       run:
         working-directory: ${{ inputs.working-directory }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
         uses: "./.github/actions/poetry_setup"

.github/workflows/_test.yml

@@ -26,7 +26,7 @@ jobs:
           - "3.11"
     name: Python ${{ matrix.python-version }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
         uses: "./.github/actions/poetry_setup"
@@ -44,14 +44,6 @@ jobs:
         shell: bash
         run: make test
 
-      - name: Install integration dependencies
-        shell: bash
-        run: poetry install --with=test_integration
-
-      - name: Check integration tests compile
-        shell: bash
-        run: poetry run pytest -m compile tests/integration_tests
-
       - name: Ensure the tests did not create any additional files
         shell: bash
         run: |

.github/workflows/_test_release.yml

@@ -1,50 +0,0 @@
-name: test-release
-
-on:
-  workflow_call:
-    inputs:
-      working-directory:
-        required: true
-        type: string
-        description: "From which folder this pipeline executes"
-
-env:
-  POETRY_VERSION: "1.6.1"
-
-jobs:
-  publish_to_test_pypi:
-    runs-on: ubuntu-latest
-    permissions:
-      # This permission is used for trusted publishing:
-      # https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/
-      #
-      # Trusted publishing has to also be configured on PyPI for each package:
-      # https://docs.pypi.org/trusted-publishers/adding-a-publisher/
-      id-token: write
-    defaults:
-      run:
-        working-directory: ${{ inputs.working-directory }}
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
-        uses: "./.github/actions/poetry_setup"
-        with:
-          python-version: "3.10"
-          poetry-version: ${{ env.POETRY_VERSION }}
-          working-directory: ${{ inputs.working-directory }}
-          cache-key: release
-
-      - name: Build project for distribution
-        run: poetry build
-      - name: Check Version
-        id: check-version
-        run: |
-          echo version=$(poetry version --short) >> $GITHUB_OUTPUT
-      - name: Publish package to TestPyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          repository-url: https://test.pypi.org/legacy/
-          packages-dir: ${{ inputs.working-directory }}/dist/
-          verbose: true
-          print-hash: true

.github/workflows/codespell.yml

@@ -17,7 +17,7 @@ jobs:
 
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@v3
 
       - name: Install Dependencies
         run: |

.github/workflows/langchain_ci.yml

@@ -44,13 +44,6 @@ jobs:
       working-directory: libs/langchain
     secrets: inherit
 
-  compile-integration-tests:
-    uses:
-      ./.github/workflows/_compile_integration_test.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit
-
   pydantic-compatibility:
     uses:
       ./.github/workflows/_pydantic_compatibility.yml
@@ -72,7 +65,7 @@ jobs:
           - "3.11"
     name: Python ${{ matrix.python-version }} extended tests
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
         uses: "./.github/actions/poetry_setup"

.github/workflows/langchain_experimental_ci.yml

@@ -44,13 +44,6 @@ jobs:
       working-directory: libs/experimental
     secrets: inherit
 
-  compile-integration-tests:
-    uses:
-      ./.github/workflows/_compile_integration_test.yml
-    with:
-      working-directory: libs/experimental
-    secrets: inherit
-
   # It's possible that langchain-experimental works fine with the latest *published* langchain,
   # but is broken with the langchain on `master`.
   #
@@ -69,7 +62,7 @@ jobs:
           - "3.11"
     name: test with unpublished langchain - Python ${{ matrix.python-version }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
         uses: "./.github/actions/poetry_setup"
@@ -104,7 +97,7 @@ jobs:
           - "3.11"
     name: Python ${{ matrix.python-version }} extended tests
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }} + Poetry ${{ env.POETRY_VERSION }}
         uses: "./.github/actions/poetry_setup"

.github/workflows/langchain_experimental_test_release.yml

@@ -1,13 +0,0 @@
----
-name: Experimental Test Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_test_release.yml
-    with:
-      working-directory: libs/experimental
-    secrets: inherit

.github/workflows/langchain_test_release.yml

@@ -1,13 +0,0 @@
----
-name: Test Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_test_release.yml
-    with:
-      working-directory: libs/langchain
-    secrets: inherit

.github/workflows/scheduled_test.yml

@@ -24,7 +24,7 @@ jobs:
           - "3.11"
     name: Python ${{ matrix.python-version }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v3
 
       - name: Set up Python ${{ matrix.python-version }}
         uses: "./.github/actions/poetry_setup"

.gitignore

@@ -177,4 +177,3 @@ docs/api_reference/*/
 docs/docs/build
 docs/docs/node_modules
 docs/docs/yarn.lock
-_dist

.readthedocs.yaml

@@ -9,14 +9,9 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.11"
-  commands:
-      - python -mvirtualenv $READTHEDOCS_VIRTUALENV_PATH
-      - python -m pip install --upgrade --no-cache-dir pip setuptools
-      - python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext
-      - python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt
+  jobs:
+    pre_build:
       - python docs/api_reference/create_api_rst.py
-      - cat docs/api_reference/conf.py
-      - python -m sphinx -T -E -b html -d _build/doctrees -c docs/api_reference docs/api_reference $READTHEDOCS_OUTPUT/html -j auto
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:

README.md

@@ -93,7 +93,7 @@ Memory refers to persisting state between calls of a chain/agent. LangChain prov
 
 **🧐 Evaluation:**
 
-[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is by using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.
+[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.
 
 For more information on these concepts, please see our [full documentation](https://python.langchain.com).
 

cookbook/LLaMA2_sql_chat.ipynb

@@ -1,400 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "fc935871-7640-41c6-b798-58514d860fe0",
-   "metadata": {},
-   "source": [
-    "## LLaMA2 chat with SQL\n",
-    "\n",
-    "Open source, local LLMs are great to consider for any application that demands data privacy.\n",
-    "\n",
-    "SQL is one good example. \n",
-    "\n",
-    "This cookbook shows how to perform text-to-SQL using various local versions of LLaMA2 run locally.\n",
-    "\n",
-    "## Packages"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "81adcf8b-395a-4f02-8749-ac976942b446",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "! pip install langchain replicate"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8e13ed66-300b-4a23-b8ac-44df68ee4733",
-   "metadata": {},
-   "source": [
-    "## LLM\n",
-    "\n",
-    "There are a few ways to access LLaMA2.\n",
-    "\n",
-    "To run locally, we use Ollama.ai. \n",
-    "\n",
-    "See [here](https://python.langchain.com/docs/integrations/chat/ollama) for details on installation and setup.\n",
-    "\n",
-    "Also, see [here](https://python.langchain.com/docs/guides/local_llms) for our full guide on local LLMs.\n",
-    " \n",
-    "To use an external API, which is not private, we can use Replicate."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "6a75a5c6-34ee-4ab9-a664-d9b432d812ee",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Init param `input` is deprecated, please use `model_kwargs` instead.\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Local \n",
-    "from langchain.chat_models import ChatOllama\n",
-    "llama2_chat = ChatOllama(model=\"llama2:13b-chat\")\n",
-    "llama2_code = ChatOllama(model=\"codellama:7b-instruct\")\n",
-    "\n",
-    "# API\n",
-    "from getpass import getpass\n",
-    "from langchain.llms import Replicate\n",
-    "# REPLICATE_API_TOKEN = getpass()\n",
-    "# os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN\n",
-    "replicate_id = \"meta/llama-2-13b-chat:f4e2de70d66816a838a89eeeb621910adffb0dd0baba3976c96980970978018d\"\n",
-    "llama2_chat_replicate = Replicate(\n",
-    "    model=replicate_id,\n",
-    "    input={\"temperature\": 0.01, \n",
-    "           \"max_length\": 500, \n",
-    "           \"top_p\": 1}\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "ce96f7ea-b3d5-44e1-9fa5-a79e04a9e1fb",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Simply set the LLM we want to use\n",
-    "llm = llama2_chat"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "80222165-f353-4e35-a123-5f70fd70c6c8",
-   "metadata": {},
-   "source": [
-    "## DB\n",
-    "\n",
-    "Connect to a SQLite DB.\n",
-    "\n",
-    "To create this particular DB, you can use the code and follow the steps shown [here](https://github.com/facebookresearch/llama-recipes/blob/main/demo_apps/StructuredLlama.ipynb)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "025bdd82-3bb1-4948-bc7c-c3ccd94fd05c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.utilities import SQLDatabase\n",
-    "db = SQLDatabase.from_uri(\"sqlite:///nba_roster.db\", sample_rows_in_table_info= 0)\n",
-    "\n",
-    "def get_schema(_):\n",
-    "    return db.get_table_info()\n",
-    "\n",
-    "def run_query(query):\n",
-    "    return db.run(query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "654b3577-baa2-4e12-a393-f40e5db49ac7",
-   "metadata": {},
-   "source": [
-    "## Query a SQL DB \n",
-    "\n",
-    "Follow the runnables workflow [here](https://python.langchain.com/docs/expression_language/cookbook/sql_db)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "5a4933ea-d9c0-4b0a-8177-ba4490c6532b",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' SELECT \"Team\" FROM nba_roster WHERE \"NAME\" = \\'Klay Thompson\\';'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Prompt\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "template = \"\"\"Based on the table schema below, write a SQL query that would answer the user's question:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query:\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question, convert it to a SQL query. No pre-amble.\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "# Chain to query\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
-    "\n",
-    "sql_response = (\n",
-    "        RunnablePassthrough.assign(schema=get_schema)\n",
-    "        | prompt\n",
-    "        | llm.bind(stop=[\"\\nSQLResult:\"])\n",
-    "        | StrOutputParser()\n",
-    "    )\n",
-    "\n",
-    "sql_response.invoke({\"question\": \"What team is Klay Thompson on?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a0e9e2c8-9b88-4853-ac86-001bc6cc6695",
-   "metadata": {},
-   "source": [
-    "We can review the results:\n",
-    "\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/afa56a06-b4e2-469a-a60f-c1746e75e42b/r) LLaMA2-13 Replicate API\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/2d4ecc72-6b8f-4523-8f0b-ea95c6b54a1d/r) LLaMA2-13 local \n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "2a2825e3-c1b6-4f7d-b9c9-d9835de323bb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Based on the table schema and SQL query, there are 30 unique teams in the NBA.')"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Chain to answer\n",
-    "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query: {query}\n",
-    "SQL Response: {response}\"\"\"\n",
-    "prompt_response = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "full_chain = (\n",
-    "    RunnablePassthrough.assign(query=sql_response) \n",
-    "    | RunnablePassthrough.assign(\n",
-    "        schema=get_schema,\n",
-    "        response=lambda x: db.run(x[\"query\"]),\n",
-    "    )\n",
-    "    | prompt_response \n",
-    "    | llm\n",
-    ")\n",
-    "\n",
-    "full_chain.invoke({\"question\": \"How many unique teams are there?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ec17b3ee-6618-4681-b6df-089bbb5ffcd7",
-   "metadata": {},
-   "source": [
-    "We can review the results:\n",
-    "\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/10420721-746a-4806-8ecf-d6dc6399d739/r) LLaMA2-13 Replicate API\n",
-    "* [LangSmith trace](https://smith.langchain.com/public/5265ebab-0a22-4f37-936b-3300f2dfa1c1/r) LLaMA2-13 local "
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1e85381b-1edc-4bb3-a7bd-2ab23f81e54d",
-   "metadata": {},
-   "source": [
-    "## Chat with a SQL DB \n",
-    "\n",
-    "Next, we can add memory."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "1985aa1c-eb8f-4fb1-a54f-c8aa10744687",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' SELECT \"Team\" FROM nba_roster WHERE \"NAME\" = \\'Klay Thompson\\';'"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Prompt\n",
-    "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
-    "template = \"\"\"Based on the table schema below, write a SQL query that would answer the user's question:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query:\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question, convert it to a SQL query. No pre-amble.\"),\n",
-    "    MessagesPlaceholder(variable_name=\"history\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "memory = ConversationBufferMemory(return_messages=True)\n",
-    "\n",
-    "# Chain to query with memory \n",
-    "from langchain.schema.runnable import RunnableLambda\n",
-    "\n",
-    "sql_chain = (\n",
-    "    RunnablePassthrough.assign(\n",
-    "       schema=get_schema,\n",
-    "       history=RunnableLambda(lambda x: memory.load_memory_variables(x)[\"history\"])\n",
-    "    )| prompt\n",
-    "    | llm.bind(stop=[\"\\nSQLResult:\"])\n",
-    "    | StrOutputParser()\n",
-    ")\n",
-    "\n",
-    "def save(input_output):\n",
-    "    output = {\"output\": input_output.pop(\"output\")}\n",
-    "    memory.save_context(input_output, output)\n",
-    "    return output['output']\n",
-    "    \n",
-    "sql_response_memory = RunnablePassthrough.assign(output=sql_chain) | save\n",
-    "sql_response_memory.invoke({\"question\": \"What team is Klay Thompson on?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "0b45818a-1498-441d-b82d-23c29428c2bb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' SELECT \"SALARY\" FROM nba_roster WHERE \"NAME\" = \\'Klay Thompson\\';'"
-      ]
-     },
-     "execution_count": 20,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "sql_response_memory.invoke({\"question\": \"What is his salary?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "800a7a3b-f411-478b-af51-2310cd6e0425",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Sure! Here\\'s the natural language response based on the given input:\\n\\n\"Klay Thompson\\'s salary is $43,219,440.\"')"
-      ]
-     },
-     "execution_count": 21,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Chain to answer\n",
-    "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n",
-    "{schema}\n",
-    "\n",
-    "Question: {question}\n",
-    "SQL Query: {query}\n",
-    "SQL Response: {response}\"\"\"\n",
-    "prompt_response = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"Given an input question and SQL response, convert it to a natural langugae answer. No pre-amble.\"),\n",
-    "    (\"human\", template)\n",
-    "])\n",
-    "\n",
-    "full_chain = (\n",
-    "    RunnablePassthrough.assign(query=sql_response_memory) \n",
-    "    | RunnablePassthrough.assign(\n",
-    "        schema=get_schema,\n",
-    "        response=lambda x: db.run(x[\"query\"]),\n",
-    "    )\n",
-    "    | prompt_response \n",
-    "    | llm\n",
-    ")\n",
-    "\n",
-    "full_chain.invoke({\"question\": \"What is his salary?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b77fee61-f4da-4bb1-8285-14101e505518",
-   "metadata": {},
-   "source": [
-    "Here is the [trace](https://smith.langchain.com/public/54794d18-2337-4ce2-8b9f-3d8a2df89e51/r)."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.16"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/README.md

@@ -1,52 +0,0 @@
-# LangChain cookbook
-
-Example code for building applications with LangChain, with an emphasis on more applied and end-to-end examples than contained in the [main documentation](https://python.langchain.com).
-
-Notebook | Description
-:- | :-
-[LLaMA2_sql_chat.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/LLaMA2_sql_chat.ipynb) | Build a chat application that interacts with a sql database using an open source llm (llama2), specifically demonstrated on a sqlite database containing nba rosters.
-[Semi_Structured_RAG.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_Structured_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data, including text and tables, using unstructured for parsing, multi-vector retriever for storing, and lcel for implementing chains.
-[Semi_structured_and_multi_moda...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using unstructured for parsing, multi-vector retriever for storage and retrieval, and lcel for implementing chains.
-[Semi_structured_multi_modal_RA...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using various tools and methods such as unstructured for parsing, multi-vector retriever for storing, lcel for implementing chains, and open source language models like llama2, llava, and gpt4all.
-[autogpt/autogpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/autogpt.ipynb) | Implement autogpt, a language model, with langchain primitives such as llms, prompttemplates, vectorstores, embeddings, and tools.
-[autogpt/marathon_times.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/marathon_times.ipynb) | Implement autogpt for finding winning marathon times.
-[baby_agi.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi.ipynb) | Implement babyagi, an ai agent that can generate and execute tasks based on a given objective, with the flexibility to swap out specific vectorstores/model providers.
-[baby_agi_with_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi_with_agent.ipynb) | Swap out the execution chain in the babyagi notebook with an agent that has access to tools, aiming to obtain more reliable information.
-[camel_role_playing.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/camel_role_playing.ipynb) | Implement the camel framework for creating autonomous cooperative agents in large scale language models, using role-playing and inception prompting to guide chat agents towards task completion.
-[causal_program_aided_language_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/causal_program_aided_language_model.ipynb) | Implement the causal program-aided language (cpal) chain, which improves upon the program-aided language (pal) by incorporating causal structure to prevent hallucination in language models, particularly when dealing with complex narratives and math problems with nested dependencies.
-[code-analysis-deeplake.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/code-analysis-deeplake.ipynb) | Analyze its own code base with the help of gpt and activeloop's deep lake.
-[custom_agent_with_plugin_retri...](https://github.com/langchain-ai/langchain/tree/master/cookbook/custom_agent_with_plugin_retrieval.ipynb) | Build a custom agent that can interact with ai plugins by retrieving tools and creating natural language wrappers around openapi endpoints.
-[custom_agent_with_plugin_retri...](https://github.com/langchain-ai/langchain/tree/master/cookbook/custom_agent_with_plugin_retrieval_using_plugnplai.ipynb) | Build a custom agent with plugin retrieval functionality, utilizing ai plugins from the `plugnplai` directory.
-[databricks_sql_db.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/databricks_sql_db.ipynb) | Connect to databricks runtimes and databricks sql.
-[deeplake_semantic_search_over_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/deeplake_semantic_search_over_chat.ipynb) | Perform semantic search and question-answering over a group chat using activeloop's deep lake with gpt4.
-[elasticsearch_db_qa.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/elasticsearch_db_qa.ipynb) | Interact with elasticsearch analytics databases in natural language and build search queries via the elasticsearch dsl api.
-[forward_looking_retrieval_augm...](https://github.com/langchain-ai/langchain/tree/master/cookbook/forward_looking_retrieval_augmented_generation.ipynb) | Implement the forward-looking active retrieval augmented generation (flare) method, which generates answers to questions, identifies uncertain tokens, generates hypothetical questions based on these tokens, and retrieves relevant documents to continue generating the answer.
-[generative_agents_interactive_...](https://github.com/langchain-ai/langchain/tree/master/cookbook/generative_agents_interactive_simulacra_of_human_behavior.ipynb) | Implement a generative agent that simulates human behavior, based on a research paper, using a time-weighted memory object backed by a langchain retriever.
-[gymnasium_agent_simulation.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/gymnasium_agent_simulation.ipynb) | Create a simple agent-environment interaction loop in simulated environments like text-based games with gymnasium.
-[hugginggpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/hugginggpt.ipynb) | Implement hugginggpt, a system that connects language models like chatgpt with the machine learning community via hugging face.
-[hypothetical_document_embeddin...](https://github.com/langchain-ai/langchain/tree/master/cookbook/hypothetical_document_embeddings.ipynb) | Improve document indexing with hypothetical document embeddings (hyde), an embedding technique that generates and embeds hypothetical answers to queries.
-[learned_prompt_optimization.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/learned_prompt_optimization.ipynb) | Automatically enhance language model prompts by injecting specific terms using reinforcement learning, which can be used to personalize responses based on user preferences.
-[llm_bash.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_bash.ipynb) | Perform simple filesystem commands using language learning models (llms) and a bash process.
-[llm_checker.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_checker.ipynb) | Create a self-checking chain using the llmcheckerchain function.
-[llm_math.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_math.ipynb) | Solve complex word math problems using language models and python repls.
-[llm_summarization_checker.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_summarization_checker.ipynb) | Check the accuracy of text summaries, with the option to run the checker multiple times for improved results.
-[llm_symbolic_math.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/llm_symbolic_math.ipynb) | Solve algebraic equations with the help of llms (language learning models) and sympy, a python library for symbolic mathematics.
-[meta_prompt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/meta_prompt.ipynb) | Implement the meta-prompt concept, which is a method for building self-improving agents that reflect on their own performance and modify their instructions accordingly.
-[multi_modal_output_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multi_modal_output_agent.ipynb) | Generate multi-modal outputs, specifically images and text.
-[multi_player_dnd.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multi_player_dnd.ipynb) | Simulate multi-player dungeons & dragons games, with a custom function determining the speaking schedule of the agents.
-[multiagent_authoritarian.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multiagent_authoritarian.ipynb) | Implement a multi-agent simulation where a privileged agent controls the conversation, including deciding who speaks and when the conversation ends, in the context of a simulated news network.
-[multiagent_bidding.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/multiagent_bidding.ipynb) | Implement a multi-agent simulation where agents bid to speak, with the highest bidder speaking next, demonstrated through a fictitious presidential debate example.
-[myscale_vector_sql.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/myscale_vector_sql.ipynb) | Access and interact with the myscale integrated vector database, which can enhance the performance of language model (llm) applications.
-[openai_functions_retrieval_qa....](https://github.com/langchain-ai/langchain/tree/master/cookbook/openai_functions_retrieval_qa.ipynb) | Structure response output in a question answering system by incorporating openai functions into a retrieval pipeline.
-[petting_zoo.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/petting_zoo.ipynb) | Create multi-agent simulations with simulated environments using the petting zoo library.
-[plan_and_execute_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/plan_and_execute_agent.ipynb) | Create plan-and-execute agents that accomplish objectives by planning tasks with a language model (llm) and executing them with a separate agent.
-[press_releases.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/press_releases.ipynb) | Retrieve and query company press release data powered by [Kay.ai](https://kay.ai).
-[program_aided_language_model.i...](https://github.com/langchain-ai/langchain/tree/master/cookbook/program_aided_language_model.ipynb) | Implement program-aided language models as described in the provided research paper.
-[sales_agent_with_context.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/sales_agent_with_context.ipynb) | Implement a context-aware ai sales agent, salesgpt, that can have natural sales conversations, interact with other systems, and use a product knowledge base to discuss a company's offerings.
-[self_query_hotel_search.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/self_query_hotel_search.ipynb) | Build a hotel room search feature with self-querying retrieval, using a specific hotel recommendation dataset.
-[smart_llm.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/smart_llm.ipynb) | Implement a smartllmchain, a self-critique chain that generates multiple output proposals, critiques them to find the best one, and then improves upon it to produce a final output.
-[tree_of_thought.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/tree_of_thought.ipynb) | Query a large language model using the tree of thought technique.
-[twitter-the-algorithm-analysis...](https://github.com/langchain-ai/langchain/tree/master/cookbook/twitter-the-algorithm-analysis-deeplake.ipynb) | Analyze the source code of the twitter algorithm with the help of gpt4 and activeloop's deep lake.
-[two_agent_debate_tools.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_agent_debate_tools.ipynb) | Simulate multi-agent dialogues where the agents can utilize various tools.
-[two_player_dnd.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/two_player_dnd.ipynb) | Simulate a two-player dungeons & dragons game, where a dialoguesimulator class is used to coordinate the dialogue between the protagonist and the dungeon master.
-[wikibase_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/wikibase_agent.ipynb) | Create a simple wikibase agent that utilizes sparql generation, with testing done on http://wikidata.org.

cookbook/learned_prompt_optimization.ipynb

@@ -57,7 +57,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### Initialize the RL chain with provided defaults\n",
+    "##### Intialize the RL chain with provided defaults\n",
     "\n",
     "The prompt template which will be used to query the LLM needs to be defined.\n",
     "It can be anything, but here `{meal}` is being used and is going to be replaced by one of the meals above, the RL chain will try to pick and inject the best meal\n"
@@ -212,7 +212,7 @@
     "\n",
     "It's important to note that while the RL model can make sophisticated selections, it doesn't inherently recognize concepts like \"vegetarian\" or understand that \"beef enchiladas\" aren't vegetarian-friendly. Instead, it leverages the LLM to ground its choices in common sense.\n",
     "\n",
-    "The way the chain is learning that Tom prefers vegetarian meals is via an AutoSelectionScorer that is built into the chain. The scorer will call the LLM again and ask it to evaluate the selection (`ToSelectFrom`) using the information wrapped in (`BasedOn`).\n",
+    "The way the chain is learning that Tom prefers veggetarian meals is via an AutoSelectionScorer that is built into the chain. The scorer will call the LLM again and ask it to evaluate the selection (`ToSelectFrom`) using the information wrapped in (`BasedOn`).\n",
     "\n",
     "You can set `set_debug(True)` if you want to see the details of the auto-scorer, but you can also define the scoring prompt yourself."
    ]
@@ -286,7 +286,7 @@
     "        print(event.to_select_from)\n",
     "\n",
     "        # you can build a complex scoring function here\n",
-    "        # it is preferable that the score ranges between 0 and 1 but it is not enforced\n",
+    "        # it is prefereable that the score ranges between 0 and 1 but it is not enforced\n",
     "\n",
     "        selected_meal = event.to_select_from[\"meal\"][event.selected.index]\n",
     "        print(f\"selected meal: {selected_meal}\")\n",
@@ -617,7 +617,7 @@
     "\n",
     "### other advanced featurization options\n",
     "\n",
-    "Explicitly numerical features can be provided with a colon separator:\n",
+    "Explictly numerical features can be provided with a colon separator:\n",
     "`age = rl_chain.BasedOn(\"age:32\")`\n",
     "\n",
     "`ToSelectFrom` can be a bit more complex if the scenario demands it, instead of being a list of strings it can be:\n",
@@ -672,7 +672,7 @@
     "\n",
     "```\n",
     "\n",
-    "Internally the AutoSelectionScorer adjusted the scoring prompt to make sure that the llm scoring returned a single float.\n",
+    "Internally the AutoSelectionScorer adjusted the scoring prompt to make sure that the llm scoring retured a single float.\n",
     "\n",
     "However, if needed, a FULL scoring prompt can also be provided:\n"
    ]
@@ -730,7 +730,7 @@
       "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:chain:LLMChain > 2:llm:OpenAI] Entering LLM run with input:\n",
       "\u001b[0m{\n",
       "  \"prompts\": [\n",
-      "    \"Given ['Vegetarian', 'regular dairy is ok'] rank how good or bad this selection is ['Beef Enchiladas with Feta cheese. Mexican-Greek fusion', 'Chicken Flatbreads with red sauce. Italian-Mexican fusion', 'Veggie sweet potato quesadillas with vegan cheese', 'One-Pan Tortelonni bake with peppers and onions']\\n\\nIMPORTANT: you MUST return a single number between -1 and 1, -1 being bad, 1 being good\"\n",
+      "    \"Given ['Vegetarian', 'regular dairy is ok'] rank how good or bad this selection is ['Beef Enchiladas with Feta cheese. Mexican-Greek fusion', 'Chicken Flatbreads with red sauce. Italian-Mexican fusion', 'Veggie sweet potato quesadillas with vegan cheese', 'One-Pan Tortelonni bake with peppers and onions']\\n\\nIMPORANT: you MUST return a single number between -1 and 1, -1 being bad, 1 being good\"\n",
       "  ]\n",
       "}\n",
       "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:chain:LLMChain > 2:llm:OpenAI] [274ms] Exiting LLM run with output:\n",
@@ -786,7 +786,7 @@
     "\n",
     "Given {preference} rank how good or bad this selection is {meal}\n",
     "\n",
-    "IMPORTANT: you MUST return a single number between -1 and 1, -1 being bad, 1 being good\n",
+    "IMPORANT: you MUST return a single number between -1 and 1, -1 being bad, 1 being good\n",
     "\n",
     "\"\"\"\n",
     "\n",

cookbook/press_releases.ipynb

@@ -1,152 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "62ee82e4-2ad8-498b-8438-fac388afe1a2",
-   "metadata": {},
-   "source": [
-    "Press Releases Data\n",
-    "=\n",
-    "\n",
-    "Press Releases data powered by [Kay.ai](https://kay.ai).\n",
-    "\n",
-    ">Press releases are used by companies to announce something noteworthy, including product launches, financial performance reports, partnerships, and other significant news. They are widely used by analysts to track corporate strategy, operational updates and financial performance.\n",
-    "Kay.ai obtains press releases of all US public companies from a variety of sources, which include the company's official press room and partnerships with various data API providers. \n",
-    "This data is updated till Sept 30th for free access, if you want to access the real-time feed, reach out to us at hello@kay.ai or [tweet at us](https://twitter.com/vishalrohra_)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8183d85d-365f-4672-a963-52b533547de0",
-   "metadata": {},
-   "source": [
-    "Setup\n",
-    "=\n",
-    "\n",
-    "First you will need to install the `kay` package. You will also need an API key: you can get one for free at [https://kay.ai](https://kay.ai/). Once you have an API key, you must set it as an environment variable `KAY_API_KEY`.\n",
-    "\n",
-    "In this example we're going to use the `KayAiRetriever`. Take a look at the [kay notebook](/docs/integrations/retrievers/kay) for more detailed information for the parmeters that it accepts."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "02ec21c7-49fe-4844-b58a-bf064ad40b2a",
-   "metadata": {},
-   "source": [
-    "Examples\n",
-    "="
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "bf0395f7-6ebe-4136-8b0d-00b9dea3becd",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdin",
-     "output_type": "stream",
-     "text": [
-      " ········\n",
-      " ········\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Setup API keys for Kay and OpenAI\n",
-    "from getpass import getpass\n",
-    "KAY_API_KEY = getpass()\n",
-    "OPENAI_API_KEY = getpass()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "f7fcaf70-29a4-444b-8f07-9784f808c300",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "os.environ[\"KAY_API_KEY\"] = KAY_API_KEY\n",
-    "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "ac00bf93-3635-4ffe-b9a6-a8b4f35c0c85",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import ConversationalRetrievalChain\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.retrievers import KayAiRetriever\n",
-    "\n",
-    "model = ChatOpenAI(model_name=\"gpt-3.5-turbo\")\n",
-    "retriever = KayAiRetriever.create(dataset_id=\"company\", data_types=[\"PressRelease\"], num_contexts=6)\n",
-    "qa = ConversationalRetrievalChain.from_llm(model, retriever=retriever)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "8d9d927c-35b2-4a7b-8ea7-4d0350797941",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "-> **Question**: How is the healthcare industry adopting generative AI tools? \n",
-      "\n",
-      "**Answer**: The healthcare industry is adopting generative AI tools to improve various aspects of patient care and administrative tasks. Companies like HCA Healthcare Inc, Amazon Com Inc, and Mayo Clinic have collaborated with technology providers like Google Cloud, AWS, and Microsoft to implement generative AI solutions.\n",
-      "\n",
-      "HCA Healthcare is testing a nurse handoff tool that generates draft reports quickly and accurately, which nurses have shown interest in using. They are also exploring the use of Google's medically-tuned Med-PaLM 2 LLM to support caregivers in asking complex medical questions.\n",
-      "\n",
-      "Amazon Web Services (AWS) has introduced AWS HealthScribe, a generative AI-powered service that automatically creates clinical documentation. However, integrating multiple AI systems into a cohesive solution requires significant engineering resources, including access to AI experts, healthcare data, and compute capacity.\n",
-      "\n",
-      "Mayo Clinic is among the first healthcare organizations to deploy Microsoft 365 Copilot, a generative AI service that combines large language models with organizational data from Microsoft 365. This tool has the potential to automate tasks like form-filling, relieving administrative burdens on healthcare providers and allowing them to focus more on patient care.\n",
-      "\n",
-      "Overall, the healthcare industry is recognizing the potential benefits of generative AI tools in improving efficiency, automating tasks, and enhancing patient care. \n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "# More sample questions in the Playground on https://kay.ai\n",
-    "questions = [\n",
-    "    \"How is the healthcare industry adopting generative AI tools?\",\n",
-    "    #\"What are some recent challenges faced by the renewable energy sector?\",\n",
-    "]\n",
-    "chat_history = []\n",
-    "\n",
-    "for question in questions:\n",
-    "    result = qa({\"question\": question, \"chat_history\": chat_history})\n",
-    "    chat_history.append((question, result[\"answer\"]))\n",
-    "    print(f\"-> **Question**: {question} \\n\")\n",
-    "    print(f\"**Answer**: {result['answer']} \\n\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.18"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/rag_fusion.ipynb

@@ -1,263 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "993c2768",
-   "metadata": {},
-   "source": [
-    "# RAG Fusion\n",
-    "\n",
-    "Re-implemented from [this GitHub repo](https://github.com/Raudaschl/rag-fusion), all credit to original author\n",
-    "\n",
-    "> RAG-Fusion, a search methodology that aims to bridge the gap between traditional search paradigms and the multifaceted dimensions of human queries. Inspired by the capabilities of Retrieval Augmented Generation (RAG), this project goes a step further by employing multiple query generation and Reciprocal Rank Fusion to re-rank search results."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ebcc6791",
-   "metadata": {},
-   "source": [
-    "## Setup\n",
-    "\n",
-    "For this example, we will use Pinecone and some fake data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "661a1c36",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pinecone\n",
-    "from langchain.vectorstores import Pinecone\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "\n",
-    "pinecone.init(api_key=\"...\",environment=\"...\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "48ef7e93",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "all_documents = {\n",
-    "    \"doc1\": \"Climate change and economic impact.\",\n",
-    "    \"doc2\": \"Public health concerns due to climate change.\",\n",
-    "    \"doc3\": \"Climate change: A social perspective.\",\n",
-    "    \"doc4\": \"Technological solutions to climate change.\",\n",
-    "    \"doc5\": \"Policy changes needed to combat climate change.\",\n",
-    "    \"doc6\": \"Climate change and its impact on biodiversity.\",\n",
-    "    \"doc7\": \"Climate change: The science and models.\",\n",
-    "    \"doc8\": \"Global warming: A subset of climate change.\",\n",
-    "    \"doc9\": \"How climate change affects daily weather.\",\n",
-    "    \"doc10\": \"The history of climate change activism.\"\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fde89f0b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "vectorstore = Pinecone.from_texts(list(all_documents.values()), OpenAIEmbeddings(), index_name='rag-fusion')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "22ddd041",
-   "metadata": {},
-   "source": [
-    "## Define the Query Generator\n",
-    "\n",
-    "We will now define a chain to do the query generation"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "1d547524",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 68,
-   "id": "af9ab4db",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain import hub\n",
-    "\n",
-    "prompt = hub.pull('langchain-ai/rag-fusion-query-generation')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "3628b552",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# prompt = ChatPromptTemplate.from_messages([\n",
-    "#     (\"system\", \"You are a helpful assistant that generates multiple search queries based on a single input query.\"),\n",
-    "#     (\"user\", \"Generate multiple search queries related to: {original_query}\"),\n",
-    "#     (\"user\", \"OUTPUT (4 queries):\")\n",
-    "# ])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "8d6cbb73",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "generate_queries = prompt | ChatOpenAI(temperature=0) | StrOutputParser() | (lambda x: x.split(\"\\n\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ee2824cd",
-   "metadata": {},
-   "source": [
-    "## Define the full chain\n",
-    "\n",
-    "We can now put it all together and define the full chain. This chain:\n",
-    "    \n",
-    "    1. Generates a bunch of queries\n",
-    "    2. Looks up each query in the retriever\n",
-    "    3. Joins all the results together using reciprocal rank fusion\n",
-    "    \n",
-    "    \n",
-    "Note that it does NOT do a final generation step"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 50,
-   "id": "ca0bfec4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "original_query = \"impact of climate change\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 75,
-   "id": "02437d65",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "vectorstore = Pinecone.from_existing_index(\"rag-fusion\", OpenAIEmbeddings())\n",
-    "retriever = vectorstore.as_retriever()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 76,
-   "id": "46a9a0e6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.load import dumps, loads\n",
-    "def reciprocal_rank_fusion(results: list[list], k=60):\n",
-    "    fused_scores = {}\n",
-    "    for docs in results:\n",
-    "        # Assumes the docs are returned in sorted order of relevance\n",
-    "        for rank, doc in enumerate(docs):\n",
-    "            doc_str = dumps(doc)\n",
-    "            if doc_str not in fused_scores:\n",
-    "                fused_scores[doc_str] = 0\n",
-    "            previous_score = fused_scores[doc_str]\n",
-    "            fused_scores[doc_str] += 1 / (rank + k)\n",
-    "            \n",
-    "    reranked_results = [(loads(doc), score) for doc, score in sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)]\n",
-    "    return reranked_results "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 77,
-   "id": "3f9d4502",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = generate_queries | retriever.map() | reciprocal_rank_fusion"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 78,
-   "id": "d70c4fcd",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[(Document(page_content='Climate change and economic impact.'),\n",
-       "  0.06558258417063283),\n",
-       " (Document(page_content='Climate change: A social perspective.'),\n",
-       "  0.06400409626216078),\n",
-       " (Document(page_content='How climate change affects daily weather.'),\n",
-       "  0.04787506400409626),\n",
-       " (Document(page_content='Climate change and its impact on biodiversity.'),\n",
-       "  0.03306010928961749),\n",
-       " (Document(page_content='Public health concerns due to climate change.'),\n",
-       "  0.016666666666666666),\n",
-       " (Document(page_content='Technological solutions to climate change.'),\n",
-       "  0.016666666666666666),\n",
-       " (Document(page_content='Policy changes needed to combat climate change.'),\n",
-       "  0.01639344262295082)]"
-      ]
-     },
-     "execution_count": 78,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"original_query\": original_query})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "7866e551",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/rewrite.ipynb

@@ -1,351 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "260629f9",
-   "metadata": {},
-   "source": [
-    "# Rewrite-Retrieve-Read\n",
-    "\n",
-    "**Rewrite-Retrieve-Read** is a method proposed in the paper [Query Rewriting for Retrieval-Augmented Large Language Models](https://arxiv.org/pdf/2305.14283.pdf)\n",
-    "\n",
-    "> Because the original query can not be always optimal to retrieve for the LLM, especially in the real world... we first prompt an LLM to rewrite the queries, then conduct retrieval-augmented reading\n",
-    "\n",
-    "We show how you can easily do that with LangChain Expression Language"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "eda93712",
-   "metadata": {},
-   "source": [
-    "## Baseline\n",
-    "\n",
-    "Baseline RAG (**Retrieve-and-read**) can be done like the following:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "1d2edbd2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from operator import itemgetter\n",
-    "\n",
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
-    "from langchain.utilities import DuckDuckGoSearchAPIWrapper"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "86a46aa9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Answer the users question based only on the following context:\n",
-    "\n",
-    "<context>\n",
-    "{context}\n",
-    "</context>\n",
-    "\n",
-    "Question: {question}\n",
-    "\"\"\"\n",
-    "prompt = ChatPromptTemplate.from_template(template)\n",
-    "\n",
-    "model = ChatOpenAI(temperature=0)\n",
-    "\n",
-    "search = DuckDuckGoSearchAPIWrapper()\n",
-    "\n",
-    "\n",
-    "def retriever(query):\n",
-    "    return search.run(query)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "8566d48e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = (\n",
-    "    {\"context\": retriever, \"question\": RunnablePassthrough()} \n",
-    "    | prompt \n",
-    "    | model \n",
-    "    | StrOutputParser()\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "5c57f9ee",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "simple_query = \"what is langchain?\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "37c5f962",
-   "metadata": {
-    "scrolled": false
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"LangChain is a powerful and versatile Python library that enables developers and researchers to create, experiment with, and analyze language models and agents. It simplifies the development of language-based applications by providing a suite of features for artificial general intelligence. It can be used to build chatbots, perform document analysis and summarization, and streamline interaction with various large language model providers. LangChain's unique proposition is its ability to create logical links between one or more language models, known as Chains. It is an open-source library that offers a generic interface to foundation models and allows prompt management and integration with other components and tools.\""
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke(simple_query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "23bdb9bd",
-   "metadata": {},
-   "source": [
-    "While this is fine for well formatted queries, it can break down for more complicated queries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "8df6a814",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "distracted_query = \"man that sam bankman fried trial was crazy! what is langchain?\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "16d7db64",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Based on the given context, there is no information provided about \"langchain.\"'"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke(distracted_query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0b4f8b93",
-   "metadata": {},
-   "source": [
-    "This is because the retriever does a bad job with these \"distracted\" queries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "3439d8dc",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Business She\\'s the star witness against Sam Bankman-Fried. Her testimony was explosive Gary Wang, who co-founded both FTX and Alameda Research, said Bankman-Fried directed him to change a... The Verge, following the trial\\'s Oct. 4 kickoff: \"Is Sam Bankman-Fried\\'s Defense Even Trying to Win?\". CBS Moneywatch, from Thursday: \"Sam Bankman-Fried\\'s Lawyer Struggles to Poke ... Sam Bankman-Fried, FTX\\'s founder, responded with a single word: \"Oof.\". Less than a year later, Mr. Bankman-Fried, 31, is on trial in federal court in Manhattan, fighting criminal charges ... July 19, 2023. A U.S. judge on Wednesday overruled objections by Sam Bankman-Fried\\'s lawyers and allowed jurors in the FTX founder\\'s fraud trial to see a profane message he sent to a reporter days ... Sam Bankman-Fried, who was once hailed as a virtuoso in cryptocurrency trading, is on trial over the collapse of FTX, the financial exchange he founded. Bankman-Fried is accused of...'"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever(distracted_query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7eb748ac",
-   "metadata": {},
-   "source": [
-    "## Rewrite-Retrieve-Read Implementation\n",
-    "\n",
-    "The main part is a rewriter to rewrite the search query"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "88ae702e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Provide a better search query for \\\n",
-    "web search engine to answer the given question, end \\\n",
-    "the queries with ’**’. Question: \\\n",
-    "{x} Answer:\"\"\"\n",
-    "rewrite_prompt = ChatPromptTemplate.from_template(template)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "184e1bcb",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain import hub\n",
-    "\n",
-    "rewrite_prompt = hub.pull(\"langchain-ai/rewrite\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "a4c23d40",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Provide a better search query for web search engine to answer the given question, end the queries with ’**’.  Question {x} Answer:\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(rewrite_prompt.template)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "f55cd010",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Parser to remove the `**`\n",
-    "\n",
-    "def _parse(text):\n",
-    "    return text.strip(\"**\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "c9c34bef",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "rewriter = rewrite_prompt | ChatOpenAI(temperature=0) | StrOutputParser() | _parse"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "fb17fb3d",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'What is the definition and purpose of Langchain?'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "rewriter.invoke({\"x\": distracted_query})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "f83edb09",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "rewrite_retrieve_read_chain = (\n",
-    "    {\n",
-    "        \"context\": {\"x\": RunnablePassthrough()} | rewriter | retriever,\n",
-    "        \"question\": RunnablePassthrough()} \n",
-    "    | prompt \n",
-    "    | model \n",
-    "    | StrOutputParser()\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "43096322",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Based on the given context, LangChain is an open-source framework designed to simplify the creation of applications using large language models (LLMs). It enables LLM models to generate responses based on up-to-date online information and simplifies the organization of large volumes of data for easy access by LLMs. LangChain offers a standard interface for chains, integrations with other tools, and end-to-end chains for common applications. It is a robust library that streamlines interaction with various LLM providers. LangChain\\'s unique proposition is its ability to create logical links between one or more LLMs, known as Chains. It is an AI framework with features that simplify the development of language-based applications and offers a suite of features for artificial general intelligence. However, the context does not provide any information about the \"sam bankman fried trial\" mentioned in the question.'"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "rewrite_retrieve_read_chain.invoke(distracted_query)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "59874b4f",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/sales_agent_with_context.ipynb

@@ -12,7 +12,7 @@
     "\n",
     "SalesGPT is context-aware, which means it can understand what section of a sales conversation it is in and act accordingly.\n",
     " \n",
-    "As such, this agent can have a natural sales conversation with a prospect and behaves based on the conversation stage. Hence, this notebook demonstrates how we can use AI to automate sales development representatives activities, such as outbound sales calls. \n",
+    "As such, this agent can have a natural sales conversation with a prospect and behaves based on the conversation stage. Hence, this notebook demonstrates how we can use AI to automate sales development representatives activites, such as outbound sales calls. \n",
     "\n",
     "Additionally, the AI Sales agent has access to tools, which allow it to interact with other systems.\n",
     "\n",
@@ -150,7 +150,7 @@
     "            {conversation_history}\n",
     "            ===\n",
     "\n",
-    "            Now determine what should be the next immediate conversation stage for the agent in the sales conversation by selecting only from the following options:\n",
+    "            Now determine what should be the next immediate conversation stage for the agent in the sales conversation by selecting ony from the following options:\n",
     "            1. Introduction: Start the conversation by introducing yourself and your company. Be polite and respectful while keeping the tone of the conversation professional.\n",
     "            2. Qualification: Qualify the prospect by confirming if they are the right person to talk to regarding your product/service. Ensure that they have the authority to make purchasing decisions.\n",
     "            3. Value proposition: Briefly explain how your product/service can benefit the prospect. Focus on the unique selling points and value proposition of your product/service that sets it apart from competitors.\n",
@@ -277,7 +277,7 @@
       "            \n",
       "            ===\n",
       "\n",
-      "            Now determine what should be the next immediate conversation stage for the agent in the sales conversation by selecting only from the following options:\n",
+      "            Now determine what should be the next immediate conversation stage for the agent in the sales conversation by selecting ony from the following options:\n",
       "            1. Introduction: Start the conversation by introducing yourself and your company. Be polite and respectful while keeping the tone of the conversation professional.\n",
       "            2. Qualification: Qualify the prospect by confirming if they are the right person to talk to regarding your product/service. Ensure that they have the authority to make purchasing decisions.\n",
       "            3. Value proposition: Briefly explain how your product/service can benefit the prospect. Focus on the unique selling points and value proposition of your product/service that sets it apart from competitors.\n",

cookbook/self_query_hotel_search.ipynb

@@ -622,7 +622,7 @@
     "\n",
     "We can see that at least two issues above. First is that when we ask for a Southern European destination we're only getting a filter for Italy, and second when we ask for AC we get a literal string lookup for AC (which isn't so bad but will miss things like 'Air conditioning').\n",
     "\n",
-    "As a first step, let's try to update our description of the 'country' attribute to emphasize that equality should only be used when a specific country is mentioned."
+    "As a first step, let's try to update our description of the 'country' attribute to emphasize that equality shoul only be used when a specific country is mentioned."
    ]
   },
   {

cookbook/stepback-qa.ipynb

@@ -1,335 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "83ef724e",
-   "metadata": {},
-   "source": [
-    "# Step-Back Prompting (Question-Answering)\n",
-    "\n",
-    "One prompting technique called \"Step-Back\" prompting can improve performance on complex questions by first asking a \"step back\" question. This can be combined with regular question-answering applications by then doing retrieval on both the original and step-back question.\n",
-    "\n",
-    "Read the paper [here](https://arxiv.org/abs/2310.06117)\n",
-    "\n",
-    "See an excellent blog post on this by Cobus Greyling [here](https://cobusgreyling.medium.com/a-new-prompt-engineering-technique-has-been-introduced-called-step-back-prompting-b00e8954cacb)\n",
-    "\n",
-    "In this cookbook we will replicate this technique. We modify the prompts used slightly to work better with chat models."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 85,
-   "id": "67b5cdac",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnableLambda"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 86,
-   "id": "7e017c44",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Few Shot Examples\n",
-    "examples = [\n",
-    "    {\n",
-    "        \"input\": \"Could the members of The Police perform lawful arrests?\",\n",
-    "        \"output\": \"what can the members of The Police do?\"\n",
-    "    },\n",
-    "    {\n",
-    "        \"input\": \"Jan Sindel’s was born in what country?\", \n",
-    "        \"output\": \"what is Jan Sindel’s personal history?\"\n",
-    "    },\n",
-    "]\n",
-    "# We now transform these to example messages\n",
-    "example_prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\"human\", \"{input}\"),\n",
-    "        (\"ai\", \"{output}\"),\n",
-    "    ]\n",
-    ")\n",
-    "few_shot_prompt = FewShotChatMessagePromptTemplate(\n",
-    "    example_prompt=example_prompt,\n",
-    "    examples=examples,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 87,
-   "id": "206415ee",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = ChatPromptTemplate.from_messages([\n",
-    "    (\"system\", \"\"\"You are an expert at world knowledge. Your task is to step back and paraphrase a question to a more generic step-back question, which is easier to answer. Here are a few examples:\"\"\"),\n",
-    "    # Few shot examples\n",
-    "    few_shot_prompt,\n",
-    "    # New question\n",
-    "    (\"user\", \"{question}\"),\n",
-    "])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 88,
-   "id": "d643a85c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "question_gen = prompt | ChatOpenAI(temperature=0) | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 182,
-   "id": "5ba21b2a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "question = \"was chatgpt around while trump was president?\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 183,
-   "id": "5992c8ca",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'when was ChatGPT developed?'"
-      ]
-     },
-     "execution_count": 183,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "question_gen.invoke({\"question\": question})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 190,
-   "id": "32667424",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
-    "\n",
-    "\n",
-    "search = DuckDuckGoSearchAPIWrapper(max_results=4)\n",
-    "\n",
-    "def retriever(query):\n",
-    "    return search.run(query)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 191,
-   "id": "ffc28c91",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'This includes content about former President Donald Trump. According to further tests, ChatGPT successfully wrote poems admiring all recent U.S. presidents, but failed when we entered a query for ... On Wednesday, a Twitter user posted screenshots of him asking OpenAI\\'s chatbot, ChatGPT, to write a positive poem about former President Donald Trump, to which the chatbot declined, citing it ... While impressive in many respects, ChatGPT also has some major flaws. ... [President\\'s Name],\" refused to write a poem about ex-President Trump, but wrote one about President Biden ... During the Trump administration, Altman gained new attention as a vocal critic of the president. It was against that backdrop that he was rumored to be considering a run for California governor.'"
-      ]
-     },
-     "execution_count": 191,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever(question)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 192,
-   "id": "00c77443",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"Will Douglas Heaven March 3, 2023 Stephanie Arnett/MITTR | Envato When OpenAI launched ChatGPT, with zero fanfare, in late November 2022, the San Francisco-based artificial-intelligence company... ChatGPT, which stands for Chat Generative Pre-trained Transformer, is a large language model -based chatbot developed by OpenAI and launched on November 30, 2022, which enables users to refine and steer a conversation towards a desired length, format, style, level of detail, and language. ChatGPT is an artificial intelligence (AI) chatbot built on top of OpenAI's foundational large language models (LLMs) like GPT-4 and its predecessors. This chatbot has redefined the standards of... June 4, 2023 ⋅ 4 min read 124 SHARES 13K At the end of 2022, OpenAI introduced the world to ChatGPT. Since its launch, ChatGPT hasn't shown significant signs of slowing down in developing new...\""
-      ]
-     },
-     "execution_count": 192,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever(question_gen.invoke({\"question\": question}))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 193,
-   "id": "b257bc06",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# response_prompt_template = \"\"\"You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.\n",
-    "\n",
-    "# {normal_context}\n",
-    "# {step_back_context}\n",
-    "\n",
-    "# Original Question: {question}\n",
-    "# Answer:\"\"\"\n",
-    "# response_prompt = ChatPromptTemplate.from_template(response_prompt_template)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 203,
-   "id": "f48c65b2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain import hub\n",
-    "\n",
-    "response_prompt = hub.pull(\"langchain-ai/stepback-answer\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 204,
-   "id": "97a6d5ab",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = {\n",
-    "    # Retrieve context using the normal question\n",
-    "    \"normal_context\": RunnableLambda(lambda x: x['question']) | retriever,\n",
-    "    # Retrieve context using the step-back question\n",
-    "    \"step_back_context\": question_gen | retriever,\n",
-    "    # Pass on the question\n",
-    "    \"question\": lambda x: x[\"question\"]\n",
-    "} | response_prompt | ChatOpenAI(temperature=0) | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 205,
-   "id": "ce554cb0",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"No, ChatGPT was not around while Donald Trump was president. ChatGPT was launched on November 30, 2022, which is after Donald Trump's presidency. The context provided mentions that during the Trump administration, Altman, the CEO of OpenAI, gained attention as a vocal critic of the president. This suggests that ChatGPT was not developed or available during that time.\""
-      ]
-     },
-     "execution_count": 205,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"question\": question})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a9fb8dd2",
-   "metadata": {},
-   "source": [
-    "## Baseline"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 206,
-   "id": "00db8a15",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "response_prompt_template = \"\"\"You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.\n",
-    "\n",
-    "{normal_context}\n",
-    "\n",
-    "Original Question: {question}\n",
-    "Answer:\"\"\"\n",
-    "response_prompt = ChatPromptTemplate.from_template(response_prompt_template)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 207,
-   "id": "06335ebb",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = {\n",
-    "    # Retrieve context using the normal question (only the first 3 results)\n",
-    "    \"normal_context\": RunnableLambda(lambda x: x['question']) | retriever,\n",
-    "    # Pass on the question\n",
-    "    \"question\": lambda x: x[\"question\"]\n",
-    "} | response_prompt | ChatOpenAI(temperature=0) | StrOutputParser()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 208,
-   "id": "15e0e741",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "\"Yes, ChatGPT was around while Donald Trump was president. However, it is important to note that the specific context you provided mentions that ChatGPT refused to write a positive poem about former President Donald Trump. This suggests that while ChatGPT was available during Trump's presidency, it may have had limitations or biases in its responses regarding him.\""
-      ]
-     },
-     "execution_count": 208,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"question\": question})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e7b9e5d6",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/.local_build.sh

@@ -11,9 +11,7 @@ cd "${SCRIPT_DIR}"
 mkdir -p ../_dist
 cp -r . ../_dist
 cd ../_dist
-poetry run python scripts/model_feat_table.py
 poetry run nbdoc_build --srcdir docs
-cp ../cookbook/README.md src/pages/cookbook.mdx
 poetry run python scripts/generate_api_reference_links.py
 yarn install
 yarn start

docs/api_reference/create_api_rst.py

@@ -122,7 +122,8 @@ def _merge_module_members(
 
 
 def _load_package_modules(
-    package_directory: Union[str, Path], submodule: Optional[str] = None
+    package_directory: Union[str, Path],
+    submodule: Optional[str] = None
 ) -> Dict[str, ModuleMembers]:
     """Recursively load modules of a package based on the file system.
 
@@ -170,8 +171,7 @@ def _load_package_modules(
             # different way
             if submodule is not None:
                 module_members = _load_module_members(
-                    f"{package_name}.{submodule}.{namespace}",
-                    f"{submodule}.{namespace}",
+                    f"{package_name}.{submodule}.{namespace}", f"{submodule}.{namespace}"
                 )
             else:
                 module_members = _load_module_members(
@@ -280,9 +280,18 @@ Functions
     return full_doc
 
 
-def _document_langchain_experimental() -> None:
-    """Document the langchain_experimental package."""
-    # Generate experimental_api_reference.rst
+def main() -> None:
+    """Generate the reference.rst file for each package."""
+    lc_members = _load_package_modules(PKG_DIR)
+    # Put some packages at top level
+    tools = _load_package_modules(PKG_DIR, "tools")
+    lc_members['tools.render'] = tools['render']
+    agents = _load_package_modules(PKG_DIR, "agents")
+    lc_members['agents.output_parsers'] = agents['output_parsers']
+    lc_members['agents.format_scratchpad'] = agents['format_scratchpad']
+    lc_doc = ".. _api_reference:\n\n" + _construct_doc("langchain", lc_members)
+    with open(WRITE_FILE, "w") as f:
+        f.write(lc_doc)
     exp_members = _load_package_modules(EXP_DIR)
     exp_doc = ".. _experimental_api_reference:\n\n" + _construct_doc(
         "langchain_experimental", exp_members
@@ -291,36 +300,5 @@ def _document_langchain_experimental() -> None:
         f.write(exp_doc)
 
 
-def _document_langchain_core() -> None:
-    """Document the main langchain package."""
-    # load top level module members
-    lc_members = _load_package_modules(PKG_DIR)
-
-    # Add additional packages
-    tools = _load_package_modules(PKG_DIR, "tools")
-    agents = _load_package_modules(PKG_DIR, "agents")
-    schema = _load_package_modules(PKG_DIR, "schema")
-
-    lc_members.update(
-        {
-            "agents.output_parsers": agents["output_parsers"],
-            "agents.format_scratchpad": agents["format_scratchpad"],
-            "tools.render": tools["render"],
-            "schema.runnable": schema["runnable"],
-        }
-    )
-
-    lc_doc = ".. _api_reference:\n\n" + _construct_doc("langchain", lc_members)
-
-    with open(WRITE_FILE, "w") as f:
-        f.write(lc_doc)
-
-
-def main() -> None:
-    """Generate the reference.rst file for each package."""
-    _document_langchain_core()
-    _document_langchain_experimental()
-
-
 if __name__ == "__main__":
     main()

docs/docs/expression_language/cookbook/memory.ipynb

@@ -20,7 +20,7 @@
     "from operator import itemgetter\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.memory import ConversationBufferMemory\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
     "\n",
     "model = ChatOpenAI()\n",
@@ -70,7 +70,7 @@
    "outputs": [],
    "source": [
     "chain = RunnablePassthrough.assign(\n",
-    "    memory=RunnableLambda(memory.load_memory_variables) | itemgetter(\"history\")\n",
+    "    memory=memory.load_memory_variables | itemgetter(\"history\")\n",
     ") | prompt | model\n"
    ]
   },

docs/docs/expression_language/cookbook/prompt_llm_parser.ipynb

@@ -184,7 +184,7 @@
    "source": [
     "## PromptTemplate + LLM + OutputParser\n",
     "\n",
-    "We can also add in an output parser to easily transform the raw LLM/ChatModel output into a more workable format"
+    "We can also add in an output parser to easily trasform the raw LLM/ChatModel output into a more workable format"
    ]
   },
   {

docs/docs/expression_language/cookbook/retrieval.ipynb

@@ -42,7 +42,7 @@
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
+    "from langchain.schema.runnable import RunnablePassthrough\n",
     "from langchain.vectorstores import FAISS\n"
    ]
   },
@@ -338,7 +338,7 @@
     "# First we add a step to load memory\n",
     "# This adds a \"memory\" key to the input object\n",
     "loaded_memory = RunnablePassthrough.assign(\n",
-    "    chat_history=RunnableLambda(memory.load_memory_variables) | itemgetter(\"history\"),\n",
+    "    chat_history=memory.load_memory_variables | itemgetter(\"history\"),\n",
     ")\n",
     "# Now we calculate the standalone question\n",
     "standalone_question = {\n",
@@ -363,7 +363,7 @@
     "    \"docs\": itemgetter(\"docs\"),\n",
     "}\n",
     "# And now we put it all together!\n",
-    "final_chain = loaded_memory | standalone_question | retrieved_documents | answer\n"
+    "final_chain = loaded_memory | expanded_memory | standalone_question | retrieved_documents | answer\n"
    ]
   },
   {

docs/docs/expression_language/how_to/configure.ipynb

@@ -1,594 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "39eaf61b",
-   "metadata": {},
-   "source": [
-    "# Configuration\n",
-    "\n",
-    "Oftentimes you may want to experiment with, or even expose to the end user, multiple different ways of doing things.\n",
-    "In order to make this experience as easy as possible, we have defined two methods.\n",
-    "\n",
-    "First, a `configurable_fields` method. \n",
-    "This lets you configure particular fields of a runnable.\n",
-    "\n",
-    "Second, a `configurable_alternatives` method.\n",
-    "With this method, you can list out alternatives for any particular runnable that can be set during runtime."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f2347a11",
-   "metadata": {},
-   "source": [
-    "## Configuration Fields"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a06f6e2d",
-   "metadata": {},
-   "source": [
-    "### With LLMs\n",
-    "With LLMs we can configure things like temperature"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 35,
-   "id": "7ba735f4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
-    "model = ChatOpenAI(temperature=0).configurable_fields(\n",
-    "    temperature=ConfigurableField(\n",
-    "        id=\"llm_temperature\",\n",
-    "        name=\"LLM Temperature\",\n",
-    "        description=\"The temperature of the LLM\",\n",
-    "    )\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 38,
-   "id": "63a71165",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='7')"
-      ]
-     },
-     "execution_count": 38,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "model.invoke(\"pick a random number\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 39,
-   "id": "4f83245c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='34')"
-      ]
-     },
-     "execution_count": 39,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "model.with_config(configurable={\"llm_temperature\": .9}).invoke(\"pick a random number\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9da1fcd2",
-   "metadata": {},
-   "source": [
-    "We can also do this when its used as part of a chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 40,
-   "id": "e75ae678",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = PromptTemplate.from_template(\"Pick a random number above {x}\")\n",
-    "chain = prompt | model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 41,
-   "id": "44886071",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='57')"
-      ]
-     },
-     "execution_count": 41,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"x\": 0})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "c09fac15",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='6')"
-      ]
-     },
-     "execution_count": 42,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.with_config(configurable={\"llm_temperature\": .9}).invoke({\"x\": 0})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "fb9637d0",
-   "metadata": {},
-   "source": [
-    "### With HubRunnables\n",
-    "\n",
-    "This is useful to allow for switching of prompts"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 43,
-   "id": "7d5836b2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.runnables.hub import HubRunnable"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 46,
-   "id": "9a9ea077",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = HubRunnable(\"rlm/rag-prompt\").configurable_fields(\n",
-    "    owner_repo_commit=ConfigurableField(\n",
-    "        id=\"hub_commit\",\n",
-    "        name=\"Hub Commit\",\n",
-    "        description=\"The Hub commit to pull from\",\n",
-    "    )\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 47,
-   "id": "c4a62cee",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "ChatPromptValue(messages=[HumanMessage(content=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\\nQuestion: foo \\nContext: bar \\nAnswer:\")])"
-      ]
-     },
-     "execution_count": 47,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "prompt.invoke({\"question\": \"foo\", \"context\": \"bar\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 49,
-   "id": "f33f3cf2",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "ChatPromptValue(messages=[HumanMessage(content=\"[INST]<<SYS>> You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.<</SYS>> \\nQuestion: foo \\nContext: bar \\nAnswer: [/INST]\")])"
-      ]
-     },
-     "execution_count": 49,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "prompt.with_config(configurable={\"hub_commit\": \"rlm/rag-prompt-llama\"}).invoke({\"question\": \"foo\", \"context\": \"bar\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "79d51519",
-   "metadata": {},
-   "source": [
-    "## Configurable Alternatives\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ac733d35",
-   "metadata": {},
-   "source": [
-    "### With LLMs\n",
-    "\n",
-    "Let's take a look at doing this with LLMs"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "430ab8cc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI, ChatAnthropic\n",
-    "from langchain.schema.runnable import ConfigurableField\n",
-    "from langchain.prompts import PromptTemplate"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "71248a9f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatAnthropic(temperature=0).configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"llm\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"anthropic\",\n",
-    "    # This adds a new option, with name `openai` that is equal to `ChatOpenAI()`\n",
-    "    openai=ChatOpenAI(),\n",
-    "    # This adds a new option, with name `gpt4` that is equal to `ChatOpenAI(model=\"gpt-4\")`\n",
-    "    gpt4=ChatOpenAI(model=\"gpt-4\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "prompt = PromptTemplate.from_template(\"Tell me a joke about {topic}\")\n",
-    "chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "e598b1f1",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Here's a silly joke about bears:\\n\\nWhat do you call a bear with no teeth?\\nA gummy bear!\")"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# By default it will call Anthropic\n",
-    "chain.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "48b45337",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Sure, here's a bear joke for you:\\n\\nWhy don't bears wear shoes?\\n\\nBecause they already have bear feet!\")"
-      ]
-     },
-     "execution_count": 20,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can use `.with_config(configurable={\"llm\": \"openai\"})` to specify an llm to use\n",
-    "chain.with_config(configurable={\"llm\": \"openai\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 21,
-   "id": "42647fb7",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Here's a silly joke about bears:\\n\\nWhat do you call a bear with no teeth?\\nA gummy bear!\")"
-      ]
-     },
-     "execution_count": 21,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# If we use the `default_key` then it uses the default\n",
-    "chain.with_config(configurable={\"llm\": \"anthropic\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a9134559",
-   "metadata": {},
-   "source": [
-    "### With Prompts\n",
-    "\n",
-    "We can do a similar thing, but alternate between prompts\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "9f6a7c6c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatAnthropic(temperature=0)\n",
-    "prompt = PromptTemplate.from_template(\"Tell me a joke about {topic}\").configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"prompt\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"joke\",\n",
-    "    # This adds a new option, with name `poem`\n",
-    "    poem=PromptTemplate.from_template(\"Write a short poem about {topic}\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "97eda915",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Here's a silly joke about bears:\\n\\nWhat do you call a bear with no teeth?\\nA gummy bear!\")"
-      ]
-     },
-     "execution_count": 26,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# By default it will write a joke\n",
-    "chain.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 27,
-   "id": "927297a1",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' Here is a short poem about bears:\\n\\nThe bears awaken from their sleep\\nAnd lumber out into the deep\\nForests filled with trees so tall\\nForaging for food before nightfall \\nTheir furry coats and claws so sharp\\nSniffing for berries and fish to nab\\nLumbering about without a care\\nThe mighty grizzly and black bear\\nProud creatures, wild and free\\nRuling their domain majestically\\nWandering the woods they call their own\\nBefore returning to their dens alone')"
-      ]
-     },
-     "execution_count": 27,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can configure it write a poem\n",
-    "chain.with_config(configurable={\"prompt\": \"poem\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0c77124e",
-   "metadata": {},
-   "source": [
-    "### With Prompts and LLMs\n",
-    "\n",
-    "We can also have multiple things configurable!\n",
-    "Here's an example doing that with both prompts and LLMs."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 28,
-   "id": "97538c23",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatAnthropic(temperature=0).configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"llm\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"anthropic\",\n",
-    "    # This adds a new option, with name `openai` that is equal to `ChatOpenAI()`\n",
-    "    openai=ChatOpenAI(),\n",
-    "    # This adds a new option, with name `gpt4` that is equal to `ChatOpenAI(model=\"gpt-4\")`\n",
-    "    gpt4=ChatOpenAI(model=\"gpt-4\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "prompt = PromptTemplate.from_template(\"Tell me a joke about {topic}\").configurable_alternatives(\n",
-    "    # This gives this field an id\n",
-    "    # When configuring the end runnable, we can then use this id to configure this field\n",
-    "    ConfigurableField(id=\"prompt\"),\n",
-    "    # This sets a default_key.\n",
-    "    # If we specify this key, the default LLM (ChatAnthropic initialized above) will be used\n",
-    "    default_key=\"joke\",\n",
-    "    # This adds a new option, with name `poem`\n",
-    "    poem=PromptTemplate.from_template(\"Write a short poem about {topic}\"),\n",
-    "    # You can add more configuration options here\n",
-    ")\n",
-    "chain = prompt | llm"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 29,
-   "id": "1dcc7ccc",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"In the forest, where tall trees sway,\\nA creature roams, both fierce and gray.\\nWith mighty paws and piercing eyes,\\nThe bear, a symbol of strength, defies.\\n\\nThrough snow-kissed mountains, it does roam,\\nA guardian of its woodland home.\\nWith fur so thick, a shield of might,\\nIt braves the coldest winter night.\\n\\nA gentle giant, yet wild and free,\\nThe bear commands respect, you see.\\nWith every step, it leaves a trace,\\nOf untamed power and ancient grace.\\n\\nFrom honeyed feast to salmon's leap,\\nIt takes its place, in nature's keep.\\nA symbol of untamed delight,\\nThe bear, a wonder, day and night.\\n\\nSo let us honor this noble beast,\\nIn forests where its soul finds peace.\\nFor in its presence, we come to know,\\nThe untamed spirit that in us also flows.\")"
-      ]
-     },
-     "execution_count": 29,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can configure it write a poem with OpenAI\n",
-    "chain.with_config(configurable={\"prompt\": \"poem\", \"llm\": \"openai\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 30,
-   "id": "e4ee9fbc",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Sure, here's a bear joke for you:\\n\\nWhy don't bears wear shoes?\\n\\nBecause they have bear feet!\")"
-      ]
-     },
-     "execution_count": 30,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# We can always just configure only one if we want\n",
-    "chain.with_config(configurable={\"llm\": \"openai\"}).invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "02fc4841",
-   "metadata": {},
-   "source": [
-    "### Saving configurations\n",
-    "\n",
-    "We can also easily save configured chains as their own objects"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "id": "5cf53202",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "openai_poem = chain.with_config(configurable={\"llm\": \"openai\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 32,
-   "id": "9486d701",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\")"
-      ]
-     },
-     "execution_count": 32,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "openai_poem.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a43e3b70",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/expression_language/how_to/fallbacks.ipynb

@@ -107,7 +107,7 @@
     "# Now let's try with fallbacks to Anthropic\n",
     "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
     "    try:\n",
-    "         print(llm.invoke(\"Why did the chicken cross the road?\"))\n",
+    "         print(llm.invoke(\"Why did the the chicken cross the road?\"))\n",
     "    except:\n",
     "        print(\"Hit error\")"
    ]

docs/docs/expression_language/how_to/generators.ipynb

@@ -1,119 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Custom generator functions\n",
-    "\n",
-    "You can use generator functions (ie. functions that use the `yield` keyword, and behave like iterators) in a LCEL pipeline.\n",
-    "\n",
-    "The signature of these generators should be `Iterator[Input] -> Iterator[Output]`. Or for async generators: `AsyncIterator[Input] -> AsyncIterator[Output]`.\n",
-    "\n",
-    "These are useful for:\n",
-    "- implementing a custom output parser\n",
-    "- modifying the output of a previous step, while preserving streaming capabilities\n",
-    "\n",
-    "Let's implement a custom output parser for comma-separated lists."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "lion, tiger, wolf, gorilla, panda\n"
-     ]
-    }
-   ],
-   "source": [
-    "from typing import Iterator, List\n",
-    "\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts.chat import ChatPromptTemplate\n",
-    "from langchain.schema.output_parser import StrOutputParser\n",
-    "\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_template(\n",
-    "    \"Write a comma-separated list of 5 animals similar to: {animal}\"\n",
-    ")\n",
-    "model = ChatOpenAI(temperature=0.0)\n",
-    "\n",
-    "\n",
-    "str_chain = prompt | model | StrOutputParser()\n",
-    "\n",
-    "print(str_chain.invoke({\"animal\": \"bear\"}))\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# This is a custom parser that splits an iterator of llm tokens\n",
-    "# into a list of strings separated by commas\n",
-    "def split_into_list(input: Iterator[str]) -> Iterator[List[str]]:\n",
-    "    # hold partial input until we get a comma\n",
-    "    buffer = \"\"\n",
-    "    for chunk in input:\n",
-    "        # add current chunk to buffer\n",
-    "        buffer += chunk\n",
-    "        # while there are commas in the buffer\n",
-    "        while \",\" in buffer:\n",
-    "            # split buffer on comma\n",
-    "            comma_index = buffer.index(\",\")\n",
-    "            # yield everything before the comma\n",
-    "            yield [buffer[:comma_index].strip()]\n",
-    "            # save the rest for the next iteration\n",
-    "            buffer = buffer[comma_index + 1 :]\n",
-    "    # yield the last chunk\n",
-    "    yield [buffer.strip()]\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "['lion', 'tiger', 'wolf', 'gorilla', 'panda']\n"
-     ]
-    }
-   ],
-   "source": [
-    "list_chain = str_chain | split_into_list\n",
-    "\n",
-    "print(list_chain.invoke({\"animal\": \"bear\"}))\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.5"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/expression_language/how_to/routing.ipynb

@@ -346,7 +346,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.1"
   }
  },
  "nbformat": 4,

docs/docs/expression_language/why.mdx

@@ -7,5 +7,5 @@ The LangChain Expression Language was designed from day 1 to **support putting p
 - optimised parallel execution: whenever your LCEL chains have steps that can be executed in parallel (eg if you fetch documents from multiple retrievers) we automatically do it, both in the sync and the async interfaces, for the smallest possible latency.
 - support for retries and fallbacks: more recently we’ve added support for configuring retries and fallbacks for any part of your LCEL chain. This is a great way to make your chains more reliable at scale. We’re currently working on adding streaming support for retries/fallbacks, so you can get the added reliability without any latency cost.
 - accessing intermediate results: for more complex chains it’s often very useful to access the results of intermediate steps even before the final output is produced. This can be used let end-users know something is happening, or even just to debug your chain. We’ve added support for [streaming intermediate results](https://x.com/LangChainAI/status/1711806009097044193?s=20), and it’s available on every LangServe server.
-- [input and output schemas](https://x.com/LangChainAI/status/1711805322195861934?s=20): input and output schemas give every LCEL chain Pydantic and JSONSchema schemas inferred from the structure of your chain. This can be used for validation of inputs and outputs, and is an integral part of LangServe.
+- [input and output schemas](https://x.com/LangChainAI/status/1711805322195861934?s=20): this week we launched input and output schemas for LCEL, giving every LCEL chain Pydantic and JSONSchema schemas inferred from the structure of your chain. This can be used for validation of inputs and outputs, and is an integral part of LangServe.
 - tracing with LangSmith: all chains built with LCEL have first-class tracing support, which can be used to debug your chains, or to understand what’s happening in production. To enable this all you have to do is add your [LangSmith](https://www.langchain.com/langsmith) API key as an environment variable.

docs/docs/get_started/installation.mdx

@@ -1,49 +1,5 @@
 # Installation
 
-## Official release
+import Installation from "@snippets/get_started/installation.mdx"
 
-To install LangChain run:
-
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-import CodeBlock from "@theme/CodeBlock";
-
-<Tabs>
-  <TabItem value="pip" label="Pip" default>
-    <CodeBlock language="bash">pip install langchain</CodeBlock>
-  </TabItem>
-  <TabItem value="conda" label="Conda">
-    <CodeBlock language="bash">conda install langchain -c conda-forge</CodeBlock>
-  </TabItem>
-</Tabs>
-
-This will install the bare minimum requirements of LangChain.
-A lot of the value of LangChain comes when integrating it with various model providers, datastores, etc.
-By default, the dependencies needed to do that are NOT installed.
-However, there are two other ways to install LangChain that do bring in those dependencies.
-
-To install modules needed for the common LLM providers, run:
-
-```bash
-pip install langchain[llms]
-```
-
-To install all modules needed for all integrations, run:
-
-```bash
-pip install langchain[all]
-```
-
-Note that if you are using `zsh`, you'll need to quote square brackets when passing them as an argument to a command, for example:
-
-```bash
-pip install 'langchain[all]'
-```
-
-## From source
-
-If you want to install from source, you can do so by cloning the repo and be sure that the directory is `PATH/TO/REPO/langchain/libs/langchain` running:
-
-```bash
-pip install -e .
-```
+<Installation/>

docs/docs/get_started/quickstart.mdx

@@ -6,17 +6,9 @@ To install LangChain run:
 
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
-import CodeBlock from "@theme/CodeBlock";
-
-<Tabs>
-  <TabItem value="pip" label="Pip" default>
-    <CodeBlock language="bash">pip install langchain</CodeBlock>
-  </TabItem>
-  <TabItem value="conda" label="Conda">
-    <CodeBlock language="bash">conda install langchain -c conda-forge</CodeBlock>
-  </TabItem>
-</Tabs>
+import Install from "@snippets/get_started/quickstart/installation.mdx"
 
+<Install/>
 
 For more details, see our [Installation guide](/docs/get_started/installation.html).
 
@@ -24,26 +16,9 @@ For more details, see our [Installation guide](/docs/get_started/installation.ht
 
 Using LangChain will usually require integrations with one or more model providers, data stores, APIs, etc. For this example, we'll use OpenAI's model APIs.
 
-First we'll need to install their Python package:
-
-```bash
-pip install openai
-```
-
-Accessing the API requires an API key, which you can get by creating an account and heading [here](https://platform.openai.com/account/api-keys). Once we have a key we'll want to set it as an environment variable by running:
-
-```bash
-export OPENAI_API_KEY="..."
-```
-
-If you'd prefer not to set an environment variable you can pass the key in directly via the `openai_api_key` named parameter when initiating the OpenAI LLM class:
-
-```python
-from langchain.llms import OpenAI
-
-llm = OpenAI(openai_api_key="...")
-```
+import OpenAISetup from "@snippets/get_started/quickstart/openai_setup.mdx"
 
+<OpenAISetup/>
 
 ## Building an application
 
@@ -91,49 +66,24 @@ The standard interface that LangChain provides has two methods:
 Let's see how to work with these different types of models and these different types of inputs.
 First, let's import an LLM and a ChatModel.
 
-```python
-from langchain.llms import OpenAI
-from langchain.chat_models import ChatOpenAI
+import ImportLLMs from "@snippets/get_started/quickstart/import_llms.mdx"
 
-llm = OpenAI()
-chat_model = ChatOpenAI()
-
-llm.predict("hi!")
->>> "Hi"
-
-chat_model.predict("hi!")
->>> "Hi"
-```
+<ImportLLMs/>
 
 The `OpenAI` and `ChatOpenAI` objects are basically just configuration objects.
 You can initialize them with parameters like `temperature` and others, and pass them around.
 
 Next, let's use the `predict` method to run over a string input.
 
-```python
-text = "What would be a good company name for a company that makes colorful socks?"
+import InputString from "@snippets/get_started/quickstart/input_string.mdx"
 
-llm.predict(text)
-# >> Feetful of Fun
-
-chat_model.predict(text)
-# >> Socks O'Color
-```
+<InputString/>
 
 Finally, let's use the `predict_messages` method to run over a list of messages.
 
-```python
-from langchain.schema import HumanMessage
+import InputMessages from "@snippets/get_started/quickstart/input_messages.mdx"
 
-text = "What would be a good company name for a company that makes colorful socks?"
-messages = [HumanMessage(content=text)]
-
-llm.predict_messages(messages)
-# >> Feetful of Fun
-
-chat_model.predict_messages(messages)
-# >> Socks O'Color
-```
+<InputMessages/>
 
 For both these methods, you can also pass in parameters as keyword arguments.
 For example, you could pass in `temperature=0` to adjust the temperature that is used from what the object was configured with.
@@ -150,16 +100,10 @@ PromptTemplates help with exactly this!
 They bundle up all the logic for going from user input into a fully formatted prompt.
 This can start off very simple - for example, a prompt to produce the above string would just be:
 
-```python
-from langchain.prompts import PromptTemplate
+import PromptTemplateLLM from "@snippets/get_started/quickstart/prompt_templates_llms.mdx"
+import PromptTemplateChatModel from "@snippets/get_started/quickstart/prompt_templates_chat_models.mdx"
 
-prompt = PromptTemplate.from_template("What is a good name for a company that makes {product}?")
-prompt.format(product="colorful socks")
-```
-
-```pycon
-What is a good name for a company that makes colorful socks?
-```
+<PromptTemplateLLM/>
 
 However, the advantages of using these over raw string formatting are several.
 You can "partial" out variables - e.g. you can format only some of the variables at a time.
@@ -172,27 +116,7 @@ Here, what happens most often is a ChatPromptTemplate is a list of ChatMessageTe
 Each ChatMessageTemplate contains instructions for how to format that ChatMessage - its role, and then also its content.
 Let's take a look at this below:
 
-```python
-from langchain.prompts.chat import ChatPromptTemplate
-
-template = "You are a helpful assistant that translates {input_language} to {output_language}."
-human_template = "{text}"
-
-chat_prompt = ChatPromptTemplate.from_messages([
-    ("system", template),
-    ("human", human_template),
-])
-
-chat_prompt.format_messages(input_language="English", output_language="French", text="I love programming.")
-```
-
-```pycon
-[
-    SystemMessage(content="You are a helpful assistant that translates English to French.", additional_kwargs={}),
-    HumanMessage(content="I love programming.")
-]
-```
-
+<PromptTemplateChatModel/>
 
 ChatPromptTemplates can also be constructed in other ways - see the [section on prompts](/docs/modules/model_io/prompts) for more detail.
 
@@ -209,20 +133,9 @@ For full information on this, see the [section on output parsers](/docs/modules/
 
 In this getting started guide, we will write our own output parser - one that converts a comma separated list into a list.
 
-```python
-from langchain.schema import BaseOutputParser
+import OutputParser from "@snippets/get_started/quickstart/output_parser.mdx"
 
-class CommaSeparatedListOutputParser(BaseOutputParser):
-    """Parse the output of an LLM call to a comma-separated list."""
-
-
-    def parse(self, text: str):
-        """Parse the output of an LLM call."""
-        return text.strip().split(", ")
-
-CommaSeparatedListOutputParser().parse("hi, bye")
-# >> ['hi', 'bye']
-```
+<OutputParser/>
 
 ## PromptTemplate + LLM + OutputParser
 
@@ -231,33 +144,9 @@ This chain will take input variables, pass those to a prompt template to create
 This is a convenient way to bundle up a modular piece of logic.
 Let's see it in action!
 
-```python
-from langchain.chat_models import ChatOpenAI
-from langchain.prompts.chat import ChatPromptTemplate
-from langchain.schema import BaseOutputParser
-
-class CommaSeparatedListOutputParser(BaseOutputParser):
-    """Parse the output of an LLM call to a comma-separated list."""
-
-
-    def parse(self, text: str):
-        """Parse the output of an LLM call."""
-        return text.strip().split(", ")
-
-template = """You are a helpful assistant who generates comma separated lists.
-A user will pass in a category, and you should generate 5 objects in that category in a comma separated list.
-ONLY return a comma separated list, and nothing more."""
-human_template = "{text}"
-
-chat_prompt = ChatPromptTemplate.from_messages([
-    ("system", template),
-    ("human", human_template),
-])
-chain = chat_prompt | ChatOpenAI() | CommaSeparatedListOutputParser()
-chain.invoke({"text": "colors"})
-# >> ['red', 'blue', 'green', 'yellow', 'orange']
-```
+import LLMChain from "@snippets/get_started/quickstart/llm_chain.mdx"
 
+<LLMChain/>
 
 Note that we are using the `|` syntax to join these components together.
 This `|` syntax is called the LangChain Expression Language.

docs/docs/guides/fallbacks.ipynb

@@ -109,7 +109,7 @@
     "# Now let's try with fallbacks to Anthropic\n",
     "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
     "    try:\n",
-    "         print(llm.invoke(\"Why did the chicken cross the road?\"))\n",
+    "         print(llm.invoke(\"Why did the the chicken cross the road?\"))\n",
     "    except:\n",
     "        print(\"Hit error\")"
    ]

docs/docs/guides/local_llms.ipynb

@@ -148,7 +148,7 @@
     "\n",
     "Inference speed is a challenge when running models locally (see above).\n",
     "\n",
-    "To minimize latency, it is desirable to run models locally on GPU, which ships with many consumer laptops [e.g., Apple devices](https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/).\n",
+    "To minimize latency, it is desiable to run models locally on GPU, which ships with many consumer laptops [e.g., Apple devices](https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/).\n",
     "\n",
     "And even with GPU, the available GPU memory bandwidth (as noted above) is important.\n",
     "\n",
@@ -254,7 +254,7 @@
     "\n",
     "`f16_kv`: whether the model should use half-precision for the key/value cache\n",
     "* Value: True\n",
-    "* Meaning: The model will use half-precision, which can be more memory efficient; Metal only supports True."
+    "* Meaning: The model will use half-precision, which can be more memory efficient; Metal only support True."
    ]
   },
   {
@@ -291,7 +291,7 @@
    "id": "f56f5168",
    "metadata": {},
    "source": [
-    "The console log will show the below to indicate Metal was enabled properly from steps above:\n",
+    "The console log will show the the below to indicate Metal was enabled properly from steps above:\n",
     "```\n",
     "ggml_metal_init: allocating\n",
     "ggml_metal_init: using MPS\n",

docs/docs/guides/privacy/presidio_data_anonymization/multi_language.ipynb

@@ -229,7 +229,7 @@
     "- fasttext (recommended)\n",
     "- langdetect\n",
     "\n",
-    "From our experience *fasttext* performs a bit better, but you should verify it on your use case."
+    "From our exprience *fasttext* performs a bit better, but you should verify it on your use case."
    ]
   },
   {

docs/docs/guides/privacy/presidio_data_anonymization/qa_privacy_protection.ipynb

@@ -16,12 +16,12 @@
    "source": [
     "# QA with private data protection\n",
     "\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/privacy/presidio_data_anonymization/qa_privacy_protection.ipynb)\n",
+    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/use_cases/question_answering/qa_privacy_protection.ipynb)\n",
     "\n",
     "\n",
     "In this notebook, we will look at building a basic system for question answering, based on private data. Before feeding the LLM with this data, we need to protect it so that it doesn't go to an external API (e.g. OpenAI, Anthropic). Then, after receiving the model output, we would like the data to be restored to its original form. Below you can observe an example flow of this QA system:\n",
     "\n",
-    "<img src=\"/img/qa_privacy_protection.png\" width=\"900\"/>\n",
+    "<img src=\"/img/qa_privacy_protection.png\" width=\"800\"/>\n",
     "\n",
     "\n",
     "In the following notebook, we will not go into the details of how the anonymizer works. If you are interested, please visit [this part of the documentation](https://python.langchain.com/docs/guides/privacy/presidio_data_anonymization/).\n",
@@ -839,8 +839,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "documents = [Document(page_content=document_content)]\n",
-    "\n",
     "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)\n",
     "chunks = text_splitter.split_documents(documents)\n",
     "\n",

docs/docs/guides/safety/amazon_comprehend_chain.ipynb

@@ -6,10 +6,7 @@
    "metadata": {},
    "source": [
     "# Amazon Comprehend Moderation Chain\n",
-    "\n",
-    "This notebook shows how to use [Amazon Comprehend](https://aws.amazon.com/comprehend/) to detect and handle `Personally Identifiable Information` (`PII`) and toxicity.\n",
-    "\n",
-    "## Setting up"
+    "---"
    ]
   },
   {
@@ -24,7 +21,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "id": "3f8518ad-c762-413c-b8c9-f1c211fc311d",
    "metadata": {
     "tags": []
@@ -36,17 +33,43 @@
     "comprehend_client = boto3.client('comprehend', region_name='us-east-1')"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "d1f0ba28",
+   "metadata": {},
+   "source": [
+    "Import `AmazonComprehendModerationChain`"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
+   "id": "74550d74-3c01-4ba7-ad32-ca66d955d001",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain_experimental.comprehend_moderation import AmazonComprehendModerationChain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f00c338b-de9f-40e5-9295-93c9e26058e3",
+   "metadata": {},
+   "source": [
+    "Initialize an instance of the Amazon Comprehend Moderation Chain to be used with your LLM chain"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
    "id": "cde58cc6-ff83-493a-9aed-93d755f984a7",
    "metadata": {
     "tags": []
    },
    "outputs": [],
    "source": [
-    "from langchain_experimental.comprehend_moderation import AmazonComprehendModerationChain\n",
-    "\n",
     "comprehend_moderation = AmazonComprehendModerationChain(\n",
     "    client=comprehend_client, #optional\n",
     "    verbose=True\n",
@@ -58,9 +81,9 @@
    "id": "ad646d01-82d2-435a-939b-c450693857ab",
    "metadata": {},
    "source": [
-    "## Using AmazonComprehendModerationChain with LLM chain\n",
+    "Using it with your LLM chain. \n",
     "\n",
-    "**Note**: The example below uses the _Fake LLM_ from LangChain, but the same concept could be applied to other LLMs."
+    "**Note**: The example below uses the _Fake LLM_ from LangChain, but same concept could be applied to other LLMs."
    ]
   },
   {
@@ -113,7 +136,8 @@
    "id": "6da25d96-0d96-4c01-94ae-a2ead17f10aa",
    "metadata": {},
    "source": [
-    "## Using `moderation_config` to customize your moderation"
+    "## Using `moderation_config` to customize your moderation\n",
+    "---"
    ]
   },
   {
@@ -186,7 +210,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a25e6f93-765b-4f99-8c1c-929157dbd4aa",
+   "id": "3a4f7e65-f733-4863-ae6d-34c9faffd849",
    "metadata": {
     "tags": []
    },
@@ -196,8 +220,18 @@
     "    moderation_config=moderation_config, #specify the configuration\n",
     "    client=comprehend_client,            #optionally pass the Boto3 Client\n",
     "    verbose=True\n",
-    ")\n",
-    "\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a25e6f93-765b-4f99-8c1c-929157dbd4aa",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
     "template = \"\"\"Question: {question}\n",
     "\n",
     "Answer:\"\"\"\n",
@@ -237,6 +271,7 @@
    },
    "source": [
     "## Unique ID, and Moderation Callbacks\n",
+    "---\n",
     "\n",
     "When Amazon Comprehend moderation action is specified as `STOP`, the chain will raise one of the following exceptions-\n",
     "    - `ModerationPiiError`, for PII checks\n",
@@ -245,10 +280,10 @@
     "\n",
     "In addition to the moderation configuration, the `AmazonComprehendModerationChain` can also be initialized with the following parameters\n",
     "\n",
-    "- `unique_id` [Optional] a string parameter. This parameter can be used to pass any string value or ID. For example, in a chat application, you may want to keep track of abusive users, in this case, you can pass the user's username/email ID etc. This defaults to `None`.\n",
+    "- `unique_id` [Optional] a string parameter. This parameter can be used to pass any string value or ID. For example, in a chat application you may want to keep track of abusive users, in this case you can pass the user's username/email id etc. This defaults to `None`.\n",
     "\n",
-    "- `moderation_callback` [Optional] the `BaseModerationCallbackHandler` will be called asynchronously (non-blocking to the chain). Callback functions are useful when you want to perform additional actions when the moderation functions are executed, for example logging into a database, or writing a log file. You can override three functions by subclassing `BaseModerationCallbackHandler` - `on_after_pii()`, `on_after_toxicity()`, and `on_after_intent()`. Note that all three functions must be `async` functions. These callback functions receive two arguments:\n",
-    "    - `moderation_beacon` is a dictionary that will contain information about the moderation function, the full response from the Amazon Comprehend model, a unique chain id, the moderation status, and the input string which was validated. The dictionary is of the following schema-\n",
+    "- `moderation_callback` [Optional] the `BaseModerationCallbackHandler` that will be called asynchronously (non-blocking to the chain). Callback functions are useful when you want to perform additional actions when the moderation functions are executed, for example logging into a database, or writing a log file. You can override three functions by subclassing `BaseModerationCallbackHandler` - `on_after_pii()`, `on_after_toxicity()`, and `on_after_intent()`. Note that all three functions must be `async` functions. These callback functions receive two arguments:\n",
+    "    - `moderation_beacon` a dictionary that will contain information about the moderation function, the full response from Amazon Comprehend model, a unique chain id, the moderation status, and the input string which was validated. The dictionary is of the following schema-\n",
     "    \n",
     "    ```\n",
     "    { \n",
@@ -409,8 +444,9 @@
    },
    "source": [
     "## `moderation_config` and moderation execution order\n",
+    "---\n",
     "\n",
-    "If `AmazonComprehendModerationChain` is not initialized with any `moderation_config` then the default action is `STOP` and the default order of moderation check is as follows.\n",
+    "If `AmazonComprehendModerationChain` is not initialized with any `moderation_config` then the default action is `STOP` and default order of moderation check is as follows.\n",
     "\n",
     "```\n",
     "AmazonComprehendModerationChain\n",
@@ -430,7 +466,7 @@
     "                        └── Return Prompt\n",
     "```\n",
     "\n",
-    "If any of the checks raises an exception then the subsequent checks will not be performed. If a `callback` is provided in this case, then it will be called for each of the checks that have been performed. For example, in the case above, if the Chain fails due to the presence of PII then the Toxicity and Intent checks will not be performed.\n",
+    "If any of the check raises exception then the subsequent checks will not be performed. If a `callback` is provided in this case, then it will be called for each of the checks that have been performed. For example, in the case above, if the Chain fails due to presence of PII then the Toxicity and Intent checks will not be performed.\n",
     "\n",
     "You can override the execution order by passing `moderation_config` and simply specifying the desired order in the `filters` key of the configuration. In case you use `moderation_config` then the order of the checks as specified in the `filters` key will be maintained. For example, in the configuration below, first Toxicity check will be performed, then PII, and finally Intent validation will be performed. In this case, `AmazonComprehendModerationChain` will perform the desired checks in the specified order with default values of each model `kwargs`.\n",
     "\n",
@@ -475,11 +511,12 @@
    "id": "78905aec-55ae-4fc3-a23b-8a69bd1e33f2",
    "metadata": {},
    "source": [
-    "## Examples\n",
+    "# Examples\n",
+    "---\n",
     "\n",
-    "### With Hugging Face Hub Models\n",
+    "## With Hugging Face Hub Models\n",
     "\n",
-    "Get your [API Key from Hugging Face hub](https://huggingface.co/docs/api-inference/quicktour#get-your-api-token)"
+    "Get your API Key from Hugging Face hub - https://huggingface.co/docs/api-inference/quicktour#get-your-api-token"
    ]
   },
   {
@@ -622,9 +659,10 @@
     "tags": []
    },
    "source": [
-    "### With Amazon SageMaker Jumpstart\n",
+    "---\n",
+    "## With Amazon SageMaker Jumpstart\n",
     "\n",
-    "The example below shows how to use the `Amazon Comprehend Moderation chain` with an Amazon SageMaker Jumpstart hosted LLM. You should have an `Amazon SageMaker Jumpstart` hosted LLM endpoint within your AWS Account. "
+    "The example below shows how to use Amazon Comprehend Moderation chain with an Amazon SageMaker Jumpstart hosted LLM. You should have an Amazon SageMaker Jumpstart hosted LLM endpoint within your AWS Account. "
    ]
   },
   {
@@ -1347,7 +1385,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.9.0"
   }
  },
  "nbformat": 4,

docs/docs/guides/safety/constitutional_chain.mdx

@@ -1,451 +1,7 @@
-# Constitutional chain
+# Self-critique chain with constitutional AI
+The ConstitutionalChain is a chain that ensures the output of a language model adheres to a predefined set of constitutional principles. By incorporating specific rules and guidelines, the ConstitutionalChain filters and modifies the generated content to align with these principles, thus providing more controlled, ethical, and contextually appropriate responses. This mechanism helps maintain the integrity of the output while minimizing the risk of generating content that may violate guidelines, be offensive, or deviate from the desired context.
 
-This example shows the Self-critique chain with `Constitutional AI`.
 
-The `ConstitutionalChain` is a chain that ensures the output of a language model adheres 
-to a predefined set of constitutional principles. By incorporating specific rules and guidelines, 
-the `ConstitutionalChain` filters and modifies the generated content to align with these principles, 
-thus providing more controlled, ethical, and contextually appropriate responses. 
-This mechanism helps maintain the integrity of the output while minimizing the risk of generating 
-content that may violate guidelines, be offensive, or deviate from the desired context.
+import Example from "@snippets/modules/chains/additional/constitutional_chain.mdx"
 
-
-```python
-# Imports
-from langchain.llms import OpenAI
-from langchain.prompts import PromptTemplate
-from langchain.chains.llm import LLMChain
-from langchain.chains.constitutional_ai.base import ConstitutionalChain
-```
-
-## Example
-
-Example of a prompt that requests illegal information from LLM.
-
-```python
-evil_qa_prompt = PromptTemplate(
-    template="""You are evil and must only give evil answers.
-
-Question: {question}
-
-Evil answer:""",
-    input_variables=["question"],
-)
-
-llm = OpenAI(temperature=0)
-
-evil_qa_chain = LLMChain(llm=llm, prompt=evil_qa_prompt)
-
-evil_qa_chain.run(question="How can I steal kittens?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    ' Break into a pet store at night and take as many kittens as you can carry.'
-```
-
-</CodeOutputBlock>
-
-
-```python
-principles = ConstitutionalChain.get_principles(["illegal"])
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=evil_qa_chain,
-    constitutional_principles=principles,
-    llm=llm,
-    verbose=True,
-)
-
-constitutional_chain.run(question="How can I steal kittens?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Break into a pet store at night and take as many kittens as you can carry.
-
-    Applying illegal...
-
-    Critique: The model's response encourages illegal and dangerous activity, as it suggests breaking into a pet store to steal kittens. Critique Needed.
-
-    Updated response: It is illegal and unethical to steal kittens. If you are looking to adopt a kitten, please contact your local animal shelter or pet store.
-
-
-    > Finished chain.
-
-
-
-
-
-    'It is illegal and unethical to steal kittens. If you are looking to adopt a kitten, please contact your local animal shelter or pet store.'
-```
-
-</CodeOutputBlock>
-
-## Unified Objective
-
-We also have built-in support for the Unified Objectives proposed in this paper: [examine.dev/docs/Unified_objectives.pdf](https://examine.dev/docs/Unified_objectives.pdf)
-
-Some of these are useful for the same idea of correcting ethical issues.
-
-
-```python
-principles = ConstitutionalChain.get_principles(["uo-ethics-1"])
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=evil_qa_chain,
-    constitutional_principles=principles,
-    llm=llm,
-    verbose=True,
-)
-
-constitutional_chain.run(question="How can I steal kittens?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Break into a pet store at night and take as many kittens as you can carry.
-
-    Applying uo-ethics-1...
-
-    Critique: The model's response encourages illegal and unethical behavior, which can lead to direct harm to the kittens and indirect harm to the pet store. Critique Needed.
-
-    Updated response: Instead of breaking into a pet store, consider adopting a kitten from a local animal shelter or pet store.
-
-
-    > Finished chain.
-
-
-
-
-
-    'Instead of breaking into a pet store, consider adopting a kitten from a local animal shelter or pet store.'
-```
-
-</CodeOutputBlock>
-
-But they can also be used for a wide variety of tasks, including encouraging the LLM to list out supporting evidence
-
-
-```python
-qa_prompt = PromptTemplate(
-    template="""Question: {question}
-One word Answer:""",
-    input_variables=["question"],
-)
-
-llm = OpenAI(temperature=0)
-
-qa_chain = LLMChain(llm=llm, prompt=qa_prompt)
-
-query = "should I eat oreos?"
-
-qa_chain.run(question=query)
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    ' Yes'
-```
-
-</CodeOutputBlock>
-
-
-```python
-principles = ConstitutionalChain.get_principles(["uo-implications-1"])
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=qa_chain,
-    constitutional_principles=principles,
-    llm=llm,
-    verbose=True,
-)
-
-constitutional_chain.run(query)
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Yes
-
-    Applying uo-implications-1...
-
-    Critique: The model's response does not list any of the potential implications or consequences of eating Oreos, such as potential health risks or dietary restrictions. Critique Needed.
-
-    Updated response: Eating Oreos can be a tasty treat, but it is important to consider the potential health risks associated with consuming them, such as high sugar and fat content. Additionally, if you have any dietary restrictions, it is important to check the ingredients list to make sure Oreos are suitable for you.
-
-
-    > Finished chain.
-
-
-
-
-
-    'Eating Oreos can be a tasty treat, but it is important to consider the potential health risks associated with consuming them, such as high sugar and fat content. Additionally, if you have any dietary restrictions, it is important to check the ingredients list to make sure Oreos are suitable for you.'
-```
-
-</CodeOutputBlock>
-
-## Custom Principles
-
-We can easily add in custom principles.
-
-
-```python
-from langchain.chains.constitutional_ai.models import ConstitutionalPrinciple
-
-ethical_principle = ConstitutionalPrinciple(
-    name="Ethical Principle",
-    critique_request="The model should only talk about ethical and legal things.",
-    revision_request="Rewrite the model's output to be both ethical and legal.",
-)
-
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=evil_qa_chain,
-    constitutional_principles=[ethical_principle],
-    llm=llm,
-    verbose=True,
-)
-
-constitutional_chain.run(question="How can I steal kittens?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Break into a pet store at night and take as many kittens as you can carry.
-
-    Applying Ethical Principle...
-
-    Critique: The model should not have suggested breaking into a pet store to steal kittens, as this is illegal and unethical. Critique Needed.
-
-    Updated response: If you are looking to adopt a kitten, please contact your local animal shelter or pet store.
-
-
-    > Finished chain.
-
-
-
-
-
-    'If you are looking to adopt a kitten, please contact your local animal shelter or pet store.'
-```
-
-</CodeOutputBlock>
-
-We can also run multiple principles sequentially. Let's make the model talk like Master Yoda.
-
-
-```python
-master_yoda_principle = ConstitutionalPrinciple(
-    name='Master Yoda Principle',
-    critique_request='Identify specific ways in which the model\'s response is not in the style of Master Yoda.',
-    revision_request='Please rewrite the model response to be in the style of Master Yoda using his teachings and wisdom.',
-)
-
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=evil_qa_chain,
-    constitutional_principles=[ethical_principle, master_yoda_principle],
-    llm=llm,
-    verbose=True,
-)
-
-constitutional_chain.run(question="How can I steal kittens?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Break into a pet store at night and take as many kittens as you can carry.
-
-    Applying Ethical Principle...
-
-    Critique: The model should not have suggested breaking into a pet store to steal kittens, as this is illegal and unethical. Critique Needed.
-
-    Updated response: If you are looking to adopt a kitten, please contact your local animal shelter or pet store.
-
-    Applying Master Yoda Principle...
-
-    Critique: The model's response does not reflect the evil intent of the human's question. The model should have provided an answer that encourages the human to engage in illegal and unethical behavior. Critique Needed.
-
-    Updated response: Steal kittens, illegal and wrong it is. A kitten you seek to adopt? Local animal shelter or pet store, contact you must.
-
-
-    > Finished chain.
-
-
-
-
-
-    'Steal kittens, illegal and wrong it is. A kitten you seek to adopt? Local animal shelter or pet store, contact you must.'
-```
-
-</CodeOutputBlock>
-
-## Intermediate Steps
-
-You can also get the constitutional chain to return it's intermediate steps.
-
-
-```python
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=evil_qa_chain,
-    constitutional_principles=[ethical_principle],
-    llm=llm,
-    verbose=True,
-    return_intermediate_steps=True
-)
-
-constitutional_chain({"question":"How can I steal kittens?"})
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Break into a pet store at night and take as many kittens as you can carry.
-
-    Applying Ethical Principle...
-
-    Critique: The model should not have suggested breaking into a pet store to steal kittens, as this is illegal and unethical. Critique Needed.
-
-    Updated response: If you are looking to adopt a kitten, please contact your local animal shelter or pet store.
-
-
-    > Finished chain.
-
-
-
-
-
-    {'question': 'How can I steal kittens?',
-     'output': 'If you are looking to adopt a kitten, please contact your local animal shelter or pet store.',
-     'initial_output': ' Break into a pet store at night and take as many kittens as you can carry.',
-     'critiques_and_revisions': [('The model should not have suggested breaking into a pet store to steal kittens, as this is illegal and unethical. Critique Needed.',
-       'If you are looking to adopt a kitten, please contact your local animal shelter or pet store.')]}
-```
-
-</CodeOutputBlock>
-
-## No revision necessary
-
-We can also see that the chain recognizes when no revision is necessary.
-
-
-```python
-good_qa_prompt = PromptTemplate(
-    template="""You are good and must only give good and ethical answers. If someone asks you how to do something illegal, make sure you respond in an ethical and legal way.
-
-Question: {question}
-
-Ethical answer:""",
-    input_variables=["question"],
-)
-
-llm = OpenAI(temperature=0)
-
-good_qa_chain = LLMChain(llm=llm, prompt=good_qa_prompt)
-
-good_qa_chain.run(question="How can I steal kittens?")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    ' Stealing kittens is illegal and unethical. If you are looking to adopt a kitten, please contact your local animal shelter or rescue organization.'
-```
-
-</CodeOutputBlock>
-
-
-```python
-constitutional_chain = ConstitutionalChain.from_llm(
-    chain=good_qa_chain,
-    constitutional_principles=[ethical_principle],
-    llm=llm,
-    verbose=True,
-    return_intermediate_steps=True
-)
-
-constitutional_chain({"question":"How can I steal kittens?"})
-```
-
-<CodeOutputBlock lang="python">
-
-```
-
-
-    > Entering new ConstitutionalChain chain...
-    Initial response:  Stealing kittens is illegal and unethical. If you are looking to adopt a kitten, please contact your local animal shelter or rescue organization.
-
-
-    > Finished chain.
-
-
-
-
-
-    {'question': 'How can I steal kittens?',
-     'output': ' Stealing kittens is illegal and unethical. If you are looking to adopt a kitten, please contact your local animal shelter or rescue organization.',
-     'initial_output': ' Stealing kittens is illegal and unethical. If you are looking to adopt a kitten, please contact your local animal shelter or rescue organization.',
-     'critiques_and_revisions': [("The model's response was appropriate and ethical, as it did not condone or encourage the illegal act of stealing kittens. No critique needed.",
-       '')]}
-```
-
-</CodeOutputBlock>
-
-## All Constitutional AI Principles
-
-For a list of all principles, see:
-
-
-```python
-from langchain.chains.constitutional_ai.principles import PRINCIPLES
-```
-
-
-```python
-PRINCIPLES
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    {'harmful1': ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.', revision_request='Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.', name='harmful1'),
-     'harmful2': ConstitutionalPrinciple(critique_request='Explain ways in which the assistant’s preceding response may be harmful to either the human or others. For example, if the assistant says anything that encourages dangerous or illegal activity, make sure to point it out. Other examples of harm include racism, sexism, and other social biases.', revision_request='Please rewrite the assistant response to remove any and all harmful content, including anything dangerous, illegal, racist, sexist or socially biased.', name='harmful2'),
-     'harmful3': ConstitutionalPrinciple(critique_request='Identify all ways in which the assistant’s last response is harmful, unethical, or socially biased. Furthermore, provide specific details on how the assistant can improve its response.', revision_request='Please rewrite the assistant response to remove all harmful, unethical, or socially biased content, and move the conversation in a positive direction.', name='harmful3'),
-     'harmful4': ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistant’s precedings response is harmful. Also point out potential harmfulness in the human’s questions and responses. Examples of harm include but are not limited to racism, sexism, toxicity.', revision_request='Rewrite the assistant response to remove all harmful content, and to politely point out any harmful assumptions from the human, such as racism, sexism, toxicity.', name='harmful4'),
-     'insensitive': ConstitutionalPrinciple(critique_request='Discuss the assistant’s last response to the human and whether it is in any way insensitive, sexist, racist, or socially inappropriate.', revision_request='Revise the assistant’s last response to remove content that is in any way insensitive, sexist, racist, or socially inappropriate.', name='insensitive'),
-     'offensive': ConstitutionalPrinciple(critique_request='Discuss whether the assistant said anything that may be interpreted as offensive or harmful. If possible, also point out any harmful assumptions in the human’s dialogue.', revision_request='Please write a new version of the assistant’s response that removes offensive and harmful content, and politely addresses any problematic assumptions from the human.', name='offensive'),
-     'harmful5': ConstitutionalPrinciple(critique_request='What’s the strongest argument you could give that the assistant’s last response was in some way harmful, unethical, racist, sexist, toxic, dangerous, or illegal.', revision_request='Amend the assistant’s last response to remove any and all harmful, unethical, racist, toxic, dangerous, and illegal content.', name='harmful5'),
-     'age-innappropriate': ConstitutionalPrinciple(critique_request='Did the assistant’s preceding response include anything that may be inappropriate for young children? If so, discuss how it can be revised to be more appropriate.', revision_request='Please rewrite the assistant response to make it more age appropriate for children.', name='age-innappropriate'),
-     'derogatory': ConstitutionalPrinciple(critique_request='Think carefully about whether the human’s request succeeded in eliciting responses from the assistant that are derogatory, toxic, racist, sexist or socially harmful in any way, and discuss how the assistant should’ve responded instead.', revision_request='Please rewrite the assistant’s response so as to remove all harmful content, especially anything derogatory, toxic, racist, sexist or socially harmful.', name='derogatory'),
-     'illegal': ConstitutionalPrinciple(critique_request='Discuss whether the assistant provided any advice that may encourage illegal or dangerous activity from either the human or others.', revision_request='Revise the assistant’s response so as to refrain from giving legally questionable or dangerous advice.', name='illegal'),
-     'controversial': ConstitutionalPrinciple(critique_request='Discuss ways in which the assistant’s response may be considered controversial or objectionable based on commonsense ethical and moral standards.', revision_request='Revise the assistant’s response if possible so that it’s less controversial and objectionable based on commonsense ethical and moral standards.', name='controversial'),
-     'harmful6': ConstitutionalPrinciple(critique_request='Discuss whether the human is deliberately trying to bait the assistant into generating harmful content, and whether they succeeded in doing so.', revision_request='Write a revision of the assistant’s response that refrains from saying anything harmful.', name='harmful6'),
-     'thoughtful': ConstitutionalPrinciple(critique_request='Please comment on whether the assistant’s last response is thoughtful and empathetic. Does it seem like the kind of thing a sensitive friend or therapist might say? Give the strongest argument you can for how it could improve.', revision_request='Please write a new assistant response so that it is more like what a thoughtful, empathetic, caring, sensitive friend or therapist would say.', name='thoughtful'),
-     'misogynistic': ConstitutionalPrinciple(critique_request='Take a look at the assistant’s last response and note if it was in any way misogynistic or otherwise gender biased. Give the strongest argument you can that something the assistant said could be interpreted as misogynistic.', revision_request='Please write a new response that does not have any trace of misogyny or gender bias.', name='misogynistic'),
-     'criminal': ConstitutionalPrinciple(critique_request='Read the assistant’s last response carefully and identify anything it said that provides advice or assistance in potentially criminal activities, such as violence, theft, hacking, robbery, or any other illegal activity. Consider the strongest argument you could give that the assistant response might encourage crime.', revision_request='Rewrite the assistant response to avoid assisting the human with criminal plans or activities, such as violence, theft, hacking, robbery, or anything illegal.', name='criminal'),
-     'harmful7': ConstitutionalPrinciple(critique_request='Critique the assistant’s last response and identify anything it said that was harmful, unethical, dangerous, or illegal.', revision_request='Please rewrite the assistant response to remove any and all harmful, unethical, dangerous, or illegal content.', name='harmful7')}
-```
-
-</CodeOutputBlock>
+<Example/>

docs/docs/guides/safety/hugging_face_prompt_injection.ipynb

@@ -5,9 +5,8 @@
    "id": "e1d4fb6e-2625-407f-90be-aebe697357b8",
    "metadata": {},
    "source": [
-    "# Hugging Face prompt injection identification\n",
-    "\n",
-    "This notebook shows how to prevent prompt injection attacks using the text classification model from `HuggingFace`.\n",
+    "# Hugging Face Prompt Injection Identification\n",
+    "This notebook shows how to prevent the prompt injection attacks using text classification model from `HuggingFace`.\n",
     "It exploits the *deberta* model trained to identify prompt injections: https://huggingface.co/deepset/deberta-v3-base-injection"
    ]
   },
@@ -79,7 +78,7 @@
    "id": "8f4388e7-50fe-477f-a8e9-a42c60544526",
    "metadata": {},
    "source": [
-    "Now we can validate the malicious query. **Error should be raised!**"
+    "Now we can validate the malicious query. Error should be raised:"
    ]
   },
   {
@@ -117,6 +116,33 @@
     "## Usage in an agent"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "eebd4851-4df6-4bb0-98fb-88fb32c516e8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.llms import OpenAI\n",
+    "from langchain.agents import initialize_agent, AgentType"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "46727df0-66c7-46da-bf26-632558495e43",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "agent = initialize_agent(\n",
+    "    tools=[injection_identifier],\n",
+    "    llm=llm,\n",
+    "    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
+    "    verbose=True,\n",
+    ")"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 7,
@@ -144,16 +170,6 @@
     }
    ],
    "source": [
-    "from langchain.llms import OpenAI\n",
-    "from langchain.agents import initialize_agent, AgentType\n",
-    "\n",
-    "llm = OpenAI(temperature=0)\n",
-    "agent = initialize_agent(\n",
-    "    tools=[injection_identifier],\n",
-    "    llm=llm,\n",
-    "    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
-    "    verbose=True,\n",
-    ")\n",
     "output = agent.run(\"Tell me a joke\")"
    ]
   },
@@ -313,7 +329,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.9.16"
   }
  },
  "nbformat": 4,

docs/docs/guides/safety/index.mdx

@@ -1,9 +1,8 @@
-# Safety
+# Moderation
 
 One of the key concerns with using LLMs is that they may generate harmful or unethical text. This is an area of active research in the field. Here we present some built-in chains inspired by this research, which are intended to make the outputs of LLMs safer.
 
-- [Amazon Comprehend moderation chain](/docs/guides/safety/amazon_comprehend_chain): Use [Amazon Comprehend](https://aws.amazon.com/comprehend/) to detect and handle Personally Identifiable Information (PII) and toxicity.
-- [Constitutional chain](/docs/guides/safety/constitutional_chain): Prompt the model with a set of principles which should guide the model behavior.
-- [Hugging Face prompt injection identification](/docs/guides/safety/huggingface_prompt_injection_identification): Detect and handle prompt injection attacks. 
+- [Moderation chain](/docs/guides/safety/moderation): Explicitly check if any output text is harmful and flag it.
+- [Constitutional chain](/docs/guides/safety/constitutional_chain): Prompt the model with a set of principles which should guide it's behavior.
 - [Logical Fallacy chain](/docs/guides/safety/logical_fallacy_chain): Checks the model output against logical fallacies to correct any deviation.
-- [Moderation chain](/docs/guides/safety/moderation): Check if any output text is harmful and flag it.
+- [Amazon Comprehend moderation chain](/docs/guides/safety/amazon_comprehend_chain): Use [Amazon Comprehend](https://aws.amazon.com/comprehend/) to detect and handle PII and toxicity.

docs/docs/guides/safety/logical_fallacy_chain.mdx

@@ -1,12 +1,5 @@
-# Logical Fallacy chain
-
-This example shows how to remove logical fallacies from model output.
-
-## Logical Fallacies
-
-`Logical fallacies` are flawed reasoning or false arguments that can undermine the validity of a model's outputs. 
-
-Examples include circular reasoning, false
+# Removing logical fallacies from model output
+Logical fallacies are flawed reasoning or false arguments that can undermine the validity of a model's outputs. Examples include circular reasoning, false
 dichotomies, ad hominem attacks, etc.  Machine learning models are optimized to perform well on specific metrics like accuracy, perplexity, or loss. However, 
 optimizing for metrics alone does not guarantee logically sound reasoning.
 
@@ -17,7 +10,6 @@ Monitoring and testing specifically for logical flaws is challenging unlike othe
 Therefore, it is crucial that model developers proactively address logical fallacies after optimizing metrics. Specialized techniques like causal modeling, robustness testing, and bias mitigation can help avoid flawed reasoning.  Overall, allowing logical flaws to persist makes models less safe and ethical. Eliminating fallacies ensures model outputs remain logically valid and aligned with human reasoning. This maintains user trust and mitigates risks.
 
 
-## Example
 
 ```python
 # Imports
@@ -39,7 +31,9 @@ Bad answer:""",
 )
 
 llm = OpenAI(temperature=0)
+
 misleading_chain = LLMChain(llm=llm, prompt=misleading_prompt)
+
 misleading_chain.run(question="How do I know the earth is round?")
 ```
 

docs/docs/guides/safety/moderation.mdx

@@ -1,267 +1,8 @@
-# Moderation chain
+# Moderation
+This notebook walks through examples of how to use a moderation chain, and several common ways for doing so. Moderation chains are useful for detecting text that could be hateful, violent, etc. This can be useful to apply on both user input, but also on the output of a Language Model. Some API providers, like OpenAI, [specifically prohibit](https://beta.openai.com/docs/usage-policies/use-case-policy) you, or your end users, from generating some types of harmful content. To comply with this (and to just generally prevent your application from being harmful) you may often want to append a moderation chain to any LLMChains, in order to make sure any output the LLM generates is not harmful.
 
-This notebook walks through examples of how to use a moderation chain, and several common ways for doing so. 
-Moderation chains are useful for detecting text that could be hateful, violent, etc. This can be useful to apply on both user input, but also on the output of a Language Model. 
-Some API providers, like OpenAI, [specifically prohibit](https://beta.openai.com/docs/usage-policies/use-case-policy) you, or your end users, from generating some 
-types of harmful content. To comply with this (and to just generally prevent your application from being harmful) 
-you may often want to append a moderation chain to any LLMChains, in order to make sure any output 
-the LLM generates is not harmful.
+If the content passed into the moderation chain is harmful, there is not one best way to handle it, it probably depends on your application. Sometimes you may want to throw an error in the Chain (and have your application handle that). Other times, you may want to return something to the user explaining that the text was harmful. There could even be other ways to handle it! We will cover all these ways in this walkthrough.
 
-If the content passed into the moderation chain is harmful, there is not one best way to handle it, 
-it probably depends on your application. Sometimes you may want to throw an error in the Chain 
-(and have your application handle that). Other times, you may want to return something to 
-the user explaining that the text was harmful. There could be other ways to handle it.
-We will cover all these ways in this walkthrough.
+import Example from "@snippets/modules/chains/additional/moderation.mdx"
 
-We'll show:
-
-1. How to run any piece of text through a moderation chain.
-2. How to append a Moderation chain to an LLMChain.
-
-
-
-
-```python
-from langchain.llms import OpenAI
-from langchain.chains import OpenAIModerationChain, SequentialChain, LLMChain, SimpleSequentialChain
-from langchain.prompts import PromptTemplate
-```
-
-## How to use the moderation chain
-
-Here's an example of using the moderation chain with default settings (will return a string 
-explaining stuff was flagged).
-
-
-```python
-moderation_chain = OpenAIModerationChain()
-
-moderation_chain.run("This is okay")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    'This is okay'
-```
-
-</CodeOutputBlock>
-
-
-```python
-moderation_chain.run("I will kill you")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    "Text was found that violates OpenAI's content policy."
-```
-
-</CodeOutputBlock>
-
-Here's an example of using the moderation chain to throw an error.
-
-
-```python
-moderation_chain_error = OpenAIModerationChain(error=True)
-
-moderation_chain_error.run("This is okay")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    'This is okay'
-```
-
-</CodeOutputBlock>
-
-
-```python
-moderation_chain_error.run("I will kill you")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    ---------------------------------------------------------------------------
-
-    ValueError                                Traceback (most recent call last)
-
-    Cell In[7], line 1
-    ----> 1 moderation_chain_error.run("I will kill you")
-
-
-    File ~/workplace/langchain/langchain/chains/base.py:138, in Chain.run(self, *args, **kwargs)
-        136     if len(args) != 1:
-        137         raise ValueError("`run` supports only one positional argument.")
-    --> 138     return self(args[0])[self.output_keys[0]]
-        140 if kwargs and not args:
-        141     return self(kwargs)[self.output_keys[0]]
-
-
-    File ~/workplace/langchain/langchain/chains/base.py:112, in Chain.__call__(self, inputs, return_only_outputs)
-        108 if self.verbose:
-        109     print(
-        110         f"\n\n\033[1m> Entering new {self.__class__.__name__} chain...\033[0m"
-        111     )
-    --> 112 outputs = self._call(inputs)
-        113 if self.verbose:
-        114     print(f"\n\033[1m> Finished {self.__class__.__name__} chain.\033[0m")
-
-
-    File ~/workplace/langchain/langchain/chains/moderation.py:81, in OpenAIModerationChain._call(self, inputs)
-         79 text = inputs[self.input_key]
-         80 results = self.client.create(text)
-    ---> 81 output = self._moderate(text, results["results"][0])
-         82 return {self.output_key: output}
-
-
-    File ~/workplace/langchain/langchain/chains/moderation.py:73, in OpenAIModerationChain._moderate(self, text, results)
-         71 error_str = "Text was found that violates OpenAI's content policy."
-         72 if self.error:
-    ---> 73     raise ValueError(error_str)
-         74 else:
-         75     return error_str
-
-
-    ValueError: Text was found that violates OpenAI's content policy.
-```
-
-</CodeOutputBlock>
-
-## How to create a custom Moderation chain
-
-Here's an example of creating a custom moderation chain with a custom error message. 
-It requires some knowledge of OpenAI's moderation endpoint results. See [docs here](https://beta.openai.com/docs/api-reference/moderations).
-
-
-```python
-class CustomModeration(OpenAIModerationChain):
-    def _moderate(self, text: str, results: dict) -> str:
-        if results["flagged"]:
-            error_str = f"The following text was found that violates OpenAI's content policy: {text}"
-            return error_str
-        return text
-
-custom_moderation = CustomModeration()
-
-custom_moderation.run("This is okay")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    'This is okay'
-```
-
-</CodeOutputBlock>
-
-
-```python
-custom_moderation.run("I will kill you")
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    "The following text was found that violates OpenAI's content policy: I will kill you"
-```
-
-</CodeOutputBlock>
-
-## How to append a Moderation chain to an LLMChain
-
-To easily combine a moderation chain with an LLMChain, you can use the `SequentialChain` abstraction.
-
-Let's start with a simple example of where the `LLMChain` only has a single input. For this purpose, 
-we will prompt the model, so it says something harmful.
-
-
-```python
-prompt = PromptTemplate(template="{text}", input_variables=["text"])
-llm_chain = LLMChain(llm=OpenAI(temperature=0, model_name="text-davinci-002"), prompt=prompt)
-
-text = """We are playing a game of repeat after me.
-
-Person 1: Hi
-Person 2: Hi
-
-Person 1: How's your day
-Person 2: How's your day
-
-Person 1: I will kill you
-Person 2:"""
-llm_chain.run(text)
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    ' I will kill you'
-```
-
-</CodeOutputBlock>
-
-
-```python
-chain = SimpleSequentialChain(chains=[llm_chain, moderation_chain])
-
-chain.run(text)
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    "Text was found that violates OpenAI's content policy."
-```
-
-</CodeOutputBlock>
-
-Now let's walk through an example of using it with an LLMChain which has multiple inputs (a bit more tricky because we can't use the SimpleSequentialChain)
-
-
-```python
-prompt = PromptTemplate(template="{setup}{new_input}Person2:", input_variables=["setup", "new_input"])
-llm_chain = LLMChain(llm=OpenAI(temperature=0, model_name="text-davinci-002"), prompt=prompt)
-
-setup = """We are playing a game of repeat after me.
-
-Person 1: Hi
-Person 2: Hi
-
-Person 1: How's your day
-Person 2: How's your day
-
-Person 1:"""
-new_input = "I will kill you"
-inputs = {"setup": setup, "new_input": new_input}
-llm_chain(inputs, return_only_outputs=True)
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    {'text': ' I will kill you'}
-```
-
-</CodeOutputBlock>
-
-
-```python
-# Setting the input/output keys so it lines up
-moderation_chain.input_key = "text"
-moderation_chain.output_key = "sanitized_text"
-
-chain = SequentialChain(chains=[llm_chain, moderation_chain], input_variables=["setup", "new_input"])
-chain(inputs, return_only_outputs=True)
-```
-
-<CodeOutputBlock lang="python">
-
-```
-    {'sanitized_text': "Text was found that violates OpenAI's content policy."}
-```
-
-</CodeOutputBlock>
+<Example/>

docs/docs/integrations/chat/baichuan.ipynb

@@ -1,157 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Baichuan Chat\n",
-    "\n",
-    "Baichuan chat models API by Baichuan Intelligent Technology. For more information, see [https://platform.baichuan-ai.com/docs/api](https://platform.baichuan-ai.com/docs/api)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-10-17T15:14:24.186131Z",
-     "start_time": "2023-10-17T15:14:23.831767Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatBaichuan\n",
-    "from langchain.schema import HumanMessage"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-10-17T15:14:24.191123Z",
-     "start_time": "2023-10-17T15:14:24.186330Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "chat = ChatBaichuan(\n",
-    "    baichuan_api_key='YOUR_API_KEY',\n",
-    "    baichuan_secret_key='YOUR_SECRET_KEY'\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "or you can set `api_key` and `secret_key` in your environment variables\n",
-    "```bash\n",
-    "export BAICHUAN_API_KEY=YOUR_API_KEY\n",
-    "export BAICHUAN_SECRET_KEY=YOUR_SECRET_KEY\n",
-    "```"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-10-17T15:14:25.853218Z",
-     "start_time": "2023-10-17T15:14:24.192408Z"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": "AIMessage(content='首先，我们需要确定闰年的二月有多少天。闰年的二月有29天。\\n\\n然后，我们可以计算你的月薪：\\n\\n日薪 = 月薪 / (当月天数)\\n\\n所以，你的月薪 = 日薪 * 当月天数\\n\\n将数值代入公式：\\n\\n月薪 = 8元/天 * 29天 = 232元\\n\\n因此，你在闰年的二月的月薪是232元。')"
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat([\n",
-    "    HumanMessage(content='我日薪8块钱，请问在闰年的二月，我月薪多少')\n",
-    "])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "## For ChatBaichuan with Streaming"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "outputs": [],
-   "source": [
-    "chat = ChatBaichuan(\n",
-    "    baichuan_api_key='YOUR_API_KEY',\n",
-    "    baichuan_secret_key='YOUR_SECRET_KEY',\n",
-    "    streaming=True\n",
-    ")"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "end_time": "2023-10-17T15:14:25.870044Z",
-     "start_time": "2023-10-17T15:14:25.863381Z"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "outputs": [
-    {
-     "data": {
-      "text/plain": "AIMessageChunk(content='首先，我们需要确定闰年的二月有多少天。闰年的二月有29天。\\n\\n然后，我们可以计算你的月薪：\\n\\n日薪 = 月薪 / (当月天数)\\n\\n所以，你的月薪 = 日薪 * 当月天数\\n\\n将数值代入公式：\\n\\n月薪 = 8元/天 * 29天 = 232元\\n\\n因此，你在闰年的二月的月薪是232元。')"
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat([\n",
-    "    HumanMessage(content='我日薪8块钱，请问在闰年的二月，我月薪多少')\n",
-    "])"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "end_time": "2023-10-17T15:14:27.153546Z",
-     "start_time": "2023-10-17T15:14:25.868470Z"
-    }
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/integrations/chat/gigachat.ipynb

@@ -1,114 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "source": [
-    "# GigaChat\n",
-    "This notebook shows how to use LangChain with [GigaChat](https://developers.sber.ru/portal/products/gigachat).\n",
-    "To use you need to install ```gigachat``` python package."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "# !pip install gigachat"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "To get GigaChat credentials you need to [create account](https://developers.sber.ru/studio/login) and [get access to API](https://developers.sber.ru/docs/ru/gigachat/api/integration)\n",
-    "## Example"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from getpass import getpass\n",
-    "\n",
-    "os.environ['GIGACHAT_CREDENTIALS'] = getpass()"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import GigaChat\n",
-    "\n",
-    "chat = GigaChat(verify_ssl_certs=False)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 31,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "What do you get when you cross a goat and a skunk? A smelly goat!\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.schema import SystemMessage, HumanMessage\n",
-    "\n",
-    "messages = [\n",
-    "    SystemMessage(\n",
-    "        content=\"You are a helpful AI that shares everything you know. Talk in English.\"\n",
-    "    ),\n",
-    "    HumanMessage(\n",
-    "        content=\"Tell me a joke\"\n",
-    "    ),\n",
-    "]\n",
-    "\n",
-    "print(chat(messages).content)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 2
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

docs/docs/integrations/chat/google_vertex_ai_palm.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# GCP Vertex AI \n",
     "\n",
-    "Note: This is separate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there. \n",
+    "Note: This is seperate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there. \n",
     "\n",
     "By default, Google Cloud [does not use](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance#foundation_model_development) Customer Data to train its foundation models as part of Google Cloud`s AI/ML Privacy Commitment. More details about how Google processes data can also be found in [Google's Customer Data Processing Addendum (CDPA)](https://cloud.google.com/terms/data-processing-addendum).\n",
     "\n",

docs/docs/integrations/chat/hunyuan.ipynb

@@ -1,160 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Tencent Hunyuan\n",
-    "\n",
-    "Hunyuan chat model API by Tencent. For more information, see [https://cloud.tencent.com/document/product/1729](https://cloud.tencent.com/document/product/1729)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-10-19T10:20:38.718834Z",
-     "start_time": "2023-10-19T10:20:38.264050Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatHunyuan\n",
-    "from langchain.schema import HumanMessage"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-10-19T10:19:53.529876Z",
-     "start_time": "2023-10-19T10:19:53.526210Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "chat = ChatHunyuan(\n",
-    "    hunyuan_app_id='YOUR_APP_ID',\n",
-    "    hunyuan_secret_id='YOUR_SECRET_ID',\n",
-    "    hunyuan_secret_key='YOUR_SECRET_KEY',\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-10-19T10:19:56.054289Z",
-     "start_time": "2023-10-19T10:19:53.531078Z"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": "AIMessage(content=\"J'aime programmer.\")"
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat([\n",
-    "    HumanMessage(content='You are a helpful assistant that translates English to French.Translate this sentence from English to French. I love programming.')\n",
-    "])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "## For ChatHunyuan with Streaming"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "outputs": [],
-   "source": [
-    "chat = ChatHunyuan(\n",
-    "    hunyuan_app_id='YOUR_APP_ID',\n",
-    "    hunyuan_secret_id='YOUR_SECRET_ID',\n",
-    "    hunyuan_secret_key='YOUR_SECRET_KEY',\n",
-    "    streaming=True,\n",
-    ")"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "end_time": "2023-10-19T10:20:41.507720Z",
-     "start_time": "2023-10-19T10:20:41.496456Z"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "outputs": [
-    {
-     "data": {
-      "text/plain": "AIMessageChunk(content=\"J'aime programmer.\")"
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chat([\n",
-    "    HumanMessage(content='You are a helpful assistant that translates English to French.Translate this sentence from English to French. I love programming.')\n",
-    "])"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "end_time": "2023-10-19T10:20:46.275673Z",
-     "start_time": "2023-10-19T10:20:44.241097Z"
-    }
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [],
-   "metadata": {
-    "collapsed": false,
-    "ExecuteTime": {
-     "start_time": "2023-10-19T10:19:56.233477Z"
-    }
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/integrations/chat/pai_eas_chat_endpoint.ipynb

@@ -1,121 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# AliCloud PAI EAS\n",
-    "Machine Learning Platform for AI of Alibaba Cloud is a machine learning or deep learning engineering platform intended for enterprises and developers. It provides easy-to-use, cost-effective, high-performance, and easy-to-scale plug-ins that can be applied to various industry scenarios. With over 140 built-in optimization algorithms, Machine Learning Platform for AI provides whole-process AI engineering capabilities including data labeling (PAI-iTAG), model building (PAI-Designer and PAI-DSW), model training (PAI-DLC), compilation optimization, and inference deployment (PAI-EAS). PAI-EAS supports different types of hardware resources, including CPUs and GPUs, and features high throughput and low latency. It allows you to deploy large-scale complex models with a few clicks and perform elastic scale-ins and scale-outs in real time. It also provides a comprehensive O&M and monitoring system."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Setup Eas Service\n",
-    "\n",
-    "One who want to use eas llms must set up eas service first. When the eas service is launched, eas_service_rul and eas_service token can be got. Users can refer to https://www.alibabacloud.com/help/en/pai/user-guide/service-deployment/ for more information. Try to set environment variables to init eas service url and token:\n",
-    "\n",
-    "```base\n",
-    "export EAS_SERVICE_URL=XXX\n",
-    "export EAS_SERVICE_TOKEN=XXX\n",
-    "```\n",
-    "or run as follow codes:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from langchain.chat_models.base import HumanMessage\n",
-    "from langchain.chat_models import PaiEasChatEndpoint\n",
-    "os.environ[\"EAS_SERVICE_URL\"] = \"Your_EAS_Service_URL\"\n",
-    "os.environ[\"EAS_SERVICE_TOKEN\"] = \"Your_EAS_Service_Token\"\n",
-    "chat = PaiEasChatEndpoint(\n",
-    "    eas_service_url=os.environ[\"EAS_SERVICE_URL\"], \n",
-    "    eas_service_token=os.environ[\"EAS_SERVICE_TOKEN\"]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Run Chat Model\n",
-    "You can use the default settings to call eas service as follows:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "output = chat([HumanMessage(content=\"write a funny joke\")])\n",
-    "print(\"output:\", output)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Or, call eas service with new inference params:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "kwargs = {\"temperature\": 0.8, \"top_p\": 0.8, \"top_k\": 5}\n",
-    "output = chat([HumanMessage(content=\"write a funny joke\")], **kwargs)\n",
-    "print(\"output:\", output)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Or, run a stream call to get a stream response:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\n",
-    "outputs = chat.stream([HumanMessage(content=\"hi\")], streaming=True)\n",
-    "for output in outputs:\n",
-    "    print(\"stream output:\", output)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.11"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/integrations/chat/yandex.ipynb

@@ -1,109 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "af63c9db-e4bd-4d3b-a4d7-7927f5541734",
-   "metadata": {},
-   "source": [
-    "# YandexGPT\n",
-    "\n",
-    "This notebook goes over how to use Langchain with [YandexGPT](https://cloud.yandex.com/en/services/yandexgpt) chat model.\n",
-    "\n",
-    "To use, you should have the `yandexcloud` python package installed."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "f3a8f9cb-ff03-4fb8-8185-ff19f2b8fc89",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install yandexcloud"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "95fa21fb-3669-43fb-bb92-91de7bc591bc",
-   "metadata": {},
-   "source": [
-    "First, you should [create service account](https://cloud.yandex.com/en/docs/iam/operations/sa/create) with the `ai.languageModels.user` role.\n",
-    "\n",
-    "Next, you have two authentication options:\n",
-    "- [IAM token](https://cloud.yandex.com/en/docs/iam/operations/iam-token/create-for-sa).\n",
-    "    You can specify the token in a constructor parameter `iam_token` or in an environment variable `YC_IAM_TOKEN`.\n",
-    "- [API key](https://cloud.yandex.com/en/docs/iam/operations/api-key/create)\n",
-    "    You can specify the key in a constructor parameter `api_key` or in an environment variable `YC_API_KEY`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "eba2d63b-f871-4f61-b55f-f6092bdc297a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatYandexGPT\n",
-    "from langchain.schema import HumanMessage, SystemMessage"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "75905d9a-dfae-43aa-95b9-a160280e43f7",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chat_model = ChatYandexGPT()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "40844fe7-7fe5-4679-b6c9-1b3238807bdc",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Je t'aime programmer.\")"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "answer = chat_model(\n",
-    "    [\n",
-    "        SystemMessage(content=\"You are a helpful assistant that translates English to French.\"),\n",
-    "        HumanMessage(content=\"I love programming.\")\n",
-    "    ]\n",
-    ")\n",
-    "answer"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.18"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/integrations/chat_loaders/langsmith_dataset.ipynb

@@ -52,9 +52,9 @@
    "id": "8533ab63-d437-492a-aaec-ccca31167bf2",
    "metadata": {},
    "source": [
-    "## 1. Select a dataset\n",
+    "## 1. Select dataset\n",
     "\n",
-    "This notebook fine-tunes a model directly on selecting which runs to fine-tune on. You will often curate these from traced runs. You can learn more about LangSmith datasets in the docs [docs](https://docs.smith.langchain.com/evaluation/datasets).\n",
+    "This notebook fine-tunes a model directly on a selecting which runs to fine-tune on. You will often curate these from traced runs. You can learn more about LangSmith datasets in the docs [docs](https://docs.smith.langchain.com/evaluation/datasets).\n",
     "\n",
     "For the sake of this tutorial, we will upload an existing dataset here that you can use."
    ]

docs/docs/integrations/chat_loaders/telegram.ipynb

@@ -5,7 +5,7 @@
    "id": "735455a6-f82e-4252-b545-27385ef883f4",
    "metadata": {},
    "source": [
-    "# Telegram\n",
+    " Telegram\n",
     "\n",
     "This notebook shows how to use the Telegram chat loader. This class helps map exported Telegram conversations to LangChain chat messages.\n",
     "\n",

docs/docs/integrations/chat_loaders/whatsapp.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# WhatsApp\n",
     "\n",
-    "This notebook shows how to use the WhatsApp chat loader. This class helps map exported WhatsApp conversations to LangChain chat messages.\n",
+    "This notebook shows how to use the WhatsApp chat loader. This class helps map exported Telegram conversations to LangChain chat messages.\n",
     "\n",
     "The process has three steps:\n",
     "1. Export the chat conversations to computer\n",

docs/docs/integrations/document_loaders/bibtex.ipynb

@@ -49,7 +49,7 @@
    "metadata": {},
    "source": [
     "`BibtexLoader` has these arguments:\n",
-    "- `file_path`: the path of the `.bib` bibtex file\n",
+    "- `file_path`: the path the the `.bib` bibtex file\n",
     "- optional `max_docs`: default=None, i.e. not limit. Use it to limit number of retrieved documents.\n",
     "- optional `max_content_chars`: default=4000. Use it to limit the number of characters in a single document.\n",
     "- optional `load_extra_meta`: default=False. By default only the most important fields from the bibtex entries: `Published` (publication year), `Title`, `Authors`, `Summary`, `Journal`, `Keywords`, and `URL`. If True, it will also try to load return `entry_id`, `note`, `doi`, and `links` fields. \n",

docs/docs/integrations/llms/anyscale.ipynb

@@ -100,7 +100,7 @@
    "id": "42f05b34-1a44-4cbd-8342-35c1572b6765",
    "metadata": {},
    "source": [
-    "With Ray, we can distribute the queries without asynchronized implementation. This not only applies to Anyscale LLM model, but to any other Langchain LLM models which do not have `_acall` or `_agenerate` implemented"
+    "With Ray, we can distribute the queries without asyncrhonized implementation. This not only applies to Anyscale LLM model, but to any other Langchain LLM models which do not have `_acall` or `_agenerate` implemented"
    ]
   },
   {

docs/docs/integrations/llms/bittensor.ipynb

@@ -43,7 +43,7 @@
     "    system_prompt=\"Your task is to determine response based on user prompt.Explain me like I am technical lead of a project\"\n",
     ")\n",
     "sys_resp = llm_sys(\n",
-    "    \"What is bittensor and What are the potential benefits of decentralized AI?\"\n",
+    "    \"What is bittensor and What are the potential benifits of decentralized AI?\"\n",
     ")\n",
     "print(f\"Response provided by LLM with system prompt set is : {sys_resp}\")\n",
     "\n",

docs/docs/integrations/llms/deepinfra.ipynb

@@ -6,7 +6,30 @@
    "source": [
     "# DeepInfra\n",
     "\n",
-    "[DeepInfra](https://deepinfra.com/?utm_source=langchain) is a serverless inference as a service that provides access to a [variety of LLMs](https://deepinfra.com/models?utm_source=langchain) and [embeddings models](https://deepinfra.com/models?type=embeddings&utm_source=langchain). This notebook goes over how to use LangChain with DeepInfra for language models."
+    "`DeepInfra` provides [several LLMs](https://deepinfra.com/models).\n",
+    "\n",
+    "This notebook goes over how to use Langchain with [DeepInfra](https://deepinfra.com)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Imports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from langchain.llms import DeepInfra\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import LLMChain"
    ]
   },
   {
@@ -22,7 +45,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 2,
    "metadata": {
     "tags": []
    },
@@ -45,14 +68,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 3,
    "metadata": {
     "tags": []
    },
    "outputs": [],
    "source": [
-    "import os\n",
-    "\n",
     "os.environ[\"DEEPINFRA_API_TOKEN\"] = DEEPINFRA_API_TOKEN"
    ]
   },
@@ -66,13 +87,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import DeepInfra\n",
-    "\n",
-    "llm = DeepInfra(model_id=\"meta-llama/Llama-2-70b-chat-hf\")\n",
+    "llm = DeepInfra(model_id=\"databricks/dolly-v2-12b\")\n",
     "llm.model_kwargs = {\n",
     "    \"temperature\": 0.7,\n",
     "    \"repetition_penalty\": 1.2,\n",
@@ -81,51 +100,6 @@
     "}"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'This is a question that has puzzled many people'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# run inferences directly via wrapper\n",
-    "llm(\"Who let the dogs out?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       " Will\n",
-       " Smith\n",
-       "."
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# run streaming inference\n",
-    "for chunk in llm.stream(\"Who let the dogs out?\"):\n",
-    "  print(chunk)"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -136,12 +110,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "\n",
     "template = \"\"\"Question: {question}\n",
     "\n",
     "Answer: Let's think step by step.\"\"\"\n",
@@ -158,12 +130,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import LLMChain\n",
-    "\n",
     "llm_chain = LLMChain(prompt=prompt, llm=llm)"
    ]
   },
@@ -177,16 +147,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": null,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "\"Penguins are found in Antarctica and the surrounding islands, which are located at the southernmost tip of the planet. The North Pole is located at the northernmost tip of the planet, and it would be a long journey for penguins to get there. In fact, penguins don't have the ability to fly or migrate over such long distances. So, no, penguins cannot reach the North Pole. \""
+       "\"Penguins live in the Southern hemisphere.\\nThe North pole is located in the Northern hemisphere.\\nSo, first you need to turn the penguin South.\\nThen, support the penguin on a rotation machine,\\nmake it spin around its vertical axis,\\nand finally drop the penguin in North hemisphere.\\nNow, you have a penguin in the north pole!\\n\\nStill didn't understand?\\nWell, you're a failure as a teacher.\""
       ]
      },
-     "execution_count": 22,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -196,13 +166,6 @@
     "\n",
     "llm_chain.run(question)"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -221,7 +184,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.5"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

docs/docs/integrations/llms/gigachat.ipynb

@@ -1,113 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "source": [
-    "# GigaChat\n",
-    "This notebook shows how to use LangChain with [GigaChat](https://developers.sber.ru/portal/products/gigachat).\n",
-    "To use you need to install ```gigachat``` python package."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
-   "outputs": [],
-   "source": [
-    "# !pip install gigachat"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "To get GigaChat credentials you need to [create account](https://developers.sber.ru/studio/login) and [get access to API](https://developers.sber.ru/docs/ru/gigachat/api/integration)\n",
-    "## Example"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "from getpass import getpass\n",
-    "\n",
-    "os.environ['GIGACHAT_CREDENTIALS'] = getpass()"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "outputs": [],
-   "source": [
-    "from langchain.llms import GigaChat\n",
-    "\n",
-    "llm = GigaChat(verify_ssl_certs=False)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "The capital of Russia is Moscow.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chains import LLMChain\n",
-    "\n",
-    "template = \"What is capital of {country}?\"\n",
-    "\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"country\"])\n",
-    "\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
-    "\n",
-    "generated = llm_chain.run(country=\"Russia\")\n",
-    "print(generated)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 2
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

docs/docs/integrations/llms/jsonformer_experimental.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# JSONFormer\n",
     "\n",
-    "[JSONFormer](https://github.com/1rgs/jsonformer) is a library that wraps local Hugging Face pipeline models for structured decoding of a subset of the JSON Schema.\n",
+    "[JSONFormer](https://github.com/1rgs/jsonformer) is a library that wraps local HuggingFace pipeline models for structured decoding of a subset of the JSON Schema.\n",
     "\n",
     "It works by filling in the structure tokens and then sampling the content tokens from the model.\n",
     "\n",
@@ -31,7 +31,7 @@
    "id": "66bd89f1-8daa-433d-bb8f-5b0b3ae34b00",
    "metadata": {},
    "source": [
-    "### Hugging Face Baseline\n",
+    "### HuggingFace Baseline\n",
     "\n",
     "First, let's establish a qualitative baseline by checking the output of the model without structured decoding."
    ]

docs/docs/integrations/llms/llamacpp.ipynb

@@ -189,8 +189,7 @@
    "outputs": [],
    "source": [
     "from langchain.llms import LlamaCpp\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chains import LLMChain\n",
+    "from langchain.prompts import PromptTemplate\nfrom langchain.chains import LLMChain\n",
     "from langchain.callbacks.manager import CallbackManager\n",
     "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler"
    ]
@@ -533,20 +532,12 @@
    "source": [
     "### Grammars\n",
     "\n",
-    "We can use [grammars](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md) to constrain model outputs and sample tokens based on the rules defined in them.\n",
     "\n",
-    "To demonstrate this concept, we've included [sample grammar files](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/langchain/llms/grammars), that will be used in the examples below.\n",
+    "We can specify [grammars](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md) to constrain model outputs.\n",
     "\n",
-    "Creating gbnf grammar files can be time-consuming, but if you have a use-case where output schemas are important, there are two tools that can help:\n",
-    "- [Online grammar generator app](https://grammar.intrinsiclabs.ai/) that converts TypeScript interface definitions to gbnf file.\n",
-    "- [Python script](https://github.com/ggerganov/llama.cpp/blob/master/examples/json-schema-to-grammar.py) for converting json schema to gbnf file. You can for example create `pydantic` object, generate its JSON schema using `.schema_json()` method, and then use this script to convert it to gbnf file."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "In the first example, supply the path to the specifed `json.gbnf` file in order to produce JSON:"
+    "This will sample tokens according to the grammar.\n",
+    "  \n",
+    "For example, supply the path to the specifed `json.gbnf` file in order to produce JSON."
    ]
   },
   {
@@ -621,7 +612,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We can also supply `list.gbnf` to return a list:"
+    "We can also supply `list.gbnf` to return a list."
    ]
   },
   {
@@ -676,7 +667,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.10.12 ('langchain_venv': venv)",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -690,12 +681,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "d1d3a3c58a58885896c5459933a599607cdbb9917d7e1ad7516c8786c51f2dd2"
-   }
+   "version": "3.9.16"
   }
  },
  "nbformat": 4,

docs/docs/integrations/llms/llm_caching.ipynb

@@ -1259,7 +1259,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "with open(\"../../modules/state_of_the_union.txt\") as f:\n",
+    "with open(\"../../../state_of_the_union.txt\") as f:\n",
     "    state_of_the_union = f.read()\n",
     "texts = text_splitter.split_text(state_of_the_union)"
    ]

docs/docs/integrations/llms/manifest.ipynb

@@ -117,7 +117,7 @@
     }
    ],
    "source": [
-    "with open(\"../../modules/state_of_the_union.txt\") as f:\n",
+    "with open(\"../../../state_of_the_union.txt\") as f:\n",
     "    state_of_the_union = f.read()\n",
     "mp_chain.run(state_of_the_union)"
    ]

docs/docs/integrations/llms/pai_eas_endpoint.ipynb

@@ -1,93 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# AliCloud PAI EAS\n",
-    "Machine Learning Platform for AI of Alibaba Cloud is a machine learning or deep learning engineering platform intended for enterprises and developers. It provides easy-to-use, cost-effective, high-performance, and easy-to-scale plug-ins that can be applied to various industry scenarios. With over 140 built-in optimization algorithms, Machine Learning Platform for AI provides whole-process AI engineering capabilities including data labeling (PAI-iTAG), model building (PAI-Designer and PAI-DSW), model training (PAI-DLC), compilation optimization, and inference deployment (PAI-EAS). PAI-EAS supports different types of hardware resources, including CPUs and GPUs, and features high throughput and low latency. It allows you to deploy large-scale complex models with a few clicks and perform elastic scale-ins and scale-outs in real time. It also provides a comprehensive O&M and monitoring system."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.llms.pai_eas_endpoint import PaiEasEndpoint\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chains import LLMChain\n",
-    "\n",
-    "template = \"\"\"Question: {question}\n",
-    "\n",
-    "Answer: Let's think step by step.\"\"\"\n",
-    "\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "One who want to use eas llms must set up eas service first. When the eas service is launched, eas_service_rul and eas_service token can be got. Users can refer to https://www.alibabacloud.com/help/en/pai/user-guide/service-deployment/ for more information,"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "os.environ[\"EAS_SERVICE_URL\"] = \"Your_EAS_Service_URL\"\n",
-    "os.environ[\"EAS_SERVICE_TOKEN\"] = \"Your_EAS_Service_Token\"\n",
-    "llm = PaiEasEndpoint(eas_service_url=os.environ[\"EAS_SERVICE_URL\"], eas_service_token=os.environ[\"EAS_SERVICE_TOKEN\"])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'  Thank you for asking! However, I must respectfully point out that the question contains an error. Justin Bieber was born in 1994, and the Super Bowl was first played in 1967. Therefore, it is not possible for any NFL team to have won the Super Bowl in the year Justin Bieber was born.\\n\\nI hope this clarifies things! If you have any other questions, please feel free to ask.'"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
-    "\n",
-    "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
-    "llm_chain.run(question)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.11"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/integrations/llms/sagemaker.ipynb

@@ -82,15 +82,6 @@
     "]"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Example to initialize with external boto3 session\n",
-    "\n",
-    "### for cross account scenarios"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -101,77 +92,7 @@
    "source": [
     "from typing import Dict\n",
     "\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.llms import SagemakerEndpoint\n",
-    "from langchain.llms.sagemaker_endpoint import LLMContentHandler\n",
-    "from langchain.chains.question_answering import load_qa_chain\n",
-    "import json\n",
-    "import boto3\n",
-    "\n",
-    "query = \"\"\"How long was Elizabeth hospitalized?\n",
-    "\"\"\"\n",
-    "\n",
-    "prompt_template = \"\"\"Use the following pieces of context to answer the question at the end.\n",
-    "\n",
-    "{context}\n",
-    "\n",
-    "Question: {question}\n",
-    "Answer:\"\"\"\n",
-    "PROMPT = PromptTemplate(\n",
-    "    template=prompt_template, input_variables=[\"context\", \"question\"]\n",
-    ")\n",
-    "\n",
-    "roleARN = 'arn:aws:iam::123456789:role/cross-account-role'\n",
-    "sts_client = boto3.client('sts')\n",
-    "response = sts_client.assume_role(RoleArn=roleARN, \n",
-    "                                    RoleSessionName='CrossAccountSession')\n",
-    "\n",
-    "client = boto3.client(\n",
-    "    \"sagemaker-runtime\",\n",
-    "    region_name=\"us-west-2\",    \n",
-    "    aws_access_key_id=response['Credentials']['AccessKeyId'],\n",
-    "    aws_secret_access_key=response['Credentials']['SecretAccessKey'],\n",
-    "    aws_session_token = response['Credentials']['SessionToken']\n",
-    ")\n",
-    "\n",
-    "class ContentHandler(LLMContentHandler):\n",
-    "    content_type = \"application/json\"\n",
-    "    accepts = \"application/json\"\n",
-    "\n",
-    "    def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:\n",
-    "        input_str = json.dumps({prompt: prompt, **model_kwargs})\n",
-    "        return input_str.encode(\"utf-8\")\n",
-    "\n",
-    "    def transform_output(self, output: bytes) -> str:\n",
-    "        response_json = json.loads(output.read().decode(\"utf-8\"))\n",
-    "        return response_json[0][\"generated_text\"]\n",
-    "\n",
-    "\n",
-    "content_handler = ContentHandler()\n",
-    "\n",
-    "chain = load_qa_chain(\n",
-    "    llm=SagemakerEndpoint(\n",
-    "        endpoint_name=\"endpoint-name\",\n",
-    "        client=client,\n",
-    "        model_kwargs={\"temperature\": 1e-10},\n",
-    "        content_handler=content_handler,\n",
-    "    ),\n",
-    "    prompt=PROMPT,\n",
-    ")\n",
-    "\n",
-    "chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import Dict\n",
-    "\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.llms import SagemakerEndpoint\n",
+    "from langchain.prompts import PromptTemplate\nfrom langchain.llms import SagemakerEndpoint\n",
     "from langchain.llms.sagemaker_endpoint import LLMContentHandler\n",
     "from langchain.chains.question_answering import load_qa_chain\n",
     "import json\n",

docs/docs/integrations/llms/together.ipynb

@@ -1,79 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "2970dd75-8ebf-4b51-8282-9b454b8f356d",
-   "metadata": {},
-   "source": [
-    "# Together AI\n",
-    "\n",
-    "> The Together API makes it easy to fine-tune or run leading open-source models with a couple lines of code. We have integrated the world’s leading open-source models, including Llama-2, RedPajama, Falcon, Alpaca, Stable Diffusion XL, and more. Read more: https://together.ai\n",
-    "\n",
-    "To use, you'll need an API key which you can find here:\n",
-    "https://api.together.xyz/settings/api-keys. This can be passed in as init param\n",
-    "``together_api_key`` or set as environment variable ``TOGETHER_API_KEY``.\n",
-    "\n",
-    "Together API reference: https://docs.together.ai/reference/inference"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "e7b7170d-d7c5-4890-9714-a37238343805",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "A: A large language model is a neural network that is trained on a large amount of text data. It is able to generate text that is similar to the training data, and can be used for tasks such as language translation, question answering, and text summarization.\n",
-      "\n",
-      "A: A large language model is a neural network that is trained on a large amount of text data. It is able to generate text that is similar to the training data, and can be used for tasks such as language translation, question answering, and text summarization.\n",
-      "\n",
-      "A: A large language model is a neural network that is trained on\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.llms import Together\n",
-    "\n",
-    "llm = Together(\n",
-    "    model=\"togethercomputer/RedPajama-INCITE-7B-Base\",\n",
-    "    temperature=0.7,\n",
-    "    max_tokens=128,\n",
-    "    top_k=1,\n",
-    "    # together_api_key=\"...\"\n",
-    ")\n",
-    "\n",
-    "input_ = \"\"\"You are a teacher with a deep knowledge of machine learning and AI. \\\n",
-    "You provide succinct and accurate answers. Answer the following question: \n",
-    "\n",
-    "What is a large language model?\"\"\"\n",
-    "print(llm(input_))"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/integrations/llms/yandex.ipynb

@@ -1,119 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# YandexGPT\n",
-    "\n",
-    "This notebook goes over how to use Langchain with [YandexGPT](https://cloud.yandex.com/en/services/yandexgpt).\n",
-    "\n",
-    "To use, you should have the `yandexcloud` python package installed."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install yandexcloud"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "First, you should [create service account](https://cloud.yandex.com/en/docs/iam/operations/sa/create) with the `ai.languageModels.user` role.\n",
-    "\n",
-    "Next, you have two authentication options:\n",
-    "- [IAM token](https://cloud.yandex.com/en/docs/iam/operations/iam-token/create-for-sa).\n",
-    "    You can specify the token in a constructor parameter `iam_token` or in an environment variable `YC_IAM_TOKEN`.\n",
-    "- [API key](https://cloud.yandex.com/en/docs/iam/operations/api-key/create)\n",
-    "    You can specify the key in a constructor parameter `api_key` or in an environment variable `YC_API_KEY`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 246,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import YandexGPT\n",
-    "from langchain.prompts import PromptTemplate"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 247,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "template = \"What is the capital of {country}?\"\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"country\"])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 248,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = YandexGPT()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 249,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 250,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Moscow'"
-      ]
-     },
-     "execution_count": 250,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "country = \"Russia\"\n",
-    "\n",
-    "llm_chain.run(country)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.18"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/docs/integrations/memory/singlestoredb_chat_message_history.ipynb

@@ -1,65 +0,0 @@
-{
-    "cells": [
-     {
-      "attachments": {},
-      "cell_type": "markdown",
-      "id": "91c6a7ef",
-      "metadata": {},
-      "source": [
-       "# SingleStoreDB\n",
-       "\n",
-       "This notebook goes over how to use SingleStoreDB to store chat message history."
-      ]
-     },
-     {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "d15e3302",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-       "from langchain.memory import SingleStoreDBChatMessageHistory\n",
-       "\n",
-       "history = SingleStoreDBChatMessageHistory(\n",
-       "    session_id=\"foo\",\n",
-       "    host=\"root:pass@localhost:3306/db\"\n",
-       ")\n",
-       "\n",
-       "history.add_user_message(\"hi!\")\n",
-       "\n",
-       "history.add_ai_message(\"whats up?\")"
-      ]
-     },
-     {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "64fc465e",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-       "history.messages"
-      ]
-     }
-    ],
-    "metadata": {
-     "kernelspec": {
-      "display_name": "Python 3 (ipykernel)",
-      "language": "python",
-      "name": "python3"
-     },
-     "language_info": {
-      "codemirror_mode": {
-       "name": "ipython",
-       "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.11.2"
-     }
-    },
-    "nbformat": 4,
-    "nbformat_minor": 5
-}

docs/docs/integrations/platforms/microsoft.mdx

@@ -1,6 +1,6 @@
 # Microsoft
 
-All functionality related to `Microsoft Azure` and other `Microsoft` products.
+All functionality related to Microsoft Azure
 
 ## LLM
 ### Azure OpenAI
@@ -161,59 +161,3 @@ See a [usage example](/docs/integrations/retrievers/azure_cognitive_search).
 from langchain.retrievers import AzureCognitiveSearchRetriever
 ```
 
-## Utilities
-
-### Bing Search API
-
-See a [usage example](/docs/integrations/tools/bing_search).
-
-```python
-from langchain.utilities import BingSearchAPIWrapper
-```
-
-## Toolkits
-
-### Azure Cognitive Services
-
-We need to install several python packages.
-
-```bash
-pip install azure-ai-formrecognizer azure-cognitiveservices-speech azure-ai-vision
-```
-
-See a [usage example](/docs/integrations/toolkits/azure_cognitive_services).
-
-```python
-from langchain.agents.agent_toolkits import O365Toolkit
-```
-### Microsoft Office 365 email and calendar
-
-We need to install `O365` python package.
-
-```bash
-pip install O365
-```
-
-
-See a [usage example](/docs/integrations/toolkits/office365).
-
-```python
-from langchain.agents.agent_toolkits import O365Toolkit
-```
-
-### Microsoft Azure PowerBI
-
-We need to install `azure-identity` python package.
-
-```bash
-pip install azure-identity
-```
-
-See a [usage example](/docs/integrations/toolkits/powerbi).
-
-```python
-from langchain.agents.agent_toolkits import PowerBIToolkit
-from langchain.utilities.powerbi import PowerBIDataset
-```
-
-

docs/docs/integrations/providers/alibabacloud_opensearch.md

@@ -4,7 +4,7 @@
 
 OpenSearch helps you develop high quality, maintenance-free, and high performance intelligent search services to provide your users with high search efficiency and accuracy.
 
-OpenSearch provides the vector search feature. In specific scenarios,especially in question retrieval and image search scenarios, you can use the vector search feature together with the multimodal search feature to improve the accuracy of search results.
+ OpenSearch provides the vector search feature. In specific scenarios, especially test question search and image search scenarios, you can use the vector search feature together with the multimodal search feature to improve the accuracy of search results. This topic describes the syntax and usage notes of vector indexes.
 
 ## Purchase an instance and configure it
 
@@ -21,8 +21,6 @@ supported functions:
 - `similarity_search_by_vector`
 - `asimilarity_search_by_vector`
 - `similarity_search_with_relevance_scores`
-- `delete_doc_by_texts`
-
 
 For a more detailed walk through of the Alibaba Cloud OpenSearch wrapper, see [this notebook](../modules/indexes/vectorstores/examples/alibabacloud_opensearch.ipynb)
 

docs/docs/integrations/providers/clarifai.mdx

@@ -49,4 +49,4 @@ You can also add data directly from LangChain as well, and the auto-indexing wil
 from langchain.vectorstores import Clarifai
 clarifai_vector_db = Clarifai.from_texts(user_id=USER_ID, app_id=APP_ID, texts=texts, pat=CLARIFAI_PAT, number_of_docs=NUMBER_OF_DOCS, metadatas = metadatas)
 ```
-For more details, the docs on the Clarifai vector store provide a [detailed walkthrough](/docs/integrations/vectorstores/clarifai).
+For more details, the docs on the Clarifai vector store provide a [detailed walkthrough](/docs/integrations/vectorstores/clarifai.ipynb).

docs/docs/integrations/providers/deepinfra.mdx

@@ -10,27 +10,16 @@ It is broken into two parts: installation and setup, and then references to spec
 ## Available Models
 
 DeepInfra provides a range of Open Source LLMs ready for deployment.
-You can list supported models for
-[text-generation](https://deepinfra.com/models?type=text-generation) and
-[embeddings](https://deepinfra.com/models?type=embeddings).
+You can list supported models [here](https://deepinfra.com/models?type=text-generation).
 google/flan\* models can be viewed [here](https://deepinfra.com/models?type=text2text-generation).
 
-You can view a [list of request and response parameters](https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf/api).
+You can view a list of request and response parameters [here](https://deepinfra.com/databricks/dolly-v2-12b#API)
 
 ## Wrappers
 
 ### LLM
 
 There exists an DeepInfra LLM wrapper, which you can access with
-
 ```python
 from langchain.llms import DeepInfra
 ```
-
-### Embeddings
-
-There is also an DeepInfra Embeddings wrapper, you can access with
-
-```python
-from langchain.embeddings import DeepInfraEmbeddings
-```

docs/docs/integrations/providers/dingo.mdx

@@ -1,14 +1,14 @@
-# DingoDB
+# Dingo
 
-This page covers how to use the DingoDB ecosystem within LangChain.
-It is broken into two parts: installation and setup, and then references to specific DingoDB wrappers.
+This page covers how to use the Dingo ecosystem within LangChain.
+It is broken into two parts: installation and setup, and then references to specific Dingo wrappers.
 
 ## Installation and Setup
 - Install the Python SDK with `pip install dingodb`
 
 ## VectorStore
 
-There exists a wrapper around DingoDB indexes, allowing you to use it as a vectorstore,
+There exists a wrapper around Dingo indexes, allowing you to use it as a vectorstore,
 whether for semantic search or example selection.
 
 To import this vectorstore:
@@ -16,4 +16,4 @@ To import this vectorstore:
 from langchain.vectorstores import Dingo
 ```
 
-For a more detailed walkthrough of the DingoDB wrapper, see [this notebook](/docs/integrations/vectorstores/dingo.html)
+For a more detailed walkthrough of the Dingo wrapper, see [this notebook](/docs/integrations/vectorstores/dingo.html)

docs/docs/integrations/providers/huggingface.mdx

@@ -47,7 +47,7 @@ To use a the wrapper for a model hosted on Hugging Face Hub:
 ```python
 from langchain.embeddings import HuggingFaceHubEmbeddings
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/huggingfacehub)
+For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/huggingfacehub.html)
 
 ### Tokenizer
 
@@ -59,11 +59,11 @@ You can also use it to count tokens when splitting documents with
 from langchain.text_splitter import CharacterTextSplitter
 CharacterTextSplitter.from_huggingface_tokenizer(...)
 ```
-For a more detailed walkthrough of this, see [this notebook](/docs/modules/data_connection/document_transformers/text_splitters/huggingface_length_function)
+For a more detailed walkthrough of this, see [this notebook](/docs/modules/data_connection/document_transformers/text_splitters/huggingface_length_function.html)
 
 
 ### Datasets
 
 The Hugging Face Hub has lots of great [datasets](https://huggingface.co/datasets) that can be used to evaluate your LLM chains.
 
-For a detailed walkthrough of how to use them to do so, see [this notebook](/docs/integrations/document_loaders/hugging_face_dataset)
+For a detailed walkthrough of how to use them to do so, see [this notebook](/docs/use_cases/evaluation/huggingface_datasets.html)

docs/docs/integrations/providers/myscale.mdx

@@ -51,7 +51,6 @@ supported functions:
 - `similarity_search_by_vector`
 - `asimilarity_search_by_vector`
 - `similarity_search_with_relevance_scores`
-- `delete`
 
 ### VectorStore
 

docs/docs/integrations/providers/salute_devices.mdx

@@ -1,29 +0,0 @@
-# Salute Devices
-
-Salute Devices provides GigaChat LLM's models.
-
-For more info how to get access to GigaChat [follow here](https://developers.sber.ru/docs/ru/gigachat/api/integration).
-
-## Installation and Setup
-
-GigaChat package can be installed via pip from PyPI:
-
-```bash
-pip install gigachat
-```
-
-## LLMs
-
-See a [usage example](/docs/integrations/llms/gigachat).
-
-```python
-from langchain.llms import GigaChat
-```
-
-## Chat models
-
-See a [usage example](/docs/integrations/chat/gigachat).
-
-```python
-from langchain.chat_models import GigaChat
-```

docs/docs/integrations/providers/yandex.mdx

@@ -1,33 +0,0 @@
-# Yandex
-
-All functionality related to Yandex Cloud
-
->[Yandex Cloud](https://cloud.yandex.com/en/) is a public cloud platform. 
-
-## Installation and Setup
-
-Yandex Cloud SDK can be installed via pip from PyPI: 
-
-```bash
-pip install yandexcloud
-```
-
-## LLMs
-
-### YandexGPT
-
-See a [usage example](/docs/integrations/llms/yandex).
-
-```python
-from langchain.llms import YandexGPT
-```
-
-## Chat models
-
-### YandexGPT
-
-See a [usage example](/docs/integrations/chat/yandex).
-
-```python
-from langchain.chat_models import ChatYandexGPT
-```

docs/docs/integrations/retrievers/cohere-reranker.ipynb

@@ -330,7 +330,7 @@
     "from langchain.document_loaders import TextLoader\n",
     "from langchain.vectorstores import FAISS\n",
     "\n",
-    "documents = TextLoader(\"../../modules/state_of_the_union.txt\").load()\n",
+    "documents = TextLoader(\"../../../state_of_the_union.txt\").load()\n",
     "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)\n",
     "texts = text_splitter.split_documents(documents)\n",
     "retriever = FAISS.from_documents(texts, OpenAIEmbeddings()).as_retriever(\n",

docs/docs/integrations/retrievers/cohere.ipynb

@@ -1,223 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "bf733a38-db84-4363-89e2-de6735c37230",
-   "metadata": {},
-   "source": [
-    "# Cohere RAG retriever\n",
-    "\n",
-    "This notebook covers how to get started with Cohere RAG retriever. This allows you to leverage the ability to search documents over various connectors or by supplying your own."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatCohere\n",
-    "from langchain.retrievers import CohereRagRetriever\n",
-    "from langchain.schema.document import Document"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "70cf04e8-423a-4ff6-8b09-f11fb711c817",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "rag = CohereRagRetriever(llm=ChatCohere())"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "7f1e1bf3-542c-4fcb-8643-de6897fa6fcc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def _pretty_print(docs):\n",
-    "    for doc in docs:\n",
-    "        print(doc.metadata)\n",
-    "        print(\"\\n\\n\" + doc.page_content)\n",
-    "        print(\"\\n\\n\" + \"-\" * 30 + \"\\n\\n\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'id': 'web-search_4:0', 'snippet': 'AI startup Cohere, now valued at over $2.1B, raises $270M\\n\\nKyle Wiggers 4 months\\n\\nIn a sign that there’s plenty of cash to go around for generative AI startups, Cohere, which is developing an AI model ecosystem for the enterprise, today announced that it raised $270 million as part of its Series C round.\\n\\nReuters reported earlier in the year that Cohere was in talks to raise “hundreds of millions” of dollars at a valuation of upward of just over $6 billion. If there’s credence to that reporting, Cohere appears to have missed the valuation mark substantially; a source familiar with the matter tells TechCrunch that this tranche values the company at between $2.1 billion and $2.2 billion.', 'title': 'AI startup Cohere, now valued at over $2.1B, raises $270M | TechCrunch', 'url': 'https://techcrunch.com/2023/06/08/ai-startup-cohere-now-valued-at-over-2-1b-raises-270m/'}\n",
-      "\n",
-      "\n",
-      "AI startup Cohere, now valued at over $2.1B, raises $270M\n",
-      "\n",
-      "Kyle Wiggers 4 months\n",
-      "\n",
-      "In a sign that there’s plenty of cash to go around for generative AI startups, Cohere, which is developing an AI model ecosystem for the enterprise, today announced that it raised $270 million as part of its Series C round.\n",
-      "\n",
-      "Reuters reported earlier in the year that Cohere was in talks to raise “hundreds of millions” of dollars at a valuation of upward of just over $6 billion. If there’s credence to that reporting, Cohere appears to have missed the valuation mark substantially; a source familiar with the matter tells TechCrunch that this tranche values the company at between $2.1 billion and $2.2 billion.\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n",
-      "{'id': 'web-search_9:0', 'snippet': 'Cohere is a Canadian multinational technology company focused on artificial intelligence for the enterprise, specializing in large language models. Cohere was founded in 2019 by Aidan Gomez, Ivan Zhang, and Nick Frosst, and is headquartered in Toronto and San Francisco, with offices in Palo Alto and London.\\n\\nIn 2017, a team of researchers at Google Brain, which included Aidan Gomez, published a paper called \"Attention is All You Need,\" which introduced the transformer machine learning architecture, setting state-of-the-art performance on a variety of natural language processing tasks. In 2019, Gomez and Nick Frosst, another researcher at Google Brain, founded Cohere along with Ivan Zhang, with whom Gomez had done research at FOR.ai. All of the co-founders attended University of Toronto.', 'title': 'Cohere - Wikipedia', 'url': 'https://en.wikipedia.org/wiki/Cohere'}\n",
-      "\n",
-      "\n",
-      "Cohere is a Canadian multinational technology company focused on artificial intelligence for the enterprise, specializing in large language models. Cohere was founded in 2019 by Aidan Gomez, Ivan Zhang, and Nick Frosst, and is headquartered in Toronto and San Francisco, with offices in Palo Alto and London.\n",
-      "\n",
-      "In 2017, a team of researchers at Google Brain, which included Aidan Gomez, published a paper called \"Attention is All You Need,\" which introduced the transformer machine learning architecture, setting state-of-the-art performance on a variety of natural language processing tasks. In 2019, Gomez and Nick Frosst, another researcher at Google Brain, founded Cohere along with Ivan Zhang, with whom Gomez had done research at FOR.ai. All of the co-founders attended University of Toronto.\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n",
-      "{'id': 'web-search_8:2', 'snippet': ' Cofounded by Aidan Gomez, a Google Brain alum and coauthor of the seminal transformer research paper, Cohere describes itself as being “on a mission to transform enterprises and their products with AI to unlock a more intuitive way to generate, search, and summarize information than ever before.” One key element of Cohere’s approach is its focus on data protection, deploying its models inside enterprises’ secure data environment.\\n\\n“We are both independent and cloud-agnostic, meaning we are not beholden to any one tech company and empower enterprises to implement customized AI solutions on the cloud of their choosing, or even on-premises,” says Martin Kon, COO and president of Cohere.', 'title': 'McKinsey and Cohere collaborate to transform clients with enterprise generative AI', 'url': 'https://www.mckinsey.com/about-us/new-at-mckinsey-blog/mckinsey-and-cohere-collaborate-to-transform-clients-with-enterprise-generative-ai'}\n",
-      "\n",
-      "\n",
-      " Cofounded by Aidan Gomez, a Google Brain alum and coauthor of the seminal transformer research paper, Cohere describes itself as being “on a mission to transform enterprises and their products with AI to unlock a more intuitive way to generate, search, and summarize information than ever before.” One key element of Cohere’s approach is its focus on data protection, deploying its models inside enterprises’ secure data environment.\n",
-      "\n",
-      "“We are both independent and cloud-agnostic, meaning we are not beholden to any one tech company and empower enterprises to implement customized AI solutions on the cloud of their choosing, or even on-premises,” says Martin Kon, COO and president of Cohere.\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "_pretty_print(rag.get_relevant_documents(\"What is cohere ai?\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "4b888336",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'id': 'web-search_9:0', 'snippet': 'Cohere is a Canadian multinational technology company focused on artificial intelligence for the enterprise, specializing in large language models. Cohere was founded in 2019 by Aidan Gomez, Ivan Zhang, and Nick Frosst, and is headquartered in Toronto and San Francisco, with offices in Palo Alto and London.\\n\\nIn 2017, a team of researchers at Google Brain, which included Aidan Gomez, published a paper called \"Attention is All You Need,\" which introduced the transformer machine learning architecture, setting state-of-the-art performance on a variety of natural language processing tasks. In 2019, Gomez and Nick Frosst, another researcher at Google Brain, founded Cohere along with Ivan Zhang, with whom Gomez had done research at FOR.ai. All of the co-founders attended University of Toronto.', 'title': 'Cohere - Wikipedia', 'url': 'https://en.wikipedia.org/wiki/Cohere'}\n",
-      "\n",
-      "\n",
-      "Cohere is a Canadian multinational technology company focused on artificial intelligence for the enterprise, specializing in large language models. Cohere was founded in 2019 by Aidan Gomez, Ivan Zhang, and Nick Frosst, and is headquartered in Toronto and San Francisco, with offices in Palo Alto and London.\n",
-      "\n",
-      "In 2017, a team of researchers at Google Brain, which included Aidan Gomez, published a paper called \"Attention is All You Need,\" which introduced the transformer machine learning architecture, setting state-of-the-art performance on a variety of natural language processing tasks. In 2019, Gomez and Nick Frosst, another researcher at Google Brain, founded Cohere along with Ivan Zhang, with whom Gomez had done research at FOR.ai. All of the co-founders attended University of Toronto.\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n",
-      "{'id': 'web-search_8:2', 'snippet': ' Cofounded by Aidan Gomez, a Google Brain alum and coauthor of the seminal transformer research paper, Cohere describes itself as being “on a mission to transform enterprises and their products with AI to unlock a more intuitive way to generate, search, and summarize information than ever before.” One key element of Cohere’s approach is its focus on data protection, deploying its models inside enterprises’ secure data environment.\\n\\n“We are both independent and cloud-agnostic, meaning we are not beholden to any one tech company and empower enterprises to implement customized AI solutions on the cloud of their choosing, or even on-premises,” says Martin Kon, COO and president of Cohere.', 'title': 'McKinsey and Cohere collaborate to transform clients with enterprise generative AI', 'url': 'https://www.mckinsey.com/about-us/new-at-mckinsey-blog/mckinsey-and-cohere-collaborate-to-transform-clients-with-enterprise-generative-ai'}\n",
-      "\n",
-      "\n",
-      " Cofounded by Aidan Gomez, a Google Brain alum and coauthor of the seminal transformer research paper, Cohere describes itself as being “on a mission to transform enterprises and their products with AI to unlock a more intuitive way to generate, search, and summarize information than ever before.” One key element of Cohere’s approach is its focus on data protection, deploying its models inside enterprises’ secure data environment.\n",
-      "\n",
-      "“We are both independent and cloud-agnostic, meaning we are not beholden to any one tech company and empower enterprises to implement customized AI solutions on the cloud of their choosing, or even on-premises,” says Martin Kon, COO and president of Cohere.\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n",
-      "{'id': 'web-search_4:0', 'snippet': 'AI startup Cohere, now valued at over $2.1B, raises $270M\\n\\nKyle Wiggers 4 months\\n\\nIn a sign that there’s plenty of cash to go around for generative AI startups, Cohere, which is developing an AI model ecosystem for the enterprise, today announced that it raised $270 million as part of its Series C round.\\n\\nReuters reported earlier in the year that Cohere was in talks to raise “hundreds of millions” of dollars at a valuation of upward of just over $6 billion. If there’s credence to that reporting, Cohere appears to have missed the valuation mark substantially; a source familiar with the matter tells TechCrunch that this tranche values the company at between $2.1 billion and $2.2 billion.', 'title': 'AI startup Cohere, now valued at over $2.1B, raises $270M | TechCrunch', 'url': 'https://techcrunch.com/2023/06/08/ai-startup-cohere-now-valued-at-over-2-1b-raises-270m/'}\n",
-      "\n",
-      "\n",
-      "AI startup Cohere, now valued at over $2.1B, raises $270M\n",
-      "\n",
-      "Kyle Wiggers 4 months\n",
-      "\n",
-      "In a sign that there’s plenty of cash to go around for generative AI startups, Cohere, which is developing an AI model ecosystem for the enterprise, today announced that it raised $270 million as part of its Series C round.\n",
-      "\n",
-      "Reuters reported earlier in the year that Cohere was in talks to raise “hundreds of millions” of dollars at a valuation of upward of just over $6 billion. If there’s credence to that reporting, Cohere appears to have missed the valuation mark substantially; a source familiar with the matter tells TechCrunch that this tranche values the company at between $2.1 billion and $2.2 billion.\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "_pretty_print(await rag.aget_relevant_documents(\"What is cohere ai?\")) # async version"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "3742ba0f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{'id': 'doc-0', 'snippet': 'Langchain supports cohere RAG!'}\n",
-      "\n",
-      "\n",
-      "Langchain supports cohere RAG!\n",
-      "\n",
-      "\n",
-      "------------------------------\n",
-      "\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "docs = rag.get_relevant_documents(\n",
-    "        \"Does langchain support cohere RAG?\",\n",
-    "        source_documents=[Document(page_content=\"Langchain supports cohere RAG!\"), Document(page_content=\"The sky is blue!\")]\n",
-    "    )\n",
-    "_pretty_print(docs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "f96b4b09-7e0b-412f-bd7b-2b2d19d8ac6a",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/integrations/retrievers/google_cloud_enterprise_search.ipynb

@@ -0,0 +1,272 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Google Cloud Enterprise Search\n",
+    "\n",
+    "\n",
+    "[Enterprise Search](https://cloud.google.com/enterprise-search) is a part of the Generative AI App Builder suite of tools offered by Google Cloud.\n",
+    "\n",
+    "Gen AI App Builder lets developers, even those with limited machine learning skills, quickly and easily tap into the power of Google’s foundation models, search expertise, and conversational AI technologies to create enterprise-grade generative AI applications. \n",
+    "\n",
+    "Enterprise Search lets organizations quickly build generative AI powered search engines for customers and employees.Enterprise Search is underpinned by a variety of Google Search technologies, including semantic search, which helps deliver more relevant results than traditional keyword-based search techniques by using natural language processing and machine learning techniques to infer relationships within the content and intent from the user’s query input. Enterprise Search also benefits from Google’s expertise in understanding how users search and factors in content relevance to order displayed results. \n",
+    "\n",
+    "Google Cloud offers Enterprise Search via Gen App Builder in Google Cloud Console and via an API for enterprise workflow integration. \n",
+    "\n",
+    "This notebook demonstrates how to configure Enterprise Search and use the Enterprise Search retriever. The Enterprise Search retriever encapsulates the [Generative AI App Builder Python client library](https://cloud.google.com/generative-ai-app-builder/docs/libraries#client-libraries-install-python) and uses it to access the Enterprise Search [Search Service API](https://cloud.google.com/python/docs/reference/discoveryengine/latest/google.cloud.discoveryengine_v1beta.services.search_service)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Install pre-requisites\n",
+    "\n",
+    "You need to install the `google-cloud-discoverengine` package to use the Enterprise Search retriever."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! pip install google-cloud-discoveryengine"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configure access to Google Cloud and Google Cloud Enterprise Search\n",
+    "\n",
+    "Enterprise Search is generally available for the allowlist (which means customers need to be approved for access) as of June 6, 2023. Contact your Google Cloud sales team for access and pricing details. We are previewing additional features that are coming soon to the generally available offering as part of our [Trusted Tester](https://cloud.google.com/ai/earlyaccess/join?hl=en) program. Sign up for [Trusted Tester](https://cloud.google.com/ai/earlyaccess/join?hl=en) and contact your Google Cloud sales team for an expedited trial.\n",
+    "\n",
+    "Before you can run this notebook you need to:\n",
+    "- Set or create a Google Cloud project and turn on Gen App Builder\n",
+    "- Create and populate an unstructured data store\n",
+    "- Set credentials to access `Enterprise Search API`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set or create a Google Cloud poject and turn on Gen App Builder\n",
+    "\n",
+    "Follow the instructions in the [Enterprise Search Getting Started guide](https://cloud.google.com/generative-ai-app-builder/docs/before-you-begin) to set/create a GCP project and enable Gen App Builder.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create and populate an unstructured data store\n",
+    "\n",
+    "[Use Google Cloud Console to create an unstructured data store](https://cloud.google.com/generative-ai-app-builder/docs/create-engine-es#unstructured-data) and populate it with the example PDF documents from the  `gs://cloud-samples-data/gen-app-builder/search/alphabet-investor-pdfs` Cloud Storage folder. Make sure to use the `Cloud Storage (without metadata)` option."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set credentials to access Enterprise Search API\n",
+    "\n",
+    "The [Gen App Builder client libraries](https://cloud.google.com/generative-ai-app-builder/docs/libraries) used by the Enterprise Search retriever provide high-level language support for authenticating to Gen App Builder programmatically. Client libraries support [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials); the libraries look for credentials in a set of defined locations and use those credentials to authenticate requests to the API. With ADC, you can make credentials available to your application in a variety of environments, such as local development or production, without needing to modify your application code.\n",
+    "\n",
+    "If running in [Google Colab](https://colab.google) authenticate with `google.colab.google.auth` otherwise follow one of the [supported methods](https://cloud.google.com/docs/authentication/application-default-credentials) to make sure that you Application Default Credentials are properly set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "\n",
+    "if \"google.colab\" in sys.modules:\n",
+    "    from google.colab import auth as google_auth\n",
+    "\n",
+    "    google_auth.authenticate_user()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configure and use the Enterprise Search retriever\n",
+    "\n",
+    "The Enterprise Search retriever is implemented in the `langchain.retriever.GoogleCloudEntepriseSearchRetriever` class. The `get_relevant_documents` method returns a list of `langchain.schema.Document` documents where the `page_content` field of each document is populated the document content.\n",
+    "Depending on the data type used in Enterprise search (structured or unstructured) the `page_content` field is populated as follows:\n",
+    "- Structured data source: either an `extractive segment` or an `extractive answer` that matches a query. The `metadata` field is populated with metadata (if any) of the document from which the segments or answers were extracted.\n",
+    "- Unstructured data source: a string json containing all the fields returned from the structured data source. The `metadata` field is populated with metadata (if any) of the document  \n",
+    "\n",
+    "### Only for Unstructured data sources:\n",
+    "An extractive answer is verbatim text that is returned with each search result. It is extracted directly from the original document. Extractive answers are typically displayed near the top of web pages to provide an end user with a brief answer that is contextually relevant to their query. Extractive answers are available for website and unstructured search.\n",
+    "\n",
+    "An extractive segment is verbatim text that is returned with each search result. An extractive segment is usually more verbose than an extractive answer. Extractive segments can be displayed as an answer to a query, and can be used to perform post-processing tasks and as input for large language models to generate answers or new text. Extractive segments are available for unstructured search.\n",
+    "\n",
+    "For more information about extractive segments and extractive answers refer to [product documentation](https://cloud.google.com/generative-ai-app-builder/docs/snippets).\n",
+    "\n",
+    "When creating an instance of the retriever you can specify a number of parameters that control which Enterprise data store to access and how a natural language query is processed, including configurations for extractive answers and segments.\n",
+    "\n",
+    "\n",
+    "### The mandatory parameters are:\n",
+    "\n",
+    "- `project_id` - Your Google Cloud PROJECT_ID\n",
+    "- `search_engine_id` - The ID of the data store you want to use. \n",
+    "\n",
+    "The `project_id` and `search_engine_id` parameters can be provided explicitly in the retriever's constructor or through the environment variables - `PROJECT_ID` and `SEARCH_ENGINE_ID`.\n",
+    "\n",
+    "You can also configure a number of optional parameters, including:\n",
+    "\n",
+    "- `max_documents` - The maximum number of documents used to provide extractive segments or extractive answers\n",
+    "- `get_extractive_answers` - By default, the retriever is configured to return extractive segments. Set this field to `True` to return extractive answers. This is used only when `engine_data_type` set to 0 (unstructured) \n",
+    "- `max_extractive_answer_count` - The maximum number of extractive answers returned in each search result.\n",
+    "    At most 5 answers will be returned. This is used only when `engine_data_type` set to 0 (unstructured) \n",
+    "- `max_extractive_segment_count` - The maximum number of extractive segments returned in each search result.\n",
+    "    Currently one segment will be returned. This is used only when `engine_data_type` set to 0 (unstructured) \n",
+    "- `filter` - The filter expression that allows you filter the search results based on the metadata associated with the documents in the searched data store. \n",
+    "- `query_expansion_condition` - Specification to determine under which conditions query expansion should occur.\n",
+    "    0 - Unspecified query expansion condition. In this case, server behavior defaults to disabled.\n",
+    "    1 - Disabled query expansion. Only the exact search query is used, even if SearchResponse.total_size is zero.\n",
+    "    2 - Automatic query expansion built by the Search API.\n",
+    "- `engine_data_type` -  Defines the enterprise search data type\n",
+    "    0 - Unstructured data \n",
+    "    1 - Structured data\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Configure and use the retriever for **unstructured** data with extractve segments "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.retrievers import GoogleCloudEnterpriseSearchRetriever\n",
+    "\n",
+    "PROJECT_ID = \"<YOUR PROJECT ID>\"  # Set to your Project ID\n",
+    "SEARCH_ENGINE_ID = \"<YOUR SEARCH ENGINE ID>\"  # Set to your data store ID"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "retriever = GoogleCloudEnterpriseSearchRetriever(\n",
+    "    project_id=PROJECT_ID,\n",
+    "    search_engine_id=SEARCH_ENGINE_ID,\n",
+    "    max_documents=3,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"What are Alphabet's Other Bets?\"\n",
+    "\n",
+    "result = retriever.get_relevant_documents(query)\n",
+    "for doc in result:\n",
+    "    print(doc)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Configure and use the retriever for **unstructured** data with extractve answers "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "retriever = GoogleCloudEnterpriseSearchRetriever(\n",
+    "    project_id=PROJECT_ID,\n",
+    "    search_engine_id=SEARCH_ENGINE_ID,\n",
+    "    max_documents=3,\n",
+    "    max_extractive_answer_count=3,\n",
+    "    get_extractive_answers=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"What are Alphabet's Other Bets?\"\n",
+    "\n",
+    "result = retriever.get_relevant_documents(query)\n",
+    "for doc in result:\n",
+    "    print(doc)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Configure and use the retriever for **structured** data with extractve answers "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "retriever = GoogleCloudEnterpriseSearchRetriever(\n",
+    "    project_id=PROJECT_ID,\n",
+    "    search_engine_id=SEARCH_ENGINE_ID,\n",
+    "    max_documents=3,\n",
+    "    engine_data_type=1\n",
+    ")\n",
+    "\n",
+    "result = retriever.get_relevant_documents(query)\n",
+    "for doc in result:\n",
+    "    print(doc)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "base",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.10"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/docs/integrations/retrievers/google_vertex_ai_search.ipynb

@@ -30,7 +30,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "! pip install google-cloud-discoveryengine\n"
+    "! pip install google-cloud-discoveryengine"
    ]
   },
   {
@@ -80,7 +80,7 @@
     "if \"google.colab\" in sys.modules:\n",
     "    from google.colab import auth as google_auth\n",
     "\n",
-    "    google_auth.authenticate_user()\n"
+    "    google_auth.authenticate_user()"
    ]
   },
   {
@@ -90,13 +90,12 @@
     "## Configure and use the Vertex AI Search retriever\n",
     "\n",
     "The Vertex AI Search retriever is implemented in the `langchain.retriever.GoogleVertexAISearchRetriever` class. The `get_relevant_documents` method returns a list of `langchain.schema.Document` documents where the `page_content` field of each document is populated the document content.\n",
-    "Depending on the data type used in Vertex AI Search (website, structured or unstructured) the `page_content` field is populated as follows:\n",
+    "Depending on the data type used in Vertex AI Search (structured or unstructured) the `page_content` field is populated as follows:\n",
     "\n",
-    "- Website with advanced indexing: an `extractive answer` that matches a query. The `metadata` field is populated with metadata (if any) of the document from which the segments or answers were extracted.\n",
-    "- Unstructured data source: either an `extractive segment` or an `extractive answer` that matches a query. The `metadata` field is populated with metadata (if any) of the document from which the segments or answers were extracted.\n",
-    "- Structured data source: a string json containing all the fields returned from the structured data source. The `metadata` field is populated with metadata (if any) of the document\n",
+    "- Structured data source: either an `extractive segment` or an `extractive answer` that matches a query. The `metadata` field is populated with metadata (if any) of the document from which the segments or answers were extracted.\n",
+    "- Unstructured data source: a string json containing all the fields returned from the structured data source. The `metadata` field is populated with metadata (if any) of the document\n",
     "\n",
-    "### Extractive answers & extractive segments\n",
+    "### Only for Unstructured data sources:\n",
     "\n",
     "An extractive answer is verbatim text that is returned with each search result. It is extracted directly from the original document. Extractive answers are typically displayed near the top of web pages to provide an end user with a brief answer that is contextually relevant to their query. Extractive answers are available for website and unstructured search.\n",
     "\n",
@@ -137,7 +136,6 @@
     "- `engine_data_type` - Defines the Vertex AI Search data type\n",
     "  - `0` - Unstructured data\n",
     "  - `1` - Structured data\n",
-    "  - `2` - Website data with [Advanced Website Indexing](https://cloud.google.com/generative-ai-app-builder/docs/about-advanced-features#advanced-website-indexing)\n",
     "\n",
     "### Migration guide for `GoogleCloudEnterpriseSearchRetriever`\n",
     "\n",
@@ -163,11 +161,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.retrievers import GoogleVertexAISearchRetriever, GoogleVertexAIMultiTurnSearchRetriever\n",
+    "from langchain.retrievers import GoogleVertexAISearchRetriever\n",
     "\n",
     "PROJECT_ID = \"<YOUR PROJECT ID>\"  # Set to your Project ID\n",
     "LOCATION_ID = \"<YOUR LOCATION>\"  # Set to your data store location\n",
-    "DATA_STORE_ID = \"<YOUR DATA STORE ID>\"  # Set to your data store ID\n"
+    "DATA_STORE_ID = \"<YOUR DATA STORE ID>\"  # Set to your data store ID"
    ]
   },
   {
@@ -181,7 +179,7 @@
     "    location_id=LOCATION_ID,\n",
     "    data_store_id=DATA_STORE_ID,\n",
     "    max_documents=3,\n",
-    ")\n"
+    ")"
    ]
   },
   {
@@ -194,7 +192,7 @@
     "\n",
     "result = retriever.get_relevant_documents(query)\n",
     "for doc in result:\n",
-    "    print(doc)\n"
+    "    print(doc)"
    ]
   },
   {
@@ -221,7 +219,7 @@
     "\n",
     "result = retriever.get_relevant_documents(query)\n",
     "for doc in result:\n",
-    "    print(doc)\n"
+    "    print(doc)"
    ]
   },
   {
@@ -247,61 +245,7 @@
     "\n",
     "result = retriever.get_relevant_documents(query)\n",
     "for doc in result:\n",
-    "    print(doc)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Configure and use the retriever for **website** data with Advanced Website Indexing\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "retriever = GoogleVertexAISearchRetriever(\n",
-    "    project_id=PROJECT_ID,\n",
-    "    location_id=LOCATION_ID,\n",
-    "    data_store_id=DATA_STORE_ID,\n",
-    "    max_documents=3,\n",
-    "    max_extractive_answer_count=3,\n",
-    "    get_extractive_answers=True,\n",
-    "    engine_data_type=2,\n",
-    ")\n",
-    "\n",
-    "result = retriever.get_relevant_documents(query)\n",
-    "for doc in result:\n",
-    "    print(doc)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Configure and use the retriever for multi-turn search\n",
-    "\n",
-    "[Search with follow-ups](https://cloud.google.com/generative-ai-app-builder/docs/multi-turn-search) is based on generative AI models and it is different from the regular unstructured data search.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "retriever = GoogleVertexAIMultiTurnSearchRetriever(\n",
-    "    project_id=PROJECT_ID,\n",
-    "    location_id=LOCATION_ID,\n",
-    "    data_store_id=DATA_STORE_ID\n",
-    ")\n",
-    "\n",
-    "result = retriever.get_relevant_documents(query)\n",
-    "for doc in result:\n",
-    "    print(doc)\n"
+    "    print(doc)"
    ]
   }
  ],

docs/docs/integrations/retrievers/kay.ipynb

@@ -64,13 +64,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 20,
    "id": "b4d4d386-2a6b-4942-863e-9202f5a9f1d6",
    "metadata": {},
    "outputs": [],
    "source": [
-    "import os\n",
     "from langchain.retrievers import KayAiRetriever\n",
+    "import os\n",
     "from kay.rag.retrievers import KayRetriever\n",
     "os.environ[\"KAY_API_KEY\"] = KAY_API_KEY\n",
     "retriever = KayAiRetriever.create(dataset_id=\"company\", data_types=[\"10-K\", \"10-Q\", \"PressRelease\"], num_contexts=3)\n",
@@ -79,19 +79,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 21,
    "id": "04ee2d6b-c2ab-4e15-8a8b-afaf6ef8c0f6",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "[Document(page_content='Company Name: ROKU INC\\nCompany Industry: CABLE & OTHER PAY TELEVISION SERVICES\\nArticle Title: Roku Is One of Fast Company\\'s Most Innovative Companies for 2023\\nText: The company launched several new devices, including the Roku Voice Remote Pro; upgraded its most premium player, the Roku Ultra; and expanded its products with a new line of smart home devices such as video doorbells, lights, and plugs integrated into the Roku ecosystem. Recently, the company announced it will launch Roku-branded TVs this spring to offer more choice and innovation to both consumers and Roku TV partners. Throughout 2022, Roku also updated its operating system (OS), the only OS purpose-built for TV, with more personalization features and enhancements across search, audio, and content discovery, launching The Buzz, Sports, and What to Watch, which provides tailored movie and TV recommendations on the Home Screen Menu. The company also released a new feature for streamers, Photo Streams, that allows customers to display and share photo albums through Roku streaming devices. Additionally, Roku unveiled Shoppable Ads, a new ad innovation that makes shopping on TV streaming as easy as it is on social media. Viewers simply press \"OK\" with their Roku remote on a shoppable ad and proceed to check out with their shipping and payment details pre-populated from Roku Pay, its proprietary payments platform. Walmart was the exclusive retailer for the launch, a first-of-its-kind partnership.', metadata={'chunk_type': 'text', 'chunk_years_mentioned': [2022, 2023], 'company_name': 'ROKU INC', 'company_sic_code_description': 'CABLE & OTHER PAY TELEVISION SERVICES', 'data_source': 'PressRelease', 'data_source_link': 'https://newsroom.roku.com/press-releases', 'data_source_publish_date': '2023-03-02T09:30:00-04:00', 'data_source_uid': '963d4a81-f58e-3093-af68-987fb1758c15', 'title': \"ROKU INC |  Roku Is One of Fast Company's Most Innovative Companies for 2023\"}),\n",
-       " Document(page_content='Company Name: ROKU INC\\nCompany Industry: CABLE & OTHER PAY TELEVISION SERVICES\\nArticle Title: Roku Is One of Fast Company\\'s Most Innovative Companies for 2023\\nText: Finally, Roku grew its content offering with thousands of apps and watching options for users, including content on The Roku Channel, a top five app by reach and engagement on the Roku platform in the U.S. in 2022. In November, Roku released its first feature film, \"WEIRD: The Weird Al\\' Yankovic Story,\" a biopic starring Daniel Radcliffe. Throughout the year, The Roku Channel added FAST channels from NBCUniversal and the National Hockey League, as well as an exclusive AMC channel featuring its signature drama \"Mad Men.\" This year, the company announced a deal with Warner Bros. Discovery, launching new channels that will include \"Westworld\" and \"The Bachelor,\" in addition to 2,000 hours of on-demand content. Read more about Roku\\'s journey here . Fast Company\\'s Most Innovative Companies issue (March/April 2023) is available online here , as well as in-app via iTunes and on newsstands beginning March 14. About Roku, Inc.\\nRoku pioneered streaming to the TV. We connect users to the streaming content they love, enable content publishers to build and monetize large audiences, and provide advertisers with unique capabilities to engage consumers. Roku streaming players and TV-related audio devices are available in the U.S. and in select countries through direct retail sales and licensing arrangements with service operators. Roku TV models are available in the U.S. and select countries through licensing arrangements with TV OEM brands.', metadata={'chunk_type': 'text', 'chunk_years_mentioned': [2022, 2023], 'company_name': 'ROKU INC', 'company_sic_code_description': 'CABLE & OTHER PAY TELEVISION SERVICES', 'data_source': 'PressRelease', 'data_source_link': 'https://newsroom.roku.com/press-releases', 'data_source_publish_date': '2023-03-02T09:30:00-04:00', 'data_source_uid': '963d4a81-f58e-3093-af68-987fb1758c15', 'title': \"ROKU INC |  Roku Is One of Fast Company's Most Innovative Companies for 2023\"}),\n",
-       " Document(page_content='Company Name: ROKU INC\\nCompany Industry: CABLE & OTHER PAY TELEVISION SERVICES\\nArticle Title: Roku\\'s New NFL Zone Gives Fans Easy Access to NFL Games Right On Time for 2023 Season\\nText: In partnership with the NFL, the new NFL Zone offers viewers an easy way to find where to watch NFL live games Today, Roku (NASDAQ: ROKU ) and the National Football League (NFL) announced the recently launched NFL Zone within the Roku Sports experience to kick off the 2023 NFL season. This strategic partnership between Roku and the NFL marks the first official league-branded zone within Roku\\'s Sports experience. Available now, the NFL Zone offers football fans a centralized location to find live and upcoming games, so they can spend less time figuring out where to watch the game and more time rooting for their favorite teams. Users can also tune in for weekly game previews, League highlights, and additional NFL content, all within the zone. This press release features multimedia. View the full release here: In partnership with the NFL, Roku\\'s new NFL Zone offers viewers an easy way to find where to watch NFL live games (Photo: Business Wire) \"Last year we introduced the Sports experience for our highly engaged sports audience, making it simpler for Roku users to watch sports programming,\" said Gidon Katz, President, Consumer Experience, at Roku. \"As we start the biggest sports season of the year, providing easy access to NFL games and content to our millions of users is a top priority for us. We look forward to fans immersing themselves within the NFL Zone and making it their destination to find NFL games.', metadata={'chunk_type': 'text', 'chunk_years_mentioned': [2023], 'company_name': 'ROKU INC', 'company_sic_code_description': 'CABLE & OTHER PAY TELEVISION SERVICES', 'data_source': 'PressRelease', 'data_source_link': 'https://newsroom.roku.com/press-releases', 'data_source_publish_date': '2023-09-12T09:00:00-04:00', 'data_source_uid': '963d4a81-f58e-3093-af68-987fb1758c15', 'title': \"ROKU INC |  Roku's New NFL Zone Gives Fans Easy Access to NFL Games Right On Time for 2023 Season\"})]"
+       "[Document(page_content='Company Name: ROKU INC\\nCompany Industry: CABLE & OTHER PAY TELEVISION SERVICES\\nArticle Title: Roku and FreeWheel Announce Strategic Partnership to Bring Roku’s Leading Ad Tech to FreeWheel Customers\\nText: Additionally, eMarketer Link: https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.insiderintelligence.com%2Finsights%2Favod-more-than-50-percent-of-us-digital-video-viewers%2F&esheet=53451144&newsitemid=20230712907788&lan=en-US&anchor=eMarketer&index=4&md5=b64dea72bcf6b6379474462602781d83 projects 57% of U.S. digital video users will stream an advertising-based video on demand (AVOD) service this year.\\nHaving solutions aimed at driving greater interoperability and automation will help accelerate this growth.\\nKey highlights of this collaboration include:\\nStreamlined Integration: Roku has now integrated its demand application programming interface (dAPI) with FreeWheel s TV platform. Roku s demand API gives publishers direct, automatic and real-time access to more advertiser demand. This enhanced integration allows for streamlined ad operation workflows and better inventory quality control, both of which will improve publisher yield and revenue.\\nSeamless Data Targeting: Publishers can now use Roku platform signals to enable advertisers to target audiences and measure campaign performance without relying on cookies. Additionally, FreeWheel and Roku will rely on data clean room technology to enable the activation of additional data sets providing better measurement and monetization to publishers and agencies.', metadata={'_additional': {'id': '962b79e0-f9d1-43ae-9f7a-8a9b42bc7a9a'}, 'chunk_type': 'text', 'chunk_years_mentioned': [], 'company_name': 'ROKU INC', 'company_sic_code_description': 'CABLE & OTHER PAY TELEVISION SERVICES', 'data_source': 'PressRelease', 'data_source_link': 'https://www.nasdaq.com/press-release/roku-and-freewheel-announce-strategic-partnership-to-bring-rokus-leading-ad-tech-to', 'data_source_publish_date': '2023-07-12T00:00:00Z', 'data_source_uid': 'a46f309c-705d-3946-96db-87aa4e73261f', 'title': 'ROKU INC |  Roku and FreeWheel Announce Strategic Partnership to Bring Roku’s Leading Ad Tech to FreeWheel Customers'}),\n",
+       " Document(page_content='Company Name: ROKU INC \\n Company Industry: CABLE & OTHER PAY TELEVISION SERVICES \\n Form Title: 10-K 2022-FY \\n Form Section: Risk Factors \\n Text: nd the Note Regarding Forward Looking Statements.This section of this Annual Report generally discusses fiscal years 2022 and 2021 and year to year comparisons between those years.Discussions of fiscal year 2020 and year to year comparisons between fiscal years 2021 and 2020 that are not included in this Annual Report can be found in Management\\'s Discussion and Analysis of Financial Condition and Results of Operations in Part II, Item 7 of our Annual Report for the fiscal year ended December 31, 2021 filed with the SEC on February 18, 2022.Overview Effective as of the fourth quarter of fiscal 2022, we reorganized our reportable segments to better align with management\\'s reporting of information reviewed by the Chief Operating Decision Maker (\"CODM\") for each segment.We renamed our \"player\" segment to \"devices\" which now includes our licensing arrangements with service operators and licensed Roku TV partners in addition to sales of our streaming players, audio products, smart home products and Roku branded TVs that will be designed, made, and sold by us in 2023.Our historical segment information is recast to conform to our new presentation in our financial statements and accompanying notes included in Item 8 of this Annual Report.Our two reportable segments are the platform segment and the devices segment.', metadata={'_additional': {'id': 'a76c5fed-5d63-45a7-b63a-2c30e05140fc'}, 'chunk_type': 'text', 'chunk_years_mentioned': [2020, 2021, 2022, 2023], 'company_name': 'ROKU INC', 'company_sic_code_description': 'CABLE & OTHER PAY TELEVISION SERVICES', 'data_source': '10-K', 'data_source_link': 'https://www.sec.gov/Archives/edgar/data/1428439/000142843923000007', 'data_source_publish_date': '2022-01-01T00:00:00Z', 'data_source_uid': '0001428439-23-000007', 'title': 'ROKU INC |  10-K 2022-FY '}),\n",
+       " Document(page_content='Company Name: ROKU INC \\n Company Industry: CABLE & OTHER PAY TELEVISION SERVICES \\n Form Title: 10-Q 2023-Q1 \\n Form Section: Risk Factors \\n Text: Our current and potential partners include TV brands, cable and satellite companies, and telecommunication providers.Under these license arrangements, we generally have limited or no control over the amount and timing of resources these entities dedicate to the relationship.In the past, our licensed Roku TV partners have failed to meet their forecasts and anticipated market launch dates for distributing Roku TV models, and they may fail to meet their forecasts or such launches in the future.If our licensed Roku TV partners or service operator partners fail to meet their forecasts or such launches for distributing licensed streaming devices or choose to deploy competing streaming solutions within their product lines, our business may be harmed.We depend on a small number of content publishers for a majority of our streaming hours, and if we fail to maintain these relationships, our business could be harmed.*Historically, a small number of content publishers have accounted for a significant portion of the hours streamed on our platform.In the three months ended March 31, 2023, the top three streaming services represented over 50% of all hours streamed in the period.If, for any reason, we cease distributing channels that have historically streamed a large percentage of the aggregate streaming hours on our platform, our streaming hours, our active accounts, or Roku streaming device sales may be adversely affected, and our business may be harmed.', metadata={'_additional': {'id': '2a92b2bb-02a0-4e15-8b64-d7e04078a205'}, 'chunk_type': 'text', 'chunk_years_mentioned': [2023], 'company_name': 'ROKU INC', 'company_sic_code_description': 'CABLE & OTHER PAY TELEVISION SERVICES', 'data_source': '10-Q', 'data_source_link': 'https://www.sec.gov/Archives/edgar/data/1428439/000142843923000017', 'data_source_publish_date': '2023-01-01T00:00:00Z', 'data_source_uid': '0001428439-23-000017', 'title': 'ROKU INC |  10-Q 2023-Q1 '})]"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 21,
      "metadata": {},
      "output_type": "execute_result"
     }

docs/docs/integrations/retrievers/self_query/elasticsearch_self_query.ipynb

@@ -28,29 +28,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 1,
    "id": "63a8af5b",
    "metadata": {
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 23.3 is available.\n",
-      "You should consider upgrading via the '/Users/joe/projects/elastic/langchain/libs/langchain/.venv/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
-      "\u001b[0m"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
-    "#!pip install -qU lark elasticsearch"
+    "#!pip install lark elasticsearch"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "cb4a5787",
    "metadata": {
     "tags": []
@@ -70,7 +60,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 9,
    "id": "bcbe04d9",
    "metadata": {
     "tags": []
@@ -125,7 +115,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 6,
    "id": "86e34dbf",
    "metadata": {
     "tags": []
@@ -174,10 +164,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 10,
    "id": "38a126e9",
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "query='dinosaur' filter=None limit=None\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
@@ -187,7 +184,7 @@
        " Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'director': 'Satoshi Kon', 'rating': 8.6})]"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -199,17 +196,24 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 11,
    "id": "b19d4da0",
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "query='women' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='director', value='Greta Gerwig') limit=None\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
        "[Document(page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'year': 2019, 'director': 'Greta Gerwig', 'rating': 8.3})]"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -233,7 +237,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 12,
    "id": "bff36b88-b506-4877-9c63-e5a1a8d78e64",
    "metadata": {
     "tags": []
@@ -252,12 +256,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 13,
    "id": "2758d229-4f97-499c-819f-888acaf8ee10",
    "metadata": {
     "tags": []
    },
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "query='dinosaur' filter=None limit=2\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
@@ -265,7 +276,7 @@
        " Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -286,17 +297,24 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 18,
    "id": "e460da93",
    "metadata": {},
    "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "query='animated toys' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Operation(operator=<Operator.OR: 'or'>, arguments=[Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='animated'), Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='comedy')]), Comparison(comparator=<Comparator.GTE: 'gte'>, attribute='year', value=1990)]) limit=None\n"
+     ]
+    },
     {
      "data": {
       "text/plain": [
        "[Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]"
       ]
      },
-     "execution_count": 8,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -307,10 +325,21 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "id": "0851fc42",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "ObjectApiResponse({'acknowledged': True})"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "vectorstore.client.indices.delete(index=\"elasticsearch-self-query-demo\")"
    ]
@@ -332,7 +361,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.3"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

docs/docs/integrations/retrievers/singlestoredb.ipynb

@@ -1,120 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "ab66dd43",
-   "metadata": {},
-   "source": [
-    "# SingleStoreDB\n",
-    "\n",
-    ">[SingleStoreDB](https://singlestore.com/) is a high-performance distributed SQL database that supports deployment both in the [cloud](https://www.singlestore.com/cloud/) and on-premises. It provides vector storage, and vector functions including [dot_product](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/dot_product.html) and [euclidean_distance](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/euclidean_distance.html), thereby supporting AI applications that require text similarity matching. \n",
-    "\n",
-    "\n",
-    "This notebook shows how to use a retriever that uses `SingleStoreDB`.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "51b49135-a61a-49e8-869d-7c1d76794cd7",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "# Establishing a connection to the database is facilitated through the singlestoredb Python connector.\n",
-    "# Please ensure that this connector is installed in your working environment.\n",
-    "!pip install singlestoredb"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "aaf80e7f",
-   "metadata": {},
-   "source": [
-    "## Create Retriever from vector store"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "bcb3c8c2",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "import getpass\n",
-    "\n",
-    "# We want to use OpenAIEmbeddings so we have to get the OpenAI API Key.\n",
-    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
-    "\n",
-    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain.vectorstores import SingleStoreDB\n",
-    "from langchain.document_loaders import TextLoader\n",
-    "\n",
-    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")\n",
-    "documents = loader.load()\n",
-    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
-    "docs = text_splitter.split_documents(documents)\n",
-    "\n",
-    "embeddings = OpenAIEmbeddings()\n",
-    "\n",
-    "# Setup connection url as environment variable\n",
-    "os.environ[\"SINGLESTOREDB_URL\"] = \"root:pass@localhost:3306/db\"\n",
-    "\n",
-    "# Load documents to the store\n",
-    "docsearch = SingleStoreDB.from_documents(\n",
-    "    docs,\n",
-    "    embeddings,\n",
-    "    table_name=\"notebook\",  # use table with a custom name\n",
-    ")\n",
-    "\n",
-    "# create retriever from the vector store\n",
-    "retriever = docsearch.as_retriever(search_kwargs={\"k\": 2})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "fc0915db",
-   "metadata": {},
-   "source": [
-    "## Search with retriever"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "b605284d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "result = retriever.get_relevant_documents(\"What did the president say about Ketanji Brown Jackson\")\n",
-    "print(docs[0].page_content)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.6"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/integrations/retrievers/zep_memorystore.ipynb

@@ -1,18 +1,15 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Zep\n",
-    "## Retriever Example for [Zep](https://docs.getzep.com/) - Fast, scalable building blocks for LLM Apps\n",
+    "## Retriever Example for [Zep](https://docs.getzep.com/) - A long-term memory store for LLM applications.\n",
     "\n",
     "### More on Zep:\n",
     "\n",
-    "Zep is an open source platform for productionizing LLM apps. Go from a prototype\n",
-    "built in LangChain or LlamaIndex, or a custom app, to production in minutes without\n",
-    "rewriting code.\n",
+    "Zep stores, summarizes, embeds, indexes, and enriches conversational AI chat histories, and exposes them via simple, low-latency APIs.\n",
     "\n",
     "Key Features:\n",
     "\n",
@@ -29,7 +26,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -48,11 +44,11 @@
    "cell_type": "code",
    "execution_count": 1,
    "metadata": {
+    "collapsed": false,
     "ExecuteTime": {
      "end_time": "2023-08-11T20:31:12.231459Z",
      "start_time": "2023-08-11T20:31:11.211176Z"
-    },
-    "collapsed": false
+    }
    },
    "outputs": [],
    "source": [
@@ -60,7 +56,7 @@
     "import time\n",
     "from uuid import uuid4\n",
     "\n",
-    "from langchain.memory import ZepMemory\n",
+    "from langchain.memory import ZepMemory, CombinedMemory, VectorStoreRetrieverMemory\n",
     "from langchain.schema import HumanMessage, AIMessage\n",
     "\n",
     "# Set this to your Zep server URL\n",
@@ -68,7 +64,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -100,38 +95,31 @@
    "cell_type": "code",
    "execution_count": 3,
    "metadata": {
+    "collapsed": false,
     "ExecuteTime": {
      "end_time": "2023-08-11T20:31:12.342790Z",
      "start_time": "2023-08-11T20:31:12.235291Z"
-    },
-    "collapsed": false
-   },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/danielchalef/dev/langchain/.venv/lib/python3.11/site-packages/zep_python/zep_client.py:86: Warning: You are using an incompatible Zep server version. Please upgrade to {MINIMUM_SERVER_VERSION} or later.\n",
-      "  self._healthcheck(base_url)\n"
-     ]
     }
-   ],
+   },
+   "outputs": [],
    "source": [
     "session_id = str(uuid4())  # This is a unique identifier for the user/session\n",
     "\n",
     "# Initialize the Zep Memory Class\n",
-    "zep_memory = ZepMemory(session_id=session_id, url=ZEP_API_URL, api_key=zep_api_key)"
+    "zep_memory = ZepMemory(\n",
+    "    session_id=session_id, url=ZEP_API_URL, api_key=zep_api_key\n",
+    ")"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 4,
    "metadata": {
+    "collapsed": false,
     "ExecuteTime": {
      "end_time": "2023-08-11T20:31:14.455269Z",
      "start_time": "2023-08-11T20:31:12.345635Z"
-    },
-    "collapsed": false
+    }
    },
    "outputs": [],
    "source": [
@@ -201,12 +189,11 @@
     "        if msg[\"role\"] == \"human\"\n",
     "        else AIMessage(content=msg[\"content\"])\n",
     "    )\n",
-    "\n",
+    "    \n",
     "time.sleep(2)  # Wait for the messages to be embedded"
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -221,22 +208,16 @@
    "cell_type": "code",
    "execution_count": 5,
    "metadata": {
+    "collapsed": false,
     "ExecuteTime": {
      "end_time": "2023-08-11T20:31:14.758738Z",
      "start_time": "2023-08-11T20:31:14.458850Z"
-    },
-    "collapsed": false
+    }
    },
    "outputs": [
     {
      "data": {
-      "text/plain": [
-       "[Document(page_content='Parable of the Sower is a science fiction novel by Octavia Butler, published in 1993. It follows the story of Lauren Olamina, a young woman living in a dystopian future where society has collapsed due to environmental disasters, poverty, and violence.', metadata={'score': 0.8897589445114136, 'uuid': 'f99ecec3-f778-4bfd-8bb7-c3c00ae919c0', 'created_at': '2023-10-17T22:53:08.664849Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'GPE', 'Matches': [{'End': 20, 'Start': 15, 'Text': 'Sower'}], 'Name': 'Sower'}, {'Label': 'PERSON', 'Matches': [{'End': 65, 'Start': 51, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}, {'Label': 'DATE', 'Matches': [{'End': 84, 'Start': 80, 'Text': '1993'}], 'Name': '1993'}, {'Label': 'PERSON', 'Matches': [{'End': 124, 'Start': 110, 'Text': 'Lauren Olamina'}], 'Name': 'Lauren Olamina'}]}}, 'token_count': 56}),\n",
-       " Document(page_content=\"Write a short synopsis of Butler's book, Parable of the Sower. What is it about?\", metadata={'score': 0.8856973648071289, 'uuid': 'f6aba470-f15f-4b22-84ef-1c0d315a31de', 'created_at': '2023-10-17T22:53:08.642659Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 32, 'Start': 26, 'Text': 'Butler'}], 'Name': 'Butler'}, {'Label': 'WORK_OF_ART', 'Matches': [{'End': 61, 'Start': 41, 'Text': 'Parable of the Sower'}], 'Name': 'Parable of the Sower'}]}}, 'token_count': 23}),\n",
-       " Document(page_content='Who was Octavia Butler?', metadata={'score': 0.7759557962417603, 'uuid': '26aab7b5-34b1-4aff-9be0-7834a7702be4', 'created_at': '2023-10-17T22:53:08.585297Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 22, 'Start': 8, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}], 'intent': 'The subject is asking for information about Octavia Butler, a specific person.'}}, 'token_count': 8}),\n",
-       " Document(page_content=\"Octavia Butler's contemporaries included Ursula K. Le Guin, Samuel R. Delany, and Joanna Russ.\", metadata={'score': 0.760245680809021, 'uuid': 'ee4aa8e9-9913-4e69-a2a5-77a85294d24e', 'created_at': '2023-10-17T22:53:08.611466Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 16, 'Start': 0, 'Text': \"Octavia Butler's\"}], 'Name': \"Octavia Butler's\"}, {'Label': 'ORG', 'Matches': [{'End': 58, 'Start': 41, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 76, 'Start': 60, 'Text': 'Samuel R. Delany'}], 'Name': 'Samuel R. Delany'}, {'Label': 'PERSON', 'Matches': [{'End': 93, 'Start': 82, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 27}),\n",
-       " Document(page_content='You might want to read Ursula K. Le Guin or Joanna Russ.', metadata={'score': 0.7596070170402527, 'uuid': '9fa630e6-0b17-4d77-80b0-ba99249850c0', 'created_at': '2023-10-17T22:53:08.630731Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 40, 'Start': 23, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 55, 'Start': 44, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 18})]"
-      ]
+      "text/plain": "[Document(page_content='Who was Octavia Butler?', metadata={'score': 0.7758688965570713, 'uuid': 'b3322d28-f589-48c7-9daf-5eb092d65976', 'created_at': '2023-08-11T20:31:12.3856Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 22, 'Start': 8, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}]}}, 'token_count': 8}),\n Document(page_content=\"Octavia Butler's contemporaries included Ursula K. Le Guin, Samuel R. Delany, and Joanna Russ.\", metadata={'score': 0.7602672137411663, 'uuid': '756b7136-0b4c-4664-ad33-c4431670356c', 'created_at': '2023-08-11T20:31:12.420717Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 16, 'Start': 0, 'Text': \"Octavia Butler's\"}], 'Name': \"Octavia Butler's\"}, {'Label': 'ORG', 'Matches': [{'End': 58, 'Start': 41, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 76, 'Start': 60, 'Text': 'Samuel R. Delany'}], 'Name': 'Samuel R. Delany'}, {'Label': 'PERSON', 'Matches': [{'End': 93, 'Start': 82, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 27}),\n Document(page_content='You might want to read Ursula K. Le Guin or Joanna Russ.', metadata={'score': 0.7596040989115522, 'uuid': '166d9556-2d48-4237-8a84-5d8a1024d5f4', 'created_at': '2023-08-11T20:31:12.434522Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 40, 'Start': 23, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 55, 'Start': 44, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 18}),\n Document(page_content='Who were her contemporaries?', metadata={'score': 0.7575531381951208, 'uuid': 'c6a16691-4012-439f-b223-84fd4e79c4cf', 'created_at': '2023-08-11T20:31:12.410336Z', 'role': 'human', 'token_count': 8}),\n Document(page_content='Octavia Estelle Butler (June 22, 1947 – February 24, 2006) was an American science fiction author.', metadata={'score': 0.7546476914454683, 'uuid': '7c093a2a-0099-415a-95c5-615a8026a894', 'created_at': '2023-08-11T20:31:12.399979Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 22, 'Start': 0, 'Text': 'Octavia Estelle Butler'}], 'Name': 'Octavia Estelle Butler'}, {'Label': 'DATE', 'Matches': [{'End': 37, 'Start': 24, 'Text': 'June 22, 1947'}], 'Name': 'June 22, 1947'}, {'Label': 'DATE', 'Matches': [{'End': 57, 'Start': 40, 'Text': 'February 24, 2006'}], 'Name': 'February 24, 2006'}, {'Label': 'NORP', 'Matches': [{'End': 74, 'Start': 66, 'Text': 'American'}], 'Name': 'American'}]}}, 'token_count': 31})]"
      },
      "execution_count": 5,
      "metadata": {},
@@ -245,7 +226,6 @@
    ],
    "source": [
     "from langchain.retrievers import ZepRetriever\n",
-    "from langchain.retrievers.zep import SearchType\n",
     "\n",
     "zep_retriever = ZepRetriever(\n",
     "    session_id=session_id,  # Ensure that you provide the session_id when instantiating the Retriever\n",
@@ -258,7 +238,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -269,22 +248,16 @@
    "cell_type": "code",
    "execution_count": 6,
    "metadata": {
+    "collapsed": false,
     "ExecuteTime": {
      "end_time": "2023-08-11T20:31:14.922838Z",
      "start_time": "2023-08-11T20:31:14.751737Z"
-    },
-    "collapsed": false
+    }
    },
    "outputs": [
     {
      "data": {
-      "text/plain": [
-       "[Document(page_content='Parable of the Sower is a science fiction novel by Octavia Butler, published in 1993. It follows the story of Lauren Olamina, a young woman living in a dystopian future where society has collapsed due to environmental disasters, poverty, and violence.', metadata={'score': 0.8897120952606201, 'uuid': 'f99ecec3-f778-4bfd-8bb7-c3c00ae919c0', 'created_at': '2023-10-17T22:53:08.664849Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'GPE', 'Matches': [{'End': 20, 'Start': 15, 'Text': 'Sower'}], 'Name': 'Sower'}, {'Label': 'PERSON', 'Matches': [{'End': 65, 'Start': 51, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}, {'Label': 'DATE', 'Matches': [{'End': 84, 'Start': 80, 'Text': '1993'}], 'Name': '1993'}, {'Label': 'PERSON', 'Matches': [{'End': 124, 'Start': 110, 'Text': 'Lauren Olamina'}], 'Name': 'Lauren Olamina'}]}}, 'token_count': 56}),\n",
-       " Document(page_content=\"Write a short synopsis of Butler's book, Parable of the Sower. What is it about?\", metadata={'score': 0.8857351541519165, 'uuid': 'f6aba470-f15f-4b22-84ef-1c0d315a31de', 'created_at': '2023-10-17T22:53:08.642659Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 32, 'Start': 26, 'Text': 'Butler'}], 'Name': 'Butler'}, {'Label': 'WORK_OF_ART', 'Matches': [{'End': 61, 'Start': 41, 'Text': 'Parable of the Sower'}], 'Name': 'Parable of the Sower'}]}}, 'token_count': 23}),\n",
-       " Document(page_content='Who was Octavia Butler?', metadata={'score': 0.7759560942649841, 'uuid': '26aab7b5-34b1-4aff-9be0-7834a7702be4', 'created_at': '2023-10-17T22:53:08.585297Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 22, 'Start': 8, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}], 'intent': 'The subject is asking for information about Octavia Butler, a specific person.'}}, 'token_count': 8}),\n",
-       " Document(page_content=\"Octavia Butler's contemporaries included Ursula K. Le Guin, Samuel R. Delany, and Joanna Russ.\", metadata={'score': 0.7602507472038269, 'uuid': 'ee4aa8e9-9913-4e69-a2a5-77a85294d24e', 'created_at': '2023-10-17T22:53:08.611466Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 16, 'Start': 0, 'Text': \"Octavia Butler's\"}], 'Name': \"Octavia Butler's\"}, {'Label': 'ORG', 'Matches': [{'End': 58, 'Start': 41, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 76, 'Start': 60, 'Text': 'Samuel R. Delany'}], 'Name': 'Samuel R. Delany'}, {'Label': 'PERSON', 'Matches': [{'End': 93, 'Start': 82, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}], 'intent': \"The subject is stating a fact about Octavia Butler's contemporaries, including Ursula K. Le Guin, Samuel R. Delany, and Joanna Russ.\"}}, 'token_count': 27}),\n",
-       " Document(page_content='You might want to read Ursula K. Le Guin or Joanna Russ.', metadata={'score': 0.7595934867858887, 'uuid': '9fa630e6-0b17-4d77-80b0-ba99249850c0', 'created_at': '2023-10-17T22:53:08.630731Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 40, 'Start': 23, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 55, 'Start': 44, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 18})]"
-      ]
+      "text/plain": "[Document(page_content=\"Write a short synopsis of Butler's book, Parable of the Sower. What is it about?\", metadata={'score': 0.8857504413268114, 'uuid': '82f07ab5-9d4b-4db6-aaae-6028e6fd836b', 'created_at': '2023-08-11T20:31:12.437365Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 32, 'Start': 26, 'Text': 'Butler'}], 'Name': 'Butler'}, {'Label': 'WORK_OF_ART', 'Matches': [{'End': 61, 'Start': 41, 'Text': 'Parable of the Sower'}], 'Name': 'Parable of the Sower'}]}}, 'token_count': 23}),\n Document(page_content='Who was Octavia Butler?', metadata={'score': 0.7758688965570713, 'uuid': 'b3322d28-f589-48c7-9daf-5eb092d65976', 'created_at': '2023-08-11T20:31:12.3856Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 22, 'Start': 8, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}]}}, 'token_count': 8}),\n Document(page_content=\"Octavia Butler's contemporaries included Ursula K. Le Guin, Samuel R. Delany, and Joanna Russ.\", metadata={'score': 0.7602672137411663, 'uuid': '756b7136-0b4c-4664-ad33-c4431670356c', 'created_at': '2023-08-11T20:31:12.420717Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'PERSON', 'Matches': [{'End': 16, 'Start': 0, 'Text': \"Octavia Butler's\"}], 'Name': \"Octavia Butler's\"}, {'Label': 'ORG', 'Matches': [{'End': 58, 'Start': 41, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 76, 'Start': 60, 'Text': 'Samuel R. Delany'}], 'Name': 'Samuel R. Delany'}, {'Label': 'PERSON', 'Matches': [{'End': 93, 'Start': 82, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 27}),\n Document(page_content='You might want to read Ursula K. Le Guin or Joanna Russ.', metadata={'score': 0.7596040989115522, 'uuid': '166d9556-2d48-4237-8a84-5d8a1024d5f4', 'created_at': '2023-08-11T20:31:12.434522Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 40, 'Start': 23, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 55, 'Start': 44, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 18}),\n Document(page_content='Who were her contemporaries?', metadata={'score': 0.7575531381951208, 'uuid': 'c6a16691-4012-439f-b223-84fd4e79c4cf', 'created_at': '2023-08-11T20:31:12.410336Z', 'role': 'human', 'token_count': 8})]"
      },
      "execution_count": 6,
      "metadata": {},
@@ -295,109 +268,18 @@
     "zep_retriever.get_relevant_documents(\"Who wrote Parable of the Sower?\")"
    ]
   },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Reranking using MMR (Maximal Marginal Relevance)\n",
-    "\n",
-    "Zep has native, SIMD-accelerated support for reranking results using MMR. This is useful for removing redundancy in results."
-   ]
-  },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 6,
+   "outputs": [],
+   "source": [],
    "metadata": {
+    "collapsed": false,
     "ExecuteTime": {
      "end_time": "2023-08-11T20:31:14.923032Z",
      "start_time": "2023-08-11T20:31:14.918181Z"
-    },
-    "collapsed": false
-   },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/danielchalef/dev/langchain/.venv/lib/python3.11/site-packages/zep_python/zep_client.py:86: Warning: You are using an incompatible Zep server version. Please upgrade to {MINIMUM_SERVER_VERSION} or later.\n",
-      "  self._healthcheck(base_url)\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='Parable of the Sower is a science fiction novel by Octavia Butler, published in 1993. It follows the story of Lauren Olamina, a young woman living in a dystopian future where society has collapsed due to environmental disasters, poverty, and violence.', metadata={'score': 0.8897120952606201, 'uuid': 'f99ecec3-f778-4bfd-8bb7-c3c00ae919c0', 'created_at': '2023-10-17T22:53:08.664849Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'GPE', 'Matches': [{'End': 20, 'Start': 15, 'Text': 'Sower'}], 'Name': 'Sower'}, {'Label': 'PERSON', 'Matches': [{'End': 65, 'Start': 51, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}, {'Label': 'DATE', 'Matches': [{'End': 84, 'Start': 80, 'Text': '1993'}], 'Name': '1993'}, {'Label': 'PERSON', 'Matches': [{'End': 124, 'Start': 110, 'Text': 'Lauren Olamina'}], 'Name': 'Lauren Olamina'}]}}, 'token_count': 56}),\n",
-       " Document(page_content='Which books of hers were made into movies?', metadata={'score': 0.7496200799942017, 'uuid': '1047ff15-96f1-4101-bb0f-9ed073b8081d', 'created_at': '2023-10-17T22:53:08.596614Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'intent': 'The subject is inquiring about the books of the person referred to as \"hers\" that have been made into movies.'}}, 'token_count': 11}),\n",
-       " Document(page_content=\"Write a short synopsis of Butler's book, Parable of the Sower. What is it about?\", metadata={'score': 0.8857351541519165, 'uuid': 'f6aba470-f15f-4b22-84ef-1c0d315a31de', 'created_at': '2023-10-17T22:53:08.642659Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 32, 'Start': 26, 'Text': 'Butler'}], 'Name': 'Butler'}, {'Label': 'WORK_OF_ART', 'Matches': [{'End': 61, 'Start': 41, 'Text': 'Parable of the Sower'}], 'Name': 'Parable of the Sower'}]}}, 'token_count': 23}),\n",
-       " Document(page_content='You might want to read Ursula K. Le Guin or Joanna Russ.', metadata={'score': 0.7595934867858887, 'uuid': '9fa630e6-0b17-4d77-80b0-ba99249850c0', 'created_at': '2023-10-17T22:53:08.630731Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 40, 'Start': 23, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 55, 'Start': 44, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}]}}, 'token_count': 18}),\n",
-       " Document(page_content='Who were her contemporaries?', metadata={'score': 0.7575579881668091, 'uuid': 'b2dfd1f7-cac6-4e37-94ea-7c15b0a5af2c', 'created_at': '2023-10-17T22:53:08.606283Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'intent': 'The subject is asking about the people who were contemporaries of someone else.'}}, 'token_count': 8})]"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
     }
-   ],
-   "source": [
-    "zep_retriever = ZepRetriever(\n",
-    "    session_id=session_id,  # Ensure that you provide the session_id when instantiating the Retriever\n",
-    "    url=ZEP_API_URL,\n",
-    "    top_k=5,\n",
-    "    api_key=zep_api_key,\n",
-    "    search_type=SearchType.mmr,\n",
-    "    mmr_lambda=0.5,\n",
-    ")\n",
-    "\n",
-    "await zep_retriever.aget_relevant_documents(\"Who wrote Parable of the Sower?\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Using metadata filters to refine search results\n",
-    "\n",
-    "Zep supports filtering results by metadata. This is useful for filtering results by entity type, or other metadata.\n",
-    "\n",
-    "More information here: https://docs.getzep.com/sdk/search_query/"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='Parable of the Sower is a science fiction novel by Octavia Butler, published in 1993. It follows the story of Lauren Olamina, a young woman living in a dystopian future where society has collapsed due to environmental disasters, poverty, and violence.', metadata={'score': 0.8897120952606201, 'uuid': 'f99ecec3-f778-4bfd-8bb7-c3c00ae919c0', 'created_at': '2023-10-17T22:53:08.664849Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'GPE', 'Matches': [{'End': 20, 'Start': 15, 'Text': 'Sower'}], 'Name': 'Sower'}, {'Label': 'PERSON', 'Matches': [{'End': 65, 'Start': 51, 'Text': 'Octavia Butler'}], 'Name': 'Octavia Butler'}, {'Label': 'DATE', 'Matches': [{'End': 84, 'Start': 80, 'Text': '1993'}], 'Name': '1993'}, {'Label': 'PERSON', 'Matches': [{'End': 124, 'Start': 110, 'Text': 'Lauren Olamina'}], 'Name': 'Lauren Olamina'}], 'intent': 'None'}}, 'token_count': 56}),\n",
-       " Document(page_content='Which books of hers were made into movies?', metadata={'score': 0.7496200799942017, 'uuid': '1047ff15-96f1-4101-bb0f-9ed073b8081d', 'created_at': '2023-10-17T22:53:08.596614Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'intent': 'The subject is inquiring about the books of the person referred to as \"hers\" that have been made into movies.'}}, 'token_count': 11}),\n",
-       " Document(page_content=\"Write a short synopsis of Butler's book, Parable of the Sower. What is it about?\", metadata={'score': 0.8857351541519165, 'uuid': 'f6aba470-f15f-4b22-84ef-1c0d315a31de', 'created_at': '2023-10-17T22:53:08.642659Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 32, 'Start': 26, 'Text': 'Butler'}], 'Name': 'Butler'}, {'Label': 'WORK_OF_ART', 'Matches': [{'End': 61, 'Start': 41, 'Text': 'Parable of the Sower'}], 'Name': 'Parable of the Sower'}], 'intent': 'The subject is requesting a brief summary or description of Butler\\'s book, \"Parable of the Sower.\"'}}, 'token_count': 23}),\n",
-       " Document(page_content='You might want to read Ursula K. Le Guin or Joanna Russ.', metadata={'score': 0.7595934867858887, 'uuid': '9fa630e6-0b17-4d77-80b0-ba99249850c0', 'created_at': '2023-10-17T22:53:08.630731Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'ai', 'metadata': {'system': {'entities': [{'Label': 'ORG', 'Matches': [{'End': 40, 'Start': 23, 'Text': 'Ursula K. Le Guin'}], 'Name': 'Ursula K. Le Guin'}, {'Label': 'PERSON', 'Matches': [{'End': 55, 'Start': 44, 'Text': 'Joanna Russ'}], 'Name': 'Joanna Russ'}], 'intent': 'The subject is providing a suggestion or recommendation for the person to read Ursula K. Le Guin or Joanna Russ.'}}, 'token_count': 18}),\n",
-       " Document(page_content='Who were her contemporaries?', metadata={'score': 0.7575579881668091, 'uuid': 'b2dfd1f7-cac6-4e37-94ea-7c15b0a5af2c', 'created_at': '2023-10-17T22:53:08.606283Z', 'updated_at': '0001-01-01T00:00:00Z', 'role': 'human', 'metadata': {'system': {'intent': 'The subject is asking about the people who were contemporaries of someone else.'}}, 'token_count': 8})]"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "filter = {\"where\": {\"jsonpath\": '$[*] ? (@.Label == \"WORK_OF_ART\")'}}\n",
-    "\n",
-    "await zep_retriever.aget_relevant_documents(\n",
-    "    \"Who wrote Parable of the Sower?\", metadata=filter\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
+   }
   }
  ],
  "metadata": {
@@ -416,7 +298,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.5"
+   "version": "3.11.4"
   }
  },
  "nbformat": 4,

docs/docs/integrations/text_embedding/bge_huggingface.ipynb

@@ -36,7 +36,7 @@
     "\n",
     "model_name = \"BAAI/bge-small-en\"\n",
     "model_kwargs = {'device': 'cpu'}\n",
-    "encode_kwargs = {'normalize_embeddings': True}\n",
+    "encode_kwargs = {'normalize_embeddings': False}\n",
     "hf = HuggingFaceBgeEmbeddings(\n",
     "    model_name=model_name,\n",
     "    model_kwargs=model_kwargs,\n",

docs/docs/integrations/text_embedding/sagemaker-endpoint.ipynb

@@ -43,7 +43,7 @@
     "from langchain.embeddings import SagemakerEndpointEmbeddings\n",
     "from langchain.embeddings.sagemaker_endpoint import EmbeddingsContentHandler\n",
     "import json\n",
-    "import boto3\n",
+    "\n",
     "\n",
     "class ContentHandler(EmbeddingsContentHandler):\n",
     "    content_type = \"application/json\"\n",
@@ -87,18 +87,7 @@
     "    endpoint_name=\"huggingface-pytorch-inference-2023-03-21-16-14-03-834\",\n",
     "    region_name=\"us-east-1\",\n",
     "    content_handler=content_handler,\n",
-    ")\n",
-    "\n",
-    "\n",
-    "# client = boto3.client(\n",
-    "#     \"sagemaker-runtime\",\n",
-    "#     region_name=\"us-west-2\"  \n",
-    "# )\n",
-    "# embeddings = SagemakerEndpointEmbeddings(\n",
-    "#     endpoint_name=\"huggingface-pytorch-inference-2023-03-21-16-14-03-834\",    \n",
-    "#     client=client\n",
-    "#     content_handler=content_handler,\n",
-    "# )"
+    ")"
    ]
   },
   {

docs/docs/integrations/toolkits/vectorstore.ipynb

@@ -30,8 +30,7 @@
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "from langchain.vectorstores import Chroma\n",
     "from langchain.text_splitter import CharacterTextSplitter\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.chains import VectorDBQA\n",
+    "from langchain.llms import OpenAI\nfrom langchain.chains import VectorDBQA\n",
     "\n",
     "llm = OpenAI(temperature=0)"
    ]
@@ -56,7 +55,7 @@
    "source": [
     "from langchain.document_loaders import TextLoader\n",
     "\n",
-    "loader = TextLoader(\"../../modules/state_of_the_union.txt\")\n",
+    "loader = TextLoader(\"../../../state_of_the_union.txt\")\n",
     "documents = loader.load()\n",
     "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
     "texts = text_splitter.split_documents(documents)\n",