Merge branch 'master' into rlm/sql-pgvector-template

.github/CONTRIBUTING.md

@@ -23,7 +23,7 @@ It's essential that we maintain great documentation and testing. If you:
   - Update any affected example notebooks and documentation. These live in `docs`.
   - Update unit and integration tests when relevant.
 - Add a feature
-  - Add a demo notebook in `docs/docs/`.
+  - Add a demo notebook in `docs/modules`.
   - Add unit and integration tests.
 
 We are a small, progress-oriented team. If there's something you'd like to add or change, opening a pull request is the
@@ -72,10 +72,9 @@ tell Poetry to use the virtualenv python environment (`poetry config virtualenvs
 
 ### Core vs. Experimental
 
-This repository contains three separate projects:
+This repository contains two separate projects:
 - `langchain`: core langchain code, abstractions, and use cases.
-- `langchain_core`: contain interfaces for key abstractions as well as logic for combining them in chains (LCEL).
-- `langchain_experimental`: see the [Experimental README](https://github.com/langchain-ai/langchain/tree/master/libs/experimental/README.md) for more information.
+- `langchain.experimental`: see the [Experimental README](https://github.com/langchain-ai/langchain/tree/master/libs/experimental/README.md) for more information.
 
 Each of these has its own development environment. Docs are run from the top-level makefile, but development
 is split across separate test & release flows.
@@ -129,24 +128,6 @@ make docker_tests
 
 There are also [integration tests and code-coverage](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/tests/README.md) available.
 
-### Only develop langchain_core or langchain_experimental
-
-If you are only developing `langchain_core` or `langchain_experimental`, you can simply install the dependencies for the respective projects and run tests:
-
-```bash
-cd libs/core
-poetry install --with test
-make test
-```
-
-Or:
-
-```bash
-cd libs/experimental
-poetry install --with test
-make test
-```
-
 ### Formatting and Linting
 
 Run these locally before submitting a PR; the CI system will check also.
@@ -233,10 +214,6 @@ ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogy
 
 Langchain relies heavily on optional dependencies to keep the Langchain package lightweight.
 
-You only need to add a new dependency if a **unit test** relies on the package.
-If your package is only required for **integration tests**, then you can skip these
-steps and leave all pyproject.toml and poetry.lock files alone.
-
 If you're adding a new dependency to Langchain, assume that it will be an optional dependency, and
 that most users won't have it installed.
 

.github/workflows/_lint.yml

@@ -68,7 +68,7 @@ jobs:
         # It doesn't matter how you change it, any change will cause a cache-bust.
         working-directory: ${{ inputs.working-directory }}
         run: |
-          poetry install --with lint,typing
+          poetry install --with dev,lint,test,typing
 
       - name: Install langchain editable
         working-directory: ${{ inputs.working-directory }}
@@ -76,7 +76,7 @@ jobs:
         env:
           LANGCHAIN_LOCATION: ${{ inputs.langchain-location }}
         run: |
-          poetry run pip install -e "$LANGCHAIN_LOCATION"
+          pip install -e "$LANGCHAIN_LOCATION"
 
       - name: Get .mypy_cache to speed up mypy
         uses: actions/cache@v3

.github/workflows/_pydantic_compatibility.yml

@@ -7,10 +7,6 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
-      langchain-location:
-        required: false
-        type: string
-        description: "Relative path to the langchain library folder"
 
 env:
   POETRY_VERSION: "1.6.1"
@@ -44,14 +40,6 @@ jobs:
         shell: bash
         run: poetry install
 
-      - name: Install langchain editable
-        working-directory: ${{ inputs.working-directory }}
-        if: ${{ inputs.langchain-location }}
-        env:
-          LANGCHAIN_LOCATION: ${{ inputs.langchain-location }}
-        run: |
-          poetry run pip install -e "$LANGCHAIN_LOCATION"
-
       - name: Install the opposite major version of pydantic
         # If normal tests use pydantic v1, here we'll use v2, and vice versa.
         shell: bash

.github/workflows/_release.yml

@@ -97,18 +97,19 @@ jobs:
         env:
           PKG_NAME: ${{ needs.build.outputs.pkg-name }}
           VERSION: ${{ needs.build.outputs.version }}
-        # Here we use:
-        # - The default regular PyPI index as the *primary* index, meaning 
-        #   that it takes priority (https://pypi.org/simple)
-        # - The test PyPI index as an extra index, so that any dependencies that
+        # Here we specify:
+        # - The test PyPI index as the *primary* index, meaning that it takes priority.
+        # - The regular PyPI index as an extra index, so that any dependencies that
         #   are not found on test PyPI can be resolved and installed anyway.
-        #   (https://test.pypi.org/simple). This will include the PKG_NAME==VERSION
-        #   package because VERSION will not have been uploaded to regular PyPI yet.
+        #
+        # Without the former, we might install the wrong langchain release.
+        # Without the latter, we might not be able to install langchain's dependencies.
         #
         # TODO: add more in-depth pre-publish tests after testing that importing works
         run: |
           pip install \
-            --extra-index-url https://test.pypi.org/simple/ \
+            --index-url https://test.pypi.org/simple/ \
+            --extra-index-url https://pypi.org/simple/ \
             "$PKG_NAME==$VERSION"
 
           # Replace all dashes in the package name with underscores,

.github/workflows/_test.yml

@@ -7,10 +7,6 @@ on:
         required: true
         type: string
         description: "From which folder this pipeline executes"
-      langchain-location:
-        required: false
-        type: string
-        description: "Relative path to the langchain library folder"
 
 env:
   POETRY_VERSION: "1.6.1"
@@ -42,20 +38,11 @@ jobs:
 
       - name: Install dependencies
         shell: bash
-        run: poetry install --with test
-
-      - name: Install langchain editable
-        working-directory: ${{ inputs.working-directory }}
-        if: ${{ inputs.langchain-location }}
-        env:
-          LANGCHAIN_LOCATION: ${{ inputs.langchain-location }}
-        run: |
-          poetry run pip install -e "$LANGCHAIN_LOCATION"
+        run: poetry install
 
       - name: Run core tests
         shell: bash
-        run: |
-          make test
+        run: make test
 
       - name: Ensure the tests did not create any additional files
         shell: bash

.github/workflows/extract_ignored_words_list.py

@@ -3,8 +3,6 @@ import toml
 pyproject_toml = toml.load("pyproject.toml")
 
 # Extract the ignore words list (adjust the key as per your TOML structure)
-ignore_words_list = (
-    pyproject_toml.get("tool", {}).get("codespell", {}).get("ignore-words-list")
-)
+ignore_words_list = pyproject_toml.get("tool", {}).get("codespell", {}).get("ignore-words-list")
 
-print(f"::set-output name=ignore_words_list::{ignore_words_list}")
+print(f"::set-output name=ignore_words_list::{ignore_words_list}")

.github/workflows/langchain_ci.yml

@@ -3,19 +3,18 @@ name: libs/langchain CI
 
 on:
   push:
-    branches: [master]
+    branches: [ master ]
   pull_request:
     paths:
-      - ".github/actions/poetry_setup/action.yml"
-      - ".github/tools/**"
-      - ".github/workflows/_lint.yml"
-      - ".github/workflows/_test.yml"
-      - ".github/workflows/_pydantic_compatibility.yml"
-      - ".github/workflows/langchain_ci.yml"
-      - "libs/*"
-      - "libs/langchain/**"
-      - "libs/core/**"
-  workflow_dispatch: # Allows to trigger the workflow manually in GitHub UI
+      - '.github/actions/poetry_setup/action.yml'
+      - '.github/tools/**'
+      - '.github/workflows/_lint.yml'
+      - '.github/workflows/_test.yml'
+      - '.github/workflows/_pydantic_compatibility.yml'
+      - '.github/workflows/langchain_ci.yml'
+      - 'libs/*'
+      - 'libs/langchain/**'
+  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
 # If another push to the same PR or branch happens while this workflow is still running,
 # cancel the earlier run in favor of the next run.
@@ -33,25 +32,29 @@ env:
 
 jobs:
   lint:
-    uses: ./.github/workflows/_lint.yml
+    uses:
+      ./.github/workflows/_lint.yml
     with:
       working-directory: libs/langchain
     secrets: inherit
 
   test:
-    uses: ./.github/workflows/_test.yml
+    uses:
+      ./.github/workflows/_test.yml
     with:
       working-directory: libs/langchain
     secrets: inherit
 
   compile-integration-tests:
-    uses: ./.github/workflows/_compile_integration_test.yml
+    uses:
+      ./.github/workflows/_compile_integration_test.yml
     with:
       working-directory: libs/langchain
     secrets: inherit
 
   pydantic-compatibility:
-    uses: ./.github/workflows/_pydantic_compatibility.yml
+    uses:
+      ./.github/workflows/_pydantic_compatibility.yml
     with:
       working-directory: libs/langchain
     secrets: inherit
@@ -86,11 +89,6 @@ jobs:
           echo "Running extended tests, installing dependencies with poetry..."
           poetry install -E extended_testing
 
-      - name: Install langchain core editable
-        shell: bash
-        run: |
-          poetry run pip install -e ../core
-
       - name: Run extended tests
         run: make extended_tests
 

.github/workflows/langchain_core_ci.yml

@@ -1,52 +0,0 @@
----
-name: libs/langchain core CI
-
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    paths:
-      - '.github/actions/poetry_setup/action.yml'
-      - '.github/tools/**'
-      - '.github/workflows/_lint.yml'
-      - '.github/workflows/_test.yml'
-      - '.github/workflows/_pydantic_compatibility.yml'
-      - '.github/workflows/langchain_core_ci.yml'
-      - 'libs/core/**'
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-# If another push to the same PR or branch happens while this workflow is still running,
-# cancel the earlier run in favor of the next run.
-#
-# There's no point in testing an outdated version of the code. GitHub only allows
-# a limited number of job runners to be active at the same time, so it's better to cancel
-# pointless jobs early so that more useful jobs can run sooner.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-env:
-  POETRY_VERSION: "1.6.1"
-  WORKDIR: "libs/core"
-
-jobs:
-  lint:
-    uses:
-      ./.github/workflows/_lint.yml
-    with:
-      working-directory: libs/core
-    secrets: inherit
-
-  test:
-    uses:
-      ./.github/workflows/_test.yml
-    with:
-      working-directory: libs/core
-    secrets: inherit
-
-  pydantic-compatibility:
-    uses:
-      ./.github/workflows/_pydantic_compatibility.yml
-    with:
-      working-directory: libs/core
-    secrets: inherit

.github/workflows/langchain_core_release.yml

@@ -1,13 +0,0 @@
----
-name: libs/core Release
-
-on:
-  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
-
-jobs:
-  release:
-    uses:
-      ./.github/workflows/_release.yml
-    with:
-      working-directory: libs/core
-    secrets: inherit

.github/workflows/langchain_experimental_ci.yml

@@ -3,19 +3,17 @@ name: libs/experimental CI
 
 on:
   push:
-    branches: [master]
+    branches: [ master ]
   pull_request:
     paths:
-      - ".github/actions/poetry_setup/action.yml"
-      - ".github/tools/**"
-      - ".github/workflows/_lint.yml"
-      - ".github/workflows/_test.yml"
-      - ".github/workflows/langchain_experimental_ci.yml"
-      - "libs/*"
-      - "libs/experimental/**"
-      - "libs/langchain/**"
-      - "libs/core/**"
-  workflow_dispatch: # Allows to trigger the workflow manually in GitHub UI
+      - '.github/actions/poetry_setup/action.yml'
+      - '.github/tools/**'
+      - '.github/workflows/_lint.yml'
+      - '.github/workflows/_test.yml'
+      - '.github/workflows/langchain_experimental_ci.yml'
+      - 'libs/*'
+      - 'libs/experimental/**'
+  workflow_dispatch:  # Allows to trigger the workflow manually in GitHub UI
 
 # If another push to the same PR or branch happens while this workflow is still running,
 # cancel the earlier run in favor of the next run.
@@ -33,19 +31,23 @@ env:
 
 jobs:
   lint:
-    uses: ./.github/workflows/_lint.yml
+    uses:
+      ./.github/workflows/_lint.yml
     with:
       working-directory: libs/experimental
+      langchain-location: ../langchain
     secrets: inherit
 
   test:
-    uses: ./.github/workflows/_test.yml
+    uses:
+      ./.github/workflows/_test.yml
     with:
       working-directory: libs/experimental
     secrets: inherit
 
   compile-integration-tests:
-    uses: ./.github/workflows/_compile_integration_test.yml
+    uses:
+      ./.github/workflows/_compile_integration_test.yml
     with:
       working-directory: libs/experimental
     secrets: inherit
@@ -86,7 +88,6 @@ jobs:
 
           echo "Editably installing langchain outside of poetry, to avoid messing up lockfile..."
           poetry run pip install -e ../langchain
-          poetry run pip install -e ../core
 
       - name: Run tests
         run: make test

LICENSE

@@ -1,6 +1,6 @@
-MIT License
+The MIT License
 
-Copyright (c) LangChain, Inc.
+Copyright (c) Harrison Chase
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -9,13 +9,13 @@ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

MIGRATE.md

@@ -1,18 +1,9 @@
-# Migrating
-
-## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28/23
-
-In an effort to make `langchain` leaner and safer, we are moving select chains to `langchain_experimental`.
-This migration has already started, but we are remaining backwards compatible until 7/28.
-On that date, we will remove functionality from `langchain`.
-Read more about the motivation and the progress [here](https://github.com/langchain-ai/langchain/discussions/8043).
-
-### Migrating to `langchain_experimental`
+# Migrating to `langchain_experimental`
 
 We are moving any experimental components of LangChain, or components with vulnerability issues, into `langchain_experimental`.
 This guide covers how to migrate.
 
-### Installation
+## Installation
 
 Previously:
 
@@ -22,7 +13,7 @@ Now (only if you want to access things in experimental):
 
 `pip install -U langchain langchain_experimental`
 
-### Things in `langchain.experimental`
+## Things in `langchain.experimental`
 
 Previously:
 
@@ -32,7 +23,7 @@ Now:
 
 `from langchain_experimental import ...`
 
-### PALChain
+## PALChain
 
 Previously:
 
@@ -42,7 +33,7 @@ Now:
 
 `from langchain_experimental.pal_chain import PALChain`
 
-### SQLDatabaseChain
+## SQLDatabaseChain
 
 Previously:
 
@@ -56,7 +47,7 @@ Alternatively, if you are just interested in using the query generation part of
 
 `from langchain.chains import create_sql_query_chain`
 
-### `load_prompt` for Python files
+## `load_prompt` for Python files
 
 Note: this only applies if you want to load Python files as prompts.
 If you want to load json/yaml files, no change is needed.

Makefile

@@ -43,11 +43,10 @@ spell_fix:
 
 lint:
 	poetry run ruff docs templates cookbook
-	poetry run ruff format docs templates cookbook --diff
-	poetry run ruff --select I docs templates cookbook
+	poetry run black docs templates cookbook --diff
 
 format format_diff:
-	poetry run ruff format docs templates cookbook
+	poetry run black docs templates cookbook
 	poetry run ruff --select I --fix docs templates cookbook
 
 ######################

README.md

@@ -15,72 +15,71 @@
 [![Dependency Status](https://img.shields.io/librariesio/github/langchain-ai/langchain)](https://libraries.io/github/langchain-ai/langchain)
 [![Open Issues](https://img.shields.io/github/issues-raw/langchain-ai/langchain)](https://github.com/langchain-ai/langchain/issues)
 
-Looking for the JS/TS library? Check out [LangChain.js](https://github.com/langchain-ai/langchainjs).
+
+Looking for the JS/TS version? Check out [LangChain.js](https://github.com/langchain-ai/langchainjs).
 
 To help you ship LangChain apps to production faster, check out [LangSmith](https://smith.langchain.com). 
 [LangSmith](https://smith.langchain.com) is a unified developer platform for building, testing, and monitoring LLM applications. 
-Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) to get off the waitlist or speak with our sales team.
+Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) to get off the waitlist or speak with our sales team
+
+## 🚨Breaking Changes for select chains (SQLDatabase) on 7/28/23
+
+In an effort to make `langchain` leaner and safer, we are moving select chains to `langchain_experimental`.
+This migration has already started, but we are remaining backwards compatible until 7/28.
+On that date, we will remove functionality from `langchain`.
+Read more about the motivation and the progress [here](https://github.com/langchain-ai/langchain/discussions/8043).
+Read how to migrate your code [here](MIGRATE.md).
 
 ## Quick Install
 
-With pip:
-```bash
-pip install langchain
-```
+`pip install langchain`
+or
+`pip install langsmith && conda install langchain -c conda-forge`
 
-With conda:
-```bash
-conda install langchain -c conda-forge
-```
+## 🤔 What is this?
 
-## 🤔 What is LangChain?
+Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. However, using these LLMs in isolation is often insufficient for creating a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.
 
-**LangChain** is a framework for developing applications powered by language models. It enables applications that:
-- **Are context-aware**: connect a language model to sources of context (prompt instructions, few shot examples, content to ground its response in, etc.)
-- **Reason**: rely on a language model to reason (about how to answer based on provided context, what actions to take, etc.)
+This library aims to assist in the development of those types of applications. Common examples of these applications include:
 
-This framework consists of several parts.
-- **LangChain Libraries**: The Python and JavaScript libraries. Contains interfaces and integrations for a myriad of components, a basic run time for combining these components into chains and agents, and off-the-shelf implementations of chains and agents.
-- **[LangChain Templates](templates)**: A collection of easily deployable reference architectures for a wide variety of tasks.
-- **[LangServe](https://github.com/langchain-ai/langserve)**: A library for deploying LangChain chains as a REST API.
-- **[LangSmith](https://smith.langchain.com)**: A developer platform that lets you debug, test, evaluate, and monitor chains built on any LLM framework and seamlessly integrates with LangChain.
-
-**This repo contains the `langchain` ([here](libs/langchain)), `langchain-experimental` ([here](libs/experimental)), and `langchain-cli` ([here](libs/cli)) Python packages, as well as [LangChain Templates](templates).**
-
-![LangChain Stack](docs/static/img/langchain_stack.png)
-
-## 🧱 What can you build with LangChain?
-**❓ Retrieval augmented generation**
+**❓ Question Answering over specific documents**
 
 - [Documentation](https://python.langchain.com/docs/use_cases/question_answering/)
-- End-to-end Example: [Chat LangChain](https://chat.langchain.com) and [repo](https://github.com/langchain-ai/chat-langchain)
+- End-to-end Example: [Question Answering over Notion Database](https://github.com/hwchase17/notion-qa)
 
-**💬 Analyzing structured data**
+**💬 Chatbots**
 
-- [Documentation](https://python.langchain.com/docs/use_cases/qa_structured/sql)
-- End-to-end Example: [SQL Llama2 Template](https://github.com/langchain-ai/langchain/tree/master/templates/sql-llama2)
+- [Documentation](https://python.langchain.com/docs/use_cases/chatbots/)
+- End-to-end Example: [Chat-LangChain](https://github.com/langchain-ai/chat-langchain)
 
-**🤖 Chatbots**
+**🤖 Agents**
 
-- [Documentation](https://python.langchain.com/docs/use_cases/chatbots)
-- End-to-end Example: [Web LangChain (web researcher chatbot)](https://weblangchain.vercel.app) and [repo](https://github.com/langchain-ai/weblangchain)
+- [Documentation](https://python.langchain.com/docs/modules/agents/)
+- End-to-end Example: [GPT+WolframAlpha](https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain)
 
-And much more! Head to the [Use cases](https://python.langchain.com/docs/use_cases/) section of the docs for more.
+## 📖 Documentation
 
-## 🚀 How does LangChain help?
-The main value props of the LangChain libraries are:
-1. **Components**: composable tools and integrations for working with language models. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not
-2. **Off-the-shelf chains**: built-in assemblages of components for accomplishing higher-level tasks
+Please see [here](https://python.langchain.com) for full documentation on:
 
-Off-the-shelf chains make it easy to get started. Components make it easy to customize existing chains and build new ones. 
+- Getting started (installation, setting up the environment, simple examples)
+- How-To examples (demos, integrations, helper functions)
+- Reference (full API docs)
+- Resources (high-level explanation of core concepts)
 
-Components fall into the following **modules**:
+## 🚀 What can this help with?
 
-**📃 Model I/O:**
+There are six main areas that LangChain is designed to help with.
+These are, in increasing order of complexity:
+
+**📃 LLMs and Prompts:**
 
 This includes prompt management, prompt optimization, a generic interface for all LLMs, and common utilities for working with LLMs.
 
-**📚 Retrieval:**
+**🔗 Chains:**
+
+Chains go beyond a single LLM call and involve sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.
+
+**📚 Data Augmented Generation:**
 
 Data Augmented Generation involves specific types of chains that first interact with an external data source to fetch data for use in the generation step. Examples include summarization of long pieces of text and question/answering over specific data sources.
 
@@ -88,23 +87,18 @@ Data Augmented Generation involves specific types of chains that first interact
 
 Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end-to-end agents.
 
-## 📖 Documentation
+**🧠 Memory:**
 
-Please see [here](https://python.langchain.com) for full documentation, which includes:
+Memory refers to persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.
 
-- [Getting started](https://python.langchain.com/docs/get_started/introduction): installation, setting up the environment, simple examples
-- Overview of the [interfaces](https://python.langchain.com/docs/expression_language/), [modules](https://python.langchain.com/docs/modules/) and [integrations](https://python.langchain.com/docs/integrations/providers)
-- [Use case](https://python.langchain.com/docs/use_cases/qa_structured/sql) walkthroughs and best practice [guides](https://python.langchain.com/docs/guides/adapters/openai)
-- [LangSmith](https://python.langchain.com/docs/langsmith/), [LangServe](https://python.langchain.com/docs/langserve), and [LangChain Template](https://python.langchain.com/docs/templates/) overviews
-- [Reference](https://api.python.langchain.com): full API docs
+**🧐 Evaluation:**
 
+[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is by using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.
+
+For more information on these concepts, please see our [full documentation](https://python.langchain.com).
 
 ## 💁 Contributing
 
 As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
 
 For detailed information on how to contribute, see [here](.github/CONTRIBUTING.md).
-
-## 🌟 Contributors
-
-[![langchain contributors](https://contrib.rocks/image?repo=langchain-ai/langchain&max=2000)](https://github.com/langchain-ai/langchain/graphs/contributors)

cookbook/LLaMA2_sql_chat.ipynb

@@ -67,6 +67,7 @@
     "llama2_code = ChatOllama(model=\"codellama:7b-instruct\")\n",
     "\n",
     "# API\n",
+    "from getpass import getpass\n",
     "from langchain.llms import Replicate\n",
     "\n",
     "# REPLICATE_API_TOKEN = getpass()\n",

cookbook/README.md

@@ -8,7 +8,6 @@ Notebook | Description
 [Semi_Structured_RAG.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_Structured_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data, including text and tables, using unstructured for parsing, multi-vector retriever for storing, and lcel for implementing chains.
 [Semi_structured_and_multi_moda...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_and_multi_modal_RAG.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using unstructured for parsing, multi-vector retriever for storage and retrieval, and lcel for implementing chains.
 [Semi_structured_multi_modal_RA...](https://github.com/langchain-ai/langchain/tree/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb) | Perform retrieval-augmented generation (rag) on documents with semi-structured data and images, using various tools and methods such as unstructured for parsing, multi-vector retriever for storing, lcel for implementing chains, and open source language models like llama2, llava, and gpt4all.
-[analyze_document.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/analyze_document.ipynb) | Analyze a single long document.
 [autogpt/autogpt.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/autogpt.ipynb) | Implement autogpt, a language model, with langchain primitives such as llms, prompttemplates, vectorstores, embeddings, and tools.
 [autogpt/marathon_times.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/autogpt/marathon_times.ipynb) | Implement autogpt for finding winning marathon times.
 [baby_agi.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/baby_agi.ipynb) | Implement babyagi, an ai agent that can generate and execute tasks based on a given objective, with the flexibility to swap out specific vectorstores/model providers.
@@ -45,7 +44,6 @@ Notebook | Description
 [plan_and_execute_agent.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/plan_and_execute_agent.ipynb) | Create plan-and-execute agents that accomplish objectives by planning tasks with a language model (llm) and executing them with a separate agent.
 [press_releases.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/press_releases.ipynb) | Retrieve and query company press release data powered by [Kay.ai](https://kay.ai).
 [program_aided_language_model.i...](https://github.com/langchain-ai/langchain/tree/master/cookbook/program_aided_language_model.ipynb) | Implement program-aided language models as described in the provided research paper.
-[qa_citations.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/qa_citations.ipynb) | Different ways to get a model to cite its sources.
 [retrieval_in_sql.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/retrieval_in_sql.ipynb) | Perform retrieval-augmented-generation (rag) on a PostgreSQL database using pgvector.
 [sales_agent_with_context.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/sales_agent_with_context.ipynb) | Implement a context-aware ai sales agent, salesgpt, that can have natural sales conversations, interact with other systems, and use a product knowledge base to discuss a company's offerings.
 [self_query_hotel_search.ipynb](https://github.com/langchain-ai/langchain/tree/master/cookbook/self_query_hotel_search.ipynb) | Build a hotel room search feature with self-querying retrieval, using a specific hotel recommendation dataset.

cookbook/Semi_Structured_RAG.ipynb

@@ -102,9 +102,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Any\n",
-    "\n",
+    "from lxml import html\n",
     "from pydantic import BaseModel\n",
+    "from typing import Any, Optional\n",
     "from unstructured.partition.pdf import partition_pdf\n",
     "\n",
     "# Get elements\n",
@@ -317,12 +317,11 @@
    "outputs": [],
    "source": [
     "import uuid\n",
-    "\n",
+    "from langchain.vectorstores import Chroma\n",
+    "from langchain.storage import InMemoryStore\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
-    "from langchain.schema.document import Document\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(collection_name=\"summaries\", embedding_function=OpenAIEmbeddings())\n",
@@ -374,6 +373,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from operator import itemgetter\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",

cookbook/Semi_structured_and_multi_modal_RAG.ipynb

@@ -92,9 +92,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Any\n",
-    "\n",
+    "from lxml import html\n",
     "from pydantic import BaseModel\n",
+    "from typing import Any, Optional\n",
     "from unstructured.partition.pdf import partition_pdf\n",
     "\n",
     "# Get elements\n",
@@ -224,7 +224,7 @@
    "outputs": [],
    "source": [
     "# Prompt\n",
-    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\\n",
+    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\ \n",
     "Give a concise summary of the table or text. Table or text chunk: {element} \"\"\"\n",
     "prompt = ChatPromptTemplate.from_template(prompt_text)\n",
     "\n",
@@ -313,7 +313,7 @@
     "    # Execute the command and save the output to the defined output file\n",
     "    /Users/rlm/Desktop/Code/llama.cpp/bin/llava -m ../models/llava-7b/ggml-model-q5_k.gguf --mmproj ../models/llava-7b/mmproj-model-f16.gguf --temp 0.1 -p \"Describe the image in detail. Be specific about graphs, such as bar plots.\" --image \"$img\" > \"$output_file\"\n",
     "\n",
-    "done\n"
+    "done"
    ]
   },
   {
@@ -337,8 +337,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import glob\n",
-    "import os\n",
+    "import os, glob\n",
     "\n",
     "# Get all .txt file summaries\n",
     "file_paths = glob.glob(os.path.expanduser(os.path.join(path, \"*.txt\")))\n",
@@ -372,12 +371,11 @@
    "outputs": [],
    "source": [
     "import uuid\n",
-    "\n",
+    "from langchain.vectorstores import Chroma\n",
+    "from langchain.storage import InMemoryStore\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
-    "from langchain.schema.document import Document\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(collection_name=\"summaries\", embedding_function=OpenAIEmbeddings())\n",
@@ -646,6 +644,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from operator import itemgetter\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",

cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb

@@ -82,9 +82,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Any\n",
-    "\n",
+    "import pandas as pd\n",
+    "from lxml import html\n",
     "from pydantic import BaseModel\n",
+    "from typing import Any, Optional\n",
     "from unstructured.partition.pdf import partition_pdf\n",
     "\n",
     "# Path to save images\n",
@@ -222,7 +223,7 @@
    "outputs": [],
    "source": [
     "# Prompt\n",
-    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\\n",
+    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text. \\ \n",
     "Give a concise summary of the table or text. Table or text chunk: {element} \"\"\"\n",
     "prompt = ChatPromptTemplate.from_template(prompt_text)\n",
     "\n",
@@ -311,7 +312,7 @@
     "    # Execute the command and save the output to the defined output file\n",
     "    /Users/rlm/Desktop/Code/llama.cpp/bin/llava -m ../models/llava-7b/ggml-model-q5_k.gguf --mmproj ../models/llava-7b/mmproj-model-f16.gguf --temp 0.1 -p \"Describe the image in detail. Be specific about graphs, such as bar plots.\" --image \"$img\" > \"$output_file\"\n",
     "\n",
-    "done\n"
+    "done"
    ]
   },
   {
@@ -321,8 +322,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import glob\n",
-    "import os\n",
+    "import os, glob\n",
     "\n",
     "# Get all .txt files in the directory\n",
     "file_paths = glob.glob(os.path.expanduser(os.path.join(path, \"*.txt\")))\n",
@@ -375,12 +375,11 @@
    ],
    "source": [
     "import uuid\n",
-    "\n",
+    "from langchain.vectorstores import Chroma\n",
+    "from langchain.storage import InMemoryStore\n",
+    "from langchain.schema.document import Document\n",
     "from langchain.embeddings import GPT4AllEmbeddings\n",
     "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
-    "from langchain.schema.document import Document\n",
-    "from langchain.storage import InMemoryStore\n",
-    "from langchain.vectorstores import Chroma\n",
     "\n",
     "# The vectorstore to use to index the child chunks\n",
     "vectorstore = Chroma(\n",
@@ -532,6 +531,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from operator import itemgetter\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt template\n",

cookbook/advanced_rag_eval.ipynb

@@ -63,13 +63,11 @@
     "\n",
     "# Load\n",
     "from langchain.document_loaders import PyPDFLoader\n",
-    "\n",
     "loader = PyPDFLoader(path + \"cpi.pdf\")\n",
     "pdf_pages = loader.load()\n",
     "\n",
     "# Split\n",
     "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "\n",
     "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
     "all_splits_pypdf = text_splitter.split_documents(pdf_pages)\n",
     "all_splits_pypdf_texts = [d.page_content for d in all_splits_pypdf]"
@@ -132,15 +130,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.vectorstores import Chroma\n",
-    "\n",
-    "baseline = Chroma.from_texts(\n",
-    "    texts=all_splits_pypdf_texts,\n",
-    "    collection_name=\"baseline\",\n",
-    "    embedding=OpenAIEmbeddings(),\n",
-    ")\n",
-    "retriever_baseline = baseline.as_retriever()"
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "baseline = Chroma.from_texts(texts=all_splits_pypdf_texts,\n",
+    "                             collection_name=\"baseline\",\n",
+    "                             embedding=OpenAIEmbeddings())\n",
+    "retriever_baseline=baseline.as_retriever()"
    ]
   },
   {
@@ -165,7 +160,7 @@
     "from langchain.schema.output_parser import StrOutputParser\n",
     "\n",
     "# Prompt\n",
-    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\\n",
+    "prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\ \n",
     "These summaries will be embedded and used to retrieve the raw text or table elements. \\\n",
     "Give a concise summary of the table or text that is well optimized for retrieval. Table or text: {element} \"\"\"\n",
     "prompt = ChatPromptTemplate.from_template(prompt_text)\n",
@@ -174,7 +169,7 @@
     "model = ChatOpenAI(temperature=0, model=\"gpt-4\")\n",
     "summarize_chain = {\"element\": lambda x: x} | prompt | model | StrOutputParser()\n",
     "\n",
-    "# Apply to text\n",
+    "# Apply to text \n",
     "text_summaries = summarize_chain.batch(texts, {\"max_concurrency\": 5})\n",
     "\n",
     "# Apply to tables\n",
@@ -197,33 +192,31 @@
    "outputs": [],
    "source": [
     "# Image summary chain\n",
-    "import base64\n",
-    "import io\n",
-    "import os\n",
+    "import os, base64, io\n",
     "from io import BytesIO\n",
-    "\n",
-    "from langchain.schema.messages import HumanMessage\n",
     "from PIL import Image\n",
-    "\n",
+    "from langchain.schema.messages import HumanMessage\n",
     "\n",
     "def encode_image(image_path):\n",
-    "    \"\"\"Getting the base64 string\"\"\"\n",
+    "    ''' Getting the base64 string '''\n",
     "    with open(image_path, \"rb\") as image_file:\n",
-    "        return base64.b64encode(image_file.read()).decode(\"utf-8\")\n",
-    "\n",
-    "\n",
-    "def image_summarize(img_base64, prompt):\n",
-    "    \"\"\"Image summary\"\"\"\n",
-    "    chat = ChatOpenAI(model=\"gpt-4-vision-preview\", max_tokens=1024)\n",
+    "        return base64.b64encode(image_file.read()).decode('utf-8') \n",
     "\n",
+    "def image_summarize(img_base64,prompt):\n",
+    "    ''' Image summary '''\n",
+    "    chat = ChatOpenAI(model=\"gpt-4-vision-preview\",\n",
+    "                      max_tokens=1024)\n",
+    "    \n",
     "    msg = chat.invoke(\n",
     "        [\n",
     "            HumanMessage(\n",
     "                content=[\n",
-    "                    {\"type\": \"text\", \"text\": prompt},\n",
+    "                    {\"type\": \"text\", \"text\":prompt},\n",
     "                    {\n",
     "                        \"type\": \"image_url\",\n",
-    "                        \"image_url\": {\"url\": f\"data:image/jpeg;base64,{img_base64}\"},\n",
+    "                        \"image_url\": {\n",
+    "                            \"url\": f\"data:image/jpeg;base64,{img_base64}\"\n",
+    "                        },\n",
     "                    },\n",
     "                ]\n",
     "            )\n",
@@ -231,7 +224,6 @@
     "    )\n",
     "    return msg.content\n",
     "\n",
-    "\n",
     "# Store base64 encoded images\n",
     "img_base64_list = []\n",
     "\n",
@@ -239,17 +231,17 @@
     "image_summaries = []\n",
     "\n",
     "# Prompt\n",
-    "prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\\n",
+    "prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\ \n",
     "These summaries will be embedded and used to retrieve the raw image. \\\n",
     "Give a concise summary of the image that is well optimized for retrieval.\"\"\"\n",
     "\n",
     "# Apply to images\n",
     "for img_file in sorted(os.listdir(path)):\n",
-    "    if img_file.endswith(\".jpg\"):\n",
+    "    if img_file.endswith('.jpg'):\n",
     "        img_path = os.path.join(path, img_file)\n",
     "        base64_image = encode_image(img_path)\n",
     "        img_base64_list.append(base64_image)\n",
-    "        image_summaries.append(image_summarize(base64_image, prompt))"
+    "        image_summaries.append(image_summarize(base64_image,prompt))"
    ]
   },
   {
@@ -271,15 +263,18 @@
    "source": [
     "import uuid\n",
     "from base64 import b64decode\n",
-    "\n",
-    "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
-    "from langchain.schema.document import Document\n",
     "from langchain.storage import InMemoryStore\n",
+    "from langchain.schema.document import Document\n",
+    "from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
     "\n",
-    "\n",
-    "def create_multi_vector_retriever(\n",
-    "    vectorstore, text_summaries, texts, table_summaries, tables, image_summaries, images\n",
-    "):\n",
+    "def create_multi_vector_retriever(vectorstore, \n",
+    "                                  text_summaries, \n",
+    "                                  texts, \n",
+    "                                  table_summaries, \n",
+    "                                  tables, \n",
+    "                                  image_summaries, \n",
+    "                                  images):\n",
+    "    \n",
     "    # Initialize the storage layer\n",
     "    store = InMemoryStore()\n",
     "    id_key = \"doc_id\"\n",
@@ -314,22 +309,18 @@
     "\n",
     "    return retriever\n",
     "\n",
-    "\n",
     "# The vectorstore to use to index the summaries\n",
-    "multi_vector_img = Chroma(\n",
-    "    collection_name=\"multi_vector_img\", embedding_function=OpenAIEmbeddings()\n",
-    ")\n",
+    "multi_vector_img = Chroma(collection_name=\"multi_vector_img\", \n",
+    "                     embedding_function=OpenAIEmbeddings())\n",
     "\n",
     "# Create retriever\n",
-    "retriever_multi_vector_img = create_multi_vector_retriever(\n",
-    "    multi_vector_img,\n",
-    "    text_summaries,\n",
-    "    texts,\n",
-    "    table_summaries,\n",
-    "    tables,\n",
-    "    image_summaries,\n",
-    "    img_base64_list,\n",
-    ")"
+    "retriever_multi_vector_img = create_multi_vector_retriever(multi_vector_img,\n",
+    "                              text_summaries,\n",
+    "                              texts,\n",
+    "                              table_summaries, \n",
+    "                              tables, \n",
+    "                              image_summaries, \n",
+    "                              img_base64_list)"
    ]
   },
   {
@@ -339,10 +330,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Testing on retrieval\n",
-    "query = \"What percentage of CPI is dedicated to Housing, and how does it compare to the combined percentage of Medical Care, Apparel, and Other Goods and Services?\"\n",
-    "suffix_for_images = \" Include any pie charts, graphs, or tables.\"\n",
-    "docs = retriever_multi_vector_img.get_relevant_documents(query + suffix_for_images)"
+    "# Testing on retrieval \n",
+    "query=\"What percentage of CPI is dedicated to Housing, and how does it compare to the combined percentage of Medical Care, Apparel, and Other Goods and Services?\"\n",
+    "suffix_for_images=\" Include any pie charts, graphs, or tables.\"\n",
+    "docs = retriever_multi_vector_img.get_relevant_documents(query+suffix_for_images)"
    ]
   },
   {
@@ -365,17 +356,15 @@
     }
    ],
    "source": [
-    "from IPython.display import HTML, display\n",
-    "\n",
-    "\n",
+    "from IPython.display import display, HTML\n",
     "def plt_img_base64(img_base64):\n",
+    "\n",
     "    # Create an HTML img tag with the base64 string as the source\n",
     "    image_html = f'<img src=\"data:image/jpeg;base64,{img_base64}\" />'\n",
-    "\n",
+    "    \n",
     "    # Display the image by rendering the HTML\n",
     "    display(HTML(image_html))\n",
     "\n",
-    "\n",
     "plt_img_base64(docs[1])"
    ]
   },
@@ -397,20 +386,17 @@
    "outputs": [],
    "source": [
     "# The vectorstore to use to index the summaries\n",
-    "multi_vector_text = Chroma(\n",
-    "    collection_name=\"multi_vector_text\", embedding_function=OpenAIEmbeddings()\n",
-    ")\n",
+    "multi_vector_text = Chroma(collection_name=\"multi_vector_text\", \n",
+    "                           embedding_function=OpenAIEmbeddings())\n",
     "\n",
     "# Create retriever\n",
-    "retriever_multi_vector_img_summary = create_multi_vector_retriever(\n",
-    "    multi_vector_text,\n",
-    "    text_summaries,\n",
-    "    texts,\n",
-    "    table_summaries,\n",
-    "    tables,\n",
-    "    image_summaries,\n",
-    "    image_summaries,\n",
-    ")"
+    "retriever_multi_vector_img_summary = create_multi_vector_retriever(multi_vector_text,\n",
+    "                              text_summaries,\n",
+    "                              texts,\n",
+    "                              table_summaries, \n",
+    "                              tables, \n",
+    "                              image_summaries, \n",
+    "                              image_summaries)"
    ]
   },
   {
@@ -432,17 +418,14 @@
     "\n",
     "# Create chroma w/ multi-modal embeddings\n",
     "multimodal_embd = Chroma(\n",
-    "    collection_name=\"multimodal_embd\", embedding_function=OpenCLIPEmbeddings()\n",
+    "    collection_name=\"multimodal_embd\",\n",
+    "    embedding_function=OpenCLIPEmbeddings()\n",
     ")\n",
     "\n",
     "# Get image URIs\n",
-    "image_uris = sorted(\n",
-    "    [\n",
-    "        os.path.join(path, image_name)\n",
-    "        for image_name in os.listdir(path)\n",
-    "        if image_name.endswith(\".jpg\")\n",
-    "    ]\n",
-    ")\n",
+    "image_uris = sorted([os.path.join(path, image_name) \n",
+    "                     for image_name in os.listdir(path) \n",
+    "                     if image_name.endswith('.jpg')])\n",
     "\n",
     "# Add images and documents\n",
     "if image_uris:\n",
@@ -452,7 +435,7 @@
     "if tables:\n",
     "    multimodal_embd.add_texts(texts=tables)\n",
     "\n",
-    "# Make retriever\n",
+    "# Make retriever \n",
     "retriever_multimodal_embd = multimodal_embd.as_retriever()"
    ]
   },
@@ -474,7 +457,6 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
-    "\n",
     "from langchain.schema.runnable import RunnablePassthrough\n",
     "\n",
     "# Prompt\n",
@@ -484,14 +466,14 @@
     "\"\"\"\n",
     "rag_prompt_text = ChatPromptTemplate.from_template(template)\n",
     "\n",
-    "\n",
-    "# Build\n",
+    "# Build \n",
     "def text_rag_chain(retriever):\n",
-    "    \"\"\"RAG chain\"\"\"\n",
+    "    \n",
+    "    ''' RAG chain '''\n",
     "\n",
     "    # LLM\n",
     "    model = ChatOpenAI(temperature=0, model=\"gpt-4\")\n",
-    "\n",
+    "    \n",
     "    # RAG pipeline\n",
     "    chain = (\n",
     "        {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
@@ -518,16 +500,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re\n",
-    "\n",
+    "import re \n",
     "from langchain.schema import Document\n",
     "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
-    "\n",
     "def looks_like_base64(sb):\n",
     "    \"\"\"Check if the string looks like base64.\"\"\"\n",
-    "    return re.match(\"^[A-Za-z0-9+/]+[=]{0,2}$\", sb) is not None\n",
-    "\n",
+    "    return re.match('^[A-Za-z0-9+/]+[=]{0,2}$', sb) is not None\n",
     "\n",
     "def is_image_data(b64data):\n",
     "    \"\"\"Check if the base64 data is an image by looking at the start of the data.\"\"\"\n",
@@ -535,7 +514,7 @@
     "        b\"\\xFF\\xD8\\xFF\": \"jpg\",\n",
     "        b\"\\x89\\x50\\x4E\\x47\\x0D\\x0A\\x1A\\x0A\": \"png\",\n",
     "        b\"\\x47\\x49\\x46\\x38\": \"gif\",\n",
-    "        b\"\\x52\\x49\\x46\\x46\": \"webp\",\n",
+    "        b\"\\x52\\x49\\x46\\x46\": \"webp\"\n",
     "    }\n",
     "    try:\n",
     "        header = base64.b64decode(b64data)[:8]  # Decode and get the first 8 bytes\n",
@@ -546,7 +525,6 @@
     "    except Exception:\n",
     "        return False\n",
     "\n",
-    "\n",
     "def split_image_text_types(docs):\n",
     "    \"\"\"Split base64-encoded images and texts.\"\"\"\n",
     "    b64_images = []\n",
@@ -561,7 +539,6 @@
     "            texts.append(doc)\n",
     "    return {\"images\": b64_images, \"texts\": texts}\n",
     "\n",
-    "\n",
     "def img_prompt_func(data_dict):\n",
     "    # Joining the context texts into a single string\n",
     "    formatted_texts = \"\\n\".join(data_dict[\"context\"][\"texts\"])\n",
@@ -573,7 +550,7 @@
     "            \"type\": \"image_url\",\n",
     "            \"image_url\": {\n",
     "                \"url\": f\"data:image/jpeg;base64,{data_dict['context']['images'][0]}\"\n",
-    "            },\n",
+    "            }\n",
     "        }\n",
     "        messages.append(image_message)\n",
     "\n",
@@ -586,24 +563,22 @@
     "            f\"User-provided question / keywords: {data_dict['question']}\\n\\n\"\n",
     "            \"Text and / or tables:\\n\"\n",
     "            f\"{formatted_texts}\"\n",
-    "        ),\n",
+    "        )\n",
     "    }\n",
     "    messages.append(text_message)\n",
     "    return [HumanMessage(content=messages)]\n",
     "\n",
-    "\n",
     "def multi_modal_rag_chain(retriever):\n",
-    "    \"\"\"Multi-modal RAG chain\"\"\"\n",
+    "    ''' Multi-modal RAG chain '''\n",
     "\n",
     "    # Multi-modal LLM\n",
-    "    model = ChatOpenAI(temperature=0, model=\"gpt-4-vision-preview\", max_tokens=1024)\n",
-    "\n",
+    "    model = ChatOpenAI(temperature=0, \n",
+    "                       model=\"gpt-4-vision-preview\", \n",
+    "                       max_tokens=1024)\n",
+    "    \n",
     "    # RAG pipeline\n",
     "    chain = (\n",
-    "        {\n",
-    "            \"context\": retriever | RunnableLambda(split_image_text_types),\n",
-    "            \"question\": RunnablePassthrough(),\n",
-    "        }\n",
+    "        {\"context\": retriever | RunnableLambda(split_image_text_types), \"question\": RunnablePassthrough()}\n",
     "        | RunnableLambda(img_prompt_func)\n",
     "        | model\n",
     "        | StrOutputParser()\n",
@@ -628,12 +603,12 @@
    "outputs": [],
    "source": [
     "# RAG chains\n",
-    "chain_baseline = text_rag_chain(retriever_baseline)\n",
-    "chain_mv_text = text_rag_chain(retriever_multi_vector_img_summary)\n",
+    "chain_baseline=text_rag_chain(retriever_baseline)\n",
+    "chain_mv_text=text_rag_chain(retriever_multi_vector_img_summary)\n",
     "\n",
     "# Multi-modal RAG chains\n",
-    "chain_multimodal_mv_img = multi_modal_rag_chain(retriever_multi_vector_img)\n",
-    "chain_multimodal_embd = multi_modal_rag_chain(retriever_multimodal_embd)"
+    "chain_multimodal_mv_img=multi_modal_rag_chain(retriever_multi_vector_img)\n",
+    "chain_multimodal_embd=multi_modal_rag_chain(retriever_multimodal_embd)"
    ]
   },
   {
@@ -719,8 +694,7 @@
    "source": [
     "# Read\n",
     "import pandas as pd\n",
-    "\n",
-    "eval_set = pd.read_csv(path + \"cpi_eval.csv\")\n",
+    "eval_set = pd.read_csv(path+'cpi_eval.csv')\n",
     "eval_set.head(3)"
    ]
   },
@@ -741,12 +715,12 @@
     "# Populate dataset\n",
     "for _, row in eval_set.iterrows():\n",
     "    # Get Q, A\n",
-    "    q = row[\"Question\"]\n",
-    "    a = row[\"Answer\"]\n",
+    "    q = row['Question']\n",
+    "    a = row['Answer']\n",
     "    # Use the values in your function\n",
-    "    client.create_example(\n",
-    "        inputs={\"question\": q}, outputs={\"answer\": a}, dataset_id=dataset.id\n",
-    "    )"
+    "    client.create_example(inputs={\"question\": q}, \n",
+    "                          outputs={\"answer\": a}, \n",
+    "                          dataset_id=dataset.id)"
    ]
   },
   {
@@ -790,22 +764,17 @@
     "    evaluators=[\"qa\"],\n",
     ")\n",
     "\n",
-    "\n",
-    "def run_eval(chain, run_name, dataset_name):\n",
+    "def run_eval(chain,run_name,dataset_name):\n",
     "    _ = client.run_on_dataset(\n",
     "        dataset_name=dataset_name,\n",
-    "        llm_or_chain_factory=lambda: (lambda x: x[\"question\"] + suffix_for_images)\n",
-    "        | chain,\n",
+    "        llm_or_chain_factory=lambda: (lambda x: x[\"question\"]+suffix_for_images) | chain,\n",
     "        evaluation=eval_config,\n",
     "        project_name=run_name,\n",
     "    )\n",
     "\n",
-    "\n",
-    "for chain, run in zip(\n",
-    "    [chain_baseline, chain_mv_text, chain_multimodal_mv_img, chain_multimodal_embd],\n",
-    "    [\"baseline\", \"mv_text\", \"mv_img\", \"mm_embd\"],\n",
-    "):\n",
-    "    run_eval(chain, dataset_name + \"-\" + run, dataset_name)"
+    "for chain, run in zip([chain_baseline, chain_mv_text, chain_multimodal_mv_img, chain_multimodal_embd], \n",
+    "                      [\"baseline\", \"mv_text\", \"mv_img\", \"mm_embd\"]):\n",
+    "    run_eval(chain, dataset_name+\"-\"+run, dataset_name)"
    ]
   }
  ],

cookbook/analyze_document.ipynb

@@ -1,105 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "f69d4a4c-137d-47e9-bea1-786afce9c1c0",
-   "metadata": {},
-   "source": [
-    "# Analyze a single long document\n",
-    "\n",
-    "The AnalyzeDocumentChain takes in a single document, splits it up, and then runs it through a CombineDocumentsChain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "2a0707ce-6d2d-471b-bc33-64da32a7b3f0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with open(\"../docs/docs/modules/state_of_the_union.txt\") as f:\n",
-    "    state_of_the_union = f.read()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "ca14d161-2d5b-4a6c-a296-77d8ce4b28cd",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import AnalyzeDocumentChain\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "9f97406c-85a9-45fb-99ce-9138c0ba3731",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains.question_answering import load_qa_chain\n",
-    "\n",
-    "qa_chain = load_qa_chain(llm, chain_type=\"map_reduce\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "0871a753-f5bb-4b4f-a394-f87f2691f659",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "qa_document_chain = AnalyzeDocumentChain(combine_docs_chain=qa_chain)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "e6f86428-3c2c-46a0-a57c-e22826fdbf91",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'The President said, \"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.\"'"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "qa_document_chain.run(\n",
-    "    input_document=state_of_the_union,\n",
-    "    question=\"what did the president say about justice breyer?\",\n",
-    ")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

cookbook/autogpt/autogpt.ipynb

@@ -27,10 +27,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import Tool\n",
-    "from langchain.tools.file_management.read import ReadFileTool\n",
-    "from langchain.tools.file_management.write import WriteFileTool\n",
     "from langchain.utilities import SerpAPIWrapper\n",
+    "from langchain.agents import Tool\n",
+    "from langchain.tools.file_management.write import WriteFileTool\n",
+    "from langchain.tools.file_management.read import ReadFileTool\n",
     "\n",
     "search = SerpAPIWrapper()\n",
     "tools = [\n",
@@ -61,9 +61,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.vectorstores import FAISS\n",
     "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.vectorstores import FAISS"
+    "from langchain.embeddings import OpenAIEmbeddings"
    ]
   },
   {
@@ -100,8 +100,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain_experimental.autonomous_agents import AutoGPT"
+    "from langchain_experimental.autonomous_agents import AutoGPT\n",
+    "from langchain.chat_models import ChatOpenAI"
    ]
   },
   {

cookbook/autogpt/marathon_times.ipynb

@@ -34,15 +34,16 @@
    "outputs": [],
    "source": [
     "# General\n",
-    "import asyncio\n",
     "import os\n",
-    "\n",
-    "import nest_asyncio\n",
     "import pandas as pd\n",
-    "from langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.docstore.document import Document\n",
     "from langchain_experimental.autonomous_agents import AutoGPT\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "\n",
+    "from langchain.agents.agent_toolkits.pandas.base import create_pandas_dataframe_agent\n",
+    "from langchain.docstore.document import Document\n",
+    "import asyncio\n",
+    "import nest_asyncio\n",
+    "\n",
     "\n",
     "# Needed synce jupyter runs an async eventloop\n",
     "nest_asyncio.apply()"
@@ -91,7 +92,6 @@
     "import os\n",
     "from contextlib import contextmanager\n",
     "from typing import Optional\n",
-    "\n",
     "from langchain.agents import tool\n",
     "from langchain.tools.file_management.read import ReadFileTool\n",
     "from langchain.tools.file_management.write import WriteFileTool\n",
@@ -223,13 +223,14 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.qa_with_sources.loading import (\n",
-    "    BaseCombineDocumentsChain,\n",
-    "    load_qa_with_sources_chain,\n",
-    ")\n",
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
     "from langchain.tools import BaseTool, DuckDuckGoSearchRun\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "\n",
     "from pydantic import Field\n",
+    "from langchain.chains.qa_with_sources.loading import (\n",
+    "    load_qa_with_sources_chain,\n",
+    "    BaseCombineDocumentsChain,\n",
+    ")\n",
     "\n",
     "\n",
     "def _get_text_splitter():\n",
@@ -310,9 +311,10 @@
    "source": [
     "# Memory\n",
     "import faiss\n",
+    "from langchain.vectorstores import FAISS\n",
     "from langchain.docstore import InMemoryDocstore\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.vectorstores import FAISS\n",
+    "from langchain.tools.human.tool import HumanInputRun\n",
     "\n",
     "embeddings_model = OpenAIEmbeddings()\n",
     "embedding_size = 1536\n",

cookbook/baby_agi.ipynb

@@ -29,10 +29,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Optional\n",
+    "import os\n",
+    "from collections import deque\n",
+    "from typing import Dict, List, Optional, Any\n",
     "\n",
+    "from langchain.chains import LLMChain\nfrom langchain.llms import OpenAI\nfrom langchain.prompts import PromptTemplate\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.llms import OpenAI\n",
+    "from langchain.llms import BaseLLM\n",
+    "from langchain.schema.vectorstore import VectorStore\n",
+    "from pydantic import BaseModel, Field\n",
+    "from langchain.chains.base import Chain\n",
     "from langchain_experimental.autonomous_agents import BabyAGI"
    ]
   },
@@ -53,8 +59,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS\n",
+    "from langchain.docstore import InMemoryDocstore"
    ]
   },
   {

cookbook/baby_agi_with_agent.ipynb

@@ -25,12 +25,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Optional\n",
+    "import os\n",
+    "from collections import deque\n",
+    "from typing import Dict, List, Optional, Any\n",
     "\n",
-    "from langchain.chains import LLMChain\n",
+    "from langchain.chains import LLMChain\nfrom langchain.llms import OpenAI\nfrom langchain.prompts import PromptTemplate\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.llms import BaseLLM\n",
+    "from langchain.schema.vectorstore import VectorStore\n",
+    "from pydantic import BaseModel, Field\n",
+    "from langchain.chains.base import Chain\n",
     "from langchain_experimental.autonomous_agents import BabyAGI"
    ]
   },
@@ -62,8 +66,8 @@
    "source": [
     "%pip install faiss-cpu > /dev/null\n",
     "%pip install google-search-results > /dev/null\n",
-    "from langchain.docstore import InMemoryDocstore\n",
-    "from langchain.vectorstores import FAISS"
+    "from langchain.vectorstores import FAISS\n",
+    "from langchain.docstore import InMemoryDocstore"
    ]
   },
   {
@@ -106,10 +110,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentExecutor, Tool, ZeroShotAgent\n",
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.utilities import SerpAPIWrapper\n",
+    "from langchain.agents import ZeroShotAgent, Tool, AgentExecutor\n",
+    "from langchain.llms import OpenAI\nfrom langchain.utilities import SerpAPIWrapper\nfrom langchain.chains import LLMChain\n",
     "\n",
     "todo_prompt = PromptTemplate.from_template(\n",
     "    \"You are a planner who is an expert at coming up with a todo list for a given objective. Come up with a todo list for this objective: {objective}\"\n",

cookbook/camel_role_playing.ipynb

@@ -35,17 +35,16 @@
    "outputs": [],
    "source": [
     "from typing import List\n",
-    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts.chat import (\n",
-    "    HumanMessagePromptTemplate,\n",
     "    SystemMessagePromptTemplate,\n",
+    "    HumanMessagePromptTemplate,\n",
     ")\n",
     "from langchain.schema import (\n",
     "    AIMessage,\n",
-    "    BaseMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
+    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/causal_program_aided_language_model.ipynb

@@ -47,9 +47,10 @@
    "outputs": [],
    "source": [
     "from IPython.display import SVG\n",
-    "from langchain.llms import OpenAI\n",
+    "\n",
     "from langchain_experimental.cpal.base import CPALChain\n",
     "from langchain_experimental.pal_chain import PALChain\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0, max_tokens=512)\n",
     "cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)\n",

cookbook/code-analysis-deeplake.ipynb

@@ -177,7 +177,7 @@
     "            try:\n",
     "                loader = TextLoader(os.path.join(dirpath, file), encoding=\"utf-8\")\n",
     "                docs.extend(loader.load_and_split())\n",
-    "            except Exception:\n",
+    "            except Exception as e:\n",
     "                pass\n",
     "print(f\"{len(docs)}\")"
    ]
@@ -648,7 +648,7 @@
     {
      "data": {
       "text/plain": [
-       "OpenAIEmbeddings(client=<class 'openai.api_resources.embedding.Embedding'>, model='text-embedding-ada-002', deployment='text-embedding-ada-002', openai_api_version='', openai_api_base='', openai_api_type='', openai_proxy='', embedding_ctx_length=8191, openai_api_key='', openai_organization='', allowed_special=set(), disallowed_special='all', chunk_size=1000, max_retries=6, request_timeout=None, headers=None, tiktoken_model_name=None, show_progress_bar=False, model_kwargs={})"
+       "OpenAIEmbeddings(client=<class 'openai.api_resources.embedding.Embedding'>, model='text-embedding-ada-002', deployment='text-embedding-ada-002', openai_api_version='', openai_api_base='', openai_api_type='', openai_proxy='', embedding_ctx_length=8191, openai_api_key='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', openai_organization='', allowed_special=set(), disallowed_special='all', chunk_size=1000, max_retries=6, request_timeout=None, headers=None, tiktoken_model_name=None, show_progress_bar=False, model_kwargs={})"
       ]
      },
      "execution_count": 13,
@@ -717,6 +717,7 @@
    "source": [
     "from langchain.vectorstores import DeepLake\n",
     "\n",
+    "\n",
     "username = \"<USERNAME_OR_ORG>\"\n",
     "\n",
     "\n",
@@ -833,8 +834,8 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains import ConversationalRetrievalChain\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains import ConversationalRetrievalChain\n",
     "\n",
     "model = ChatOpenAI(\n",
     "    model_name=\"gpt-3.5-turbo-0613\"\n",

cookbook/custom_agent_with_plugin_retrieval.ipynb

@@ -32,20 +32,19 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re\n",
-    "from typing import Union\n",
-    "\n",
     "from langchain.agents import (\n",
+    "    Tool,\n",
     "    AgentExecutor,\n",
-    "    AgentOutputParser,\n",
     "    LLMSingleActionAgent,\n",
+    "    AgentOutputParser,\n",
     ")\n",
-    "from langchain.agents.agent_toolkits import NLAToolkit\n",
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import StringPromptTemplate\n",
+    "from langchain.llms import OpenAI\nfrom langchain.utilities import SerpAPIWrapper\nfrom langchain.chains import LLMChain\n",
+    "from typing import List, Union\n",
     "from langchain.schema import AgentAction, AgentFinish\n",
-    "from langchain.tools.plugin import AIPlugin"
+    "from langchain.agents.agent_toolkits import NLAToolkit\n",
+    "from langchain.tools.plugin import AIPlugin\n",
+    "import re"
    ]
   },
   {
@@ -114,9 +113,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.vectorstores import FAISS\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema import Document\n",
-    "from langchain.vectorstores import FAISS"
+    "from langchain.schema import Document"
    ]
   },
   {

cookbook/custom_agent_with_plugin_retrieval_using_plugnplai.ipynb

@@ -56,21 +56,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re\n",
-    "from typing import Union\n",
-    "\n",
-    "import plugnplai\n",
     "from langchain.agents import (\n",
+    "    Tool,\n",
     "    AgentExecutor,\n",
-    "    AgentOutputParser,\n",
     "    LLMSingleActionAgent,\n",
+    "    AgentOutputParser,\n",
     ")\n",
-    "from langchain.agents.agent_toolkits import NLAToolkit\n",
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import OpenAI\n",
     "from langchain.prompts import StringPromptTemplate\n",
+    "from langchain.llms import OpenAI\nfrom langchain.utilities import SerpAPIWrapper\nfrom langchain.chains import LLMChain\n",
+    "from typing import List, Union\n",
     "from langchain.schema import AgentAction, AgentFinish\n",
-    "from langchain.tools.plugin import AIPlugin"
+    "from langchain.agents.agent_toolkits import NLAToolkit\n",
+    "from langchain.tools.plugin import AIPlugin\n",
+    "import re\n",
+    "import plugnplai"
    ]
   },
   {
@@ -138,9 +137,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.vectorstores import FAISS\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.schema import Document\n",
-    "from langchain.vectorstores import FAISS"
+    "from langchain.schema import Document"
    ]
   },
   {

cookbook/deeplake_semantic_search_over_chat.ipynb

@@ -48,17 +48,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import getpass\n",
     "import os\n",
-    "\n",
-    "from langchain.chains import RetrievalQA\n",
+    "import getpass\n",
+    "from langchain.document_loaders import PyPDFLoader, TextLoader\n",
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain.llms import OpenAI\n",
     "from langchain.text_splitter import (\n",
-    "    CharacterTextSplitter,\n",
     "    RecursiveCharacterTextSplitter,\n",
+    "    CharacterTextSplitter,\n",
     ")\n",
     "from langchain.vectorstores import DeepLake\n",
+    "from langchain.chains import ConversationalRetrievalChain, RetrievalQA\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
     "activeloop_token = getpass.getpass(\"Activeloop Token:\")\n",

cookbook/elasticsearch_db_qa.ipynb

@@ -38,8 +38,9 @@
    "outputs": [],
    "source": [
     "from elasticsearch import Elasticsearch\n",
-    "from langchain.chains.elasticsearch_database import ElasticsearchDatabaseChain\n",
-    "from langchain.chat_models import ChatOpenAI"
+    "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains.elasticsearch_database import ElasticsearchDatabaseChain"
    ]
   },
   {
@@ -111,6 +112,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from langchain.chains.elasticsearch_database.prompts import DEFAULT_DSL_TEMPLATE\n",
     "from langchain.prompts.prompt import PromptTemplate\n",
     "\n",
     "PROMPT_TEMPLATE = \"\"\"Given an input question, create a syntactically correct Elasticsearch query to run. Unless the user specifies in their question a specific number of examples they wish to obtain, always limit your query to at most {top_k} results. You can order the results by a relevant column to return the most interesting examples in the database.\n",

cookbook/extraction_openai_tools.ipynb

@@ -19,11 +19,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import List, Optional\n",
-    "\n",
-    "from langchain.chains.openai_tools import create_extraction_chain_pydantic\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.pydantic_v1 import BaseModel"
+    "from langchain.pydantic_v1 import BaseModel\n",
+    "from typing import Optional, List\n",
+    "from langchain.chains.openai_tools import create_extraction_chain_pydantic"
    ]
   },
   {

cookbook/fake_llm.ipynb

@@ -30,7 +30,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentType, initialize_agent, load_tools"
+    "from langchain.agents import load_tools\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.agents import AgentType"
    ]
   },
   {

cookbook/forward_looking_retrieval_augmented_generation.ipynb

@@ -56,8 +56,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"SERPER_API_KEY\"] = \"\"\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"\""
+    "os.environ[\"SERPER_API_KEY\"] = \"\"os.environ[\"OPENAI_API_KEY\"] = \"\""
    ]
   },
   {
@@ -67,16 +66,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Any, List\n",
+    "import re\n",
     "\n",
+    "import numpy as np\n",
+    "\n",
+    "from langchain.schema import BaseRetriever\n",
     "from langchain.callbacks.manager import (\n",
     "    AsyncCallbackManagerForRetrieverRun,\n",
     "    CallbackManagerForRetrieverRun,\n",
     ")\n",
+    "from langchain.utilities import GoogleSerperAPIWrapper\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.llms import OpenAI\n",
-    "from langchain.schema import BaseRetriever, Document\n",
-    "from langchain.utilities import GoogleSerperAPIWrapper"
+    "from langchain.schema import Document\n",
+    "from typing import Any, List"
    ]
   },
   {

cookbook/generative_agents_interactive_simulacra_of_human_behavior.ipynb

@@ -46,13 +46,14 @@
    "source": [
     "from datetime import datetime, timedelta\n",
     "from typing import List\n",
+    "from termcolor import colored\n",
+    "\n",
     "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.docstore import InMemoryDocstore\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.retrievers import TimeWeightedVectorStoreRetriever\n",
-    "from langchain.vectorstores import FAISS\n",
-    "from termcolor import colored"
+    "from langchain.vectorstores import FAISS"
    ]
   },
   {
@@ -152,7 +153,6 @@
    "outputs": [],
    "source": [
     "import math\n",
-    "\n",
     "import faiss\n",
     "\n",
     "\n",

cookbook/gymnasium_agent_simulation.ipynb

@@ -27,12 +27,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import gymnasium as gym\n",
+    "import inspect\n",
     "import tenacity\n",
-    "from langchain.output_parsers import RegexParser\n",
+    "\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
+    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")"
+    "    BaseMessage,\n",
+    ")\n",
+    "from langchain.output_parsers import RegexParser"
    ]
   },
   {
@@ -125,7 +131,7 @@
     "            ):\n",
     "                with attempt:\n",
     "                    action = self._act()\n",
-    "        except tenacity.RetryError:\n",
+    "        except tenacity.RetryError as e:\n",
     "            action = self.random_action()\n",
     "        return action"
    ]

cookbook/human_input_chat_model.ipynb

@@ -55,7 +55,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentType, initialize_agent, load_tools"
+    "from langchain.agents import load_tools\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.agents import AgentType"
    ]
   },
   {

cookbook/human_input_llm.ipynb

@@ -28,7 +28,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentType, initialize_agent, load_tools"
+    "from langchain.agents import load_tools\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.agents import AgentType"
    ]
   },
   {

cookbook/hypothetical_document_embeddings.ipynb

@@ -20,9 +20,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import HypotheticalDocumentEmbedder, LLMChain\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.llms import OpenAI\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.chains import LLMChain, HypotheticalDocumentEmbedder\n",
     "from langchain.prompts import PromptTemplate"
    ]
   },

cookbook/learned_prompt_optimization.ipynb

@@ -790,8 +790,8 @@
     }
    ],
    "source": [
-    "from langchain.globals import set_debug\n",
     "from langchain.prompts.prompt import PromptTemplate\n",
+    "from langchain.globals import set_debug\n",
     "\n",
     "set_debug(True)\n",
     "\n",

cookbook/llm_bash.ipynb

@@ -43,8 +43,8 @@
     }
    ],
    "source": [
-    "from langchain.llms import OpenAI\n",
     "from langchain_experimental.llm_bash.base import LLMBashChain\n",
+    "from langchain.llms import OpenAI\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "\n",
@@ -70,7 +70,7 @@
    "outputs": [],
    "source": [
     "from langchain.prompts.prompt import PromptTemplate\n",
-    "from langchain_experimental.llm_bash.prompt import BashOutputParser\n",
+    "from langchain.chains.llm_bash.prompt import BashOutputParser\n",
     "\n",
     "_PROMPT_TEMPLATE = \"\"\"If someone asks you to perform a task, your job is to come up with a series of bash commands that will perform the task. There is no need to put \"#!/bin/bash\" in your answer. Make sure to reason step by step, using this format:\n",
     "Question: \"copy the files in the directory named 'target' into a new directory at the same level as target called 'myNewDirectory'\"\n",
@@ -185,6 +185,7 @@
    "source": [
     "from langchain_experimental.llm_bash.bash import BashProcess\n",
     "\n",
+    "\n",
     "persistent_process = BashProcess(persistent=True)\n",
     "bash_chain = LLMBashChain.from_llm(llm, bash_process=persistent_process, verbose=True)\n",
     "\n",

cookbook/llm_math.ipynb

@@ -45,8 +45,7 @@
     }
    ],
    "source": [
-    "from langchain.chains import LLMMathChain\n",
-    "from langchain.llms import OpenAI\n",
+    "from langchain.llms import OpenAI\nfrom langchain.chains import LLMMathChain\n",
     "\n",
     "llm = OpenAI(temperature=0)\n",
     "llm_math = LLMMathChain.from_llm(llm, verbose=True)\n",

cookbook/meta_prompt.ipynb

@@ -56,10 +56,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import LLMChain\n",
-    "from langchain.llms import OpenAI\n",
-    "from langchain.memory import ConversationBufferWindowMemory\n",
-    "from langchain.prompts import PromptTemplate"
+    "from langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\n",
+    "from langchain.memory import ConversationBufferWindowMemory"
    ]
   },
   {
@@ -154,13 +152,13 @@
     "        for j in range(max_iters):\n",
     "            print(f\"(Step {j+1}/{max_iters})\")\n",
     "            print(f\"Assistant: {output}\")\n",
-    "            print(\"Human: \")\n",
+    "            print(f\"Human: \")\n",
     "            human_input = input()\n",
     "            if any(phrase in human_input.lower() for phrase in key_phrases):\n",
     "                break\n",
     "            output = chain.predict(human_input=human_input)\n",
     "        if success_phrase in human_input.lower():\n",
-    "            print(\"You succeeded! Thanks for playing!\")\n",
+    "            print(f\"You succeeded! Thanks for playing!\")\n",
     "            return\n",
     "        meta_chain = initialize_meta_chain()\n",
     "        meta_output = meta_chain.predict(chat_history=get_chat_history(chain.memory))\n",
@@ -168,7 +166,7 @@
     "        instructions = get_new_instructions(meta_output)\n",
     "        print(f\"New Instructions: {instructions}\")\n",
     "        print(\"\\n\" + \"#\" * 80 + \"\\n\")\n",
-    "    print(\"You failed! Thanks for playing!\")"
+    "    print(f\"You failed! Thanks for playing!\")"
    ]
   },
   {

cookbook/multi_modal_QA.ipynb

@@ -40,13 +40,12 @@
     }
    ],
    "source": [
-    "import base64\n",
-    "import io\n",
     "import os\n",
-    "\n",
+    "import io\n",
+    "import base64\n",
     "import numpy as np\n",
-    "from IPython.display import HTML, display\n",
     "from PIL import Image\n",
+    "from IPython.display import display, HTML\n",
     "\n",
     "\n",
     "def encode_image(image_path):\n",

cookbook/multi_modal_RAG_chroma.ipynb

@@ -115,7 +115,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Folder with pdf and extracted images\n",
+    "# Folder with pdf and extracted images \n",
     "path = \"/Users/rlm/Desktop/photos/\""
    ]
   },
@@ -128,10 +128,9 @@
    "source": [
     "# Extract images, tables, and chunk text\n",
     "from unstructured.partition.pdf import partition_pdf\n",
-    "\n",
     "raw_pdf_elements = partition_pdf(\n",
     "    filename=path + \"photos.pdf\",\n",
-    "    extract_images_in_pdf=True,\n",
+    "    extract_images_in_pdf=True, \n",
     "    infer_table_structure=True,\n",
     "    chunking_strategy=\"by_title\",\n",
     "    max_characters=4000,\n",
@@ -184,26 +183,22 @@
    "source": [
     "import os\n",
     "import uuid\n",
-    "\n",
     "import chromadb\n",
     "import numpy as np\n",
+    "from PIL import Image as _PILImage\n",
     "from langchain.vectorstores import Chroma\n",
     "from langchain_experimental.open_clip import OpenCLIPEmbeddings\n",
-    "from PIL import Image as _PILImage\n",
     "\n",
     "# Create chroma\n",
     "vectorstore = Chroma(\n",
-    "    collection_name=\"mm_rag_clip_photos\", embedding_function=OpenCLIPEmbeddings()\n",
+    "    collection_name=\"mm_rag_clip_photos\",\n",
+    "    embedding_function=OpenCLIPEmbeddings()\n",
     ")\n",
     "\n",
     "# Get image URIs with .jpg extension only\n",
-    "image_uris = sorted(\n",
-    "    [\n",
-    "        os.path.join(path, image_name)\n",
-    "        for image_name in os.listdir(path)\n",
-    "        if image_name.endswith(\".jpg\")\n",
-    "    ]\n",
-    ")\n",
+    "image_uris = sorted([os.path.join(path, image_name) \n",
+    "                     for image_name in os.listdir(path) \n",
+    "                     if image_name.endswith('.jpg')])\n",
     "\n",
     "# Add images\n",
     "vectorstore.add_images(uris=image_uris)\n",
@@ -211,7 +206,7 @@
     "# Add documents\n",
     "vectorstore.add_texts(texts=texts)\n",
     "\n",
-    "# Make retriever\n",
+    "# Make retriever \n",
     "retriever = vectorstore.as_retriever()"
    ]
   },
@@ -234,14 +229,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import base64\n",
     "import io\n",
-    "from io import BytesIO\n",
-    "\n",
     "import numpy as np\n",
+    "import base64\n",
+    "from io import BytesIO\n",
     "from PIL import Image\n",
     "\n",
-    "\n",
     "def resize_base64_image(base64_string, size=(128, 128)):\n",
     "    \"\"\"\n",
     "    Resize an image encoded as a Base64 string.\n",
@@ -265,31 +258,30 @@
     "    resized_img.save(buffered, format=img.format)\n",
     "\n",
     "    # Encode the resized image to Base64\n",
-    "    return base64.b64encode(buffered.getvalue()).decode(\"utf-8\")\n",
-    "\n",
+    "    return base64.b64encode(buffered.getvalue()).decode('utf-8')\n",
     "\n",
     "def is_base64(s):\n",
-    "    \"\"\"Check if a string is Base64 encoded\"\"\"\n",
+    "    ''' Check if a string is Base64 encoded '''\n",
     "    try:\n",
     "        return base64.b64encode(base64.b64decode(s)) == s.encode()\n",
     "    except Exception:\n",
     "        return False\n",
-    "\n",
-    "\n",
+    "        \n",
     "def split_image_text_types(docs):\n",
-    "    \"\"\"Split numpy array images and texts\"\"\"\n",
+    "    ''' Split numpy array images and texts '''\n",
     "    images = []\n",
     "    text = []\n",
     "    for doc in docs:\n",
-    "        doc = doc.page_content  # Extract Document contents\n",
+    "        doc = doc.page_content # Extract Document contents \n",
     "        if is_base64(doc):\n",
     "            # Resize image to avoid OAI server error\n",
-    "            images.append(\n",
-    "                resize_base64_image(doc, size=(250, 250))\n",
-    "            )  # base64 encoded str\n",
+    "            images.append(resize_base64_image(doc, size=(250, 250)))  # base64 encoded str \n",
     "        else:\n",
-    "            text.append(doc)\n",
-    "    return {\"images\": images, \"texts\": text}"
+    "            text.append(doc) \n",
+    "    return {\n",
+    "        \"images\": images,\n",
+    "        \"texts\": text\n",
+    "    }"
    ]
   },
   {
@@ -314,12 +306,10 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
-    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.schema.messages import HumanMessage, SystemMessage\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
-    "\n",
+    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
+    "from langchain.schema.messages import HumanMessage, SystemMessage\n",
     "\n",
     "def prompt_func(data_dict):\n",
     "    # Joining the context texts into a single string\n",
@@ -332,7 +322,7 @@
     "            \"type\": \"image_url\",\n",
     "            \"image_url\": {\n",
     "                \"url\": f\"data:image/jpeg;base64,{data_dict['context']['images'][0]}\"\n",
-    "            },\n",
+    "            }\n",
     "        }\n",
     "        messages.append(image_message)\n",
     "\n",
@@ -352,21 +342,17 @@
     "            f\"User-provided keywords: {data_dict['question']}\\n\\n\"\n",
     "            \"Text and / or tables:\\n\"\n",
     "            f\"{formatted_texts}\"\n",
-    "        ),\n",
+    "        )\n",
     "    }\n",
     "    messages.append(text_message)\n",
     "\n",
     "    return [HumanMessage(content=messages)]\n",
-    "\n",
-    "\n",
+    "    \n",
     "model = ChatOpenAI(temperature=0, model=\"gpt-4-vision-preview\", max_tokens=1024)\n",
     "\n",
     "# RAG pipeline\n",
     "chain = (\n",
-    "    {\n",
-    "        \"context\": retriever | RunnableLambda(split_image_text_types),\n",
-    "        \"question\": RunnablePassthrough(),\n",
-    "    }\n",
+    "    {\"context\": retriever | RunnableLambda(split_image_text_types), \"question\": RunnablePassthrough()}\n",
     "    | RunnableLambda(prompt_func)\n",
     "    | model\n",
     "    | StrOutputParser()\n",
@@ -424,18 +410,17 @@
     }
    ],
    "source": [
-    "from IPython.display import HTML, display\n",
-    "\n",
+    "from IPython.display import display, HTML\n",
     "\n",
     "def plt_img_base64(img_base64):\n",
+    "\n",
     "    # Create an HTML img tag with the base64 string as the source\n",
     "    image_html = f'<img src=\"data:image/jpeg;base64,{img_base64}\" />'\n",
-    "\n",
+    "    \n",
     "    # Display the image by rendering the HTML\n",
     "    display(HTML(image_html))\n",
     "\n",
-    "\n",
-    "docs = retriever.get_relevant_documents(\"Woman with children\", k=10)\n",
+    "docs = retriever.get_relevant_documents(\"Woman with children\",k=10)\n",
     "for doc in docs:\n",
     "    if is_base64(doc.page_content):\n",
     "        plt_img_base64(doc.page_content)\n",
@@ -461,7 +446,9 @@
     }
    ],
    "source": [
-    "chain.invoke(\"Woman with children\")"
+    "chain.invoke(\n",
+    "    \"Woman with children\"\n",
+    ")"
    ]
   },
   {

cookbook/multi_modal_output_agent.ipynb

@@ -29,10 +29,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from steamship import Block, Steamship\n",
     "import re\n",
-    "\n",
-    "from IPython.display import Image\n",
-    "from steamship import Block, Steamship"
+    "from IPython.display import Image"
    ]
   },
   {
@@ -42,8 +41,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentType, initialize_agent\n",
     "from langchain.llms import OpenAI\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.agents import AgentType\n",
     "from langchain.tools import SteamshipImageGenerationTool"
    ]
   },

cookbook/multi_player_dnd.ipynb

@@ -26,12 +26,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Callable, List\n",
-    "\n",
+    "from typing import List, Dict, Callable\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
+    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
+    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/multiagent_authoritarian.ipynb

@@ -27,20 +27,26 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from collections import OrderedDict\n",
     "import functools\n",
     "import random\n",
-    "from collections import OrderedDict\n",
-    "from typing import Callable, List\n",
-    "\n",
+    "import re\n",
     "import tenacity\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.output_parsers import RegexParser\n",
+    "from typing import List, Dict, Callable\n",
+    "\n",
     "from langchain.prompts import (\n",
+    "    ChatPromptTemplate,\n",
+    "    HumanMessagePromptTemplate,\n",
     "    PromptTemplate,\n",
     ")\n",
+    "from langchain.chains import LLMChain\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.output_parsers import RegexParser\n",
     "from langchain.schema import (\n",
+    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
+    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/multiagent_bidding.ipynb

@@ -24,15 +24,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Callable, List\n",
-    "\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "import re\n",
     "import tenacity\n",
+    "from typing import List, Dict, Callable\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.output_parsers import RegexParser\n",
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.schema import (\n",
+    "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
+    "    BaseMessage,\n",
     ")"
    ]
   },

cookbook/myscale_vector_sql.ipynb

@@ -27,15 +27,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import getpass\n",
     "from os import environ\n",
-    "\n",
-    "from langchain.chains import LLMChain\n",
+    "import getpass\n",
+    "from typing import Dict, Any\n",
     "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
     "from langchain.utilities import SQLDatabase\n",
+    "from langchain.chains import LLMChain\n",
     "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
-    "from sqlalchemy import MetaData, create_engine\n",
+    "from sqlalchemy import create_engine, Column, MetaData\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "\n",
+    "\n",
+    "from sqlalchemy import create_engine\n",
     "\n",
     "MYSCALE_HOST = \"msc-4a9e710a.us-east-1.aws.staging.myscale.cloud\"\n",
     "MYSCALE_PORT = 443\n",
@@ -74,8 +77,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.callbacks import StdOutCallbackHandler\n",
     "from langchain.llms import OpenAI\n",
+    "from langchain.callbacks import StdOutCallbackHandler\n",
+    "\n",
     "from langchain.utilities.sql_database import SQLDatabase\n",
     "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
     "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
@@ -116,16 +120,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.qa_with_sources.retrieval import RetrievalQAWithSourcesChain\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains.qa_with_sources.retrieval import RetrievalQAWithSourcesChain\n",
+    "\n",
+    "from langchain_experimental.sql.vector_sql import VectorSQLDatabaseChain\n",
     "from langchain_experimental.retrievers.vector_sql_database import (\n",
     "    VectorSQLDatabaseChainRetriever,\n",
     ")\n",
     "from langchain_experimental.sql.prompt import MYSCALE_PROMPT\n",
-    "from langchain_experimental.sql.vector_sql import (\n",
-    "    VectorSQLDatabaseChain,\n",
-    "    VectorSQLRetrieveAllOutputParser,\n",
-    ")\n",
+    "from langchain_experimental.sql.vector_sql import VectorSQLRetrieveAllOutputParser\n",
     "\n",
     "output_parser_retrieve_all = VectorSQLRetrieveAllOutputParser.from_embeddings(\n",
     "    output_parser.model\n",

cookbook/openai_functions_retrieval_qa.ipynb

@@ -50,10 +50,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import create_qa_with_sources_chain\n",
-    "from langchain.chains.combine_documents.stuff import StuffDocumentsChain\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.prompts import PromptTemplate"
+    "from langchain.chains.combine_documents.stuff import StuffDocumentsChain\n",
+    "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chains import create_qa_with_sources_chain"
    ]
   },
   {
@@ -230,8 +230,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import ConversationalRetrievalChain, LLMChain\n",
+    "from langchain.chains import ConversationalRetrievalChain\n",
     "from langchain.memory import ConversationBufferMemory\n",
+    "from langchain.chains import LLMChain\n",
     "\n",
     "memory = ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True)\n",
     "_template = \"\"\"Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.\\\n",
@@ -356,10 +357,12 @@
    "source": [
     "from typing import List\n",
     "\n",
+    "from pydantic import BaseModel, Field\n",
+    "\n",
     "from langchain.chains.openai_functions import create_qa_with_structure_chain\n",
+    "\n",
     "from langchain.prompts.chat import ChatPromptTemplate, HumanMessagePromptTemplate\n",
-    "from langchain.schema import HumanMessage, SystemMessage\n",
-    "from pydantic import BaseModel, Field"
+    "from langchain.schema import SystemMessage, HumanMessage"
    ]
   },
   {

cookbook/openai_v1_cookbook.ipynb

@@ -17,7 +17,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# need openai>=1.1.0, langchain>=0.0.335, langchain-experimental>=0.0.39\n",
+    "# need openai>=1.1.0, langchain>=0.0.333, langchain-experimental>=0.0.39\n",
     "!pip install -U openai langchain langchain-experimental"
    ]
   },
@@ -109,7 +109,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents.openai_assistant import OpenAIAssistantRunnable"
+    "from langchain_experimental.openai_assistant import OpenAIAssistantRunnable"
    ]
   },
   {
@@ -167,7 +167,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.tools import DuckDuckGoSearchRun, E2BDataAnalysisTool\n",
+    "from langchain.tools import E2BDataAnalysisTool, DuckDuckGoSearchRun\n",
     "\n",
     "tools = [E2BDataAnalysisTool(api_key=\"...\"), DuckDuckGoSearchRun()]"
    ]
@@ -419,7 +419,7 @@
     "\n",
     "\n",
     "RECOMMENDED CHANGES:\n",
-    "- When using `AzureChatOpenAI` or `AzureOpenAI`, if passing in an Azure endpoint (eg https://example-resource.azure.openai.com/) this should be specified via the `azure_endpoint` parameter or the `AZURE_OPENAI_ENDPOINT`. We're maintaining backwards compatibility for now with specifying this via `openai_api_base`/`base_url` or env var `OPENAI_API_BASE` but this shouldn't be relied upon.\n",
+    "- When using AzureChatOpenAI, if passing in an Azure endpoint (eg https://example-resource.azure.openai.com/) this should be specified via the `azure_endpoint` parameter or the `AZURE_OPENAI_ENDPOINT`. We're maintaining backwards compatibility for now with specifying this via `openai_api_base`/`base_url` or env var `OPENAI_API_BASE` but this shouldn't be relied upon.\n",
     "- When using Azure chat or embedding models, pass in API keys either via `openai_api_key` parameter or `AZURE_OPENAI_API_KEY` parameter. We're maintaining backwards compatibility for now with specifying this via `OPENAI_API_KEY` but this shouldn't be relied upon."
    ]
   },
@@ -456,9 +456,9 @@
     "from typing import Literal\n",
     "\n",
     "from langchain.output_parsers.openai_tools import PydanticToolsParser\n",
+    "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
     "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.pydantic_v1 import BaseModel, Field\n",
-    "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
     "\n",
     "\n",
     "class GetCurrentWeather(BaseModel):\n",

cookbook/petting_zoo.ipynb

@@ -45,14 +45,14 @@
    "source": [
     "import collections\n",
     "import inspect\n",
-    "\n",
     "import tenacity\n",
+    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.output_parsers import RegexParser\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
-    ")"
+    ")\n",
+    "from langchain.output_parsers import RegexParser"
    ]
   },
   {
@@ -146,7 +146,7 @@
     "            ):\n",
     "                with attempt:\n",
     "                    action = self._act()\n",
-    "        except tenacity.RetryError:\n",
+    "        except tenacity.RetryError as e:\n",
     "            action = self.random_action()\n",
     "        return action"
    ]

cookbook/program_aided_language_model.ipynb

@@ -17,8 +17,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.llms import OpenAI\n",
-    "from langchain_experimental.pal_chain import PALChain"
+    "from langchain_experimental.pal_chain import PALChain\n",
+    "from langchain.llms import OpenAI"
    ]
   },
   {

cookbook/qianfan_baidu_elasticesearch_RAG.ipynb

@@ -54,13 +54,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from baidubce.auth.bce_credentials import BceCredentials\n",
     "from baidubce.bce_client_configuration import BceClientConfiguration\n",
+    "from baidubce.auth.bce_credentials import BceCredentials\n",
     "from langchain.document_loaders.baiducloud_bos_directory import BaiduBOSDirectoryLoader\n",
-    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
-    "from langchain.llms.baidu_qianfan_endpoint import QianfanLLMEndpoint\n",
     "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "from langchain.vectorstores import BESVectorStore"
+    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
+    "from langchain.vectorstores import BESVectorStore\n",
+    "from langchain.llms.baidu_qianfan_endpoint import QianfanLLMEndpoint"
    ]
   },
   {
@@ -82,9 +82,7 @@
     "secret_access_key = \"your bos access sk\"\n",
     "\n",
     "# create BceClientConfiguration\n",
-    "config = BceClientConfiguration(\n",
-    "    credentials=BceCredentials(access_key_id, secret_access_key), endpoint=bos_host\n",
-    ")\n",
+    "config = BceClientConfiguration(credentials=BceCredentials(access_key_id, secret_access_key), endpoint = bos_host)\n",
     "\n",
     "loader = BaiduBOSDirectoryLoader(conf=config, bucket=\"llm-test\", prefix=\"llm/\")\n",
     "documents = loader.load()\n",
@@ -111,14 +109,10 @@
     "embeddings.client = sentence_transformers.SentenceTransformer(embeddings.model_name)\n",
     "\n",
     "db = BESVectorStore.from_documents(\n",
-    "    documents=split_docs,\n",
-    "    embedding=embeddings,\n",
-    "    bes_url=\"your bes url\",\n",
-    "    index_name=\"test-index\",\n",
-    "    vector_query_field=\"vector\",\n",
-    ")\n",
+    "  documents=split_docs, embedding=embeddings, bes_url=\"your bes url\", index_name='test-index', vector_query_field='vector'\n",
+    " )\n",
     "\n",
-    "db.client.indices.refresh(index=\"test-index\")\n",
+    "db.client.indices.refresh(index='test-index')\n",
     "retriever = db.as_retriever()"
    ]
   },
@@ -136,15 +130,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "llm = QianfanLLMEndpoint(\n",
-    "    model=\"ERNIE-Bot\",\n",
-    "    qianfan_ak=\"your qianfan ak\",\n",
-    "    qianfan_sk=\"your qianfan sk\",\n",
-    "    streaming=True,\n",
-    ")\n",
-    "qa = RetrievalQA.from_chain_type(\n",
-    "    llm=llm, chain_type=\"refine\", retriever=retriever, return_source_documents=True\n",
-    ")\n",
+    "llm = QianfanLLMEndpoint(model=\"ERNIE-Bot\", qianfan_ak='your qianfan ak', qianfan_sk='your qianfan sk', streaming=True)\n",
+    "qa = RetrievalQA.from_chain_type(llm=llm, chain_type=\"refine\", retriever=retriever, return_source_documents=True)\n",
     "\n",
     "query = \"什么是张量?\"\n",
     "print(qa.run(query))"

cookbook/rag_fusion.ipynb

@@ -30,8 +30,8 @@
    "outputs": [],
    "source": [
     "import pinecone\n",
-    "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.vectorstores import Pinecone\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
     "\n",
     "pinecone.init(api_key=\"...\", environment=\"...\")"
    ]
@@ -87,6 +87,7 @@
    "outputs": [],
    "source": [
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
     "from langchain.schema.output_parser import StrOutputParser"
    ]
   },

cookbook/retrieval_in_sql.ipynb

@@ -28,8 +28,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import getpass\n",
     "import os\n",
+    "import getpass\n",
     "\n",
     "os.environ[\"OPENAI_API_KEY\"] = os.environ.get(\"OPENAI_API_KEY\") or getpass.getpass(\n",
     "    \"OpenAI API Key:\"\n",
@@ -42,8 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.sql_database import SQLDatabase\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "CONNECTION_STRING = \"postgresql+psycopg2://postgres:test@localhost:5432/vectordb\"  # Replace with your own\n",
     "db = SQLDatabase.from_uri(CONNECTION_STRING)"
@@ -323,7 +323,6 @@
    "outputs": [],
    "source": [
     "import re\n",
-    "\n",
     "from langchain.schema.runnable import RunnableLambda\n",
     "\n",
     "\n",

cookbook/rewrite.ipynb

@@ -31,10 +31,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
+    "from operator import itemgetter\n",
+    "\n",
     "from langchain.prompts import ChatPromptTemplate\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.runnable import RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
     "from langchain.utilities import DuckDuckGoSearchAPIWrapper"
    ]
   },

cookbook/sales_agent_with_context.ipynb

@@ -42,22 +42,22 @@
     "OPENAI_API_KEY = \"sk-xx\"\n",
     "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY\n",
     "\n",
-    "from typing import Any, Callable, Dict, List, Union\n",
-    "\n",
-    "from langchain.agents import AgentExecutor, LLMSingleActionAgent, Tool\n",
-    "from langchain.agents.agent import AgentOutputParser\n",
-    "from langchain.agents.conversational.prompt import FORMAT_INSTRUCTIONS\n",
-    "from langchain.chains import LLMChain, RetrievalQA\n",
+    "from typing import Dict, List, Any, Union, Callable\n",
+    "from pydantic import BaseModel, Field\n",
+    "from langchain.chains import LLMChain\nfrom langchain.prompts import PromptTemplate\n",
+    "from langchain.llms import BaseLLM\n",
     "from langchain.chains.base import Chain\n",
     "from langchain.chat_models import ChatOpenAI\n",
-    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain.llms import BaseLLM, OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.prompts.base import StringPromptTemplate\n",
-    "from langchain.schema import AgentAction, AgentFinish\n",
+    "from langchain.agents import Tool, LLMSingleActionAgent, AgentExecutor\n",
     "from langchain.text_splitter import CharacterTextSplitter\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.chains import RetrievalQA\n",
     "from langchain.vectorstores import Chroma\n",
-    "from pydantic import BaseModel, Field"
+    "from langchain.llms import OpenAI\n",
+    "from langchain.prompts.base import StringPromptTemplate\n",
+    "from langchain.agents.agent import AgentOutputParser\n",
+    "from langchain.agents.conversational.prompt import FORMAT_INSTRUCTIONS\n",
+    "from langchain.schema import AgentAction, AgentFinish"
    ]
   },
   {

cookbook/selecting_llms_based_on_context_length.ipynb

@@ -17,10 +17,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
+    "from langchain.schema.prompt import PromptValue\n",
+    "from langchain.schema.messages import BaseMessage\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
-    "from langchain.schema.prompt import PromptValue"
+    "from typing import Union, Sequence"
    ]
   },
   {

cookbook/self_query_hotel_search.ipynb

@@ -1084,6 +1084,7 @@
    "outputs": [],
    "source": [
     "from langchain.embeddings import OpenAIEmbeddings\n",
+    "from langchain.schema import Document\n",
     "from langchain.vectorstores import ElasticsearchStore\n",
     "\n",
     "embeddings = OpenAIEmbeddings()"

cookbook/smart_llm.ipynb

@@ -51,8 +51,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import PromptTemplate\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain_experimental.smart_llm import SmartLLMChain"
    ]
   },

cookbook/stepback-qa.ipynb

@@ -131,6 +131,7 @@
    "source": [
     "from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
     "\n",
+    "\n",
     "search = DuckDuckGoSearchAPIWrapper(max_results=4)\n",
     "\n",
     "\n",

cookbook/tree_of_thought.ipynb

@@ -84,11 +84,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re\n",
     "from typing import Tuple\n",
-    "\n",
     "from langchain_experimental.tot.checker import ToTChecker\n",
     "from langchain_experimental.tot.thought import ThoughtValidity\n",
+    "import re\n",
     "\n",
     "\n",
     "class MyChecker(ToTChecker):\n",

cookbook/twitter-the-algorithm-analysis-deeplake.ipynb

@@ -34,8 +34,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import getpass\n",
     "import os\n",
+    "import getpass\n",
     "\n",
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "from langchain.vectorstores import DeepLake\n",
@@ -109,7 +109,6 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "\n",
     "from langchain.document_loaders import TextLoader\n",
     "\n",
     "root_dir = \"./the-algorithm\"\n",
@@ -119,7 +118,7 @@
     "        try:\n",
     "            loader = TextLoader(os.path.join(dirpath, file), encoding=\"utf-8\")\n",
     "            docs.extend(loader.load_and_split())\n",
-    "        except Exception:\n",
+    "        except Exception as e:\n",
     "            pass"
    ]
   },
@@ -3808,8 +3807,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import ConversationalRetrievalChain\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains import ConversationalRetrievalChain\n",
     "\n",
     "model = ChatOpenAI(model_name=\"gpt-3.5-turbo-0613\")  # switch to 'gpt-4'\n",
     "qa = ConversationalRetrievalChain.from_llm(model, retriever=retriever)"

cookbook/two_agent_debate_tools.ipynb

@@ -22,14 +22,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Callable, List\n",
-    "\n",
+    "from typing import List, Dict, Callable\n",
+    "from langchain.chains import ConversationChain\n",
     "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.llms import OpenAI\n",
     "from langchain.memory import ConversationBufferMemory\n",
+    "from langchain.prompts.prompt import PromptTemplate\n",
     "from langchain.schema import (\n",
     "    AIMessage,\n",
     "    HumanMessage,\n",
     "    SystemMessage,\n",
+    "    BaseMessage,\n",
     ")"
    ]
   },
@@ -46,7 +49,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentType, initialize_agent, load_tools"
+    "from langchain.agents import Tool\n",
+    "from langchain.agents import initialize_agent\n",
+    "from langchain.agents import AgentType\n",
+    "from langchain.agents import load_tools"
    ]
   },
   {

cookbook/two_player_dnd.ipynb

@@ -22,8 +22,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from typing import Callable, List\n",
-    "\n",
+    "from typing import List, Dict, Callable\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.schema import (\n",
     "    HumanMessage,\n",

cookbook/wikibase_agent.ipynb

@@ -192,10 +192,10 @@
     "    return current\n",
     "\n",
     "\n",
-    "from typing import Optional\n",
-    "\n",
     "import requests\n",
     "\n",
+    "from typing import Optional\n",
+    "\n",
     "\n",
     "def vocab_lookup(\n",
     "    search: str,\n",
@@ -319,10 +319,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import json\n",
-    "from typing import Any, Dict, List\n",
-    "\n",
     "import requests\n",
+    "from typing import List, Dict, Any\n",
+    "import json\n",
     "\n",
     "\n",
     "def run_sparql(\n",
@@ -390,18 +389,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import re\n",
-    "from typing import List, Union\n",
-    "\n",
     "from langchain.agents import (\n",
-    "    AgentExecutor,\n",
-    "    AgentOutputParser,\n",
-    "    LLMSingleActionAgent,\n",
     "    Tool,\n",
+    "    AgentExecutor,\n",
+    "    LLMSingleActionAgent,\n",
+    "    AgentOutputParser,\n",
     ")\n",
-    "from langchain.chains import LLMChain\n",
     "from langchain.prompts import StringPromptTemplate\n",
-    "from langchain.schema import AgentAction, AgentFinish"
+    "from langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\n",
+    "from typing import List, Union\n",
+    "from langchain.schema import AgentAction, AgentFinish\n",
+    "import re"
    ]
   },
   {

docs/.local_build.sh

@@ -9,15 +9,13 @@ SCRIPT_DIR="$(cd "$(dirname "$0")"; pwd)"
 cd "${SCRIPT_DIR}"
 
 mkdir -p ../_dist
-rsync -ruv --exclude node_modules . ../_dist
+cp -r . ../_dist
 cd ../_dist
 poetry run python scripts/model_feat_table.py
+poetry run nbdoc_build --srcdir docs
 cp ../cookbook/README.md src/pages/cookbook.mdx
 cp ../.github/CONTRIBUTING.md docs/contributing.md
-mkdir -p docs/templates
-cp ../templates/docs/INDEX.md docs/templates/index.md
 wget https://raw.githubusercontent.com/langchain-ai/langserve/main/README.md -O docs/langserve.md
-
-yarn
-
-quarto preview docs
+poetry run python scripts/generate_api_reference_links.py
+yarn install
+yarn start

docs/api_reference/_static/css/custom.css

@@ -15,11 +15,3 @@ pre {
 #my-component-root *, #headlessui-portal-root * {
   z-index: 10000;
 }
-
-table.longtable code {
-  white-space: normal;
-}
-
-table.longtable td {
-  max-width: 600px;
-}

docs/api_reference/create_api_rst.py

@@ -13,10 +13,8 @@ HERE = Path(__file__).parent
 
 PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
 EXP_DIR = ROOT_DIR / "libs" / "experimental" / "langchain_experimental"
-CORE_DIR = ROOT_DIR / "libs" / "core" / "langchain_core"
 WRITE_FILE = HERE / "api_reference.rst"
 EXP_WRITE_FILE = HERE / "experimental_api_reference.rst"
-CORE_WRITE_FILE = HERE / "core_api_reference.rst"
 
 
 ClassKind = Literal["TypedDict", "Regular", "Pydantic", "enum"]
@@ -294,17 +292,6 @@ def _document_langchain_experimental() -> None:
 
 
 def _document_langchain_core() -> None:
-    """Document the langchain_core package."""
-    # Generate core_api_reference.rst
-    core_members = _load_package_modules(CORE_DIR)
-    core_doc = ".. _core_api_reference:\n\n" + _construct_doc(
-        "langchain_core", core_members
-    )
-    with open(CORE_WRITE_FILE, "w") as f:
-        f.write(core_doc)
-
-
-def _document_langchain() -> None:
     """Document the main langchain package."""
     # load top level module members
     lc_members = _load_package_modules(PKG_DIR)
@@ -319,6 +306,7 @@ def _document_langchain() -> None:
             "agents.output_parsers": agents["output_parsers"],
             "agents.format_scratchpad": agents["format_scratchpad"],
             "tools.render": tools["render"],
+            "schema.runnable": schema["runnable"],
         }
     )
 
@@ -330,9 +318,8 @@ def _document_langchain() -> None:
 
 def main() -> None:
     """Generate the reference.rst file for each package."""
-    _document_langchain()
-    _document_langchain_experimental()
     _document_langchain_core()
+    _document_langchain_experimental()
 
 
 if __name__ == "__main__":

docs/api_reference/requirements.txt

@@ -1,6 +1,5 @@
 -e libs/langchain
 -e libs/experimental
--e libs/core
 pydantic<2
 autodoc_pydantic==1.8.0
 myst_parser

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -34,9 +34,6 @@
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
         </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('core_api_reference') }}">Core</a>
-        </li>
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('experimental_api_reference') }}">Experimental</a>
         </li>

docs/docs/_static/css/custom.css

@@ -0,0 +1,21 @@
+pre {
+  white-space: break-spaces;
+}
+
+@media (min-width: 1200px) {
+  .container,
+  .container-lg,
+  .container-md,
+  .container-sm,
+  .container-xl {
+    max-width: 2560px !important;
+  }
+}
+
+#my-component-root *, #headlessui-portal-root * {
+  z-index: 10000;
+}
+
+.content-container p {
+    margin: revert;
+}

docs/docs/_static/js/mendablesearch.js

@@ -0,0 +1,56 @@
+document.addEventListener('DOMContentLoaded', () => {
+  // Load the external dependencies
+  function loadScript(src, onLoadCallback) {
+    const script = document.createElement('script');
+    script.src = src;
+    script.onload = onLoadCallback;
+    document.head.appendChild(script);
+  }
+
+  function createRootElement() {
+    const rootElement = document.createElement('div');
+    rootElement.id = 'my-component-root';
+    document.body.appendChild(rootElement);
+    return rootElement;
+  }
+
+  
+
+  function initializeMendable() {
+    const rootElement = createRootElement();
+    const { MendableFloatingButton } = Mendable;
+    
+
+    const iconSpan1 = React.createElement('span', {
+    }, '🦜');
+
+    const iconSpan2 = React.createElement('span', {
+    }, '🔗');
+
+    const icon = React.createElement('p', {
+      style: { color: '#ffffff', fontSize: '22px',width: '48px', height: '48px', margin: '0px', padding: '0px', display: 'flex', alignItems: 'center', justifyContent: 'center', textAlign: 'center' },
+    }, [iconSpan1, iconSpan2]);
+    
+    const mendableFloatingButton = React.createElement(
+      MendableFloatingButton,
+      {
+        style: { darkMode: false, accentColor: '#010810' },
+        floatingButtonStyle: { color: '#ffffff', backgroundColor: '#010810' },
+        anon_key: '82842b36-3ea6-49b2-9fb8-52cfc4bde6bf', // Mendable Search Public ANON key, ok to be public
+        messageSettings: {
+          openSourcesInNewTab: false,
+          prettySources: true // Prettify the sources displayed now
+        },
+        icon: icon,
+      }
+    );
+
+    ReactDOM.render(mendableFloatingButton, rootElement);
+  }
+
+  loadScript('https://unpkg.com/react@17/umd/react.production.min.js', () => {
+    loadScript('https://unpkg.com/react-dom@17/umd/react-dom.production.min.js', () => {
+      loadScript('https://unpkg.com/@mendable/search@0.0.102/dist/umd/mendable.min.js', initializeMendable);
+    });
+  });
+});

docs/docs/expression_language/cookbook/agent.ipynb

@@ -17,7 +17,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.agents import AgentExecutor, XMLAgent, tool\n",
+    "from langchain.agents import XMLAgent, tool, AgentExecutor\n",
     "from langchain.chat_models import ChatAnthropic"
    ]
   },

docs/docs/expression_language/cookbook/code_writing.ipynb

@@ -20,6 +20,8 @@
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import (\n",
     "    ChatPromptTemplate,\n",
+    "    SystemMessagePromptTemplate,\n",
+    "    HumanMessagePromptTemplate,\n",
     ")\n",
     "from langchain.schema.output_parser import StrOutputParser\n",
     "from langchain_experimental.utilities import PythonREPL"

docs/docs/expression_language/cookbook/embedding_router.ipynb

@@ -26,6 +26,7 @@
     "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
     "from langchain.utils.math import cosine_similarity\n",
     "\n",
+    "\n",
     "physics_template = \"\"\"You are a very smart physics professor. \\\n",
     "You are great at answering questions about physics in a concise and easy to understand manner. \\\n",
     "When you don't know the answer to a question you admit that you don't know.\n",

docs/docs/expression_language/cookbook/index.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 2
 ---
 
 # Cookbook

docs/docs/expression_language/cookbook/memory.ipynb

@@ -18,11 +18,10 @@
    "outputs": [],
    "source": [
     "from operator import itemgetter\n",
-    "\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.memory import ConversationBufferMemory\n",
+    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
     "from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
-    "from langchain.schema.runnable import RunnableLambda, RunnablePassthrough\n",
     "\n",
     "model = ChatOpenAI()\n",
     "prompt = ChatPromptTemplate.from_messages(\n",

docs/docs/expression_language/cookbook/multiple_chains.ipynb

@@ -69,7 +69,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.schema.runnable import RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnableMap, RunnablePassthrough\n",
     "\n",
     "prompt1 = ChatPromptTemplate.from_template(\n",
     "    \"generate a {attribute} color. Return the name of the color and nothing else:\"\n",
@@ -146,7 +146,7 @@
    "source": [
     "### Branching and Merging\n",
     "\n",
-    "You may want the output of one component to be processed by 2 or more other components. [RunnableParallels](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.RunnableParallel.html#langchain_core.runnables.base.RunnableParallel) let you split or fork the chain so multiple components can process the input in parallel. Later, other components can join or merge the results to synthesize a final response. This type of chain creates a computation graph that looks like the following:\n",
+    "You may want the output of one component to be processed by 2 or more other components. [RunnableMaps](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.RunnableMap.html) let you split or fork the chain so multiple components can process the input in parallel. Later, other components can join or merge the results to synthesize a final response. This type of chain creates a computation graph that looks like the following:\n",
     "\n",
     "```text\n",
     "     Input\n",

docs/docs/expression_language/cookbook/prompt_llm_parser.ipynb

@@ -42,8 +42,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.prompts import ChatPromptTemplate\n",
+    "from langchain.chat_models import ChatOpenAI\n",
     "\n",
     "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {foo}\")\n",
     "model = ChatOpenAI()\n",
@@ -317,7 +317,7 @@
    "source": [
     "## Simplifying input\n",
     "\n",
-    "To make invocation even simpler, we can add a `RunnableParallel` to take care of creating the prompt input dict for us:"
+    "To make invocation even simpler, we can add a `RunnableMap` to take care of creating the prompt input dict for us:"
    ]
   },
   {
@@ -327,9 +327,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.schema.runnable import RunnableParallel, RunnablePassthrough\n",
+    "from langchain.schema.runnable import RunnableMap, RunnablePassthrough\n",
     "\n",
-    "map_ = RunnableParallel(foo=RunnablePassthrough())\n",
+    "map_ = RunnableMap(foo=RunnablePassthrough())\n",
     "chain = (\n",
     "    map_\n",
     "    | prompt\n",