Merge branch 'master' into eugene/expand_documentation

.devcontainer/README.md

@@ -15,11 +15,7 @@ You may use the button above, or follow these steps to open this repo in a Codes
 For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
   
 ## VS Code Dev Containers
-[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/langchain-ai/langchain)
-
-Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name: 
-https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/<yourusername>/<yourclonedreponame>
-
+[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 
 If you already have VS Code and Docker installed, you can use the button above to get started. This will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
 
@@ -29,7 +25,7 @@ You can also follow these steps to open this repo in a container using the VS Co
 
 2. Open a locally cloned copy of the code:
 
-   - Fork and Clone this repository to your local filesystem.
+   - Clone this repository to your local filesystem.
    - Press <kbd>F1</kbd> and select the **Dev Containers: Open Folder in Container...** command.
    - Select the cloned copy of this folder, wait for the container to start, and try things out!
 

.github/workflows/_release.yml

@@ -37,7 +37,6 @@ jobs:
           echo version=$(poetry version --short) >> $GITHUB_OUTPUT
       - name: Create Release
         uses: ncipollo/release-action@v1
-        if: ${{ inputs.working-directory == 'libs/langchain' }}
         with:
           artifacts: "dist/*"
           token: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -162,7 +162,6 @@ docs/.docusaurus/
 docs/.cache-loader/
 docs/_dist
 docs/api_reference/api_reference.rst
-docs/api_reference/experimental_api_reference.rst
 docs/api_reference/_build
 docs/api_reference/*/
 !docs/api_reference/_static/

MIGRATE.md

@@ -43,10 +43,6 @@ Now:
 
 `from langchain_experimental.sql import SQLDatabaseChain`
 
-Alternatively, if you are just interested in using the query generation part of the SQL chain, you can check out [`create_sql_query_chain`](https://github.com/langchain-ai/langchain/blob/master/docs/extras/use_cases/tabular/sql_query.ipynb)
-
-`from langchain.chains import create_sql_query_chain`
-
 ## `load_prompt` for Python files
 
 Note: this only applies if you want to load Python files as prompts.

README.md

@@ -12,7 +12,7 @@
 [![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/hwchase17/langchain)
 [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hwchase17/langchain)
 [![GitHub star chart](https://img.shields.io/github/stars/hwchase17/langchain?style=social)](https://star-history.com/#hwchase17/langchain)
-[![Dependency Status](https://img.shields.io/librariesio/github/langchain-ai/langchain)](https://libraries.io/github/langchain-ai/langchain)
+[![Dependency Status](https://img.shields.io/librariesio/github/hwchase17/langchain)](https://libraries.io/github/hwchase17/langchain)
 [![Open Issues](https://img.shields.io/github/issues-raw/hwchase17/langchain)](https://github.com/hwchase17/langchain/issues)
 
 

docs/.local_build.sh

@@ -13,6 +13,5 @@ cp -r {docs_skeleton,snippets} _dist
 cp -r extras/* _dist/docs_skeleton/docs
 cd _dist/docs_skeleton
 poetry run nbdoc_build
-poetry run python generate_api_reference_links.py
 yarn install
 yarn start

docs/api_reference/conf.py

@@ -23,7 +23,6 @@ from sphinx.util.docutils import SphinxDirective
 _DIR = Path(__file__).parent.absolute()
 sys.path.insert(0, os.path.abspath("."))
 sys.path.insert(0, os.path.abspath("../../libs/langchain"))
-sys.path.insert(0, os.path.abspath("../../libs/experimental"))
 
 with (_DIR.parents[1] / "libs" / "langchain" / "pyproject.toml").open("r") as f:
     data = toml.load(f)

docs/api_reference/create_api_rst.py

@@ -5,15 +5,13 @@ from pathlib import Path
 
 ROOT_DIR = Path(__file__).parents[2].absolute()
 PKG_DIR = ROOT_DIR / "libs" / "langchain" / "langchain"
-EXP_DIR = ROOT_DIR / "libs" / "experimental" / "langchain_experimental"
 WRITE_FILE = Path(__file__).parent / "api_reference.rst"
-EXP_WRITE_FILE = Path(__file__).parent / "experimental_api_reference.rst"
 
 
-def load_members(dir: Path) -> dict:
+def load_members() -> dict:
     members: dict = {}
-    for py in glob.glob(str(dir) + "/**/*.py", recursive=True):
-        module = py[len(str(dir)) + 1 :].replace(".py", "").replace("/", ".")
+    for py in glob.glob(str(PKG_DIR) + "/**/*.py", recursive=True):
+        module = py[len(str(PKG_DIR)) + 1 :].replace(".py", "").replace("/", ".")
         top_level = module.split(".")[0]
         if top_level not in members:
             members[top_level] = {"classes": [], "functions": []}
@@ -28,10 +26,12 @@ def load_members(dir: Path) -> dict:
     return members
 
 
-def construct_doc(pkg: str, members: dict) -> str:
-    full_doc = f"""\
+def construct_doc(members: dict) -> str:
+    full_doc = """\
+.. _api_reference:
+
 =============
-``{pkg}`` API Reference
+API Reference
 =============
 
 """
@@ -40,12 +40,12 @@ def construct_doc(pkg: str, members: dict) -> str:
         functions = _members["functions"]
         if not (classes or functions):
             continue
-        section = f":mod:`{pkg}.{module}`"
+        section = f":mod:`langchain.{module}`"
         full_doc += f"""\
 {section}
 {'=' * (len(section) + 1)}
 
-.. automodule:: {pkg}.{module}
+.. automodule:: langchain.{module}
     :no-members:
     :no-inherited-members:
 
@@ -56,7 +56,7 @@ def construct_doc(pkg: str, members: dict) -> str:
             full_doc += f"""\
 Classes
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
@@ -70,7 +70,7 @@ Classes
             full_doc += f"""\
 Functions
 --------------
-.. currentmodule:: {pkg}
+.. currentmodule:: langchain
 
 .. autosummary::
     :toctree: {module}
@@ -83,14 +83,10 @@ Functions
 
 
 def main() -> None:
-    lc_members = load_members(PKG_DIR)
-    lc_doc = ".. _api_reference:\n\n" + construct_doc("langchain", lc_members)
+    members = load_members()
+    full_doc = construct_doc(members)
     with open(WRITE_FILE, "w") as f:
-        f.write(lc_doc)
-    exp_members = load_members(EXP_DIR)
-    exp_doc = ".. _experimental_api_reference:\n\n" + construct_doc("langchain_experimental", exp_members)
-    with open(EXP_WRITE_FILE, "w") as f:
-        f.write(exp_doc)
+        f.write(full_doc)
 
 
 if __name__ == "__main__":

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -45,9 +45,6 @@
         <li class="nav-item">
           <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
         </li>
-        <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('experimental_api_reference') }}">Experimental</a>
-        </li>
         <li class="nav-item">
           <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
         </li>

docs/docs_skeleton/docs/guides/expression_language/index.mdx

@@ -1,9 +0,0 @@
-# LangChain Expression Language
-
-import DocCardList from "@theme/DocCardList";
-
-LangChain Expression Language is a declarative way to easily compose chains together.
-Any chain constructed this way will automatically have full sync, async, and streaming support.
-See guides below for how to interact with chains constructed this way as well as cookbook examples.
-
-<DocCardList />

docs/docs_skeleton/docs/guides/safety/index.mdx

@@ -1,6 +0,0 @@
-# Preventing harmful outputs
-
-One of the key concerns with using LLMs is that they may generate harmful or unethical text. This is an area of active research in the field. Here we present some built-in chains inspired by this research, which are intended to make the outputs of LLMs safer.
-
-- [Moderation chain](/docs/use_cases/safety/moderation): Explicitly check if any output text is harmful and flag it.
-- [Constitutional chain](/docs/use_cases/safety/constitutional_chain): Prompt the model with a set of principles which should guide it's behavior.

docs/docs_skeleton/docs/modules/chains/additional/index.mdx

@@ -0,0 +1,8 @@
+---
+sidebar_position: 4
+---
+# Additional
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/chains/additional/multi_prompt_router.mdx

@@ -0,0 +1,7 @@
+# Dynamically selecting from multiple prompts
+
+This notebook demonstrates how to use the `RouterChain` paradigm to create a chain that dynamically selects the prompt to use for a given input. Specifically we show how to use the `MultiPromptChain` to create a question-answering chain that selects the prompt which is most relevant for a given question, and then answers the question using that prompt.
+
+import Example from "@snippets/modules/chains/additional/multi_prompt_router.mdx"
+
+<Example/>

docs/docs_skeleton/docs/modules/chains/additional/multi_retrieval_qa_router.mdx

@@ -1,4 +1,4 @@
-# Dynamically select from multiple retrievers
+# Dynamically selecting from multiple retrievers
 
 This notebook demonstrates how to use the `RouterChain` paradigm to create a chain that dynamically selects which Retrieval system to use. Specifically we show how to use the `MultiRetrievalQAChain` to create a question-answering chain that selects the retrieval QA chain which is most relevant for a given question, and then answers the question using it.
 

docs/docs_skeleton/docs/modules/chains/additional/question_answering.mdx

@@ -1,4 +1,4 @@
-# QA over in-memory documents
+# Document QA
 
 Here we walk through how to use LangChain for question answering over a list of documents. Under the hood we'll be using our [Document chains](/docs/modules/chains/document/).
 

docs/docs_skeleton/docs/modules/chains/foundational/sequential_chains.mdx

@@ -1,6 +1,6 @@
 # Sequential
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 The next step after calling a language model is make a series of calls to a language model. This is particularly useful when you want to take the output from one call and use it as the input to another.
 

docs/docs_skeleton/docs/modules/chains/popular/chat_vector_db.mdx

@@ -2,7 +2,7 @@
 sidebar_position: 2
 ---
 
-# Store and reference chat history
+# Conversational Retrieval QA
 The ConversationalRetrievalQA chain builds on RetrievalQAChain to provide a chat history component.
 
 It first combines the chat history (either explicitly passed in or retrieved from the provided memory) and the question into a standalone question, then looks up relevant documents from the retriever, and finally passes those documents and the question to a question answering chain to return a response.

docs/docs_skeleton/docs/modules/chains/popular/index.mdx

@@ -0,0 +1,8 @@
+---
+sidebar_position: 3
+---
+# Popular
+
+import DocCardList from "@theme/DocCardList";
+
+<DocCardList />

docs/docs_skeleton/docs/modules/chains/popular/vector_db_qa.mdx

@@ -1,7 +1,7 @@
 ---
 sidebar_position: 1
 ---
-# QA using a Retriever
+# Retrieval QA
 
 This example showcases question answering over an index.
 

docs/docs_skeleton/docs/modules/data_connection/retrievers/integrations/_category_.yml

@@ -0,0 +1 @@
+label: 'Integrations'

docs/docs_skeleton/docs/modules/memory/buffer.mdx

@@ -4,6 +4,6 @@ This notebook shows how to use `ConversationBufferMemory`. This memory allows fo
 
 We can first extract it as a string.
 
-import Example from "@snippets/modules/memory/types/buffer.mdx"
+import Example from "@snippets/modules/memory/how_to/buffer.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/buffer_window.mdx

@@ -4,6 +4,6 @@
 
 Let's first explore the basic functionality of this type of memory.
 
-import Example from "@snippets/modules/memory/types/buffer_window.mdx"
+import Example from "@snippets/modules/memory/how_to/buffer_window.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/chat_messages/index.mdx

@@ -1,17 +0,0 @@
----
-sidebar_position: 1
----
-# Chat Messages
-
-:::info
-Head to [Integrations](/docs/integrations/memory/) for documentation on built-in memory integrations with 3rd-party databases and tools.
-:::
-
-One of the core utility classes underpinning most (if not all) memory modules is the `ChatMessageHistory` class.
-This is a super lightweight wrapper which exposes convenience methods for saving Human messages, AI messages, and then fetching them all.
-
-You may want to use this class directly if you are managing memory outside of a chain.
-
-import GetStarted from "@snippets/modules/memory/chat_messages/get_started.mdx"
-
-<GetStarted/>

docs/docs_skeleton/docs/modules/memory/entity_summary_memory.mdx

@@ -4,6 +4,6 @@ Entity Memory remembers given facts about specific entities in a conversation. I
 
 Let's first walk through using this functionality.
 
-import Example from "@snippets/modules/memory/types/entity_summary_memory.mdx"
+import Example from "@snippets/modules/memory/how_to/entity_summary_memory.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/index.mdx

@@ -1,62 +1,34 @@
 ---
 sidebar_position: 3
 ---
+
 # Memory
 
-Most LLM applications have a conversational interface. An essential component of a conversation is being able to refer to information introduced earlier in the conversation.
-At bare minimum, a conversational system should be able to access some window of past messages directly.
-A more complex system will need to have a world model that it is constantly updating, which allows it to do things like maintain information about entities and their relationships.
+🚧 _Docs under construction_ 🚧
 
-We call this ability to store information about past interactions "memory".
-LangChain provides a lot of utilities for adding memory to a system.
-These utilities can be used by themselves or incorporated seamlessly into a chain.
+:::info
+Head to [Integrations](/docs/integrations/memory/) for documentation on built-in memory integrations with 3rd-party tools.
+:::
 
-A memory system needs to support two basic actions: reading and writing.
-Recall that every chain defines some core execution logic that expects certain inputs.
-Some of these inputs come directly from the user, but some of these inputs can come from memory.
-A chain will interact with its memory system twice in a given run.
-1. AFTER receiving the initial user inputs but BEFORE executing the core logic, a chain will READ from its memory system and augment the user inputs.
-2. AFTER executing the core logic but BEFORE returning the answer, a chain will WRITE the inputs and outputs of the current run to memory, so that they can be referred to in future runs.
+By default, Chains and Agents are stateless,
+meaning that they treat each incoming query independently (like the underlying LLMs and chat models themselves).
+In some applications, like chatbots, it is essential
+to remember previous interactions, both in the short and long-term.
+The **Memory** class does exactly that.
 
-![memory-diagram](/img/memory_diagram.png)
-
-
-## Building memory into a system
-The two core design decisions in any memory system are:
-- How state is stored
-- How state is queried
-
-### Storing: List of chat messages
-Underlying any memory is a history of all chat interactions.
-Even if these are not all used directly, they need to be stored in some form.
-One of the key parts of the LangChain memory module is a series of integrations for storing these chat messages,
-from in-memory lists to persistent databases.
-
-- [Chat message storage](/docs/modules/memory/chat_messages/): How to work with Chat Messages, and the various integrations offered
-
-### Querying: Data structures and algorithms on top of chat messages
-Keeping a list of chat messages is fairly straight-forward.
-What is less straight-forward are the data structures and algorithms built on top of chat messages that serve a view of those messages that is most useful.
-
-A very simply memory system might just return the most recent messages each run. A slightly more complex memory system might return a succinct summary of the past K messages.
-An even more sophisticated system might extract entities from stored messages and only return information about entities referenced in the current run.
-
-Each application can have different requirements for how memory is queried. The memory module should make it easy to both get started with simple memory systems and write your own custom systems if needed.
-
-- [Memory types](/docs/modules/memory/types/): The various data structures and algorithms that make up the memory types LangChain supports
+LangChain provides memory components in two forms.
+First, LangChain provides helper utilities for managing and manipulating previous chat messages.
+These are designed to be modular and useful regardless of how they are used.
+Secondly, LangChain provides easy ways to incorporate these utilities into chains.
 
 ## Get started
 
-Let's take a look at what Memory actually looks like in LangChain.
-Here we'll cover the basics of interacting with an arbitrary memory class.
+Memory involves keeping a concept of state around throughout a user's interactions with an language model. A user's interactions with a language model are captured in the concept of ChatMessages, so this boils down to ingesting, capturing, transforming and extracting knowledge from a sequence of chat messages. There are many different ways to do this, each of which exists as its own memory type.
+
+In general, for each type of memory there are two ways to understanding using memory. These are the standalone functions which extract information from a sequence of messages, and then there is the way you can use this type of memory in a chain.
+
+Memory can return multiple pieces of information (for example, the most recent N messages and a summary of all previous messages). The returned information can either be a string or a list of messages.
 
 import GetStarted from "@snippets/modules/memory/get_started.mdx"
 
 <GetStarted/>
-
-## Next steps
-
-And that's it for getting started!
-Please see the other sections for walkthroughs of more advanced topics,
-like custom memory, multiple memories, and more.
-

docs/docs_skeleton/docs/modules/memory/summary.mdx

@@ -4,6 +4,6 @@ Conversation summary memory summarizes the conversation as it happens and stores
 
 Let's first explore the basic functionality of this type of memory.
 
-import Example from "@snippets/modules/memory/types/summary.mdx"
+import Example from "@snippets/modules/memory/how_to/summary.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/memory/types/index.mdx

@@ -1,8 +0,0 @@
----
-sidebar_position: 2
----
-# Memory Types
-
-There are many different types of memory.
-Each have their own parameters, their own return types, and are useful in different scenarios.
-Please see their individual page for more detail on each one.

docs/docs_skeleton/docs/modules/memory/vectorstore_retriever_memory.mdx

@@ -6,6 +6,6 @@ This differs from most of the other Memory classes in that it doesn't explicitly
 
 In this case, the "docs" are previous conversation snippets. This can be useful to refer to relevant pieces of information that the AI was told earlier in the conversation.
 
-import Example from "@snippets/modules/memory/types/vectorstore_retriever_memory.mdx"
+import Example from "@snippets/modules/memory/how_to/vectorstore_retriever_memory.mdx"
 
 <Example/>

docs/docs_skeleton/docs/use_cases/question_answering/how_to/_category_.yml

@@ -1 +0,0 @@
-label: 'How to'

docs/docs_skeleton/generate_api_reference_links.py

@@ -5,7 +5,6 @@ import logging
 import os
 import re
 from pathlib import Path
-import argparse
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
@@ -15,12 +14,7 @@ _BASE_URL = "https://api.python.langchain.com/en/latest/"
 # Regular expression to match Python code blocks
 code_block_re = re.compile(r"^(```python\n)(.*?)(```\n)", re.DOTALL | re.MULTILINE)
 # Regular expression to match langchain import lines
-_IMPORT_RE = re.compile(
-    r"from\s+(langchain\.\w+(\.\w+)*?)\s+import\s+"
-    r"((?:\w+(?:,\s*)?)*"  # Match zero or more words separated by a comma+optional ws
-    r"(?:\s*\(.*?\))?)",  # Match optional parentheses block
-    re.DOTALL,  # Match newlines as well
-)
+_IMPORT_RE = re.compile(r"(from\s+(langchain\.\w+(\.\w+)*?)\s+import\s+)(\w+)")
 
 _CURRENT_PATH = Path(__file__).parent.absolute()
 # Directory where generated markdown files are stored
@@ -30,10 +24,6 @@ _JSON_PATH = _CURRENT_PATH.parent / "api_reference" / "guide_imports.json"
 
 def find_files(path):
     """Find all MDX files in the given path"""
-    # Check if is file first
-    if os.path.isfile(path):
-        yield path
-        return
     for root, _, files in os.walk(path):
         for file in files:
             if file.endswith(".mdx") or file.endswith(".md"):
@@ -47,33 +37,20 @@ def get_full_module_name(module_path, class_name):
     return inspect.getmodule(class_).__name__
 
 
-def get_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "--docs_dir",
-        type=str,
-        default=_DOCS_DIR,
-        help="Directory where generated markdown files are stored",
-    )
-    return parser.parse_args()
-
-
 def main():
     """Main function"""
-    args = get_args()
     global_imports = {}
 
-    for file in find_files(args.docs_dir):
+    for file in find_files(_DOCS_DIR):
         print(f"Adding links for imports in {file}")
+
+        # replace_imports now returns the import information rather than writing it to a file
         file_imports = replace_imports(file)
 
         if file_imports:
             # Use relative file path as key
-            relative_path = (
-                os.path.relpath(file, _DOCS_DIR).replace(".mdx", "").replace(".md", "")
-            )
-
-            doc_url = f"https://python.langchain.com/docs/{relative_path}"
+            relative_path = os.path.relpath(file, _DOCS_DIR)
+            doc_url = f"https://python.langchain.com/docs/{relative_path.replace('.mdx', '').replace('.md', '')}"
             for import_info in file_imports:
                 doc_title = import_info["title"]
                 class_name = import_info["imported"]
@@ -82,7 +59,6 @@ def main():
                 global_imports[class_name][doc_title] = doc_url
 
     # Write the global imports information to a JSON file
-    _JSON_PATH.parent.mkdir(parents=True, exist_ok=True)
     with _JSON_PATH.open("w") as f:
         json.dump(global_imports, f)
 
@@ -100,8 +76,7 @@ def _get_doc_title(data: str, file_name: str) -> str:
 
 
 def replace_imports(file):
-    """Replace imports in each Python code block with links to their
-    documentation and append the import info in a comment"""
+    """Replace imports in each Python code block with links to their documentation and append the import info in a comment"""
     all_imports = []
     with open(file, "r") as f:
         data = f.read()
@@ -121,45 +96,37 @@ def replace_imports(file):
         # Process imports in the code block
         imports = []
         for import_match in _IMPORT_RE.finditer(code):
-            module = import_match.group(1)
-            imports_str = (
-                import_match.group(3).replace("(\n", "").replace("\n)", "")
-            )  # Handle newlines within parentheses
-            # remove any newline and spaces, then split by comma
-            imported_classes = [
-                imp.strip()
-                for imp in re.split(r",\s*", imports_str.replace("\n", ""))
-                if imp.strip()
-            ]
-            for class_name in imported_classes:
-                try:
-                    module_path = get_full_module_name(module, class_name)
-                except AttributeError as e:
-                    logger.warning(f"Could not find module for {class_name}, {e}")
-                    continue
-                except ImportError as e:
-                    logger.warning(f"Failed to load for class {class_name}, {e}")
-                    continue
+            class_name = import_match.group(4)
+            try:
+                module_path = get_full_module_name(import_match.group(2), class_name)
+            except AttributeError as e:
+                logger.warning(f"Could not find module for {class_name}, {e}")
+                continue
+            except ImportError as e:
+                # Some CentOS OpenSSL issues can cause this to fail
+                logger.warning(f"Failed to load for class {class_name}, {e}")
+                continue
 
-                url = (
-                    _BASE_URL
-                    + module_path.split(".")[1]
-                    + "/"
-                    + module_path
-                    + "."
-                    + class_name
-                    + ".html"
-                )
+            url = (
+                _BASE_URL
+                + "/"
+                + module_path.split(".")[1]
+                + "/"
+                + module_path
+                + "."
+                + class_name
+                + ".html"
+            )
 
-                # Add the import information to our list
-                imports.append(
-                    {
-                        "imported": class_name,
-                        "source": module,
-                        "docs": url,
-                        "title": _DOC_TITLE,
-                    }
-                )
+            # Add the import information to our list
+            imports.append(
+                {
+                    "imported": class_name,
+                    "source": import_match.group(2),
+                    "docs": url,
+                    "title": _DOC_TITLE,
+                }
+            )
 
         if imports:
             all_imports.extend(imports)

docs/docs_skeleton/vercel.json

@@ -1610,59 +1610,59 @@
     },
     {
       "source": "/en/latest/modules/chains/examples/flare.html",
-      "destination": "/docs/use_cases/question_answering/how_to/flare"
+      "destination": "/docs/modules/chains/additional/flare"
     },
     {
       "source": "/en/latest/modules/chains/examples/graph_cypher_qa.html",
-      "destination": "/docs/use_cases/graph/graph_cypher_qa"
+      "destination": "/docs/modules/chains/additional/graph_cypher_qa"
     },
     {
       "source": "/en/latest/modules/chains/examples/graph_nebula_qa.html",
-      "destination": "/docs/use_cases/graph/graph_nebula_qa"
+      "destination": "/docs/modules/chains/additional/graph_nebula_qa"
     },
     {
       "source": "/en/latest/modules/chains/index_examples/graph_qa.html",
-      "destination": "/docs/use_cases/graph/graph_qa"
+      "destination": "/docs/modules/chains/additional/graph_qa"
     },
     {
       "source": "/en/latest/modules/chains/index_examples/hyde.html",
-      "destination": "/docs/use_cases/question_answering/how_to/hyde"
+      "destination": "/docs/modules/chains/additional/hyde"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_bash.html",
-      "destination": "/docs/use_cases/code_writing/llm_bash"
+      "destination": "/docs/modules/chains/additional/llm_bash"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_checker.html",
-      "destination": "/docs/use_cases/self_check/llm_checker"
+      "destination": "/docs/modules/chains/additional/llm_checker"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_math.html",
-      "destination": "/docs/use_cases/code_writing/llm_math"
+      "destination": "/docs/modules/chains/additional/llm_math"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_requests.html",
-      "destination": "/docs/use_cases/apis/llm_requests"
+      "destination": "/docs/modules/chains/additional/llm_requests"
     },
     {
       "source": "/en/latest/modules/chains/examples/llm_summarization_checker.html",
-      "destination": "/docs/use_cases/self_check/llm_summarization_checker"
+      "destination": "/docs/modules/chains/additional/llm_summarization_checker"
     },
     {
       "source": "/en/latest/modules/chains/examples/openapi.html",
-      "destination": "/docs/use_cases/apis/openapi"
+      "destination": "/docs/modules/chains/additional/openapi"
     },
     {
       "source": "/en/latest/modules/chains/examples/pal.html",
-      "destination": "/docs/use_cases/code_writing/pal"
+      "destination": "/docs/modules/chains/additional/pal"
     },
     {
       "source": "/en/latest/modules/chains/examples/tagging.html",
-      "destination": "/docs/use_cases/tagging"
+      "destination": "/docs/modules/chains/additional/tagging"
     },
     {
       "source": "/en/latest/modules/chains/index_examples/vector_db_text_generation.html",
-      "destination": "/docs/use_cases/question_answering/how_to/vector_db_text_generation"
+      "destination": "/docs/modules/chains/additional/vector_db_text_generation"
     },
     {
       "source": "/en/latest/modules/chains/generic/router.html",
@@ -3448,10 +3448,6 @@
       "source": "/docs/modules/model_io/models/llms/integrations/writer",
       "destination": "/docs/integrations/llms/writer"
     },
-    {
-      "source": "/en/latest/modules/prompts.html",
-      "destination": "/docs/modules/model_io/prompts"
-    },
     {
       "source": "/en/latest/modules/prompts/output_parsers.html",
       "destination": "/docs/modules/model_io/output_parsers/"
@@ -3476,10 +3472,6 @@
       "source": "/en/latest/modules/prompts/output_parsers/examples/retry.html",
       "destination": "/docs/modules/model_io/output_parsers/retry"
     },
-    {
-      "source": "/en/latest/modules/prompts/example_selectors.html",
-      "destination": "/docs/modules/model_io/prompts/example_selectors"
-    },
     {
       "source": "/en/latest/modules/prompts/example_selectors/examples/custom_example_selector.html",
       "destination": "/docs/modules/model_io/prompts/example_selectors/custom_example_selector"
@@ -3492,10 +3484,6 @@
       "source": "/en/latest/modules/prompts/example_selectors/examples/ngram_overlap.html",
       "destination": "/docs/modules/model_io/prompts/example_selectors/ngram_overlap"
     },
-    {
-      "source": "/en/latest/modules/prompts/prompt_templates.html",
-      "destination": "/docs/modules/model_io/prompts/prompt_templates"
-    },
     {
       "source": "/en/latest/modules/prompts/prompt_templates/examples/connecting_to_a_feature_store.html",
       "destination": "/docs/modules/model_io/prompts/prompt_templates/connecting_to_a_feature_store"
@@ -3748,10 +3736,6 @@
       "source": "/docs/modules/evaluation/:path*(/?)",
       "destination": "/docs/guides/evaluation/:path*"
     },
-    {
-      "source": "/en/latest/modules/indexes.html",
-      "destination": "/docs/modules/data_connection"
-    },
     {
       "source": "/en/latest/modules/indexes/:path*",
       "destination": "/docs/modules/data_connection/:path*"
@@ -3787,174 +3771,6 @@
     {
       "source": "/en/latest/:path*",
       "destination": "/docs/:path*"
-    },
-    {
-      "source": "/docs/modules/chains/additional/constitutional_chain",
-      "destination": "/docs/guides/safety/constitutional_chain"
-    },
-    {
-      "source": "/docs/modules/chains/additional/moderation",
-      "destination": "/docs/guides/safety/moderation"
-    },
-    {
-      "source": "/docs/modules/chains/popular/api",
-      "destination": "/docs/use_cases/apis/api"
-    },
-    {
-      "source": "/docs/modules/chains/additional/analyze_document",
-      "destination": "/docs/use_cases/question_answering/how_to/analyze_document"
-    },
-    {
-      "source": "/docs/modules/chains/popular/chat_vector_db",
-      "destination": "/docs/use_cases/question_answering/how_to/chat_vector_db"
-    },
-    {
-      "source": "/docs/modules/chains/additional/multi_retrieval_qa_router",
-      "destination": "/docs/use_cases/question_answering/how_to/multi_retrieval_qa_router"
-    },
-    {
-      "source": "/docs/modules/chains/additional/question_answering",
-      "destination": "/docs/use_cases/question_answering/how_to/question_answering"
-    },
-    {
-      "source": "/docs/modules/chains/popular/vector_db_qa",
-      "destination": "/docs/use_cases/question_answering/how_to/vector_db_qa"
-    },
-    {
-      "source": "/docs/modules/chains/popular/summarize",
-      "destination": "/docs/use_cases/summarization/summarize"
-    },
-    {
-      "source": "/docs/modules/chains/popular/sqlite",
-      "destination": "/docs/use_cases/tabular/sqlite"
-    },
-    {
-      "source": "/docs/modules/chains/popular/openai_functions",
-      "destination": "/docs/modules/chains/how_to/openai_functions"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_requests",
-      "destination": "/docs/use_cases/apis/llm_requests"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openai_openapi",
-      "destination": "/docs/use_cases/apis/openai_openapi"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openapi",
-      "destination": "/docs/use_cases/apis/openapi"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openapi_openai",
-      "destination": "/docs/use_cases/apis/openapi_openai"
-    },
-    {
-      "source": "/docs/modules/chains/additional/cpal",
-      "destination": "/docs/use_cases/code_writing/cpal"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_bash",
-      "destination": "/docs/use_cases/code_writing/llm_bash"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_math",
-      "destination": "/docs/use_cases/code_writing/llm_math"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_symbolic_math",
-      "destination": "/docs/use_cases/code_writing/llm_symbolic_math"
-    },
-    {
-      "source": "/docs/modules/chains/additional/pal",
-      "destination": "/docs/use_cases/code_writing/pal"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_arangodb_qa",
-      "destination": "/docs/use_cases/graph/graph_arangodb_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_cypher_qa",
-      "destination": "/docs/use_cases/graph/graph_cypher_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_hugegraph_qa",
-      "destination": "/docs/use_cases/graph/graph_hugegraph_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_kuzu_qa",
-      "destination": "/docs/use_cases/graph/graph_kuzu_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_nebula_qa",
-      "destination": "/docs/use_cases/graph/graph_nebula_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_qa",
-      "destination": "/docs/use_cases/graph/graph_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/graph_sparql_qa",
-      "destination": "/docs/use_cases/graph/graph_sparql_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/neptune_cypher_qa",
-      "destination": "/docs/use_cases/graph/neptune_cypher_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/tot",
-      "destination": "/docs/use_cases/graph/tot"
-    },
-    {
-      "source": "/docs/use_cases/question_answering//document-context-aware-QA",
-      "destination": "/docs/use_cases/question_answering/how_to/document-context-aware-QA"
-    },
-    {
-      "source": "/docs/modules/chains/additional/flare",
-      "destination": "/docs/use_cases/question_answering/how_to/flare"
-    },
-    {
-      "source": "/docs/modules/chains/additional/hyde",
-      "destination": "/docs/use_cases/question_answering/how_to/hyde"
-    },
-    {
-      "source": "/docs/use_cases/question_answering//local_retrieval_qa",
-      "destination": "/docs/use_cases/question_answering/how_to/local_retrieval_qa"
-    },
-    {
-      "source": "/docs/modules/chains/additional/qa_citations",
-      "destination": "/docs/use_cases/question_answering/how_to/qa_citations"
-    },
-    {
-      "source": "/docs/modules/chains/additional/vector_db_text_generation",
-      "destination": "/docs/use_cases/question_answering/how_to/vector_db_text_generation"
-    },
-    {
-      "source": "/docs/modules/chains/additional/openai_functions_retrieval_qa",
-      "destination": "/docs/use_cases/question_answering/integrations/openai_functions_retrieval_qa"
-    },
-    {
-      "source": "/docs/use_cases/question_answering//semantic-search-over-chat",
-      "destination": "/docs/use_cases/question_answering/integrations/semantic-search-over-chat"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_checker",
-      "destination": "/docs/use_cases/self_check/llm_checker"
-    },
-    {
-      "source": "/docs/modules/chains/additional/llm_summarization_checker",
-      "destination": "/docs/use_cases/self_check/llm_summarization_checker"
-    },
-    {
-      "source": "/docs/modules/chains/additional/elasticsearch_database",
-      "destination": "/docs/use_cases/tabular/elasticsearch_database"
-    },
-    {
-      "source": "/docs/modules/chains/additional/tagging",
-      "destination": "/docs/use_cases/tagging"
-    },
-    {
-      "source": "docs/integrations/providers/agent_with_wandb_tracing",
-      "destination": "docs/integrations/providers/wandb_tracing"
     }
   ]
 }

docs/docs_skeleton/vercel_build.sh

@@ -1,47 +1,26 @@
 #!/bin/bash
 
-version_compare() {
-    local v1=(${1//./ })
-    local v2=(${2//./ })
-    for i in {0..2}; do
-        if (( ${v1[i]} < ${v2[i]} )); then
-            return 1
-        fi
-    done
-    return 0
-}
-
-openssl_version=$(openssl version | awk '{print $2}')
-required_openssl_version="1.1.1"
-
-python_version=$(python3 --version 2>&1 | awk '{print $2}')
-required_python_version="3.10"
-
-echo "OpenSSL Version"
-echo $openssl_version
-echo "Python Version"
-echo $python_version
-# If openssl version is less than 1.1.1 AND python version is less than 3.10
-if ! version_compare $openssl_version $required_openssl_version && ! version_compare $python_version $required_python_version; then
 ### See: https://github.com/urllib3/urllib3/issues/2168
 # Requests lib breaks for old SSL versions,
 # which are defaults on Amazon Linux 2 (which Vercel uses for builds)
-    yum -y update
-    yum remove openssl-devel -y
-    yum install gcc bzip2-devel libffi-devel zlib-devel wget tar -y
-    yum install openssl11 -y
-    yum install openssl11-devel -y
-
-    wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz
-    tar xzf Python-3.11.4.tgz
-    cd Python-3.11.4
-    ./configure
-    make altinstall
-    echo "Python Version"
-    python3.11 --version
-    cd ..
-fi
+yum -y update
+yum remove openssl-devel -y
+yum install gcc bzip2-devel libffi-devel zlib-devel wget tar -y
+yum install openssl11 -y
+yum install openssl11-devel -y
+# Install python 3.11 to connect with openSSL 1.1.1
+wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz 
+tar xzf Python-3.11.4.tgz 
+cd Python-3.11.4 
+./configure 
+make altinstall
+# Check python version
+echo "Python Version"
+python3.11 --version
+cd ..
+###
 
+# Install nbdev and generate docs
 cd ..
 python3.11 -m venv .venv
 source .venv/bin/activate

docs/extras/additional_resources/tutorials.mdx

@@ -1,6 +1,5 @@
 # Tutorials
 
-Below are links to video tutorials and courses on LangChain. For written guides on common use cases for LangChain, check out the [use cases guides](/docs/use_cases).
 
 ⛓ icon marks a new addition [last update 2023-07-05]
 

docs/extras/guides/debugging.md

@@ -4,7 +4,7 @@ If you're building with LLMs, at some point something will break, and you'll nee
 
 Here's a few different tools and functionalities to aid in debugging.
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 ## Tracing
 

docs/extras/guides/expression_language/interface.ipynb

@@ -1,269 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "9a9acd2e",
-   "metadata": {},
-   "source": [
-    "# Interface\n",
-    "\n",
-    "In an effort to make it as easy as possible to create custom chains, we've implemented a [\"Runnable\"](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.Runnable.html#langchain.schema.runnable.Runnable) protocol that most components implement. This is a standard interface with a few different methods, which makes it easy to define custom chains as well as making it possible to invoke them in a standard way. The standard interface exposed includes:\n",
-    "\n",
-    "- `stream`: stream back chunks of the response\n",
-    "- `invoke`: call the chain on an input\n",
-    "- `batch`: call the chain on a list of inputs\n",
-    "\n",
-    "These also have corresponding async methods:\n",
-    "\n",
-    "- `astream`: stream back chunks of the response async\n",
-    "- `ainvoke`: call the chain on an input async\n",
-    "- `abatch`: call the chain on a list of inputs async\n",
-    "\n",
-    "The type of the input varies by component. For a prompt it is a dictionary, for a retriever it is a single string, for a model either a single string, a list of chat messages, or a PromptValue.\n",
-    "\n",
-    "The output type also varies by component. For an LLM it is a string, for a ChatModel it's a ChatMessage, for a prompt it's a PromptValue, for a retriever it's a list of documents.\n",
-    "\n",
-    "Let's take a look at these methods! To do so, we'll create a super simple PromptTemplate + ChatModel chain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 20,
-   "id": "466b65b3",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "from langchain.chat_models import ChatOpenAI"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "3c634ef0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = ChatOpenAI()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "d1850a1f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "56d0669f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = prompt | model"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "daf2b2b2",
-   "metadata": {},
-   "source": [
-    "## Stream"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "bea9639d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Sure, here's a bear-themed joke for you:\n",
-      "\n",
-      "Why don't bears wear shoes?\n",
-      "\n",
-      "Because they have bear feet!"
-     ]
-    }
-   ],
-   "source": [
-    "for s in chain.stream({\"topic\": \"bears\"}):\n",
-    "    print(s.content, end=\"\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cbf1c782",
-   "metadata": {},
-   "source": [
-    "## Invoke"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "470e483f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "88f0c279",
-   "metadata": {},
-   "source": [
-    "## Batch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "9685de67",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears ever wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False),\n",
-       " AIMessage(content=\"Why don't cats play poker in the wild?\\n\\nToo many cheetahs!\", additional_kwargs={}, example=False)]"
-      ]
-     },
-     "execution_count": 19,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b960cbfe",
-   "metadata": {},
-   "source": [
-    "## Async Stream"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "ea35eee4",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Why don't bears wear shoes?\n",
-      "\n",
-      "Because they have bear feet!"
-     ]
-    }
-   ],
-   "source": [
-    "async for s in chain.astream({\"topic\": \"bears\"}):\n",
-    "    print(s.content, end=\"\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "04cb3324",
-   "metadata": {},
-   "source": [
-    "## Async Invoke"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "ef8c9b20",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\"Sure, here you go:\\n\\nWhy don't bears wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "await chain.ainvoke({\"topic\": \"bears\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3da288d5",
-   "metadata": {},
-   "source": [
-    "## Async Batch"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "eba2a103",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\", additional_kwargs={}, example=False)]"
-      ]
-     },
-     "execution_count": 18,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "await chain.abatch([{\"topic\": \"bears\"}])"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/guides/model_laboratory.ipynb

@@ -5,7 +5,7 @@
    "id": "920a3c1a",
    "metadata": {},
    "source": [
-    "# Model comparison\n",
+    "# Model Comparison\n",
     "\n",
     "Constructing your language model application will likely involved choosing between many different options of prompts, models, and even chains to use. When doing so, you will want to compare these different options on different inputs in an easy, flexible, and intuitive way. \n",
     "\n",
@@ -254,7 +254,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.10.9"
   }
  },
  "nbformat": 4,

docs/extras/integrations/callbacks/argilla.ipynb

@@ -14,7 +14,7 @@
     "> using both human and machine feedback. We provide support for each step in the MLOps cycle, \n",
     "> from data labeling to model monitoring.\n",
     "\n",
-    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/integrations/callbacks/argilla.html\">\n",
+    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/hwchase17/langchain/blob/master/docs/modules/callbacks/integrations/argilla.html\">\n",
     "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
     "</a>"
    ]

docs/extras/integrations/chat/anthropic_functions.ipynb

@@ -1,287 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "5125a1e3",
-   "metadata": {},
-   "source": [
-    "# Anthropic Functions\n",
-    "\n",
-    "This notebook shows how to use an experimental wrapper around Anthropic that gives it the same API as OpenAI Functions."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "378be79b",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/Users/harrisonchase/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.14) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.\n",
-      "  warnings.warn(\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain_experimental.llms.anthropic_functions import AnthropicFunctions"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "65499965",
-   "metadata": {},
-   "source": [
-    "## Initialize Model\n",
-    "\n",
-    "You can initialize this wrapper the same way you'd initialize ChatAnthropic"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "e1d535f6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = AnthropicFunctions(model='claude-2')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "fcc9eaf4",
-   "metadata": {},
-   "source": [
-    "## Passing in functions\n",
-    "\n",
-    "You can now pass in functions in a similar way"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "0779c320",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "functions=[\n",
-    "    {\n",
-    "      \"name\": \"get_current_weather\",\n",
-    "      \"description\": \"Get the current weather in a given location\",\n",
-    "      \"parameters\": {\n",
-    "        \"type\": \"object\",\n",
-    "        \"properties\": {\n",
-    "          \"location\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"description\": \"The city and state, e.g. San Francisco, CA\"\n",
-    "          },\n",
-    "          \"unit\": {\n",
-    "            \"type\": \"string\",\n",
-    "            \"enum\": [\"celsius\", \"fahrenheit\"]\n",
-    "          }\n",
-    "        },\n",
-    "        \"required\": [\"location\"]\n",
-    "      }\n",
-    "    }\n",
-    "  ]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "ad75a933",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.schema import HumanMessage"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "fc703085",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "response = model.predict_messages(\n",
-    "    [HumanMessage(content=\"whats the weater in boston?\")], \n",
-    "    functions=functions\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "04d7936a",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=' ', additional_kwargs={'function_call': {'name': 'get_current_weather', 'arguments': '{\"location\": \"Boston, MA\", \"unit\": \"fahrenheit\"}'}}, example=False)"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "response"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0072fdba",
-   "metadata": {},
-   "source": [
-    "## Using for extraction\n",
-    "\n",
-    "You can now use this for extraction."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "7af5c567",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import create_extraction_chain\n",
-    "schema = {\n",
-    "    \"properties\": {\n",
-    "        \"name\": {\"type\": \"string\"},\n",
-    "        \"height\": {\"type\": \"integer\"},\n",
-    "        \"hair_color\": {\"type\": \"string\"},\n",
-    "    },\n",
-    "    \"required\": [\"name\", \"height\"],\n",
-    "}\n",
-    "inp = \"\"\"\n",
-    "Alex is 5 feet tall. Claudia is 1 feet taller Alex and jumps higher than him. Claudia is a brunette and Alex is blonde.\n",
-    "        \"\"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "bd01082a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = create_extraction_chain(schema, model)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "b5a23e9f",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[{'name': 'Alex', 'height': '5', 'hair_color': 'blonde'},\n",
-       " {'name': 'Claudia', 'height': '6', 'hair_color': 'brunette'}]"
-      ]
-     },
-     "execution_count": 10,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.run(inp)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "90ec959e",
-   "metadata": {},
-   "source": [
-    "## Using for tagging\n",
-    "\n",
-    "You can now use this for tagging"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "03c1eb0d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chains import create_tagging_chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "581c0ece",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "schema = {\n",
-    "    \"properties\": {\n",
-    "        \"sentiment\": {\"type\": \"string\"},\n",
-    "        \"aggressiveness\": {\"type\": \"integer\"},\n",
-    "        \"language\": {\"type\": \"string\"},\n",
-    "    }\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "d9a8570e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "chain = create_tagging_chain(schema, model)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "cf37d679",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'sentiment': 'positive', 'aggressiveness': '0', 'language': 'english'}"
-      ]
-     },
-     "execution_count": 15,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.run(\"this is really cool\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/chat/azureml_chat_endpoint.ipynb

@@ -1,95 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# AzureML Chat Online Endpoint\n",
-    "\n",
-    "[AzureML](https://azure.microsoft.com/en-us/products/machine-learning/) is a platform used to build, train, and deploy machine learning models. Users can explore the types of models to deploy in the Model Catalog, which provides Azure Foundation Models and OpenAI Models. Azure Foundation Models include various open-source models and popular Hugging Face models. Users can also import models of their liking into AzureML.\n",
-    "\n",
-    "This notebook goes over how to use a chat model hosted on an `AzureML online endpoint`"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models.azureml_endpoint import AzureMLChatOnlineEndpoint"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Set up\n",
-    "\n",
-    "To use the wrapper, you must [deploy a model on AzureML](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-foundation-models?view=azureml-api-2#deploying-foundation-models-to-endpoints-for-inferencing) and obtain the following parameters:\n",
-    "\n",
-    "* `endpoint_api_key`: The API key provided by the endpoint\n",
-    "* `endpoint_url`: The REST endpoint url provided by the endpoint"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Content Formatter\n",
-    "\n",
-    "The `content_formatter` parameter is a handler class for transforming the request and response of an AzureML endpoint to match with required schema. Since there are a wide range of models in the model catalog, each of which may process data differently from one another, a `ContentFormatterBase` class is provided to allow users to transform data to their liking. The following content formatters are provided:\n",
-    "\n",
-    "* `LLamaContentFormatter`: Formats request and response data for LLaMa2-chat"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content='  The Collatz Conjecture is one of the most famous unsolved problems in mathematics, and it has been the subject of much study and research for many years. While it is impossible to predict with certainty whether the conjecture will ever be solved, there are several reasons why it is considered a challenging and important problem:\\n\\n1. Simple yet elusive: The Collatz Conjecture is a deceptively simple statement that has proven to be extraordinarily difficult to prove or disprove. Despite its simplicity, the conjecture has eluded some of the brightest minds in mathematics, and it remains one of the most famous open problems in the field.\\n2. Wide-ranging implications: The Collatz Conjecture has far-reaching implications for many areas of mathematics, including number theory, algebra, and analysis. A solution to the conjecture could have significant impacts on these fields and potentially lead to new insights and discoveries.\\n3. Computational evidence: While the conjecture remains unproven, extensive computational evidence supports its validity. In fact, no counterexample to the conjecture has been found for any starting value up to 2^64 (a number', additional_kwargs={}, example=False)"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.chat_models.azureml_endpoint import LlamaContentFormatter\n",
-    "from langchain.schema import HumanMessage\n",
-    "\n",
-    "chat = AzureMLChatOnlineEndpoint(content_formatter=LlamaContentFormatter())\n",
-    "response = chat(messages=[\n",
-    "    HumanMessage(content=\"Will the Collatz conjecture ever be solved?\")\n",
-    "])\n",
-    "response"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.11"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/chat/google_vertex_ai_palm.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -9,7 +8,11 @@
     "\n",
     "Note: This is seperate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there. \n",
     "\n",
-    "By default, Google Cloud [does not use](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance#foundation_model_development) Customer Data to train its foundation models as part of Google Cloud`s AI/ML Privacy Commitment. More details about how Google processes data can also be found in [Google's Customer Data Processing Addendum (CDPA)](https://cloud.google.com/terms/data-processing-addendum).\n",
+    "PaLM API on Vertex AI is a Preview offering, subject to the Pre-GA Offerings Terms of the [GCP Service Specific Terms](https://cloud.google.com/terms/service-terms). \n",
+    "\n",
+    "Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the [launch stage descriptions](https://cloud.google.com/products#product-launch-stages). Further, by using PaLM API on Vertex AI, you agree to the Generative AI Preview [terms and conditions](https://cloud.google.com/trustedtester/aitos) (Preview Terms).\n",
+    "\n",
+    "For PaLM API on Vertex AI, you can process personal data as outlined in the Cloud Data Processing Addendum, subject to applicable restrictions and obligations in the Agreement (as defined in the Preview Terms).\n",
     "\n",
     "To use Vertex AI PaLM you must have the `google-cloud-aiplatform` Python package installed and either:\n",
     "- Have credentials configured for your environment (gcloud, workload identity, etc...)\n",
@@ -87,7 +90,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -140,7 +142,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "execution": {

docs/extras/integrations/document_loaders/concurrent.ipynb

@@ -1,94 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "23c6e167",
-   "metadata": {},
-   "source": [
-    "# Concurrent Loader\n",
-    "\n",
-    "Works just like the GenericLoader but concurrently for those who choose to optimize their workflow.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "6ff3fb1f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import ConcurrentLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "ce96fa20",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = ConcurrentLoader.from_filesystem('example_data/', glob=\"**/*.txt\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "06a6cf5d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "files = loader.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "b87d3e58",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "2"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "len(files)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "668f1ee5",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/document_loaders/example_data/notebook.md

@@ -2,7 +2,7 @@
 
 This notebook covers how to load data from an .ipynb notebook into a format suitable by LangChain.
 
-
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! Instead, edit the notebook w/the location & name as this file. -->
 
 
 ```python

docs/extras/integrations/document_loaders/huawei_obs_directory.ipynb

@@ -1,178 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "c83b6a4c",
-   "metadata": {},
-   "source": [
-    "# Huawei OBS Directory\n",
-    "The following code demonstrates how to load objects from the Huawei OBS (Object Storage Service) as documents."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c2191935",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install the required package\n",
-    "# pip install esdk-obs-python"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "55fca3b4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import OBSDirectoryLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "c3ed419f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "endpoint = \"your-endpoint\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "3428fd4e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Configure your access credentials\\n\n",
-    "config = {\n",
-    "    \"ak\": \"your-access-key\",\n",
-    "    \"sk\": \"your-secret-key\"\n",
-    "}\n",
-    "loader = OBSDirectoryLoader(\"your-bucket-name\", endpoint=endpoint, config=config)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9beede9f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1e20a839",
-   "metadata": {},
-   "source": [
-    "## Specify a Prefix for Loading\n",
-    "If you want to load objects with a specific prefix from the bucket, you can use the following code:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "125f311d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = OBSDirectoryLoader(\"your-bucket-name\", endpoint=endpoint, config=config, prefix=\"test_prefix\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b3488037",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "84c82c0a",
-   "metadata": {},
-   "source": [
-    "## Get Authentication Information from ECS\n",
-    "If your langchain is deployed on Huawei Cloud ECS and [Agency is set up](https://support.huaweicloud.com/intl/en-us/usermanual-ecs/ecs_03_0166.html#section7), the loader can directly get the security token from ECS without needing access key and secret key. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "1db99969",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "config = {\"get_token_from_ecs\": True}\n",
-    "loader = OBSDirectoryLoader(\"your-bucket-name\", endpoint=endpoint, config=config)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "57dd9f35",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "30205d25",
-   "metadata": {},
-   "source": [
-    "## Use a Public Bucket\n",
-    "If your bucket's bucket policy allows anonymous access (anonymous users have `listBucket` and `GetObject` permissions), you can directly load the objects without configuring the `config` parameter."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "4dfa2ef0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = OBSDirectoryLoader(\"your-bucket-name\", endpoint=endpoint)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "67d4c1d0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.7"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/document_loaders/huawei_obs_file.ipynb

@@ -1,180 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "4394a872",
-   "metadata": {},
-   "source": [
-    "# Huawei OBS File\n",
-    "The following code demonstrates how to load an object from the Huawei OBS (Object Storage Service) as document."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c43d811b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install the required package\n",
-    "# pip install esdk-obs-python"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "5e16bae6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders.obs_file import OBSFileLoader"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "75cc7e7c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "endpoint = \"your-endpoint\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "f9816984",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from obs import ObsClient\n",
-    "obs_client = ObsClient(access_key_id=\"your-access-key\", secret_access_key=\"your-secret-key\", server=endpoint)\n",
-    "loader = OBSFileLoader(\"your-bucket-name\", \"your-object-key\", client=obs_client)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6143b39b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "633e05ca",
-   "metadata": {},
-   "source": [
-    "## Each Loader with Separate Authentication Information\n",
-    "If you don't need to reuse OBS connections between different loaders, you can directly configure the `config`. The loader will use the config information to initialize its own OBS client."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "a5dd6a5d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Configure your access credentials\\n\n",
-    "config = {\n",
-    "    \"ak\": \"your-access-key\",\n",
-    "    \"sk\": \"your-secret-key\"\n",
-    "}\n",
-    "loader = OBSFileLoader(\"your-bucket-name\", \"your-object-key\",endpoint=endpoint, config=config)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9a741f1c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1e2e611c",
-   "metadata": {},
-   "source": [
-    "## Get Authentication Information from ECS\n",
-    "If your langchain is deployed on Huawei Cloud ECS and [Agency is set up](https://support.huaweicloud.com/intl/en-us/usermanual-ecs/ecs_03_0166.html#section7), the loader can directly get the security token from ECS without needing access key and secret key. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "338fafef",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "config = {\"get_token_from_ecs\": True}\n",
-    "loader = OBSFileLoader(\"your-bucket-name\", \"your-object-key\", endpoint=endpoint, config=config)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "73976c55",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b77aa18c",
-   "metadata": {},
-   "source": [
-    "## Access a Publicly Accessible Object\n",
-    "If the object you want to access allows anonymous user access (anonymous users have `GetObject` permission), you can directly load the object without configuring the `config` parameter."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "df83d121",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader = OBSFileLoader(\"your-bucket-name\", \"your-object-key\", endpoint=endpoint)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "82a844ba",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "loader.load()"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.7"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/document_loaders/mediawikidump.ipynb

@@ -57,13 +57,7 @@
     }
    ],
    "source": [
-    "loader = MWDumpLoader(\n",
-    "    file_path = \"example_data/testmw_pages_current.xml\", \n",
-    "    encoding=\"utf8\",\n",
-    "    #namespaces = [0,2,3] Optional list to load only specific namespaces. Loads all namespaces by default.\n",
-    "    skip_redirects = True, #will skip over pages that just redirect to other pages (or not if False)\n",
-    "    stop_on_error = False #will skip over pages that cause parsing errors (or not if False)\n",
-    "     )\n",
+    "loader = MWDumpLoader(\"example_data/testmw_pages_current.xml\", encoding=\"utf8\")\n",
     "documents = loader.load()\n",
     "print(f\"You have {len(documents)} document(s) in your data \")"
    ]

docs/extras/integrations/document_loaders/tomarkdown.ipynb

@@ -113,7 +113,7 @@
       "\n",
       "The modules are (from least to most complex):\n",
       "\n",
-      "- [Models](https://python.langchain.com/docs/modules/model_io/models/): Supported model types and integrations.\n",
+      "- [Models](https://python.langchain.com/en/latest/modules/models.html): Supported model types and integrations.\n",
       "\n",
       "- [Prompts](https://python.langchain.com/en/latest/modules/prompts.html): Prompt management, optimization, and serialization.\n",
       "\n",

docs/extras/integrations/document_loaders/unstructured_file.ipynb

@@ -18,7 +18,8 @@
    "outputs": [],
    "source": [
     "# # Install package\n",
-    "!pip install \"unstructured[all-docs]\"\n"
+    "!pip install \"unstructured[local-inference]\"\n",
+    "!pip install layoutparser[layoutmodels,tesseract]"
    ]
   },
   {

docs/extras/integrations/document_loaders/youtube_audio.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "e48afb8d",
    "metadata": {},
@@ -12,8 +11,7 @@
     "\n",
     "Below we show how to easily go from a YouTube url to text to chat!\n",
     "\n",
-    "We wil use the `OpenAIWhisperParser`, which will use the OpenAI Whisper API to transcribe audio to text, \n",
-    "and the  `OpenAIWhisperParserLocal` for local support and running on private clouds or on premise.\n",
+    "We wil use the `OpenAIWhisperParser`, which will use the OpenAI Whisper API to transcribe audio to text.\n",
     "\n",
     "Note: You will need to have an `OPENAI_API_KEY` supplied."
    ]
@@ -26,7 +24,7 @@
    "outputs": [],
    "source": [
     "from langchain.document_loaders.generic import GenericLoader\n",
-    "from langchain.document_loaders.parsers import OpenAIWhisperParser, OpenAIWhisperParserLocal\n",
+    "from langchain.document_loaders.parsers import OpenAIWhisperParser\n",
     "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader"
    ]
   },
@@ -48,8 +46,7 @@
    "outputs": [],
    "source": [
     "! pip install yt_dlp\n",
-    "! pip install pydub\n",
-    "! pip install librosa"
+    "! pip install pydub"
    ]
   },
   {
@@ -66,18 +63,6 @@
     "Let's take the first lecture of Andrej Karpathy's YouTube course as an example! "
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "8682f256",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# set a flag to switch between local and remote parsing\n",
-    "# change this to True if you want to use local parsing\n",
-    "local = False"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -117,10 +102,7 @@
     "save_dir = \"~/Downloads/YouTube\"\n",
     "\n",
     "# Transcribe the videos to text\n",
-    "if local:\n",
-    "    loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParserLocal())\n",
-    "else:\n",
-    "    loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParser())\n",
+    "loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParser())\n",
     "docs = loader.load()"
    ]
   },
@@ -293,7 +275,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -307,12 +289,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "97cc609b13305c559618ec78a438abc56230b9381f827f22d070313b9a1f3777"
-   }
+   "version": "3.9.16"
   }
  },
  "nbformat": 4,

docs/extras/integrations/llms/Fireworks.ipynb

@@ -1,231 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "cc6caafa",
-   "metadata": {},
-   "source": [
-    "# Fireworks\n",
-    "\n",
-    ">[Fireworks](https://www.fireworks.ai/) is an AI startup focused on accelerating product development on generative AI by creating an innovative AI experiment and production platform. \n",
-    "\n",
-    "This example goes over how to use LangChain to interact with `Fireworks` models."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 25,
-   "id": "60b6dbb2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.llms.fireworks import Fireworks, FireworksChat\n",
-    "from langchain import PromptTemplate, LLMChain\n",
-    "from langchain.prompts.chat import (\n",
-    "    ChatPromptTemplate,\n",
-    "    HumanMessagePromptTemplate,\n",
-    ")\n",
-    "import os"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ccff689e",
-   "metadata": {},
-   "source": [
-    "# Setup\n",
-    "\n",
-    "Contact Fireworks AI for the an API Key to access our models\n",
-    "\n",
-    "Set up your model using a model id. If the model is not set, the default model is fireworks-llama-v2-13b-chat."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "9ca87a2e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Initialize a Fireworks LLM\n",
-    "os.environ['FIREWORKS_API_KEY'] = \"\" #change this to your own API KEY\n",
-    "llm = Fireworks(model_id=\"fireworks-llama-v2-13b-chat\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 19,
-   "id": "43a11ba8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Create LLM chain\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "acc24d0c",
-   "metadata": {},
-   "source": [
-    "# Calling the Model\n",
-    "\n",
-    "You can use the LLMs to call the model for specified prompt(s). \n",
-    "\n",
-    "Current Specified Models: \n",
-    "* fireworks-falcon-7b, fireworks-falcon-40b-w8a16\n",
-    "* fireworks-guanaco-30b, fireworks-guanaco-33b-w8a16\n",
-    "* fireworks-llama-7b, fireworks-llama-13b, fireworks-llama-30b-w8a16\n",
-    "* fireworks-llama-v2-13b, fireworks-llama-v2-13b-chat, fireworks-llama-v2-13b-w8a16, fireworks-llama-v2-13b-chat-w8a16\n",
-    "* fireworks-llama-v2-7b, fireworks-llama-v2-7b-chat, fireworks-llama-v2-7b-w8a16, fireworks-llama-v2-7b-chat-w8a16"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "bf0a425c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "It's a question that has been debated for years, with different analysts and fans making their cases for various signal-callers. Here are some of the top contenders for the title of best quarterback in the NFL:\n",
-      "\n",
-      "1. Tom Brady: The New England Patriots legend has won six Super Bowls and has been named Super Bowl MVP four times. He's known for his precision passing, pocket presence, and ability to read defenses.\n",
-      "2. Aaron Rodgers: The Green Bay Packers quarterback has won two Super Bowls and has been named NFL MVP twice. He's known for his quick release, accuracy, and ability to extend plays with his feet.\n",
-      "3. Drew Brees: The New Orleans Saints quarterback has won a Super Bowl and has been named NFL MVP once. He's known for his accuracy, pocket presence, and ability to read defenses.\n",
-      "4. Patrick Mahomes: The Kansas City Chiefs quarterback has won a Super Bowl and has been named NFL MVP twice. He's known for his arm strength, athleticism, and ability to make plays outside of the pocket.\n",
-      "5. Russell Wilson: The Seattle Seahawks quarterback has won a Super Bowl and has been named NFL MVP once. He's known for his mobility, accuracy, and ability to extend plays with his feet.\n",
-      "\n",
-      "Of course, there are other talented quarterbacks in the league, such as Lamar Jackson, Deshaun Watson, and Carson Wentz, who could also be considered among the best. Ultimately, the answer to the question of who's the best quarterback in the NFL is subjective and can vary depending on individual perspectives and criteria.\n"
-     ]
-    }
-   ],
-   "source": [
-    "#single prompt\n",
-    "output = llm(\"Who's the best quarterback in the NFL?\")\n",
-    "print(output)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "afc7de6f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "generations=[[Generation(text=\"\\nWho is the best cricket player in the world in 2016?\\nThe best cricket player in the world in 2016 is Virat Kohli. The Indian captain has had a fabulous year, scoring heavily in all formats of the game, leading India to several victories, and breaking several records. In Test cricket, Kohli has scored 1215 runs at an average of 75.33 with 6 centuries and 4 fifties, which is the highest number of runs scored by any player in a calendar year. In ODI cricket, he has scored 1143 runs at an average of 83.42 with 7 centuries and 6 fifties, which is also the highest number of runs scored by any player in a calendar year. Additionally, Kohli has led India to the number one ranking in Test cricket, and has been named the ICC Test Player of the Year and the ICC ODI Player of the Year.\\nVirat Kohli has been in incredible form in 2016, and his performances have made him the standout player of the year. Other players who have had a great year include Steve Smith, Joe Root, and Kane Williamson, but Kohli's consistency and dominance in all formats of the game make him the best cricket player in the world in 2016.\", generation_info=None)], [Generation(text=\"\\n\\nA: LeBron James.\\n\\nB: Kevin Durant.\\n\\nC: Steph Curry.\\n\\nD: James Harden.\\n\\nE: Other (please specify).\\n\\nWhat's your answer?\", generation_info=None)]] llm_output={'token_usage': {}, 'model_id': 'fireworks-llama-v2-13b-chat'} run=[RunInfo(run_id=UUID('d14b6bee-7692-46ad-8798-acb6f72fc7fb')), RunInfo(run_id=UUID('b9f5b3b5-9e62-4eaf-b269-ecf0cbbcfb82'))]\n"
-     ]
-    }
-   ],
-   "source": [
-    "#calling multiple prompts\n",
-    "output = llm.generate([\"Who's the best cricket player in 2016?\", \"Who's the best basketball player in the league?\"])\n",
-    "print(output)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "b801c20d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "Kansas City in December can be quite chilly, with average high\n"
-     ]
-    }
-   ],
-   "source": [
-    "#setting parameters: model_id, temperature, max_tokens, top_p\n",
-    "llm = Fireworks(model_id=\"fireworks-llama-v2-13b-chat\", temperature=0.7, max_tokens=15, top_p=1.0)\n",
-    "print(llm(\"What's the weather like in Kansas City in December?\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "137662a6",
-   "metadata": {},
-   "source": [
-    "# Create and Run Chain\n",
-    "\n",
-    "Create a prompt template to be used with the LLM Chain. Once this prompt template is created, initialize the chain with the LLM and prompt template, and run the chain with the specified prompts."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 26,
-   "id": "fd2c6bc1",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "(Note: I'm just an AI and not a branding expert, so take this as a starting point for your own research and brainstorming.)\n",
-      "A good name for a company that makes football helmets could be:\n",
-      "\n",
-      "1. Helix Pro: This name plays off the idea of a helix, or spiral, shape that is commonly associated with football helmets. \"Pro\" implies a professional-level product.\n",
-      "2. Gridiron Gear: \"Gridiron\" is a term used to describe a football field, and \"gear\" highlights the company's focus on producing high-quality football helmets.\n",
-      "3. Linebacker Lab: \"Linebacker\" is a position on the football field, and \"Lab\" suggests a focus on research and development.\n",
-      "4. Helmet Hut: This name is simple and easy to remember, and it immediately conveys the company's focus on football helmets.\n",
-      "5. Tackle Tech: \"Tackle\" is a term used in football to describe a hit or collision, and \"Tech\" implies a focus on advanced technology and innovation.\n",
-      "6. Victory Vest: \"Victory\" implies a focus on winning and success, and \"Vest\" could suggest a protective or armored design.\n",
-      "7. Pigskin Pro: \"Pigskin\" is a term used to describe a football, and \"Pro\" implies a professional-level product.\n",
-      "8. Football Fusion: This name could suggest a combination of different materials or technologies to create a high-quality football helmet.\n",
-      "9. Endzone Edge: \"Endzone\" is the area of the football field where a team scores a touchdown, and \"Edge\" implies a competitive advantage.\n",
-      "10. MVP Masks: \"MVP\" stands for \"Most Valuable Player,\" and \"Masks\" highlights the protective nature of the company's football helmets.\n",
-      "\n",
-      "Remember, the name you choose for your company should be memorable, easy to pronounce and spell, and convey a sense of quality and professionalism. It's also important to check that the name isn't already in use by another company, and to consider any potential trademark issues.\n"
-     ]
-    }
-   ],
-   "source": [
-    "human_message_prompt = HumanMessagePromptTemplate(\n",
-    "    prompt=PromptTemplate(\n",
-    "        template=\"What is a good name for a company that makes {product}?\",\n",
-    "        input_variables=[\"product\"],\n",
-    "    )\n",
-    ")\n",
-    "\n",
-    "chat_prompt_template = ChatPromptTemplate.from_messages([human_message_prompt])\n",
-    "chat = Fireworks()\n",
-    "chain = LLMChain(llm=chat, prompt=chat_prompt_template)\n",
-    "output = chain.run(\"football helmets\")\n",
-    "\n",
-    "print(output)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/llms/azureml_endpoint_example.ipynb

@@ -28,9 +28,9 @@
     "\n",
     "To use the wrapper, you must [deploy a model on AzureML](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-foundation-models?view=azureml-api-2#deploying-foundation-models-to-endpoints-for-inferencing) and obtain the following parameters:\n",
     "\n",
-    "* `endpoint_api_key`: Required - The API key provided by the endpoint\n",
-    "* `endpoint_url`: Required - The REST endpoint url provided by the endpoint\n",
-    "* `deployment_name`: Not required - The deployment name of the model using the endpoint"
+    "* `endpoint_api_key`: The API key provided by the endpoint\n",
+    "* `endpoint_url`: The REST endpoint url provided by the endpoint\n",
+    "* `deployment_name`: The deployment name of the endpoint"
    ]
   },
   {
@@ -39,14 +39,11 @@
    "source": [
     "## Content Formatter\n",
     "\n",
-    "The `content_formatter` parameter is a handler class for transforming the request and response of an AzureML endpoint to match with required schema. Since there are a wide range of models in the model catalog, each of which may process data differently from one another, a `ContentFormatterBase` class is provided to allow users to transform data to their liking. The following content formatters are provided:\n",
+    "The `content_formatter` parameter is a handler class for transforming the request and response of an AzureML endpoint to match with required schema. Since there are a wide range of models in the model catalog, each of which may process data differently from one another, a `ContentFormatterBase` class is provided to allow users to transform data to their liking. Additionally, there are three content formatters already provided:\n",
     "\n",
-    "* `GPT2ContentFormatter`: Formats request and response data for GPT2\n",
-    "* `DollyContentFormatter`: Formats request and response data for the Dolly-v2\n",
+    "* `OSSContentFormatter`: Formats request and response data for models from the Open Source category in the Model Catalog. Note, that not all models in the Open Source category may follow the same schema\n",
+    "* `DollyContentFormatter`: Formats request and response data for the `dolly-v2-12b` model\n",
     "* `HFContentFormatter`: Formats request and response data for text-generation Hugging Face models\n",
-    "* `LLamaContentFormatter`: Formats request and response data for LLaMa2\n",
-    "\n",
-    "*Note: `OSSContentFormatter` is being deprecated and replaced with `GPT2ContentFormatter`. The logic is the same but `GPT2ContentFormatter` is a more suitable name. You can still continue to use `OSSContentFormatter` as the changes are backwards compatibile.*\n",
     "\n",
     "Below is an example using a summarization model from Hugging Face."
    ]
@@ -103,6 +100,7 @@
     "llm = AzureMLOnlineEndpoint(\n",
     "    endpoint_api_key=os.getenv(\"BART_ENDPOINT_API_KEY\"),\n",
     "    endpoint_url=os.getenv(\"BART_ENDPOINT_URL\"),\n",
+    "    deployment_name=\"linydub-bart-large-samsum-3\",\n",
     "    model_kwargs={\"temperature\": 0.8, \"max_new_tokens\": 400},\n",
     "    content_formatter=content_formatter,\n",
     ")\n",
@@ -169,6 +167,7 @@
     "llm = AzureMLOnlineEndpoint(\n",
     "    endpoint_api_key=os.getenv(\"DOLLY_ENDPOINT_API_KEY\"),\n",
     "    endpoint_url=os.getenv(\"DOLLY_ENDPOINT_URL\"),\n",
+    "    deployment_name=\"databricks-dolly-v2-12b-4\",\n",
     "    model_kwargs={\"temperature\": 0.8, \"max_tokens\": 300},\n",
     "    content_formatter=content_formatter,\n",
     ")\n",

docs/extras/integrations/llms/google_vertex_ai_palm.ipynb

@@ -1,15 +1,18 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Google Cloud Platform Vertex AI PaLM \n",
     "\n",
-    "Note: This is seperate from the Google PaLM integration, it exposes [Vertex AI PaLM API](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) on Google Cloud. \n",
+    "Note: This is seperate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there. \n",
     "\n",
-    "By default, Google Cloud [does not use](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance#foundation_model_development) Customer Data to train its foundation models as part of Google Cloud`s AI/ML Privacy Commitment. More details about how Google processes data can also be found in [Google's Customer Data Processing Addendum (CDPA)](https://cloud.google.com/terms/data-processing-addendum).\n",
+    "PaLM API on Vertex AI is a Preview offering, subject to the Pre-GA Offerings Terms of the [GCP Service Specific Terms](https://cloud.google.com/terms/service-terms). \n",
+    "\n",
+    "Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the [launch stage descriptions](https://cloud.google.com/products#product-launch-stages). Further, by using PaLM API on Vertex AI, you agree to the Generative AI Preview [terms and conditions](https://cloud.google.com/trustedtester/aitos) (Preview Terms).\n",
+    "\n",
+    "For PaLM API on Vertex AI, you can process personal data as outlined in the Cloud Data Processing Addendum, subject to applicable restrictions and obligations in the Agreement (as defined in the Preview Terms).\n",
     "\n",
     "To use Vertex AI PaLM you must have the `google-cloud-aiplatform` Python package installed and either:\n",
     "- Have credentials configured for your environment (gcloud, workload identity, etc...)\n",
@@ -98,7 +101,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [

docs/extras/integrations/llms/jsonformer_experimental.ipynb

@@ -216,7 +216,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain_experimental.llms import JsonFormer\n",
+    "from langchain.experimental.llms import JsonFormer\n",
     "\n",
     "json_former = JsonFormer(json_schema=decoder_schema, pipeline=hf_model)"
    ]

docs/extras/integrations/llms/minimax.ipynb

@@ -1,176 +0,0 @@
-{
- "cells": [
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Minimax\n",
-    "\n",
-    "[Minimax](https://api.minimax.chat) is a Chinese startup that provides natural language processing models for companies and individuals.\n",
-    "\n",
-    "This example demonstrates using Langchain to interact with Minimax."
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Setup\n",
-    "\n",
-    "To run this notebook, you'll need a [Minimax account](https://api.minimax.chat), an [API key](https://api.minimax.chat/user-center/basic-information/interface-key), and a [Group ID](https://api.minimax.chat/user-center/basic-information)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Single model call"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 33,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.llms import Minimax"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 34,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Load the model\n",
-    "minimax = Minimax(minimax_api_key=\"YOUR_API_KEY\", minimax_group_id=\"YOUR_GROUP_ID\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "# Prompt the model\n",
-    "minimax(\"What is the difference between panda and bear?\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Chained model calls"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "# get api_key and group_id: https://api.minimax.chat/user-center/basic-information\n",
-    "# We need `MINIMAX_API_KEY` and `MINIMAX_GROUP_ID`\n",
-    "\n",
-    "import os\n",
-    "\n",
-    "os.environ[\"MINIMAX_API_KEY\"] = \"YOUR_API_KEY\"\n",
-    "os.environ[\"MINIMAX_GROUP_ID\"] = \"YOUR_GROUP_ID\""
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "from langchain.llms import Minimax\n",
-    "from langchain import PromptTemplate, LLMChain"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "template = \"\"\"Question: {question}\n",
-    "\n",
-    "Answer: Let's think step by step.\"\"\"\n",
-    "\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "llm = Minimax()"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "outputs": [],
-   "source": [
-    "question = \"What NBA team won the Championship in the year Jay Zhou was born?\"\n",
-    "\n",
-    "llm_chain.run(question)"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.4"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/llms/petals_example.ipynb

@@ -16,9 +16,7 @@
    "metadata": {},
    "source": [
     "## Install petals\n",
-    "The `petals` package is required to use the Petals API. Install `petals` using `pip3 install petals`.\n",
-    "\n",
-    "For Apple Silicon(M1/M2) users please follow this guide [https://github.com/bigscience-workshop/petals/issues/147#issuecomment-1365379642](https://github.com/bigscience-workshop/petals/issues/147#issuecomment-1365379642) to install petals "
+    "The `petals` package is required to use the Petals API. Install `petals` using `pip3 install petals`."
    ]
   },
   {
@@ -64,7 +62,7 @@
    },
    "outputs": [
     {
-     "name": "stdout",
+     "name": "stdin",
      "output_type": "stream",
      "text": [
       " ········\n"

docs/extras/integrations/llms/rellm_experimental.ipynb

@@ -162,7 +162,7 @@
     }
    ],
    "source": [
-    "from langchain_experimental.llms import RELLM\n",
+    "from langchain.experimental.llms import RELLM\n",
     "\n",
     "model = RELLM(pipeline=hf_model, regex=pattern, max_new_tokens=200)\n",
     "\n",

docs/extras/integrations/llms/xinference.ipynb

@@ -1,176 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Xorbits Inference (Xinference)\n",
-    "\n",
-    "[Xinference](https://github.com/xorbitsai/inference) is a powerful and versatile library designed to serve LLMs, \n",
-    "speech recognition models, and multimodal models, even on your laptop. It supports a variety of models compatible with GGML, such as chatglm, baichuan, whisper, vicuna, orca, and many others. This notebook demonstrates how to use Xinference with LangChain."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Installation\n",
-    "\n",
-    "Install `Xinference` through PyPI:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install \"xinference[all]\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Deploy Xinference Locally or in a Distributed Cluster.\n",
-    "\n",
-    "For local deployment, run `xinference`. \n",
-    "\n",
-    "To deploy Xinference in a cluster, first start an Xinference supervisor using the `xinference-supervisor`. You can also use the option -p to specify the port and -H to specify the host. The default port is 9997.\n",
-    "\n",
-    "Then, start the Xinference workers using `xinference-worker` on each server you want to run them on. \n",
-    "\n",
-    "You can consult the README file from [Xinference](https://github.com/xorbitsai/inference) for more information.\n",
-    "## Wrapper\n",
-    "\n",
-    "To use Xinference with LangChain, you need to first launch a model. You can use command line interface (CLI) to do so:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Model uid: 7167b2b0-2a04-11ee-83f0-d29396a3f064\n"
-     ]
-    }
-   ],
-   "source": [
-    "!xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "A model UID is returned for you to use. Now you can use Xinference with LangChain:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "' You can visit the Eiffel Tower, Notre-Dame Cathedral, the Louvre Museum, and many other historical sites in Paris, the capital of France.'"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain.llms import Xinference\n",
-    "\n",
-    "llm = Xinference(\n",
-    "    server_url=\"http://0.0.0.0:9997\",\n",
-    "    model_uid = \"7167b2b0-2a04-11ee-83f0-d29396a3f064\"\n",
-    ")\n",
-    "\n",
-    "llm(\n",
-    "    prompt=\"Q: where can we visit in the capital of France? A:\",\n",
-    "    generate_config={\"max_tokens\": 1024, \"stream\": True},\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Integrate with a LLMChain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "A: You can visit many places in Paris, such as the Eiffel Tower, the Louvre Museum, Notre-Dame Cathedral, the Champs-Elysées, Montmartre, Sacré-Cœur, and the Palace of Versailles.\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain import PromptTemplate, LLMChain\n",
-    "\n",
-    "template = \"Where can we visit in the capital of {country}?\"\n",
-    "\n",
-    "prompt = PromptTemplate(template=template, input_variables=[\"country\"])\n",
-    "\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
-    "\n",
-    "generated = llm_chain.run(country=\"France\")\n",
-    "print(generated)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Lastly, terminate the model when you do not need to use it:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!xinference terminate --model-uid \"7167b2b0-2a04-11ee-83f0-d29396a3f064\""
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "myenv3.9",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.11"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/memory/streamlit_chat_message_history.ipynb

@@ -1,61 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "91c6a7ef",
-   "metadata": {},
-   "source": [
-    "# Streamlit Chat Message History\n",
-    "\n",
-    "This notebook goes over how to use Streamlit to store chat message history. Note, StreamlitChatMessageHistory only works when run in a Streamlit app. For more on Streamlit check out their\n",
-    "[getting started documentation](https://docs.streamlit.io/library/get-started)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d15e3302",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.memory import StreamlitChatMessageHistory\n",
-    "\n",
-    "history = StreamlitChatMessageHistory(\"foo\")\n",
-    "\n",
-    "history.add_user_message(\"hi!\")\n",
-    "history.add_ai_message(\"whats up?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "64fc465e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "history.messages"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "poetry-venv",
-   "language": "python",
-   "name": "poetry-venv"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/providers/agent_with_wandb_tracing.ipynb

@@ -177,7 +177,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.3"
   }
  },
  "nbformat": 4,

docs/extras/integrations/providers/arangodb.mdx

@@ -13,7 +13,7 @@ pip install python-arango
 
 Connect your ArangoDB Database with a Chat Model to get insights on your data. 
 
-See the notebook example [here](/docs/use_cases/graph/graph_arangodb_qa.html).
+See the notebook example [here](/docs/modules/chains/additional/graph_arangodb_qa.html).
 
 ```python
 from arango import ArangoClient

docs/extras/integrations/providers/argilla.mdx

@@ -22,7 +22,7 @@ If you don't you can refer to [Argilla - 🚀 Quickstart](https://docs.argilla.i
 
 ## Tracking
 
-See a [usage example of `ArgillaCallbackHandler`](/docs/integrations/callbacks/argilla.html).
+See a [usage example of `ArgillaCallbackHandler`](/docs/modules/callbacks/integrations/argilla.html).
 
 ```python
 from langchain.callbacks import ArgillaCallbackHandler

docs/extras/integrations/providers/cassandra.mdx

@@ -28,7 +28,7 @@ from langchain.memory import CassandraChatMessageHistory
 
 ## Memory
 
-See a [usage example](/docs/integrations/memory/cassandra_chat_message_history).
+See a [usage example](/docs/modules/memory/integrations/cassandra_chat_message_history).
 
 ```python
 from langchain.memory import CassandraChatMessageHistory

docs/extras/integrations/providers/databricks.ipynb

@@ -166,7 +166,7 @@
    "source": [
     "### SQL Database Agent example\n",
     "\n",
-    "This example demonstrates the use of the [SQL Database Agent](/docs/integrations/toolkits/sql_database.html) for answering questions over a Databricks database."
+    "This example demonstrates the use of the [SQL Database Agent](/docs/modules/agents/toolkits/sql_database.html) for answering questions over a Databricks database."
    ]
   },
   {

docs/extras/integrations/providers/databricks.md

@@ -32,11 +32,11 @@ See [MLflow AI Gateway](/docs/ecosystem/integrations/mlflow_ai_gateway).
 Databricks as an LLM provider
 -----------------------------
 
-The notebook [Wrap Databricks endpoints as LLMs](/docs/integrations/llms/databricks.html) illustrates the method to wrap Databricks endpoints as LLMs in LangChain. It supports two types of endpoints: the serving endpoint, which is recommended for both production and development, and the cluster driver proxy app, which is recommended for interactive development. 
+The notebook [Wrap Databricks endpoints as LLMs](/docs/modules/model_io/models/llms/integrations/databricks.html) illustrates the method to wrap Databricks endpoints as LLMs in LangChain. It supports two types of endpoints: the serving endpoint, which is recommended for both production and development, and the cluster driver proxy app, which is recommended for interactive development. 
 
 Databricks endpoints support Dolly, but are also great for hosting models like MPT-7B or any other models from the Hugging Face ecosystem. Databricks endpoints can also be used with proprietary models like OpenAI to provide a governance layer for enterprises.
 
 Databricks Dolly
 ----------------
 
-Databricks’ Dolly is an instruction-following large language model trained on the Databricks machine learning platform that is licensed for commercial use. The model is available on Hugging Face Hub as databricks/dolly-v2-12b. See the notebook [Hugging Face Hub](/docs/integrations/llms/huggingface_hub.html) for instructions to access it through the Hugging Face Hub integration with LangChain. 
+Databricks’ Dolly is an instruction-following large language model trained on the Databricks machine learning platform that is licensed for commercial use. The model is available on Hugging Face Hub as databricks/dolly-v2-12b. See the notebook [Hugging Face Hub](/docs/modules/model_io/models/llms/integrations/huggingface_hub.html) for instructions to access it through the Hugging Face Hub integration with LangChain. 

docs/extras/integrations/providers/dataforseo.mdx

@@ -13,7 +13,7 @@ This page provides instructions on how to use the DataForSEO search APIs within
 The DataForSEO utility wraps the API. To import this utility, use:
 
 ```python
-from langchain.utilities.dataforseo_api_search import DataForSeoAPIWrapper
+from langchain.utilities import DataForSeoAPIWrapper
 ```
 
 For a detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/dataforseo.ipynb).

docs/extras/integrations/providers/fireworks.md

@@ -1,22 +0,0 @@
-# Fireworks
-
-This page covers how to use the Fireworks models within Langchain.
-
-## Installation and Setup
-
-- To use the Fireworks model, you need to have a Fireworks API key. To generate one, sign up at platform.fireworks.ai
-- Authenticate by setting the FIREWORKS_API_KEY environment variable.
-
-## LLM
-
-Fireworks integrates with Langchain through the LLM module, which allows for standardized usage of any models deployed on the Fireworks models.
-
-In this example, we'll work the llama-v2-13b. 
-```python
-from langchain.llms.fireworks import Fireworks 
-
-llm = Fireworks(model="fireworks-llama-v2-13b-chat", max_tokens=256, temperature=0.4)
-llm("Name 3 sports.")
-```
-
-For a more detailed walkthrough, see [here](/docs/extras/modules/model_io/models/llms/integrations/Fireworks.ipynb).

docs/extras/integrations/providers/minimax.mdx

@@ -1,25 +0,0 @@
-# Minimax
-
->[Minimax](https://api.minimax.chat) is a Chinese startup that provides natural language processing models
-> for companies and individuals.
-
-## Installation and Setup
-Get a [Minimax api key](https://api.minimax.chat/user-center/basic-information/interface-key) and set it as an environment variable (`MINIMAX_API_KEY`)
-Get a [Minimax group id](https://api.minimax.chat/user-center/basic-information) and set it as an environment variable (`MINIMAX_GROUP_ID`)
-
-
-## LLM
-
-There exists a Minimax LLM wrapper, which you can access with
-See a [usage example](/docs/modules/model_io/models/llms/integrations/minimax.html).
-
-```python
-from langchain.llms import Minimax
-```
-
-## Text Embedding Model
-
-There exists a Minimax Embedding model, which you can access with
-```python
-from langchain.embeddings import MiniMaxEmbeddings
-```

docs/extras/integrations/providers/momento.mdx

@@ -51,4 +51,4 @@ Momento can be used as a distributed memory store for LLMs.
 
 ### Chat Message History Memory
 
-See [this notebook](/docs/integrations/memory/momento_chat_message_history.html) for a walkthrough of how to use Momento as a memory store for chat message history.
+See [this notebook](/docs/modules/memory/integrations/momento_chat_message_history.html) for a walkthrough of how to use Momento as a memory store for chat message history.

docs/extras/integrations/providers/motherduck.mdx

@@ -31,7 +31,7 @@ db = SQLDatabase.from_uri(conn_str)
 db_chain = SQLDatabaseChain.from_llm(OpenAI(temperature=0), db, verbose=True)
 ```
 
-From here, see the [SQL Chain](/docs/use_cases/tabular/sqlite.html) documentation on how to use.
+From here, see the [SQL Chain](/docs/modules/chains/popular/sqlite.html) documentation on how to use.
 
 
 ## LLMCache

docs/extras/integrations/providers/openai.mdx

@@ -58,7 +58,7 @@ For a more detailed walkthrough of this, see [this notebook](/docs/modules/data_
 
 ## Chain
 
-See a [usage example](/docs/guides/safety/moderation).
+See a [usage example](/docs/modules/chains/additional/moderation).
 
 ```python
 from langchain.chains import OpenAIModerationChain

docs/extras/integrations/providers/rebuff.ipynb

@@ -177,9 +177,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains import TransformChain, SimpleSequentialChain\n",
-    "from langchain.sql_database import SQLDatabase\n",
-    "from langchain_experimental.sql import SQLDatabaseChain"
+    "from langchain.chains import TransformChain, SQLDatabaseChain, SimpleSequentialChain\n",
+    "from langchain.sql_database import SQLDatabase"
    ]
   },
   {

docs/extras/integrations/providers/redis.mdx

@@ -106,4 +106,4 @@ Redis can be used to persist LLM conversations.
 For a more detailed walkthrough of the `VectorStoreRetrieverMemory` wrapper, see [this notebook](/docs/modules/memory/integrations/vectorstore_retriever_memory.html).
 
 #### Chat Message History Memory
-For a detailed example of Redis to cache conversation message history, see [this notebook](/docs/integrations/memory/redis_chat_message_history.html).
+For a detailed example of Redis to cache conversation message history, see [this notebook](/docs/modules/memory/integrations/redis_chat_message_history.html).

docs/extras/integrations/providers/rockset.mdx

@@ -15,7 +15,7 @@ pip install rockset
 See a [usage example](/docs/integrations/vectorstores/rockset).
 
 ```python
-from langchain.vectorstores import Rockset 
+from langchain.vectorstores import RocksetDB
 ```
 
 ## Document Loader

docs/extras/integrations/providers/sagemaker_tracking.ipynb

@@ -1,916 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "ef3909cf-72ca-4841-85c6-ef4e0eae3aaf",
-   "metadata": {},
-   "source": [
-    "# SageMaker Tracking\n",
-    "\n",
-    "This notebook shows how LangChain Callback can be used to log and track prompts and other LLM hyperparameters into SageMaker Experiments. Here, we use different scenarios to showcase the capability:\n",
-    "* **Scenario 1**: *Single LLM* - A case where a single LLM model is used to generate output based on a given prompt.\n",
-    "* **Scenario 2**: *Sequential Chain* - A case where a sequential chain of two LLM models is used.\n",
-    "* **Scenario 3**: *Agent with Tools (Chain of Thought)* - A case where multiple tools (search and math) are used in addition to an LLM.\n",
-    "\n",
-    "[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a fully managed service that is used to quickly and easily build, train and deploy machine learning (ML) models. \n",
-    "\n",
-    "[Amazon SageMaker Experiments](https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html) is a capability of Amazon SageMaker that lets you organize, track, compare and evaluate ML experiments and model versions.\n",
-    "\n",
-    "In this notebook, we will create a single experiment to log the prompts from each scenario."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "94c22cb4-3b1c-432b-b3be-0235eec79c5c",
-   "metadata": {},
-   "source": [
-    "## Installation and Setup"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2353436d-17fe-4f58-a2f9-c299d56393fd",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "!pip install sagemaker\n",
-    "!pip install openai\n",
-    "!pip install google-search-results"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "65dcf62e-7a38-4119-adb9-d9e884e82499",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "First, setup the required API keys\n",
-    "* OpenAI: https://platform.openai.com/account/api-keys (For OpenAI LLM model)\n",
-    "* Google SERP API: https://serpapi.com/manage-api-key (For Google Search Tool)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "5ec2b898-0cfc-4308-8e86-569cd7b7cf41",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "## Add your API keys below\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"<ADD-KEY-HERE>\"\n",
-    "os.environ[\"SERPAPI_API_KEY\"] = \"<ADD-KEY-HERE>\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "80968ebf-519f-46de-8703-97532ac39e3e",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.llms import OpenAI\n",
-    "from langchain.prompts import PromptTemplate\n",
-    "from langchain.chains import LLMChain, SimpleSequentialChain\n",
-    "from langchain.agents import initialize_agent, load_tools\n",
-    "from langchain.agents import Tool\n",
-    "from langchain.callbacks import SageMakerCallbackHandler\n",
-    "\n",
-    "from sagemaker.analytics import ExperimentAnalytics\n",
-    "from sagemaker.session import Session\n",
-    "from sagemaker.experiments.run import Run"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b67d031f-a01f-4009-ad29-c80ab8ad50ea",
-   "metadata": {},
-   "source": [
-    "## LLM Prompt Tracking"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "da2d70ee-173b-469d-a718-54c33d862844",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "#LLM Hyperparameters\n",
-    "HPARAMS = {\n",
-    "    \"temperature\": 0.1,\n",
-    "    \"model_name\": \"text-davinci-003\",\n",
-    "}\n",
-    "\n",
-    "#Bucket used to save prompt logs (Use `None` is used to save the default bucket or otherwise change it)\n",
-    "BUCKET_NAME = None\n",
-    "\n",
-    "#Experiment name\n",
-    "EXPERIMENT_NAME = \"langchain-sagemaker-tracker\"\n",
-    "\n",
-    "#Create SageMaker Session with the given bucket\n",
-    "session = Session(default_bucket=BUCKET_NAME)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7239a39a-08d8-43cb-8922-81abdd5d9ebf",
-   "metadata": {},
-   "source": [
-    "### Scenario 1 - LLM"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "abc00335-50c8-4119-adb8-4c4ab8522e23",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "RUN_NAME = \"run-scenario-1\"\n",
-    "PROMPT_TEMPLATE = \"tell me a joke about {topic}\"\n",
-    "INPUT_VARIABLES = {\"topic\": \"fish\"}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "4a3a3cbe-db85-4255-8d8b-eaafdca8c6e2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with Run(experiment_name=EXPERIMENT_NAME, run_name=RUN_NAME, sagemaker_session=session) as run:\n",
-    "\n",
-    "    # Create SageMaker Callback\n",
-    "    sagemaker_callback = SageMakerCallbackHandler(run)\n",
-    "\n",
-    "    # Define LLM model with callback\n",
-    "    llm = OpenAI(callbacks=[sagemaker_callback], **HPARAMS)\n",
-    "\n",
-    "    # Create prompt template\n",
-    "    prompt = PromptTemplate.from_template(template=PROMPT_TEMPLATE)\n",
-    "\n",
-    "    # Create LLM Chain\n",
-    "    chain = LLMChain(llm=llm, prompt=prompt, callbacks=[sagemaker_callback])\n",
-    "\n",
-    "    # Run chain\n",
-    "    chain.run(**INPUT_VARIABLES)\n",
-    "\n",
-    "    # Reset the callback\n",
-    "    sagemaker_callback.flush_tracker()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7dc69934-9f42-40b7-9931-36a3371a38da",
-   "metadata": {},
-   "source": [
-    "### Scenario 2 - Sequential Chain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "50b75ef9-9825-4ccc-8414-4cd7525a1b68",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "RUN_NAME = \"run-scenario-2\"\n",
-    "\n",
-    "PROMPT_TEMPLATE_1 = \"\"\"You are a playwright. Given the title of play, it is your job to write a synopsis for that title.\n",
-    "Title: {title}\n",
-    "Playwright: This is a synopsis for the above play:\"\"\"\n",
-    "PROMPT_TEMPLATE_2 = \"\"\"You are a play critic from the New York Times. Given the synopsis of play, it is your job to write a review for that play.\n",
-    "Play Synopsis: {synopsis}\n",
-    "Review from a New York Times play critic of the above play:\"\"\"\n",
-    "\n",
-    "INPUT_VARIABLES = {\n",
-    "    \"input\": \"documentary about good video games that push the boundary of game design\"\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fb7fff5f-e89f-40e2-96b4-3641a0b6e9b4",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with Run(experiment_name=EXPERIMENT_NAME, run_name=RUN_NAME, sagemaker_session=session) as run:\n",
-    "\n",
-    "    # Create SageMaker Callback\n",
-    "    sagemaker_callback = SageMakerCallbackHandler(run)\n",
-    "\n",
-    "    # Create prompt templates for the chain\n",
-    "    prompt_template1 = PromptTemplate.from_template(template=PROMPT_TEMPLATE_1)\n",
-    "    prompt_template2 = PromptTemplate.from_template(template=PROMPT_TEMPLATE_2)\n",
-    "\n",
-    "    # Define LLM model with callback\n",
-    "    llm = OpenAI(callbacks=[sagemaker_callback], **HPARAMS)\n",
-    "\n",
-    "    # Create chain1\n",
-    "    chain1 = LLMChain(llm=llm, prompt=prompt_template1, callbacks=[sagemaker_callback])\n",
-    "\n",
-    "    # Create chain2\n",
-    "    chain2 = LLMChain(llm=llm, prompt=prompt_template2, callbacks=[sagemaker_callback])\n",
-    "\n",
-    "    # Create Sequential chain\n",
-    "    overall_chain = SimpleSequentialChain(chains=[chain1, chain2], callbacks=[sagemaker_callback])\n",
-    "\n",
-    "    # Run overall sequential chain\n",
-    "    overall_chain.run(**INPUT_VARIABLES)\n",
-    "\n",
-    "    # Reset the callback\n",
-    "    sagemaker_callback.flush_tracker()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "6b82bd0e-c626-4797-bb06-c1983f176315",
-   "metadata": {},
-   "source": [
-    "### Scenario 3 - Agent with Tools"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b5066f03-49dc-4868-be8e-d21ce22063fe",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "RUN_NAME = \"run-scenario-3\"\n",
-    "PROMPT_TEMPLATE = \"Who is the oldest person alive? And what is their current age raised to the power of 1.51?\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "98385c42-9e44-4b03-b76d-007cb4797864",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "with Run(experiment_name=EXPERIMENT_NAME, run_name=RUN_NAME, sagemaker_session=session) as run:\n",
-    "\n",
-    "    # Create SageMaker Callback\n",
-    "    sagemaker_callback = SageMakerCallbackHandler(run)\n",
-    "\n",
-    "    # Define LLM model with callback\n",
-    "    llm = OpenAI(callbacks=[sagemaker_callback], **HPARAMS)\n",
-    "\n",
-    "    # Define tools\n",
-    "    tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm, callbacks=[sagemaker_callback])\n",
-    "\n",
-    "    # Initialize agent with all the tools\n",
-    "    agent = initialize_agent(tools, llm, agent=\"zero-shot-react-description\", callbacks=[sagemaker_callback])\n",
-    "\n",
-    "    # Run agent\n",
-    "    agent.run(input=PROMPT_TEMPLATE)\n",
-    "\n",
-    "    # Reset the callback\n",
-    "    sagemaker_callback.flush_tracker()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c306a1c9-99f8-476d-96db-347746f5cfe0",
-   "metadata": {
-    "tags": []
-   },
-   "source": [
-    "## Load Log Data\n",
-    "\n",
-    "Once the prompts are logged, we can easily load and convert them to Pandas DataFrame as follows."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "ec7b4af2-e01d-4f6c-9de5-70d2b4acb9e6",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "#Load\n",
-    "logs = ExperimentAnalytics(experiment_name=EXPERIMENT_NAME)\n",
-    "\n",
-    "#Convert as pandas dataframe\n",
-    "df = logs.dataframe(force_refresh=True)\n",
-    "\n",
-    "print(df.shape)\n",
-    "df.head()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "29991c75-f9cf-4c36-abfd-903c09fb170d",
-   "metadata": {},
-   "source": [
-    "As can be seen above, there are three runs (rows) in the experiment corresponding to each scenario. Each run logs the prompts and related LLM settings/hyperparameters as json and are saved in s3 bucket. Feel free to load and explore the log data from each json path."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "61a695d6-0aef-4284-9e12-eea8bc143dbd",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "availableInstances": [
-   {
-    "_defaultOrder": 0,
-    "_isFastLaunch": true,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 4,
-    "name": "ml.t3.medium",
-    "vcpuNum": 2
-   },
-   {
-    "_defaultOrder": 1,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 8,
-    "name": "ml.t3.large",
-    "vcpuNum": 2
-   },
-   {
-    "_defaultOrder": 2,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.t3.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 3,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.t3.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 4,
-    "_isFastLaunch": true,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 8,
-    "name": "ml.m5.large",
-    "vcpuNum": 2
-   },
-   {
-    "_defaultOrder": 5,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.m5.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 6,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.m5.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 7,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 64,
-    "name": "ml.m5.4xlarge",
-    "vcpuNum": 16
-   },
-   {
-    "_defaultOrder": 8,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 128,
-    "name": "ml.m5.8xlarge",
-    "vcpuNum": 32
-   },
-   {
-    "_defaultOrder": 9,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 192,
-    "name": "ml.m5.12xlarge",
-    "vcpuNum": 48
-   },
-   {
-    "_defaultOrder": 10,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 256,
-    "name": "ml.m5.16xlarge",
-    "vcpuNum": 64
-   },
-   {
-    "_defaultOrder": 11,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 384,
-    "name": "ml.m5.24xlarge",
-    "vcpuNum": 96
-   },
-   {
-    "_defaultOrder": 12,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 8,
-    "name": "ml.m5d.large",
-    "vcpuNum": 2
-   },
-   {
-    "_defaultOrder": 13,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.m5d.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 14,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.m5d.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 15,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 64,
-    "name": "ml.m5d.4xlarge",
-    "vcpuNum": 16
-   },
-   {
-    "_defaultOrder": 16,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 128,
-    "name": "ml.m5d.8xlarge",
-    "vcpuNum": 32
-   },
-   {
-    "_defaultOrder": 17,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 192,
-    "name": "ml.m5d.12xlarge",
-    "vcpuNum": 48
-   },
-   {
-    "_defaultOrder": 18,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 256,
-    "name": "ml.m5d.16xlarge",
-    "vcpuNum": 64
-   },
-   {
-    "_defaultOrder": 19,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 384,
-    "name": "ml.m5d.24xlarge",
-    "vcpuNum": 96
-   },
-   {
-    "_defaultOrder": 20,
-    "_isFastLaunch": false,
-    "category": "General purpose",
-    "gpuNum": 0,
-    "hideHardwareSpecs": true,
-    "memoryGiB": 0,
-    "name": "ml.geospatial.interactive",
-    "supportedImageNames": [
-     "sagemaker-geospatial-v1-0"
-    ],
-    "vcpuNum": 0
-   },
-   {
-    "_defaultOrder": 21,
-    "_isFastLaunch": true,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 4,
-    "name": "ml.c5.large",
-    "vcpuNum": 2
-   },
-   {
-    "_defaultOrder": 22,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 8,
-    "name": "ml.c5.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 23,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.c5.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 24,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.c5.4xlarge",
-    "vcpuNum": 16
-   },
-   {
-    "_defaultOrder": 25,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 72,
-    "name": "ml.c5.9xlarge",
-    "vcpuNum": 36
-   },
-   {
-    "_defaultOrder": 26,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 96,
-    "name": "ml.c5.12xlarge",
-    "vcpuNum": 48
-   },
-   {
-    "_defaultOrder": 27,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 144,
-    "name": "ml.c5.18xlarge",
-    "vcpuNum": 72
-   },
-   {
-    "_defaultOrder": 28,
-    "_isFastLaunch": false,
-    "category": "Compute optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 192,
-    "name": "ml.c5.24xlarge",
-    "vcpuNum": 96
-   },
-   {
-    "_defaultOrder": 29,
-    "_isFastLaunch": true,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.g4dn.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 30,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.g4dn.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 31,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 64,
-    "name": "ml.g4dn.4xlarge",
-    "vcpuNum": 16
-   },
-   {
-    "_defaultOrder": 32,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 128,
-    "name": "ml.g4dn.8xlarge",
-    "vcpuNum": 32
-   },
-   {
-    "_defaultOrder": 33,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 4,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 192,
-    "name": "ml.g4dn.12xlarge",
-    "vcpuNum": 48
-   },
-   {
-    "_defaultOrder": 34,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 256,
-    "name": "ml.g4dn.16xlarge",
-    "vcpuNum": 64
-   },
-   {
-    "_defaultOrder": 35,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 61,
-    "name": "ml.p3.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 36,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 4,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 244,
-    "name": "ml.p3.8xlarge",
-    "vcpuNum": 32
-   },
-   {
-    "_defaultOrder": 37,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 8,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 488,
-    "name": "ml.p3.16xlarge",
-    "vcpuNum": 64
-   },
-   {
-    "_defaultOrder": 38,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 8,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 768,
-    "name": "ml.p3dn.24xlarge",
-    "vcpuNum": 96
-   },
-   {
-    "_defaultOrder": 39,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.r5.large",
-    "vcpuNum": 2
-   },
-   {
-    "_defaultOrder": 40,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.r5.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 41,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 64,
-    "name": "ml.r5.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 42,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 128,
-    "name": "ml.r5.4xlarge",
-    "vcpuNum": 16
-   },
-   {
-    "_defaultOrder": 43,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 256,
-    "name": "ml.r5.8xlarge",
-    "vcpuNum": 32
-   },
-   {
-    "_defaultOrder": 44,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 384,
-    "name": "ml.r5.12xlarge",
-    "vcpuNum": 48
-   },
-   {
-    "_defaultOrder": 45,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 512,
-    "name": "ml.r5.16xlarge",
-    "vcpuNum": 64
-   },
-   {
-    "_defaultOrder": 46,
-    "_isFastLaunch": false,
-    "category": "Memory Optimized",
-    "gpuNum": 0,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 768,
-    "name": "ml.r5.24xlarge",
-    "vcpuNum": 96
-   },
-   {
-    "_defaultOrder": 47,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 16,
-    "name": "ml.g5.xlarge",
-    "vcpuNum": 4
-   },
-   {
-    "_defaultOrder": 48,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 32,
-    "name": "ml.g5.2xlarge",
-    "vcpuNum": 8
-   },
-   {
-    "_defaultOrder": 49,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 64,
-    "name": "ml.g5.4xlarge",
-    "vcpuNum": 16
-   },
-   {
-    "_defaultOrder": 50,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 128,
-    "name": "ml.g5.8xlarge",
-    "vcpuNum": 32
-   },
-   {
-    "_defaultOrder": 51,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 1,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 256,
-    "name": "ml.g5.16xlarge",
-    "vcpuNum": 64
-   },
-   {
-    "_defaultOrder": 52,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 4,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 192,
-    "name": "ml.g5.12xlarge",
-    "vcpuNum": 48
-   },
-   {
-    "_defaultOrder": 53,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 4,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 384,
-    "name": "ml.g5.24xlarge",
-    "vcpuNum": 96
-   },
-   {
-    "_defaultOrder": 54,
-    "_isFastLaunch": false,
-    "category": "Accelerated computing",
-    "gpuNum": 8,
-    "hideHardwareSpecs": false,
-    "memoryGiB": 768,
-    "name": "ml.g5.48xlarge",
-    "vcpuNum": 192
-   }
-  ],
-  "instance_type": "ml.t3.large",
-  "kernelspec": {
-   "display_name": "conda_pytorch_p310",
-   "language": "python",
-   "name": "conda_pytorch_p310"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.10"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/providers/spacy.mdx

@@ -16,5 +16,5 @@ pip install spacy
 See a [usage example](/docs/modules/data_connection/document_transformers/text_splitters/split_by_token.html#spacy).
 
 ```python
-from langchain.text_splitter import SpacyTextSplitter
+from langchain.llms import SpacyTextSplitter
 ```

docs/extras/integrations/providers/unstructured.mdx

@@ -11,9 +11,7 @@ ecosystem within LangChain.
 If you are using a loader that runs locally, use the following steps to get `unstructured` and
 its dependencies running locally.
 
-- Install the Python SDK with `pip install unstructured`.
-    - You can install document specific dependencies with extras, i.e. `pip install "unstructured[docx]"`.
-    - To install the dependencies for all document types, use `pip install "unstructured[all-docs]"`.
+- Install the Python SDK with `pip install "unstructured[local-inference]"`
 - Install the following system dependencies if they are not already available on your system.
   Depending on what document types you're parsing, you may not need all of these.
     - `libmagic-dev` (filetype detection)

docs/extras/integrations/providers/xinference.mdx

@@ -1,102 +0,0 @@
-# Xorbits Inference (Xinference)
-
-This page demonstrates how to use [Xinference](https://github.com/xorbitsai/inference)
-with LangChain.
-
-`Xinference` is a powerful and versatile library designed to serve LLMs, 
-speech recognition models, and multimodal models, even on your laptop. 
-With Xorbits Inference, you can effortlessly deploy and serve your or 
-state-of-the-art built-in models using just a single command.
-
-## Installation and Setup
-
-Xinference can be installed via pip from PyPI: 
-
-```bash
-pip install "xinference[all]"
-```
-
-## LLM
-
-Xinference supports various models compatible with GGML, including chatglm, baichuan, whisper, 
-vicuna, and orca. To view the builtin models, run the command:
-
-```bash
-xinference list --all
-```
-
-
-### Wrapper for Xinference
-
-You can start a local instance of Xinference by running:
-
-```bash
-xinference
-```
-
-You can also deploy Xinference in a distributed cluster. To do so, first start an Xinference supervisor
-on the server you want to run it:
-
-```bash
-xinference-supervisor -H "${supervisor_host}"
-```
-
-
-Then, start the Xinference workers on each of the other servers where you want to run them on:
-
-```bash
-xinference-worker -e "http://${supervisor_host}:9997"
-```
-
-You can also start a local instance of Xinference by running:
-
-```bash
-xinference
-```
-
-Once Xinference is running, an endpoint will be accessible for model management via CLI or 
-Xinference client. 
-
-For local deployment, the endpoint will be http://localhost:9997. 
-
-
-For cluster deployment, the endpoint will be http://${supervisor_host}:9997.
-
-
-Then, you need to launch a model. You can specify the model names and other attributes 
-including model_size_in_billions and quantization. You can use command line interface (CLI) to 
-do it. For example, 
-
-```bash
-xinference launch -n orca -s 3 -q q4_0
-```
-
-A model uid will be returned.
-
-Example usage:
-
-```python
-from langchain.llms import Xinference
-
-llm = Xinference(
-    server_url="http://0.0.0.0:9997",
-    model_uid = {model_uid} # replace model_uid with the model UID return from launching the model
-)
-
-llm(
-    prompt="Q: where can we visit in the capital of France? A:",
-    generate_config={"max_tokens": 1024, "stream": True},
-)
-
-```
-
-### Usage
-
-For more information and detailed examples, refer to the
-[example notebook for xinference](../modules/models/llms/integrations/xinference.ipynb)
-
-### Embeddings
-
-Xinference also supports embedding queries and documents. See
-[example notebook for xinference embeddings](../modules/data_connection/text_embedding/integrations/xinference.ipynb) 
-for a more detailed demo.

docs/extras/integrations/text_embedding/Awa.ipynb

@@ -1,109 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "b14a24db",
-   "metadata": {},
-   "source": [
-    "# AwaEmbedding\n",
-    "\n",
-    "This notebook explains how to use AwaEmbedding, which is included in [awadb](https://github.com/awa-ai/awadb), to embedding texts in langchain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "0ab948fc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# pip install awadb"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "67c637ca",
-   "metadata": {},
-   "source": [
-    "## import the library"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "5709b030",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.embeddings import AwaEmbeddings"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "1756b1ba",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "Embedding = AwaEmbeddings()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4a2a098d",
-   "metadata": {},
-   "source": [
-    "# Set embedding model\n",
-    "Users can use `Embedding.set_model()` to specify the embedding model. \\\n",
-    "The input of this function is a string which represents the model's name. \\\n",
-    "The list of currently supported models can be obtained [here](https://github.com/awa-ai/awadb) \\ \\ \n",
-    "\n",
-    "The **default model** is `all-mpnet-base-v2`, it can be used without setting."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "584b9af5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "text = \"our embedding test\"\n",
-    "\n",
-    "Embedding.set_model(\"all-mpnet-base-v2\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "be18b873",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "res_query = Embedding.embed_query(\"The test information\")\n",
-    "res_document = Embedding.embed_documents([\"test1\", \"another test\"])"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/integrations/text_embedding/google_vertex_ai_palm.ipynb

@@ -1,15 +1,18 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Google Cloud Platform Vertex AI PaLM \n",
     "\n",
-    "Note: This is seperate from the Google PaLM integration, it exposes [Vertex AI PaLM API](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) on Google Cloud. \n",
+    "Note: This is seperate from the Google PaLM integration. Google has chosen to offer an enterprise version of PaLM through GCP, and this supports the models made available through there. \n",
     "\n",
-    "By default, Google Cloud [does not use](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance#foundation_model_development) Customer Data to train its foundation models as part of Google Cloud`s AI/ML Privacy Commitment. More details about how Google processes data can also be found in [Google's Customer Data Processing Addendum (CDPA)](https://cloud.google.com/terms/data-processing-addendum).\n",
+    "PaLM API on Vertex AI is a Preview offering, subject to the Pre-GA Offerings Terms of the [GCP Service Specific Terms](https://cloud.google.com/terms/service-terms). \n",
+    "\n",
+    "Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the [launch stage descriptions](https://cloud.google.com/products#product-launch-stages). Further, by using PaLM API on Vertex AI, you agree to the Generative AI Preview [terms and conditions](https://cloud.google.com/trustedtester/aitos) (Preview Terms).\n",
+    "\n",
+    "For PaLM API on Vertex AI, you can process personal data as outlined in the Cloud Data Processing Addendum, subject to applicable restrictions and obligations in the Agreement (as defined in the Preview Terms).\n",
     "\n",
     "To use Vertex AI PaLM you must have the `google-cloud-aiplatform` Python package installed and either:\n",
     "- Have credentials configured for your environment (gcloud, workload identity, etc...)\n",

docs/extras/integrations/text_embedding/localai.ipynb

@@ -77,7 +77,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.embeddings import LocalAIEmbeddings"
+    "from langchain.embeddings.openai import LocalAIEmbeddings"
    ]
   },
   {

docs/extras/integrations/text_embedding/xinference.ipynb

@@ -1,144 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Xorbits inference (Xinference)\n",
-    "\n",
-    "This notebook goes over how to use Xinference embeddings within LangChain"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Installation\n",
-    "\n",
-    "Install `Xinference` through PyPI:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install \"xinference[all]\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Deploy Xinference Locally or in a Distributed Cluster.\n",
-    "\n",
-    "For local deployment, run `xinference`. \n",
-    "\n",
-    "To deploy Xinference in a cluster, first start an Xinference supervisor using the `xinference-supervisor`. You can also use the option -p to specify the port and -H to specify the host. The default port is 9997.\n",
-    "\n",
-    "Then, start the Xinference workers using `xinference-worker` on each server you want to run them on. \n",
-    "\n",
-    "You can consult the README file from [Xinference](https://github.com/xorbitsai/inference) for more information.\n",
-    "\n",
-    "## Wrapper\n",
-    "\n",
-    "To use Xinference with LangChain, you need to first launch a model. You can use command line interface (CLI) to do so:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Model uid: 915845ee-2a04-11ee-8ed4-d29396a3f064\n"
-     ]
-    }
-   ],
-   "source": [
-    "!xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "A model UID is returned for you to use. Now you can use Xinference embeddings with LangChain:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.embeddings import XinferenceEmbeddings\n",
-    "\n",
-    "xinference = XinferenceEmbeddings(\n",
-    "    server_url=\"http://0.0.0.0:9997\",\n",
-    "    model_uid = \"915845ee-2a04-11ee-8ed4-d29396a3f064\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "query_result = xinference.embed_query(\"This is a test query\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "doc_result = xinference.embed_documents([\"text A\", \"text B\"])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Lastly, terminate the model when you do not need to use it:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!xinference terminate --model-uid \"915845ee-2a04-11ee-8ed4-d29396a3f064\""
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "base",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.11"
-  },
-  "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/toolkits/github.ipynb

@@ -4,41 +4,18 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Github Toolkit\n",
+    "# GitHub\n",
     "\n",
-    "The Github toolkit contains tools that enable an LLM agent to interact with a github repository. The tools are a wrapper for the [PyGitHub](https://github.com/PyGithub/PyGithub) library. \n",
+    "This notebook goes over how to use the GitHub tool.\n",
+    "The GitHub tool allows agents to interact with a given GitHub repository. It implements CRUD operations for modifying files and can read/comment on Issues. The tool wraps the [PyGitHub](https://github.com/PyGithub/PyGithub) library.\n",
     "\n",
-    "## Quickstart\n",
-    "1. Install the pygithub library\n",
-    "2. Create a Github app\n",
-    "3. Set your environmental variables\n",
-    "4. Pass the tools to your agent with `toolkit.get_tools()`\n",
-    "\n",
-    "Each of these steps will be explained in greate detail below.\n",
-    "\n",
-    "1. **Get Issues**- fetches issues from the repository.\n",
-    "\n",
-    "2. **Get Issue**- feteches details about a specific issue.\n",
-    "\n",
-    "3. **Comment on Issue**- posts a comment on a specific issue.\n",
-    "\n",
-    "4. **Create Pull Request**- creates a pull request from the bot's working branch to the base branch.\n",
-    "\n",
-    "5. **Create File**- creates a new file in the repository.\n",
-    "\n",
-    "6. **Read File**- reads a file from the repository.\n",
-    "\n",
-    "7. **Update File**- updates a file in the repository.\n",
-    "\n",
-    "8. **Delete File**- deletes a file from the repository.\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 1. Install the pygithub library"
+    "In order to interact with the GitHub API you must create a [GitHub app](https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/about-creating-github-apps). Next, you must set the following environment variables:\n",
+    "```\n",
+    "GITHUB_APP_ID\n",
+    "GITHUB_APP_PRIVATE_KEY\n",
+    "GITHUB_REPOSITORY\n",
+    "GITHUB_BRANCH\n",
+    "```"
    ]
   },
   {
@@ -54,44 +31,9 @@
     "%pip install pygithub"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## 2. Create a Github App\n",
-    "\n",
-    "[Follow the instructions here](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app) to create and register a Github app. Make sure your app has the following [repository permissions:](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps?apiVersion=2022-11-28)\n",
-    "* Commit statuses (read only)\n",
-    "* Contents (read and write)\n",
-    "* Issues (read and write)\n",
-    "* Metadata (read only)\n",
-    "* Pull requests (read and write)\n",
-    "\n",
-    "\n",
-    "\n",
-    "Once the app has been registered, add it to the repository you wish the bot to act upon.\n",
-    "\n",
-    "## 3. Set Environmental Variables\n",
-    "\n",
-    "Before initializing your agent, the following environmental variables need to be set:\n",
-    "\n",
-    "* **GITHUB_APP_ID**- A six digit number found in your app's general settings\n",
-    "* **GITHUB_APP_PRIVATE_KEY**- The location of your app's private key .pem file\n",
-    "* **GITHUB_REPOSITORY**- The name of the Github repository you want your bot to act upon. Must follow the format {username}/{repo-name}. Make sure the app has been added to this repository first!\n",
-    "* **GITHUB_BRANCH**- The branch where the bot will make its commits. Defaults to 'master.'\n",
-    "* **GITHUB_BASE_BRANCH**- The base branch of your repo, usually either 'main' or 'master.' This is where pull requests will base from. Defaults to 'master.'\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Example Usage- Simple Agent"
-   ]
-  },
   {
    "cell_type": "code",
-   "execution_count": 47,
+   "execution_count": 2,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -105,25 +47,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 53,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Set your environment variables using os.environ\n",
-    "os.environ[\"GITHUB_APP_ID\"] = \"123456\"\n",
-    "os.environ[\"GITHUB_APP_PRIVATE_KEY\"] = \"path/to/your/private-key.pem\"\n",
-    "os.environ[\"GITHUB_REPOSITORY\"] = \"username/repo-name\"\n",
-    "os.environ[\"GITHUB_BRANCH\"] = \"bot-branch-name\"\n",
-    "os.environ[\"GITHUB_BASE_BRANCH\"] = \"main\"\n",
-    "\n",
-    "# This example also requires an OpenAI API key\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
-    "\n"
+    "os.environ[\"GITHUB_APP_ID\"] = \"your-github-app-id\"\n",
+    "os.environ[\"GITHUB_APP_PRIVATE_KEY\"] = \"/path/to/your/private/key\"\n",
+    "os.environ[\"GITHUB_REPOSITORY\"] = \"user/repo\"\n",
+    "os.environ[\"GITHUB_BRANCH\"] = \"branch-name\"\n",
+    "os.environ[\"OPENAI_API_KEY\"] = \"your-openai-api-key\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 54,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -137,7 +74,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 55,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [
     {
@@ -147,46 +84,42 @@
       "\n",
       "\n",
       "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m I need to figure out what issues need to be completed.\n",
+      "\u001b[32;1m\u001b[1;3m I need to figure out what issues need to be completed and how to complete them.\n",
       "Action: Get Issues\n",
       "Action Input: N/A\u001b[0m\n",
       "Observation: \u001b[36;1m\u001b[1;3mFound 1 issues:\n",
-      "[{'title': 'Update README file', 'number': 9}]\u001b[0m\n",
+      "[{'title': 'Change the main script to print Hello AI!', 'number': 1}]\u001b[0m\n",
       "Thought:\u001b[32;1m\u001b[1;3m I need to get more information about this issue.\n",
       "Action: Get Issue\n",
-      "Action Input: 9\u001b[0m\n",
-      "Observation: \u001b[33;1m\u001b[1;3m{\"title\": \"Update README file\", \"body\": \"Find what the most popular frontend framework is right now and add a short blurb to the readme.md file about how this website will take advantage of it.\", \"comments\": \"[]\"}\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to update the README file.\n",
-      "Action: Create File\n",
-      "Action Input: README.md\u001b[0m\n",
-      "Observation: \u001b[33;1m\u001b[1;3mFile already exists at README.md. Use update_file instead\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to update the existing README file.\n",
+      "Action Input: 1\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3m{'title': 'Change the main script to print Hello AI!', 'body': None, 'comments': '[]'}\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to update the main script to print Hello AI!\n",
       "Action: Update File\n",
-      "Action Input: README.md\n",
+      "Action Input: main.py\n",
       "OLD <<<<\n",
-      "This is a sample website\n",
+      "print(\"Hello World!\")\n",
       ">>>> OLD\n",
       "NEW <<<<\n",
-      "This is a sample website that uses the most popular frontend framework.\n",
+      "print(\"Hello AI!\")\n",
       ">>>> NEW\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mFile content was not updated because old content was not found.It may be helpful to use the read_file action to get the current file contents.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to get the current file contents.\n",
+      "Observation: \u001b[38;5;200m\u001b[1;3mFile content was not updated because the old content was not found. It may be helpful to use the read_file action to get the current file contents.\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to read the current file contents.\n",
       "Action: Read File\n",
-      "Action Input: README.md\u001b[0m\n",
-      "Observation: \u001b[38;5;200m\u001b[1;3mThis is my awesome website!\n",
+      "Action Input: main.py\u001b[0m\n",
+      "Observation: \u001b[33;1m\u001b[1;3mprint(\"Hello world!\")\n",
       "\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I need to update the README file with the new content.\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I need to update the main script to print Hello AI!\n",
       "Action: Update File\n",
-      "Action Input: README.md\n",
+      "Action Input: main.py\n",
       "OLD <<<<\n",
-      "This is my awesome website!\n",
+      "print(\"Hello world!\")\n",
       ">>>> OLD\n",
       "NEW <<<<\n",
-      "This is my awesome website that uses the most popular frontend framework.\n",
+      "print(\"Hello AI!\")\n",
       ">>>> NEW\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mUpdated file README.md\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
-      "Final Answer: The README.md file has been updated with the new content.\u001b[0m\n",
+      "Observation: \u001b[38;5;200m\u001b[1;3mUpdated file main.py\u001b[0m\n",
+      "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer\n",
+      "Final Answer: The main script has been updated to print \"Hello AI!\"\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -194,10 +127,10 @@
     {
      "data": {
       "text/plain": [
-       "'The README.md file has been updated with the new content.'"
+       "'The main script has been updated to print \"Hello AI!\"'"
       ]
      },
-     "execution_count": 55,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -207,161 +140,11 @@
     "    \"You have the software engineering capabilities of a Google Principle engineer. You are tasked with completing issues on a github repository. Please look at the existing issues and complete them.\"\n",
     ")"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Example Usage- Advanced Agent\n",
-    "\n",
-    "If your agent does not need to use all 8 tools, you can build tools individually to use. For this example, we'll make an agent that does not use the create_file, delete_file or create_pull_request tools, but can also use duckduckgo-search."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "scrolled": true
-   },
-   "outputs": [],
-   "source": [
-    "%pip install duckduckgo-search"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 72,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.tools.github.tool import GitHubAction\n",
-    "from langchain.tools import DuckDuckGoSearchRun\n",
-    "from langchain.agents import Tool\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "tools = []\n",
-    "unwanted_tools = ['Get Issue','Delete File', 'Create File', 'Create Pull Request']\n",
-    "\n",
-    "for tool in toolkit.get_tools():\n",
-    "    if tool.name not in unwanted_tools:\n",
-    "        tools.append(tool)\n",
-    "tools+= [\n",
-    "    Tool(\n",
-    "        name = \"Search\",\n",
-    "        func = DuckDuckGoSearchRun().run,\n",
-    "        description = \"useful for when you need to search the web\"\n",
-    "    )]\n",
-    "        \n",
-    "agent = initialize_agent(\n",
-    "    tools = tools,\n",
-    "    llm = ChatOpenAI(temperature=0.1),\n",
-    "    agent = AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
-    "    verbose = True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Finally let's build a prompt and test it out!"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 73,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mTo complete this issue, I need to find the most popular frontend framework and add a blurb about how this website will utilize it to the readme.md file. I should start by researching the most popular frontend frameworks and then update the readme file accordingly. I will use the \"Search\" tool to research the most popular frontend framework.\n",
-      "\n",
-      "Action: Search\n",
-      "Action Input: \"most popular frontend framework\"\u001b[0m\n",
-      "Observation: \u001b[33;1m\u001b[1;3mAlex Ivanovs February 25, 2023 Table of Contents What are the current Front-end trends? Top Front-end Frameworks for 2023 #1 - React #2 - Angular #3 - Vue #4 - Svelte #5 - Preact #6 - Ember #7 - Solid #8 - Lit #9 - Alpine #10 - Stencil #11 - Qwik Front-end Frameworks: A Summary Top 6 Frontend Frameworks To Use in 2022 by Nwose Lotanna Victor August 26, 2022 Web 0 Comments This post reveals the top six frontend libraries to use in 2022. The list is fresh and very different from the previous years. State of JS Though React is the most popular framework for frontend development, it also has some shortcomings. Due to its limitations, the idea was to design a small-size framework that will offer the same features as React. This is how a tiny version of React — Preact — appeared. Top 10 Popular Frontend Frameworks to Use in 2023 Sep 26, 2022 10 min Сontents 1. What is a framework? 2. Front-end frameworks vs backend frameworks 3. The best front-end frameworks in 2023 React Vue.js Angular Svelte JQuery Ember Backbone Semantic UI 4. Final words Technostacks Jan 11 2023 Top Frontend Frameworks of 2023 for Web Development Developing what the users see on their screens is the role of a front-end web developer. Unarguably, front-end developers worldwide are trying to use the best front-end frameworks to provide the best user experience.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mBased on my research, the most popular frontend framework right now is React. I will now update the readme.md file to include a blurb about how this website will take advantage of React.\n",
-      "\n",
-      "Action: Update File\n",
-      "Action Input:\n",
-      "README.md\n",
-      "OLD <<<<\n",
-      "This is the readme file for the website.\n",
-      ">>>> OLD\n",
-      "NEW <<<<\n",
-      "This is the readme file for the website.\n",
-      "\n",
-      "This website takes advantage of the React framework, which allows for efficient and reusable UI components. With React, we can easily manage the state of our application and create interactive user interfaces. It provides a smooth and seamless user experience, making this website highly responsive and dynamic.\n",
-      ">>>> NEW\n",
-      "\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mFile content was not updated because old content was not found.It may be helpful to use the read_file action to get the current file contents.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI need to first read the contents of the README.md file to get the current content. Then I can update the file with the new content.\n",
-      "\n",
-      "Action: Read File\n",
-      "Action Input: README.md\u001b[0m\n",
-      "Observation: \u001b[38;5;200m\u001b[1;3mThis is my awesome website that uses the most popular frontend framework.\n",
-      "\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mThe current content of the README.md file is \"This is my awesome website that uses the most popular frontend framework.\" I can now update the file with the new content.\n",
-      "\n",
-      "Action: Update File\n",
-      "Action Input:\n",
-      "README.md\n",
-      "OLD <<<<\n",
-      "This is my awesome website that uses the most popular frontend framework.\n",
-      ">>>> OLD\n",
-      "NEW <<<<\n",
-      "This is my awesome website that uses the most popular frontend framework.\n",
-      "\n",
-      "This website takes advantage of the React framework, which allows for efficient and reusable UI components. With React, we can easily manage the state of our application and create interactive user interfaces. It provides a smooth and seamless user experience, making this website highly responsive and dynamic.\n",
-      ">>>> NEW\n",
-      "\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3mUpdated file README.md\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI have successfully updated the README.md file with the blurb about how this website will take advantage of the React framework.\n",
-      "\n",
-      "Final Answer: The most popular frontend framework right now is React. This website takes advantage of React to create efficient and reusable UI components, manage application state, and provide a smooth and seamless user experience.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'The most popular frontend framework right now is React. This website takes advantage of React to create efficient and reusable UI components, manage application state, and provide a smooth and seamless user experience.'"
-      ]
-     },
-     "execution_count": 73,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# The GitHubAPIWrapper can be used outside of an agent, too\n",
-    "# This gets the info about issue number 9, since we want to\n",
-    "# force the agent to address this specific issue.\n",
-    "\n",
-    "issue = github.get_issue(9)\n",
-    "\n",
-    "prompt = f\"\"\"\n",
-    "You are a seinor frontend developer who is experienced in HTML, CSS, and JS- especially React.\n",
-    "You have been assigned the below issue. Complete it to the best of your ability.\n",
-    "Remember to first make a plan and pay attention to details like file names and commonsense.\n",
-    "Then execute the plan and use tools appropriately.\n",
-    "Finally, make a pull request to merge your changes.\n",
-    "Issue: {issue[\"title\"]}\n",
-    "Issue Description: {issue['body']}\n",
-    "Comments: {issue['comments']}\"\"\"\n",
-    "\n",
-    "agent.run(prompt)\n"
-   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
@@ -375,8 +158,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.16"
-  }
+   "version": "3.11.4"
+  },
+  "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 2

docs/extras/integrations/toolkits/openapi_nla.ipynb

@@ -9,7 +9,7 @@
     "\n",
     "Natural Language API Toolkits (NLAToolkits) permit LangChain Agents to efficiently plan and combine calls across endpoints. This notebook demonstrates a sample composition of the Speak, Klarna, and Spoonacluar APIs.\n",
     "\n",
-    "For a detailed walkthrough of the OpenAPI chains wrapped within the NLAToolkit, see the [OpenAPI Operation Chain](/docs/use_cases/apis/openapi.html) notebook.\n",
+    "For a detailed walkthrough of the OpenAPI chains wrapped within the NLAToolkit, see the [OpenAPI Operation Chain](/docs/modules/chains/additional/openapi.html) notebook.\n",
     "\n",
     "### First, import dependencies and load the LLM"
    ]

docs/extras/integrations/toolkits/spark_sql.ipynb

@@ -6,7 +6,7 @@
    "source": [
     "# Spark SQL Agent\n",
     "\n",
-    "This notebook shows how to use agents to interact with a Spark SQL. Similar to [SQL Database Agent](https://python.langchain.com/docs/integrations/toolkits/sql_database), it is designed to address general inquiries about Spark SQL and facilitate error recovery.\n",
+    "This notebook shows how to use agents to interact with a Spark SQL. Similar to [SQL Database Agent](https://python.langchain.com/en/latest/modules/agents/toolkits/examples/sql_database.html), it is designed to address general inquiries about Spark SQL and facilitate error recovery.\n",
     "\n",
     "**NOTE: Note that, as this agent is in active development, all answers might not be correct. Additionally, it is not guaranteed that the agent won't perform DML statements on your Spark cluster given certain questions. Be careful running it on sensitive data!**"
    ]

docs/extras/integrations/toolkits/sql_database.ipynb

@@ -8,7 +8,7 @@
    "source": [
     "# SQL Database Agent\n",
     "\n",
-    "This notebook showcases an agent designed to interact with a sql databases. The agent builds off of [SQLDatabaseChain](https://python.langchain.com/docs/use_cases/tabular/sqlite) and is designed to answer more general questions about a database, as well as recover from errors.\n",
+    "This notebook showcases an agent designed to interact with a sql databases. The agent builds off of [SQLDatabaseChain](https://python.langchain.com/docs/modules/chains/popular/sqlite) and is designed to answer more general questions about a database, as well as recover from errors.\n",
     "\n",
     "Note that, as this agent is in active development, all answers might not be correct. Additionally, it is not guaranteed that the agent won't perform DML statements on your database given certain questions. Be careful running it on sensitive data!\n",
     "\n",
@@ -63,7 +63,7 @@
    "source": [
     "## Using ZERO_SHOT_REACT_DESCRIPTION\n",
     "\n",
-    "This shows how to initialize the agent using the ZERO_SHOT_REACT_DESCRIPTION agent type."
+    "This shows how to initialize the agent using the ZERO_SHOT_REACT_DESCRIPTION agent type. Note that this is an alternative to the above."
    ]
   },
   {

docs/extras/integrations/tools/dataforseo.ipynb

@@ -15,7 +15,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.utilities.dataforseo_api_search import DataForSeoAPIWrapper"
+    "from langchain.utilities import DataForSeoAPIWrapper"
    ]
   },
   {

docs/extras/integrations/vectorstores/chroma.ipynb

@@ -215,23 +215,10 @@
     "Chroma has the ability to handle multiple `Collections` of documents, but the LangChain interface expects one, so we need to specify the collection name. The default collection name used by LangChain is \"langchain\".\n",
     "\n",
     "Here is how to clone, build, and run the Docker Image:\n",
-    "```sh\n",
+    "```\n",
     "git clone git@github.com:chroma-core/chroma.git\n",
-    "```\n",
-    "\n",
-    "Edit the `docker-compose.yml` file and add `ALLOW_RESET=TRUE` under `environment`\n",
-    "```yaml\n",
-    "    ...\n",
-    "    command: uvicorn chromadb.app:app --reload --workers 1 --host 0.0.0.0 --port 8000 --log-config log_config.yml\n",
-    "    environment:\n",
-    "      - IS_PERSISTENT=TRUE\n",
-    "      - ALLOW_RESET=TRUE\n",
-    "    ports:\n",
-    "      - 8000:8000\n",
-    "    ...\n",
-    "```\n",
-    "\n",
-    "Then run `docker-compose up -d --build`"
+    "docker-compose up -d --build\n",
+    "```"
    ]
   },
   {
@@ -269,9 +256,8 @@
     "    )\n",
     "\n",
     "# tell LangChain to use our client and collection name\n",
-    "db4 = Chroma(client=client, collection_name=\"my_collection\", embedding_function=embedding_function)\n",
-    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
-    "docs = db4.similarity_search(query)\n",
+    "db4 = Chroma(client=client, collection_name=\"my_collection\")\n",
+    "docs = db.similarity_search(query)\n",
     "print(docs[0].page_content)"
    ]
   },

docs/extras/integrations/vectorstores/faiss.ipynb

@@ -23,9 +23,9 @@
    },
    "outputs": [],
    "source": [
-    "!pip install faiss-gpu # For CUDA 7.5+ Supported GPU's.\n",
+    "#!pip install faiss\n",
     "# OR\n",
-    "!pip install faiss-cpu # For CPU Installation"
+    "!pip install faiss-cpu"
    ]
   },
   {

docs/extras/integrations/vectorstores/meilisearch.ipynb

@@ -1,306 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Meilisearch\n",
-    "\n",
-    "> [Meilisearch](https://meilisearch.com) is an open-source, lightning-fast, and hyper relevant search engine. It comes with great defaults to help developers build snappy search experiences. \n",
-    ">\n",
-    "> You can [self-host Meilisearch](https://www.meilisearch.com/docs/learn/getting_started/installation#local-installation) or run on [Meilisearch Cloud](https://www.meilisearch.com/pricing).\n",
-    "\n",
-    "Meilisearch v1.3 supports vector search. This page guides you through integrating Meilisearch as a vector store and using it to perform vector search."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Setup\n",
-    "\n",
-    "### Launching a Meilisearch instance\n",
-    "\n",
-    "You will need a running Meilisearch instance to use as your vector store. You can run [Meilisearch in local](https://www.meilisearch.com/docs/learn/getting_started/installation#local-installation) or create a [Meilisearch Cloud](https://cloud.meilisearch.com/) account.\n",
-    "\n",
-    "As of Meilisearch v1.3, vector storage is an experimental feature. After launching your Meilisearch instance, you need to **enable vector storage**. For self-hosted Meilisearch, read the docs on [enabling experimental features](https://www.meilisearch.com/docs/learn/experimental/vector-search). On **Meilisearch Cloud**, enable _Vector Store_ via your project _Settings_ page.\n",
-    "\n",
-    "You should now have a running Meilisearch instance with vector storage enabled. 🎉\n",
-    "\n",
-    "### Credentials\n",
-    "\n",
-    "To interact with your Meilisearch instance, the Meilisearch SDK needs a host (URL of your instance) and an API key.\n",
-    "\n",
-    "**Host**\n",
-    "\n",
-    "- In **local**, the default host is `localhost:7700`\n",
-    "- On **Meilisearch Cloud**, find the host in your project _Settings_ page\n",
-    "\n",
-    "**API keys**\n",
-    "\n",
-    "Meilisearch instance provides you with three API keys out of the box: \n",
-    "- A `MASTER KEY` — it should only be used to create your Meilisearch instance\n",
-    "- A `ADMIN KEY` — use it only server-side to update your database and its settings\n",
-    "- A `SEARCH KEY` — a key that you can safely share in front-end applications\n",
-    "\n",
-    "You can create [additional API keys](https://www.meilisearch.com/docs/learn/security/master_api_keys) as needed."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Installing dependencies\n",
-    "\n",
-    "This guide uses the [Meilisearch Python SDK](https://github.com/meilisearch/meilisearch-python). You can install it by running:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install meilisearch"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "For more information, refer to the [Meilisearch Python SDK documentation](https://meilisearch.github.io/meilisearch-python/)."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Examples\n",
-    "\n",
-    "There are multiple ways to initialize the Meilisearch vector store: providing a Meilisearch client or the _URL_ and _API key_ as needed. In our examples, the credentials will be loaded from the environment.\n",
-    "\n",
-    "You can make environment variables available in your Notebook environment by using `os` and `getpass`. You can use this technique for all the following examples."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "import getpass\n",
-    "\n",
-    "os.environ[\"MEILI_HTTP_ADDR\"] = getpass.getpass(\"Meilisearch HTTP address and port:\")\n",
-    "os.environ[\"MEILI_MASTER_KEY\"] = getpass.getpass(\"Meilisearch API Key:\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We want to use OpenAIEmbeddings so we have to get the OpenAI API Key."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Adding text and embeddings\n",
-    "\n",
-    "This example adds text to the Meilisearch vector database without having to initialize a Meilisearch vector store."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.vectorstores import Meilisearch\n",
-    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
-    "from langchain.text_splitter import CharacterTextSplitter\n",
-    "\n",
-    "embeddings = OpenAIEmbeddings()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "with open(\"../../../state_of_the_union.txt\") as f:\n",
-    "    state_of_the_union = f.read()\n",
-    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
-    "texts = text_splitter.split_text(state_of_the_union)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Use Meilisearch vector store to store texts & associated embeddings as vector\n",
-    "vector_store = Meilisearch.from_texts(texts=texts, embedding=embeddings)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Behind the scenes, Meilisearch will convert the text to multiple vectors. This will bring us to the same result as the following example."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Adding documents and embeddings\n",
-    "\n",
-    "In this example, we'll use Langchain TextSplitter to split the text in multiple documents. Then, we'll store these documents along with their embeddings."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.document_loaders import TextLoader\n",
-    "\n",
-    "# Load text\n",
-    "loader = TextLoader(\"../../../state_of_the_union.txt\")\n",
-    "documents = loader.load()\n",
-    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
-    "\n",
-    "# Create documents\n",
-    "docs = text_splitter.split_documents(documents)\n",
-    "\n",
-    "# Import documents & embeddings in the vector store\n",
-    "vector_store = Meilisearch.from_documents(documents=documents, embedding=embeddings)\n",
-    "\n",
-    "# Search in our vector store\n",
-    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
-    "docs = vector_store.similarity_search(query)\n",
-    "print(docs[0].page_content)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Add documents by creating a Meilisearch Vectorstore"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "In this approach, we create a vector store object and add documents to it."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.vectorstores import Meilisearch\n",
-    "import meilisearch\n",
-    "\n",
-    "client = meilisearch.Client(url=\"http://127.0.0.1:7700\", api_key=\"***\")\n",
-    "vector_store = Meilisearch(\n",
-    "    embedding=embeddings, client=client, index_name=\"langchain_demo\", text_key=\"text\"\n",
-    ")\n",
-    "vector_store.add_documents(documents)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Similarity Search with score\n",
-    "\n",
-    "This specific method allows you to return the documents and the distance score of the query to them."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs_and_scores = vector_store.similarity_search_with_score(query)\n",
-    "docs_and_scores[0]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Similarity Search by vector"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "embedding_vector = embeddings.embed_query(query)\n",
-    "docs_and_scores = vector_store.similarity_search_by_vector(embedding_vector)\n",
-    "docs_and_scores[0]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Additional resources\n",
-    "\n",
-    "Documentation\n",
-    "- [Meilisearch](https://www.meilisearch.com/docs/)\n",
-    "- [Meilisearch Python SDK](https://python-sdk.meilisearch.com)\n",
-    "\n",
-    "Open-source repositories\n",
-    "- [Meilisearch repository](https://github.com/meilisearch/meilisearch)\n",
-    "- [Meilisearch Python SDK](https://github.com/meilisearch/meilisearch-python)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/extras/integrations/vectorstores/pinecone.ipynb

@@ -25,7 +25,7 @@
    },
    "outputs": [],
    "source": [
-    "!pip install pinecone-client openai tiktoken langchain"
+    "!pip install pinecone-client openai tiktoken"
    ]
   },
   {
@@ -38,7 +38,7 @@
     "import os\n",
     "import getpass\n",
     "\n",
-    "os.environ[\"PINECONE_API_KEY\"] = getpass.getpass(\"Pinecone API Key:\")"
+    "PINECONE_API_KEY = getpass.getpass(\"Pinecone API Key:\")"
    ]
   },
   {
@@ -48,7 +48,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "os.environ[\"PINECONE_ENV\"] = getpass.getpass(\"Pinecone Environment:\")"
+    "PINECONE_ENV = getpass.getpass(\"Pinecone Environment:\")"
    ]
   },
   {
@@ -113,22 +113,12 @@
     "\n",
     "# initialize pinecone\n",
     "pinecone.init(\n",
-    "    api_key=os.getenv(\"PINECONE_API_KEY\"),  # find at app.pinecone.io\n",
-    "    environment=os.getenv(\"PINECONE_ENV\"),  # next to api key in console\n",
+    "    api_key=PINECONE_API_KEY,  # find at app.pinecone.io\n",
+    "    environment=PINECONE_ENV,  # next to api key in console\n",
     ")\n",
     "\n",
     "index_name = \"langchain-demo\"\n",
     "\n",
-    "# First, check if our index already exists. If it doesn't, we create it\n",
-    "if index_name not in pinecone.list_indexes():\n",
-    "    # we create a new index\n",
-    "    pinecone.create_index(\n",
-    "      name=index_name,\n",
-    "      metric='cosine',\n",
-    "      dimension=1536  \n",
-    ")\n",
-    "# The OpenAI embedding model `text-embedding-ada-002 uses 1536 dimensions`\n",
-    
     "docsearch = Pinecone.from_documents(docs, embeddings, index_name=index_name)\n",
     "\n",
     "# if you already have an index, you can load it like this\n",

docs/extras/integrations/vectorstores/qdrant.ipynb

@@ -202,7 +202,7 @@
         "qdrant = Qdrant.from_documents(\n",
         "    docs,\n",
         "    embeddings,\n",
-        "    url=url,\n",
+        "    url,\n",
         "    prefer_grpc=True,\n",
         "    collection_name=\"my_documents\",\n",
         ")"
@@ -236,7 +236,7 @@
         "qdrant = Qdrant.from_documents(\n",
         "    docs,\n",
         "    embeddings,\n",
-        "    url=url,\n",
+        "    url,\n",
         "    prefer_grpc=True,\n",
         "    api_key=api_key,\n",
         "    collection_name=\"my_documents\",\n",
@@ -270,7 +270,7 @@
         "qdrant = Qdrant.from_documents(\n",
         "    docs,\n",
         "    embeddings,\n",
-        "    url=url,\n",
+        "    url,\n",
         "    prefer_grpc=True,\n",
         "    collection_name=\"my_documents\",\n",
         "    force_recreate=True,\n",

docs/extras/integrations/vectorstores/rockset.ipynb

@@ -2,141 +2,131 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "9787b308",
+   "id": "20b588b4",
    "metadata": {},
    "source": [
     "# Rockset\n",
     "\n",
-    ">[Rockset](https://rockset.com/) is a real-time search and analytics database built for the cloud. Rockset uses a [Converged Index™](https://rockset.com/blog/converged-indexing-the-secret-sauce-behind-rocksets-fast-queries/) with an efficient store for vector embeddings to serve low latency, high concurrency search queries at scale. Rockset has full support for metadata filtering and  handles real-time ingestion for constantly updating, streaming data.\n",
+    ">[Rockset](https://rockset.com/product/) is a real-time analytics database service for serving low latency, high concurrency analytical queries at scale. It builds a Converged Index™ on structured and semi-structured data with an efficient store for vector embeddings. Its support for running SQL on schemaless data makes it a perfect choice for running vector search with metadata filters. \n",
     "\n",
-    "This notebook demonstrates how to use `Rockset` as a vector store in LangChain. Before getting started, make sure you have access to a `Rockset` account and an API key available. [Start your free trial today.](https://rockset.com/create/)\n"
+    "This notebook demonstrates how to use `Rockset` as a vectorstore in langchain. To get started, make sure you have a `Rockset` account and an API key available."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b823d64a",
+   "id": "e290ddc0",
    "metadata": {},
    "source": [
-    "## Setting Up Your Environment[](https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/rockset#setting-up-environment)\n",
+    "## Setting up environment\n",
     "\n",
-    "1. Leverage the `Rockset` console to create a [collection](https://rockset.com/docs/collections/) with the Write API as your source. In this walkthrough, we create a collection named `langchain_demo`. \n",
+    "1. Make sure you have Rockset account and go to the web console to get the API key. Details can be found on [the website](https://rockset.com/docs/rest-api/). For the purpose of this notebook, we will assume you're using Rockset from `Oregon(us-west-2)`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7d77bbbe",
+   "metadata": {},
+   "source": [
+    "2. Now you will need to create a Rockset collection to write to, use the Rockset web console to do this. For the purpose of this exercise, we will create a collection called `langchain_demo`. Since Rockset supports schemaless ingest, you don't need to inform us of the shape of metadata for your texts. However, you do need to decide on two columns upfront:\n",
+    "- Where to store the text. We will use the column `description` for this.\n",
+    "- Where to store the vector-embedding for the text. We will use the column `description_embedding` for this.\n",
+    "\n",
+    "Also you will need to inform Rockset that `description_embedding` is a vector-embedding, so that we can optimize its format. You can do this using a **Rockset ingest transformation** while creating your collection:"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "3daa76ba",
+   "metadata": {},
+   "source": [
+    "SELECT\n",
+    "    _input.* EXCEPT(_meta),\n",
+    "    VECTOR_ENFORCE(_input.description_embedding, #length_of_vector_embedding, 'float') as description_embedding\n",
+    "FROM\n",
+    "    _input\n",
     "    \n",
-    "    Configure the following [ingest transformation](https://rockset.com/docs/ingest-transformation/) to mark your embeddings field and take advantage of performance and storage optimizations:"
+    "// We used OpenAI `text-embedding-ada-002` for this examples, where #length_of_vector_embedding = 1536"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7951c9cd",
+   "metadata": {},
+   "source": [
+    "3. Now let's install the [rockset-python-client](https://github.com/rockset/rockset-python-client). This is used by langchain to talk to the Rockset database."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "aac58387",
-   "metadata": {
-    "vscode": {
-     "languageId": "sql"
-    }
-   },
+   "id": "2aac7ae6",
+   "metadata": {},
    "outputs": [],
    "source": [
-    "SELECT _input.* EXCEPT(_meta), \n",
-    "VECTOR_ENFORCE(_input.description_embedding, #length_of_vector_embedding, 'float') as description_embedding \n",
-    "FROM _input"
+    "!pip install rockset"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "df380e1c",
+   "id": "8600900d",
    "metadata": {},
    "source": [
-    "2. After creating your collection, use the console to retrieve an [API key](https://rockset.com/docs/iam/#users-api-keys-and-roles). For the purpose of this notebook, we assume you are using the `Oregon(us-west-2)` region.\n",
-    "\n",
-    "3. Install the [rockset-python-client](https://github.com/rockset/rockset-python-client) to enable LangChain to communicate directly with `Rockset`."
+    "This is it! Now you're ready to start writing some python code to store vector embeddings in Rockset, and querying the database to find texts similar to your query! We support 3 distance functions: `COSINE_SIM`, `EUCLIDEAN_DIST` and `DOT_PRODUCT`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3bf2f818",
+   "metadata": {},
+   "source": [
+    "## Example"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "00d16b83",
+   "id": "a7b39626",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "pip install rockset"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e79550eb",
-   "metadata": {},
-   "source": [
-    "## LangChain Tutorial\n",
-    "\n",
-    "Follow along in your own Python notebook to generate and store vector embeddings in Rockset.\n",
-    "Start using Rockset to search for documents similar to your search queries.\n",
-    "\n",
-    "### 1. Define Key Variables"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "29505c1e",
-   "metadata": {},
-   "outputs": [
-    {
-     "ename": "InitializationException",
-     "evalue": "The rockset client was initialized incorrectly: An api key must be provided as a parameter to the RocksetClient or the Configuration object.",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[0;31mInitializationException\u001b[0m                   Traceback (most recent call last)",
-      "Cell \u001b[0;32mIn[5], line 6\u001b[0m\n\u001b[1;32m      4\u001b[0m ROCKSET_API_KEY \u001b[39m=\u001b[39m os\u001b[39m.\u001b[39menviron\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mROCKSET_API_KEY\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39m# Verify ROCKSET_API_KEY environment variable\u001b[39;00m\n\u001b[1;32m      5\u001b[0m ROCKSET_API_SERVER \u001b[39m=\u001b[39m rockset\u001b[39m.\u001b[39mRegions\u001b[39m.\u001b[39musw2a1 \u001b[39m# Verify Rockset region\u001b[39;00m\n\u001b[0;32m----> 6\u001b[0m rockset_client \u001b[39m=\u001b[39m rockset\u001b[39m.\u001b[39;49mRocksetClient(ROCKSET_API_SERVER, ROCKSET_API_KEY)\n\u001b[1;32m      8\u001b[0m COLLECTION_NAME\u001b[39m=\u001b[39m\u001b[39m'\u001b[39m\u001b[39mlangchain_demo\u001b[39m\u001b[39m'\u001b[39m\n\u001b[1;32m      9\u001b[0m TEXT_KEY\u001b[39m=\u001b[39m\u001b[39m'\u001b[39m\u001b[39mdescription\u001b[39m\u001b[39m'\u001b[39m\n",
-      "File \u001b[0;32m~/Library/Python/3.9/lib/python/site-packages/rockset/rockset_client.py:242\u001b[0m, in \u001b[0;36mRocksetClient.__init__\u001b[0;34m(self, host, api_key, max_workers, config)\u001b[0m\n\u001b[1;32m    239\u001b[0m     config\u001b[39m.\u001b[39mhost \u001b[39m=\u001b[39m host\n\u001b[1;32m    241\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m config\u001b[39m.\u001b[39mapi_key:\n\u001b[0;32m--> 242\u001b[0m     \u001b[39mraise\u001b[39;00m InitializationException(\n\u001b[1;32m    243\u001b[0m         \u001b[39m\"\u001b[39m\u001b[39mAn api key must be provided as a parameter to the RocksetClient or the Configuration object.\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m    244\u001b[0m     )\n\u001b[1;32m    246\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mapi_client \u001b[39m=\u001b[39m ApiClient(config, max_workers\u001b[39m=\u001b[39mmax_workers)\n\u001b[1;32m    248\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mAliases \u001b[39m=\u001b[39m AliasesApiWrapper(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mapi_client)\n",
-      "\u001b[0;31mInitializationException\u001b[0m: The rockset client was initialized incorrectly: An api key must be provided as a parameter to the RocksetClient or the Configuration object."
-     ]
-    }
-   ],
    "source": [
     "import os\n",
     "import rockset\n",
     "\n",
-    "ROCKSET_API_KEY = os.environ.get(\"ROCKSET_API_KEY\") # Verify ROCKSET_API_KEY environment variable\n",
-    "ROCKSET_API_SERVER = rockset.Regions.usw2a1 # Verify Rockset region\n",
+    "# Make sure env variable ROCKSET_API_KEY is set\n",
+    "ROCKSET_API_KEY = os.environ.get(\"ROCKSET_API_KEY\")\n",
+    "ROCKSET_API_SERVER = (\n",
+    "    rockset.Regions.usw2a1\n",
+    ")  # Make sure this points to the correct Rockset region\n",
     "rockset_client = rockset.RocksetClient(ROCKSET_API_SERVER, ROCKSET_API_KEY)\n",
     "\n",
-    "COLLECTION_NAME='langchain_demo'\n",
-    "TEXT_KEY='description'\n",
-    "EMBEDDING_KEY='description_embedding'"
+    "COLLECTION_NAME = \"langchain_demo\"\n",
+    "TEXT_KEY = \"description\"\n",
+    "EMBEDDING_KEY = \"description_embedding\""
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "07625be2",
+   "id": "474636a2",
    "metadata": {},
    "source": [
-    "### 2. Prepare Documents"
+    "Now let's use this client to create a Rockset Langchain Vectorstore!\n",
+    "\n",
+    "### 1. Inserting texts"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9740d8c4",
+   "id": "0d73c5bb",
    "metadata": {},
-   "outputs": [
-    {
-     "ename": "",
-     "evalue": "",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[1;31mRunning cells with '/opt/local/bin/python3.11' requires the ipykernel package.\n",
-      "\u001b[1;31mRun the following command to install 'ipykernel' into the Python environment. \n",
-      "\u001b[1;31mCommand: '/opt/local/bin/python3.11 -m pip install ipykernel -U --user --force-reinstall'"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from langchain.embeddings.openai import OpenAIEmbeddings\n",
     "from langchain.text_splitter import CharacterTextSplitter\n",
     "from langchain.document_loaders import TextLoader\n",
-    "from langchain.vectorstores import Rockset\n",
+    "from langchain.vectorstores.rocksetdb import RocksetDB\n",
     "\n",
-    "loader = TextLoader('../../../state_of_the_union.txt')\n",
+    "loader = TextLoader(\"../../../state_of_the_union.txt\")\n",
     "documents = loader.load()\n",
     "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
     "docs = text_splitter.split_documents(documents)"
@@ -144,33 +134,23 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a068be18",
+   "id": "1404cada",
    "metadata": {},
    "source": [
-    "### 3. Insert Documents"
+    "Now we have the documents we want to insert. Let's create a Rockset vectorstore and insert these docs into the Rockset collection. We will use `OpenAIEmbeddings` to create embeddings for the texts, but you're free to use whatever you want."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "85b6a6c5",
+   "id": "63c98bac",
    "metadata": {},
-   "outputs": [
-    {
-     "ename": "",
-     "evalue": "",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[1;31mRunning cells with '/opt/local/bin/python3.11' requires the ipykernel package.\n",
-      "\u001b[1;31mRun the following command to install 'ipykernel' into the Python environment. \n",
-      "\u001b[1;31mCommand: '/opt/local/bin/python3.11 -m pip install ipykernel -U --user --force-reinstall'"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
-    "embeddings = OpenAIEmbeddings() # Verify OPENAI_KEY environment variable\n",
+    "# Make sure the environment variable OPENAI_API_KEY is set up\n",
+    "embeddings = OpenAIEmbeddings()\n",
     "\n",
-    "docsearch = Rockset(\n",
+    "docsearch = RocksetDB(\n",
     "    client=rockset_client,\n",
     "    embeddings=embeddings,\n",
     "    collection_name=COLLECTION_NAME,\n",
@@ -178,46 +158,38 @@
     "    embedding_key=EMBEDDING_KEY,\n",
     ")\n",
     "\n",
-    "ids=docsearch.add_texts(\n",
+    "ids = docsearch.add_texts(\n",
     "    texts=[d.page_content for d in docs],\n",
     "    metadatas=[d.metadata for d in docs],\n",
-    ")"
+    ")\n",
+    "\n",
+    "## If you go to the Rockset console now, you should be able to see this docs along with the metadata `source`"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "56eef48d",
+   "id": "f1290844",
    "metadata": {},
    "source": [
-    "### 4. Search for Similar Documents"
+    "### 2. Searching similar texts\n",
+    "\n",
+    "Now let's try to search Rockset to find strings similar to our query string!"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "id": "0bbf3df0",
+   "execution_count": null,
+   "id": "96e73ac1",
    "metadata": {},
-   "outputs": [
-    {
-     "ename": "NameError",
-     "evalue": "name 'docsearch' is not defined",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
-      "Cell \u001b[0;32mIn[1], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m query \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mWhat did the president say about Ketanji Brown Jackson?\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m output \u001b[39m=\u001b[39m docsearch\u001b[39m.\u001b[39msimilarity_search_with_relevance_scores(query, \u001b[39m4\u001b[39m, Rockset\u001b[39m.\u001b[39mDistanceFunction\u001b[39m.\u001b[39mCOSINE_SIM)\n\u001b[1;32m      4\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39moutput length:\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mlen\u001b[39m(output))\n\u001b[1;32m      5\u001b[0m \u001b[39mfor\u001b[39;00m d, dist \u001b[39min\u001b[39;00m output:\n",
-      "\u001b[0;31mNameError\u001b[0m: name 'docsearch' is not defined"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "query = \"What did the president say about Ketanji Brown Jackson\"\n",
     "output = docsearch.similarity_search_with_relevance_scores(\n",
-    "    query, 4, Rockset.DistanceFunction.COSINE_SIM\n",
+    "    query, 4, RocksetDB.DistanceFunction.COSINE_SIM\n",
     ")\n",
     "print(\"output length:\", len(output))\n",
     "for d, dist in output:\n",
-    "    print(dist, d.metadata, d.page_content[:20] + '...')\n",
+    "    print(dist, d.metadata, d.page_content[:20] + \"...\")\n",
     "\n",
     "##\n",
     "# output length: 4\n",
@@ -229,28 +201,32 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7037a22f",
+   "id": "5e15d630",
    "metadata": {},
    "source": [
-    "### 5. Search for Similar Documents with Filtering"
+    "You can also use a where filter to prune your search space. You can add filters on text key, or any of the metadata fields. \n",
+    "\n",
+    "> **Note**: Since Rockset stores each metadata field as a separate column internally, these filters are much faster than other vector databases which store all metadata as a single JSON.\n",
+    "\n",
+    "For eg, to find all texts NOT containing the substring \"and\", you can use the following code:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b64a290f",
+   "id": "c1c44d41",
    "metadata": {},
    "outputs": [],
    "source": [
     "output = docsearch.similarity_search_with_relevance_scores(\n",
     "    query,\n",
     "    4,\n",
-    "    Rockset.DistanceFunction.COSINE_SIM,\n",
+    "    RocksetDB.DistanceFunction.COSINE_SIM,\n",
     "    where_str=\"{} NOT LIKE '%citizens%'\".format(TEXT_KEY),\n",
     ")\n",
     "print(\"output length:\", len(output))\n",
     "for d, dist in output:\n",
-    "    print(dist, d.metadata, d.page_content[:20] + '...')\n",
+    "    print(dist, d.metadata, d.page_content[:20] + \"...\")\n",
     "\n",
     "##\n",
     "# output length: 4\n",
@@ -261,23 +237,21 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
-   "id": "13a52b38",
+   "id": "0765b822",
    "metadata": {},
    "source": [
-    "### 6. [Optional] Delete Inserted Documents\n",
+    "### 3. [Optional] Drop all inserted documents\n",
     "\n",
-    "You must have the unique ID associated with each document to delete them from your collection.\n",
-    "Define IDs when inserting documents with `Rockset.add_texts()`. Rockset will otherwise generate a unique ID for each document. Regardless, `Rockset.add_texts()` returns the IDs of inserted documents.\n",
+    "In order to delete texts from the Rockset collection, you need to know the unique ID associated with each document inside Rockset. These ids can either be supplied directly by the user while inserting the texts (in the `RocksetDB.add_texts()` function), else Rockset will generate a unique ID or each document. Either way, `Rockset.add_texts()` returns the ids for the inserted documents.\n",
     "\n",
-    "To delete these docs, simply use the `Rockset.delete_texts()` function."
+    "To delete these docs, simply use the `RocksetDB.delete_texts()` function."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1f755924",
+   "id": "31738966",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -286,15 +260,23 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d468f431",
+   "id": "03fa12a9",
    "metadata": {},
    "source": [
-    "## Summary\n",
+    "## Congratulations!\n",
     "\n",
-    "In this tutorial, we successfully created a `Rockset` collection, `inserted` documents with  OpenAI embeddings, and searched for similar documents with and without metadata filters.\n",
+    "Voila! In this example you successfuly created a Rockset collection, inserted documents along with their OpenAI vector embeddings, and searched for similar docs both with and without any metadata filters.\n",
     "\n",
-    "Keep an eye on https://rockset.com/ for future updates in this space."
+    "Keep an eye on https://rockset.com/blog/introducing-vector-search-on-rockset/ for future updates in this space!"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2763dddb-e87d-4d3b-b0bf-c246b0573d87",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -313,7 +295,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,