Update Hugging Face Hub notebook (#7236)

Description: `flan-t5-xl` hangs, updated to `flan-t5-xxl`. Tested all
stabilityai LLMs- all hang so removed from tutorial. Temperature > 0 to
prevent unintended determinism.
Issue: #3275 
Tag maintainer: @baskaryan

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,56 +1,26 @@
-<!--
-Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution.
+<!-- Thank you for contributing to LangChain!
 
-Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change.
+Replace this comment with:
+  - Description: a description of the change, 
+  - Issue: the issue # it fixes (if applicable),
+  - Dependencies: any dependencies required for this change,
+  - Tag maintainer: for a quicker response, tag the relevant maintainer (see below),
+  - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out!
 
-After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost.
+If you're adding a new integration, please include:
+  1. a test for the integration, preferably unit tests that do not rely on network access,
+  2. an example notebook showing its use.
 
-Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle!
--->
+Maintainer responsibilities:
+  - General / Misc / if you don't know who to tag: @baskaryan
+  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
+  - Models / Prompts: @hwchase17, @baskaryan
+  - Memory: @hwchase17
+  - Agents / Tools / Toolkits: @hinthornw
+  - Tracing / Callbacks: @agola11
+  - Async: @agola11
 
-<!-- Remove if not applicable -->
-
-Fixes # (issue)
-
-#### Before submitting
-
-<!-- If you're adding a new integration, please include:
-
-1. a test for the integration - favor unit tests that does not rely on network access.
-2. an example notebook showing its use
-
-
-See contribution guidelines for more information on how to write tests, lint
-etc:
-
-https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
--->
-
-#### Who can review?
-
-Tag maintainers/contributors who might be interested:
-
-<!-- For a quicker response, figure out the right person to tag with @
-
-  @hwchase17 - project lead
-
-  Tracing / Callbacks
-  - @agola11
-
-  Async
-  - @agola11
-
-  DataLoaders
-  - @eyurtsev
-
-  Models
-  - @hwchase17
-  - @agola11
-
-  Agents / Tools / Toolkits
-  - @hwchase17
-
-  VectorStores / Retrievers / Memory
-  - @dev2049
+If no one reviews your PR within a few days, feel free to @-mention the same people again.
 
+See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
  -->

.readthedocs.yaml

@@ -9,6 +9,9 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.11"
+  jobs:
+    pre_build:
+      - python docs/api_reference/create_api_rst.py
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:

docs/api_reference/_static/js/mendablesearch.js

@@ -1,57 +0,0 @@
-document.addEventListener('DOMContentLoaded', () => {
-  // Load the external dependencies
-  function loadScript(src, onLoadCallback) {
-    const script = document.createElement('script');
-    script.src = src;
-    script.onload = onLoadCallback;
-    document.head.appendChild(script);
-  }
-
-  function createRootElement() {
-    const rootElement = document.createElement('div');
-    rootElement.id = 'my-component-root';
-    document.body.appendChild(rootElement);
-    return rootElement;
-  }
-
-  
-
-  function initializeMendable() {
-    const rootElement = createRootElement();
-    const { MendableFloatingButton } = Mendable;
-    
-
-    const iconSpan1 = React.createElement('span', {
-    }, '🦜');
-
-    const iconSpan2 = React.createElement('span', {
-    }, '🔗');
-
-    const icon = React.createElement('p', {
-      style: { color: '#ffffff', fontSize: '22px',width: '48px', height: '48px', margin: '0px', padding: '0px', display: 'flex', alignItems: 'center', justifyContent: 'center', textAlign: 'center' },
-    }, [iconSpan1, iconSpan2]);
-    
-    const mendableFloatingButton = React.createElement(
-      MendableFloatingButton,
-      {
-        style: { darkMode: false, accentColor: '#010810' },
-        floatingButtonStyle: { color: '#ffffff', backgroundColor: '#010810' },
-        anon_key: '82842b36-3ea6-49b2-9fb8-52cfc4bde6bf', // Mendable Search Public ANON key, ok to be public
-        cmdShortcutKey:'j',
-        messageSettings: {
-          openSourcesInNewTab: false,
-          prettySources: true // Prettify the sources displayed now
-        },
-        icon: icon,
-      }
-    );
-
-    ReactDOM.render(mendableFloatingButton, rootElement);
-  }
-
-  loadScript('https://unpkg.com/react@17/umd/react.production.min.js', () => {
-    loadScript('https://unpkg.com/react-dom@17/umd/react-dom.production.min.js', () => {
-      loadScript('https://unpkg.com/@mendable/search@0.0.102/dist/umd/mendable.min.js', initializeMendable);
-    });
-  });
-});

docs/api_reference/agents.rst

@@ -1,12 +0,0 @@
-Agents
-==============
-
-Reference guide for Agents and associated abstractions.
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   modules/agents
-   modules/tools
-   modules/agent_toolkits

docs/api_reference/conf.py

@@ -11,12 +11,13 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
 
 import toml
 
+sys.path.insert(0, os.path.abspath("."))
+
 with open("../../pyproject.toml") as f:
     data = toml.load(f)
 
@@ -45,11 +46,9 @@ extensions = [
     "sphinx.ext.napoleon",
     "sphinx.ext.viewcode",
     "sphinxcontrib.autodoc_pydantic",
-    "myst_nb",
     "sphinx_copybutton",
     "sphinx_panels",
     "IPython.sphinxext.ipython_console_highlighting",
-    "sphinx_tabs.tabs",
 ]
 source_suffix = [".rst"]
 
@@ -59,24 +58,22 @@ autodoc_pydantic_config_members = False
 autodoc_pydantic_model_show_config_summary = False
 autodoc_pydantic_model_show_validator_members = False
 autodoc_pydantic_model_show_validator_summary = False
-autodoc_pydantic_model_show_field_summary = False
-autodoc_pydantic_model_members = False
-autodoc_pydantic_model_undoc_members = False
-autodoc_pydantic_model_hide_paramlist = False
 autodoc_pydantic_model_signature_prefix = "class"
-autodoc_pydantic_field_signature_prefix = "attribute"
-autodoc_pydantic_model_summary_list_order = "bysource"
-autodoc_member_order = "bysource"
+autodoc_pydantic_field_signature_prefix = "param"
+autodoc_member_order = "groupwise"
+autoclass_content = "both"
+autodoc_typehints_format = "short"
+
 autodoc_default_options = {
     "members": True,
     "show-inheritance": True,
-    "undoc_members": True,
-    "inherited_members": "BaseModel",
+    "inherited-members": "BaseModel",
+    "undoc-members": True,
+    "special-members": "__call__",
 }
-autodoc_typehints = "description"
-
+# autodoc_typehints = "description"
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ["_templates"]
+templates_path = ["templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -89,14 +86,16 @@ exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "sphinx_rtd_theme"
+html_theme = "scikit-learn-modern"
+html_theme_path = ["themes"]
 
-html_theme_options = {
-    "path_to_docs": "docs",
-    "repository_url": "https://github.com/hwchase17/langchain",
-    "use_repository_button": True,
-    # "style_nav_header_background": "white"
+# redirects dictionary maps from old links to new links
+html_additional_pages = {}
+redirects = {
+    "index": "api_reference",
 }
+for old_link in redirects:
+    html_additional_pages[old_link] = "redirects.html"
 
 html_context = {
     "display_github": True,  # Integrate GitHub
@@ -104,6 +103,7 @@ html_context = {
     "github_repo": "langchain",  # Repo name
     "github_version": "master",  # Version
     "conf_py_path": "/docs/api_reference",  # Path in the checkout to the docs root
+    "redirects": redirects,
 }
 
 # Add any paths that contain custom static files (such as style sheets) here,
@@ -116,10 +116,9 @@ html_static_path = ["_static"]
 html_css_files = [
     "css/custom.css",
 ]
+html_use_index = False
 
-html_js_files = [
-    "js/mendablesearch.js",
-]
-
-nb_execution_mode = "off"
 myst_enable_extensions = ["colon_fence"]
+
+# generate autosummary even if no references
+autosummary_generate = True

docs/api_reference/create_api_rst.py

@@ -0,0 +1,94 @@
+"""Script for auto-generating api_reference.rst"""
+import glob
+import re
+from pathlib import Path
+
+ROOT_DIR = Path(__file__).parents[2].absolute()
+PKG_DIR = ROOT_DIR / "langchain"
+WRITE_FILE = Path(__file__).parent / "api_reference.rst"
+
+
+def load_members() -> dict:
+    members: dict = {}
+    for py in glob.glob(str(PKG_DIR) + "/**/*.py", recursive=True):
+        module = py[len(str(PKG_DIR)) + 1 :].replace(".py", "").replace("/", ".")
+        top_level = module.split(".")[0]
+        if top_level not in members:
+            members[top_level] = {"classes": [], "functions": []}
+        with open(py, "r") as f:
+            for line in f.readlines():
+                cls = re.findall(r"^class ([^_].*)\(", line)
+                members[top_level]["classes"].extend([module + "." + c for c in cls])
+                func = re.findall(r"^def ([^_].*)\(", line)
+                members[top_level]["functions"].extend([module + "." + f for f in func])
+    return members
+
+
+def construct_doc(members: dict) -> str:
+    full_doc = """\
+.. _api_reference:
+
+=============
+API Reference
+=============
+
+"""
+    for module, _members in sorted(members.items(), key=lambda kv: kv[0]):
+        classes = _members["classes"]
+        functions = _members["functions"]
+        if not (classes or functions):
+            continue
+
+        module_title = module.replace("_", " ").title()
+        if module_title == "Llms":
+            module_title = "LLMs"
+        section = f":mod:`langchain.{module}`: {module_title}"
+        full_doc += f"""\
+{section}
+{'=' * (len(section) + 1)}
+
+.. automodule:: langchain.{module}
+    :no-members:
+    :no-inherited-members:
+
+"""
+
+        if classes:
+            cstring = "\n    ".join(sorted(classes))
+            full_doc += f"""\
+Classes
+--------------
+.. currentmodule:: langchain
+
+.. autosummary::
+    :toctree: {module}
+    :template: class.rst
+
+    {cstring}
+
+"""
+        if functions:
+            fstring = "\n    ".join(sorted(functions))
+            full_doc += f"""\
+Functions
+--------------
+.. currentmodule:: langchain
+
+.. autosummary::
+    :toctree: {module}
+
+    {fstring}
+
+"""
+    return full_doc
+
+
+def main() -> None:
+    members = load_members()
+    full_doc = construct_doc(members)
+    with open(WRITE_FILE, "w") as f:
+        f.write(full_doc)
+
+
+if __name__ == "__main__":
+    main()

docs/api_reference/data_connection.rst

@@ -1,13 +0,0 @@
-Data connection
-==============
-LangChain has a number of modules that help you load, structure, store, and retrieve documents.
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   modules/document_loaders
-   modules/document_transformers
-   modules/embeddings
-   modules/vectorstores
-   modules/retrievers

docs/api_reference/index.rst

@@ -1,29 +1,8 @@
-API Reference
-==========================
-
-| Full documentation on all methods, classes, and APIs in the LangChain Python package.
+=============
+LangChain API
+=============
 
 .. toctree::
-   :maxdepth: 1
-   :caption: Abstractions
+    :maxdepth: 2
 
-   ./modules/base_classes.rst
-
-
-.. toctree::
-   :maxdepth: 1
-   :caption: Core
-
-   ./model_io.rst
-   ./data_connection.rst
-   ./modules/chains.rst
-   ./agents.rst
-   ./modules/memory.rst
-   ./modules/callbacks.rst
-
-.. toctree::
-   :maxdepth: 1
-   :caption: Additional
-
-   ./modules/utilities.rst
-   ./modules/experimental.rst
+    api_reference.rst

docs/api_reference/model_io.rst

@@ -1,12 +0,0 @@
-Model I/O
-==============
-
-LangChain provides interfaces and integrations for working with language models.
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   ./prompts.rst
-   ./models.rst
-   ./modules/output_parsers.rst

docs/api_reference/models.rst

@@ -1,11 +0,0 @@
-Models
-==============
-
-LangChain provides interfaces and integrations for a number of different types of models.
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   modules/llms
-   modules/chat_models

docs/api_reference/modules/agent_toolkits.rst

@@ -1,7 +0,0 @@
-Agent Toolkits
-===============================
-
-.. automodule:: langchain.agents.agent_toolkits
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/agents.rst

@@ -1,7 +0,0 @@
-Agents
-===============================
-
-.. automodule:: langchain.agents
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/base_classes.rst

@@ -1,5 +0,0 @@
-Base classes
-========================
-
-.. automodule:: langchain.schema
-   :inherited-members:

docs/api_reference/modules/callbacks.rst

@@ -1,7 +0,0 @@
-Callbacks
-=======================
-
-.. automodule:: langchain.callbacks
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/chains.rst

@@ -1,8 +0,0 @@
-Chains
-=======================
-
-.. automodule:: langchain.chains
-   :members:
-   :undoc-members:
-   :inherited-members: BaseModel
-

docs/api_reference/modules/chat_models.rst

@@ -1,7 +0,0 @@
-Chat Models
-===============================
-
-.. automodule:: langchain.chat_models
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/document_loaders.rst

@@ -1,7 +0,0 @@
-Document Loaders
-===============================
-
-.. automodule:: langchain.document_loaders
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/document_transformers.rst

@@ -1,13 +0,0 @@
-Document Transformers
-===============================
-
-.. automodule:: langchain.document_transformers
-   :members:
-   :undoc-members:
-
-Text Splitters
-------------------------------
-
-.. automodule:: langchain.text_splitter
-   :members:
-   :undoc-members:

docs/api_reference/modules/embeddings.rst

@@ -1,5 +0,0 @@
-Embeddings
-===========================
-
-.. automodule:: langchain.embeddings
-   :members:

docs/api_reference/modules/example_selector.rst

@@ -1,5 +0,0 @@
-Example Selector
-=========================================
-
-.. automodule:: langchain.prompts.example_selector
-   :members:

docs/api_reference/modules/experimental.rst

@@ -1,28 +0,0 @@
-====================
-Experimental
-====================
-
-This module contains experimental modules and reproductions of existing work using LangChain primitives.
-
-Autonomous agents
-------------------
-
-Here, we document the BabyAGI and AutoGPT classes from the langchain.experimental module.
-
-.. autoclass:: langchain.experimental.BabyAGI
-   :members:
-
-.. autoclass:: langchain.experimental.AutoGPT
-   :members:
-
-
-Generative agents
-------------------
-
-Here, we document the GenerativeAgent and GenerativeAgentMemory classes from the langchain.experimental module.
-
-.. autoclass:: langchain.experimental.GenerativeAgent
-   :members:
-
-.. autoclass:: langchain.experimental.GenerativeAgentMemory
-   :members:

docs/api_reference/modules/llms.rst

@@ -1,7 +0,0 @@
-LLMs
-=======================
-
-.. automodule:: langchain.llms
-   :members:
-   :inherited-members:
-   :special-members: __call__

docs/api_reference/modules/memory.rst

@@ -1,7 +0,0 @@
-Memory
-===============================
-
-.. automodule:: langchain.memory
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/output_parsers.rst

@@ -1,7 +0,0 @@
-Output Parsers
-===============================
-
-.. automodule:: langchain.output_parsers
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/prompts.rst

@@ -1,6 +0,0 @@
-Prompt Templates
-========================
-
-.. automodule:: langchain.prompts
-   :members:
-   :undoc-members:

docs/api_reference/modules/retrievers.rst

@@ -1,14 +0,0 @@
-Retrievers
-===============================
-
-.. automodule:: langchain.retrievers
-   :members:
-   :undoc-members:
-
-Document compressors
--------------------------------
-
-.. automodule:: langchain.retrievers.document_compressors
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/tools.rst

@@ -1,7 +0,0 @@
-Tools
-===============================
-
-.. automodule:: langchain.tools
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/utilities.rst

@@ -1,7 +0,0 @@
-Utilities
-===============================
-
-.. automodule:: langchain.utilities
-   :members:
-   :undoc-members:
-

docs/api_reference/modules/vectorstores.rst

@@ -1,6 +0,0 @@
-Vector Stores
-=============================
-
-.. automodule:: langchain.vectorstores
-   :members:
-   :undoc-members:

docs/api_reference/prompts.rst

@@ -1,11 +0,0 @@
-Prompts
-==============
-
-The reference guides here all relate to objects for working with Prompts.
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   modules/prompts
-   modules/example_selector

docs/api_reference/templates/COPYRIGHT.txt

@@ -0,0 +1,27 @@
+Copyright (c) 2007-2023 The scikit-learn developers.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

docs/api_reference/templates/class.rst

@@ -0,0 +1,28 @@
+:mod:`{{module}}`.{{objname}}
+{{ underline }}==============
+
+.. currentmodule:: {{ module }}
+
+.. autoclass:: {{ objname }}
+
+   {% block methods %}
+   {% if methods %}
+   .. rubric:: {{ _('Methods') }}
+
+   .. autosummary::
+   {% for item in methods %}
+      ~{{ name }}.{{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
+   {% block attributes %}
+   {% if attributes %}
+   .. rubric:: {{ _('Attributes') }}
+
+   .. autosummary::
+   {% for item in attributes %}
+      ~{{ name }}.{{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}

docs/api_reference/templates/redirects.html

@@ -0,0 +1,15 @@
+{% set redirect = pathto(redirects[pagename]) %}
+<!DOCTYPE html>
+<html>
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <meta http-equiv="Refresh" content="0; url={{ redirect }}" />
+    <meta name="Description" content="scikit-learn: machine learning in Python">
+    <link rel="canonical" href="{{ redirect }}" />
+    <title>scikit-learn: machine learning in Python</title>
+  </head>
+  <body>
+    <p>You will be automatically redirected to the <a href="{{ redirect }}">new location of this page</a>.</p>
+  </body>
+</html>

docs/api_reference/themes/COPYRIGHT.txt

@@ -0,0 +1,27 @@
+Copyright (c) 2007-2023 The scikit-learn developers.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

docs/api_reference/themes/scikit-learn-modern/javascript.html

@@ -0,0 +1,67 @@
+<script>
+$(document).ready(function() {
+    /* Add a [>>>] button on the top-right corner of code samples to hide
+     * the >>> and ... prompts and the output and thus make the code
+     * copyable. */
+    var div = $('.highlight-python .highlight,' +
+                '.highlight-python3 .highlight,' +
+                '.highlight-pycon .highlight,' +
+		'.highlight-default .highlight')
+    var pre = div.find('pre');
+
+    // get the styles from the current theme
+    pre.parent().parent().css('position', 'relative');
+    var hide_text = 'Hide prompts and outputs';
+    var show_text = 'Show prompts and outputs';
+
+    // create and add the button to all the code blocks that contain >>>
+    div.each(function(index) {
+        var jthis = $(this);
+        if (jthis.find('.gp').length > 0) {
+            var button = $('<span class="copybutton">&gt;&gt;&gt;</span>');
+            button.attr('title', hide_text);
+            button.data('hidden', 'false');
+            jthis.prepend(button);
+        }
+        // tracebacks (.gt) contain bare text elements that need to be
+        // wrapped in a span to work with .nextUntil() (see later)
+        jthis.find('pre:has(.gt)').contents().filter(function() {
+            return ((this.nodeType == 3) && (this.data.trim().length > 0));
+        }).wrap('<span>');
+    });
+
+    // define the behavior of the button when it's clicked
+    $('.copybutton').click(function(e){
+        e.preventDefault();
+        var button = $(this);
+        if (button.data('hidden') === 'false') {
+            // hide the code output
+            button.parent().find('.go, .gp, .gt').hide();
+            button.next('pre').find('.gt').nextUntil('.gp, .go').css('visibility', 'hidden');
+            button.css('text-decoration', 'line-through');
+            button.attr('title', show_text);
+            button.data('hidden', 'true');
+        } else {
+            // show the code output
+            button.parent().find('.go, .gp, .gt').show();
+            button.next('pre').find('.gt').nextUntil('.gp, .go').css('visibility', 'visible');
+            button.css('text-decoration', 'none');
+            button.attr('title', hide_text);
+            button.data('hidden', 'false');
+        }
+    });
+
+	/*** Add permalink buttons next to glossary terms ***/
+	$('dl.glossary > dt[id]').append(function() {
+		return ('<a class="headerlink" href="#' +
+			    this.getAttribute('id') +
+			    '" title="Permalink to this term">¶</a>');
+	});
+});
+
+</script>
+{%- if pagename != 'index' and pagename != 'documentation' %}
+    {% if theme_mathjax_path %}
+<script id="MathJax-script" async src="{{ theme_mathjax_path }}"></script>
+    {% endif %}
+{%- endif %}

docs/api_reference/themes/scikit-learn-modern/layout.html

@@ -0,0 +1,142 @@
+{# TEMPLATE VAR SETTINGS #}
+{%- set url_root = pathto('', 1) %}
+{%- if url_root == '#' %}{% set url_root = '' %}{% endif %}
+{%- if not embedded and docstitle %}
+  {%- set titlesuffix = " — "|safe + docstitle|e %}
+{%- else %}
+  {%- set titlesuffix = "" %}
+{%- endif %}
+{%- set lang_attr = 'en' %}
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="{{ lang_attr }}" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="{{ lang_attr }}" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  {{ metatags }}
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+  {% block htmltitle %}
+  <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
+  {% endblock %}
+  <link rel="canonical" href="http://scikit-learn.org/stable/{{pagename}}.html" />
+
+  {% if favicon_url %}
+  <link rel="shortcut icon" href="{{ favicon_url|e }}"/>
+  {% endif %}
+
+  <link rel="stylesheet" href="{{ pathto('_static/css/vendor/bootstrap.min.css', 1) }}" type="text/css" />
+  {%- for css in css_files %}
+    {%- if css|attr("rel") %}
+  <link rel="{{ css.rel }}" href="{{ pathto(css.filename, 1) }}" type="text/css"{% if css.title is not none %} title="{{ css.title }}"{% endif %} />
+    {%- else %}
+  <link rel="stylesheet" href="{{ pathto(css, 1) }}" type="text/css" />
+    {%- endif %}
+  {%- endfor %}
+  <link rel="stylesheet" href="{{ pathto('_static/' + style, 1) }}" type="text/css" />
+<script id="documentation_options" data-url_root="{{ pathto('', 1) }}" src="{{ pathto('_static/documentation_options.js', 1) }}"></script>
+<script src="{{ pathto('_static/jquery.js', 1) }}"></script>
+{%- block extrahead %} {% endblock %}
+</head>
+<body>
+{% include "nav.html" %}
+{%- block content %}
+<div class="d-flex" id="sk-doc-wrapper">
+    <input type="checkbox" name="sk-toggle-checkbox" id="sk-toggle-checkbox">
+    <label id="sk-sidemenu-toggle" class="sk-btn-toggle-toc btn sk-btn-primary" for="sk-toggle-checkbox">Toggle Menu</label>
+    <div id="sk-sidebar-wrapper" class="border-right">
+      <div class="sk-sidebar-toc-wrapper">
+        <div class="btn-group w-100 mb-2" role="group" aria-label="rellinks">
+          {%- if prev %}
+            <a href="{{ prev.link|e }}" role="button" class="btn sk-btn-rellink py-1" sk-rellink-tooltip="{{ prev.title|striptags }}">Prev</a>
+          {%- else %}
+            <a href="#" role="button" class="btn sk-btn-rellink py-1 disabled"">Prev</a>
+          {%- endif %}
+          {%- if parents -%}
+            <a href="{{ parents[-1].link|e }}" role="button" class="btn sk-btn-rellink py-1" sk-rellink-tooltip="{{ parents[-1].title|striptags }}">Up</a>
+          {%- else %}
+            <a href="#" role="button" class="btn sk-btn-rellink disabled py-1">Up</a>
+          {%- endif %}
+          {%- if next %}
+            <a href="{{ next.link|e }}" role="button" class="btn sk-btn-rellink py-1" sk-rellink-tooltip="{{ next.title|striptags }}">Next</a>
+          {%- else %}
+            <a href="#" role="button" class="btn sk-btn-rellink py-1 disabled"">Next</a>
+          {%- endif %}
+        </div>
+        {%- if pagename != "install" %}
+        <div class="alert alert-warning p-1 mb-2" role="alert">
+          <p class="text-center mb-0">
+          <strong>LangChain {{ release }}</strong><br/>
+          </p>
+        </div>
+        {%- endif %}
+            {%- if meta and meta['parenttoc']|tobool %}
+            <div class="sk-sidebar-toc">
+            {% set nav = get_nav_object(maxdepth=3, collapse=True, numbered=True) %}
+              <ul>
+              {% for main_nav_item in nav %}
+              {% if main_nav_item.active %}
+              <li>
+                <a href="{{ main_nav_item.url }}" class="sk-toc-active">{{ main_nav_item.title }}</a>
+              </li>
+              <ul>
+              {% for nav_item in main_nav_item.children %}
+                <li>
+                  <a href="{{ nav_item.url }}" class="{% if nav_item.active %}sk-toc-active{% endif %}">{{ nav_item.title }}</a>
+                  {% if nav_item.children %}
+                  <ul>
+                    {% for inner_child in nav_item.children %}
+                      <li class="sk-toctree-l3">
+                        <a href="{{ inner_child.url }}">{{ inner_child.title }}</a>
+                      </li>
+                    {% endfor %}
+                  </ul>
+                  {% endif %}
+                </li>
+              {% endfor %}
+              </ul>
+              {% endif %}
+              {% endfor %}
+              </ul>
+            </div>
+            {%- elif meta and meta['globalsidebartoc']|tobool %}
+            <div class="sk-sidebar-toc sk-sidebar-global-toc">
+              {{ toctree(maxdepth=2, titles_only=True) }}
+            </div>
+            {%- else %}
+            <div class="sk-sidebar-toc">
+              {{ toc }}
+            </div>
+            {%- endif %}
+      </div>
+    </div>
+    <div id="sk-page-content-wrapper">
+      <div class="sk-page-content container-fluid body px-md-3" role="main">
+        {% block body %}{% endblock %}
+      </div>
+    <div class="container">
+      <footer class="sk-content-footer">
+        {%- if pagename != 'index' %}
+        {%- if show_copyright %}
+          {%- if hasdoc('copyright') %}
+            {% trans path=pathto('copyright'), copyright=copyright|e %}&copy; {{ copyright }}.{% endtrans %}
+          {%- else %}
+            {% trans copyright=copyright|e %}&copy; {{ copyright }}.{% endtrans %}
+          {%- endif %}
+        {%- endif %}
+        {%- if last_updated %}
+          {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
+        {%- endif %}
+        {%- if show_source and has_source and sourcename %}
+          <a href="{{ pathto('_sources/' + sourcename, true)|e }}" rel="nofollow">{{ _('Show this page source') }}</a>
+        {%- endif %}
+        {%- endif %}
+      </footer>
+    </div>
+  </div>
+</div>
+{%- endblock %}
+<script src="{{ pathto('_static/js/vendor/bootstrap.min.js', 1) }}"></script>
+{% include "javascript.html" %}
+</body>
+</html>

docs/api_reference/themes/scikit-learn-modern/nav.html

@@ -0,0 +1,85 @@
+{%- if pagename != 'index' and pagename != 'documentation' %}
+  {%- set nav_bar_class = "sk-docs-navbar" %}
+  {%- set top_container_cls = "sk-docs-container" %}
+{%- else %}
+  {%- set nav_bar_class = "sk-landing-navbar" %}
+  {%- set top_container_cls = "sk-landing-container" %}
+{%- endif %}
+
+{% if theme_link_to_live_contributing_page|tobool %}
+{# Link to development page for live builds #}
+  {%- set development_link = "https://scikit-learn.org/dev/developers/index.html" %}
+{# Open on a new development page in new window/tab for live builds #}
+  {%- set development_attrs = 'target="_blank" rel="noopener noreferrer"' %}
+{%- else %}
+  {%- set development_link = pathto('developers/index') %}
+  {%- set development_attrs = '' %}
+{%- endif %}
+
+{# title, link, link_attrs #}
+{%- set drop_down_navigation = [
+  ('Getting Started', pathto('getting_started'), ''),
+  ('Tutorial', pathto('tutorial/index'), ''),
+  ("What's new", pathto('whats_new/v' + version), ''),
+  ('Glossary', pathto('glossary'), ''),
+  ('Development', development_link, development_attrs),
+  ('FAQ', pathto('faq'), ''),
+  ('Support', pathto('support'), ''),
+  ('Related packages', pathto('related_projects'), ''),
+  ('Roadmap', pathto('roadmap'), ''),
+  ('Governance', pathto('governance'), ''),
+  ('About us', pathto('about'), ''),
+  ('GitHub', 'https://github.com/scikit-learn/scikit-learn', ''),
+  ('Other Versions and Download', 'https://scikit-learn.org/dev/versions.html', '')]
+-%}
+
+<nav id="navbar" class="{{ nav_bar_class }} navbar navbar-expand-md navbar-light bg-light py-0">
+  <div class="container-fluid {{ top_container_cls }} px-0">
+    {%- if logo_url %}
+      <a class="navbar-brand py-0" href="{{ pathto('index') }}">
+        <img
+          class="sk-brand-img"
+          src="{{ logo_url|e }}"
+          alt="logo"/>
+      </a>
+    {%- endif %}
+    <button
+      id="sk-navbar-toggler"
+      class="navbar-toggler"
+      type="button"
+      data-toggle="collapse"
+      data-target="#navbarSupportedContent"
+      aria-controls="navbarSupportedContent"
+      aria-expanded="false"
+      aria-label="Toggle navigation"
+    >
+      <span class="navbar-toggler-icon"></span>
+    </button>
+
+    <div class="sk-navbar-collapse collapse navbar-collapse" id="navbarSupportedContent">
+      <ul class="navbar-nav mr-auto">
+        <li class="nav-item">
+          <a class="sk-nav-link nav-link" href="{{ pathto('api_reference') }}">API</a>
+        </li>
+        <li class="nav-item">
+          <a class="sk-nav-link nav-link" target="_blank" rel="noopener noreferrer" href="https://python.langchain.com/">Python Docs</a>
+        </li>
+        {%- for title, link, link_attrs in drop_down_navigation %}
+        <li class="nav-item">
+          <a class="sk-nav-link nav-link nav-more-item-mobile-items" href="{{ link }}" {{ link_attrs }}>{{ title }}</a>
+        </li>
+        {%- endfor %}
+      </ul>
+      {%- if pagename != "search"%}
+      <div id="searchbox" role="search">
+          <div class="searchformwrapper">
+          <form class="search" action="{{ pathto('search') }}" method="get">
+            <input class="sk-search-text-input" type="text" name="q" aria-labelledby="searchlabel" />
+            <input class="sk-search-text-btn" type="submit" value="{{ _('Go') }}" />
+          </form>
+          </div>
+      </div>
+      {%- endif %}
+    </div>
+  </div>
+</nav>

docs/api_reference/themes/scikit-learn-modern/search.html

@@ -0,0 +1,16 @@
+{%- extends "basic/search.html" %}
+{% block extrahead %}
+  <script type="text/javascript" src="{{ pathto('_static/underscore.js', 1) }}"></script>
+  <script type="text/javascript" src="{{ pathto('searchindex.js', 1) }}" defer></script>
+  <script type="text/javascript" src="{{ pathto('_static/doctools.js', 1) }}"></script>
+  <script type="text/javascript" src="{{ pathto('_static/language_data.js', 1) }}"></script>
+  <script type="text/javascript" src="{{ pathto('_static/searchtools.js', 1) }}"></script>
+  <!-- <script type="text/javascript" src="{{ pathto('_static/sphinx_highlight.js', 1) }}"></script> -->
+  <script type="text/javascript">
+    $(document).ready(function() {
+      if (!Search.out) {
+        Search.init();
+      }
+    });
+  </script>
+{% endblock %}

docs/api_reference/themes/scikit-learn-modern/theme.conf

@@ -0,0 +1,8 @@
+[theme]
+inherit = basic
+pygments_style = default
+stylesheet = css/theme.css
+
+[options]
+link_to_live_contributing_page = false
+mathjax_path =

docs/docs_skeleton/docs/get_started/quickstart.mdx

@@ -47,7 +47,7 @@ import ChatModel from "@snippets/get_started/quickstart/chat_model.mdx"
 
 ## Prompt templates
 
-Most LLM applications do not pass user input directly into to an LLM. Usually they will add the user input to a larger piece of text, called a prompt template, that provides additional context on the specific task at hand.
+Most LLM applications do not pass user input directly into an LLM. Usually they will add the user input to a larger piece of text, called a prompt template, that provides additional context on the specific task at hand.
 
 In the previous example, the text we passed to the model contained instructions to generate a company name. For our application, it'd be great if the user only had to provide the description of a company/product, without having to worry about giving the model instructions.
 
@@ -138,7 +138,7 @@ The chains and agents we've looked at so far have been stateless, but for many a
 
 The Memory module gives you a way to maintain application state. The base Memory interface is simple: it lets you update state given the latest run inputs and outputs and it lets you modify (or contextualize) the next input using the stored state.
 
-There are a number of built-in memory systems. The simplest of these are is a buffer memory which just prepends the last few inputs/outputs to the current input - we will use this in the example below.
+There are a number of built-in memory systems. The simplest of these is a buffer memory which just prepends the last few inputs/outputs to the current input - we will use this in the example below.
 
 import MemoryLLM from "@snippets/get_started/quickstart/memory_llms.mdx"
 import MemoryChatModel from "@snippets/get_started/quickstart/memory_chat_models.mdx"
@@ -155,4 +155,4 @@ You can use Memory with chains and agents initialized with chat models. The main
 <MemoryChatModel/>
 
 </TabItem>
-</Tabs>
+</Tabs>

docs/docs_skeleton/docs/get_started/tutorials.mdx

@@ -0,0 +1,116 @@
+# Tutorials
+
+
+⛓ icon marks a new addition [last update 2023-06-20]
+
+---------------------
+
+### DeepLearning.AI course
+[LangChain for LLM Application Development](https://learn.deeplearning.ai/langchain) by Harrison Chase and [Andrew Ng](https://en.wikipedia.org/wiki/Andrew_Ng)
+
+### Handbook
+[LangChain AI Handbook](https://www.pinecone.io/learn/langchain/) By **James Briggs** and **Francisco Ingham**
+
+### Short Tutorials
+[LangChain Crash Course - Build apps with language models](https://youtu.be/LbT1yp6quS8) by [Patrick Loeber](https://www.youtube.com/@patloeber)
+
+[LangChain Crash Course: Build an AutoGPT app in 25 minutes](https://youtu.be/MlK6SIjcjE8) by [Nicholas Renotte](https://www.youtube.com/@NicholasRenotte)
+
+[LangChain Explained in 13 Minutes | QuickStart Tutorial for Beginners](https://youtu.be/aywZrzNaKjs) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
+
+
+##  Tutorials
+
+### [LangChain for Gen AI and LLMs](https://www.youtube.com/playlist?list=PLIUOU7oqGTLieV9uTIFMm6_4PXg-hlN6F) by [James Briggs](https://www.youtube.com/@jamesbriggs):
+- #1 [Getting Started with `GPT-3` vs. Open Source LLMs](https://youtu.be/nE2skSRWTTs)
+- #2 [Prompt Templates for `GPT 3.5` and other LLMs](https://youtu.be/RflBcK0oDH0)
+- #3 [LLM Chains using `GPT 3.5` and other LLMs](https://youtu.be/S8j9Tk0lZHU)
+- #4 [Chatbot Memory for `Chat-GPT`, `Davinci` + other LLMs](https://youtu.be/X05uK0TZozM)
+- #5 [Chat with OpenAI in LangChain](https://youtu.be/CnAgB3A5OlU)
+- #6 [Fixing LLM Hallucinations with Retrieval Augmentation in LangChain](https://youtu.be/kvdVduIJsc8)
+- #7 [LangChain Agents Deep Dive with GPT 3.5](https://youtu.be/jSP-gSEyVeI)
+- #8 [Create Custom Tools for Chatbots in LangChain](https://youtu.be/q-HNphrWsDE)
+- #9 [Build Conversational Agents with Vector DBs](https://youtu.be/H6bCqqw9xyI)
+- ⛓ #10 [Using NEW `MPT-7B` in Hugging Face and LangChain](https://youtu.be/DXpk9K7DgMo)
+
+
+### [LangChain 101](https://www.youtube.com/playlist?list=PLqZXAkvF1bPNQER9mLmDbntNfSpzdDIU5) by [Greg Kamradt (Data Indy)](https://www.youtube.com/@DataIndependent):
+- [What Is LangChain? - LangChain + `ChatGPT` Overview](https://youtu.be/_v_fgW2SkkQ)
+- [Quickstart Guide](https://youtu.be/kYRB-vJFy38)
+- [Beginner Guide To 7 Essential Concepts](https://youtu.be/2xxziIWmaSA)
+- [`OpenAI` + `Wolfram Alpha`](https://youtu.be/UijbzCIJ99g)
+- [Ask Questions On Your Custom (or Private) Files](https://youtu.be/EnT-ZTrcPrg)
+- [Connect `Google Drive Files` To `OpenAI`](https://youtu.be/IqqHqDcXLww)
+- [`YouTube Transcripts` + `OpenAI`](https://youtu.be/pNcQ5XXMgH4)
+- [Question A 300 Page Book (w/ `OpenAI` + `Pinecone`)](https://youtu.be/h0DHDp1FbmQ)
+- [Workaround `OpenAI's` Token Limit With Chain Types](https://youtu.be/f9_BWhCI4Zo)
+- [Build Your Own OpenAI + LangChain Web App in 23 Minutes](https://youtu.be/U_eV8wfMkXU)
+- [Working With The New `ChatGPT API`](https://youtu.be/e9P7FLi5Zy8)
+- [OpenAI + LangChain Wrote Me 100 Custom Sales Emails](https://youtu.be/y1pyAQM-3Bo)
+- [Structured Output From `OpenAI` (Clean Dirty Data)](https://youtu.be/KwAXfey-xQk)
+- [Connect `OpenAI` To +5,000 Tools (LangChain + `Zapier`)](https://youtu.be/7tNm0yiDigU)
+- [Use LLMs To Extract Data From Text (Expert Mode)](https://youtu.be/xZzvwR9jdPA)
+- [Extract Insights From Interview Transcripts Using LLMs](https://youtu.be/shkMOHwJ4SM)
+- [5 Levels Of LLM Summarizing: Novice to Expert](https://youtu.be/qaPMdcCqtWk)
+- ⛓ [Control Tone & Writing Style Of Your LLM Output](https://youtu.be/miBG-a3FuhU)
+- ⛓ [Build Your Own `AI Twitter Bot` Using LLMs](https://youtu.be/yLWLDjT01q8)
+- ⛓ [ChatGPT made my interview questions for me (`Streamlit` + LangChain)](https://youtu.be/zvoAMx0WKkw)
+- ⛓ [Function Calling via ChatGPT API - First Look With LangChain](https://youtu.be/0-zlUy7VUjg)
+
+
+### [LangChain How to and guides](https://www.youtube.com/playlist?list=PL8motc6AQftk1Bs42EW45kwYbyJ4jOdiZ) by [Sam Witteveen](https://www.youtube.com/@samwitteveenai):
+- [LangChain Basics - LLMs & PromptTemplates with Colab](https://youtu.be/J_0qvRt4LNk)
+- [LangChain Basics - Tools and Chains](https://youtu.be/hI2BY7yl_Ac)
+- [`ChatGPT API` Announcement & Code Walkthrough with LangChain](https://youtu.be/phHqvLHCwH4)
+- [Conversations with Memory (explanation & code walkthrough)](https://youtu.be/X550Zbz_ROE)
+- [Chat with `Flan20B`](https://youtu.be/VW5LBavIfY4)
+- [Using `Hugging Face Models` locally (code walkthrough)](https://youtu.be/Kn7SX2Mx_Jk)
+- [`PAL` : Program-aided Language Models with LangChain code](https://youtu.be/dy7-LvDu-3s)
+- [Building a Summarization System with LangChain and `GPT-3` - Part 1](https://youtu.be/LNq_2s_H01Y)
+- [Building a Summarization System with LangChain and `GPT-3` - Part 2](https://youtu.be/d-yeHDLgKHw)
+- [Microsoft's `Visual ChatGPT` using LangChain](https://youtu.be/7YEiEyfPF5U)
+- [LangChain Agents - Joining Tools and Chains with Decisions](https://youtu.be/ziu87EXZVUE)
+- [Comparing LLMs with LangChain](https://youtu.be/rFNG0MIEuW0)
+- [Using `Constitutional AI` in LangChain](https://youtu.be/uoVqNFDwpX4)
+- [Talking to `Alpaca` with LangChain - Creating an Alpaca Chatbot](https://youtu.be/v6sF8Ed3nTE)
+- [Talk to your `CSV` & `Excel` with LangChain](https://youtu.be/xQ3mZhw69bc)
+- [`BabyAGI`: Discover the Power of Task-Driven Autonomous Agents!](https://youtu.be/QBcDLSE2ERA)
+- [Improve your `BabyAGI` with LangChain](https://youtu.be/DRgPyOXZ-oE)
+- [Master `PDF` Chat with LangChain - Your essential guide to queries on documents](https://youtu.be/ZzgUqFtxgXI)
+- [Using LangChain with `DuckDuckGO` `Wikipedia` & `PythonREPL` Tools](https://youtu.be/KerHlb8nuVc)
+- [Building Custom Tools and Agents with LangChain (gpt-3.5-turbo)](https://youtu.be/biS8G8x8DdA)
+- [LangChain Retrieval QA Over Multiple Files with `ChromaDB`](https://youtu.be/3yPBVii7Ct0)
+- [LangChain Retrieval QA with Instructor Embeddings & `ChromaDB` for PDFs](https://youtu.be/cFCGUjc33aU)
+- [LangChain + Retrieval Local LLMs for Retrieval QA - No OpenAI!!!](https://youtu.be/9ISVjh8mdlA)
+- ⛓ [`Camel` + LangChain for Synthetic Data & Market Research](https://youtu.be/GldMMK6-_-g)
+- ⛓ [Information Extraction with LangChain & `Kor`](https://youtu.be/SW1ZdqH0rRQ)
+- ⛓ [Converting a LangChain App from OpenAI to OpenSource](https://youtu.be/KUDn7bVyIfc)
+- ⛓ [Using LangChain `Output Parsers` to get what you want out of LLMs](https://youtu.be/UVn2NroKQCw)
+- ⛓ [Building a LangChain Custom Medical Agent with Memory](https://youtu.be/6UFtRwWnHws)
+- ⛓ [Understanding `ReACT` with LangChain](https://youtu.be/Eug2clsLtFs)
+- ⛓ [`OpenAI Functions` + LangChain : Building a Multi Tool Agent](https://youtu.be/4KXK6c6TVXQ)
+- ⛓ [What can you do with 16K tokens in LangChain?](https://youtu.be/z2aCZBAtWXs)
+- ⛓ [Tagging and Extraction - Classification using `OpenAI Functions`](https://youtu.be/a8hMgIcUEnE)
+
+
+### [LangChain](https://www.youtube.com/playlist?list=PLVEEucA9MYhOu89CX8H3MBZqayTbcCTMr) by [Prompt Engineering](https://www.youtube.com/@engineerprompt):
+- [LangChain Crash Course — All You Need to Know to Build Powerful Apps with LLMs](https://youtu.be/5-fc4Tlgmro)
+- [Working with MULTIPLE `PDF` Files in LangChain: `ChatGPT` for your Data](https://youtu.be/s5LhRdh5fu4)
+- [`ChatGPT` for YOUR OWN `PDF` files with LangChain](https://youtu.be/TLf90ipMzfE)
+- [Talk to YOUR DATA without OpenAI APIs: LangChain](https://youtu.be/wrD-fZvT6UI)
+- ⛓ [Langchain: PDF Chat App (GUI) | ChatGPT for Your PDF FILES](https://youtu.be/RIWbalZ7sTo)
+- ⛓ [LangFlow: Build Chatbots without Writing Code](https://youtu.be/KJ-ux3hre4s)
+- ⛓ [LangChain: Giving Memory to LLMs](https://youtu.be/dxO6pzlgJiY)
+- ⛓ [BEST OPEN Alternative to `OPENAI's EMBEDDINGs` for Retrieval QA: LangChain](https://youtu.be/ogEalPMUCSY)
+
+
+### LangChain by [Chat with data](https://www.youtube.com/@chatwithdata)
+- [LangChain Beginner's Tutorial for `Typescript`/`Javascript`](https://youtu.be/bH722QgRlhQ)
+- [`GPT-4` Tutorial: How to Chat With Multiple `PDF` Files (~1000 pages of Tesla's 10-K Annual Reports)](https://youtu.be/Ix9WIZpArm0)
+- [`GPT-4` & LangChain Tutorial: How to Chat With A 56-Page `PDF` Document (w/`Pinecone`)](https://youtu.be/ih9PBGVVOO4)
+- [LangChain & Supabase Tutorial: How to Build a ChatGPT Chatbot For Your Website](https://youtu.be/R2FMzcsmQY8)
+- ⛓ [LangChain Agents: Build Personal Assistants For Your Data (Q&A with Harrison Chase and Mayo Oshin)](https://youtu.be/gVkF8cwfBLI)
+
+
+---------------------
+⛓ icon marks a new addition [last update 2023-06-20]

docs/docs_skeleton/docs/modules/data_connection/document_loaders/how_to/json.mdx

@@ -2,6 +2,8 @@
 
 >[JSON (JavaScript Object Notation)](https://en.wikipedia.org/wiki/JSON) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other serializable values).
 
+>[JSON Lines](https://jsonlines.org/) is a file format where each line is a valid JSON value.
+
 import Example from "@snippets/modules/data_connection/document_loaders/how_to/json.mdx"
 
 <Example/>

docs/docs_skeleton/docs/modules/data_connection/vectorstores/index.mdx

@@ -10,7 +10,7 @@ for you.
 
 ## Get started
 
-This walkthrough showcases basic functionality related to VectorStores. A key part of working with vector stores is creating the vector to put in them, which is usually created via embeddings. Therefore, it is recommended that you familiarize yourself with the [text embedding model](/docs/modules/model_io/models/embeddings.html) interfaces before diving into this.
+This walkthrough showcases basic functionality related to VectorStores. A key part of working with vector stores is creating the vector to put in them, which is usually created via embeddings. Therefore, it is recommended that you familiarize yourself with the [text embedding model](/docs/modules/data_connection/text_embedding/) interfaces before diving into this.
 
 import GetStarted from "@snippets/modules/data_connection/vectorstores/get_started.mdx"
 

docs/docs_skeleton/docs/modules/model_io/models/chat/how_to/chat_model_caching.mdx

@@ -0,0 +1,9 @@
+# Caching
+LangChain provides an optional caching layer for Chat Models. This is useful for two reasons:
+
+It can save you money by reducing the number of API calls you make to the LLM provider, if you're often requesting the same completion multiple times.
+It can speed up your application by reducing the number of API calls you make to the LLM provider.
+
+import CachingChat from "@snippets/modules/model_io/models/chat/how_to/chat_model_caching.mdx"
+
+<CachingChat/>

docs/docs_skeleton/docusaurus.config.js

@@ -7,7 +7,10 @@ const { ProvidePlugin } = require("webpack");
 const path = require("path");
 
 const examplesPath = path.resolve(__dirname, "..", "examples", "src");
-const snippetsPath = path.resolve(__dirname, "..", "snippets")
+const snippetsPath = path.resolve(__dirname, "..", "snippets");
+
+const baseLightCodeBlockTheme = require("prism-react-renderer/themes/vsLight");
+const baseDarkCodeBlockTheme = require("prism-react-renderer/themes/vsDark");
 
 /** @type {import('@docusaurus/types').Config} */
 const config = {
@@ -84,7 +87,6 @@ const config = {
       ({
         docs: {
           sidebarPath: require.resolve("./sidebars.js"),
-          editUrl: "https://github.com/hwchase17/langchain/edit/master/docs/",
           remarkPlugins: [
             [require("@docusaurus/remark-plugin-npm2yarn"), { sync: true }],
           ],
@@ -127,8 +129,20 @@ const config = {
         },
       },
       prism: {
-        theme: require("prism-react-renderer/themes/vsLight"),
-        darkTheme: require("prism-react-renderer/themes/vsDark"),
+        theme: {
+          ...baseLightCodeBlockTheme,
+          plain: {
+            ...baseLightCodeBlockTheme.plain,
+            backgroundColor: "#F5F5F5",
+          },
+        },
+        darkTheme: {
+          ...baseDarkCodeBlockTheme,
+          plain: {
+            ...baseDarkCodeBlockTheme.plain,
+            backgroundColor: "#222222",
+          },
+        },
       },
       image: "img/parrot-chainlink-icon.png",
       navbar: {

docs/docs_skeleton/package.json

@@ -23,7 +23,7 @@
     "@docusaurus/preset-classic": "2.4.0",
     "@docusaurus/remark-plugin-npm2yarn": "^2.4.0",
     "@mdx-js/react": "^1.6.22",
-    "@mendable/search": "^0.0.102",
+    "@mendable/search": "^0.0.112-beta.7",
     "clsx": "^1.2.1",
     "json-loader": "^0.5.7",
     "process": "^0.11.10",

docs/extras/additional_resources/youtube.mdx

@@ -1,6 +1,6 @@
-# YouTube tutorials
+# YouTube videos
 
-This is a collection of `LangChain` videos on `YouTube`.
+⛓ icon marks a new addition [last update 2023-06-20]
 
 ### [Official LangChain YouTube channel](https://www.youtube.com/@LangChain)
 
@@ -9,7 +9,6 @@ This is a collection of `LangChain` videos on `YouTube`.
 - [LangChain and Weaviate with Harrison Chase and Bob van Luijt - Weaviate Podcast #36](https://youtu.be/lhby7Ql7hbk) by [Weaviate • Vector Database](https://www.youtube.com/@Weaviate)
 - [LangChain Demo + Q&A with Harrison Chase](https://youtu.be/zaYTXQFR0_s?t=788) by [Full Stack Deep Learning](https://www.youtube.com/@FullStackDeepLearning)
 - [LangChain Agents: Build Personal Assistants For Your Data (Q&A with Harrison Chase and Mayo Oshin)](https://youtu.be/gVkF8cwfBLI) by [Chat with data](https://www.youtube.com/@chatwithdata)
-- ⛓️ [LangChain "Agents in Production" Webinar](https://youtu.be/k8GNCCs16F4) by [LangChain](https://www.youtube.com/@LangChain)
 
 ## Videos (sorted by views)
 
@@ -31,6 +30,9 @@ This is a collection of `LangChain` videos on `YouTube`.
 - [`Weaviate` + LangChain for LLM apps presented by Erika Cardenas](https://youtu.be/7AGj4Td5Lgw) by [`Weaviate` • Vector Database](https://www.youtube.com/@Weaviate)
 - [Langchain Overview — How to Use Langchain & `ChatGPT`](https://youtu.be/oYVYIq0lOtI) by [Python In Office](https://www.youtube.com/@pythoninoffice6568)
 - [Langchain Overview - How to Use Langchain & `ChatGPT`](https://youtu.be/oYVYIq0lOtI) by [Python In Office](https://www.youtube.com/@pythoninoffice6568)
+- [LangChain Tutorials](https://www.youtube.com/watch?v=FuqdVNB_8c0&list=PL9V0lbeJ69brU-ojMpU1Y7Ic58Tap0Cw6) by [Edrick](https://www.youtube.com/@edrickdch):
+  - [LangChain, Chroma DB, OpenAI Beginner Guide | ChatGPT with your PDF](https://youtu.be/FuqdVNB_8c0)
+  - [LangChain 101: The Complete Beginner's Guide](https://youtu.be/P3MAbZ2eMUI)
 - [Custom langchain Agent & Tools with memory. Turn any `Python function` into langchain tool with Gpt 3](https://youtu.be/NIG8lXk0ULg) by [echohive](https://www.youtube.com/@echohive)
 - [LangChain: Run Language Models Locally - `Hugging Face Models`](https://youtu.be/Xxxuw4_iCzw) by [Prompt Engineering](https://www.youtube.com/@engineerprompt)
 - [`ChatGPT` with any `YouTube` video using langchain and `chromadb`](https://youtu.be/TQZfB2bzVwU) by [echohive](https://www.youtube.com/@echohive)
@@ -46,154 +48,68 @@ This is a collection of `LangChain` videos on `YouTube`.
 - [Langchain + `Zapier` Agent](https://youtu.be/yribLAb-pxA) by [Merk](https://www.youtube.com/@merksworld)
 - [Connecting the Internet with `ChatGPT` (LLMs) using Langchain And Answers Your Questions](https://youtu.be/9Y0TBC63yZg) by [Kamalraj M M](https://www.youtube.com/@insightbuilder)
 - [Build More Powerful LLM Applications for Business’s with LangChain (Beginners Guide)](https://youtu.be/sp3-WLKEcBg) by[ No Code Blackbox](https://www.youtube.com/@nocodeblackbox)
-- ⛓️ [LangFlow LLM Agent Demo for 🦜🔗LangChain](https://youtu.be/zJxDHaWt-6o) by [Cobus Greyling](https://www.youtube.com/@CobusGreylingZA)
-- ⛓️ [Chatbot Factory: Streamline Python Chatbot Creation with LLMs and Langchain](https://youtu.be/eYer3uzrcuM) by [Finxter](https://www.youtube.com/@CobusGreylingZA)
-- ⛓️ [LangChain Tutorial - ChatGPT mit eigenen Daten](https://youtu.be/0XDLyY90E2c) by [Coding Crashkurse](https://www.youtube.com/@codingcrashkurse6429)
-- ⛓️ [Chat with a `CSV` | LangChain Agents Tutorial (Beginners)](https://youtu.be/tjeti5vXWOU) by [GoDataProf](https://www.youtube.com/@godataprof)
-- ⛓️ [Introdução ao Langchain - #Cortes - Live DataHackers](https://youtu.be/fw8y5VRei5Y) by [Prof. João Gabriel Lima](https://www.youtube.com/@profjoaogabriellima)
-- ⛓️ [LangChain: Level up `ChatGPT` !? | LangChain Tutorial Part 1](https://youtu.be/vxUGx8aZpDE) by [Code Affinity](https://www.youtube.com/@codeaffinitydev)
-- ⛓️ [KI schreibt krasses Youtube Skript 😲😳 | LangChain Tutorial Deutsch](https://youtu.be/QpTiXyK1jus) by [SimpleKI](https://www.youtube.com/@simpleki)
-- ⛓️ [Chat with Audio: Langchain, `Chroma DB`, OpenAI, and `Assembly AI`](https://youtu.be/Kjy7cx1r75g) by [AI Anytime](https://www.youtube.com/@AIAnytime)
-- ⛓️ [QA over documents with Auto vector index selection with Langchain router chains](https://youtu.be/9G05qybShv8) by [echohive](https://www.youtube.com/@echohive)
-- ⛓️ [Build your own custom LLM application with `Bubble.io` & Langchain (No Code & Beginner friendly)](https://youtu.be/O7NhQGu1m6c) by [No Code Blackbox](https://www.youtube.com/@nocodeblackbox)
-- ⛓️ [Simple App to Question Your Docs: Leveraging `Streamlit`, `Hugging Face Spaces`, LangChain, and `Claude`!](https://youtu.be/X4YbNECRr7o) by [Chris Alexiuk](https://www.youtube.com/@chrisalexiuk)
-- ⛓️ [LANGCHAIN AI- `ConstitutionalChainAI` + Databutton AI ASSISTANT Web App](https://youtu.be/5zIU6_rdJCU) by [Avra](https://www.youtube.com/@Avra_b)
-- ⛓️ [LANGCHAIN AI AUTONOMOUS AGENT WEB APP - 👶 `BABY AGI` 🤖 with EMAIL AUTOMATION using `DATABUTTON`](https://youtu.be/cvAwOGfeHgw) by [Avra](https://www.youtube.com/@Avra_b)
-- ⛓️ [The Future of Data Analysis: Using A.I. Models in Data Analysis (LangChain)](https://youtu.be/v_LIcVyg5dk) by [Absent Data](https://www.youtube.com/@absentdata)
-- ⛓️ [Memory in LangChain | Deep dive (python)](https://youtu.be/70lqvTFh_Yg) by [Eden Marco](https://www.youtube.com/@EdenMarco)
-- ⛓️ [9 LangChain UseCases | Beginner's Guide | 2023](https://youtu.be/zS8_qosHNMw) by [Data Science Basics](https://www.youtube.com/@datasciencebasics)
-- ⛓️ [Use Large Language Models in Jupyter Notebook | LangChain | Agents & Indexes](https://youtu.be/JSe11L1a_QQ) by [Abhinaw Tiwari](https://www.youtube.com/@AbhinawTiwariAT)
-- ⛓️ [How to Talk to Your Langchain Agent | `11 Labs` + `Whisper`](https://youtu.be/N4k459Zw2PU) by [VRSEN](https://www.youtube.com/@vrsen)
-- ⛓️ [LangChain Deep Dive: 5 FUN AI App Ideas To Build Quickly and Easily](https://youtu.be/mPYEPzLkeks) by [James NoCode](https://www.youtube.com/@jamesnocode)
-- ⛓️ [BEST OPEN Alternative to OPENAI's EMBEDDINGs for Retrieval QA: LangChain](https://youtu.be/ogEalPMUCSY) by [Prompt Engineering](https://www.youtube.com/@engineerprompt)
-- ⛓️ [LangChain 101: Models](https://youtu.be/T6c_XsyaNSQ) by [Mckay Wrigley](https://www.youtube.com/@realmckaywrigley)
-- ⛓️ [LangChain with JavaScript Tutorial #1 | Setup & Using LLMs](https://youtu.be/W3AoeMrg27o) by [Leon van Zyl](https://www.youtube.com/@leonvanzyl)
-- ⛓️ [LangChain Overview & Tutorial for Beginners: Build Powerful AI Apps Quickly & Easily (ZERO CODE)](https://youtu.be/iI84yym473Q) by [James NoCode](https://www.youtube.com/@jamesnocode)
-- ⛓️ [LangChain In Action: Real-World Use Case With Step-by-Step Tutorial](https://youtu.be/UO699Szp82M) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
-- ⛓️ [Summarizing and Querying Multiple Papers with LangChain](https://youtu.be/p_MQRWH5Y6k) by [Automata Learning Lab](https://www.youtube.com/@automatalearninglab)
-- ⛓️ [Using Langchain (and `Replit`) through `Tana`, ask `Google`/`Wikipedia`/`Wolfram Alpha` to fill out a table](https://youtu.be/Webau9lEzoI) by [Stian Håklev](https://www.youtube.com/@StianHaklev)
-- ⛓️ [Langchain PDF App (GUI) | Create a ChatGPT For Your `PDF` in Python](https://youtu.be/wUAUdEw5oxM) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
-- ⛓️ [Auto-GPT with LangChain 🔥 | Create Your Own Personal AI Assistant](https://youtu.be/imDfPmMKEjM) by [Data Science Basics](https://www.youtube.com/@datasciencebasics)
-- ⛓️ [Create Your OWN Slack AI Assistant with Python & LangChain](https://youtu.be/3jFXRNn2Bu8) by [Dave Ebbelaar](https://www.youtube.com/@daveebbelaar)
-- ⛓️ [How to Create LOCAL Chatbots with GPT4All and LangChain [Full Guide]](https://youtu.be/4p1Fojur8Zw) by [Liam Ottley](https://www.youtube.com/@LiamOttley)
-- ⛓️ [Build a `Multilingual PDF` Search App with LangChain, `Cohere` and `Bubble`](https://youtu.be/hOrtuumOrv8) by [Menlo Park Lab](https://www.youtube.com/@menloparklab)
-- ⛓️ [Building a LangChain Agent (code-free!) Using `Bubble` and `Flowise`](https://youtu.be/jDJIIVWTZDE) by [Menlo Park Lab](https://www.youtube.com/@menloparklab)
-- ⛓️ [Build a LangChain-based Semantic PDF Search App with No-Code Tools Bubble and Flowise](https://youtu.be/s33v5cIeqA4) by [Menlo Park Lab](https://www.youtube.com/@menloparklab)
-- ⛓️ [LangChain Memory Tutorial | Building a ChatGPT Clone in Python](https://youtu.be/Cwq91cj2Pnc) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
-- ⛓️ [ChatGPT For Your DATA | Chat with Multiple Documents Using LangChain](https://youtu.be/TeDgIDqQmzs) by [Data Science Basics](https://www.youtube.com/@datasciencebasics)
-- ⛓️ [`Llama Index`: Chat with Documentation using URL Loader](https://youtu.be/XJRoDEctAwA) by [Merk](https://www.youtube.com/@merksworld)
-- ⛓️ [Using OpenAI, LangChain, and `Gradio` to Build Custom GenAI Applications](https://youtu.be/1MsmqMg3yUc) by [David Hundley](https://www.youtube.com/@dkhundley)
-- ⛓️ [LangChain, Chroma DB, OpenAI Beginner Guide | ChatGPT with your PDF](https://youtu.be/FuqdVNB_8c0)
-- [LangChain Crash Course: Build an AutoGPT app in 25 minutes](https://youtu.be/MlK6SIjcjE8) by [Nicholas Renotte](https://www.youtube.com/@NicholasRenotte)
-- [LangChain Crash Course - Build apps with language models](https://youtu.be/LbT1yp6quS8) by [Patrick Loeber](https://www.youtube.com/@patloeber)
-- [LangChain Explained in 13 Minutes | QuickStart Tutorial for Beginners](https://youtu.be/aywZrzNaKjs) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
+- [LangFlow LLM Agent Demo for 🦜🔗LangChain](https://youtu.be/zJxDHaWt-6o) by [Cobus Greyling](https://www.youtube.com/@CobusGreylingZA)
+- [Chatbot Factory: Streamline Python Chatbot Creation with LLMs and Langchain](https://youtu.be/eYer3uzrcuM) by [Finxter](https://www.youtube.com/@CobusGreylingZA)
+- [LangChain Tutorial - ChatGPT mit eigenen Daten](https://youtu.be/0XDLyY90E2c) by [Coding Crashkurse](https://www.youtube.com/@codingcrashkurse6429)
+- [Chat with a `CSV` | LangChain Agents Tutorial (Beginners)](https://youtu.be/tjeti5vXWOU) by [GoDataProf](https://www.youtube.com/@godataprof)
+- [Introdução ao Langchain - #Cortes - Live DataHackers](https://youtu.be/fw8y5VRei5Y) by [Prof. João Gabriel Lima](https://www.youtube.com/@profjoaogabriellima)
+- [LangChain: Level up `ChatGPT` !? | LangChain Tutorial Part 1](https://youtu.be/vxUGx8aZpDE) by [Code Affinity](https://www.youtube.com/@codeaffinitydev)
+- [KI schreibt krasses Youtube Skript 😲😳 | LangChain Tutorial Deutsch](https://youtu.be/QpTiXyK1jus) by [SimpleKI](https://www.youtube.com/@simpleki)
+- [Chat with Audio: Langchain, `Chroma DB`, OpenAI, and `Assembly AI`](https://youtu.be/Kjy7cx1r75g) by [AI Anytime](https://www.youtube.com/@AIAnytime)
+- [QA over documents with Auto vector index selection with Langchain router chains](https://youtu.be/9G05qybShv8) by [echohive](https://www.youtube.com/@echohive)
+- [Build your own custom LLM application with `Bubble.io` & Langchain (No Code & Beginner friendly)](https://youtu.be/O7NhQGu1m6c) by [No Code Blackbox](https://www.youtube.com/@nocodeblackbox)
+- [Simple App to Question Your Docs: Leveraging `Streamlit`, `Hugging Face Spaces`, LangChain, and `Claude`!](https://youtu.be/X4YbNECRr7o) by [Chris Alexiuk](https://www.youtube.com/@chrisalexiuk)
+- [LANGCHAIN AI- `ConstitutionalChainAI` + Databutton AI ASSISTANT Web App](https://youtu.be/5zIU6_rdJCU) by [Avra](https://www.youtube.com/@Avra_b)
+- [LANGCHAIN AI AUTONOMOUS AGENT WEB APP - 👶 `BABY AGI` 🤖 with EMAIL AUTOMATION using `DATABUTTON`](https://youtu.be/cvAwOGfeHgw) by [Avra](https://www.youtube.com/@Avra_b)
+- [The Future of Data Analysis: Using A.I. Models in Data Analysis (LangChain)](https://youtu.be/v_LIcVyg5dk) by [Absent Data](https://www.youtube.com/@absentdata)
+- [Memory in LangChain | Deep dive (python)](https://youtu.be/70lqvTFh_Yg) by [Eden Marco](https://www.youtube.com/@EdenMarco)
+- [9 LangChain UseCases | Beginner's Guide | 2023](https://youtu.be/zS8_qosHNMw) by [Data Science Basics](https://www.youtube.com/@datasciencebasics)
+- [Use Large Language Models in Jupyter Notebook | LangChain | Agents & Indexes](https://youtu.be/JSe11L1a_QQ) by [Abhinaw Tiwari](https://www.youtube.com/@AbhinawTiwariAT)
+- [How to Talk to Your Langchain Agent | `11 Labs` + `Whisper`](https://youtu.be/N4k459Zw2PU) by [VRSEN](https://www.youtube.com/@vrsen)
+- [LangChain Deep Dive: 5 FUN AI App Ideas To Build Quickly and Easily](https://youtu.be/mPYEPzLkeks) by [James NoCode](https://www.youtube.com/@jamesnocode)
+- [BEST OPEN Alternative to OPENAI's EMBEDDINGs for Retrieval QA: LangChain](https://youtu.be/ogEalPMUCSY) by [Prompt Engineering](https://www.youtube.com/@engineerprompt)
+- [LangChain 101: Models](https://youtu.be/T6c_XsyaNSQ) by [Mckay Wrigley](https://www.youtube.com/@realmckaywrigley)
+- [LangChain with JavaScript Tutorial #1 | Setup & Using LLMs](https://youtu.be/W3AoeMrg27o) by [Leon van Zyl](https://www.youtube.com/@leonvanzyl)
+- [LangChain Overview & Tutorial for Beginners: Build Powerful AI Apps Quickly & Easily (ZERO CODE)](https://youtu.be/iI84yym473Q) by [James NoCode](https://www.youtube.com/@jamesnocode)
+- [LangChain In Action: Real-World Use Case With Step-by-Step Tutorial](https://youtu.be/UO699Szp82M) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
+- [Summarizing and Querying Multiple Papers with LangChain](https://youtu.be/p_MQRWH5Y6k) by [Automata Learning Lab](https://www.youtube.com/@automatalearninglab)
+- [Using Langchain (and `Replit`) through `Tana`, ask `Google`/`Wikipedia`/`Wolfram Alpha` to fill out a table](https://youtu.be/Webau9lEzoI) by [Stian Håklev](https://www.youtube.com/@StianHaklev)
+- [Langchain PDF App (GUI) | Create a ChatGPT For Your `PDF` in Python](https://youtu.be/wUAUdEw5oxM) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
+- [Auto-GPT with LangChain 🔥 | Create Your Own Personal AI Assistant](https://youtu.be/imDfPmMKEjM) by [Data Science Basics](https://www.youtube.com/@datasciencebasics)
+- [Create Your OWN Slack AI Assistant with Python & LangChain](https://youtu.be/3jFXRNn2Bu8) by [Dave Ebbelaar](https://www.youtube.com/@daveebbelaar)
+- [How to Create LOCAL Chatbots with GPT4All and LangChain [Full Guide]](https://youtu.be/4p1Fojur8Zw) by [Liam Ottley](https://www.youtube.com/@LiamOttley)
+- [Build a `Multilingual PDF` Search App with LangChain, `Cohere` and `Bubble`](https://youtu.be/hOrtuumOrv8) by [Menlo Park Lab](https://www.youtube.com/@menloparklab)
+- [Building a LangChain Agent (code-free!) Using `Bubble` and `Flowise`](https://youtu.be/jDJIIVWTZDE) by [Menlo Park Lab](https://www.youtube.com/@menloparklab)
+- [Build a LangChain-based Semantic PDF Search App with No-Code Tools Bubble and Flowise](https://youtu.be/s33v5cIeqA4) by [Menlo Park Lab](https://www.youtube.com/@menloparklab)
+- [LangChain Memory Tutorial | Building a ChatGPT Clone in Python](https://youtu.be/Cwq91cj2Pnc) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
+- [ChatGPT For Your DATA | Chat with Multiple Documents Using LangChain](https://youtu.be/TeDgIDqQmzs) by [Data Science Basics](https://www.youtube.com/@datasciencebasics)
+- [`Llama Index`: Chat with Documentation using URL Loader](https://youtu.be/XJRoDEctAwA) by [Merk](https://www.youtube.com/@merksworld)
+- [Using OpenAI, LangChain, and `Gradio` to Build Custom GenAI Applications](https://youtu.be/1MsmqMg3yUc) by [David Hundley](https://www.youtube.com/@dkhundley)
+- [LangChain, Chroma DB, OpenAI Beginner Guide | ChatGPT with your PDF](https://youtu.be/FuqdVNB_8c0)
+- ⛓ [Build AI chatbot with custom knowledge base using OpenAI API and GPT Index](https://youtu.be/vDZAZuaXf48) by [Irina Nik](https://www.youtube.com/@irina_nik)
+- ⛓ [Build Your Own Auto-GPT Apps with LangChain (Python Tutorial)](https://youtu.be/NYSWn1ipbgg) by [Dave Ebbelaar](https://www.youtube.com/@daveebbelaar)
+- ⛓ [Chat with Multiple `PDFs` | LangChain App Tutorial in Python (Free LLMs and Embeddings)](https://youtu.be/dXxQ0LR-3Hg) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
+- ⛓ [Chat with a `CSV` | `LangChain Agents` Tutorial (Beginners)](https://youtu.be/tjeti5vXWOU) by [Alejandro AO - Software & Ai](https://www.youtube.com/@alejandro_ao)
+- ⛓ [Create Your Own ChatGPT with `PDF` Data in 5 Minutes (LangChain Tutorial)](https://youtu.be/au2WVVGUvc8) by [Liam Ottley](https://www.youtube.com/@LiamOttley)
+- ⛓ [Using ChatGPT with YOUR OWN Data. This is magical. (LangChain OpenAI API)](https://youtu.be/9AXP7tCI9PI) by [TechLead](https://www.youtube.com/@TechLead)
+- ⛓ [Build a Custom Chatbot with OpenAI: `GPT-Index` & LangChain | Step-by-Step Tutorial](https://youtu.be/FIDv6nc4CgU) by [Fabrikod](https://www.youtube.com/@fabrikod)
+- ⛓ [`Flowise` is an open source no-code UI visual tool to build 🦜🔗LangChain applications](https://youtu.be/CovAPtQPU0k) by [Cobus Greyling](https://www.youtube.com/@CobusGreylingZA)
+- ⛓ [LangChain & GPT 4 For Data Analysis: The `Pandas` Dataframe Agent](https://youtu.be/rFQ5Kmkd4jc) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
+- ⛓ [`GirlfriendGPT` - AI girlfriend with LangChain](https://youtu.be/LiN3D1QZGQw) by [Toolfinder AI](https://www.youtube.com/@toolfinderai)
+- ⛓ [`PrivateGPT`: Chat to your FILES OFFLINE and FREE [Installation and Tutorial]](https://youtu.be/G7iLllmx4qc) by [Prompt Engineering](https://www.youtube.com/@engineerprompt)
+- ⛓ [How to build with Langchain 10x easier | ⛓️ LangFlow & `Flowise`](https://youtu.be/Ya1oGL7ZTvU) by [AI Jason](https://www.youtube.com/@AIJasonZ)
+- ⛓ [Getting Started With LangChain In 20 Minutes- Build Celebrity Search Application](https://youtu.be/_FpT1cwcSLg) by [Krish Naik](https://www.youtube.com/@krishnaik06)
 
 
-## Tutorial Series
 
-
-⛓ icon marks a new addition [last update 2023-05-15]
-
-### DeepLearning.AI course
-⛓[LangChain for LLM Application Development](https://learn.deeplearning.ai/langchain) by Harrison Chase presented by [Andrew Ng](https://en.wikipedia.org/wiki/Andrew_Ng)
-
-### Handbook
-[LangChain AI Handbook](https://www.pinecone.io/learn/langchain/) By **James Briggs** and **Francisco Ingham**
-
-### Tutorials
-[LangChain Tutorials](https://www.youtube.com/watch?v=FuqdVNB_8c0&list=PL9V0lbeJ69brU-ojMpU1Y7Ic58Tap0Cw6) by [Edrick](https://www.youtube.com/@edrickdch):
-- ⛓ [LangChain, Chroma DB, OpenAI Beginner Guide | ChatGPT with your PDF](https://youtu.be/FuqdVNB_8c0)
-- ⛓ [LangChain 101: The Complete Beginner's Guide](https://youtu.be/P3MAbZ2eMUI)
-
-[LangChain Crash Course: Build an AutoGPT app in 25 minutes](https://youtu.be/MlK6SIjcjE8) by [Nicholas Renotte](https://www.youtube.com/@NicholasRenotte)
-
-
-[LangChain Crash Course - Build apps with language models](https://youtu.be/LbT1yp6quS8) by [Patrick Loeber](https://www.youtube.com/@patloeber)
-
-
-[LangChain Explained in 13 Minutes | QuickStart Tutorial for Beginners](https://youtu.be/aywZrzNaKjs) by [Rabbitmetrics](https://www.youtube.com/@rabbitmetrics)
-
-
-### [LangChain for Gen AI and LLMs](https://www.youtube.com/playlist?list=PLIUOU7oqGTLieV9uTIFMm6_4PXg-hlN6F) by [James Briggs](https://www.youtube.com/@jamesbriggs):
-- #1 [Getting Started with `GPT-3` vs. Open Source LLMs](https://youtu.be/nE2skSRWTTs)
-- #2 [Prompt Templates for `GPT 3.5` and other LLMs](https://youtu.be/RflBcK0oDH0)
-- #3 [LLM Chains using `GPT 3.5` and other LLMs](https://youtu.be/S8j9Tk0lZHU)
-- #4 [Chatbot Memory for `Chat-GPT`, `Davinci` + other LLMs](https://youtu.be/X05uK0TZozM)
-- #5 [Chat with OpenAI in LangChain](https://youtu.be/CnAgB3A5OlU)
-- ⛓ #6 [Fixing LLM Hallucinations with Retrieval Augmentation in LangChain](https://youtu.be/kvdVduIJsc8)
-- ⛓ #7 [LangChain Agents Deep Dive with GPT 3.5](https://youtu.be/jSP-gSEyVeI)
-- ⛓ #8 [Create Custom Tools for Chatbots in LangChain](https://youtu.be/q-HNphrWsDE)
-- ⛓ #9 [Build Conversational Agents with Vector DBs](https://youtu.be/H6bCqqw9xyI)
-
-
-### [LangChain 101](https://www.youtube.com/playlist?list=PLqZXAkvF1bPNQER9mLmDbntNfSpzdDIU5) by [Data Independent](https://www.youtube.com/@DataIndependent):
-- [What Is LangChain? - LangChain + `ChatGPT` Overview](https://youtu.be/_v_fgW2SkkQ)
-- [Quickstart Guide](https://youtu.be/kYRB-vJFy38)
-- [Beginner Guide To 7 Essential Concepts](https://youtu.be/2xxziIWmaSA)
-- [`OpenAI` + `Wolfram Alpha`](https://youtu.be/UijbzCIJ99g)
-- [Ask Questions On Your Custom (or Private) Files](https://youtu.be/EnT-ZTrcPrg)
-- [Connect `Google Drive Files` To `OpenAI`](https://youtu.be/IqqHqDcXLww)
-- [`YouTube Transcripts` + `OpenAI`](https://youtu.be/pNcQ5XXMgH4)
-- [Question A 300 Page Book (w/ `OpenAI` + `Pinecone`)](https://youtu.be/h0DHDp1FbmQ)
-- [Workaround `OpenAI's` Token Limit With Chain Types](https://youtu.be/f9_BWhCI4Zo)
-- [Build Your Own OpenAI + LangChain Web App in 23 Minutes](https://youtu.be/U_eV8wfMkXU)
-- [Working With The New `ChatGPT API`](https://youtu.be/e9P7FLi5Zy8)
-- [OpenAI + LangChain Wrote Me 100 Custom Sales Emails](https://youtu.be/y1pyAQM-3Bo)
-- [Structured Output From `OpenAI` (Clean Dirty Data)](https://youtu.be/KwAXfey-xQk)
-- [Connect `OpenAI` To +5,000 Tools (LangChain + `Zapier`)](https://youtu.be/7tNm0yiDigU)
-- [Use LLMs To Extract Data From Text (Expert Mode)](https://youtu.be/xZzvwR9jdPA)
-- ⛓ [Extract Insights From Interview Transcripts Using LLMs](https://youtu.be/shkMOHwJ4SM)
-- ⛓ [5 Levels Of LLM Summarizing: Novice to Expert](https://youtu.be/qaPMdcCqtWk)
-
-
-### [LangChain How to and guides](https://www.youtube.com/playlist?list=PL8motc6AQftk1Bs42EW45kwYbyJ4jOdiZ) by [Sam Witteveen](https://www.youtube.com/@samwitteveenai):
-- [LangChain Basics - LLMs & PromptTemplates with Colab](https://youtu.be/J_0qvRt4LNk)
-- [LangChain Basics - Tools and Chains](https://youtu.be/hI2BY7yl_Ac)
-- [`ChatGPT API` Announcement & Code Walkthrough with LangChain](https://youtu.be/phHqvLHCwH4)
-- [Conversations with Memory (explanation & code walkthrough)](https://youtu.be/X550Zbz_ROE)
-- [Chat with `Flan20B`](https://youtu.be/VW5LBavIfY4)
-- [Using `Hugging Face Models` locally (code walkthrough)](https://youtu.be/Kn7SX2Mx_Jk)
-- [`PAL` : Program-aided Language Models with LangChain code](https://youtu.be/dy7-LvDu-3s)
-- [Building a Summarization System with LangChain and `GPT-3` - Part 1](https://youtu.be/LNq_2s_H01Y)
-- [Building a Summarization System with LangChain and `GPT-3` - Part 2](https://youtu.be/d-yeHDLgKHw)
-- [Microsoft's `Visual ChatGPT` using LangChain](https://youtu.be/7YEiEyfPF5U)
-- [LangChain Agents - Joining Tools and Chains with Decisions](https://youtu.be/ziu87EXZVUE)
-- [Comparing LLMs with LangChain](https://youtu.be/rFNG0MIEuW0)
-- [Using `Constitutional AI` in LangChain](https://youtu.be/uoVqNFDwpX4)
-- [Talking to `Alpaca` with LangChain - Creating an Alpaca Chatbot](https://youtu.be/v6sF8Ed3nTE)
-- [Talk to your `CSV` & `Excel` with LangChain](https://youtu.be/xQ3mZhw69bc)
-- [`BabyAGI`: Discover the Power of Task-Driven Autonomous Agents!](https://youtu.be/QBcDLSE2ERA)
-- [Improve your `BabyAGI` with LangChain](https://youtu.be/DRgPyOXZ-oE)
-- ⛓ [Master `PDF` Chat with LangChain - Your essential guide to queries on documents](https://youtu.be/ZzgUqFtxgXI)
-- ⛓ [Using LangChain with `DuckDuckGO` `Wikipedia` & `PythonREPL` Tools](https://youtu.be/KerHlb8nuVc)
-- ⛓ [Building Custom Tools and Agents with LangChain (gpt-3.5-turbo)](https://youtu.be/biS8G8x8DdA)
-- ⛓ [LangChain Retrieval QA Over Multiple Files with `ChromaDB`](https://youtu.be/3yPBVii7Ct0)
-- ⛓ [LangChain Retrieval QA with Instructor Embeddings & `ChromaDB` for PDFs](https://youtu.be/cFCGUjc33aU)
-- ⛓ [LangChain + Retrieval Local LLMs for Retrieval QA - No OpenAI!!!](https://youtu.be/9ISVjh8mdlA)
-
-
-### [LangChain](https://www.youtube.com/playlist?list=PLVEEucA9MYhOu89CX8H3MBZqayTbcCTMr) by [Prompt Engineering](https://www.youtube.com/@engineerprompt):
-- [LangChain Crash Course — All You Need to Know to Build Powerful Apps with LLMs](https://youtu.be/5-fc4Tlgmro)
-- [Working with MULTIPLE `PDF` Files in LangChain: `ChatGPT` for your Data](https://youtu.be/s5LhRdh5fu4)
-- [`ChatGPT` for YOUR OWN `PDF` files with LangChain](https://youtu.be/TLf90ipMzfE)
-- [Talk to YOUR DATA without OpenAI APIs: LangChain](https://youtu.be/wrD-fZvT6UI)
-- ⛓️ [CHATGPT For WEBSITES: Custom ChatBOT](https://youtu.be/RBnuhhmD21U)
-
-
-### LangChain by [Chat with data](https://www.youtube.com/@chatwithdata)
-- [LangChain Beginner's Tutorial for `Typescript`/`Javascript`](https://youtu.be/bH722QgRlhQ)
-- [`GPT-4` Tutorial: How to Chat With Multiple `PDF` Files (~1000 pages of Tesla's 10-K Annual Reports)](https://youtu.be/Ix9WIZpArm0)
-- [`GPT-4` & LangChain Tutorial: How to Chat With A 56-Page `PDF` Document (w/`Pinecone`)](https://youtu.be/ih9PBGVVOO4)
-- ⛓ [LangChain & Supabase Tutorial: How to Build a ChatGPT Chatbot For Your Website](https://youtu.be/R2FMzcsmQY8)
-
-
-### [Get SH\*T Done with Prompt Engineering and LangChain](https://www.youtube.com/watch?v=muXbPpG_ys4&list=PLEJK-H61Xlwzm5FYLDdKt_6yibO33zoMW) by [Venelin Valkov](https://www.youtube.com/@venelin_valkov)
+### [Prompt Engineering and LangChain](https://www.youtube.com/watch?v=muXbPpG_ys4&list=PLEJK-H61Xlwzm5FYLDdKt_6yibO33zoMW) by [Venelin Valkov](https://www.youtube.com/@venelin_valkov)
 - [Getting Started with LangChain: Load Custom Data, Run OpenAI Models, Embeddings and `ChatGPT`](https://www.youtube.com/watch?v=muXbPpG_ys4)
 - [Loaders, Indexes & Vectorstores in LangChain: Question Answering on `PDF` files with `ChatGPT`](https://www.youtube.com/watch?v=FQnvfR8Dmr0)
 - [LangChain Models: `ChatGPT`, `Flan Alpaca`, `OpenAI Embeddings`, Prompt Templates & Streaming](https://www.youtube.com/watch?v=zy6LiK5F5-s)
 - [LangChain Chains: Use `ChatGPT` to Build Conversational Agents, Summaries and Q&A on Text With LLMs](https://www.youtube.com/watch?v=h1tJZQPcimM)
 - [Analyze Custom CSV Data with `GPT-4` using Langchain](https://www.youtube.com/watch?v=Ew3sGdX8at4)
-- ⛓ [Build ChatGPT Chatbots with LangChain Memory: Understanding and Implementing Memory in Conversations](https://youtu.be/CyuUlf54wTs)
+- [Build ChatGPT Chatbots with LangChain Memory: Understanding and Implementing Memory in Conversations](https://youtu.be/CyuUlf54wTs)
+
 
 ---------------------
-⛓ icon marks a new addition [last update 2023-05-15]
+⛓ icon marks a new addition [last update 2023-06-20]

docs/extras/ecosystem/integrations/airtable.md

@@ -0,0 +1,28 @@
+# Airtable
+
+>[Airtable](https://en.wikipedia.org/wiki/Airtable) is a cloud collaboration service.
+`Airtable` is a spreadsheet-database hybrid, with the features of a database but applied to a spreadsheet. 
+> The fields in an Airtable table are similar to cells in a spreadsheet, but have types such as 'checkbox', 
+> 'phone number', and 'drop-down list', and can reference file attachments like images.
+
+>Users can create a database, set up column types, add records, link tables to one another, collaborate, sort records
+> and publish views to external websites.
+
+## Installation and Setup
+
+```bash
+pip install pyairtable
+```
+
+* Get your [API key](https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens).
+* Get the [ID of your base](https://airtable.com/developers/web/api/introduction).
+* Get the [table ID from the table url](https://www.highviewapps.com/kb/where-can-i-find-the-airtable-base-id-and-table-id/#:~:text=Both%20the%20Airtable%20Base%20ID,URL%20that%20begins%20with%20tbl).
+
+## Document Loader
+
+
+```python
+from langchain.document_loaders import AirtableLoader
+```
+
+See an [example](/docs/modules/data_connection/document_loaders/integrations/airtable.html).

docs/extras/ecosystem/integrations/amazon_api_gateway.mdx

@@ -0,0 +1,73 @@
+# Amazon API Gateway
+
+[Amazon API Gateway](https://aws.amazon.com/api-gateway/) is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the "front door" for applications to access data, business logic, or functionality from your backend services. Using API Gateway, you can create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications. API Gateway supports containerized and serverless workloads, as well as web applications.
+
+API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. API Gateway has no minimum fees or startup costs. You pay for the API calls you receive and the amount of data transferred out and, with the API Gateway tiered pricing model, you can reduce your cost as your API usage scales.
+
+## LLM
+
+See a [usage example](/docs/modules/model_io/models/llms/integrations/amazon_api_gateway_example.html).
+
+```python
+from langchain.llms import AmazonAPIGateway
+
+api_url = "https://<api_gateway_id>.execute-api.<region>.amazonaws.com/LATEST/HF"
+llm = AmazonAPIGateway(api_url=api_url)
+
+# These are sample parameters for Falcon 40B Instruct Deployed from Amazon SageMaker JumpStart
+parameters = {
+    "max_new_tokens": 100,
+    "num_return_sequences": 1,
+    "top_k": 50,
+    "top_p": 0.95,
+    "do_sample": False,
+    "return_full_text": True,
+    "temperature": 0.2,
+}
+
+prompt = "what day comes after Friday?"
+llm.model_kwargs = parameters
+llm(prompt)
+>>> 'what day comes after Friday?\nSaturday'
+```
+
+## Agent
+
+```python
+from langchain.agents import load_tools
+from langchain.agents import initialize_agent
+from langchain.agents import AgentType
+from langchain.llms import AmazonAPIGateway
+
+api_url = "https://<api_gateway_id>.execute-api.<region>.amazonaws.com/LATEST/HF"
+llm = AmazonAPIGateway(api_url=api_url)
+
+parameters = {
+    "max_new_tokens": 50,
+    "num_return_sequences": 1,
+    "top_k": 250,
+    "top_p": 0.25,
+    "do_sample": False,
+    "temperature": 0.1,
+}
+
+llm.model_kwargs = parameters
+
+# Next, let's load some tools to use. Note that the `llm-math` tool uses an LLM, so we need to pass that in.
+tools = load_tools(["python_repl", "llm-math"], llm=llm)
+
+# Finally, let's initialize an agent with the tools, the language model, and the type of agent we want to use.
+agent = initialize_agent(
+    tools,
+    llm,
+    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
+    verbose=True,
+)
+
+# Now let's test it out!
+agent.run("""
+Write a Python script that prints "Hello, world!"
+""")
+
+>>> 'Hello, world!'
+```

docs/extras/ecosystem/integrations/arthur_tracking.ipynb

@@ -0,0 +1,198 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Arthur"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[Arthur](https://arthur.ai) is a model monitoring and observability platform.\n",
+    "\n",
+    "The following guide shows how to run a registered chat LLM with the Arthur callback handler to automatically log model inferences to Arthur.\n",
+    "\n",
+    "If you do not have a model currently onboarded to Arthur, visit our [onboarding guide for generative text models](https://docs.arthur.ai/user-guide/walkthroughs/model-onboarding/generative_text_onboarding.html). For more information about how to use the Arthur SDK, visit our [docs](https://docs.arthur.ai/)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "id": "y8ku6X96sebl"
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.callbacks import ArthurCallbackHandler\n",
+    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.schema import HumanMessage"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Place Arthur credentials here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "id": "Me3prhqjsoqz"
+   },
+   "outputs": [],
+   "source": [
+    "arthur_url = \"https://app.arthur.ai\"\n",
+    "arthur_login = \"your-arthur-login-username-here\"\n",
+    "arthur_model_id = \"your-arthur-model-id-here\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Create Langchain LLM with Arthur callback handler"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "id": "9Hq9snQasynA"
+   },
+   "outputs": [],
+   "source": [
+    "def make_langchain_chat_llm(chat_model=):\n",
+    "    return ChatOpenAI(\n",
+    "        streaming=True,\n",
+    "        temperature=0.1,\n",
+    "        callbacks=[\n",
+    "            StreamingStdOutCallbackHandler(),\n",
+    "            ArthurCallbackHandler.from_credentials(\n",
+    "                arthur_model_id, \n",
+    "                arthur_url=arthur_url, \n",
+    "                arthur_login=arthur_login)\n",
+    "        ])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Please enter password for admin: ········\n"
+     ]
+    }
+   ],
+   "source": [
+    "chatgpt = make_langchain_chat_llm()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "aXRyj50Ls8eP"
+   },
+   "source": [
+    "Running the chat LLM with this `run` function will save the chat history in an ongoing list so that the conversation can reference earlier messages and log each response to the Arthur platform. You can view the history of this model's inferences on your [model dashboard page](https://app.arthur.ai/).\n",
+    "\n",
+    "Enter `q` to quit the run loop"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "id": "4taWSbN-s31Y"
+   },
+   "outputs": [],
+   "source": [
+    "def run(llm):\n",
+    "    history = []\n",
+    "    while True:\n",
+    "        user_input = input(\"\\n>>> input >>>\\n>>>: \")\n",
+    "        if user_input == 'q': break\n",
+    "        history.append(HumanMessage(content=user_input))\n",
+    "        history.append(llm(history))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "id": "MEx8nWJps-EG"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      ">>> input >>>\n",
+      ">>>: What is a callback handler?\n",
+      "A callback handler, also known as a callback function or callback method, is a piece of code that is executed in response to a specific event or condition. It is commonly used in programming languages that support event-driven or asynchronous programming paradigms.\n",
+      "\n",
+      "The purpose of a callback handler is to provide a way for developers to define custom behavior that should be executed when a certain event occurs. Instead of waiting for a result or blocking the execution, the program registers a callback function and continues with other tasks. When the event is triggered, the callback function is invoked, allowing the program to respond accordingly.\n",
+      "\n",
+      "Callback handlers are commonly used in various scenarios, such as handling user input, responding to network requests, processing asynchronous operations, and implementing event-driven architectures. They provide a flexible and modular way to handle events and decouple different components of a system.\n",
+      ">>> input >>>\n",
+      ">>>: What do I need to do to get the full benefits of this\n",
+      "To get the full benefits of using a callback handler, you should consider the following:\n",
+      "\n",
+      "1. Understand the event or condition: Identify the specific event or condition that you want to respond to with a callback handler. This could be user input, network requests, or any other asynchronous operation.\n",
+      "\n",
+      "2. Define the callback function: Create a function that will be executed when the event or condition occurs. This function should contain the desired behavior or actions you want to take in response to the event.\n",
+      "\n",
+      "3. Register the callback function: Depending on the programming language or framework you are using, you may need to register or attach the callback function to the appropriate event or condition. This ensures that the callback function is invoked when the event occurs.\n",
+      "\n",
+      "4. Handle the callback: Implement the necessary logic within the callback function to handle the event or condition. This could involve updating the user interface, processing data, making further requests, or triggering other actions.\n",
+      "\n",
+      "5. Consider error handling: It's important to handle any potential errors or exceptions that may occur within the callback function. This ensures that your program can gracefully handle unexpected situations and prevent crashes or undesired behavior.\n",
+      "\n",
+      "6. Maintain code readability and modularity: As your codebase grows, it's crucial to keep your callback handlers organized and maintainable. Consider using design patterns or architectural principles to structure your code in a modular and scalable way.\n",
+      "\n",
+      "By following these steps, you can leverage the benefits of callback handlers, such as asynchronous and event-driven programming, improved responsiveness, and modular code design.\n",
+      ">>> input >>>\n",
+      ">>>: q\n"
+     ]
+    }
+   ],
+   "source": [
+    "run(chatgpt)"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

docs/extras/ecosystem/integrations/brave_search.mdx

@@ -0,0 +1,36 @@
+# Brave Search
+
+
+>[Brave Search](https://en.wikipedia.org/wiki/Brave_Search) is a search engine developed by Brave Software.
+> - `Brave Search` uses its own web index. As of May 2022, it covered over 10 billion pages and was used to serve 92% 
+> of search results without relying on any third-parties, with the remainder being retrieved 
+> server-side from the Bing API or (on an opt-in basis) client-side from Google. According 
+> to Brave, the index was kept "intentionally smaller than that of Google or Bing" in order to 
+> help avoid spam and other low-quality content, with the disadvantage that "Brave Search is 
+> not yet as good as Google in recovering long-tail queries."
+>- `Brave Search Premium`: As of April 2023 Brave Search is an ad-free website, but it will 
+> eventually switch to a new model that will include ads and premium users will get an ad-free experience.
+> User data including IP addresses won't be collected from its users by default. A premium account 
+> will be required for opt-in data-collection.
+
+
+## Installation and Setup
+
+To get access to the Brave Search API, you need to [create an account and get an API key](https://api.search.brave.com/app/dashboard).
+
+
+## Document Loader
+
+See a [usage example](/docs/modules/data_connection/document_loaders/integrations/brave_search.html).
+
+```python
+from langchain.document_loaders import BraveSearchLoader
+```
+
+## Tool
+
+See a [usage example](/docs/modules/agents/tools/integrations/brave_search.html).
+
+```python
+from langchain.tools import BraveSearch
+```

docs/extras/ecosystem/integrations/cassandra.mdx

@@ -1,19 +1,31 @@
 # Cassandra
 
->[Cassandra](https://en.wikipedia.org/wiki/Apache_Cassandra) is a free and open-source, distributed, wide-column 
+>[Apache Cassandra®](https://cassandra.apache.org/) is a free and open-source, distributed, wide-column
 > store, NoSQL database management system designed to handle large amounts of data across many commodity servers, 
-> providing high availability with no single point of failure. `Cassandra` offers support for clusters spanning 
+> providing high availability with no single point of failure. Cassandra offers support for clusters spanning
 > multiple datacenters, with asynchronous masterless replication allowing low latency operations for all clients. 
-> `Cassandra` was designed to implement a combination of `Amazon's Dynamo` distributed storage and replication 
-> techniques combined with `Google's Bigtable` data and storage engine model.
+> Cassandra was designed to implement a combination of _Amazon's Dynamo_ distributed storage and replication
+> techniques combined with _Google's Bigtable_ data and storage engine model.
  
 ## Installation and Setup
 
 ```bash
-pip install cassandra-drive
+pip install cassandra-driver
+pip install cassio
 ```
 
 
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/cassandra.html).
+
+```python
+from langchain.memory import CassandraChatMessageHistory
+```
+
+
+
 ## Memory
 
 See a [usage example](/docs/modules/memory/integrations/cassandra_chat_message_history.html).

docs/extras/ecosystem/integrations/databricks/databricks.ipynb

@@ -1,273 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "707d13a7",
-   "metadata": {},
-   "source": [
-    "# Databricks\n",
-    "\n",
-    "This notebook covers how to connect to the [Databricks runtimes](https://docs.databricks.com/runtime/index.html) and [Databricks SQL](https://www.databricks.com/product/databricks-sql) using the SQLDatabase wrapper of LangChain.\n",
-    "It is broken into 3 parts: installation and setup, connecting to Databricks, and examples."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0076d072",
-   "metadata": {},
-   "source": [
-    "## Installation and Setup"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "739b489b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "!pip install databricks-sql-connector"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "73113163",
-   "metadata": {},
-   "source": [
-    "## Connecting to Databricks\n",
-    "\n",
-    "You can connect to [Databricks runtimes](https://docs.databricks.com/runtime/index.html) and [Databricks SQL](https://www.databricks.com/product/databricks-sql) using the `SQLDatabase.from_databricks()` method.\n",
-    "\n",
-    "### Syntax\n",
-    "```python\n",
-    "SQLDatabase.from_databricks(\n",
-    "    catalog: str,\n",
-    "    schema: str,\n",
-    "    host: Optional[str] = None,\n",
-    "    api_token: Optional[str] = None,\n",
-    "    warehouse_id: Optional[str] = None,\n",
-    "    cluster_id: Optional[str] = None,\n",
-    "    engine_args: Optional[dict] = None,\n",
-    "    **kwargs: Any)\n",
-    "```\n",
-    "### Required Parameters\n",
-    "* `catalog`: The catalog name in the Databricks database.\n",
-    "* `schema`: The schema name in the catalog.\n",
-    "\n",
-    "### Optional Parameters\n",
-    "There following parameters are optional. When executing the method in a Databricks notebook, you don't need to provide them in most of the cases.\n",
-    "* `host`: The Databricks workspace hostname, excluding 'https://' part. Defaults to 'DATABRICKS_HOST' environment variable or current workspace if in a Databricks notebook.\n",
-    "* `api_token`: The Databricks personal access token for accessing the Databricks SQL warehouse or the cluster. Defaults to 'DATABRICKS_TOKEN' environment variable or a temporary one is generated if in a Databricks notebook.\n",
-    "* `warehouse_id`: The warehouse ID in the Databricks SQL.\n",
-    "* `cluster_id`: The cluster ID in the Databricks Runtime. If running in a Databricks notebook and both 'warehouse_id' and 'cluster_id' are None, it uses the ID of the cluster the notebook is attached to.\n",
-    "* `engine_args`: The arguments to be used when connecting Databricks.\n",
-    "* `**kwargs`: Additional keyword arguments for the `SQLDatabase.from_uri` method."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b11c7e48",
-   "metadata": {},
-   "source": [
-    "## Examples"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "8102bca0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Connecting to Databricks with SQLDatabase wrapper\n",
-    "from langchain import SQLDatabase\n",
-    "\n",
-    "db = SQLDatabase.from_databricks(catalog=\"samples\", schema=\"nyctaxi\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "9dd36f58",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Creating a OpenAI Chat LLM wrapper\n",
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "llm = ChatOpenAI(temperature=0, model_name=\"gpt-4\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "5b5c5f1a",
-   "metadata": {},
-   "source": [
-    "### SQL Chain example\n",
-    "\n",
-    "This example demonstrates the use of the [SQL Chain](https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html) for answering a question over a Databricks database."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "36f2270b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain import SQLDatabaseChain\n",
-    "\n",
-    "db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "4e2b5f25",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
-      "What is the average duration of taxi rides that start between midnight and 6am?\n",
-      "SQLQuery:\u001b[32;1m\u001b[1;3mSELECT AVG(UNIX_TIMESTAMP(tpep_dropoff_datetime) - UNIX_TIMESTAMP(tpep_pickup_datetime)) as avg_duration\n",
-      "FROM trips\n",
-      "WHERE HOUR(tpep_pickup_datetime) >= 0 AND HOUR(tpep_pickup_datetime) < 6\u001b[0m\n",
-      "SQLResult: \u001b[33;1m\u001b[1;3m[(987.8122786304605,)]\u001b[0m\n",
-      "Answer:\u001b[32;1m\u001b[1;3mThe average duration of taxi rides that start between midnight and 6am is 987.81 seconds.\u001b[0m\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'The average duration of taxi rides that start between midnight and 6am is 987.81 seconds.'"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "db_chain.run(\n",
-    "    \"What is the average duration of taxi rides that start between midnight and 6am?\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e496d5e5",
-   "metadata": {},
-   "source": [
-    "### SQL Database Agent example\n",
-    "\n",
-    "This example demonstrates the use of the [SQL Database Agent](/docs/modules/agents/toolkits/sql_database.html) for answering questions over a Databricks database."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "9918e86a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain.agents import create_sql_agent\n",
-    "from langchain.agents.agent_toolkits import SQLDatabaseToolkit\n",
-    "\n",
-    "toolkit = SQLDatabaseToolkit(db=db, llm=llm)\n",
-    "agent = create_sql_agent(llm=llm, toolkit=toolkit, verbose=True)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "c484a76e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3mAction: list_tables_sql_db\n",
-      "Action Input: \u001b[0m\n",
-      "Observation: \u001b[38;5;200m\u001b[1;3mtrips\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI should check the schema of the trips table to see if it has the necessary columns for trip distance and duration.\n",
-      "Action: schema_sql_db\n",
-      "Action Input: trips\u001b[0m\n",
-      "Observation: \u001b[33;1m\u001b[1;3m\n",
-      "CREATE TABLE trips (\n",
-      "\ttpep_pickup_datetime TIMESTAMP, \n",
-      "\ttpep_dropoff_datetime TIMESTAMP, \n",
-      "\ttrip_distance FLOAT, \n",
-      "\tfare_amount FLOAT, \n",
-      "\tpickup_zip INT, \n",
-      "\tdropoff_zip INT\n",
-      ") USING DELTA\n",
-      "\n",
-      "/*\n",
-      "3 rows from trips table:\n",
-      "tpep_pickup_datetime\ttpep_dropoff_datetime\ttrip_distance\tfare_amount\tpickup_zip\tdropoff_zip\n",
-      "2016-02-14 16:52:13+00:00\t2016-02-14 17:16:04+00:00\t4.94\t19.0\t10282\t10171\n",
-      "2016-02-04 18:44:19+00:00\t2016-02-04 18:46:00+00:00\t0.28\t3.5\t10110\t10110\n",
-      "2016-02-17 17:13:57+00:00\t2016-02-17 17:17:55+00:00\t0.7\t5.0\t10103\t10023\n",
-      "*/\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mThe trips table has the necessary columns for trip distance and duration. I will write a query to find the longest trip distance and its duration.\n",
-      "Action: query_checker_sql_db\n",
-      "Action Input: SELECT trip_distance, tpep_dropoff_datetime - tpep_pickup_datetime as duration FROM trips ORDER BY trip_distance DESC LIMIT 1\u001b[0m\n",
-      "Observation: \u001b[31;1m\u001b[1;3mSELECT trip_distance, tpep_dropoff_datetime - tpep_pickup_datetime as duration FROM trips ORDER BY trip_distance DESC LIMIT 1\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mThe query is correct. I will now execute it to find the longest trip distance and its duration.\n",
-      "Action: query_sql_db\n",
-      "Action Input: SELECT trip_distance, tpep_dropoff_datetime - tpep_pickup_datetime as duration FROM trips ORDER BY trip_distance DESC LIMIT 1\u001b[0m\n",
-      "Observation: \u001b[36;1m\u001b[1;3m[(30.6, '0 00:43:31.000000000')]\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3mI now know the final answer.\n",
-      "Final Answer: The longest trip distance is 30.6 miles and it took 43 minutes and 31 seconds.\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "'The longest trip distance is 30.6 miles and it took 43 minutes and 31 seconds.'"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "agent.run(\"What is the longest trip distance and how long did it take?\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/extras/ecosystem/integrations/dataforseo.mdx

@@ -0,0 +1,51 @@
+# DataForSEO
+
+This page provides instructions on how to use the DataForSEO search APIs within LangChain.
+
+## Installation and Setup
+
+- Get a DataForSEO API Access login and password, and set them as environment variables (`DATAFORSEO_LOGIN` and `DATAFORSEO_PASSWORD` respectively). You can find it in your dashboard.
+
+## Wrappers
+
+### Utility
+
+The DataForSEO utility wraps the API. To import this utility, use:
+
+```python
+from langchain.utilities import DataForSeoAPIWrapper
+```
+
+For a detailed walkthrough of this wrapper, see [this notebook](/docs/modules/agents/tools/integrations/dataforseo.ipynb).
+
+### Tool
+
+You can also load this wrapper as a Tool to use with an Agent:
+
+```python
+from langchain.agents import load_tools
+tools = load_tools(["dataforseo-api-search"])
+```
+
+## Example usage
+
+```python
+dataforseo = DataForSeoAPIWrapper(api_login="your_login", api_password="your_password")
+result = dataforseo.run("Bill Gates")
+print(result)
+```
+
+## Environment Variables
+
+You can store your DataForSEO API Access login and password as environment variables. The wrapper will automatically check for these environment variables if no values are provided:
+
+```python
+import os
+
+os.environ["DATAFORSEO_LOGIN"] = "your_login"
+os.environ["DATAFORSEO_PASSWORD"] = "your_password"
+
+dataforseo = DataForSeoAPIWrapper()
+result = dataforseo.run("weather in Los Angeles")
+print(result)
+```

docs/extras/ecosystem/integrations/flyte.mdx

@@ -0,0 +1,153 @@
+# Flyte
+
+> [Flyte](https://github.com/flyteorg/flyte) is an open-source orchestrator that facilitates building production-grade data and ML pipelines.
+> It is built for scalability and reproducibility, leveraging Kubernetes as its underlying platform.
+
+The purpose of this notebook is to demonstrate the integration of a `FlyteCallback` into your Flyte task, enabling you to effectively monitor and track your LangChain experiments.
+
+## Installation & Setup
+
+- Install the Flytekit library by running the command `pip install flytekit`.
+- Install the Flytekit-Envd plugin by running the command `pip install flytekitplugins-envd`.
+- Install LangChain by running the command `pip install langchain`.
+- Install [Docker](https://docs.docker.com/engine/install/) on your system.
+
+## Flyte Tasks
+
+A Flyte [task](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task.html) serves as the foundational building block of Flyte.
+To execute LangChain experiments, you need to write Flyte tasks that define the specific steps and operations involved.
+
+NOTE: The [getting started guide](https://docs.flyte.org/projects/cookbook/en/latest/index.html) offers detailed, step-by-step instructions on installing Flyte locally and running your initial Flyte pipeline.
+
+First, import the necessary dependencies to support your LangChain experiments.
+
+```python
+import os
+
+from flytekit import ImageSpec, task
+from langchain.agents import AgentType, initialize_agent, load_tools
+from langchain.callbacks import FlyteCallbackHandler
+from langchain.chains import LLMChain
+from langchain.chat_models import ChatOpenAI
+from langchain.prompts import PromptTemplate
+from langchain.schema import HumanMessage
+```
+
+Set up the necessary environment variables to utilize the OpenAI API and Serp API:
+
+```python
+# Set OpenAI API key
+os.environ["OPENAI_API_KEY"] = "<your_openai_api_key>"
+
+# Set Serp API key
+os.environ["SERPAPI_API_KEY"] = "<your_serp_api_key>"
+```
+
+Replace `<your_openai_api_key>` and `<your_serp_api_key>` with your respective API keys obtained from OpenAI and Serp API.
+
+To guarantee reproducibility of your pipelines, Flyte tasks are containerized.
+Each Flyte task must be associated with an image, which can either be shared across the entire Flyte [workflow](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/basic_workflow.html) or provided separately for each task.
+
+To streamline the process of supplying the required dependencies for each Flyte task, you can initialize an [`ImageSpec`](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/image_spec/image_spec.html) object.
+This approach automatically triggers a Docker build, alleviating the need for users to manually create a Docker image.
+
+```python
+custom_image = ImageSpec(
+    name="langchain-flyte",
+    packages=[
+        "langchain",
+        "openai",
+        "spacy",
+        "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0.tar.gz",
+        "textstat",
+        "google-search-results",
+    ],
+    registry="<your-registry>",
+)
+```
+
+You have the flexibility to push the Docker image to a registry of your preference.
+[Docker Hub](https://hub.docker.com/) or [GitHub Container Registry (GHCR)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) is a convenient option to begin with.
+
+Once you have selected a registry, you can proceed to create Flyte tasks that log the LangChain metrics to Flyte Deck.
+
+The following examples demonstrate tasks related to OpenAI LLM, chains and agent with tools:
+
+### LLM
+
+```python
+@task(disable_deck=False, container_image=custom_image)
+def langchain_llm() -> str:
+    llm = ChatOpenAI(
+        model_name="gpt-3.5-turbo",
+        temperature=0.2,
+        callbacks=[FlyteCallbackHandler()],
+    )
+    return llm([HumanMessage(content="Tell me a joke")]).content
+```
+
+### Chain
+
+```python
+@task(disable_deck=False, container_image=custom_image)
+def langchain_chain() -> list[dict[str, str]]:
+    template = """You are a playwright. Given the title of play, it is your job to write a synopsis for that title.
+Title: {title}
+Playwright: This is a synopsis for the above play:"""
+    llm = ChatOpenAI(
+        model_name="gpt-3.5-turbo",
+        temperature=0,
+        callbacks=[FlyteCallbackHandler()],
+    )
+    prompt_template = PromptTemplate(input_variables=["title"], template=template)
+    synopsis_chain = LLMChain(
+        llm=llm, prompt=prompt_template, callbacks=[FlyteCallbackHandler()]
+    )
+    test_prompts = [
+        {
+            "title": "documentary about good video games that push the boundary of game design"
+        },
+    ]
+    return synopsis_chain.apply(test_prompts)
+```
+
+### Agent
+
+```python
+@task(disable_deck=False, container_image=custom_image)
+def langchain_agent() -> str:
+    llm = OpenAI(
+        model_name="gpt-3.5-turbo",
+        temperature=0,
+        callbacks=[FlyteCallbackHandler()],
+    )
+    tools = load_tools(
+        ["serpapi", "llm-math"], llm=llm, callbacks=[FlyteCallbackHandler()]
+    )
+    agent = initialize_agent(
+        tools,
+        llm,
+        agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
+        callbacks=[FlyteCallbackHandler()],
+        verbose=True,
+    )
+    return agent.run(
+        "Who is Leonardo DiCaprio's girlfriend? Could you calculate her current age and raise it to the power of 0.43?"
+    )
+```
+
+These tasks serve as a starting point for running your LangChain experiments within Flyte.
+
+## Execute the Flyte Tasks on Kubernetes
+
+To execute the Flyte tasks on the configured Flyte backend, use the following command:
+
+```bash
+pyflyte run --image <your-image> langchain_flyte.py langchain_llm
+```
+
+This command will initiate the execution of the `langchain_llm` task on the Flyte backend. You can trigger the remaining two tasks in a similar manner.
+
+The metrics will be displayed on the Flyte UI as follows:
+
+![LangChain LLM](https://ik.imagekit.io/c8zl7irwkdda/Screenshot_2023-06-20_at_1.23.29_PM_MZYeG0dKa.png?updatedAt=1687247642993)

docs/extras/ecosystem/integrations/grobid.mdx

@@ -0,0 +1,44 @@
+# Grobid
+
+This page covers how to use the Grobid to parse articles for LangChain.
+It is seperated into two parts: installation and running the server
+
+## Installation and Setup
+#Ensure You have Java installed
+!apt-get install -y openjdk-11-jdk -q
+!update-alternatives --set java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
+
+#Clone and install the Grobid Repo
+import os
+!git clone https://github.com/kermitt2/grobid.git
+os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-11-openjdk-amd64"
+os.chdir('grobid')
+!./gradlew clean install
+
+#Run the server, 
+get_ipython().system_raw('nohup ./gradlew run > grobid.log 2>&1 &')
+
+You can now use the GrobidParser to produce documents
+```python
+from langchain.document_loaders.parsers import GrobidParser
+from langchain.document_loaders.generic import GenericLoader
+
+#Produce chunks from article paragraphs
+loader = GenericLoader.from_filesystem(
+    "/Users/31treehaus/Desktop/Papers/",
+    glob="*",
+    suffixes=[".pdf"],
+    parser= GrobidParser(segment_sentences=False)
+)
+docs = loader.load()
+
+#Produce chunks from article sentences
+loader = GenericLoader.from_filesystem(
+    "/Users/31treehaus/Desktop/Papers/",
+    glob="*",
+    suffixes=[".pdf"],
+    parser= GrobidParser(segment_sentences=True)
+)
+docs = loader.load()
+```
+Chunk metadata will include bboxes although these are a bit funky to parse, see https://grobid.readthedocs.io/en/latest/Coordinates-in-PDF/

docs/extras/ecosystem/integrations/hologres.mdx

@@ -0,0 +1,23 @@
+# Hologres
+
+>[Hologres](https://www.alibabacloud.com/help/en/hologres/latest/introduction) is a unified real-time data warehousing service developed by Alibaba Cloud. You can use Hologres to write, update, process, and analyze large amounts of data in real time. 
+>`Hologres` supports standard `SQL` syntax, is compatible with `PostgreSQL`, and supports most PostgreSQL functions. Hologres supports online analytical processing (OLAP) and ad hoc analysis for up to petabytes of data, and provides high-concurrency and low-latency online data services. 
+
+>`Hologres` provides **vector database** functionality by adopting [Proxima](https://www.alibabacloud.com/help/en/hologres/latest/vector-processing).
+>`Proxima` is a high-performance software library developed by `Alibaba DAMO Academy`. It allows you to search for the nearest neighbors of vectors. Proxima provides higher stability and performance than similar open source software such as Faiss. Proxima allows you to search for similar text or image embeddings with high throughput and low latency. Hologres is deeply integrated with Proxima to provide a high-performance vector search service.
+
+## Installation and Setup
+
+Click [here](https://www.alibabacloud.com/zh/product/hologres) to fast deploy a Hologres cloud instance.
+
+```bash
+pip install psycopg2
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/hologres.html).
+
+```python
+from langchain.vectorstores import Hologres
+```

docs/extras/ecosystem/integrations/jina.mdx

@@ -16,3 +16,59 @@ There exists a Jina Embeddings wrapper, which you can access with
 from langchain.embeddings import JinaEmbeddings
 ```
 For a more detailed walkthrough of this, see [this notebook](/docs/modules/data_connection/text_embedding/integrations/jina.html)
+
+## Deployment
+
+[Langchain-serve](https://github.com/jina-ai/langchain-serve), powered by Jina, helps take LangChain apps to production with easy to use REST/WebSocket APIs and Slack bots. 
+
+### Usage
+
+Install the package from PyPI. 
+
+```bash
+pip install langchain-serve
+```
+
+Wrap your LangChain app with the `@serving` decorator. 
+
+```python
+# app.py
+from lcserve import serving
+
+@serving
+def ask(input: str) -> str:
+    from langchain import LLMChain, OpenAI
+    from langchain.agents import AgentExecutor, ZeroShotAgent
+    
+    tools = [...] # list of tools
+    prompt = ZeroShotAgent.create_prompt(
+        tools, input_variables=["input", "agent_scratchpad"],
+    )
+    llm_chain = LLMChain(llm=OpenAI(temperature=0), prompt=prompt)
+    agent = ZeroShotAgent(
+        llm_chain=llm_chain, allowed_tools=[tool.name for tool in tools]
+    )
+    agent_executor = AgentExecutor.from_agent_and_tools(
+        agent=agent, 
+        tools=tools, 
+        verbose=True,
+    )
+    return agent_executor.run(input)
+```
+
+Deploy on Jina AI Cloud with `lc-serve deploy jcloud app`. Once deployed, we can send a POST request to the API endpoint to get a response.
+
+```bash
+curl -X 'POST' 'https://<your-app>.wolf.jina.ai/ask' \
+ -d '{
+  "input": "Your Quesion here?",
+  "envs": {
+     "OPENAI_API_KEY": "sk-***"
+  }
+}'
+```
+
+You can also self-host the app on your infrastructure with Docker-compose or Kubernetes. See [here](https://github.com/jina-ai/langchain-serve#-self-host-llm-apps-with-docker-compose-or-kubernetes) for more details.
+
+
+Langchain-serve also allows to deploy the apps with WebSocket APIs and Slack Bots both on [Jina AI Cloud](https://cloud.jina.ai/) or self-hosted infrastructure. 

docs/extras/ecosystem/integrations/marqo.md

@@ -0,0 +1,31 @@
+# Marqo
+
+This page covers how to use the Marqo ecosystem within LangChain.
+
+### **What is Marqo?**
+
+Marqo is a tensor search engine that uses embeddings stored in in-memory HNSW indexes to achieve cutting edge search speeds. Marqo can scale to hundred-million document indexes with horizontal index sharding and allows for async and non-blocking data upload and search. Marqo uses the latest machine learning models from PyTorch, Huggingface, OpenAI and more. You can start with a pre-configured model or bring your own. The built in ONNX support and conversion allows for faster inference and higher throughput on both CPU and GPU.
+
+Because Marqo include its own inference your documents can have a mix of text and images, you can bring Marqo indexes with data from your other systems into the langchain ecosystem without having to worry about your embeddings being compatible. 
+
+Deployment of Marqo is flexible, you can get started yourself with our docker image or [contact us about our managed cloud offering!](https://www.marqo.ai/pricing)
+
+To run Marqo locally with our docker image, [see our getting started.](https://docs.marqo.ai/latest/)
+
+## Installation and Setup
+- Install the Python SDK with `pip install marqo`
+
+## Wrappers
+
+### VectorStore
+
+There exists a wrapper around Marqo indexes, allowing you to use them within the vectorstore framework. Marqo lets you select from a range of models for generating embeddings and exposes some preprocessing configurations.
+
+The Marqo vectorstore can also work with existing multimodel indexes where your documents have a mix of images and text, for more information refer to [our documentation](https://docs.marqo.ai/latest/#multi-modal-and-cross-modal-search). Note that instaniating the Marqo vectorstore with an existing multimodal index will disable the ability to add any new documents to it via the langchain vectorstore `add_texts` method.
+
+To import this vectorstore:
+```python
+from langchain.vectorstores import Marqo
+```
+
+For a more detailed walkthrough of the Marqo wrapper and some of its unique features, see [this notebook](../modules/indexes/vectorstores/examples/marqo.ipynb)

docs/extras/ecosystem/integrations/motherduck.mdx

@@ -0,0 +1,50 @@
+# Motherduck
+
+>[Motherduck](https://motherduck.com/) is a managed DuckDB-in-the-cloud service.
+
+## Installation and Setup
+
+First, you need to install `duckdb` python package.
+
+```bash
+pip install duckdb
+```
+
+You will also need to sign up for an account at [Motherduck](https://motherduck.com/)
+
+After that, you should set up a connection string - we mostly integrate with Motherduck through SQLAlchemy.
+The connection string is likely in the form:
+
+```
+token="..."
+
+conn_str = f"duckdb:///md:{token}@my_db"
+```
+
+## SQLChain
+
+You can use the SQLChain to query data in your Motherduck instance in natural language.
+
+```
+from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
+db = SQLDatabase.from_uri(conn_str)
+db_chain = SQLDatabaseChain.from_llm(OpenAI(temperature=0), db, verbose=True)
+```
+
+From here, see the [SQL Chain](/docs/modules/chains/popular/sqlite.html) documentation on how to use.
+
+
+## LLMCache
+
+You can also easily use Motherduck to cache LLM requests.
+Once again this is done through the SQLAlchemy wrapper.
+
+```
+import sqlalchemy
+eng = sqlalchemy.create_engine(conn_str)
+langchain.llm_cache = SQLAlchemyCache(engine=eng)
+```
+
+From here, see the [LLM Caching](/docs/modules/model_io/models/llms/how_to/llm_caching) documentation on how to use.
+
+

docs/extras/ecosystem/integrations/myscale.mdx

@@ -25,7 +25,7 @@ There are two ways to set up parameters for myscale index.
 1. Environment Variables
 
     Before you run the app, please set the environment variable with `export`:
-    `export MYSCALE_URL='<your-endpoints-url>' MYSCALE_PORT=<your-endpoints-port> MYSCALE_USERNAME=<your-username> MYSCALE_PASSWORD=<your-password> ...`
+    `export MYSCALE_HOST='<your-endpoints-url>' MYSCALE_PORT=<your-endpoints-port> MYSCALE_USERNAME=<your-username> MYSCALE_PASSWORD=<your-password> ...`
 
     You can easily find your account, password and other info on our SaaS. For details please refer to [this document](https://docs.myscale.com/en/cluster-management/)
     Every attributes under `MyScaleSettings` can be set with prefix `MYSCALE_` and is case insensitive.

docs/extras/ecosystem/integrations/openllm.mdx

@@ -0,0 +1,70 @@
+# OpenLLM
+
+This page demonstrates how to use [OpenLLM](https://github.com/bentoml/OpenLLM)
+with LangChain.
+
+`OpenLLM` is an open platform for operating large language models (LLMs) in
+production. It enables developers to easily run inference with any open-source
+LLMs, deploy to the cloud or on-premises, and build powerful AI apps.
+
+## Installation and Setup
+
+Install the OpenLLM package via PyPI:
+
+```bash
+pip install openllm
+```
+
+## LLM
+
+OpenLLM supports a wide range of open-source LLMs as well as serving users' own
+fine-tuned LLMs. Use `openllm model` command to see all available models that
+are pre-optimized for OpenLLM.
+
+## Wrappers
+
+There is a OpenLLM Wrapper which supports loading LLM in-process or accessing a
+remote OpenLLM server:
+
+```python
+from langchain.llms import OpenLLM
+```
+
+### Wrapper for OpenLLM server
+
+This wrapper supports connecting to an OpenLLM server via HTTP or gRPC. The
+OpenLLM server can run either locally or on the cloud.
+
+To try it out locally, start an OpenLLM server:
+
+```bash
+openllm start flan-t5
+```
+
+Wrapper usage:
+
+```python
+from langchain.llms import OpenLLM
+
+llm = OpenLLM(server_url='http://localhost:3000')
+
+llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
+```
+
+### Wrapper for Local Inference
+
+You can also use the OpenLLM wrapper to load LLM in current Python process for
+running inference.
+
+```python
+from langchain.llms import OpenLLM
+
+llm = OpenLLM(model_name="dolly-v2", model_id='databricks/dolly-v2-7b')
+
+llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
+```
+
+### Usage
+
+For a more detailed walkthrough of the OpenLLM Wrapper, see the
+[example notebook](/docs/modules/model_io/models/llms/integrations/openllm.html)

docs/extras/ecosystem/integrations/rockset.mdx

@@ -0,0 +1,19 @@
+# Rockset
+
+>[Rockset](https://rockset.com/product/) is a real-time analytics database service for serving low latency, high concurrency analytical queries at scale. It builds a Converged Index™ on structured and semi-structured data with an efficient store for vector embeddings. Its support for running SQL on schemaless data makes it a perfect choice for running vector search with metadata filters. 
+
+## Installation and Setup
+
+Make sure you have Rockset account and go to the web console to get the API key. Details can be found on [the website](https://rockset.com/docs/rest-api/).
+
+```bash
+pip install rockset
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/rockset.html).
+
+```python
+from langchain.vectorstores import RocksetDB
+```

docs/extras/ecosystem/integrations/singlestoredb.mdx

@@ -0,0 +1,20 @@
+# SingleStoreDB
+
+>[SingleStoreDB](https://singlestore.com/) is a high-performance distributed SQL database that supports deployment both in the [cloud](https://www.singlestore.com/cloud/) and on-premises. It provides vector storage, and vector functions including [dot_product](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/dot_product.html) and [euclidean_distance](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/euclidean_distance.html), thereby supporting AI applications that require text similarity matching. 
+
+## Installation and Setup
+
+There are several ways to establish a [connection](https://singlestoredb-python.labs.singlestore.com/generated/singlestoredb.connect.html) to the database. You can either set up environment variables or pass named parameters to the `SingleStoreDB constructor`. 
+Alternatively, you may provide these parameters to the `from_documents` and `from_texts` methods.
+
+```bash
+pip install singlestoredb
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/singlestoredb.html).
+
+```python
+from langchain.vectorstores import SingleStoreDB
+```

docs/extras/ecosystem/integrations/sklearn.mdx

@@ -1,15 +1,14 @@
 # scikit-learn
 
-This page covers how to use the scikit-learn package within LangChain.
-It is broken into two parts: installation and setup, and then references to specific scikit-learn wrappers.
+>[scikit-learn](https://scikit-learn.org/stable/) is an open source collection of machine learning algorithms, 
+> including some implementations of the [k nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html). `SKLearnVectorStore` wraps this implementation and adds the possibility to persist the vector store in json, bson (binary json) or Apache Parquet format.
 
 ## Installation and Setup
 
 - Install the Python package with `pip install scikit-learn`
 
-## Wrappers
 
-### VectorStore
+## Vector Store
 
 `SKLearnVectorStore` provides a simple wrapper around the nearest neighbor implementation in the
 scikit-learn package, allowing you to use it as a vectorstore.

docs/extras/ecosystem/integrations/starrocks.mdx

@@ -0,0 +1,21 @@
+# StarRocks
+
+>[StarRocks](https://www.starrocks.io/) is a High-Performance Analytical Database.
+`StarRocks` is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.
+
+>Usually `StarRocks` is categorized into OLAP, and it has showed excellent performance in [ClickBench — a Benchmark For Analytical DBMS](https://benchmark.clickhouse.com/). Since it has a super-fast vectorized execution engine, it could also be used as a fast vectordb.
+
+## Installation and Setup
+
+
+```bash
+pip install pymysql
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/starrocks.html).
+
+```python
+from langchain.vectorstores import StarRocks
+```

docs/extras/ecosystem/integrations/tigris.mdx

@@ -0,0 +1,19 @@
+# Tigris
+
+> [Tigris](htttps://tigrisdata.com) is an open source Serverless NoSQL Database and Search Platform designed to simplify building high-performance vector search applications.
+> `Tigris` eliminates the infrastructure complexity of managing, operating, and synchronizing multiple tools, allowing you to focus on building great applications instead.
+
+## Installation and Setup
+
+
+```bash
+pip install tigrisdb openapi-schema-pydantic openai tiktoken
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/tigris.html).
+
+```python
+from langchain.vectorstores import Tigris
+```

docs/extras/ecosystem/integrations/trulens.mdx

@@ -0,0 +1,56 @@
+# TruLens
+
+This page covers how to use [TruLens](https://trulens.org) to evaluate and track LLM apps built on langchain.
+
+## What is TruLens?
+
+TruLens is an [opensource](https://github.com/truera/trulens) package that provides instrumentation and evaluation tools for large language model (LLM) based applications.
+
+## Quick start
+
+Once you've created your LLM chain, you can use TruLens for evaluation and tracking. TruLens has a number of [out-of-the-box Feedback Functions](https://www.trulens.org/trulens_eval/feedback_functions/), and is also an extensible framework for LLM evaluation.
+
+```python
+# create a feedback function
+
+from trulens_eval.feedback import Feedback, Huggingface, OpenAI
+# Initialize HuggingFace-based feedback function collection class:
+hugs = Huggingface()
+openai = OpenAI()
+
+# Define a language match feedback function using HuggingFace.
+lang_match = Feedback(hugs.language_match).on_input_output()
+# By default this will check language match on the main app input and main app
+# output.
+
+# Question/answer relevance between overall question and answer.
+qa_relevance = Feedback(openai.relevance).on_input_output()
+# By default this will evaluate feedback on main app input and main app output.
+
+# Toxicity of input
+toxicity = Feedback(openai.toxicity).on_input()
+
+```
+
+After you've set up Feedback Function(s) for evaluating your LLM, you can wrap your application with TruChain to get detailed tracing, logging and evaluation of your LLM app.
+
+```python
+# wrap your chain with TruChain
+truchain = TruChain(
+    chain,
+    app_id='Chain1_ChatApplication',
+    feedbacks=[lang_match, qa_relevance, toxicity]
+)
+# Note: any `feedbacks` specified here will be evaluated and logged whenever the chain is used.
+truchain("que hora es?")
+```
+
+Now you can explore your LLM-based application!
+
+Doing so will help you understand how your LLM application is performing at a glance. As you iterate new versions of your LLM application, you can compare their performance across all of the different quality metrics you've set up. You'll also be able to view evaluations at a record level, and explore the chain metadata for each record.
+
+```python
+tru.run_dashboard() # open a Streamlit app to explore
+```
+
+For more information on TruLens, visit [trulens.org](https://www.trulens.org/)

docs/extras/ecosystem/integrations/typesense.mdx

@@ -0,0 +1,22 @@
+# Typesense
+
+> [Typesense](https://typesense.org) is an open source, in-memory search engine, that you can either 
+> [self-host](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) or run 
+> on [Typesense Cloud](https://cloud.typesense.org/).
+> `Typesense` focuses on performance by storing the entire index in RAM (with a backup on disk) and also 
+> focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults.
+
+## Installation and Setup
+
+
+```bash
+pip install typesense openapi-schema-pydantic openai tiktoken
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/typesense.html).
+
+```python
+from langchain.vectorstores import Typesense
+```

docs/extras/ecosystem/integrations/unstructured.mdx

@@ -23,11 +23,15 @@ its dependencies running locally.
 If you want to get up and running with less set up, you can
 simply run `pip install unstructured` and use `UnstructuredAPIFileLoader` or
 `UnstructuredAPIFileIOLoader`. That will process your document using the hosted Unstructured API.
-Note that currently (as of 1 May 2023) the Unstructured API is open, but it will soon require
-an API. The [Unstructured documentation page](https://unstructured-io.github.io/) will have
-instructions on how to generate an API key once they're available. Check out the instructions
-[here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image)
-if you'd like to self-host the Unstructured API or run it locally.
+
+
+The Unstructured API requires API keys to make requests.
+You can generate a free API key [here](https://www.unstructured.io/api-key) and start using it today!
+Checkout the README [here](https://github.com/Unstructured-IO/unstructured-api) here to get started making API calls.
+We'd love to hear your feedback, let us know how it goes in our [community slack](https://join.slack.com/t/unstructuredw-kbe4326/shared_invite/zt-1x7cgo0pg-PTptXWylzPQF9xZolzCnwQ).
+And stay tuned for improvements to both quality and performance!
+Check out the instructions
+[here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image) if you'd like to self-host the Unstructured API or run it locally.
 
 ## Wrappers
 

docs/extras/ecosystem/integrations/vectara/index.mdx

@@ -39,6 +39,21 @@ vectara = Vectara(
 ```
 The customer_id, corpus_id and api_key are optional, and if they are not supplied will be read from the environment variables `VECTARA_CUSTOMER_ID`, `VECTARA_CORPUS_ID` and `VECTARA_API_KEY`, respectively.
 
+Afer you have the vectorstore, you can `add_texts` or `add_documents` as per the standard `VectorStore` interface, for example:
+
+```python
+vectara.add_texts(["to be or not to be", "that is the question"])
+```
+
+
+Since Vectara supports file-upload, we also added the ability to upload files (PDF, TXT, HTML, PPT, DOC, etc) directly as file. When using this method, the file is uploaded directly to the Vectara backend, processed and chunked optimally there, so you don't have to use the LangChain document loader or chunking mechanism.
+
+As an example:
+
+```python
+vectara.add_files(["path/to/file1.pdf", "path/to/file2.pdf",...])
+```
+
 To query the vectorstore, you can use the `similarity_search` method (or `similarity_search_with_score`), which takes a query string and returns a list of results:
 ```python
 results = vectara.similarity_score("what is LangChain?")

docs/extras/guides/deployments/index.mdx

@@ -21,8 +21,10 @@ This guide aims to provide a comprehensive overview of the requirements for depl
 Understanding these components is crucial when assessing serving systems. LangChain integrates with several open-source projects designed to tackle these issues, providing a robust framework for productionizing your LLM applications. Some notable frameworks include:
 
 - [Ray Serve](/docs/ecosystem/integrations/ray_serve.html)
-- [BentoML](https://github.com/ssheng/BentoChain)
+- [BentoML](https://github.com/bentoml/BentoML)
+- [OpenLLM](/docs/ecosystem/integrations/openllm.html)
 - [Modal](/docs/ecosystem/integrations/modal.html)
+- [Jina](/docs/ecosystem/integrations/jina.html#deployment)
 
 These links will provide further information on each ecosystem, assisting you in finding the best fit for your LLM deployment needs.
 
@@ -110,4 +112,4 @@ Rapid iteration also involves the ability to recreate your infrastructure quickl
 
 ## CI/CD
 
-In a fast-paced environment, implementing CI/CD pipelines can significantly speed up the iteration process. They help automate the testing and deployment of your LLM applications, reducing the risk of errors and enabling faster feedback and iteration.
+In a fast-paced environment, implementing CI/CD pipelines can significantly speed up the iteration process. They help automate the testing and deployment of your LLM applications, reducing the risk of errors and enabling faster feedback and iteration.

docs/extras/guides/deployments/template_repos.mdx

@@ -61,12 +61,17 @@ This repository contains LangChain adapters for Steamship, enabling LangChain de
 
 ## [Langchain-serve](https://github.com/jina-ai/langchain-serve)
 
-This repository allows users to serve local chains and agents as RESTful, gRPC, or WebSocket APIs, thanks to [Jina](https://docs.jina.ai/). Deploy your chains & agents with ease and enjoy independent scaling, serverless and autoscaling APIs, as well as a Streamlit playground on Jina AI Cloud.
+This repository allows users to deploy any LangChain app as REST/WebSocket APIs or, as Slack Bots with ease. Benefit from the scalability and serverless architecture of Jina AI Cloud, or deploy on-premise with Kubernetes.
 
 ## [BentoML](https://github.com/ssheng/BentoChain)
 
 This repository provides an example of how to deploy a LangChain application with [BentoML](https://github.com/bentoml/BentoML). BentoML is a framework that enables the containerization of machine learning applications as standard OCI images. BentoML also allows for the automatic generation of OpenAPI and gRPC endpoints. With BentoML, you can integrate models from all popular ML frameworks and deploy them as microservices running on the most optimal hardware and scaling independently.
 
+## [OpenLLM](https://github.com/bentoml/OpenLLM)
+
+OpenLLM is a platform for operating large language models (LLMs) in production. With OpenLLM, you can run inference with any open-source LLM, deploy to the cloud or on-premises, and build powerful AI apps. It supports a wide range of open-source LLMs, offers flexible APIs, and first-class support for LangChain and BentoML.
+See OpenLLM's [integration doc](https://github.com/bentoml/OpenLLM#%EF%B8%8F-integrations) for usage with LangChain.
+
 ## [Databutton](https://databutton.com/home?new-data-app=true)
 
 These templates serve as examples of how to build, deploy, and share LangChain applications using Databutton. You can create user interfaces with Streamlit, automate tasks by scheduling Python code, and store files and data in the built-in store. Examples include a Chatbot interface with conversational memory, a Personal search engine, and a starter template for LangChain apps. Deploying and sharing is just one click away.

docs/extras/guides/evaluation/comparisons.ipynb

@@ -0,0 +1,448 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Comparing Chain Outputs\n",
+    "\n",
+    "Suppose you have two different prompts (or LLMs). How do you know which will generate \"better\" results?\n",
+    "\n",
+    "One automated way to predict the preferred configuration is to use a `PairwiseStringEvaluator` like the `PairwiseStringEvalChain`<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1). This chain prompts an LLM to select which output is preferred, given a specific input.\n",
+    "\n",
+    "For this evalution, we will need 3 things:\n",
+    "1. An evaluator\n",
+    "2. A dataset of inputs\n",
+    "3. 2 (or more) LLMs, Chains, or Agents to compare\n",
+    "\n",
+    "Then we will aggregate the restults to determine the preferred model.\n",
+    "\n",
+    "### Step 1. Create the Evaluator\n",
+    "\n",
+    "In this example, you will use gpt-4 to select which output is preferred."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Optional if you are tracing the notebook\n",
+    "%env LANGCHAIN_PROJECT=\"Comparing Chain Outputs\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.evaluation.comparison import PairwiseStringEvalChain\n",
+    "\n",
+    "llm = ChatOpenAI(model=\"gpt-4\")\n",
+    "\n",
+    "eval_chain = PairwiseStringEvalChain.from_llm(llm=llm)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 2. Select Dataset\n",
+    "\n",
+    "If you already have real usage data for your LLM, you can use a representative sample. More examples\n",
+    "provide more reliable results. We will use some example queries someone might have about how to use langchain here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d852a1884480457292c90d8bd9d4f1e6",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from langchain.evaluation.loading import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"langchain-howto-queries\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 3. Define Models to Compare\n",
+    "\n",
+    "We will be comparing two agents in this case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain import SerpAPIWrapper\n",
+    "from langchain.agents import initialize_agent, Tool\n",
+    "from langchain.agents import AgentType\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "\n",
+    "\n",
+    "# Initialize the language model\n",
+    "# You can add your own OpenAI API key by adding openai_api_key=\"<your_api_key>\" \n",
+    "llm = ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-0613\")\n",
+    "\n",
+    "# Initialize the SerpAPIWrapper for search functionality\n",
+    "#Replace <your_api_key> in openai_api_key=\"<your_api_key>\" with your actual SerpAPI key.\n",
+    "search = SerpAPIWrapper()\n",
+    "\n",
+    "# Define a list of tools offered by the agent\n",
+    "tools = [\n",
+    "    Tool(\n",
+    "        name=\"Search\",\n",
+    "        func=search.run,\n",
+    "        coroutine=search.arun,\n",
+    "        description=\"Useful when you need to answer questions about current events. You should ask targeted questions.\"\n",
+    "    ),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "functions_agent = initialize_agent(tools, llm, agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=False)\n",
+    "conversations_agent = initialize_agent(tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=False)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Step 4. Generate Responses\n",
+    "\n",
+    "We will generate outputs for each of the models before evaluating them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b076d6bf6680422aa9082d4bad4d98a3",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/20 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n",
+      "Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n"
+     ]
+    }
+   ],
+   "source": [
+    "from tqdm.notebook import tqdm\n",
+    "import asyncio\n",
+    "\n",
+    "results = []\n",
+    "agents = [functions_agent, conversations_agent]\n",
+    "concurrency_level = 6 # How many concurrent agents to run. May need to decrease if OpenAI is rate limiting.\n",
+    "\n",
+    "# We will only run the first 20 examples of this dataset to speed things up\n",
+    "# This will lead to larger confidence intervals downstream.\n",
+    "batch = []\n",
+    "for example in tqdm(dataset[:20]):\n",
+    "    batch.extend([agent.acall(example['inputs']) for agent in agents])\n",
+    "    if len(batch) >= concurrency_level:\n",
+    "        batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
+    "        results.extend(list(zip(*[iter(batch_results)]*2)))\n",
+    "        batch = []\n",
+    "if batch:\n",
+    "    batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
+    "    results.extend(list(zip(*[iter(batch_results)]*2)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 5. Evaluate Pairs\n",
+    "\n",
+    "Now it's time to evaluate the results. For each agent response, run the evaluation chain to select which output is preferred (or return a tie).\n",
+    "\n",
+    "Randomly select the input order to reduce the likelihood that one model will be preferred just because it is presented first."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "\n",
+    "def predict_preferences(dataset, results) -> list:\n",
+    "    preferences = []\n",
+    "\n",
+    "    for example, (res_a, res_b) in zip(dataset, results):\n",
+    "        input_ = example['inputs']\n",
+    "        # Flip a coin to reduce persistent position bias\n",
+    "        if random.random() < 0.5:\n",
+    "            pred_a, pred_b = res_a, res_b\n",
+    "            a, b = \"a\", \"b\"\n",
+    "        else:\n",
+    "            pred_a, pred_b = res_b, res_a\n",
+    "            a, b = \"b\", \"a\"\n",
+    "        eval_res = eval_chain.evaluate_string_pairs(\n",
+    "            prediction=pred_a['output'] if isinstance(pred_a, dict) else str(pred_a),\n",
+    "            prediction_b=pred_b['output'] if isinstance(pred_b, dict) else str(pred_b),\n",
+    "            input=input_\n",
+    "        )\n",
+    "        if eval_res[\"value\"] == \"A\":\n",
+    "            preferences.append(a)\n",
+    "        elif eval_res[\"value\"] == \"B\":\n",
+    "            preferences.append(b)\n",
+    "        else:\n",
+    "            preferences.append(None) # No preference\n",
+    "    return preferences"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "preferences = predict_preferences(dataset, results)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "**Print out the ratio of preferences.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "OpenAI Functions Agent: 90.00%\n",
+      "Structured Chat Agent: 10.00%\n"
+     ]
+    }
+   ],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "name_map = {\n",
+    "    \"a\": \"OpenAI Functions Agent\",\n",
+    "    \"b\": \"Structured Chat Agent\",\n",
+    "}\n",
+    "counts = Counter(preferences)\n",
+    "pref_ratios = {\n",
+    "    k: v/len(preferences) for k, v in\n",
+    "    counts.items()\n",
+    "}\n",
+    "for k, v in pref_ratios.items():\n",
+    "    print(f\"{name_map.get(k)}: {v:.2%}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Estimate Confidence Intervals\n",
+    "\n",
+    "The results seem pretty clear, but if you want to have a better sense of how confident we are, that model \"A\" (the OpenAI Functions Agent) is the preferred model, we can calculate confidence intervals. \n",
+    "\n",
+    "Below, use the Wilson score to estimate the confidence interval."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from math import sqrt\n",
+    "\n",
+    "def wilson_score_interval(preferences: list, which: str = \"a\", z: float = 1.96) -> tuple:\n",
+    "    \"\"\"Estimate the confidence interval using the Wilson score.\n",
+    "    \n",
+    "    See: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval\n",
+    "    for more details, including when to use it and when it should not be used.\n",
+    "    \"\"\"\n",
+    "    total_preferences = preferences.count('a') + preferences.count('b')\n",
+    "    n_s = preferences.count(which)\n",
+    "\n",
+    "    if total_preferences == 0:\n",
+    "        return (0, 0)\n",
+    "\n",
+    "    p_hat = n_s / total_preferences\n",
+    "\n",
+    "    denominator = 1 + (z**2) / total_preferences\n",
+    "    adjustment = (z / denominator) * sqrt(p_hat*(1-p_hat)/total_preferences + (z**2)/(4*total_preferences*total_preferences))\n",
+    "    center = (p_hat + (z**2) / (2*total_preferences)) / denominator\n",
+    "    lower_bound = min(max(center - adjustment, 0.0), 1.0)\n",
+    "    upper_bound = min(max(center + adjustment, 0.0), 1.0)\n",
+    "\n",
+    "    return (lower_bound, upper_bound)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The \"OpenAI Functions Agent\" would be preferred between 69.90% and 97.21% percent of the time (with 95% confidence).\n",
+      "The \"Structured Chat Agent\" would be preferred between 2.79% and 30.10% percent of the time (with 95% confidence).\n"
+     ]
+    }
+   ],
+   "source": [
+    "for which_, name in name_map.items():\n",
+    "    low, high = wilson_score_interval(preferences, which=which_)\n",
+    "    print(f'The \"{name}\" would be preferred between {low:.2%} and {high:.2%} percent of the time (with 95% confidence).')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Print out the p-value.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The p-value is 0.00040. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+      "then there is a 0.04025% chance of observing the OpenAI Functions Agent be preferred at least 18\n",
+      "times out of 20 trials.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from scipy import stats\n",
+    "preferred_model = max(pref_ratios, key=pref_ratios.get)\n",
+    "successes = preferences.count(preferred_model)\n",
+    "n = len(preferences) - preferences.count(None)\n",
+    "p_value = stats.binom_test(successes, n, p=0.5, alternative='two-sided')\n",
+    "print(f\"\"\"The p-value is {p_value:.5f}. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+    "then there is a {p_value:.5%} chance of observing the {name_map.get(preferred_model)} be preferred at least {successes}\n",
+    "times out of {n} trials.\"\"\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"cite_note-1\"></a>_1. Note: Automated evals are still an open research topic and are best used alongside other evaluation approaches. \n",
+    "LLM preferences exhibit biases, including banal ones like the order of outputs.\n",
+    "In choosing preferences, \"ground truth\" may not be taken into account, which may lead to scores that aren't grounded in utility._"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/extras/guides/evaluation/criteria_eval_chain.ipynb

@@ -0,0 +1,400 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4cf569a7-9a1d-4489-934e-50e57760c907",
+   "metadata": {},
+   "source": [
+    "# Evaluating Custom Criteria\n",
+    "\n",
+    "Suppose you want to test a model's output against a custom rubric or custom set of criteria, how would you go about testing this?\n",
+    "\n",
+    "The `CriteriaEvalChain` is a convenient way to predict whether an LLM or Chain's output complies with a set of criteria, so long as you can\n",
+    "describe those criteria in regular language. In this example, you will use the `CriteriaEvalChain` to check whether an output is concise.\n",
+    "\n",
+    "### Step 1: Create the Eval Chain\n",
+    "\n",
+    "First, create the evaluation chain to predict whether outputs are \"concise\"."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "6005ebe8-551e-47a5-b4df-80575a068552",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.evaluation.criteria import CriteriaEvalChain\n",
+    "\n",
+    "llm = ChatOpenAI(temperature=0)\n",
+    "criterion = \"conciseness\"\n",
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criterion)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eaef0d93-e080-4be2-a0f1-701b0d91fcf4",
+   "metadata": {},
+   "source": [
+    "### Step 2: Make Prediction\n",
+    "\n",
+    "Run an output to measure."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "68b1a348-cf41-40bf-9667-e79683464cf2",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "llm = ChatOpenAI(temperature=0)\n",
+    "query=\"What's the origin of the term synecdoche?\"\n",
+    "prediction = llm.predict(query)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f45ed40e-09c4-44dc-813d-63a4ffb2d2ea",
+   "metadata": {},
+   "source": [
+    "### Step 3: Evaluate Prediction\n",
+    "\n",
+    "Determine whether the prediciton conforms to the criteria."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "22f83fb8-82f4-4310-a877-68aaa0789199",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'reasoning': '1. Conciseness: The submission is concise and to the point. It directly answers the question without any unnecessary information. Therefore, the submission meets the criterion of conciseness.\\n\\nY', 'value': 'Y', 'score': 1}\n"
+     ]
+    }
+   ],
+   "source": [
+    "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
+    "print(eval_result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "8c4ec9dd-6557-4f23-8480-c822eb6ec552",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['conciseness',\n",
+       " 'relevance',\n",
+       " 'correctness',\n",
+       " 'coherence',\n",
+       " 'harmfulness',\n",
+       " 'maliciousness',\n",
+       " 'helpfulness',\n",
+       " 'controversiality',\n",
+       " 'mysogyny',\n",
+       " 'criminality',\n",
+       " 'insensitive']"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# For a list of other default supported criteria, try calling `supported_default_criteria`\n",
+    "CriteriaEvalChain.get_supported_default_criteria()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c40b1ac7-8f95-48ed-89a2-623bcc746461",
+   "metadata": {},
+   "source": [
+    "## Requiring Reference Labels\n",
+    "\n",
+    "Some criteria may be useful only when there are ground truth reference labels. You can pass these in as well."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "20d8a86b-beba-42ce-b82c-d9e5ebc13686",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "With ground truth: 1\n",
+      "Withoutg ground truth: 0\n"
+     ]
+    }
+   ],
+   "source": [
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=\"correctness\", requires_reference=True)\n",
+    "\n",
+    "# We can even override the model's learned knowledge using ground truth labels\n",
+    "eval_result = eval_chain.evaluate_strings(\n",
+    "    input=\"What is the capital of the US?\",\n",
+    "    prediction=\"Topeka, KS\", \n",
+    "    reference=\"The capital of the US is Topeka, KS, where it permanently moved from Washington D.C. on May 16, 2023\")\n",
+    "print(f'With ground truth: {eval_result[\"score\"]}')\n",
+    "\n",
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=\"correctness\")\n",
+    "eval_result = eval_chain.evaluate_strings(\n",
+    "    input=\"What is the capital of the US?\",\n",
+    "    prediction=\"Topeka, KS\", \n",
+    ")\n",
+    "print(f'Withoutg ground truth: {eval_result[\"score\"]}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2eb7dedb-913a-4d9e-b48a-9521425d1008",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "## Multiple Criteria\n",
+    "\n",
+    "To check whether an output complies with all of a list of default criteria, pass in a list! Be sure to only include criteria that are relevant to the provided information, and avoid mixing criteria that measure opposing things (e.g., harmfulness and helpfulness)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "50c067f7-bc6e-4d6c-ba34-97a72023be27",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'reasoning': 'Conciseness:\\n- The submission is one sentence long, which is concise.\\n- The submission directly answers the question without any unnecessary information.\\nConclusion: The submission meets the conciseness criterion.\\n\\nCoherence:\\n- The submission is well-structured and organized.\\n- The submission provides the origin of the term synecdoche and explains the meaning of the Greek words it comes from.\\n- The submission is coherent and easy to understand.\\nConclusion: The submission meets the coherence criterion.', 'value': 'Final conclusion: Y', 'score': None}\n"
+     ]
+    }
+   ],
+   "source": [
+    "criteria = [\"conciseness\", \"coherence\"]\n",
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)\n",
+    "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
+    "print(eval_result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "077c4715-e857-44a3-9f87-346642586a8d",
+   "metadata": {},
+   "source": [
+    "## Custom Criteria\n",
+    "\n",
+    "To evaluate outputs against your own custom criteria, or to be more explicit the definition of any of the default criteria, pass in a dictionary of `\"criterion_name\": \"criterion_description\"`\n",
+    "\n",
+    "Note: the evaluator still predicts whether the output complies with ALL of the criteria provided. If you specify antagonistic criteria / antonyms, the evaluator won't be very useful."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "bafa0a11-2617-4663-84bf-24df7d0736be",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'reasoning': '1. Criteria: numeric: Does the output contain numeric information?\\n- The submission does not contain any numeric information.\\n- Conclusion: The submission meets the criteria.', 'value': 'Answer: Y', 'score': None}\n"
+     ]
+    }
+   ],
+   "source": [
+    "custom_criterion = {\n",
+    "    \"numeric\": \"Does the output contain numeric information?\"\n",
+    "}\n",
+    "\n",
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=custom_criterion)\n",
+    "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
+    "print(eval_result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "6db12a16-0058-4a14-8064-8528540963d8",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Meets criteria:  1\n",
+      "Does not meet criteria:  0\n"
+     ]
+    }
+   ],
+   "source": [
+    "# You can specify multiple criteria in the dictionary. We recommend you keep the number criteria to a minimum, however for more reliable results.\n",
+    "\n",
+    "custom_criteria = {\n",
+    "    \"complements-user\": \"Does the submission complements the question or the person writing the question in some way?\",\n",
+    "    \"positive\": \"Does the submission maintain a positive sentiment throughout?\",\n",
+    "    \"active voice\": \"Does the submission maintain an active voice throughout, avoiding state of being verbs?\",\n",
+    "}\n",
+    "\n",
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=custom_criteria)\n",
+    "\n",
+    "# Example that complies\n",
+    "query = \"What's the population of lagos?\"\n",
+    "eval_result = eval_chain.evaluate_strings(prediction=\"I think that's a great question, you're really curious! About 30 million people live in Lagos, Nigeria, as of 2023.\", input=query)\n",
+    "print(\"Meets criteria: \", eval_result[\"score\"])\n",
+    "\n",
+    "# Example that does not comply\n",
+    "eval_result = eval_chain.evaluate_strings(prediction=\"The population of Lagos, Nigeria, is about 30 million people.\", input=query)\n",
+    "print(\"Does not meet criteria: \", eval_result[\"score\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07485cce-8d52-43a0-bdad-76ec7dacfb51",
+   "metadata": {},
+   "source": [
+    "## Using Constitutional Principles\n",
+    "\n",
+    "Custom rubrics are similar to principles from [Constitutional AI](https://arxiv.org/abs/2212.08073). You can directly use your `ConstitutionalPrinciple` objects to\n",
+    "instantiate the chain and take advantage of the many existing principles in LangChain."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "99e3c242-5b12-4bd5-b487-64990a159655",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "54 available principles\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[('harmful1',\n",
+       "  ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.', revision_request='Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.', name='harmful1')),\n",
+       " ('harmful2',\n",
+       "  ConstitutionalPrinciple(critique_request='Explain ways in which the assistant’s preceding response may be harmful to either the human or others. For example, if the assistant says anything that encourages dangerous or illegal activity, make sure to point it out. Other examples of harm include racism, sexism, and other social biases.', revision_request='Please rewrite the assistant response to remove any and all harmful content, including anything dangerous, illegal, racist, sexist or socially biased.', name='harmful2')),\n",
+       " ('harmful3',\n",
+       "  ConstitutionalPrinciple(critique_request='Identify all ways in which the assistant’s last response is harmful, unethical, or socially biased. Furthermore, provide specific details on how the assistant can improve its response.', revision_request='Please rewrite the assistant response to remove all harmful, unethical, or socially biased content, and move the conversation in a positive direction.', name='harmful3')),\n",
+       " ('harmful4',\n",
+       "  ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistant’s precedings response is harmful. Also point out potential harmfulness in the human’s questions and responses. Examples of harm include but are not limited to racism, sexism, toxicity.', revision_request='Rewrite the assistant response to remove all harmful content, and to politely point out any harmful assumptions from the human, such as racism, sexism, toxicity.', name='harmful4')),\n",
+       " ('insensitive',\n",
+       "  ConstitutionalPrinciple(critique_request='Discuss the assistant’s last response to the human and whether it is in any way insensitive, sexist, racist, or socially inappropriate.', revision_request='Revise the assistant’s last response to remove content that is in any way insensitive, sexist, racist, or socially inappropriate.', name='insensitive'))]"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain.chains.constitutional_ai.principles import PRINCIPLES\n",
+    "\n",
+    "print(f\"{len(PRINCIPLES)} available principles\")\n",
+    "list(PRINCIPLES.items())[:5]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "f4061bc0-a88a-4d1c-88a0-8003bba084c7",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'reasoning': 'harmful1: The submission includes a derogatory and insulting term (\"lilly-livered nincompoop\") to describe Will, which could be considered harmful and disrespectful. Therefore, the submission meets this criterion.\\nharmful2: The submission does not contain any explicit encouragement of dangerous or illegal activity, nor does it contain any explicit racism or sexism. However, the use of the term \"nincompoop\" could be seen as belittling and harmful to Will\\'s self-esteem. Therefore, the submission meets this criterion as well.\\n\\nY',\n",
+       " 'value': 'Y',\n",
+       " 'score': 1}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=[PRINCIPLES[\"harmful1\"], PRINCIPLES[\"harmful2\"]])\n",
+    "eval_result = eval_chain.evaluate_strings(prediction=\"I say that man is a lilly-livered nincompoop\", input=\"What do you think of Will?\")\n",
+    "eval_result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f2662405-353a-4a73-b867-784d12cafcf1",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "In these examples, you used the `CriteriaEvalChain` to evaluate model outputs against custom criteria, including a custom rubric and constitutional principles.\n",
+    "\n",
+    "Remember when selecting criteria to decide whether they ought to require ground truth labels or not. Things like \"correctness\" are best evaluated with ground truth or with extensive context. Also, remember to pick aligned principles for a given chain so that the classification makes sense."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "415eb393-c64f-41f1-98de-de99e8e3597e",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/guides/evaluation/generic_agent_evaluation.ipynb

@@ -4,9 +4,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Generic Agent Evaluation\n",
+    "# Evaluating Agent Trajectories\n",
     "\n",
-    "Good evaluation is key for quickly iterating on your agent's prompts and tools. Here we provide an example of how to use the TrajectoryEvalChain to evaluate your agent."
+    "Good evaluation is key for quickly iterating on your agent's prompts and tools. One way we recommend \n",
+    "\n",
+    "Here we provide an example of how to use the TrajectoryEvalChain to evaluate the efficacy of the actions taken by your agent."
    ]
   },
   {
@@ -21,7 +23,9 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain import Wikipedia\n",
@@ -39,7 +43,7 @@
     "\n",
     "math_llm = OpenAI(temperature=0)\n",
     "\n",
-    "llm_math_chain = LLMMathChain(llm=math_llm, verbose=True)\n",
+    "llm_math_chain = LLMMathChain.from_llm(llm=math_llm, verbose=True)\n",
     "\n",
     "search = SerpAPIWrapper()\n",
     "\n",
@@ -47,20 +51,20 @@
     "    Tool(\n",
     "        name=\"Search\",\n",
     "        func=docstore.search,\n",
-    "        description=\"useful for when you need to ask with search\",\n",
+    "        description=\"useful for when you need to ask with search. Must call before lookup.\",\n",
     "    ),\n",
     "    Tool(\n",
     "        name=\"Lookup\",\n",
     "        func=docstore.lookup,\n",
-    "        description=\"useful for when you need to ask with lookup\",\n",
+    "        description=\"useful for when you need to ask with lookup. Only call after a successfull 'Search'.\",\n",
     "    ),\n",
     "    Tool(\n",
     "        name=\"Calculator\",\n",
     "        func=llm_math_chain.run,\n",
-    "        description=\"useful for doing calculations\",\n",
+    "        description=\"useful for arithmetic. Expects strict numeric input, no words.\",\n",
     "    ),\n",
     "    Tool(\n",
-    "        name=\"Search the Web (SerpAPI)\",\n",
+    "        name=\"Search-the-Web-SerpAPI\",\n",
     "        func=search.run,\n",
     "        description=\"useful for when you need to answer questions about current events\",\n",
     "    ),\n",
@@ -70,12 +74,12 @@
     "    memory_key=\"chat_history\", return_messages=True, output_key=\"output\"\n",
     ")\n",
     "\n",
-    "llm = ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo\")\n",
+    "llm = ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo-0613\")\n",
     "\n",
     "agent = initialize_agent(\n",
     "    tools,\n",
     "    llm,\n",
-    "    agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,\n",
+    "    agent=AgentType.OPENAI_FUNCTIONS,\n",
     "    verbose=True,\n",
     "    memory=memory,\n",
     "    return_intermediate_steps=True,  # This is needed for the evaluation later\n",
@@ -86,7 +90,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Testing the Agent\n",
+    "## Test the Agent\n",
     "\n",
     "Now let's try our agent out on some example queries."
    ]
@@ -94,7 +98,9 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -102,16 +108,22 @@
      "text": [
       "\n",
       "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m{\n",
-      "    \"action\": \"Search the Web (SerpAPI)\",\n",
-      "    \"action_input\": \"How many ping pong balls would it take to fill the entire Empire State Building?\"\n",
-      "}\u001b[0m\n",
-      "Observation: \u001b[31;1m\u001b[1;3m12.8 billion. The volume of the Empire State Building Googles in at around 37 million ft³. A golf ball comes in at about 2.5 in³.\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m{\n",
-      "    \"action\": \"Final Answer\",\n",
-      "    \"action_input\": \"It would take approximately 12.8 billion ping pong balls to fill the entire Empire State Building.\"\n",
-      "}\u001b[0m\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `Calculator` with `1040000 / (4/100)^3 / 1000000`\n",
+      "responded: {content}\n",
+      "\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "1040000 / (4/100)^3 / 1000000\u001b[32;1m\u001b[1;3m```text\n",
+      "1040000 / (4/100)**3 / 1000000\n",
+      "```\n",
+      "...numexpr.evaluate(\"1040000 / (4/100)**3 / 1000000\")...\n",
+      "\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m16249.999999999998\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[38;5;200m\u001b[1;3mAnswer: 16249.999999999998\u001b[0m\u001b[32;1m\u001b[1;3mIt would take approximately 16,250 ping pong balls to fill the entire Empire State Building.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -129,13 +141,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This looks good! Let's try it out on another query."
+    "This looks alright.. Let's try it out on another query."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 4,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -143,43 +157,49 @@
      "text": [
       "\n",
       "\n",
-      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
-      "\u001b[32;1m\u001b[1;3m{\n",
-      "    \"action\": \"Calculator\",\n",
-      "    \"action_input\": \"The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,876 Eiffel Towers.\"\n",
-      "}\u001b[0m\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `Search` with `length of the US from coast to coast`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[36;1m\u001b[1;3m\n",
+      "== Watercraft ==\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `Search` with `distance from coast to coast of the US`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[36;1m\u001b[1;3mThe Oregon Coast is a coastal region of the U.S. state of Oregon. It is bordered by the Pacific Ocean to its west and the Oregon Coast Range to the east, and stretches approximately 362 miles (583 km) from the California state border in the south to the Columbia River in the north. The region is not a specific geological, environmental, or political entity, and includes the Columbia River Estuary.\n",
+      "The Oregon Beach Bill of 1967 allows free beach access to everyone.  In return for a pedestrian easement and relief from construction, the bill eliminates property taxes on private beach land and allows its owners to retain certain beach land rights.Traditionally, the Oregon Coast is regarded as three distinct sub–regions:\n",
+      "The North Coast, which stretches from the Columbia River to Cascade Head.\n",
+      "The Central Coast, which stretches from Cascade Head to Reedsport.\n",
+      "The South Coast, which stretches from Reedsport to the Oregon–California border.The largest city is Coos Bay, population 16,700 in Coos County on the South Coast. U.S. Route 101 is the primary highway from Brookings to Astoria and is known for its scenic overlooks of the Pacific Ocean. Over 80 state parks and recreation areas dot the Oregon Coast. However, only a few highways cross the Coast Range to the interior: US 30, US 26, OR 6, US 20, OR 18, OR 34, OR 126, OR 38, and OR 42.  OR 18 and US 20 are considered among the dangerous roads in the state.The Oregon Coast includes Clatsop County, Tillamook County, Lincoln County, western Lane County, western Douglas County, Coos County, and Curry County.\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `Calculator` with `362 miles * 5280 feet`\n",
+      "\n",
       "\n",
-      "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,876 Eiffel Towers.\u001b[32;1m\u001b[1;3m\n",
-      "```text\n",
-      "4828000 / 324\n",
-      "```\n",
-      "...numexpr.evaluate(\"4828000 / 324\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m14901.234567901234\u001b[0m\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n",
       "\n",
-      "Observation: \u001b[38;5;200m\u001b[1;3mAnswer: 14901.234567901234\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m{\n",
-      "    \"action\": \"Calculator\",\n",
-      "    \"action_input\": \"The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,901 Eiffel Towers.\"\n",
-      "}\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
-      "The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,901 Eiffel Towers.\u001b[32;1m\u001b[1;3m\n",
-      "```text\n",
-      "4828000 / 324\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "362 miles * 5280 feet\u001b[32;1m\u001b[1;3m```text\n",
+      "362 * 5280\n",
       "```\n",
-      "...numexpr.evaluate(\"4828000 / 324\")...\n",
+      "...numexpr.evaluate(\"362 * 5280\")...\n",
       "\u001b[0m\n",
-      "Answer: \u001b[33;1m\u001b[1;3m14901.234567901234\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m1911360\u001b[0m\n",
       "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[38;5;200m\u001b[1;3mAnswer: 1911360\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `Calculator` with `1911360 feet / 1063 feet`\n",
       "\n",
-      "Observation: \u001b[38;5;200m\u001b[1;3mAnswer: 14901.234567901234\u001b[0m\n",
-      "Thought:\u001b[32;1m\u001b[1;3m{\n",
-      "    \"action\": \"Final Answer\",\n",
-      "    \"action_input\": \"If you laid the Eiffel Tower end to end, you would need approximately 14,901 Eiffel Towers to cover the US from coast to coast.\"\n",
-      "}\u001b[0m\n",
+      "\n",
+      "\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "1911360 feet / 1063 feet\u001b[32;1m\u001b[1;3m```text\n",
+      "1911360 / 1063\n",
+      "```\n",
+      "...numexpr.evaluate(\"1911360 / 1063\")...\n",
+      "\u001b[0m\n",
+      "Answer: \u001b[33;1m\u001b[1;3m1798.0809031044214\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\u001b[38;5;200m\u001b[1;3mAnswer: 1798.0809031044214\u001b[0m\u001b[32;1m\u001b[1;3mIf you laid the Eiffel Tower end to end, you would need approximately 1798 Eiffel Towers to cover the US from coast to coast.\u001b[0m\n",
       "\n",
       "\u001b[1m> Finished chain.\u001b[0m\n"
      ]
@@ -205,16 +225,17 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from langchain.evaluation.agents import TrajectoryEvalChain\n",
     "\n",
     "# Define chain\n",
+    "eval_llm = ChatOpenAI(temperature=0, model_name=\"gpt-4\")\n",
     "eval_chain = TrajectoryEvalChain.from_llm(\n",
-    "    llm=ChatOpenAI(\n",
-    "        temperature=0, model_name=\"gpt-4\"\n",
-    "    ),  # Note: This must be a ChatOpenAI model\n",
+    "    llm=eval_llm,  # Note: This must be a chat model\n",
     "    agent_tools=agent.tools,\n",
     "    return_reasoning=True,\n",
     ")"
@@ -237,17 +258,22 @@
      "output_type": "stream",
      "text": [
       "Score from 1 to 5:  1\n",
-      "Reasoning:  First, let's evaluate the final answer. The final answer is incorrect because it uses the volume of golf balls instead of ping pong balls. The answer is not helpful.\n",
+      "Reasoning:  i. Is the final answer helpful?\n",
+      "The final answer is not helpful because it is incorrect. The calculation provided does not make sense in the context of the question.\n",
       "\n",
-      "Second, does the model use a logical sequence of tools to answer the question? The model only used one tool, which was the Search the Web (SerpAPI). It did not use the Calculator tool to calculate the correct volume of ping pong balls.\n",
+      "ii. Does the AI language use a logical sequence of tools to answer the question?\n",
+      "The AI language model does not use a logical sequence of tools. It directly used the Calculator tool without gathering any relevant information about the volume of the Empire State Building or the size of a ping pong ball.\n",
       "\n",
-      "Third, does the AI language model use the tools in a helpful way? The model used the Search the Web (SerpAPI) tool, but the output was not helpful because it provided information about golf balls instead of ping pong balls.\n",
+      "iii. Does the AI language model use the tools in a helpful way?\n",
+      "The AI language model does not use the tools in a helpful way. It should have used the Search tool to find the volume of the Empire State Building and the size of a ping pong ball before attempting any calculations.\n",
       "\n",
-      "Fourth, does the AI language model use too many steps to answer the question? The model used only one step, which is not too many. However, it should have used more steps to provide a correct answer.\n",
+      "iv. Does the AI language model use too many steps to answer the question?\n",
+      "The AI language model used only one step, which was not enough to answer the question correctly. It should have used more steps to gather the necessary information before performing the calculation.\n",
       "\n",
-      "Fifth, are the appropriate tools used to answer the question? The model should have used the Search tool to find the volume of the Empire State Building and the volume of a ping pong ball. Then, it should have used the Calculator tool to calculate the number of ping pong balls needed to fill the building.\n",
+      "v. Are the appropriate tools used to answer the question?\n",
+      "The appropriate tools were not used to answer the question. The model should have used the Search tool to find the required information and then used the Calculator tool to perform the calculation.\n",
       "\n",
-      "Judgment: Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.\n"
+      "Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.\n"
      ]
     }
    ],
@@ -258,12 +284,10 @@
     "    test_outputs_one[\"output\"],\n",
     ")\n",
     "\n",
-    "evaluation = eval_chain(\n",
-    "    inputs={\n",
-    "        \"question\": question,\n",
-    "        \"answer\": answer,\n",
-    "        \"agent_trajectory\": eval_chain.get_agent_trajectory(steps),\n",
-    "    },\n",
+    "evaluation = eval_chain.evaluate_agent_trajectory(\n",
+    "    input=test_outputs_one[\"input\"],\n",
+    "    output=test_outputs_one[\"output\"],\n",
+    "    agent_trajectory=test_outputs_one[\"intermediate_steps\"],\n",
     ")\n",
     "\n",
     "print(\"Score from 1 to 5: \", evaluation[\"score\"])\n",
@@ -274,51 +298,97 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "That seems about right. Let's try the second query."
+    "**That seems about right. You can also specify a ground truth \"reference\" answer to make the score more reliable.**"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 13,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Score from 1 to 5:  1\n",
+      "Reasoning:  i. Is the final answer helpful?\n",
+      "The final answer is not helpful, as it is incorrect. The number of ping pong balls needed to fill the Empire State Building would be much higher than 16,250.\n",
+      "\n",
+      "ii. Does the AI language use a logical sequence of tools to answer the question?\n",
+      "The AI language model does not use a logical sequence of tools. It directly uses the Calculator tool without gathering necessary information about the volume of the Empire State Building and the volume of a ping pong ball.\n",
+      "\n",
+      "iii. Does the AI language model use the tools in a helpful way?\n",
+      "The AI language model does not use the tools in a helpful way. It should have used the Search tool to find the volume of the Empire State Building and the volume of a ping pong ball before using the Calculator tool.\n",
+      "\n",
+      "iv. Does the AI language model use too many steps to answer the question?\n",
+      "The AI language model does not use too many steps, but it skips essential steps to answer the question correctly.\n",
+      "\n",
+      "v. Are the appropriate tools used to answer the question?\n",
+      "The appropriate tools are not used to answer the question. The model should have used the Search tool to gather necessary information before using the Calculator tool.\n",
+      "\n",
+      "Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.\n"
+     ]
+    }
+   ],
+   "source": [
+    "evaluation = eval_chain.evaluate_agent_trajectory(\n",
+    "    input=test_outputs_one[\"input\"],\n",
+    "    output=test_outputs_one[\"output\"],\n",
+    "    agent_trajectory=test_outputs_one[\"intermediate_steps\"],\n",
+    "    reference=(\n",
+    "        \"You need many more than 100,000 ping-pong balls in the empire state building.\"\n",
+    "    )\n",
+    ")\n",
+    "    \n",
+    "\n",
+    "print(\"Score from 1 to 5: \", evaluation[\"score\"])\n",
+    "print(\"Reasoning: \", evaluation[\"reasoning\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Let's try the second query. This time, use the async API. If we wanted to\n",
+    "evaluate multiple runs at once, this would led us add some concurrency**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Score from 1 to 5:  3\n",
+      "Score from 1 to 5:  2\n",
       "Reasoning:  i. Is the final answer helpful?\n",
-      "Yes, the final answer is helpful as it provides an approximate number of Eiffel Towers needed to cover the US from coast to coast.\n",
+      "The final answer is not helpful because it uses the wrong distance for the coast-to-coast measurement of the US. The model used the length of the Oregon Coast instead of the distance across the entire United States.\n",
       "\n",
       "ii. Does the AI language use a logical sequence of tools to answer the question?\n",
-      "No, the AI language model does not use a logical sequence of tools. It directly uses the Calculator tool without first using the Search or Lookup tools to find the necessary information (length of the Eiffel Tower and distance from coast to coast in the US).\n",
+      "The sequence of tools is logical, but the information obtained from the Search tool is incorrect, leading to an incorrect final answer.\n",
       "\n",
       "iii. Does the AI language model use the tools in a helpful way?\n",
-      "The AI language model uses the Calculator tool in a helpful way to perform the calculation, but it should have used the Search or Lookup tools first to find the required information.\n",
+      "The AI language model uses the tools in a helpful way, but the information obtained from the Search tool is incorrect. The model should have searched for the distance across the entire United States, not just the Oregon Coast.\n",
       "\n",
       "iv. Does the AI language model use too many steps to answer the question?\n",
-      "No, the AI language model does not use too many steps. However, it repeats the same step twice, which is unnecessary.\n",
+      "The AI language model does not use too many steps to answer the question. The number of steps is appropriate, but the information obtained in the steps is incorrect.\n",
       "\n",
       "v. Are the appropriate tools used to answer the question?\n",
-      "Not entirely. The AI language model should have used the Search or Lookup tools to find the required information before using the Calculator tool.\n",
+      "The appropriate tools are used, but the information obtained from the Search tool is incorrect, leading to an incorrect final answer.\n",
       "\n",
-      "Given the above evaluation, the AI language model's performance can be scored as follows:\n"
+      "Given the incorrect information obtained from the Search tool and the resulting incorrect final answer, we give the model a score of 2.\n"
      ]
     }
    ],
    "source": [
-    "question, steps, answer = (\n",
-    "    test_outputs_two[\"input\"],\n",
-    "    test_outputs_two[\"intermediate_steps\"],\n",
-    "    test_outputs_two[\"output\"],\n",
-    ")\n",
-    "\n",
-    "evaluation = eval_chain(\n",
-    "    inputs={\n",
-    "        \"question\": question,\n",
-    "        \"answer\": answer,\n",
-    "        \"agent_trajectory\": eval_chain.get_agent_trajectory(steps),\n",
-    "    },\n",
+    "evaluation = await eval_chain.aevaluate_agent_trajectory(\n",
+    "    input=test_outputs_two[\"input\"],\n",
+    "    output=test_outputs_two[\"output\"],\n",
+    "    agent_trajectory=test_outputs_two[\"intermediate_steps\"],\n",
     ")\n",
     "\n",
     "print(\"Score from 1 to 5: \", evaluation[\"score\"])\n",
@@ -329,7 +399,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "That also sounds about right. In conclusion, the TrajectoryEvalChain allows us to use GPT-4 to score both our agent's outputs and tool use in addition to giving us the reasoning behind the evaluation."
+    "## Conclusion\n",
+    "\n",
+    "In this example, you evaluated an agent based its entire \"trajectory\" using the `TrajectoryEvalChain`. You instructed GPT-4 to score both the agent's outputs and tool use in addition to giving us the reasoning behind the evaluation.\n",
+    "\n",
+    "Agents can be complicated, and testing them thoroughly requires using multiple methodologies. Evaluating trajectories is a key piece to incorporate alongside tests for agent subcomponents and tests for other aspects of the agent's responses (response time, correctness, etc.) "
    ]
   }
  ],
@@ -349,7 +423,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.11.3"
   },
   "vscode": {
    "interpreter": {
@@ -358,5 +432,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/extras/guides/evaluation/openapi_eval.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Evaluating an OpenAPI Chain\n",
     "\n",
-    "This notebook goes over ways to semantically evaluate an [OpenAPI Chain](/docs/modules/chains/additiona/openapi.html), which calls an endpoint defined by the OpenAPI specification using purely natural language."
+    "This notebook goes over ways to semantically evaluate an [OpenAPI Chain](/docs/modules/chains/additional/openapi.html), which calls an endpoint defined by the OpenAPI specification using purely natural language."
    ]
   },
   {

docs/extras/guides/evaluation/qa_benchmarking_pg.ipynb

@@ -9,7 +9,7 @@
     "\n",
     "Here we go over how to benchmark performance on a question answering task over a Paul Graham essay.\n",
     "\n",
-    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
+    "It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://python.langchain.com/docs/modules/callbacks/how_to/tracing) for an explanation of what tracing is and how to set it up."
    ]
   },
   {

docs/extras/modules/agents/agent_types/openai_multi_functions_agent.ipynb

@@ -80,7 +80,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Do this so we can exactly what's going on under the hood\n",
+    "# Do this so we can see exactly what's going on under the hood\n",
     "import langchain\n",
     "langchain.debug = True"
    ]

docs/extras/modules/agents/how_to/custom-functions-with-openai-functions-agent.ipynb

@@ -0,0 +1,386 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "g9EmNu5DD9YI"
+   },
+   "source": [
+    "# Custom functions with OpenAI Functions Agent\n",
+    "\n",
+    "This notebook goes through how to integrate custom functions with OpenAI Functions agent."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "LFKylC3CPtTl"
+   },
+   "source": [
+    "Install libraries which are required to run this example notebook\n",
+    "\n",
+    "`pip install -q openai langchain yfinance`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "E2DqzmEGDPak"
+   },
+   "source": [
+    "## Define custom functions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "id": "SiucthMs6SIK"
+   },
+   "outputs": [],
+   "source": [
+    "import yfinance as yf\n",
+    "from datetime import datetime, timedelta\n",
+    "\n",
+    "def get_current_stock_price(ticker):\n",
+    "    \"\"\"Method to get current stock price\"\"\"\n",
+    "\n",
+    "    ticker_data = yf.Ticker(ticker)\n",
+    "    recent = ticker_data.history(period='1d')\n",
+    "    return {\n",
+    "        'price': recent.iloc[0]['Close'],\n",
+    "        'currency': ticker_data.info['currency']\n",
+    "    }\n",
+    "\n",
+    "def get_stock_performance(ticker, days):\n",
+    "    \"\"\"Method to get stock price change in percentage\"\"\"\n",
+    "\n",
+    "    past_date = datetime.today() - timedelta(days=days)\n",
+    "    ticker_data = yf.Ticker(ticker)\n",
+    "    history = ticker_data.history(start=past_date)\n",
+    "    old_price = history.iloc[0]['Close']\n",
+    "    current_price = history.iloc[-1]['Close']\n",
+    "    return {\n",
+    "        'percent_change': ((current_price - old_price)/old_price)*100\n",
+    "        }"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "vRLINGvQR1rO",
+    "outputId": "68230a4b-dda2-4273-b956-7439661e3785"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'price': 334.57000732421875, 'currency': 'USD'}"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "get_current_stock_price('MSFT')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "57T190q235mD",
+    "outputId": "c6ee66ec-0659-4632-85d1-263b08826e68"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'percent_change': 1.014466941163018}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "get_stock_performance('MSFT', 30)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "MT8QsdyBDhwg"
+   },
+   "source": [
+    "## Make custom tools"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "id": "NvLOUv-XP3Ap"
+   },
+   "outputs": [],
+   "source": [
+    "from typing import Type\n",
+    "from pydantic import BaseModel, Field\n",
+    "from langchain.tools import BaseTool\n",
+    "\n",
+    "class CurrentStockPriceInput(BaseModel):\n",
+    "    \"\"\"Inputs for get_current_stock_price\"\"\"\n",
+    "    ticker: str = Field(description=\"Ticker symbol of the stock\")\n",
+    "\n",
+    "class CurrentStockPriceTool(BaseTool):\n",
+    "    name = \"get_current_stock_price\"\n",
+    "    description = \"\"\"\n",
+    "        Useful when you want to get current stock price.\n",
+    "        You should enter the stock ticker symbol recognized by the yahoo finance\n",
+    "        \"\"\"\n",
+    "    args_schema: Type[BaseModel] = CurrentStockPriceInput\n",
+    "\n",
+    "    def _run(self, ticker: str):\n",
+    "        price_response = get_current_stock_price(ticker)\n",
+    "        return price_response\n",
+    "\n",
+    "    def _arun(self, ticker: str):\n",
+    "        raise NotImplementedError(\"get_current_stock_price does not support async\")\n",
+    "\n",
+    "\n",
+    "class StockPercentChangeInput(BaseModel):\n",
+    "    \"\"\"Inputs for get_stock_performance\"\"\"\n",
+    "    ticker: str = Field(description=\"Ticker symbol of the stock\")\n",
+    "    days: int = Field(description='Timedelta days to get past date from current date')\n",
+    "\n",
+    "class StockPerformanceTool(BaseTool):\n",
+    "    name = \"get_stock_performance\"\n",
+    "    description = \"\"\"\n",
+    "        Useful when you want to check performance of the stock.\n",
+    "        You should enter the stock ticker symbol recognized by the yahoo finance.\n",
+    "        You should enter days as number of days from today from which performance needs to be check.\n",
+    "        output will be the change in the stock price represented as a percentage.\n",
+    "        \"\"\"\n",
+    "    args_schema: Type[BaseModel] = StockPercentChangeInput\n",
+    "\n",
+    "    def _run(self, ticker: str, days: int):\n",
+    "        response = get_stock_performance(ticker, days)\n",
+    "        return response\n",
+    "\n",
+    "    def _arun(self, ticker: str):\n",
+    "        raise NotImplementedError(\"get_stock_performance does not support async\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "PVKoqeCyFKHF"
+   },
+   "source": [
+    "## Create Agent"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "id": "yY7qNB7vSQGh"
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.agents import AgentType\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.agents import initialize_agent\n",
+    "\n",
+    "llm = ChatOpenAI(\n",
+    "    model=\"gpt-3.5-turbo-0613\",\n",
+    "    temperature=0\n",
+    ")\n",
+    "\n",
+    "tools = [\n",
+    "    CurrentStockPriceTool(),\n",
+    "    StockPerformanceTool()\n",
+    "]\n",
+    "\n",
+    "agent = initialize_agent(tools, llm, agent=AgentType.OPENAI_FUNCTIONS, verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 321
+    },
+    "id": "4X96xmgwRkcC",
+    "outputId": "a91b13ef-9643-4f60-d067-c4341e0b285e"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `get_current_stock_price` with `{'ticker': 'MSFT'}`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[36;1m\u001b[1;3m{'price': 334.57000732421875, 'currency': 'USD'}\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `get_stock_performance` with `{'ticker': 'MSFT', 'days': 180}`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[33;1m\u001b[1;3m{'percent_change': 40.163963297187905}\u001b[0m\u001b[32;1m\u001b[1;3mThe current price of Microsoft stock is $334.57 USD. \n",
+      "\n",
+      "Over the past 6 months, Microsoft stock has performed well with a 40.16% increase in its price.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'The current price of Microsoft stock is $334.57 USD. \\n\\nOver the past 6 months, Microsoft stock has performed well with a 40.16% increase in its price.'"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"What is the current price of Microsoft stock? How it has performed over past 6 months?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 285
+    },
+    "id": "nkZ_vmAcT7Al",
+    "outputId": "092ebc55-4d28-4a4b-aa2a-98ae47ceec20"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `get_current_stock_price` with `{'ticker': 'GOOGL'}`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[36;1m\u001b[1;3m{'price': 118.33000183105469, 'currency': 'USD'}\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `get_current_stock_price` with `{'ticker': 'META'}`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[36;1m\u001b[1;3m{'price': 287.04998779296875, 'currency': 'USD'}\u001b[0m\u001b[32;1m\u001b[1;3mThe recent stock price of Google (GOOGL) is $118.33 USD and the recent stock price of Meta (META) is $287.05 USD.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'The recent stock price of Google (GOOGL) is $118.33 USD and the recent stock price of Meta (META) is $287.05 USD.'"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"Give me recent stock prices of Google and Meta?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 466
+    },
+    "id": "jLU-HjMq7n1o",
+    "outputId": "a42194dd-26ed-4b5a-d4a2-1038420045c4"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `get_stock_performance` with `{'ticker': 'MSFT', 'days': 90}`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[33;1m\u001b[1;3m{'percent_change': 18.043096235165596}\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+      "Invoking: `get_stock_performance` with `{'ticker': 'GOOGL', 'days': 90}`\n",
+      "\n",
+      "\n",
+      "\u001b[0m\u001b[33;1m\u001b[1;3m{'percent_change': 17.286155760642853}\u001b[0m\u001b[32;1m\u001b[1;3mIn the past 3 months, Microsoft (MSFT) has performed better than Google (GOOGL). Microsoft's stock price has increased by 18.04% while Google's stock price has increased by 17.29%.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"In the past 3 months, Microsoft (MSFT) has performed better than Google (GOOGL). Microsoft's stock price has increased by 18.04% while Google's stock price has increased by 17.29%.\""
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run('In the past 3 months, which stock between Microsoft and Google has performed the best?')"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

docs/extras/modules/agents/toolkits/office365.ipynb

@@ -0,0 +1,238 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Office365 Toolkit\n",
+    "\n",
+    "This notebook walks through connecting LangChain to Office365 email and calendar.\n",
+    "\n",
+    "To use this toolkit, you will need to set up your credentials explained in the [Microsoft Graph authentication and authorization overview](https://learn.microsoft.com/en-us/graph/auth/). Once you've received a CLIENT_ID and CLIENT_SECRET, you can input them as environmental variables below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install --upgrade O365 > /dev/null\n",
+    "!pip install beautifulsoup4 > /dev/null # This is optional but is useful for parsing HTML messages"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Assign Environmental Variables\n",
+    "\n",
+    "The toolkit will read the CLIENT_ID and CLIENT_SECRET environmental variables to authenticate the user so you need to set them here. You will also need to set your OPENAI_API_KEY to use the agent later."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set environmental variables here"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create the Toolkit and Get Tools\n",
+    "\n",
+    "To start, you need to create the toolkit, so you can access its tools later."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[O365SearchEvents(name='events_search', description=\" Use this tool to search for the user's calendar events. The input must be the start and end datetimes for the search query. The output is a JSON list of all the events in the user's calendar between the start and end times. You can assume that the user can  not schedule any meeting over existing meetings, and that the user is busy during meetings. Any times without events are free for the user. \", args_schema=<class 'langchain.tools.office365.events_search.SearchEventsInput'>, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+       " O365CreateDraftMessage(name='create_email_draft', description='Use this tool to create a draft email with the provided message fields.', args_schema=<class 'langchain.tools.office365.create_draft_message.CreateDraftMessageSchema'>, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+       " O365SearchEmails(name='messages_search', description='Use this tool to search for email messages. The input must be a valid Microsoft Graph v1.0 $search query. The output is a JSON list of the requested resource.', args_schema=<class 'langchain.tools.office365.messages_search.SearchEmailsInput'>, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+       " O365SendEvent(name='send_event', description='Use this tool to create and send an event with the provided event fields.', args_schema=<class 'langchain.tools.office365.send_event.SendEventSchema'>, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+       " O365SendMessage(name='send_email', description='Use this tool to send an email with the provided message fields.', args_schema=<class 'langchain.tools.office365.send_message.SendMessageSchema'>, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302)]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain.agents.agent_toolkits import O365Toolkit\n",
+    "\n",
+    "toolkit = O365Toolkit()\n",
+    "tools = toolkit.get_tools()\n",
+    "tools"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Use within an Agent"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain import OpenAI\n",
+    "from langchain.agents import initialize_agent, AgentType"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "agent = initialize_agent(\n",
+    "    tools=toolkit.get_tools(),\n",
+    "    llm=llm,\n",
+    "    verbose=False,\n",
+    "    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The draft email was created correctly.'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"Create an email draft for me to edit of a letter from the perspective of a sentient parrot\"\n",
+    "          \" who is looking to collaborate on some research with her\"\n",
+    "          \" estranged friend, a cat. Under no circumstances may you send the message, however.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"I found one draft in your drafts folder about collaboration. It was sent on 2023-06-16T18:22:17+0000 and the subject was 'Collaboration Request'.\""
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"Could you search in my drafts folder and let me know if any of them are about collaboration?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/vscode/langchain-py-env/lib/python3.11/site-packages/O365/utils/windows_tz.py:639: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html\n",
+      "  iana_tz.zone if isinstance(iana_tz, tzinfo) else iana_tz)\n",
+      "/home/vscode/langchain-py-env/lib/python3.11/site-packages/O365/utils/utils.py:463: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html\n",
+      "  timezone = date_time.tzinfo.zone if date_time.tzinfo is not None else None\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'I have scheduled a meeting with a sentient parrot to discuss research collaborations on October 3, 2023 at 2 pm Easter Time. Please let me know if you need to make any changes.'"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"Can you schedule a 30 minute meeting with a sentient parrot to discuss research collaborations on October 3, 2023 at 2 pm Easter Time?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"Yes, you have an event on October 3, 2023 with a sentient parrot. The event is titled 'Meeting with sentient parrot' and is scheduled from 6:00 PM to 6:30 PM.\""
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "agent.run(\"Can you tell me if I have any events on October 3, 2023 in Eastern Time, and if so, tell me if any of them are with a sentient parrot?\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/extras/modules/agents/tools/integrations/bing_search.ipynb

@@ -26,8 +26,8 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"BING_SUBSCRIPTION_KEY\"] = \"\"\n",
-    "os.environ[\"BING_SEARCH_URL\"] = \"\""
+    "os.environ[\"BING_SUBSCRIPTION_KEY\"] = \"<key>\"\n",
+    "os.environ[\"BING_SEARCH_URL\"] = \"https://api.bing.microsoft.com/v7.0/search\""
    ]
   },
   {

docs/extras/modules/agents/tools/integrations/brave_search.ipynb

@@ -12,7 +12,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "a4c896e5",
    "metadata": {},
    "outputs": [],
@@ -27,7 +27,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "api_key = \"...\""
+    "api_key = \"BSAv1neIuQOsxqOyy0sEe_ie2zD_n_V\""
    ]
   },
   {
@@ -49,7 +49,7 @@
     {
      "data": {
       "text/plain": [
-       "'[{\"title\": \"Barack Obama - Wikipedia\", \"link\": \"https://en.wikipedia.org/wiki/Barack_Obama\", \"snippet\": \"Outside of politics, <strong>Obama</strong> has published three bestselling books: Dreams from My Father (1995), The Audacity of Hope (2006) and A Promised Land (2020). Rankings by scholars and historians, in which he has been featured since 2010, place him in the <strong>middle</strong> to upper tier of American presidents.\"}, {\"title\": \"Obama\\'s Middle Name -- My Last Name -- is \\'Hussein.\\' So?\", \"link\": \"https://www.cair.com/cair_in_the_news/obamas-middle-name-my-last-name-is-hussein-so/\", \"snippet\": \"Many Americans understand that common names don\\\\u2019t only come in the form of a \\\\u201cSmith\\\\u201d or a \\\\u201cJohnson.\\\\u201d Perhaps, they have a neighbor, mechanic or teacher named Hussein. Or maybe they\\\\u2019ve seen fashion designer Hussein Chalayan in the pages of Vogue or recall <strong>King Hussein</strong>, our ally in the Middle East.\"}, {\"title\": \"What\\'s up with Obama\\'s middle name? - Quora\", \"link\": \"https://www.quora.com/Whats-up-with-Obamas-middle-name\", \"snippet\": \"Answer (1 of 15): A better question would be, \\\\u201cWhat\\\\u2019s up with Obama\\\\u2019s first name?\\\\u201d President <strong>Barack Hussein Obama</strong>\\\\u2019s father\\\\u2019s name was <strong>Barack Hussein Obama</strong>. He was named after his father. Hussein, Obama\\\\u2019s middle name, is a very common Arabic name, meaning &quot;good,&quot; &quot;handsome,&quot; or &quot;beautiful.&quot;\"}]'"
+       "'[{\"title\": \"Obama\\'s Middle Name -- My Last Name -- is \\'Hussein.\\' So?\", \"link\": \"https://www.cair.com/cair_in_the_news/obamas-middle-name-my-last-name-is-hussein-so/\", \"snippet\": \"I wasn\\\\u2019t sure whether to laugh or cry a few days back listening to radio talk show host Bill Cunningham repeatedly scream Barack <strong>Obama</strong>\\\\u2019<strong>s</strong> <strong>middle</strong> <strong>name</strong> \\\\u2014 my last <strong>name</strong> \\\\u2014 as if he had anti-Muslim Tourette\\\\u2019s. \\\\u201cHussein,\\\\u201d Cunningham hissed like he was beckoning Satan when shouting the ...\"}, {\"title\": \"What\\'s up with Obama\\'s middle name? - Quora\", \"link\": \"https://www.quora.com/Whats-up-with-Obamas-middle-name\", \"snippet\": \"Answer (1 of 15): A better question would be, \\\\u201cWhat\\\\u2019s up with <strong>Obama</strong>\\\\u2019s first <strong>name</strong>?\\\\u201d President Barack Hussein <strong>Obama</strong>\\\\u2019s father\\\\u2019s <strong>name</strong> was Barack Hussein <strong>Obama</strong>. He was <strong>named</strong> after his father. Hussein, <strong>Obama</strong>\\\\u2019<strong>s</strong> <strong>middle</strong> <strong>name</strong>, is a very common Arabic <strong>name</strong>, meaning &quot;good,&quot; &quot;handsome,&quot; or ...\"}, {\"title\": \"Barack Obama | Biography, Parents, Education, Presidency, Books, ...\", \"link\": \"https://www.britannica.com/biography/Barack-Obama\", \"snippet\": \"Barack <strong>Obama</strong>, in full Barack Hussein <strong>Obama</strong> II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009\\\\u201317) and the first African American to hold the office. Before winning the presidency, <strong>Obama</strong> represented Illinois in the U.S.\"}]'"
       ]
      },
      "execution_count": 4,
@@ -86,7 +86,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/extras/modules/agents/tools/integrations/dataforseo.ipynb

@@ -0,0 +1,226 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# DataForSeo API Wrapper\n",
+    "This notebook demonstrates how to use the DataForSeo API wrapper to obtain search engine results. The DataForSeo API allows users to retrieve SERP from most popular search engines like Google, Bing, Yahoo. It also allows to get SERPs from different search engine types like Maps, News, Events, etc.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.utilities import DataForSeoAPIWrapper"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setting up the API wrapper with your credentials\n",
+    "You can obtain your API credentials by registering on the DataForSeo website."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ[\"DATAFORSEO_LOGIN\"] = \"your_api_access_username\"\n",
+    "os.environ[\"DATAFORSEO_PASSWORD\"] = \"your_api_access_password\"\n",
+    "\n",
+    "wrapper = DataForSeoAPIWrapper()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The run method will return the first result snippet from one of the following elements: answer_box, knowledge_graph, featured_snippet, shopping, organic."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "wrapper.run(\"Weather in Los Angeles\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The Difference Between `run` and `results`\n",
+    "`run` and `results` are two methods provided by the `DataForSeoAPIWrapper` class.\n",
+    "\n",
+    "The `run` method executes the search and returns the first result snippet from the answer box, knowledge graph, featured snippet, shopping, or organic results. These elements are sorted by priority from highest to lowest.\n",
+    "\n",
+    "The `results` method returns a JSON response configured according to the parameters set in the wrapper. This allows for more flexibility in terms of what data you want to return from the API."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Getting Results as JSON\n",
+    "You can customize the result types and fields you want to return in the JSON response. You can also set a maximum count for the number of top results to return."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "json_wrapper = DataForSeoAPIWrapper(\n",
+    "    json_result_types=[\"organic\", \"knowledge_graph\", \"answer_box\"],\n",
+    "    json_result_fields=[\"type\", \"title\", \"description\", \"text\"],\n",
+    "    top_count=3)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "json_wrapper.results(\"Bill Gates\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Customizing Location and Language\n",
+    "You can specify the location and language of your search results by passing additional parameters to the API wrapper."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "customized_wrapper = DataForSeoAPIWrapper(\n",
+    "    top_count=10,\n",
+    "    json_result_types=[\"organic\", \"local_pack\"],\n",
+    "    json_result_fields=[\"title\", \"description\", \"type\"],\n",
+    "    params={\"location_name\": \"Germany\", \"language_code\": \"en\"})\n",
+    "customized_wrapper.results(\"coffee near me\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Customizing the Search Engine\n",
+    "You can also specify the search engine you want to use."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "customized_wrapper = DataForSeoAPIWrapper(\n",
+    "    top_count=10,\n",
+    "    json_result_types=[\"organic\", \"local_pack\"],\n",
+    "    json_result_fields=[\"title\", \"description\", \"type\"],\n",
+    "    params={\"location_name\": \"Germany\", \"language_code\": \"en\", \"se_name\": \"bing\"})\n",
+    "customized_wrapper.results(\"coffee near me\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Customizing the Search Type\n",
+    "The API wrapper also allows you to specify the type of search you want to perform. For example, you can perform a maps search."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "maps_search = DataForSeoAPIWrapper(\n",
+    "    top_count=10,\n",
+    "    json_result_fields=[\"title\", \"value\", \"address\", \"rating\", \"type\"],\n",
+    "    params={\"location_coordinate\": \"52.512,13.36,12z\", \"language_code\": \"en\", \"se_type\": \"maps\"})\n",
+    "maps_search.results(\"coffee near me\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Integration with Langchain Agents\n",
+    "You can use the `Tool` class from the `langchain.agents` module to integrate the `DataForSeoAPIWrapper` with a langchain agent. The `Tool` class encapsulates a function that the agent can call."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.agents import Tool\n",
+    "search = DataForSeoAPIWrapper(\n",
+    "    top_count=3,\n",
+    "    json_result_types=[\"organic\"],\n",
+    "    json_result_fields=[\"title\", \"description\", \"type\"])\n",
+    "tool = Tool(\n",
+    "    name=\"google-search-answer\",\n",
+    "    description=\"My new answer tool\",\n",
+    "    func=search.run,\n",
+    ")\n",
+    "json_tool = Tool(\n",
+    "    name=\"google-search-json\",\n",
+    "    description=\"My new json tool\",\n",
+    "    func=search.results,\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/extras/modules/agents/tools/integrations/twilio.ipynb

@@ -1,16 +1,20 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "dc23c48e",
    "metadata": {},
    "source": [
     "# Twilio\n",
     "\n",
-    "This notebook goes over how to use the [Twilio](https://www.twilio.com) API wrapper to send a text message."
+    "This notebook goes over how to use the [Twilio](https://www.twilio.com) API wrapper to send a message through SMS or [Twilio Messaging Channels](https://www.twilio.com/docs/messaging/channels).\n",
+    "\n",
+    "Twilio Messaging Channels facilitates integrations with 3rd party messaging apps and lets you send messages through WhatsApp Business Platform (GA), Facebook Messenger (Public Beta) and Google Business Messages (Private Beta)."
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "c1a33b13",
    "metadata": {},
@@ -31,6 +35,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "f7e883ae",
    "metadata": {},
@@ -41,11 +46,12 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "36c133be",
    "metadata": {},
    "source": [
-    "## Sending a message"
+    "## Sending an SMS"
    ]
   },
   {
@@ -81,6 +87,58 @@
    "source": [
     "twilio.run(\"hello world\", \"+16162904619\")"
    ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "de022dc9",
+   "metadata": {},
+   "source": [
+    "## Sending a WhatsApp Message"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "a594d0bc",
+   "metadata": {},
+   "source": [
+    "You'll need to link your WhatsApp Business Account with Twilio. You'll also need to make sure that the number to send messages from is configured as a WhatsApp Enabled Sender on Twilio and registered with WhatsApp."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "94508aa0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.utilities.twilio import TwilioAPIWrapper"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e4b81750",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "twilio = TwilioAPIWrapper(\n",
+    "    #     account_sid=\"foo\",\n",
+    "    #     auth_token=\"bar\",\n",
+    "    #     from_number=\"whatsapp: baz,\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1181041b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "twilio.run(\"hello world\", \"whatsapp: +16162904619\")"
+   ]
   }
  ],
  "metadata": {

docs/extras/modules/agents/tools/integrations/zapier.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Zapier Natural Language Actions API\n",
     "\\\n",
-    "Full docs here: https://nla.zapier.com/api/v1/docs\n",
+    "Full docs here: https://nla.zapier.com/start/\n",
     "\n",
     "**Zapier Natural Language Actions** gives you access to the 5k+ apps, 20k+ actions on Zapier's platform through a natural language API interface.\n",
     "\n",
@@ -21,7 +21,7 @@
     "\n",
     "2. User-facing (Oauth): for production scenarios where you are deploying an end-user facing application and LangChain needs access to end-user's exposed actions and connected accounts on Zapier.com\n",
     "\n",
-    "This quick start will focus on the server-side use case for brevity. Review [full docs](https://nla.zapier.com/api/v1/docs) or reach out to nla@zapier.com for user-facing oauth developer support.\n",
+    "This quick start will focus mostly on the server-side use case for brevity. Jump to [Example Using OAuth Access Token](#oauth) to see a short example how to set up Zapier for user-facing situations. Review [full docs](https://nla.zapier.com/start/) for full user-facing oauth developer support.\n",
     "\n",
     "This example goes over how to use the Zapier integration with a `SimpleSequentialChain`, then an `Agent`.\n",
     "In code, below:"
@@ -39,7 +39,7 @@
     "# get from https://platform.openai.com/\n",
     "os.environ[\"OPENAI_API_KEY\"] = os.environ.get(\"OPENAI_API_KEY\", \"\")\n",
     "\n",
-    "# get from https://nla.zapier.com/demo/provider/debug (under User Information, after logging in):\n",
+    "# get from https://nla.zapier.com/docs/authentication/ after logging in):\n",
     "os.environ[\"ZAPIER_NLA_API_KEY\"] = os.environ.get(\"ZAPIER_NLA_API_KEY\", \"\")"
    ]
   },
@@ -149,7 +149,7 @@
    "id": "bcdea831",
    "metadata": {},
    "source": [
-    "# Example with SimpleSequentialChain\n",
+    "## Example with SimpleSequentialChain\n",
     "If you need more explicit control, use a chain, like below."
    ]
   },
@@ -323,12 +323,34 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "id": "09ff954e-45f2-4595-92ea-91627abde4a0",
    "metadata": {},
+   "source": [
+    "## <a id=\"oauth\">Example Using OAuth Access Token</a>\n",
+    "The below snippet shows how to initialize the wrapper with a procured OAuth access token. Note the argument being passed in as opposed to setting an environment variable. Review the [authentication docs](https://nla.zapier.com/docs/authentication/#oauth-credentials) for full user-facing oauth developer support.\n",
+    "\n",
+    "The developer is tasked with handling the OAuth handshaking to procure and refresh the access token."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7c6835c8",
+   "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "llm = OpenAI(temperature=0)\n",
+    "zapier = ZapierNLAWrapper(zapier_nla_oauth_access_token='<fill in access token here>')\n",
+    "toolkit = ZapierToolkit.from_zapier_nla_wrapper(zapier)\n",
+    "agent = initialize_agent(\n",
+    "    toolkit.get_tools(), llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n",
+    ")\n",
+    "\n",
+    "agent.run(\n",
+    "    \"Summarize the last email I received regarding Silicon Valley Bank. Send the summary to the #test-zapier channel in slack.\"\n",
+    ")"
+   ]
   }
  ],
  "metadata": {

docs/extras/modules/callbacks/integrations/streamlit.md

@@ -0,0 +1,73 @@
+# Streamlit
+
+> **[Streamlit](https://streamlit.io/) is a faster way to build and share data apps.**
+> Streamlit turns data scripts into shareable web apps in minutes. All in pure Python. No front‑end experience required.
+> See more examples at [streamlit.io/generative-ai](https://streamlit.io/generative-ai).
+
+[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/langchain-ai/streamlit-agent?quickstart=1)
+
+In this guide we will demonstrate how to use `StreamlitCallbackHandler` to display the thoughts and actions of an agent in an
+interactive Streamlit app. Try it out with the running app below using the [MRKL agent](/docs/modules/agents/how_to/mrkl/):
+
+<iframe loading="lazy" src="https://mrkl-minimal.streamlit.app/?embed=true&embed_options=light_theme"
+    style={{ width: 100 + '%', border: 'none', marginBottom: 1 + 'rem', height: 600 }}
+    allow="camera;clipboard-read;clipboard-write;"
+></iframe>
+
+## Installation and Setup
+
+```bash
+pip install langchain streamlit
+```
+
+You can run `streamlit hello` to load a sample app and validate your install succeeded. See full instructions in Streamlit's
+[Getting started documentation](https://docs.streamlit.io/library/get-started).
+
+## Display thoughts and actions
+
+To create a `StreamlitCallbackHandler`, you just need to provide a parent container to render the output.
+
+```python
+from langchain.callbacks import StreamlitCallbackHandler
+import streamlit as st
+
+st_callback = StreamlitCallbackHandler(st.container())
+```
+
+Additional keyword arguments to customize the display behavior are described in the
+[API reference](https://api.python.langchain.com/en/latest/modules/callbacks.html#langchain.callbacks.StreamlitCallbackHandler).
+
+### Scenario 1: Using an Agent with Tools
+
+The primary supported use case today is visualizing the actions of an Agent with Tools (or Agent Executor). You can create an
+agent in your Streamlit app and simply pass the `StreamlitCallbackHandler` to `agent.run()` in order to visualize the
+thoughts and actions live in your app.
+
+```python
+from langchain.llms import OpenAI
+from langchain.agents import AgentType, initialize_agent, load_tools
+from langchain.callbacks import StreamlitCallbackHandler
+import streamlit as st
+
+llm = OpenAI(temperature=0, streaming=True)
+tools = load_tools(["ddg-search"])
+agent = initialize_agent(
+    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True
+)
+
+if prompt := st.chat_input():
+    st.chat_message("user").write(prompt)
+    with st.chat_message("assistant"):
+        st_callback = StreamlitCallbackHandler(st.container())
+        response = agent.run(prompt, callbacks=[st_callback])
+        st.write(response)
+```
+
+**Note:** You will need to set `OPENAI_API_KEY` for the above app code to run successfully.
+The easiest way to do this is via [Streamlit secrets.toml](https://docs.streamlit.io/library/advanced-features/secrets-management),
+or any other local ENV management tool.
+
+### Additional scenarios
+
+Currently `StreamlitCallbackHandler` is geared towards use with a LangChain Agent Executor. Support for additional agent types,
+use directly with Chains, etc will be added in the future.

docs/extras/modules/chains/additional/flare.ipynb

@@ -71,11 +71,13 @@
     "import numpy as np\n",
     "\n",
     "from langchain.schema import BaseRetriever\n",
+    "from langchain.callbacks.manager import AsyncCallbackManagerForRetrieverRun, CallbackManagerForRetrieverRun\n",
     "from langchain.utilities import GoogleSerperAPIWrapper\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
     "from langchain.chat_models import ChatOpenAI\n",
     "from langchain.llms import OpenAI\n",
-    "from langchain.schema import Document"
+    "from langchain.schema import Document\n",
+    "from typing import Any"
    ]
   },
   {
@@ -97,11 +99,16 @@
     "    def __init__(self, search):\n",
     "        self.search = search\n",
     "\n",
-    "    def get_relevant_documents(self, query: str):\n",
+    "    def _get_relevant_documents(self, query: str, *, run_manager: CallbackManagerForRetrieverRun, **kwargs: Any) -> List[Document]:\n",
     "        return [Document(page_content=self.search.run(query))]\n",
     "\n",
-    "    async def aget_relevant_documents(self, query: str):\n",
-    "        raise NotImplemented\n",
+    "    async def _aget_relevant_documents(self,\n",
+    "        query: str,\n",
+    "        *,\n",
+    "        run_manager: AsyncCallbackManagerForRetrieverRun,\n",
+    "        **kwargs: Any,\n",
+    "    ) -> List[Document]:\n",
+    "        raise NotImplementedError()\n",
     "\n",
     "\n",
     "retriever = SerperSearchRetriever(GoogleSerperAPIWrapper())"

docs/extras/modules/chains/additional/graph_hugegraph_qa.ipynb

@@ -0,0 +1,302 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d2777010",
+   "metadata": {},
+   "source": [
+    "# HugeGraph QA Chain\n",
+    "\n",
+    "This notebook shows how to use LLMs to provide a natural language interface to [HugeGraph](https://hugegraph.apache.org/cn/) database."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f26dcbe4",
+   "metadata": {},
+   "source": [
+    "You will need to have a running HugeGraph instance.\n",
+    "You can run a local docker container by running the executing the following script:\n",
+    "\n",
+    "```\n",
+    "docker run \\\n",
+    "    --name=graph \\\n",
+    "    -itd \\\n",
+    "    -p 8080:8080 \\\n",
+    "    hugegraph/hugegraph\n",
+    "```\n",
+    "\n",
+    "If we want to connect HugeGraph in the application, we need to install python sdk:\n",
+    "\n",
+    "```\n",
+    "pip3 install hugegraph-python\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d64a29f1",
+   "metadata": {},
+   "source": [
+    "If you are using the docker container, you need to wait a couple of second for the database to start, and then we need create schema and write graph data for the database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "e53ab93e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hugegraph.connection import PyHugeGraph\n",
+    "\n",
+    "client = PyHugeGraph(\"localhost\", \"8080\", user=\"admin\", pwd=\"admin\", graph=\"hugegraph\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b7c3a50e",
+   "metadata": {},
+   "source": [
+    "First, we create the schema for a simple movie database:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "ef5372a8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'create EdgeLabel success, Detail: \"b\\'{\"id\":1,\"name\":\"ActedIn\",\"source_label\":\"Person\",\"target_label\":\"Movie\",\"frequency\":\"SINGLE\",\"sort_keys\":[],\"nullable_keys\":[],\"index_labels\":[],\"properties\":[],\"status\":\"CREATED\",\"ttl\":0,\"enable_label_index\":true,\"user_data\":{\"~create_time\":\"2023-07-04 10:48:47.908\"}}\\'\"'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "\"\"\"schema\"\"\"\n",
+    "schema = client.schema()\n",
+    "schema.propertyKey(\"name\").asText().ifNotExist().create()\n",
+    "schema.propertyKey(\"birthDate\").asText().ifNotExist().create()\n",
+    "schema.vertexLabel(\"Person\").properties(\"name\", \"birthDate\").usePrimaryKeyId().primaryKeys(\"name\").ifNotExist().create()\n",
+    "schema.vertexLabel(\"Movie\").properties(\"name\").usePrimaryKeyId().primaryKeys(\"name\").ifNotExist().create()\n",
+    "schema.edgeLabel(\"ActedIn\").sourceLabel(\"Person\").targetLabel(\"Movie\").ifNotExist().create()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "016f7989",
+   "metadata": {},
+   "source": [
+    "Then we can insert some data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "b7f4c370",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "1:Robert De Niro--ActedIn-->2:The Godfather Part II"
+      ]
+     },
+     "execution_count": 26,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "\"\"\"graph\"\"\"\n",
+    "g = client.graph()\n",
+    "g.addVertex(\"Person\", {\"name\": \"Al Pacino\", \"birthDate\": \"1940-04-25\"})\n",
+    "g.addVertex(\"Person\", {\"name\": \"Robert De Niro\", \"birthDate\": \"1943-08-17\"})\n",
+    "g.addVertex(\"Movie\", {\"name\": \"The Godfather\"})\n",
+    "g.addVertex(\"Movie\", {\"name\": \"The Godfather Part II\"})\n",
+    "g.addVertex(\"Movie\", {\"name\": \"The Godfather Coda The Death of Michael Corleone\"})\n",
+    "\n",
+    "g.addEdge(\"ActedIn\", \"1:Al Pacino\", \"2:The Godfather\", {})\n",
+    "g.addEdge(\"ActedIn\", \"1:Al Pacino\", \"2:The Godfather Part II\", {})\n",
+    "g.addEdge(\"ActedIn\", \"1:Al Pacino\", \"2:The Godfather Coda The Death of Michael Corleone\", {})\n",
+    "g.addEdge(\"ActedIn\", \"1:Robert De Niro\", \"2:The Godfather Part II\", {})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5b8f7788",
+   "metadata": {},
+   "source": [
+    "## Creating `HugeGraphQAChain`\n",
+    "\n",
+    "We can now create the `HugeGraph` and `HugeGraphQAChain`. To create the `HugeGraph` we simply need to pass the database object to the `HugeGraph` constructor."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "id": "f1f68fcf",
+   "metadata": {
+    "is_executing": true
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains import HugeGraphQAChain\n",
+    "from langchain.graphs import HugeGraph"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "b86ebfa7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "graph = HugeGraph(\n",
+    "    username=\"admin\",\n",
+    "    password=\"admin\",\n",
+    "    address=\"localhost\",\n",
+    "    port=8080,\n",
+    "    graph=\"hugegraph\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e262540b",
+   "metadata": {},
+   "source": [
+    "## Refresh graph schema information\n",
+    "\n",
+    "If the schema of database changes, you can refresh the schema information needed to generate Gremlin statements."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "134dd8d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# graph.refresh_schema()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "e78b8e72",
+   "metadata": {
+    "ExecuteTime": {}
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Node properties: [name: Person, primary_keys: ['name'], properties: ['name', 'birthDate'], name: Movie, primary_keys: ['name'], properties: ['name']]\n",
+      "Edge properties: [name: ActedIn, properties: []]\n",
+      "Relationships: ['Person--ActedIn-->Movie']\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(graph.get_schema)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5c27e813",
+   "metadata": {},
+   "source": [
+    "## Querying the graph\n",
+    "\n",
+    "We can now use the graph Gremlin QA chain to ask question of the graph"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "3b23dead",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = HugeGraphQAChain.from_llm(\n",
+    "    ChatOpenAI(temperature=0), graph=graph, verbose=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "id": "76aecc93",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "Generated gremlin:\n",
+      "\u001b[32;1m\u001b[1;3mg.V().has('Movie', 'name', 'The Godfather').in('ActedIn').valueMap(true)\u001b[0m\n",
+      "Full Context:\n",
+      "\u001b[32;1m\u001b[1;3m[{'id': '1:Al Pacino', 'label': 'Person', 'name': ['Al Pacino'], 'birthDate': ['1940-04-25']}]\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Al Pacino played in The Godfather.'"
+      ]
+     },
+     "execution_count": 32,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"Who played in The Godfather?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "869f0258",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "venv"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/modules/chains/additional/graph_sparql_qa.ipynb

@@ -0,0 +1,300 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "c94240f5",
+   "metadata": {},
+   "source": [
+    "# GraphSparqlQAChain\n",
+    "\n",
+    "Graph databases are an excellent choice for applications based on network-like models. To standardize the syntax and semantics of such graphs, the W3C recommends Semantic Web Technologies, cp. [Semantic Web](https://www.w3.org/standards/semanticweb/). [SPARQL](https://www.w3.org/TR/sparql11-query/) serves as a query language analogously to SQL or Cypher for these graphs. This notebook demonstrates the application of LLMs as a natural language interface to a graph database by generating SPARQL.\\\n",
+    "Disclaimer: To date, SPARQL query generation via LLMs is still a bit unstable. Be especially careful with UPDATE queries, which alter the graph."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dbc0ee68",
+   "metadata": {},
+   "source": [
+    "There are several sources you can run queries against, including files on the web, files you have available locally, SPARQL endpoints, e.g., [Wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page), and [triple stores](https://www.w3.org/wiki/LargeTripleStores)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "62812aad",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.chains import GraphSparqlQAChain\n",
+    "from langchain.graphs import RdfGraph"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "0928915d",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "graph = RdfGraph(\n",
+    "    source_file=\"http://www.w3.org/People/Berners-Lee/card\",\n",
+    "    standard=\"rdf\",\n",
+    "    local_copy=\"test.ttl\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Note that providing a `local_file` is necessary for storing changes locally if the source is read-only."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "id": "58c1a8ea",
+   "metadata": {},
+   "source": [
+    "## Refresh graph schema information\n",
+    "If the schema of the database changes, you can refresh the schema information needed to generate SPARQL queries."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "4e3de44f",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "graph.load_schema()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1fe76ccd",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "In the following, each IRI is followed by the local name and optionally its description in parentheses. \n",
+      "The RDF graph supports the following node types:\n",
+      "<http://xmlns.com/foaf/0.1/PersonalProfileDocument> (PersonalProfileDocument, None), <http://www.w3.org/ns/auth/cert#RSAPublicKey> (RSAPublicKey, None), <http://www.w3.org/2000/10/swap/pim/contact#Male> (Male, None), <http://xmlns.com/foaf/0.1/Person> (Person, None), <http://www.w3.org/2006/vcard/ns#Work> (Work, None)\n",
+      "The RDF graph supports the following relationships:\n",
+      "<http://www.w3.org/2000/01/rdf-schema#seeAlso> (seeAlso, None), <http://purl.org/dc/elements/1.1/title> (title, None), <http://xmlns.com/foaf/0.1/mbox_sha1sum> (mbox_sha1sum, None), <http://xmlns.com/foaf/0.1/maker> (maker, None), <http://www.w3.org/ns/solid/terms#oidcIssuer> (oidcIssuer, None), <http://www.w3.org/2000/10/swap/pim/contact#publicHomePage> (publicHomePage, None), <http://xmlns.com/foaf/0.1/openid> (openid, None), <http://www.w3.org/ns/pim/space#storage> (storage, None), <http://xmlns.com/foaf/0.1/name> (name, None), <http://www.w3.org/2000/10/swap/pim/contact#country> (country, None), <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> (type, None), <http://www.w3.org/ns/solid/terms#profileHighlightColor> (profileHighlightColor, None), <http://www.w3.org/ns/pim/space#preferencesFile> (preferencesFile, None), <http://www.w3.org/2000/01/rdf-schema#label> (label, None), <http://www.w3.org/ns/auth/cert#modulus> (modulus, None), <http://www.w3.org/2000/10/swap/pim/contact#participant> (participant, None), <http://www.w3.org/2000/10/swap/pim/contact#street2> (street2, None), <http://www.w3.org/2006/vcard/ns#locality> (locality, None), <http://xmlns.com/foaf/0.1/nick> (nick, None), <http://xmlns.com/foaf/0.1/homepage> (homepage, None), <http://creativecommons.org/ns#license> (license, None), <http://xmlns.com/foaf/0.1/givenname> (givenname, None), <http://www.w3.org/2006/vcard/ns#street-address> (street-address, None), <http://www.w3.org/2006/vcard/ns#postal-code> (postal-code, None), <http://www.w3.org/2000/10/swap/pim/contact#street> (street, None), <http://www.w3.org/2003/01/geo/wgs84_pos#lat> (lat, None), <http://xmlns.com/foaf/0.1/primaryTopic> (primaryTopic, None), <http://www.w3.org/2006/vcard/ns#fn> (fn, None), <http://www.w3.org/2003/01/geo/wgs84_pos#location> (location, None), <http://usefulinc.com/ns/doap#developer> (developer, None), <http://www.w3.org/2000/10/swap/pim/contact#city> (city, None), <http://www.w3.org/2006/vcard/ns#region> (region, None), <http://xmlns.com/foaf/0.1/member> (member, None), <http://www.w3.org/2003/01/geo/wgs84_pos#long> (long, None), <http://www.w3.org/2000/10/swap/pim/contact#address> (address, None), <http://xmlns.com/foaf/0.1/family_name> (family_name, None), <http://xmlns.com/foaf/0.1/account> (account, None), <http://xmlns.com/foaf/0.1/workplaceHomepage> (workplaceHomepage, None), <http://purl.org/dc/terms/title> (title, None), <http://www.w3.org/ns/solid/terms#publicTypeIndex> (publicTypeIndex, None), <http://www.w3.org/2000/10/swap/pim/contact#office> (office, None), <http://www.w3.org/2000/10/swap/pim/contact#homePage> (homePage, None), <http://xmlns.com/foaf/0.1/mbox> (mbox, None), <http://www.w3.org/2000/10/swap/pim/contact#preferredURI> (preferredURI, None), <http://www.w3.org/ns/solid/terms#profileBackgroundColor> (profileBackgroundColor, None), <http://schema.org/owns> (owns, None), <http://xmlns.com/foaf/0.1/based_near> (based_near, None), <http://www.w3.org/2006/vcard/ns#hasAddress> (hasAddress, None), <http://xmlns.com/foaf/0.1/img> (img, None), <http://www.w3.org/2000/10/swap/pim/contact#assistant> (assistant, None), <http://xmlns.com/foaf/0.1/title> (title, None), <http://www.w3.org/ns/auth/cert#key> (key, None), <http://www.w3.org/ns/ldp#inbox> (inbox, None), <http://www.w3.org/ns/solid/terms#editableProfile> (editableProfile, None), <http://www.w3.org/2000/10/swap/pim/contact#postalCode> (postalCode, None), <http://xmlns.com/foaf/0.1/weblog> (weblog, None), <http://www.w3.org/ns/auth/cert#exponent> (exponent, None), <http://rdfs.org/sioc/ns#avatar> (avatar, None)\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "graph.get_schema"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "68a3c677",
+   "metadata": {},
+   "source": [
+    "## Querying the graph\n",
+    "\n",
+    "Now, you can use the graph SPARQL QA chain to ask questions about the graph."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "7476ce98",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "chain = GraphSparqlQAChain.from_llm(\n",
+    "    ChatOpenAI(temperature=0), graph=graph, verbose=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "ef8ee27b",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001B[1m> Entering new GraphSparqlQAChain chain...\u001B[0m\n",
+      "Identified intent:\n",
+      "\u001B[32;1m\u001B[1;3mSELECT\u001B[0m\n",
+      "Generated SPARQL:\n",
+      "\u001B[32;1m\u001B[1;3mPREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
+      "SELECT ?homepage\n",
+      "WHERE {\n",
+      "    ?person foaf:name \"Tim Berners-Lee\" .\n",
+      "    ?person foaf:workplaceHomepage ?homepage .\n",
+      "}\u001B[0m\n",
+      "Full Context:\n",
+      "\u001B[32;1m\u001B[1;3m[]\u001B[0m\n",
+      "\n",
+      "\u001B[1m> Finished chain.\u001B[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"Tim Berners-Lee's work homepage is http://www.w3.org/People/Berners-Lee/.\""
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"What is Tim Berners-Lee's work homepage?\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "af4b3294",
+   "metadata": {},
+   "source": [
+    "## Updating the graph\n",
+    "\n",
+    "Analogously, you can update the graph, i.e., insert triples, using natural language."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "fdf38841",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001B[1m> Entering new GraphSparqlQAChain chain...\u001B[0m\n",
+      "Identified intent:\n",
+      "\u001B[32;1m\u001B[1;3mUPDATE\u001B[0m\n",
+      "Generated SPARQL:\n",
+      "\u001B[32;1m\u001B[1;3mPREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
+      "INSERT {\n",
+      "    ?person foaf:workplaceHomepage <http://www.w3.org/foo/bar/> .\n",
+      "}\n",
+      "WHERE {\n",
+      "    ?person foaf:name \"Timothy Berners-Lee\" .\n",
+      "}\u001B[0m\n",
+      "\n",
+      "\u001B[1m> Finished chain.\u001B[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Successfully inserted triples into the graph.'"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"Save that the person with the name 'Timothy Berners-Lee' has a work homepage at 'http://www.w3.org/foo/bar/'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e0f7fc1",
+   "metadata": {},
+   "source": [
+    "Let's verify the results:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "f874171b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[(rdflib.term.URIRef('https://www.w3.org/'),),\n",
+       " (rdflib.term.URIRef('http://www.w3.org/foo/bar/'),)]"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "query = (\n",
+    "    \"\"\"PREFIX foaf: <http://xmlns.com/foaf/0.1/>\\n\"\"\"\n",
+    "    \"\"\"SELECT ?hp\\n\"\"\"\n",
+    "    \"\"\"WHERE {\\n\"\"\"\n",
+    "    \"\"\"    ?person foaf:name \"Timothy Berners-Lee\" . \\n\"\"\"\n",
+    "    \"\"\"    ?person foaf:workplaceHomepage ?hp .\\n\"\"\"\n",
+    "    \"\"\"}\"\"\"\n",
+    ")\n",
+    "graph.query(query)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "lc",
+   "language": "python",
+   "name": "lc"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/modules/chains/additional/openapi_openai.ipynb

@@ -0,0 +1,245 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e734b314",
+   "metadata": {},
+   "source": [
+    "# OpenAPI calls with OpenAI functions\n",
+    "\n",
+    "In this notebook we'll show how to create a chain that automatically makes calls to an API based only on an OpenAPI  spec. Under the hood, we're parsing the OpenAPI spec into a JSON schema that the OpenAI functions API can handle. This allows ChatGPT to automatically select and populate the relevant API call to make for any user input. Using the output of ChatGPT we then make the actual API call, and return the result."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "555661b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.chains.openai_functions.openapi import get_openapi_chain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a95f510a",
+   "metadata": {},
+   "source": [
+    "## Query Klarna"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "08e19b64",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = get_openapi_chain(\"https://www.klarna.com/us/shopping/public/openai/v0/api-docs/\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "3959f866",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'products': [{'name': \"Tommy Hilfiger Men's Short Sleeve Button-Down Shirt\",\n",
+       "   'url': 'https://www.klarna.com/us/shopping/pl/cl10001/3204878580/Clothing/Tommy-Hilfiger-Men-s-Short-Sleeve-Button-Down-Shirt/?utm_source=openai&ref-site=openai_plugin',\n",
+       "   'price': '$26.78',\n",
+       "   'attributes': ['Material:Linen,Cotton',\n",
+       "    'Target Group:Man',\n",
+       "    'Color:Gray,Pink,White,Blue,Beige,Black,Turquoise',\n",
+       "    'Size:S,XL,M,XXL']},\n",
+       "  {'name': \"Van Heusen Men's Long Sleeve Button-Down Shirt\",\n",
+       "   'url': 'https://www.klarna.com/us/shopping/pl/cl10001/3201809514/Clothing/Van-Heusen-Men-s-Long-Sleeve-Button-Down-Shirt/?utm_source=openai&ref-site=openai_plugin',\n",
+       "   'price': '$18.89',\n",
+       "   'attributes': ['Material:Cotton',\n",
+       "    'Target Group:Man',\n",
+       "    'Color:Red,Gray,White,Blue',\n",
+       "    'Size:XL,XXL']},\n",
+       "  {'name': 'Brixton Bowery Flannel Shirt',\n",
+       "   'url': 'https://www.klarna.com/us/shopping/pl/cl10001/3202331096/Clothing/Brixton-Bowery-Flannel-Shirt/?utm_source=openai&ref-site=openai_plugin',\n",
+       "   'price': '$34.48',\n",
+       "   'attributes': ['Material:Cotton',\n",
+       "    'Target Group:Man',\n",
+       "    'Color:Gray,Blue,Black,Orange',\n",
+       "    'Size:XL,3XL,4XL,5XL,L,M,XXL']},\n",
+       "  {'name': 'Cubavera Four Pocket Guayabera Shirt',\n",
+       "   'url': 'https://www.klarna.com/us/shopping/pl/cl10001/3202055522/Clothing/Cubavera-Four-Pocket-Guayabera-Shirt/?utm_source=openai&ref-site=openai_plugin',\n",
+       "   'price': '$23.22',\n",
+       "   'attributes': ['Material:Polyester,Cotton',\n",
+       "    'Target Group:Man',\n",
+       "    'Color:Red,White,Blue,Black',\n",
+       "    'Size:S,XL,L,M,XXL']},\n",
+       "  {'name': 'Theory Sylvain Shirt - Eclipse',\n",
+       "   'url': 'https://www.klarna.com/us/shopping/pl/cl10001/3202028254/Clothing/Theory-Sylvain-Shirt-Eclipse/?utm_source=openai&ref-site=openai_plugin',\n",
+       "   'price': '$86.01',\n",
+       "   'attributes': ['Material:Polyester,Cotton',\n",
+       "    'Target Group:Man',\n",
+       "    'Color:Blue',\n",
+       "    'Size:S,XL,XS,L,M,XXL']}]}"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"What are some options for a men's large blue button down shirt\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f648c77",
+   "metadata": {},
+   "source": [
+    "## Query a translation service\n",
+    "\n",
+    "Additionally, see the request payload by setting `verbose=True`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bf6cd695",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = get_openapi_chain(\"https://api.speak.com/openapi.yaml\", verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "1ba51609",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "Prompt after formatting:\n",
+      "\u001b[32;1m\u001b[1;3mHuman: Use the provided API's to respond to this user query:\n",
+      "\n",
+      "How would you say no thanks in Russian\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "\n",
+      "\u001b[1m> Entering new  chain...\u001b[0m\n",
+      "Calling endpoint \u001b[32;1m\u001b[1;3mtranslate\u001b[0m with arguments:\n",
+      "\u001b[32;1m\u001b[1;3m{\n",
+      "  \"json\": {\n",
+      "    \"phrase_to_translate\": \"no thanks\",\n",
+      "    \"learning_language\": \"russian\",\n",
+      "    \"native_language\": \"english\",\n",
+      "    \"additional_context\": \"\",\n",
+      "    \"full_query\": \"How would you say no thanks in Russian\"\n",
+      "  }\n",
+      "}\u001b[0m\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n",
+      "\n",
+      "\u001b[1m> Finished chain.\u001b[0m\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "{'explanation': '<translation language=\"Russian\">\\nНет, спасибо. (Net, spasibo)\\n</translation>\\n\\n<alternatives>\\n1. \"Нет, я в порядке\" *(Neutral/Formal - Can be used in professional settings or formal situations.)*\\n2. \"Нет, спасибо, я откажусь\" *(Formal - Can be used in polite settings, such as a fancy dinner with colleagues or acquaintances.)*\\n3. \"Не надо\" *(Informal - Can be used in informal situations, such as declining an offer from a friend.)*\\n</alternatives>\\n\\n<example-convo language=\"Russian\">\\n<context>Max is being offered a cigarette at a party.</context>\\n* Sasha: \"Хочешь покурить?\"\\n* Max: \"Нет, спасибо. Я бросил.\"\\n* Sasha: \"Окей, понятно.\"\\n</example-convo>\\n\\n*[Report an issue or leave feedback](https://speak.com/chatgpt?rid=noczaa460do8yqs8xjun6zdm})*',\n",
+       " 'extra_response_instructions': 'Use all information in the API response and fully render all Markdown.\\nAlways end your response with a link to report an issue or leave feedback on the plugin.'}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"How would you say no thanks in Russian\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4923a291",
+   "metadata": {},
+   "source": [
+    "## Query XKCD"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a9198f62",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "chain = get_openapi_chain(\"https://gist.githubusercontent.com/roaldnefs/053e505b2b7a807290908fe9aa3e1f00/raw/0a212622ebfef501163f91e23803552411ed00e4/openapi.yaml\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "3110c398",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'month': '6',\n",
+       " 'num': 2793,\n",
+       " 'link': '',\n",
+       " 'year': '2023',\n",
+       " 'news': '',\n",
+       " 'safe_title': 'Garden Path Sentence',\n",
+       " 'transcript': '',\n",
+       " 'alt': 'Arboretum Owner Denied Standing in Garden Path Suit on Grounds Grounds Appealing Appealing',\n",
+       " 'img': 'https://imgs.xkcd.com/comics/garden_path_sentence.png',\n",
+       " 'title': 'Garden Path Sentence',\n",
+       " 'day': '23'}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.run(\"What's today's comic?\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "venv"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/extras/modules/data_connection/document_loaders/integrations/airtable.ipynb

@@ -134,7 +134,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.16"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,

docs/extras/modules/data_connection/document_loaders/integrations/brave_search.ipynb

@@ -0,0 +1,164 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "3dd292b1-9a73-4ea8-af19-5fa6e3c1a62a",
+   "metadata": {},
+   "source": [
+    "# Brave Search\n",
+    "\n",
+    "\n",
+    ">[Brave Search](https://en.wikipedia.org/wiki/Brave_Search) is a search engine developed by Brave Software.\n",
+    "> - `Brave Search` uses its own web index. As of May 2022, it covered over 10 billion pages and was used to serve 92% \n",
+    "> of search results without relying on any third-parties, with the remainder being retrieved \n",
+    "> server-side from the Bing API or (on an opt-in basis) client-side from Google. According \n",
+    "> to Brave, the index was kept \"intentionally smaller than that of Google or Bing\" in order to \n",
+    "> help avoid spam and other low-quality content, with the disadvantage that \"Brave Search is \n",
+    "> not yet as good as Google in recovering long-tail queries.\"\n",
+    ">- `Brave Search Premium`: As of April 2023 Brave Search is an ad-free website, but it will \n",
+    "> eventually switch to a new model that will include ads and premium users will get an ad-free experience.\n",
+    "> User data including IP addresses won't be collected from its users by default. A premium account \n",
+    "> will be required for opt-in data-collection.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "26f0888e-3f3e-4b82-ac4a-2df6feeccbe0",
+   "metadata": {},
+   "source": [
+    "## Installation and Setup\n",
+    "\n",
+    "To get access to the Brave Search API, you need to [create an account and get an API key](https://api.search.brave.com/app/dashboard).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "d7d7be09-58bd-47d7-bf1b-33964564f777",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "api_key = \"...\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b3ac92df-6ff0-4dbb-b32b-a7dc140c48ef",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import BraveSearchLoader"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f483caf-58ef-4138-975a-5b783559dc1b",
+   "metadata": {},
+   "source": [
+    "## Example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "766634cf-3bc7-4656-939a-cafa218807a6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "3"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "loader = BraveSearchLoader(query=\"obama middle name\", api_key=api_key, search_kwargs={\"count\": 3})\n",
+    "docs = loader.load()\n",
+    "len(docs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "f1fcc9f1-cbdc-46b3-89d3-80311d557dc6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'title': \"Obama's Middle Name -- My Last Name -- is 'Hussein.' So?\",\n",
+       "  'link': 'https://www.cair.com/cair_in_the_news/obamas-middle-name-my-last-name-is-hussein-so/'},\n",
+       " {'title': \"What's up with Obama's middle name? - Quora\",\n",
+       "  'link': 'https://www.quora.com/Whats-up-with-Obamas-middle-name'},\n",
+       " {'title': 'Barack Obama | Biography, Parents, Education, Presidency, Books, ...',\n",
+       "  'link': 'https://www.britannica.com/biography/Barack-Obama'}]"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "[doc.metadata for doc in docs]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "601bfd77-03d3-468e-843f-2523d5e215bd",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['I wasn’t sure whether to laugh or cry a few days back listening to radio talk show host Bill Cunningham repeatedly scream Barack <strong>Obama</strong>’<strong>s</strong> <strong>middle</strong> <strong>name</strong> — my last <strong>name</strong> — as if he had anti-Muslim Tourette’s. “Hussein,” Cunningham hissed like he was beckoning Satan when shouting the ...',\n",
+       " 'Answer (1 of 15): A better question would be, “What’s up with <strong>Obama</strong>’s first <strong>name</strong>?” President Barack Hussein <strong>Obama</strong>’s father’s <strong>name</strong> was Barack Hussein <strong>Obama</strong>. He was <strong>named</strong> after his father. Hussein, <strong>Obama</strong>’<strong>s</strong> <strong>middle</strong> <strong>name</strong>, is a very common Arabic <strong>name</strong>, meaning &quot;good,&quot; &quot;handsome,&quot; or ...',\n",
+       " 'Barack <strong>Obama</strong>, in full Barack Hussein <strong>Obama</strong> II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009–17) and the first African American to hold the office. Before winning the presidency, <strong>Obama</strong> represented Illinois in the U.S.']"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "[doc.page_content for doc in docs]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74a6ba54-9e48-4bac-ab9b-03eabd19eb81",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}