core[patch]: Release 0.3.27 (#28799)

- Convert developer openai messages to SystemMessage
- store additional_kwargs={"__openai_role__": "developer"} so that the
correct role can be reconstructed if needed
- update ChatOpenAI to read in openai_role

---------

Co-authored-by: Erick Friis <erick@langchain.dev>

.github/scripts/check_diff.py

@@ -273,6 +273,8 @@ if __name__ == "__main__":
             # note: won't run on external repo partners
             dirs_to_run["lint"].add("libs/standard-tests")
             dirs_to_run["test"].add("libs/standard-tests")
+            dirs_to_run["lint"].add("libs/cli")
+            dirs_to_run["test"].add("libs/cli")
             dirs_to_run["test"].add("libs/partners/mistralai")
             dirs_to_run["test"].add("libs/partners/openai")
             dirs_to_run["test"].add("libs/partners/anthropic")

.github/workflows/check_diffs.yml

@@ -5,6 +5,7 @@ on:
   push:
     branches: [master]
   pull_request:
+  merge_group:
 
 # If another push to the same PR or branch happens while this workflow is still running,
 # cancel the earlier run in favor of the next run.

README.md

@@ -38,18 +38,21 @@ conda install langchain -c conda-forge
 
 For these applications, LangChain simplifies the entire application lifecycle:
 
-- **Open-source libraries**: Build your applications using LangChain's open-source [building blocks](https://python.langchain.com/docs/concepts/#langchain-expression-language-lcel), [components](https://python.langchain.com/docs/concepts/), and [third-party integrations](https://python.langchain.com/docs/integrations/providers/).
+
+- **Open-source libraries**: Build your applications using LangChain's open-source
+[components](https://python.langchain.com/docs/concepts/) and
+[third-party integrations](https://python.langchain.com/docs/integrations/providers/).
   Use [LangGraph](https://langchain-ai.github.io/langgraph/) to build stateful agents with first-class streaming and human-in-the-loop support.
 - **Productionization**: Inspect, monitor, and evaluate your apps with [LangSmith](https://docs.smith.langchain.com/) so that you can constantly optimize and deploy with confidence.
-- **Deployment**: Turn your LangGraph applications into production-ready APIs and Assistants with [LangGraph Cloud](https://langchain-ai.github.io/langgraph/cloud/).
+- **Deployment**: Turn your LangGraph applications into production-ready APIs and Assistants with [LangGraph Platform](https://langchain-ai.github.io/langgraph/cloud/).
 
 ### Open-source libraries
 
-- **`langchain-core`**: Base abstractions and LangChain Expression Language.
-- **`langchain-community`**: Third party integrations.
-  - Some integrations have been further split into **partner packages** that only rely on **`langchain-core`**. Examples include **`langchain_openai`** and **`langchain_anthropic`**.
+- **`langchain-core`**: Base abstractions.
+- **Integration packages** (e.g. **`langchain-openai`**, **`langchain-anthropic`**, etc.): Important integrations have been split into lightweight packages that are co-maintained by the LangChain team and the integration developers.
 - **`langchain`**: Chains, agents, and retrieval strategies that make up an application's cognitive architecture.
-- **[`LangGraph`](https://langchain-ai.github.io/langgraph/)**: A library for building robust and stateful multi-actor applications with LLMs by modeling steps as edges and nodes in a graph. Integrates smoothly with LangChain, but can be used without it. To learn more about LangGraph, check out our first LangChain Academy course, *Introduction to LangGraph*, available [here](https://academy.langchain.com/courses/intro-to-langgraph).
+- **`langchain-community`**: Third-party integrations that are community maintained.
+- **[LangGraph](https://langchain-ai.github.io/langgraph)**: Build robust and stateful multi-actor applications with LLMs by modeling steps as edges and nodes in a graph. Integrates smoothly with LangChain, but can be used without it. To learn more about LangGraph, check out our first LangChain Academy course, *Introduction to LangGraph*, available [here](https://academy.langchain.com/courses/intro-to-langgraph).
 
 ### Productionization:
 
@@ -57,7 +60,7 @@ For these applications, LangChain simplifies the entire application lifecycle:
 
 ### Deployment:
 
-- **[LangGraph Cloud](https://langchain-ai.github.io/langgraph/cloud/)**: Turn your LangGraph applications into production-ready APIs and Assistants.
+- **[LangGraph Platform](https://langchain-ai.github.io/langgraph/cloud/)**: Turn your LangGraph applications into production-ready APIs and Assistants.
 
 ![Diagram outlining the hierarchical organization of the LangChain framework, displaying the interconnected parts across multiple layers.](docs/static/svg/langchain_stack_112024.svg#gh-light-mode-only "LangChain Architecture Overview")
 ![Diagram outlining the hierarchical organization of the LangChain framework, displaying the interconnected parts across multiple layers.](docs/static/svg/langchain_stack_112024_dark.svg#gh-dark-mode-only "LangChain Architecture Overview")
@@ -72,7 +75,7 @@ For these applications, LangChain simplifies the entire application lifecycle:
 **🧱 Extracting structured output**
 
 - [Documentation](https://python.langchain.com/docs/tutorials/extraction/)
-- End-to-end Example: [SQL Llama2 Template](https://github.com/langchain-ai/langchain-extract/)
+- End-to-end Example: [LangChain Extract](https://github.com/langchain-ai/langchain-extract/)
 
 **🤖 Chatbots**
 
@@ -85,19 +88,12 @@ And much more! Head to the [Tutorials](https://python.langchain.com/docs/tutoria
 
 The main value props of the LangChain libraries are:
 
-1. **Components**: composable building blocks, tools and integrations for working with language models. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not
-2. **Off-the-shelf chains**: built-in assemblages of components for accomplishing higher-level tasks
-
-Off-the-shelf chains make it easy to get started. Components make it easy to customize existing chains and build new ones.
-
-## LangChain Expression Language (LCEL)
-
-LCEL is a key part of LangChain, allowing you to build and organize chains of processes in a straightforward, declarative manner. It was designed to support taking prototypes directly into production without needing to alter any code. This means you can use LCEL to set up everything from basic "prompt + LLM" setups to intricate, multi-step workflows.
-
-- **[Overview](https://python.langchain.com/docs/concepts/#langchain-expression-language-lcel)**: LCEL and its benefits
-- **[Interface](https://python.langchain.com/docs/concepts/#runnable-interface)**: The standard Runnable interface for LCEL objects
-- **[Primitives](https://python.langchain.com/docs/how_to/#langchain-expression-language-lcel)**: More on the primitives LCEL includes
-- **[Cheatsheet](https://python.langchain.com/docs/how_to/lcel_cheatsheet/)**: Quick overview of the most common usage patterns
+1. **Components**: composable building blocks, tools and integrations for working with language models. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not.
+2. **Easy orchestration with LangGraph**: [LangGraph](https://langchain-ai.github.io/langgraph/),
+built on top of `langchain-core`, has built-in support for [messages](https://python.langchain.com/docs/concepts/messages/), [tools](https://python.langchain.com/docs/concepts/tools/),
+and other LangChain abstractions. This makes it easy to combine components into
+production-ready applications with persistence, streaming, and other key features.
+Check out the LangChain [tutorials page](https://python.langchain.com/docs/tutorials/#orchestration) for examples.
 
 ## Components
 
@@ -105,15 +101,19 @@ Components fall into the following **modules**:
 
 **📃 Model I/O**
 
-This includes [prompt management](https://python.langchain.com/docs/concepts/#prompt-templates), [prompt optimization](https://python.langchain.com/docs/concepts/#example-selectors), a generic interface for [chat models](https://python.langchain.com/docs/concepts/#chat-models) and [LLMs](https://python.langchain.com/docs/concepts/#llms), and common utilities for working with [model outputs](https://python.langchain.com/docs/concepts/#output-parsers).
+This includes [prompt management](https://python.langchain.com/docs/concepts/prompt_templates/)
+and a generic interface for [chat models](https://python.langchain.com/docs/concepts/chat_models/), including a consistent interface for [tool-calling](https://python.langchain.com/docs/concepts/tool_calling/) and [structured output](https://python.langchain.com/docs/concepts/structured_outputs/) across model providers.
 
 **📚 Retrieval**
 
-Retrieval Augmented Generation involves [loading data](https://python.langchain.com/docs/concepts/#document-loaders) from a variety of sources, [preparing it](https://python.langchain.com/docs/concepts/#text-splitters), then [searching over (a.k.a. retrieving from)](https://python.langchain.com/docs/concepts/#retrievers) it for use in the generation step.
+Retrieval Augmented Generation involves [loading data](https://python.langchain.com/docs/concepts/document_loaders/) from a variety of sources, [preparing it](https://python.langchain.com/docs/concepts/text_splitters/), then [searching over (a.k.a. retrieving from)](https://python.langchain.com/docs/concepts/retrievers/) it for use in the generation step.
 
 **🤖 Agents**
 
-Agents allow an LLM autonomy over how a task is accomplished. Agents make decisions about which Actions to take, then take that Action, observe the result, and repeat until the task is complete. LangChain provides a [standard interface for agents](https://python.langchain.com/docs/concepts/#agents), along with [LangGraph](https://github.com/langchain-ai/langgraph) for building custom agents.
+Agents allow an LLM autonomy over how a task is accomplished. Agents make decisions about which Actions to take, then take that Action, observe the result, and repeat until the task is complete. [LangGraph](https://langchain-ai.github.io/langgraph/) makes it easy to use
+LangChain components to build both [custom](https://langchain-ai.github.io/langgraph/tutorials/)
+and [built-in](https://langchain-ai.github.io/langgraph/how-tos/create-react-agent/)
+LLM agents.
 
 ## 📖 Documentation
 

docs/Makefile

@@ -13,28 +13,21 @@ OUTPUT_NEW_DOCS_DIR = $(OUTPUT_NEW_DIR)/docs
 
 PYTHON = .venv/bin/python
 
-PARTNER_DEPS_LIST := $(shell find ../libs/partners -mindepth 1 -maxdepth 1 -type d -exec sh -c ' \
-for dir; do \
-    if find "$$dir" -maxdepth 1 -type f \( -name "pyproject.toml" -o -name "setup.py" \) | grep -q .; then \
-        echo "$$dir"; \
-    fi \
-done' sh {} + | grep -vE "airbyte|ibm|databricks" | tr '\n' ' ')
-
 PORT ?= 3001
 
 clean:
 	rm -rf build
 
 install-vercel-deps:
-	yum -y update
-	yum install gcc bzip2-devel libffi-devel zlib-devel wget tar gzip rsync -y
+	yum -y -q update
+	yum -y -q install gcc bzip2-devel libffi-devel zlib-devel wget tar gzip rsync -y
 
 install-py-deps:
 	python3 -m venv .venv
-	$(PYTHON) -m pip install --upgrade pip
-	$(PYTHON) -m pip install --upgrade uv
-	$(PYTHON) -m uv pip install --pre -r vercel_requirements.txt
-	$(PYTHON) -m uv pip install --pre --editable $(PARTNER_DEPS_LIST)
+	$(PYTHON) -m pip install -q --upgrade pip
+	$(PYTHON) -m pip install -q --upgrade uv
+	$(PYTHON) -m uv pip install -q --pre -r vercel_requirements.txt
+	$(PYTHON) -m uv pip install -q --pre $$($(PYTHON) scripts/partner_deps_list.py)
 
 generate-files:
 	mkdir -p $(INTERMEDIATE_DIR)
@@ -60,6 +53,7 @@ copy-infra:
 	cp package.json $(OUTPUT_NEW_DIR)
 	cp sidebars.js $(OUTPUT_NEW_DIR)
 	cp -r static $(OUTPUT_NEW_DIR)
+	cp -r ../libs/cli/langchain_cli/integration_template $(OUTPUT_NEW_DIR)/src/theme
 	cp yarn.lock $(OUTPUT_NEW_DIR)
 
 render:
@@ -81,6 +75,7 @@ build: install-py-deps generate-files copy-infra render md-sync append-related
 vercel-build: install-vercel-deps build generate-references
 	rm -rf docs
 	mv $(OUTPUT_NEW_DOCS_DIR) docs
+	cp -r ../libs/cli/langchain_cli/integration_template src/theme
 	rm -rf build
 	mkdir static/api_reference
 	git clone --depth=1 https://github.com/langchain-ai/langchain-api-docs-html.git

docs/api_reference/_static/css/custom.css

@@ -80,6 +80,8 @@
 html {
     --pst-font-family-base: 'Inter';
     --pst-font-family-heading: 'Inter Tight', sans-serif;
+
+    --pst-icon-versionmodified-deprecated: var(--pst-icon-exclamation-triangle);
 }
 
 /*******************************************************************************
@@ -92,7 +94,7 @@ html {
  * https://sass-lang.com/documentation/interpolation
  */
 /* Defaults to light mode if data-theme is not set */
-html:not([data-theme]) {
+html:not([data-theme]), html[data-theme=light] {
   --pst-color-primary: #287977;
   --pst-color-primary-bg: #80D6D3;
   --pst-color-secondary: #6F3AED;
@@ -122,58 +124,8 @@ html:not([data-theme]) {
   --pst-color-on-background: #F4F9F8;
   --pst-color-surface: #F4F9F8;
   --pst-color-on-surface: #222832;
-}
-html:not([data-theme]) {
-  --pst-color-link: var(--pst-color-primary);
-  --pst-color-link-hover: var(--pst-color-secondary);
-}
-html:not([data-theme]) .only-dark,
-html:not([data-theme]) .only-dark ~ figcaption {
-  display: none !important;
-}
-
-/* NOTE: @each {...} is like a for-loop
- * https://sass-lang.com/documentation/at-rules/control/each
- */
-html[data-theme=light] {
-  --pst-color-primary: #287977;
-  --pst-color-primary-bg: #80D6D3;
-  --pst-color-secondary: #6F3AED;
-  --pst-color-secondary-bg: #DAD6FE;
-  --pst-color-accent: #c132af;
-  --pst-color-accent-bg: #f8dff5;
-  --pst-color-info: #276be9;
-  --pst-color-info-bg: #dce7fc;
-  --pst-color-warning: #f66a0a;
-  --pst-color-warning-bg: #f8e3d0;
-  --pst-color-success: #00843f;
-  --pst-color-success-bg: #d6ece1;
-  --pst-color-attention: var(--pst-color-warning);
-  --pst-color-attention-bg: var(--pst-color-warning-bg);
-  --pst-color-danger: #d72d47;
-  --pst-color-danger-bg: #f9e1e4;
-  --pst-color-text-base: #222832;
-  --pst-color-text-muted: #48566b;
-  --pst-color-heading-color: #ffffff;
-  --pst-color-shadow: rgba(0, 0, 0, 0.1);
-  --pst-color-border: #d1d5da;
-  --pst-color-border-muted: rgba(23, 23, 26, 0.2);
-  --pst-color-inline-code: #912583;
-  --pst-color-inline-code-links: #246161;
-  --pst-color-target: #f3cf95;
-  --pst-color-background: #ffffff;
-  --pst-color-on-background: #F4F9F8;
-  --pst-color-surface: #F4F9F8;
-  --pst-color-on-surface: #222832;
-  color-scheme: light;
-}
-html[data-theme=light] {
-  --pst-color-link: var(--pst-color-primary);
-  --pst-color-link-hover: var(--pst-color-secondary);
-}
-html[data-theme=light] .only-dark,
-html[data-theme=light] .only-dark ~ figcaption {
-  display: none !important;
+  --pst-color-deprecated: #f47d2e;
+  --pst-color-deprecated-bg: #fff3e8;
 }
 
 html[data-theme=dark] {
@@ -206,6 +158,8 @@ html[data-theme=dark] {
   --pst-color-on-background: #222832;
   --pst-color-surface: #29313d;
   --pst-color-on-surface: #f3f4f5;
+  --pst-color-deprecated: #b46f3e;
+  --pst-color-deprecated-bg: #341906;
   /* Adjust images in dark mode (unless they have class .only-dark or
    * .dark-light, in which case assume they're already optimized for dark
    * mode).
@@ -216,6 +170,30 @@ html[data-theme=dark] {
   */
   color-scheme: dark;
 }
+
+html:not([data-theme]) {
+  --pst-color-link: var(--pst-color-primary);
+  --pst-color-link-hover: var(--pst-color-secondary);
+}
+html:not([data-theme]) .only-dark,
+html:not([data-theme]) .only-dark ~ figcaption {
+  display: none !important;
+}
+
+/* NOTE: @each {...} is like a for-loop
+ * https://sass-lang.com/documentation/at-rules/control/each
+ */
+html[data-theme=light] {
+  color-scheme: light;
+}
+html[data-theme=light] {
+  --pst-color-link: var(--pst-color-primary);
+  --pst-color-link-hover: var(--pst-color-secondary);
+}
+html[data-theme=light] .only-dark,
+html[data-theme=light] .only-dark ~ figcaption {
+  display: none !important;
+}
 html[data-theme=dark] {
   --pst-color-link: var(--pst-color-primary);
   --pst-color-link-hover: var(--pst-color-secondary);
@@ -389,6 +367,13 @@ html[data-theme=dark] .MathJax_SVG * {
 div.deprecated {
   margin-top: 0.5em;
   margin-bottom: 2em;
+
+  background-color: var(--pst-color-deprecated-bg);
+  border-color: var(--pst-color-deprecated);
+}
+
+span.versionmodified.deprecated:before {
+  color: var(--pst-color-deprecated);
 }
 
 .admonition-beta.admonition, div.admonition-beta.admonition {
@@ -408,4 +393,4 @@ dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.
 p {
   font-size: 0.9rem;
   margin-bottom: 0.5rem;
-}
+}

docs/api_reference/conf.py

@@ -87,6 +87,18 @@ class Beta(BaseAdmonition):
 def setup(app):
     app.add_directive("example_links", ExampleLinksDirective)
     app.add_directive("beta", Beta)
+    app.connect("autodoc-skip-member", skip_private_members)
+
+
+def skip_private_members(app, what, name, obj, skip, options):
+    if skip:
+        return True
+    if hasattr(obj, "__doc__") and obj.__doc__ and ":private:" in obj.__doc__:
+        return True
+    if name == "__init__" and obj.__objclass__ is object:
+        # dont document default init
+        return True
+    return None
 
 
 # -- Project information -----------------------------------------------------
@@ -116,6 +128,7 @@ extensions = [
     "_extensions.gallery_directive",
     "sphinx_design",
     "sphinx_copybutton",
+    "sphinxcontrib.googleanalytics",
 ]
 source_suffix = [".rst", ".md"]
 
@@ -255,6 +268,8 @@ html_show_sourcelink = False
 # Set canonical URL from the Read the Docs Domain
 html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "")
 
+googleanalytics_id = "G-9B66JQQH2F"
+
 # Tell Jinja2 templates the build is running on Read the Docs
 if os.environ.get("READTHEDOCS", "") == "True":
     html_context["READTHEDOCS"] = True

docs/api_reference/create_api_rst.py

@@ -72,14 +72,21 @@ def _load_module_members(module_path: str, namespace: str) -> ModuleMembers:
     Returns:
         list: A list of loaded module objects.
     """
+
     classes_: List[ClassInfo] = []
     functions: List[FunctionInfo] = []
     module = importlib.import_module(module_path)
+
+    if ":private:" in (module.__doc__ or ""):
+        return ModuleMembers(classes_=[], functions=[])
+
     for name, type_ in inspect.getmembers(module):
         if not hasattr(type_, "__module__"):
             continue
         if type_.__module__ != module_path:
             continue
+        if ":private:" in (type_.__doc__ or ""):
+            continue
 
         if inspect.isclass(type_):
             # The type of the class is used to select a template

docs/api_reference/requirements.txt

@@ -9,3 +9,4 @@ pyyaml
 sphinx-design
 sphinx-copybutton
 beautifulsoup4
+sphinxcontrib-googleanalytics

docs/docs/additional_resources/tutorials.mdx

@@ -12,6 +12,7 @@
 ### [by Mayo Oshin](https://www.youtube.com/@chatwithdata/search?query=langchain)
 ### [by 1 little Coder](https://www.youtube.com/playlist?list=PLpdmBGJ6ELUK-v0MK-t4wZmVEbxM5xk6L)
 ### [by BobLin (Chinese language)](https://www.youtube.com/playlist?list=PLbd7ntv6PxC3QMFQvtWfk55p-Op_syO1C)
+### [by Total Technology Zonne](https://youtube.com/playlist?list=PLI8raxzYtfGyE02fAxiM1CPhLUuqcTLWg&si=fkAye16rQKBJVHc9)
 
 ## Courses
 

docs/docs/concepts/architecture.mdx

@@ -65,7 +65,7 @@ A package to deploy LangChain chains as REST APIs. Makes it easy to get a produc
 :::important
 LangServe is designed to primarily deploy simple Runnables and work with well-known primitives in langchain-core.
 
-If you need a deployment option for LangGraph, you should instead be looking at LangGraph Cloud (beta) which will be better suited for deploying LangGraph applications.
+If you need a deployment option for LangGraph, you should instead be looking at LangGraph Platform (beta) which will be better suited for deploying LangGraph applications.
 :::
 
 For more information, see the [LangServe documentation](/docs/langserve).

docs/docs/concepts/index.mdx

@@ -48,7 +48,7 @@ The conceptual guide does not cover step-by-step instructions or specific implem
 - **[AIMessage](/docs/concepts/messages#aimessage)**: Represents a complete response from an AI model.
 - **[astream_events](/docs/concepts/chat_models#key-methods)**: Stream granular information from [LCEL](/docs/concepts/lcel) chains.
 - **[BaseTool](/docs/concepts/tools/#tool-interface)**: The base class for all tools in LangChain.
-- **[batch](/docs/concepts/runnables)**: Use to execute a runnable with batch inputs a Runnable.
+- **[batch](/docs/concepts/runnables)**: Use to execute a runnable with batch inputs.
 - **[bind_tools](/docs/concepts/tool_calling/#tool-binding)**: Allows models to interact with tools.
 - **[Caching](/docs/concepts/chat_models#caching)**: Storing results to avoid redundant calls to a chat model.
 - **[Chat models](/docs/concepts/multimodality/#multimodality-in-chat-models)**: Chat models that handle multiple data modalities.
@@ -70,7 +70,7 @@ The conceptual guide does not cover step-by-step instructions or specific implem
 - **[langchain-core](/docs/concepts/architecture#langchain-core)**: Core langchain package. Includes base interfaces and in-memory implementations.
 - **[langchain](/docs/concepts/architecture#langchain)**: A package for higher level components (e.g., some pre-built chains).
 - **[langgraph](/docs/concepts/architecture#langgraph)**: Powerful orchestration layer for LangChain. Use to build complex pipelines and workflows.
-- **[langserve](/docs/concepts/architecture#langserve)**: Use to deploy LangChain Runnables as REST endpoints. Uses FastAPI. Works primarily for LangChain Runnables, does not currently integrate with LangGraph.
+- **[langserve](/docs/concepts/architecture#langserve)**: Used to deploy LangChain Runnables as REST endpoints. Uses FastAPI. Works primarily for LangChain Runnables, does not currently integrate with LangGraph.
 - **[LLMs (legacy)](/docs/concepts/text_llms)**: Older language models that take a string as input and return a string as output.
 - **[Managing chat history](/docs/concepts/chat_history#managing-chat-history)**: Techniques to maintain and manage the chat history.
 - **[OpenAI format](/docs/concepts/messages#openai-format)**: OpenAI's message format for chat models.
@@ -79,7 +79,7 @@ The conceptual guide does not cover step-by-step instructions or specific implem
 - **[RemoveMessage](/docs/concepts/messages/#removemessage)**: An abstraction used to remove a message from chat history, used primarily in LangGraph.
 - **[role](/docs/concepts/messages#role)**: Represents the role (e.g., user, assistant) of a chat message.
 - **[RunnableConfig](/docs/concepts/runnables/#runnableconfig)**: Use to pass run time information to Runnables (e.g., `run_name`, `run_id`, `tags`, `metadata`, `max_concurrency`, `recursion_limit`, `configurable`).
-- **[Standard parameters for chat models](/docs/concepts/chat_models#standard-parameters)**: Parameters such as API key, `temperature`, and `max_tokens`,
+- **[Standard parameters for chat models](/docs/concepts/chat_models#standard-parameters)**: Parameters such as API key, `temperature`, and `max_tokens`.
 - **[Standard tests](/docs/concepts/testing#standard-tests)**: A defined set of unit and integration tests that all integrations must pass.
 - **[stream](/docs/concepts/streaming)**: Use to stream output from a Runnable or a graph.
 - **[Tokenization](/docs/concepts/tokens)**: The process of converting data into tokens and vice versa.

docs/docs/concepts/runnables.mdx

@@ -114,7 +114,7 @@ Please see the [Configurable Runnables](#configurable-runnables) section for mor
 | Method                  | Description                                                      |
 |-------------------------|------------------------------------------------------------------|
 | `get_input_schema`      | Gives the Pydantic Schema of the input schema for the Runnable.  |
-| `get_output_chema`      | Gives the Pydantic Schema of the output schema for the Runnable. |
+| `get_output_schema`      | Gives the Pydantic Schema of the output schema for the Runnable. |
 | `config_schema`         | Gives the Pydantic Schema of the config schema for the Runnable. |
 | `get_input_jsonschema`  | Gives the JSONSchema of the input schema for the Runnable.       |
 | `get_output_jsonschema` | Gives the JSONSchema of the output schema for the Runnable.      |
@@ -323,7 +323,7 @@ multiple Runnables and you need to add custom processing logic in one of the ste
 
 There are two ways to create a custom Runnable from a function:
 
-* `RunnableLambda`: Use this simple transformations where streaming is not required.
+* `RunnableLambda`: Use this for simple transformations where streaming is not required.
 * `RunnableGenerator`: use this for more complex transformations when streaming is needed.
 
 See the [How to run custom functions](/docs/how_to/functions) guide for more information on how to use `RunnableLambda` and `RunnableGenerator`.
@@ -347,6 +347,6 @@ Sometimes you may want to experiment with, or even expose to the end user, multi
 To simplify this process, the Runnable interface provides two methods for creating configurable Runnables at runtime:
 
 * `configurable_fields`: This method allows you to configure specific **attributes** in a Runnable. For example, the `temperature` attribute of a chat model.
-* `configurable_alternatives`: This method enables you to specify **alternative** Runnables that can be run during run time. For example, you could specify a list of different chat models that can be used.
+* `configurable_alternatives`: This method enables you to specify **alternative** Runnables that can be run during runtime. For example, you could specify a list of different chat models that can be used.
 
 See the [How to configure runtime chain internals](/docs/how_to/configure) guide for more information on how to configure runtime chain internals.

docs/docs/contributing/how_to/integrations/index.mdx

@@ -1,17 +1,20 @@
 ---
+pagination_prev: null
 pagination_next: contributing/how_to/integrations/package
 ---
 
 # Contribute Integrations
 
-LangChain integrations are packages that provide access to language models, vector stores, and other components that can be used in LangChain.
+Integrations are a core component of LangChain.
+LangChain provides standard interfaces for several different components (language models, vector stores, etc) that are crucial when building LLM applications.
 
-This guide will walk you through how to contribute new integrations to LangChain, by
-publishing an integration package to PyPi, and adding documentation for it
-to the LangChain Monorepo.
 
-These instructions will evolve over the next few months as we improve our integration
-processes.
+## Why contribute an integration to LangChain?
+
+- **Discoverability:** LangChain is the most used framework for building LLM applications, with over 20 million monthly downloads. LangChain integrations are discoverable by a large community of GenAI builders.
+- **Interoperability:** LangChain components expose a standard interface, allowing developers to easily swap them for each other. If you implement a LangChain integration, any developer using a different component will easily be able to swap yours in.
+- **Best Practices:** Through their standard interface, LangChain components encourage and facilitate best practices (streaming, async, etc)
+
 
 ## Components to Integrate
 
@@ -22,8 +25,7 @@ supported in LangChain
 
 :::
 
-While any component can be integrated into LangChain, at this time we are only accepting
-new integrations in the docs of the following kinds:
+While any component can be integrated into LangChain, there are specific types of integrations we encourage more:
 
 <table>
   <tr>
@@ -36,7 +38,6 @@ new integrations in the docs of the following kinds:
         <li>Chat Models</li>
         <li>Tools/Toolkits</li>
         <li>Retrievers</li>
-        <li>Document Loaders</li>
         <li>Vector Stores</li>
         <li>Embedding Models</li>
       </ul>
@@ -44,6 +45,7 @@ new integrations in the docs of the following kinds:
     <td>
       <ul>
         <li>LLMs (Text-Completion Models)</li>
+        <li>Document Loaders</li>
         <li>Key-Value Stores</li>
         <li>Document Transformers</li>
         <li>Model Caches</li>
@@ -60,18 +62,30 @@ new integrations in the docs of the following kinds:
 
 ## How to contribute an integration
 
-The only step necessary to "be" a LangChain integration is to add documentation
-that will render on this site (https://python.langchain.com/).
+In order to contribute an integration, you should follow these steps:
 
-As a prerequisite to adding your integration to our documentation, you must:
-
-1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently accepting.
-2. [Implement your package](./package.mdx) and publish it to a public github repository.
+1. Confirm that your integration is in the [list of components](#components-to-integrate) we are currently encouraging.
+2. [Implement your package](/docs/contributing/how_to/integrations/package/) and publish it to a public github repository.
 3. [Implement the standard tests](./standard_tests) for your integration and successfully run them.
 4. [Publish your integration](./publish.mdx) by publishing the package to PyPi and add docs in the `docs/docs/integrations` directory of the LangChain monorepo.
+5. [Optional] Open and merge a PR to add documentation for your integration to the official LangChain docs.
+6. [Optional] Engage with the LangChain team for joint co-marketing ([see below](#co-marketing)).
 
-Once you have completed these steps, you can submit a PR to the LangChain monorepo to add your integration to the documentation.
+## Co-Marketing
+
+With over 20 million monthly downloads, LangChain has a large audience of developers building LLM applications.
+Besides just adding integrations, we also like to show them examples of cool tools or APIs they can use.
+
+While traditionally called "co-marketing", we like to think of this more as "co-education".
+For that reason, while we are happy to highlight your integration through our social media channels, we prefer to highlight examples that also serve some educational purpose.
+Our main social media channels are Twitter and LinkedIn.
+
+Here are some heuristics for types of content we are excited to promote:
+
+- **Integration announcement:** If you announce the integration with a link to the LangChain documentation page, we are happy to re-tweet/re-share on Twitter/LinkedIn.
+- **Educational content:** We highlight good educational content on the weekends - if you write a good blog or make a good YouTube video, we are happy to share there! Note that we prefer content that is NOT framed as "here's how to use integration XYZ", but rather "here's how to do ABC", as we find that is more educational and helpful for developers.
+- **End-to-end applications:** End-to-end applications are great resources for developers looking to build. We prefer to highlight applications that are more complex/agentic in nature, and that use [LangGraph](https://github.com/langchain-ai/langgraph) as the orchestration framework. We get particularly excited about anything involving long-term memory, human-in-the-loop interaction patterns, or multi-agent architectures.
+- **Research:** We love highlighting novel research! Whether it is research built on top of LangChain or that integrates with it.
 
 ## Further Reading
-
-To get started, let's learn [how to bootstrap a new integration package](./package.mdx) for LangChain.
+To get started, let's learn [how to implement an integration package](/docs/contributing/how_to/integrations/package/) for LangChain.

docs/docs/contributing/how_to/integrations/package.mdx

@@ -2,23 +2,109 @@
 pagination_next: contributing/how_to/integrations/standard_tests
 pagination_prev: contributing/how_to/integrations/index
 ---
-# How to bootstrap a new integration package
+# How to implement an integration package
 
-This guide walks through the process of publishing a new LangChain integration 
-package to PyPi.
+This guide walks through the process of implementing a LangChain integration 
+package.
 
 Integration packages are just Python packages that can be installed with `pip install <your-package>`,
 which contain classes that are compatible with LangChain's core interfaces.
 
-In this guide, we will be using [Poetry](https://python-poetry.org/) for
-dependency management and packaging, and you're welcome to use any other tools you prefer.
+We will cover:
 
-## **Prerequisites**
+1. (Optional) How to bootstrap a new integration package
+2. How to implement components, such as [chat models](/docs/concepts/chat_models/) and [vector stores](/docs/concepts/vectorstores/), that adhere
+to the LangChain interface;  
+
+## (Optional) bootstrapping a new integration package
+
+In this section, we will outline 2 options for bootstrapping a new integration package, 
+and you're welcome to use other tools if you prefer!
+
+1. **langchain-cli**: This is a command-line tool that can be used to bootstrap a new integration package with a template for LangChain components and Poetry for dependency management.
+2. **Poetry**: This is a Python dependency management tool that can be used to bootstrap a new Python package with dependencies. You can then add LangChain components to this package.
+
+<details>
+    <summary>Option 1: langchain-cli (recommended)</summary>
+
+In this guide, we will be using the `langchain-cli` to create a new integration package
+from a template, which can be edited to implement your LangChain components.
+
+### **Prerequisites**
 
 - [GitHub](https://github.com) account
 - [PyPi](https://pypi.org/) account
 
-## Boostrapping a new Python package with Poetry
+### Boostrapping a new Python package with langchain-cli
+
+First, install `langchain-cli` and `poetry`:
+
+```bash
+pip install langchain-cli poetry
+```
+
+Next, come up with a name for your package. For this guide, we'll use `langchain-parrot-link`.
+You can confirm that the name is available on PyPi by searching for it on the [PyPi website](https://pypi.org/).
+
+Next, create your new Python package with `langchain-cli`, and navigate into the new directory with `cd`:
+
+```bash
+langchain-cli integration new
+
+> The name of the integration to create (e.g. `my-integration`): parrot-link
+> Name of integration in PascalCase [ParrotLink]:
+
+cd parrot-link
+```
+
+Next, let's add any dependencies we need
+
+```bash
+poetry add my-integration-sdk
+```
+
+We can also add some `typing` or `test` dependencies in a separate poetry dependency group.
+
+```
+poetry add --group typing my-typing-dep
+poetry add --group test my-test-dep
+```
+
+And finally, have poetry set up a virtual environment with your dependencies, as well
+as your integration package:
+
+```bash
+poetry install --with lint,typing,test,test_integration
+```
+
+You now have a new Python package with a template for LangChain components! This
+template comes with files for each integration type, and you're welcome to duplicate or
+delete any of these files as needed (including the associated test files).
+
+To create any individual files from the [template], you can run e.g.:
+
+```bash
+langchain-cli integration new \
+    --name parrot-link \
+    --name-class ParrotLink \
+    --src integration_template/chat_models.py \
+    --dst langchain_parrot_link/chat_models_2.py
+```
+
+</details>
+
+<details>
+    <summary>Option 2: Poetry (manual)</summary>
+
+In this guide, we will be using [Poetry](https://python-poetry.org/) for
+dependency management and packaging, and you're welcome to use any other tools you prefer.
+
+### **Prerequisites**
+
+- [GitHub](https://github.com) account
+- [PyPi](https://pypi.org/) account
+
+### Boostrapping a new Python package with Poetry
 
 First, install Poetry:
 
@@ -64,7 +150,7 @@ poetry install --with test
 
 You're now ready to start writing your integration package!
 
-## Writing your integration
+### Writing your integration
 
 Let's say you're building a simple integration package that provides a `ChatParrotLink`
 chat model integration for LangChain. Here's a simple example of what your project
@@ -86,183 +172,12 @@ All of these files should already exist from step 1, except for
 `chat_models.py` and `test_chat_models.py`! We will implement `test_chat_models.py` 
 later, following the [standard tests](../standard_tests) guide.
 
-To implement `chat_models.py`, let's copy the implementation from our
-[Custom Chat Model Guide](../../../../how_to/custom_chat_model).
+For `chat_models.py`, simply paste the contents of the chat model implementation
+[above](#implementing-langchain-components).
 
-<details>
-    <summary>chat_models.py</summary>
-```python title="langchain_parrot_link/chat_models.py"
-from typing import Any, Dict, Iterator, List, Optional
-
-from langchain_core.callbacks import (
-    CallbackManagerForLLMRun,
-)
-from langchain_core.language_models import BaseChatModel
-from langchain_core.messages import (
-    AIMessage,
-    AIMessageChunk,
-    BaseMessage,
-)
-from langchain_core.messages.ai import UsageMetadata
-from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult
-from pydantic import Field
-
-
-class ChatParrotLink(BaseChatModel):
-    """A custom chat model that echoes the first `parrot_buffer_length` characters
-    of the input.
-
-    When contributing an implementation to LangChain, carefully document
-    the model including the initialization parameters, include
-    an example of how to initialize the model and include any relevant
-    links to the underlying models documentation or API.
-
-    Example:
-
-        .. code-block:: python
-
-            model = ChatParrotLink(parrot_buffer_length=2, model="bird-brain-001")
-            result = model.invoke([HumanMessage(content="hello")])
-            result = model.batch([[HumanMessage(content="hello")],
-                                 [HumanMessage(content="world")]])
-    """
-
-    model_name: str = Field(alias="model")
-    """The name of the model"""
-    parrot_buffer_length: int
-    """The number of characters from the last message of the prompt to be echoed."""
-    temperature: Optional[float] = None
-    max_tokens: Optional[int] = None
-    timeout: Optional[int] = None
-    stop: Optional[List[str]] = None
-    max_retries: int = 2
-
-    def _generate(
-        self,
-        messages: List[BaseMessage],
-        stop: Optional[List[str]] = None,
-        run_manager: Optional[CallbackManagerForLLMRun] = None,
-        **kwargs: Any,
-    ) -> ChatResult:
-        """Override the _generate method to implement the chat model logic.
-
-        This can be a call to an API, a call to a local model, or any other
-        implementation that generates a response to the input prompt.
-
-        Args:
-            messages: the prompt composed of a list of messages.
-            stop: a list of strings on which the model should stop generating.
-                  If generation stops due to a stop token, the stop token itself
-                  SHOULD BE INCLUDED as part of the output. This is not enforced
-                  across models right now, but it's a good practice to follow since
-                  it makes it much easier to parse the output of the model
-                  downstream and understand why generation stopped.
-            run_manager: A run manager with callbacks for the LLM.
-        """
-        # Replace this with actual logic to generate a response from a list
-        # of messages.
-        last_message = messages[-1]
-        tokens = last_message.content[: self.parrot_buffer_length]
-        ct_input_tokens = sum(len(message.content) for message in messages)
-        ct_output_tokens = len(tokens)
-        message = AIMessage(
-            content=tokens,
-            additional_kwargs={},  # Used to add additional payload to the message
-            response_metadata={  # Use for response metadata
-                "time_in_seconds": 3,
-            },
-            usage_metadata={
-                "input_tokens": ct_input_tokens,
-                "output_tokens": ct_output_tokens,
-                "total_tokens": ct_input_tokens + ct_output_tokens,
-            },
-        )
-        ##
-
-        generation = ChatGeneration(message=message)
-        return ChatResult(generations=[generation])
-
-    def _stream(
-        self,
-        messages: List[BaseMessage],
-        stop: Optional[List[str]] = None,
-        run_manager: Optional[CallbackManagerForLLMRun] = None,
-        **kwargs: Any,
-    ) -> Iterator[ChatGenerationChunk]:
-        """Stream the output of the model.
-
-        This method should be implemented if the model can generate output
-        in a streaming fashion. If the model does not support streaming,
-        do not implement it. In that case streaming requests will be automatically
-        handled by the _generate method.
-
-        Args:
-            messages: the prompt composed of a list of messages.
-            stop: a list of strings on which the model should stop generating.
-                  If generation stops due to a stop token, the stop token itself
-                  SHOULD BE INCLUDED as part of the output. This is not enforced
-                  across models right now, but it's a good practice to follow since
-                  it makes it much easier to parse the output of the model
-                  downstream and understand why generation stopped.
-            run_manager: A run manager with callbacks for the LLM.
-        """
-        last_message = messages[-1]
-        tokens = str(last_message.content[: self.parrot_buffer_length])
-        ct_input_tokens = sum(len(message.content) for message in messages)
-
-        for token in tokens:
-            usage_metadata = UsageMetadata(
-                {
-                    "input_tokens": ct_input_tokens,
-                    "output_tokens": 1,
-                    "total_tokens": ct_input_tokens + 1,
-                }
-            )
-            ct_input_tokens = 0
-            chunk = ChatGenerationChunk(
-                message=AIMessageChunk(content=token, usage_metadata=usage_metadata)
-            )
-
-            if run_manager:
-                # This is optional in newer versions of LangChain
-                # The on_llm_new_token will be called automatically
-                run_manager.on_llm_new_token(token, chunk=chunk)
-
-            yield chunk
-
-        # Let's add some other information (e.g., response metadata)
-        chunk = ChatGenerationChunk(
-            message=AIMessageChunk(content="", response_metadata={"time_in_sec": 3})
-        )
-        if run_manager:
-            # This is optional in newer versions of LangChain
-            # The on_llm_new_token will be called automatically
-            run_manager.on_llm_new_token(token, chunk=chunk)
-        yield chunk
-
-    @property
-    def _llm_type(self) -> str:
-        """Get the type of language model used by this chat model."""
-        return "echoing-chat-model-advanced"
-
-    @property
-    def _identifying_params(self) -> Dict[str, Any]:
-        """Return a dictionary of identifying parameters.
-
-        This information is used by the LangChain callback system, which
-        is used for tracing purposes make it possible to monitor LLMs.
-        """
-        return {
-            # The model name allows users to specify custom token counting
-            # rules in LLM monitoring applications (e.g., in LangSmith users
-            # can provide per token pricing for their model and monitor
-            # costs for the given LLM.)
-            "model_name": self.model_name,
-        }
-```
 </details>
 
-## Push your package to a public Github repository
+### Push your package to a public Github repository
 
 This is only required if you want to publish your integration in the LangChain documentation.
 
@@ -270,6 +185,290 @@ This is only required if you want to publish your integration in the LangChain d
 2. Push your code to the repository.
 3. Confirm that your repository is viewable by the public (e.g. in a private browsing window, where you're not logged into Github).
 
+## Implementing LangChain components
+
+LangChain components are subclasses of base classes in [langchain-core](/docs/concepts/architecture/#langchain-core).
+Examples include [chat models](/docs/concepts/chat_models/),
+[vector stores](/docs/concepts/vectorstores/), [tools](/docs/concepts/tools/),
+[embedding models](/docs/concepts/embedding_models/) and [retrievers](/docs/concepts/retrievers/).
+
+Your integration package will typically implement a subclass of at least one of these
+components. Expand the tabs below to see details on each.
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+<Tabs>
+
+    <TabItem value="chat_models" label="Chat models">
+        
+        Refer to the [Custom Chat Model Guide](/docs/how_to/custom_chat_model) guide for
+        detail on a starter chat model [implementation](/docs/how_to/custom_chat_model/#implementation).
+
+        You can start from the following template or langchain-cli command:
+
+        ```bash
+        langchain-cli integration new \
+            --name parrot-link \
+            --name-class ParrotLink \
+            --src integration_template/chat_models.py \
+            --dst langchain_parrot_link/chat_models.py
+        ```
+
+        <details>
+            <summary>Example chat model code</summary>
+
+import ChatModelSource from '../../../../src/theme/integration_template/integration_template/chat_models.py';
+
+        <CodeBlock language="python" title="langchain_parrot_link/chat_models.py">
+            {
+                ChatModelSource.replaceAll('__ModuleName__', 'ParrotLink')
+                    .replaceAll('__package_name__', 'langchain-parrot-link')
+                    .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+                    .replaceAll('__module_name__', 'langchain_parrot_link')
+            }
+        </CodeBlock>
+
+        </details>
+
+    </TabItem>
+    <TabItem value="vector_stores" label="Vector stores">
+
+        Your vector store implementation will depend on your chosen database technology.
+        `langchain-core` includes a minimal
+        [in-memory vector store](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.in_memory.InMemoryVectorStore.html)
+        that we can use as a guide. You can access the code [here](https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/vectorstores/in_memory.py).
+
+        All vector stores must inherit from the [VectorStore](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html)
+        base class. This interface consists of methods for writing, deleting and searching
+        for documents in the vector store.
+
+        `VectorStore` supports a variety of synchronous and asynchronous search types (e.g., 
+        nearest-neighbor or maximum marginal relevance), as well as interfaces for adding
+        documents to the store. See the [API Reference](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html)
+        for all supported methods. The required methods are tabulated below:
+
+        | Method/Property         | Description                                          |
+        |------------------------ |------------------------------------------------------|
+        | `add_documents`         | Add documents to the vector store.                   |
+        | `delete`                | Delete selected documents from vector store (by IDs) |
+        | `get_by_ids`            | Get selected documents from vector store (by IDs)    |
+        | `similarity_search`     | Get documents most similar to a query.               |
+        | `embeddings` (property) | Embeddings object for vector store.                  |
+        | `from_texts`            | Instantiate vector store via adding texts.           |
+
+        Note that `InMemoryVectorStore` implements some optional search types, as well as
+        convenience methods for loading and dumping the object to a file, but this is not
+        necessary for all implementations.
+
+        :::tip
+
+        The [in-memory vector store](https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/vectorstores/in_memory.py)
+        is tested against the standard tests in the LangChain Github repository.
+
+        :::
+
+        <details>
+            <summary>Example vector store code</summary>
+
+import VectorstoreSource from '../../../../src/theme/integration_template/integration_template/vectorstores.py';
+
+        <CodeBlock language="python" title="langchain_parrot_link/vectorstores.py">
+            {
+                VectorstoreSource.replaceAll('__ModuleName__', 'ParrotLink')
+                    .replaceAll('__package_name__', 'langchain-parrot-link')
+                    .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+                    .replaceAll('__module_name__', 'langchain_parrot_link')
+            }
+        </CodeBlock>
+
+        </details>
+
+    </TabItem>
+    <TabItem value="embeddings" label="Embeddings">
+
+Embeddings are used to convert `str` objects from `Document.page_content` fields
+into a vector representation (represented as a list of floats).
+
+Refer to the [Custom Embeddings Guide](/docs/how_to/custom_embeddings) guide for
+detail on a starter embeddings [implementation](/docs/how_to/custom_embeddings/#implementation).
+
+You can start from the following template or langchain-cli command:
+
+```bash
+langchain-cli integration new \
+    --name parrot-link \
+    --name-class ParrotLink \
+    --src integration_template/embeddings.py \
+    --dst langchain_parrot_link/embeddings.py
+```
+
+        <details>
+            <summary>Example embeddings code</summary>
+
+import EmbeddingsSource from '/src/theme/integration_template/integration_template/embeddings.py';
+
+        <CodeBlock language="python" title="langchain_parrot_link/embeddings.py">
+            {
+                EmbeddingsSource.replaceAll('__ModuleName__', 'ParrotLink')
+                    .replaceAll('__package_name__', 'langchain-parrot-link')
+                    .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+                    .replaceAll('__module_name__', 'langchain_parrot_link')
+            }
+        </CodeBlock>
+
+        </details>
+
+    </TabItem>
+    <TabItem value="tools" label="Tools">
+
+Tools are used in 2 main ways:
+
+1. To define an "input schema" or "args schema" to pass to a chat model's tool calling
+feature along with a text request, such that the chat model can generate a "tool call",
+or parameters to call the tool with.
+2. To take a "tool call" as generated above, and take some action and return a response
+that can be passed back to the chat model as a ToolMessage.
+
+The `Tools` class must inherit from the [BaseTool](https://python.langchain.com/api_reference/core/tools/langchain_core.tools.base.BaseTool.html#langchain_core.tools.base.BaseTool) base class. This interface has 3 properties and 2 methods that should be implemented in a 
+subclass.
+
+| Method/Property         | Description                                          |
+|------------------------ |------------------------------------------------------|
+| `name`                  | Name of the tool (passed to the LLM too).            |
+| `description`           | Description of the tool (passed to the LLM too).     |
+| `args_schema`           | Define the schema for the tool's input arguments.    |
+| `_run`                  | Run the tool with the given arguments.               |
+| `_arun`                 | Asynchronously run the tool with the given arguments.|
+
+### Properties
+
+`name`, `description`, and `args_schema` are all properties that should be implemented
+in the subclass. `name` and `description` are strings that are used to identify the tool
+and provide a description of what the tool does. Both of these are passed to the LLM,
+and users may override these values depending on the LLM they are using as a form of
+"prompt engineering." Giving these a concise and LLM-usable name and description is
+important for the initial user experience of the tool.
+
+`args_schema` is a Pydantic `BaseModel` that defines the schema for the tool's input
+arguments. This is used to validate the input arguments to the tool, and to provide
+a schema for the LLM to fill out when calling the tool. Similar to the `name` and
+`description` of the overall Tool class, the fields' names (the variable name) and
+description (part of `Field(..., description="description")`) are passed to the LLM, 
+and the values in these fields should be concise and LLM-usable.
+
+### Run Methods
+
+`_run` is the main method that should be implemented in the subclass. This method
+takes in the arguments from `args_schema` and runs the tool, returning a string
+response. This method is usually called in a LangGraph [`ToolNode`](https://langchain-ai.github.io/langgraph/how-tos/tool-calling/), and can also be called in a legacy
+`langchain.agents.AgentExecutor`.
+
+`_arun` is optional because by default, `_run` will be run in an async executor.
+However, if your tool is calling any apis or doing any async work, you should implement
+this method to run the tool asynchronously in addition to `_run`.
+
+### Implementation
+
+You can start from the following template or langchain-cli command:
+
+```bash
+langchain-cli integration new \
+    --name parrot-link \
+    --name-class ParrotLink \
+    --src integration_template/tools.py \
+    --dst langchain_parrot_link/tools.py
+```
+
+        <details>
+            <summary>Example tool code</summary>
+
+import ToolSource from '/src/theme/integration_template/integration_template/tools.py';
+
+        <CodeBlock language="python" title="langchain_parrot_link/tools.py">
+            {
+                ToolSource.replaceAll('__ModuleName__', 'ParrotLink')
+                    .replaceAll('__package_name__', 'langchain-parrot-link')
+                    .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+                    .replaceAll('__module_name__', 'langchain_parrot_link')
+            }
+        </CodeBlock>
+
+        </details>
+
+    </TabItem>
+    <TabItem value="retrievers" label="Retrievers">
+
+Retrievers are used to retrieve documents from APIs, databases, or other sources
+based on a query. The `Retriever` class must inherit from the [BaseRetriever](https://python.langchain.com/api_reference/core/retrievers/langchain_core.retrievers.BaseRetriever.html) base class. This interface has 1 attribute and 2 methods that should be implemented in a subclass.
+
+| Method/Property         | Description                                          |
+|------------------------ |------------------------------------------------------|
+| `k`                     | Default number of documents to retrieve (configurable). |
+| `_get_relevant_documents`| Retrieve documents based on a query.                 |
+| `_aget_relevant_documents`| Asynchronously retrieve documents based on a query.  |
+
+### Attributes
+
+`k` is an attribute that should be implemented in the subclass. This attribute
+can simply be defined at the top of the class with a default value like
+`k: int = 5`. This attribute is the default number of documents to retrieve
+from the retriever, and can be overridden by the user when constructing or calling
+the retriever.
+
+### Methods
+
+`_get_relevant_documents` is the main method that should be implemented in the subclass.
+
+This method takes in a query and returns a list of `Document` objects, which have 2
+main properties:
+
+- `page_content` - the text content of the document
+- `metadata` - a dictionary of metadata about the document
+
+Retrievers are typically directly invoked by a user, e.g. as
+`MyRetriever(k=4).invoke("query")`, which will automatically call `_get_relevant_documents`
+under the hood.
+
+`_aget_relevant_documents` is optional because by default, `_get_relevant_documents` will
+be run in an async executor. However, if your retriever is calling any apis or doing
+any async work, you should implement this method to run the retriever asynchronously
+in addition to `_get_relevant_documents` for performance reasons.
+
+### Implementation
+
+You can start from the following template or langchain-cli command:
+
+```bash
+langchain-cli integration new \
+    --name parrot-link \
+    --name-class ParrotLink \
+    --src integration_template/retrievers.py \
+    --dst langchain_parrot_link/retrievers.py
+```
+
+        <details>
+            <summary>Example retriever code</summary>
+
+import RetrieverSource from '/src/theme/integration_template/integration_template/retrievers.py';
+
+        <CodeBlock language="python" title="langchain_parrot_link/retrievers.py">
+            {
+                RetrieverSource.replaceAll('__ModuleName__', 'ParrotLink')
+                    .replaceAll('__package_name__', 'langchain-parrot-link')
+                    .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+                    .replaceAll('__module_name__', 'langchain_parrot_link')
+            }
+        </CodeBlock>
+
+        </details>
+
+    </TabItem>
+</Tabs>
+
+---
+
 ## Next Steps
 
 Now that you've implemented your package, you can move on to [testing your integration](../standard_tests) for your integration and successfully run them.

docs/docs/contributing/how_to/integrations/standard_tests.ipynb

@@ -1,497 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "metadata": {
-    "vscode": {
-     "languageId": "raw"
-    }
-   },
-   "source": [
-    "---\n",
-    "pagination_next: contributing/how_to/integrations/publish\n",
-    "pagination_prev: contributing/how_to/integrations/package\n",
-    "---"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# How to add standard tests to an integration\n",
-    "\n",
-    "When creating either a custom class for yourself or to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to a custom chat model, and you can **[Skip to the test templates](#standard-test-templates-per-component)** for implementing tests for each integration type.\n",
-    "\n",
-    "## Setup\n",
-    "\n",
-    "If you're coming from the [previous guide](../package), you have already installed these dependencies, and you can skip this section.\n",
-    "\n",
-    "First, let's install 2 dependencies:\n",
-    "\n",
-    "- `langchain-core` will define the interfaces we want to import to define our custom tool.\n",
-    "- `langchain-tests` will provide the standard tests we want to use. Recommended to pin to the latest version: <img src=\"https://img.shields.io/pypi/v/langchain-tests\" style={{position:\"relative\",top:4,left:3}} />\n",
-    "\n",
-    ":::note\n",
-    "\n",
-    "Because added tests in new versions of `langchain-tests` can break your CI/CD pipelines, we recommend pinning the \n",
-    "version of `langchain-tests` to avoid unexpected changes.\n",
-    "\n",
-    ":::\n",
-    "\n",
-    "import Tabs from '@theme/Tabs';\n",
-    "import TabItem from '@theme/TabItem';\n",
-    "\n",
-    "<Tabs>\n",
-    "    <TabItem value=\"poetry\" label=\"Poetry\" default>\n",
-    "If you followed the [previous guide](../package), you should already have these dependencies installed!\n",
-    "\n",
-    "```bash\n",
-    "poetry add langchain-core\n",
-    "poetry add --group test pytest pytest-socket pytest-asyncio langchain-tests==<latest_version>\n",
-    "poetry install --with test\n",
-    "```\n",
-    "    </TabItem>\n",
-    "    <TabItem value=\"pip\" label=\"Pip\">\n",
-    "```bash\n",
-    "pip install -U langchain-core pytest pytest-socket pytest-asyncio langchain-tests\n",
-    "\n",
-    "# install current package in editable mode\n",
-    "pip install --editable .\n",
-    "```\n",
-    "    </TabItem>\n",
-    "</Tabs>"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Let's say we're publishing a package, `langchain_parrot_link`, that exposes the chat model from the [guide on implementing the package](../package). We can add the standard tests to the package by following the steps below."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "And we'll assume you've structured your package the same way as the main LangChain\n",
-    "packages:\n",
-    "\n",
-    "```plaintext\n",
-    "langchain-parrot-link/\n",
-    "├── langchain_parrot_link/\n",
-    "│   ├── __init__.py\n",
-    "│   └── chat_models.py\n",
-    "├── tests/\n",
-    "│   ├── __init__.py\n",
-    "│   └── test_chat_models.py\n",
-    "├── pyproject.toml\n",
-    "└── README.md\n",
-    "```\n",
-    "\n",
-    "## Add and configure standard tests\n",
-    "\n",
-    "There are 2 namespaces in the `langchain-tests` package: \n",
-    "\n",
-    "- [unit tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.unit_tests`): designed to be used to test the component in isolation and without access to external services\n",
-    "- [integration tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.integration_tests`): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with).\n",
-    "\n",
-    "Both types of tests are implemented as [`pytest` class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class).\n",
-    "\n",
-    "By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you\n",
-    "can override the properties that the test suite uses to configure the tests.\n",
-    "\n",
-    "### Standard chat model tests\n",
-    "\n",
-    "Here's how you would configure the standard unit tests for the custom chat model:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/unit_tests/test_chat_models.py\"\n",
-    "from typing import Tuple, Type\n",
-    "\n",
-    "from langchain_parrot_link.chat_models import ChatParrotLink\n",
-    "from langchain_tests.unit_tests import ChatModelUnitTests\n",
-    "\n",
-    "\n",
-    "class TestChatParrotLinkUnit(ChatModelUnitTests):\n",
-    "    @property\n",
-    "    def chat_model_class(self) -> Type[ChatParrotLink]:\n",
-    "        return ChatParrotLink\n",
-    "\n",
-    "    @property\n",
-    "    def chat_model_params(self) -> dict:\n",
-    "        return {\n",
-    "            \"model\": \"bird-brain-001\",\n",
-    "            \"temperature\": 0,\n",
-    "            \"parrot_buffer_length\": 50,\n",
-    "        }"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/integration_tests/test_chat_models.py\"\n",
-    "from typing import Type\n",
-    "\n",
-    "from langchain_parrot_link.chat_models import ChatParrotLink\n",
-    "from langchain_tests.integration_tests import ChatModelIntegrationTests\n",
-    "\n",
-    "\n",
-    "class TestChatParrotLinkIntegration(ChatModelIntegrationTests):\n",
-    "    @property\n",
-    "    def chat_model_class(self) -> Type[ChatParrotLink]:\n",
-    "        return ChatParrotLink\n",
-    "\n",
-    "    @property\n",
-    "    def chat_model_params(self) -> dict:\n",
-    "        return {\n",
-    "            \"model\": \"bird-brain-001\",\n",
-    "            \"temperature\": 0,\n",
-    "            \"parrot_buffer_length\": 50,\n",
-    "        }"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "and you would run these with the following commands from your project root\n",
-    "\n",
-    "<Tabs>\n",
-    "    <TabItem value=\"poetry\" label=\"Poetry\" default>\n",
-    "\n",
-    "```bash\n",
-    "# run unit tests without network access\n",
-    "poetry run pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests\n",
-    "\n",
-    "# run integration tests\n",
-    "poetry run pytest --asyncio-mode=auto tests/integration_tests\n",
-    "```\n",
-    "\n",
-    "    </TabItem>\n",
-    "    <TabItem value=\"pip\" label=\"Pip\">\n",
-    "\n",
-    "```bash\n",
-    "# run unit tests without network access\n",
-    "pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests\n",
-    "\n",
-    "# run integration tests\n",
-    "pytest --asyncio-mode=auto tests/integration_tests\n",
-    "```\n",
-    "\n",
-    "    </TabItem>\n",
-    "</Tabs>"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Test suite information and troubleshooting\n",
-    "\n",
-    "For a full list of the standard test suites that are available, as well as\n",
-    "information on which tests are included and how to troubleshoot common issues,\n",
-    "see the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html).\n",
-    "\n",
-    "An increasing number of troubleshooting guides are being added to this documentation,\n",
-    "and if you're interested in contributing, feel free to add docstrings to tests in \n",
-    "[Github](https://github.com/langchain-ai/langchain/tree/master/libs/standard-tests/langchain_tests)!"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Standard test templates per component:\n",
-    "\n",
-    "Above, we implement the **unit** and **integration** standard tests for a tool. Below are the templates for implementing the standard tests for each component:\n",
-    "\n",
-    "<details>\n",
-    "    <summary>Chat Models</summary>\n",
-    "    <p>Note: The standard tests for chat models are implemented in the example in the main body of this guide too.</p>"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/unit_tests/test_chat_models.py\"\n",
-    "from typing import Type\n",
-    "\n",
-    "from langchain_parrot_link.chat_models import ChatParrotLink\n",
-    "from langchain_tests.unit_tests import ChatModelUnitTests\n",
-    "\n",
-    "\n",
-    "class TestChatParrotLinkUnit(ChatModelUnitTests):\n",
-    "    @property\n",
-    "    def chat_model_class(self) -> Type[ChatParrotLink]:\n",
-    "        return ChatParrotLink\n",
-    "\n",
-    "    @property\n",
-    "    def chat_model_params(self) -> dict:\n",
-    "        return {\n",
-    "            \"model\": \"bird-brain-001\",\n",
-    "            \"temperature\": 0,\n",
-    "            \"parrot_buffer_length\": 50,\n",
-    "        }"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/integration_tests/test_chat_models.py\"\n",
-    "from typing import Type\n",
-    "\n",
-    "from langchain_parrot_link.chat_models import ChatParrotLink\n",
-    "from langchain_tests.integration_tests import ChatModelIntegrationTests\n",
-    "\n",
-    "\n",
-    "class TestChatParrotLinkIntegration(ChatModelIntegrationTests):\n",
-    "    @property\n",
-    "    def chat_model_class(self) -> Type[ChatParrotLink]:\n",
-    "        return ChatParrotLink\n",
-    "\n",
-    "    @property\n",
-    "    def chat_model_params(self) -> dict:\n",
-    "        return {\n",
-    "            \"model\": \"bird-brain-001\",\n",
-    "            \"temperature\": 0,\n",
-    "            \"parrot_buffer_length\": 50,\n",
-    "        }"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "</details>\n",
-    "<details>\n",
-    "    <summary>Embedding Models</summary>"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/unit_tests/test_embeddings.py\"\n",
-    "from typing import Tuple, Type\n",
-    "\n",
-    "from langchain_parrot_link.embeddings import ParrotLinkEmbeddings\n",
-    "from langchain_tests.unit_tests import EmbeddingsUnitTests\n",
-    "\n",
-    "\n",
-    "class TestParrotLinkEmbeddingsUnit(EmbeddingsUnitTests):\n",
-    "    @property\n",
-    "    def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:\n",
-    "        return ParrotLinkEmbeddings\n",
-    "\n",
-    "    @property\n",
-    "    def embedding_model_params(self) -> dict:\n",
-    "        return {\"model\": \"nest-embed-001\", \"temperature\": 0}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/integration_tests/test_embeddings.py\"\n",
-    "from typing import Type\n",
-    "\n",
-    "from langchain_parrot_link.embeddings import ParrotLinkEmbeddings\n",
-    "from langchain_tests.integration_tests import EmbeddingsIntegrationTests\n",
-    "\n",
-    "\n",
-    "class TestParrotLinkEmbeddingsIntegration(EmbeddingsIntegrationTests):\n",
-    "    @property\n",
-    "    def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:\n",
-    "        return ParrotLinkEmbeddings\n",
-    "\n",
-    "    @property\n",
-    "    def embedding_model_params(self) -> dict:\n",
-    "        return {\"model\": \"nest-embed-001\"}"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "</details>\n",
-    "<details>\n",
-    "    <summary>Tools/Toolkits</summary>"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/unit_tests/test_tools.py\"\n",
-    "from typing import Type\n",
-    "\n",
-    "from langchain_parrot_link.tools import ParrotMultiplyTool\n",
-    "from langchain_tests.unit_tests import ToolsUnitTests\n",
-    "\n",
-    "\n",
-    "class TestParrotMultiplyToolUnit(ToolsUnitTests):\n",
-    "    @property\n",
-    "    def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n",
-    "        return ParrotMultiplyTool\n",
-    "\n",
-    "    @property\n",
-    "    def tool_constructor_params(self) -> dict:\n",
-    "        # if your tool constructor instead required initialization arguments like\n",
-    "        # `def __init__(self, some_arg: int):`, you would return those here\n",
-    "        # as a dictionary, e.g.: `return {'some_arg': 42}`\n",
-    "        return {}\n",
-    "\n",
-    "    @property\n",
-    "    def tool_invoke_params_example(self) -> dict:\n",
-    "        \"\"\"\n",
-    "        Returns a dictionary representing the \"args\" of an example tool call.\n",
-    "\n",
-    "        This should NOT be a ToolCall dict - i.e. it should not\n",
-    "        have {\"name\", \"id\", \"args\"} keys.\n",
-    "        \"\"\"\n",
-    "        return {\"a\": 2, \"b\": 3}"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/integration_tests/test_tools.py\"\n",
-    "from typing import Type\n",
-    "\n",
-    "from langchain_parrot_link.tools import ParrotMultiplyTool\n",
-    "from langchain_tests.integration_tests import ToolsIntegrationTests\n",
-    "\n",
-    "\n",
-    "class TestParrotMultiplyToolIntegration(ToolsIntegrationTests):\n",
-    "    @property\n",
-    "    def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n",
-    "        return ParrotMultiplyTool\n",
-    "\n",
-    "    @property\n",
-    "    def tool_constructor_params(self) -> dict:\n",
-    "        # if your tool constructor instead required initialization arguments like\n",
-    "        # `def __init__(self, some_arg: int):`, you would return those here\n",
-    "        # as a dictionary, e.g.: `return {'some_arg': 42}`\n",
-    "        return {}\n",
-    "\n",
-    "    @property\n",
-    "    def tool_invoke_params_example(self) -> dict:\n",
-    "        \"\"\"\n",
-    "        Returns a dictionary representing the \"args\" of an example tool call.\n",
-    "\n",
-    "        This should NOT be a ToolCall dict - i.e. it should not\n",
-    "        have {\"name\", \"id\", \"args\"} keys.\n",
-    "        \"\"\"\n",
-    "        return {\"a\": 2, \"b\": 3}"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "</details>\n",
-    "<details>\n",
-    "    <summary>Vector Stores</summary>"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# title=\"tests/integration_tests/test_vectorstores_sync.py\"\n",
-    "\n",
-    "from typing import AsyncGenerator, Generator\n",
-    "\n",
-    "import pytest\n",
-    "from langchain_core.vectorstores import VectorStore\n",
-    "from langchain_parrot_link.vectorstores import ParrotVectorStore\n",
-    "from langchain_standard_tests.integration_tests.vectorstores import (\n",
-    "    AsyncReadWriteTestSuite,\n",
-    "    ReadWriteTestSuite,\n",
-    ")\n",
-    "\n",
-    "\n",
-    "class TestSync(ReadWriteTestSuite):\n",
-    "    @pytest.fixture()\n",
-    "    def vectorstore(self) -> Generator[VectorStore, None, None]:  # type: ignore\n",
-    "        \"\"\"Get an empty vectorstore for unit tests.\"\"\"\n",
-    "        store = ParrotVectorStore()\n",
-    "        # note: store should be EMPTY at this point\n",
-    "        # if you need to delete data, you may do so here\n",
-    "        try:\n",
-    "            yield store\n",
-    "        finally:\n",
-    "            # cleanup operations, or deleting data\n",
-    "            pass\n",
-    "\n",
-    "\n",
-    "class TestAsync(AsyncReadWriteTestSuite):\n",
-    "    @pytest.fixture()\n",
-    "    async def vectorstore(self) -> AsyncGenerator[VectorStore, None]:  # type: ignore\n",
-    "        \"\"\"Get an empty vectorstore for unit tests.\"\"\"\n",
-    "        store = ParrotVectorStore()\n",
-    "        # note: store should be EMPTY at this point\n",
-    "        # if you need to delete data, you may do so here\n",
-    "        try:\n",
-    "            yield store\n",
-    "        finally:\n",
-    "            # cleanup operations, or deleting data\n",
-    "            pass"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "</details>"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".venv",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.4"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

docs/docs/contributing/how_to/integrations/standard_tests.mdx

@@ -0,0 +1,393 @@
+---
+pagination_next: contributing/how_to/integrations/publish
+pagination_prev: contributing/how_to/integrations/package
+---
+# How to add standard tests to an integration
+
+When creating either a custom class for yourself or to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to each integration type.
+
+## Setup
+
+First, let's install 2 dependencies:
+
+- `langchain-core` will define the interfaces we want to import to define our custom tool.
+- `langchain-tests` will provide the standard tests we want to use, as well as pytest plugins necessary to run them. Recommended to pin to the latest version: <img src="https://img.shields.io/pypi/v/langchain-tests" style={{position:"relative",top:4,left:3}} />
+
+:::note
+
+Because added tests in new versions of `langchain-tests` can break your CI/CD pipelines, we recommend pinning the 
+version of `langchain-tests` to avoid unexpected changes.
+
+:::
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+<Tabs>
+    <TabItem value="poetry" label="Poetry" default>
+If you followed the [previous guide](../package), you should already have these dependencies installed!
+
+```bash
+poetry add langchain-core
+poetry add --group test langchain-tests==<latest_version>
+poetry install --with test
+```
+    </TabItem>
+    <TabItem value="pip" label="Pip">
+```bash
+pip install -U langchain-core langchain-tests
+
+# install current package in editable mode
+pip install --editable .
+```
+    </TabItem>
+</Tabs>
+
+## Add and configure standard tests
+
+There are 2 namespaces in the `langchain-tests` package: 
+
+- [unit tests](../../../concepts/testing.mdx#unit-tests) (`langchain_tests.unit_tests`): designed to be used to test the component in isolation and without access to external services
+- [integration tests](../../../concepts/testing.mdx#integration-tests) (`langchain_tests.integration_tests`): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with).
+
+Both types of tests are implemented as [`pytest` class-based test suites](https://docs.pytest.org/en/7.1.x/getting-started.html#group-multiple-tests-in-a-class).
+
+By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you
+can override the properties that the test suite uses to configure the tests.
+
+In order to run the tests in the same way as this guide, we recommend subclassing these
+classes in test files under two test subdirectories:
+
+- `tests/unit_tests` for unit tests
+- `tests/integration_tests` for integration tests
+
+### Implementing standard tests
+
+import CodeBlock from '@theme/CodeBlock';
+
+In the following tabs, we show how to implement the standard tests for
+each component type:
+
+<Tabs>
+
+    <TabItem value="chat_models" label="Chat models">
+
+To configure standard tests for a chat model, we subclass `ChatModelUnitTests` and `ChatModelIntegrationTests`. On each subclass, we override the following `@property` methods to specify the chat model to be tested and the chat model's configuration:
+
+| Property | Description |
+| --- | --- |
+| `chat_model_class` | The class for the chat model to be tested |
+| `chat_model_params` | The parameters to pass to the chat
+model's constructor |
+
+Additionally, chat model standard tests test a range of behaviors, from the most basic requirements (generating a response to a query) to optional capabilities like multi-modal support and tool-calling. For a test run to be successful:
+
+1. If a feature is intended to be supported by the model, it should pass;
+2. If a feature is not intended to be supported by the model, it should be skipped.
+
+Tests for "optional" capabilities are controlled via a set of properties that can be overridden on the test model subclass.
+
+You can see the **entire list of configurable capabilities** in the API references for
+[unit tests](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelUnitTests.html)
+and [integration tests](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.chat_models.ChatModelIntegrationTests.html).
+
+For example, to enable integration tests for image inputs, we can implement
+
+```python
+@property
+def supports_image_inputs(self) -> bool:
+    return True
+```
+
+on the integration test class.
+
+:::note
+
+Details on what tests are run, how each test can be skipped, and troubleshooting tips for each test can be found in the API references. See details:
+
+- [Unit tests API reference](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.chat_models.ChatModelUnitTests.html)
+- [Integration tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.chat_models.ChatModelIntegrationTests.html)
+
+:::
+
+Unit test example:
+
+import ChatUnitSource from '../../../../src/theme/integration_template/tests/unit_tests/test_chat_models.py';
+
+<CodeBlock language="python" title="tests/unit_tests/test_chat_models.py">
+{
+    ChatUnitSource.replaceAll('__ModuleName__', 'ParrotLink')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+Integration test example:
+
+
+import ChatIntegrationSource from '../../../../src/theme/integration_template/tests/integration_tests/test_chat_models.py';
+
+<CodeBlock language="python" title="tests/integration_tests/test_chat_models.py">
+{
+    ChatIntegrationSource.replaceAll('__ModuleName__', 'ParrotLink')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+    </TabItem>
+    <TabItem value="vector_stores" label="Vector stores">
+
+
+Here's how you would configure the standard tests for a typical vector store (using
+`ParrotVectorStore` as a placeholder):
+
+Vector store tests do not have optional capabilities to be configured at this time.
+
+import VectorStoreIntegrationSource from '../../../../src/theme/integration_template/tests/integration_tests/test_vectorstores.py';
+
+<CodeBlock language="python" title="tests/integration_tests/test_vectorstores.py">
+{
+    VectorStoreIntegrationSource.replaceAll('__ModuleName__', 'Parrot')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+Configuring the tests consists of implementing pytest fixtures for setting up an
+empty vector store and tearing down the vector store after the test run ends.
+
+| Fixture | Description |
+| --- | --- |
+| `vectorstore` | A generator that yields an empty vector store for unit tests. The vector store is cleaned up after the test run ends. |
+
+For example, below is the `VectorStoreIntegrationTests` class for the [Chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma/)
+integration:
+
+```python
+from typing import Generator
+
+import pytest
+from langchain_core.vectorstores import VectorStore
+from langchain_tests.integration_tests.vectorstores import VectorStoreIntegrationTests
+
+from langchain_chroma import Chroma
+
+
+class TestChromaStandard(VectorStoreIntegrationTests):
+    @pytest.fixture()
+    def vectorstore(self) -> Generator[VectorStore, None, None]:  # type: ignore
+        """Get an empty vectorstore for unit tests."""
+        store = Chroma(embedding_function=self.get_embeddings())
+        try:
+            yield store
+        finally:
+            store.delete_collection()
+            pass
+
+```
+
+Note that before the initial `yield`, we instantiate the vector store with an
+[embeddings](/docs/concepts/embedding_models/) object. This is a pre-defined
+["fake" embeddings model](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.VectorStoreIntegrationTests.html#langchain_tests.integration_tests.vectorstores.VectorStoreIntegrationTests.get_embeddings)
+that will generate short, arbitrary vectors for documents. You can use a different
+embeddings object if desired.
+
+In the `finally` block, we call whatever integration-specific logic is needed to
+bring the vector store to a clean state. This logic is executed in between each test
+(e.g., even if tests fail).
+
+:::note
+
+Details on what tests are run and troubleshooting tips for each test can be found in the [API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.vectorstores.VectorStoreIntegrationTests.html).
+
+:::
+
+
+    </TabItem>
+    <TabItem value="embeddings" label="Embeddings">
+
+To configure standard tests for an embeddings model, we subclass `EmbeddingsUnitTests` and `EmbeddingsIntegrationTests`. On each subclass, we override the following `@property` methods to specify the embeddings model to be tested and the embeddings model's configuration:
+
+| Property | Description |
+| --- | --- |
+| `embeddings_class` | The class for the embeddings model to be tested |
+| `embedding_model_params` | The parameters to pass to the embeddings model's constructor |
+
+:::note
+
+Details on what tests are run, how each test can be skipped, and troubleshooting tips for each test can be found in the API references. See details:
+
+- [Unit tests API reference](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.embeddings.EmbeddingsUnitTests.html)
+- [Integration tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.embeddings.EmbeddingsIntegrationTests.html)
+
+:::
+
+Unit test example:
+
+import EmbeddingsUnitSource from '../../../../src/theme/integration_template/tests/unit_tests/test_embeddings.py';
+
+<CodeBlock language="python" title="tests/unit_tests/test_embeddings.py">
+{
+    EmbeddingsUnitSource.replaceAll('__ModuleName__', 'ParrotLink')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+Integration test example:
+
+
+```python title="tests/integration_tests/test_embeddings.py"
+from typing import Type
+
+from langchain_parrot_link.embeddings import ParrotLinkEmbeddings
+from langchain_tests.integration_tests import EmbeddingsIntegrationTests
+
+
+class TestParrotLinkEmbeddingsIntegration(EmbeddingsIntegrationTests):
+    @property
+    def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:
+        return ParrotLinkEmbeddings
+
+    @property
+    def embedding_model_params(self) -> dict:
+        return {"model": "nest-embed-001"}
+```
+
+import EmbeddingsIntegrationSource from '../../../../src/theme/integration_template/tests/integration_tests/test_embeddings.py';
+
+<CodeBlock language="python" title="tests/integration_tests/test_embeddings.py">
+{
+    EmbeddingsIntegrationSource.replaceAll('__ModuleName__', 'ParrotLink')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT_LINK')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+    </TabItem>
+    <TabItem value="tools" label="Tools">
+
+To configure standard tests for a tool, we subclass `ToolsUnitTests` and
+`ToolsIntegrationTests`. On each subclass, we override the following `@property` methods
+to specify the tool to be tested and the tool's configuration:
+
+| Property | Description |
+| --- | --- |
+| `tool_constructor` | The constructor for the tool to be tested, or an instantiated tool. |
+| `tool_constructor_params` | The parameters to pass to the tool (optional). |
+| `tool_invoke_params_example` | An example of the parameters to pass to the tool's `invoke` method. |
+
+If you are testing a tool class and pass a class like `MyTool` to `tool_constructor`, you can pass the parameters to the constructor in `tool_constructor_params`. 
+
+If you are testing an instantiated tool, you can pass the instantiated tool to `tool_constructor` and do not
+override `tool_constructor_params`.
+
+:::note
+
+Details on what tests are run, how each test can be skipped, and troubleshooting tips for each test can be found in the API references. See details:
+
+- [Unit tests API reference](https://python.langchain.com/api_reference/standard_tests/unit_tests/langchain_tests.unit_tests.tools.ToolsUnitTests.html)
+- [Integration tests API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.tools.ToolsIntegrationTests.html)
+
+:::
+
+import ToolsUnitSource from '../../../../src/theme/integration_template/tests/unit_tests/test_tools.py';
+
+<CodeBlock language="python" title="tests/unit_tests/test_tools.py">
+{
+    ToolsUnitSource.replaceAll('__ModuleName__', 'Parrot')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+import ToolsIntegrationSource from '../../../../src/theme/integration_template/tests/integration_tests/test_tools.py';
+
+<CodeBlock language="python" title="tests/integration_tests/test_tools.py">
+{
+    ToolsIntegrationSource.replaceAll('__ModuleName__', 'Parrot')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+    </TabItem>
+
+    <TabItem value="retrievers" label="Retrievers">
+
+To configure standard tests for a retriever, we subclass `RetrieversUnitTests` and
+`RetrieversIntegrationTests`. On each subclass, we override the following `@property` methods
+
+| Property | Description |
+| --- | --- |
+| `retriever_constructor` | The class for the retriever to be tested |
+| `retriever_constructor_params` | The parameters to pass to the retriever's constructor |
+| `retriever_query_example` | An example of the query to pass to the retriever's `invoke` method |
+
+:::note
+
+Details on what tests are run and troubleshooting tips for each test can be found in the [API reference](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.retrievers.RetrieversIntegrationTests.html).
+
+:::
+
+import RetrieverIntegrationSource from '../../../../src/theme/integration_template/tests/integration_tests/test_retrievers.py';
+
+<CodeBlock language="python" title="tests/integration_tests/test_retrievers.py">
+{
+    RetrieverIntegrationSource.replaceAll('__ModuleName__', 'Parrot')
+        .replaceAll('__package_name__', 'langchain-parrot-link')
+        .replaceAll('__MODULE_NAME__', 'PARROT')
+        .replaceAll('__module_name__', 'langchain_parrot_link')
+}
+</CodeBlock>
+
+    </TabItem>
+</Tabs>
+
+---
+
+### Running the tests
+
+You can run these with the following commands from your project root
+
+<Tabs>
+    <TabItem value="poetry" label="Poetry" default>
+
+```bash
+# run unit tests without network access
+poetry run pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests
+
+# run integration tests
+poetry run pytest --asyncio-mode=auto tests/integration_tests
+```
+
+    </TabItem>
+    <TabItem value="pip" label="Pip">
+
+```bash
+# run unit tests without network access
+pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests
+
+# run integration tests
+pytest --asyncio-mode=auto tests/integration_tests
+```
+
+    </TabItem>
+</Tabs>
+
+## Test suite information and troubleshooting
+
+For a full list of the standard test suites that are available, as well as
+information on which tests are included and how to troubleshoot common issues,
+see the [Standard Tests API Reference](https://python.langchain.com/api_reference/standard_tests/index.html).
+
+You can see troubleshooting guides under the individual test suites listed in that API Reference. For example,
+[here is the guide for `ChatModelIntegrationTests.test_usage_metadata`](https://python.langchain.com/api_reference/standard_tests/integration_tests/langchain_tests.integration_tests.chat_models.ChatModelIntegrationTests.html#langchain_tests.integration_tests.chat_models.ChatModelIntegrationTests.test_usage_metadata).

docs/docs/contributing/index.mdx

@@ -37,7 +37,11 @@ If you are able to help answer questions, please do so! This will allow the main
 
 Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests.
 
-There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues.
+There is a [taxonomy of labels](https://github.com/langchain-ai/langchain/labels?sort=count-desc)
+to help with sorting and discovery of issues of interest. Please use these to help
+organize issues. Check out the [Help Wanted](https://github.com/langchain-ai/langchain/labels/help%20wanted)
+and [Good First Issue](https://github.com/langchain-ai/langchain/labels/good%20first%20issue)
+tags for recommendations.
 
 If you start working on an issue, please assign it to yourself.
 

docs/docs/how_to/agent_executor.ipynb

@@ -802,7 +802,7 @@
     "That's a wrap! In this quick start we covered how to create a simple agent. Agents are a complex topic, and there's lot to learn! \n",
     "\n",
     ":::important\n",
-    "This section covered building with LangChain Agents. LangChain Agents are fine for getting started, but past a certain point you will likely want flexibility and control that they do not offer. For working with more advanced agents, we'd reccommend checking out [LangGraph](/docs/concepts/architecture/#langgraph)\n",
+    "This section covered building with LangChain Agents. They are fine for getting started, but past a certain point you will likely want flexibility and control which they do not offer. To develop more advanced agents, we recommend checking out [LangGraph](/docs/concepts/architecture/#langgraph)\n",
     ":::\n",
     "\n",
     "If you want to continue using LangChain agents, some good advanced guides are:\n",

docs/docs/how_to/caching_embeddings.ipynb

@@ -23,7 +23,7 @@
     "**Attention**:\n",
     "\n",
     "- Be sure to set the `namespace` parameter to avoid collisions of the same text embedded using different embeddings models.\n",
-    "- `CacheBackedEmbeddings` does not cache query embeddings by default. To enable query caching, one need to specify a `query_embedding_cache`."
+    "- `CacheBackedEmbeddings` does not cache query embeddings by default. To enable query caching, one needs to specify a `query_embedding_cache`."
    ]
   },
   {

docs/docs/how_to/callbacks_runtime.ipynb

@@ -15,7 +15,7 @@
     "\n",
     ":::\n",
     "\n",
-    "In many cases, it is advantageous to pass in handlers instead when running the object. When we pass through [`CallbackHandlers`](https://python.langchain.com/api_reference/core/callbacks/langchain_core.callbacks.base.BaseCallbackHandler.html#langchain-core-callbacks-base-basecallbackhandler) using the `callbacks` keyword arg when executing an run, those callbacks will be issued by all nested objects involved in the execution. For example, when a handler is passed through to an Agent, it will be used for all callbacks related to the agent and all the objects involved in the agent's execution, in this case, the Tools and LLM.\n",
+    "In many cases, it is advantageous to pass in handlers instead when running the object. When we pass through [`CallbackHandlers`](https://python.langchain.com/api_reference/core/callbacks/langchain_core.callbacks.base.BaseCallbackHandler.html#langchain-core-callbacks-base-basecallbackhandler) using the `callbacks` keyword arg when executing a run, those callbacks will be issued by all nested objects involved in the execution. For example, when a handler is passed through to an Agent, it will be used for all callbacks related to the agent and all the objects involved in the agent's execution, in this case, the Tools and LLM.\n",
     "\n",
     "This prevents us from having to manually attach the handlers to each individual nested object. Here's an example:"
    ]

docs/docs/how_to/chat_model_rate_limiting.ipynb

@@ -37,7 +37,7 @@
     "\n",
     "Langchain comes with a built-in in memory rate limiter. This rate limiter is thread safe and can be shared by multiple threads in the same process.\n",
     "\n",
-    "The provided rate limiter can only limit the number of requests per unit time. It will not help if you need to also limited based on the size\n",
+    "The provided rate limiter can only limit the number of requests per unit time. It will not help if you need to also limit based on the size\n",
     "of the requests."
    ]
   },

docs/docs/how_to/custom_embeddings.ipynb

@@ -0,0 +1,222 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "c160026f-aadb-4e9f-8642-b4a9e8479d77",
+   "metadata": {},
+   "source": [
+    "# Custom Embeddings\n",
+    "\n",
+    "LangChain is integrated with many [3rd party embedding models](/docs/integrations/text_embedding/). In this guide we'll show you how to create a custom Embedding class, in case a built-in one does not already exist. Embeddings are critical in natural language processing applications as they convert text into a numerical form that algorithms can understand, thereby enabling a wide range of applications such as similarity search, text classification, and clustering.\n",
+    "\n",
+    "Implementing embeddings using the standard [Embeddings](https://python.langchain.com/api_reference/core/embeddings/langchain_core.embeddings.embeddings.Embeddings.html) interface will allow your embeddings to be utilized in existing `LangChain` abstractions (e.g., as the embeddings powering a [VectorStore](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html) or cached using [CacheBackedEmbeddings](/docs/how_to/caching_embeddings/)).\n",
+    "\n",
+    "## Interface\n",
+    "\n",
+    "The current `Embeddings` abstraction in LangChain is designed to operate on text data. In this implementation, the inputs are either single strings or lists of strings, and the outputs are lists of numerical arrays (vectors), where each vector represents\n",
+    "an embedding of the input text into some n-dimensional space.\n",
+    "\n",
+    "Your custom embedding must implement the following methods:\n",
+    "\n",
+    "| Method/Property                 | Description                                                                | Required/Optional |\n",
+    "|---------------------------------|----------------------------------------------------------------------------|-------------------|\n",
+    "| `embed_documents(texts)`        | Generates embeddings for a list of strings.                                | Required          |\n",
+    "| `embed_query(text)`             | Generates an embedding for a single text query.                            | Required          |\n",
+    "| `aembed_documents(texts)`       | Asynchronously generates embeddings for a list of strings.                 | Optional          |\n",
+    "| `aembed_query(text)`            | Asynchronously generates an embedding for a single text query.             | Optional          |\n",
+    "\n",
+    "These methods ensure that your embedding model can be integrated seamlessly into the LangChain framework, providing both synchronous and asynchronous capabilities for scalability and performance optimization.\n",
+    "\n",
+    "\n",
+    ":::note\n",
+    "`Embeddings` do not currently implement the [Runnable](/docs/concepts/runnables/) interface and are also **not** instances of pydantic `BaseModel`.\n",
+    ":::\n",
+    "\n",
+    "### Embedding queries vs documents\n",
+    "\n",
+    "The `embed_query` and `embed_documents` methods are required. These methods both operate\n",
+    "on string inputs. The accessing of `Document.page_content` attributes is handled\n",
+    "by the vector store using the embedding model for legacy reasons.\n",
+    "\n",
+    "`embed_query` takes in a single string and returns a single embedding as a list of floats.\n",
+    "If your model has different modes for embedding queries vs the underlying documents, you can\n",
+    "implement this method to handle that. \n",
+    "\n",
+    "`embed_documents` takes in a list of strings and returns a list of embeddings as a list of lists of floats.\n",
+    "\n",
+    ":::note\n",
+    "`embed_documents` takes in a list of plain text, not a list of LangChain `Document` objects. The name of this method\n",
+    "may change in future versions of LangChain.\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2162547f-4577-47e8-b12f-e9aa3c243797",
+   "metadata": {},
+   "source": [
+    "## Implementation\n",
+    "\n",
+    "As an example, we'll implement a simple embeddings model that returns a constant vector. This model is for illustrative purposes only."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "6b838062-552c-43f8-94f8-d17e4ae4c221",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from typing import List\n",
+    "\n",
+    "from langchain_core.embeddings import Embeddings\n",
+    "\n",
+    "\n",
+    "class ParrotLinkEmbeddings(Embeddings):\n",
+    "    \"\"\"ParrotLink embedding model integration.\n",
+    "\n",
+    "    # TODO: Populate with relevant params.\n",
+    "    Key init args — completion params:\n",
+    "        model: str\n",
+    "            Name of ParrotLink model to use.\n",
+    "\n",
+    "    See full list of supported init args and their descriptions in the params section.\n",
+    "\n",
+    "    # TODO: Replace with relevant init params.\n",
+    "    Instantiate:\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "            from langchain_parrot_link import ParrotLinkEmbeddings\n",
+    "\n",
+    "            embed = ParrotLinkEmbeddings(\n",
+    "                model=\"...\",\n",
+    "                # api_key=\"...\",\n",
+    "                # other params...\n",
+    "            )\n",
+    "\n",
+    "    Embed single text:\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "            input_text = \"The meaning of life is 42\"\n",
+    "            embed.embed_query(input_text)\n",
+    "\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "            # TODO: Example output.\n",
+    "\n",
+    "    # TODO: Delete if token-level streaming isn't supported.\n",
+    "    Embed multiple text:\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "             input_texts = [\"Document 1...\", \"Document 2...\"]\n",
+    "            embed.embed_documents(input_texts)\n",
+    "\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "            # TODO: Example output.\n",
+    "\n",
+    "    # TODO: Delete if native async isn't supported.\n",
+    "    Async:\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "            await embed.aembed_query(input_text)\n",
+    "\n",
+    "            # multiple:\n",
+    "            # await embed.aembed_documents(input_texts)\n",
+    "\n",
+    "        .. code-block:: python\n",
+    "\n",
+    "            # TODO: Example output.\n",
+    "\n",
+    "    \"\"\"\n",
+    "\n",
+    "    def __init__(self, model: str):\n",
+    "        self.model = model\n",
+    "\n",
+    "    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n",
+    "        \"\"\"Embed search docs.\"\"\"\n",
+    "        return [[0.5, 0.6, 0.7] for _ in texts]\n",
+    "\n",
+    "    def embed_query(self, text: str) -> List[float]:\n",
+    "        \"\"\"Embed query text.\"\"\"\n",
+    "        return self.embed_documents([text])[0]\n",
+    "\n",
+    "    # optional: add custom async implementations here\n",
+    "    # you can also delete these, and the base class will\n",
+    "    # use the default implementation, which calls the sync\n",
+    "    # version in an async executor:\n",
+    "\n",
+    "    # async def aembed_documents(self, texts: List[str]) -> List[List[float]]:\n",
+    "    #     \"\"\"Asynchronous Embed search docs.\"\"\"\n",
+    "    #     ...\n",
+    "\n",
+    "    # async def aembed_query(self, text: str) -> List[float]:\n",
+    "    #     \"\"\"Asynchronous Embed query text.\"\"\"\n",
+    "    #     ..."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "47a19044-5c3f-40da-889a-1a1cfffc137c",
+   "metadata": {},
+   "source": [
+    "### Let's test it 🧪"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "21c218fe-8f91-437f-b523-c2b6e5cf749e",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[[0.5, 0.6, 0.7], [0.5, 0.6, 0.7]]\n",
+      "[0.5, 0.6, 0.7]\n"
+     ]
+    }
+   ],
+   "source": [
+    "embeddings = ParrotLinkEmbeddings(\"test-model\")\n",
+    "print(embeddings.embed_documents([\"Hello\", \"world\"]))\n",
+    "print(embeddings.embed_query(\"Hello\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "de50f690-178e-4561-af98-14967b3c8501",
+   "metadata": {},
+   "source": [
+    "## Contributing\n",
+    "\n",
+    "We welcome contributions of Embedding models to the LangChain code base.\n",
+    "\n",
+    "If you aim to contribute an embedding model for a new provider (e.g., with a new set of dependencies or SDK), we encourage you to publish your implementation in a separate `langchain-*` integration package. This will enable you to appropriately manage dependencies and version your package. Please refer to our [contributing guide](/docs/contributing/how_to/integrations/) for a walkthrough of this process."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/docs/how_to/custom_llm.ipynb

@@ -9,10 +9,16 @@
     "\n",
     "This notebook goes over how to create a custom LLM wrapper, in case you want to use your own LLM or a different wrapper than one that is supported in LangChain.\n",
     "\n",
-    "Wrapping your LLM with the standard `LLM` interface allow you to use your LLM in existing LangChain programs with minimal code modifications!\n",
+    "Wrapping your LLM with the standard `LLM` interface allow you to use your LLM in existing LangChain programs with minimal code modifications.\n",
     "\n",
     "As an bonus, your LLM will automatically become a LangChain `Runnable` and will benefit from some optimizations out of the box, async support, the `astream_events` API, etc.\n",
     "\n",
+    ":::caution\n",
+    "You are currently on a page documenting the use of [text completion models](/docs/concepts/text_llms). Many of the latest and most popular models are [chat completion models](/docs/concepts/chat_models).\n",
+    "\n",
+    "Unless you are specifically using more advanced prompting techniques, you are probably looking for [this page instead](/docs/how_to/custom_chat_model/).\n",
+    ":::\n",
+    "\n",
     "## Implementation\n",
     "\n",
     "There are only two required things that a custom LLM needs to implement:\n",

docs/docs/how_to/custom_tools.ipynb

@@ -162,7 +162,7 @@
     "\n",
     "@tool\n",
     "def multiply_by_max(\n",
-    "    a: Annotated[str, \"scale factor\"],\n",
+    "    a: Annotated[int, \"scale factor\"],\n",
     "    b: Annotated[List[int], \"list of ints over which to take maximum\"],\n",
     ") -> int:\n",
     "    \"\"\"Multiply a by the maximum of b.\"\"\"\n",
@@ -294,7 +294,7 @@
    "metadata": {},
    "source": [
     ":::caution\n",
-    "By default, `@tool(parse_docstring=True)` will raise `ValueError` if the docstring does not parse correctly. See [API Reference](https://python.langchain.com/api_reference/core/tools/langchain_core.tools.tool.html) for detail and examples.\n",
+    "By default, `@tool(parse_docstring=True)` will raise `ValueError` if the docstring does not parse correctly. See [API Reference](https://python.langchain.com/api_reference/core/tools/langchain_core.tools.convert.tool.html) for detail and examples.\n",
     ":::"
    ]
   },

docs/docs/how_to/functions.ipynb

@@ -121,7 +121,7 @@
    "source": [
     "## The convenience `@chain` decorator\n",
     "\n",
-    "You can also turn an arbitrary function into a chain by adding a `@chain` decorator. This is functionaly equivalent to wrapping the function in a `RunnableLambda` constructor as shown above. Here's an example:"
+    "You can also turn an arbitrary function into a chain by adding a `@chain` decorator. This is functionally equivalent to wrapping the function in a `RunnableLambda` constructor as shown above. Here's an example:"
    ]
   },
   {

docs/docs/how_to/graph_mapping.ipynb

@@ -1,459 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "id": "5e61b0f2-15b9-4241-9ab5-ff0f3f732232",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 1\n",
-    "---"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "846ef4f4-ee38-4a42-a7d3-1a23826e4830",
-   "metadata": {},
-   "source": [
-    "# How to map values to a graph database\n",
-    "\n",
-    "In this guide we'll go over strategies to improve graph database query generation by mapping values from user inputs to database.\n",
-    "When using the built-in graph chains, the LLM is aware of the graph schema, but has no information about the values of properties stored in the database.\n",
-    "Therefore, we can introduce a new step in graph database QA system to accurately map values.\n",
-    "\n",
-    "## Setup\n",
-    "\n",
-    "First, get required packages and set environment variables:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "18294435-182d-48da-bcab-5b8945b6d9cf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%pip install --upgrade --quiet  langchain langchain-neo4j langchain-openai neo4j"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d86dd771-4001-4a34-8680-22e9b50e1e88",
-   "metadata": {},
-   "source": [
-    "We default to OpenAI models in this guide, but you can swap them out for the model provider of your choice."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "9346f8e9-78bf-4667-b3d3-72807a73b718",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdin",
-     "output_type": "stream",
-     "text": [
-      " ········\n"
-     ]
-    }
-   ],
-   "source": [
-    "import getpass\n",
-    "import os\n",
-    "\n",
-    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()\n",
-    "\n",
-    "# Uncomment the below to use LangSmith. Not required.\n",
-    "# os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
-    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "271c8a23-e51c-4ead-a76e-cf21107db47e",
-   "metadata": {},
-   "source": [
-    "Next, we need to define Neo4j credentials.\n",
-    "Follow [these installation steps](https://neo4j.com/docs/operations-manual/current/installation/) to set up a Neo4j database."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "a2a3bb65-05c7-4daf-bac2-b25ae7fe2751",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "os.environ[\"NEO4J_URI\"] = \"bolt://localhost:7687\"\n",
-    "os.environ[\"NEO4J_USERNAME\"] = \"neo4j\"\n",
-    "os.environ[\"NEO4J_PASSWORD\"] = \"password\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "50fa4510-29b7-49b6-8496-5e86f694e81f",
-   "metadata": {},
-   "source": [
-    "The below example will create a connection with a Neo4j database and will populate it with example data about movies and their actors."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "4ee9ef7a-eef9-4289-b9fd-8fbc31041688",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[]"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain_neo4j import Neo4jGraph\n",
-    "\n",
-    "graph = Neo4jGraph()\n",
-    "\n",
-    "# Import movie information\n",
-    "\n",
-    "movies_query = \"\"\"\n",
-    "LOAD CSV WITH HEADERS FROM \n",
-    "'https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/movies/movies_small.csv'\n",
-    "AS row\n",
-    "MERGE (m:Movie {id:row.movieId})\n",
-    "SET m.released = date(row.released),\n",
-    "    m.title = row.title,\n",
-    "    m.imdbRating = toFloat(row.imdbRating)\n",
-    "FOREACH (director in split(row.director, '|') | \n",
-    "    MERGE (p:Person {name:trim(director)})\n",
-    "    MERGE (p)-[:DIRECTED]->(m))\n",
-    "FOREACH (actor in split(row.actors, '|') | \n",
-    "    MERGE (p:Person {name:trim(actor)})\n",
-    "    MERGE (p)-[:ACTED_IN]->(m))\n",
-    "FOREACH (genre in split(row.genres, '|') | \n",
-    "    MERGE (g:Genre {name:trim(genre)})\n",
-    "    MERGE (m)-[:IN_GENRE]->(g))\n",
-    "\"\"\"\n",
-    "\n",
-    "graph.query(movies_query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0cb0ea30-ca55-4f35-aad6-beb57453de66",
-   "metadata": {},
-   "source": [
-    "## Detecting entities in the user input\n",
-    "We have to extract the types of entities/values we want to map to a graph database. In this example, we are dealing with a movie graph, so we can map movies and people to the database."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "e1a19424-6046-40c2-81d1-f3b88193a293",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import List, Optional\n",
-    "\n",
-    "from langchain_core.prompts import ChatPromptTemplate\n",
-    "from langchain_openai import ChatOpenAI\n",
-    "from pydantic import BaseModel, Field\n",
-    "\n",
-    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n",
-    "\n",
-    "\n",
-    "class Entities(BaseModel):\n",
-    "    \"\"\"Identifying information about entities.\"\"\"\n",
-    "\n",
-    "    names: List[str] = Field(\n",
-    "        ...,\n",
-    "        description=\"All the person or movies appearing in the text\",\n",
-    "    )\n",
-    "\n",
-    "\n",
-    "prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\n",
-    "            \"system\",\n",
-    "            \"You are extracting person and movies from the text.\",\n",
-    "        ),\n",
-    "        (\n",
-    "            \"human\",\n",
-    "            \"Use the given format to extract information from the following \"\n",
-    "            \"input: {question}\",\n",
-    "        ),\n",
-    "    ]\n",
-    ")\n",
-    "\n",
-    "\n",
-    "entity_chain = prompt | llm.with_structured_output(Entities)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9c14084c-37a7-4a9c-a026-74e12961c781",
-   "metadata": {},
-   "source": [
-    "We can test the entity extraction chain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "bbfe0d8f-982e-46e6-88fb-8a4f0d850b07",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "Entities(names=['Casino'])"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "entities = entity_chain.invoke({\"question\": \"Who played in Casino movie?\"})\n",
-    "entities"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a8afbf13-05d0-4383-8050-f88b8c2f6fab",
-   "metadata": {},
-   "source": [
-    "We will utilize a simple `CONTAINS` clause to match entities to database. In practice, you might want to use a fuzzy search or a fulltext index to allow for minor misspellings."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "6f92929f-74fb-4db2-b7e1-eb1e9d386a67",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Casino maps to Casino Movie in database\\n'"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "match_query = \"\"\"MATCH (p:Person|Movie)\n",
-    "WHERE p.name CONTAINS $value OR p.title CONTAINS $value\n",
-    "RETURN coalesce(p.name, p.title) AS result, labels(p)[0] AS type\n",
-    "LIMIT 1\n",
-    "\"\"\"\n",
-    "\n",
-    "\n",
-    "def map_to_database(entities: Entities) -> Optional[str]:\n",
-    "    result = \"\"\n",
-    "    for entity in entities.names:\n",
-    "        response = graph.query(match_query, {\"value\": entity})\n",
-    "        try:\n",
-    "            result += f\"{entity} maps to {response[0]['result']} {response[0]['type']} in database\\n\"\n",
-    "        except IndexError:\n",
-    "            pass\n",
-    "    return result\n",
-    "\n",
-    "\n",
-    "map_to_database(entities)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f66c6756-6efb-4b1e-9b5d-87ed914a5212",
-   "metadata": {},
-   "source": [
-    "## Custom Cypher generating chain\n",
-    "\n",
-    "We need to define a custom Cypher prompt that takes the entity mapping information along with the schema and the user question to construct a Cypher statement.\n",
-    "We will be using the LangChain expression language to accomplish that."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "8ef3e21d-f1c2-45e2-9511-4920d1cf6e7e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnablePassthrough\n",
-    "\n",
-    "# Generate Cypher statement based on natural language input\n",
-    "cypher_template = \"\"\"Based on the Neo4j graph schema below, write a Cypher query that would answer the user's question:\n",
-    "{schema}\n",
-    "Entities in the question map to the following database values:\n",
-    "{entities_list}\n",
-    "Question: {question}\n",
-    "Cypher query:\"\"\"\n",
-    "\n",
-    "cypher_prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\n",
-    "            \"system\",\n",
-    "            \"Given an input question, convert it to a Cypher query. No pre-amble.\",\n",
-    "        ),\n",
-    "        (\"human\", cypher_template),\n",
-    "    ]\n",
-    ")\n",
-    "\n",
-    "cypher_response = (\n",
-    "    RunnablePassthrough.assign(names=entity_chain)\n",
-    "    | RunnablePassthrough.assign(\n",
-    "        entities_list=lambda x: map_to_database(x[\"names\"]),\n",
-    "        schema=lambda _: graph.get_schema,\n",
-    "    )\n",
-    "    | cypher_prompt\n",
-    "    | llm.bind(stop=[\"\\nCypherResult:\"])\n",
-    "    | StrOutputParser()\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "1f0011e3-9660-4975-af2a-486b1bc3b954",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'MATCH (:Movie {title: \"Casino\"})<-[:ACTED_IN]-(actor)\\nRETURN actor.name'"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "cypher = cypher_response.invoke({\"question\": \"Who played in Casino movie?\"})\n",
-    "cypher"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "38095678-611f-4847-a4de-e51ef7ef727c",
-   "metadata": {},
-   "source": [
-    "## Generating answers based on database results\n",
-    "\n",
-    "Now that we have a chain that generates the Cypher statement, we need to execute the Cypher statement against the database and send the database results back to an LLM to generate the final answer.\n",
-    "Again, we will be using LCEL."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "d1fa97c0-1c9c-41d3-9ee1-5f1905d17434",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_neo4j.chains.graph_qa.cypher_utils import (\n",
-    "    CypherQueryCorrector,\n",
-    "    Schema,\n",
-    ")\n",
-    "\n",
-    "graph.refresh_schema()\n",
-    "# Cypher validation tool for relationship directions\n",
-    "corrector_schema = [\n",
-    "    Schema(el[\"start\"], el[\"type\"], el[\"end\"])\n",
-    "    for el in graph.structured_schema.get(\"relationships\")\n",
-    "]\n",
-    "cypher_validation = CypherQueryCorrector(corrector_schema)\n",
-    "\n",
-    "# Generate natural language response based on database results\n",
-    "response_template = \"\"\"Based on the the question, Cypher query, and Cypher response, write a natural language response:\n",
-    "Question: {question}\n",
-    "Cypher query: {query}\n",
-    "Cypher Response: {response}\"\"\"\n",
-    "\n",
-    "response_prompt = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        (\n",
-    "            \"system\",\n",
-    "            \"Given an input question and Cypher response, convert it to a natural\"\n",
-    "            \" language answer. No pre-amble.\",\n",
-    "        ),\n",
-    "        (\"human\", response_template),\n",
-    "    ]\n",
-    ")\n",
-    "\n",
-    "chain = (\n",
-    "    RunnablePassthrough.assign(query=cypher_response)\n",
-    "    | RunnablePassthrough.assign(\n",
-    "        response=lambda x: graph.query(cypher_validation(x[\"query\"])),\n",
-    "    )\n",
-    "    | response_prompt\n",
-    "    | llm\n",
-    "    | StrOutputParser()\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "918146e5-7918-46d2-a774-53f9547d8fcb",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'Robert De Niro, James Woods, Joe Pesci, and Sharon Stone played in the movie \"Casino\".'"
-      ]
-     },
-     "execution_count": 11,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke({\"question\": \"Who played in Casino movie?\"})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c7ba75cd-8399-4e54-a6f8-8a411f159f56",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.18"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/how_to/graph_prompting.ipynb

@@ -1,548 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "raw",
-   "metadata": {},
-   "source": [
-    "---\n",
-    "sidebar_position: 2\n",
-    "---"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# How to best prompt for Graph-RAG\n",
-    "\n",
-    "In this guide we'll go over prompting strategies to improve graph database query generation. We'll largely focus on methods for getting relevant database-specific information in your prompt.\n",
-    "\n",
-    "## Setup\n",
-    "\n",
-    "First, get required packages and set environment variables:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Note: you may need to restart the kernel to use updated packages.\n"
-     ]
-    }
-   ],
-   "source": [
-    "%pip install --upgrade --quiet  langchain langchain-neo4j langchain-openai neo4j"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We default to OpenAI models in this guide, but you can swap them out for the model provider of your choice."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdin",
-     "output_type": "stream",
-     "text": [
-      " ········\n"
-     ]
-    }
-   ],
-   "source": [
-    "import getpass\n",
-    "import os\n",
-    "\n",
-    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()\n",
-    "\n",
-    "# Uncomment the below to use LangSmith. Not required.\n",
-    "# os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
-    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Next, we need to define Neo4j credentials.\n",
-    "Follow [these installation steps](https://neo4j.com/docs/operations-manual/current/installation/) to set up a Neo4j database."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "os.environ[\"NEO4J_URI\"] = \"bolt://localhost:7687\"\n",
-    "os.environ[\"NEO4J_USERNAME\"] = \"neo4j\"\n",
-    "os.environ[\"NEO4J_PASSWORD\"] = \"password\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The below example will create a connection with a Neo4j database and will populate it with example data about movies and their actors."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[]"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from langchain_neo4j import Neo4jGraph\n",
-    "\n",
-    "graph = Neo4jGraph()\n",
-    "\n",
-    "# Import movie information\n",
-    "\n",
-    "movies_query = \"\"\"\n",
-    "LOAD CSV WITH HEADERS FROM \n",
-    "'https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/movies/movies_small.csv'\n",
-    "AS row\n",
-    "MERGE (m:Movie {id:row.movieId})\n",
-    "SET m.released = date(row.released),\n",
-    "    m.title = row.title,\n",
-    "    m.imdbRating = toFloat(row.imdbRating)\n",
-    "FOREACH (director in split(row.director, '|') | \n",
-    "    MERGE (p:Person {name:trim(director)})\n",
-    "    MERGE (p)-[:DIRECTED]->(m))\n",
-    "FOREACH (actor in split(row.actors, '|') | \n",
-    "    MERGE (p:Person {name:trim(actor)})\n",
-    "    MERGE (p)-[:ACTED_IN]->(m))\n",
-    "FOREACH (genre in split(row.genres, '|') | \n",
-    "    MERGE (g:Genre {name:trim(genre)})\n",
-    "    MERGE (m)-[:IN_GENRE]->(g))\n",
-    "\"\"\"\n",
-    "\n",
-    "graph.query(movies_query)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Filtering graph schema\n",
-    "\n",
-    "At times, you may need to focus on a specific subset of the graph schema while generating Cypher statements.\n",
-    "Let's say we are dealing with the following graph schema:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Node properties are the following:\n",
-      "Movie {imdbRating: FLOAT, id: STRING, released: DATE, title: STRING},Person {name: STRING},Genre {name: STRING}\n",
-      "Relationship properties are the following:\n",
-      "\n",
-      "The relationships are the following:\n",
-      "(:Movie)-[:IN_GENRE]->(:Genre),(:Person)-[:DIRECTED]->(:Movie),(:Person)-[:ACTED_IN]->(:Movie)\n"
-     ]
-    }
-   ],
-   "source": [
-    "graph.refresh_schema()\n",
-    "print(graph.schema)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Let's say we want to exclude the _Genre_ node from the schema representation we pass to an LLM.\n",
-    "We can achieve that using the `exclude` parameter of the GraphCypherQAChain chain."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_neo4j import GraphCypherQAChain\n",
-    "from langchain_openai import ChatOpenAI\n",
-    "\n",
-    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n",
-    "chain = GraphCypherQAChain.from_llm(\n",
-    "    graph=graph,\n",
-    "    llm=llm,\n",
-    "    exclude_types=[\"Genre\"],\n",
-    "    verbose=True,\n",
-    "    allow_dangerous_requests=True,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Node properties are the following:\n",
-      "Movie {imdbRating: FLOAT, id: STRING, released: DATE, title: STRING},Person {name: STRING}\n",
-      "Relationship properties are the following:\n",
-      "\n",
-      "The relationships are the following:\n",
-      "(:Person)-[:DIRECTED]->(:Movie),(:Person)-[:ACTED_IN]->(:Movie)\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(chain.graph_schema)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Few-shot examples\n",
-    "\n",
-    "Including examples of natural language questions being converted to valid Cypher queries against our database in the prompt will often improve model performance, especially for complex queries.\n",
-    "\n",
-    "Let's say we have the following examples:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "examples = [\n",
-    "    {\n",
-    "        \"question\": \"How many artists are there?\",\n",
-    "        \"query\": \"MATCH (a:Person)-[:ACTED_IN]->(:Movie) RETURN count(DISTINCT a)\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"Which actors played in the movie Casino?\",\n",
-    "        \"query\": \"MATCH (m:Movie {{title: 'Casino'}})<-[:ACTED_IN]-(a) RETURN a.name\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"How many movies has Tom Hanks acted in?\",\n",
-    "        \"query\": \"MATCH (a:Person {{name: 'Tom Hanks'}})-[:ACTED_IN]->(m:Movie) RETURN count(m)\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"List all the genres of the movie Schindler's List\",\n",
-    "        \"query\": \"MATCH (m:Movie {{title: 'Schindler\\\\'s List'}})-[:IN_GENRE]->(g:Genre) RETURN g.name\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"Which actors have worked in movies from both the comedy and action genres?\",\n",
-    "        \"query\": \"MATCH (a:Person)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g1:Genre), (a)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g2:Genre) WHERE g1.name = 'Comedy' AND g2.name = 'Action' RETURN DISTINCT a.name\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"Which directors have made movies with at least three different actors named 'John'?\",\n",
-    "        \"query\": \"MATCH (d:Person)-[:DIRECTED]->(m:Movie)<-[:ACTED_IN]-(a:Person) WHERE a.name STARTS WITH 'John' WITH d, COUNT(DISTINCT a) AS JohnsCount WHERE JohnsCount >= 3 RETURN d.name\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"Identify movies where directors also played a role in the film.\",\n",
-    "        \"query\": \"MATCH (p:Person)-[:DIRECTED]->(m:Movie), (p)-[:ACTED_IN]->(m) RETURN m.title, p.name\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"question\": \"Find the actor with the highest number of movies in the database.\",\n",
-    "        \"query\": \"MATCH (a:Actor)-[:ACTED_IN]->(m:Movie) RETURN a.name, COUNT(m) AS movieCount ORDER BY movieCount DESC LIMIT 1\",\n",
-    "    },\n",
-    "]"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can create a few-shot prompt with them like so:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_core.prompts import FewShotPromptTemplate, PromptTemplate\n",
-    "\n",
-    "example_prompt = PromptTemplate.from_template(\n",
-    "    \"User input: {question}\\nCypher query: {query}\"\n",
-    ")\n",
-    "prompt = FewShotPromptTemplate(\n",
-    "    examples=examples[:5],\n",
-    "    example_prompt=example_prompt,\n",
-    "    prefix=\"You are a Neo4j expert. Given an input question, create a syntactically correct Cypher query to run.\\n\\nHere is the schema information\\n{schema}.\\n\\nBelow are a number of examples of questions and their corresponding Cypher queries.\",\n",
-    "    suffix=\"User input: {question}\\nCypher query: \",\n",
-    "    input_variables=[\"question\", \"schema\"],\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "You are a Neo4j expert. Given an input question, create a syntactically correct Cypher query to run.\n",
-      "\n",
-      "Here is the schema information\n",
-      "foo.\n",
-      "\n",
-      "Below are a number of examples of questions and their corresponding Cypher queries.\n",
-      "\n",
-      "User input: How many artists are there?\n",
-      "Cypher query: MATCH (a:Person)-[:ACTED_IN]->(:Movie) RETURN count(DISTINCT a)\n",
-      "\n",
-      "User input: Which actors played in the movie Casino?\n",
-      "Cypher query: MATCH (m:Movie {title: 'Casino'})<-[:ACTED_IN]-(a) RETURN a.name\n",
-      "\n",
-      "User input: How many movies has Tom Hanks acted in?\n",
-      "Cypher query: MATCH (a:Person {name: 'Tom Hanks'})-[:ACTED_IN]->(m:Movie) RETURN count(m)\n",
-      "\n",
-      "User input: List all the genres of the movie Schindler's List\n",
-      "Cypher query: MATCH (m:Movie {title: 'Schindler\\'s List'})-[:IN_GENRE]->(g:Genre) RETURN g.name\n",
-      "\n",
-      "User input: Which actors have worked in movies from both the comedy and action genres?\n",
-      "Cypher query: MATCH (a:Person)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g1:Genre), (a)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g2:Genre) WHERE g1.name = 'Comedy' AND g2.name = 'Action' RETURN DISTINCT a.name\n",
-      "\n",
-      "User input: How many artists are there?\n",
-      "Cypher query: \n"
-     ]
-    }
-   ],
-   "source": [
-    "print(prompt.format(question=\"How many artists are there?\", schema=\"foo\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Dynamic few-shot examples\n",
-    "\n",
-    "If we have enough examples, we may want to only include the most relevant ones in the prompt, either because they don't fit in the model's context window or because the long tail of examples distracts the model. And specifically, given any input we want to include the examples most relevant to that input.\n",
-    "\n",
-    "We can do just this using an ExampleSelector. In this case we'll use a [SemanticSimilarityExampleSelector](https://python.langchain.com/api_reference/core/example_selectors/langchain_core.example_selectors.semantic_similarity.SemanticSimilarityExampleSelector.html), which will store the examples in the vector database of our choosing. At runtime it will perform a similarity search between the input and our examples, and return the most semantically similar ones: "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_core.example_selectors import SemanticSimilarityExampleSelector\n",
-    "from langchain_neo4j import Neo4jVector\n",
-    "from langchain_openai import OpenAIEmbeddings\n",
-    "\n",
-    "example_selector = SemanticSimilarityExampleSelector.from_examples(\n",
-    "    examples,\n",
-    "    OpenAIEmbeddings(),\n",
-    "    Neo4jVector,\n",
-    "    k=5,\n",
-    "    input_keys=[\"question\"],\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[{'query': 'MATCH (a:Person)-[:ACTED_IN]->(:Movie) RETURN count(DISTINCT a)',\n",
-       "  'question': 'How many artists are there?'},\n",
-       " {'query': \"MATCH (a:Person {{name: 'Tom Hanks'}})-[:ACTED_IN]->(m:Movie) RETURN count(m)\",\n",
-       "  'question': 'How many movies has Tom Hanks acted in?'},\n",
-       " {'query': \"MATCH (a:Person)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g1:Genre), (a)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g2:Genre) WHERE g1.name = 'Comedy' AND g2.name = 'Action' RETURN DISTINCT a.name\",\n",
-       "  'question': 'Which actors have worked in movies from both the comedy and action genres?'},\n",
-       " {'query': \"MATCH (d:Person)-[:DIRECTED]->(m:Movie)<-[:ACTED_IN]-(a:Person) WHERE a.name STARTS WITH 'John' WITH d, COUNT(DISTINCT a) AS JohnsCount WHERE JohnsCount >= 3 RETURN d.name\",\n",
-       "  'question': \"Which directors have made movies with at least three different actors named 'John'?\"},\n",
-       " {'query': 'MATCH (a:Actor)-[:ACTED_IN]->(m:Movie) RETURN a.name, COUNT(m) AS movieCount ORDER BY movieCount DESC LIMIT 1',\n",
-       "  'question': 'Find the actor with the highest number of movies in the database.'}]"
-      ]
-     },
-     "execution_count": 12,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "example_selector.select_examples({\"question\": \"how many artists are there?\"})"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "To use it, we can pass the ExampleSelector directly in to our FewShotPromptTemplate:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "prompt = FewShotPromptTemplate(\n",
-    "    example_selector=example_selector,\n",
-    "    example_prompt=example_prompt,\n",
-    "    prefix=\"You are a Neo4j expert. Given an input question, create a syntactically correct Cypher query to run.\\n\\nHere is the schema information\\n{schema}.\\n\\nBelow are a number of examples of questions and their corresponding Cypher queries.\",\n",
-    "    suffix=\"User input: {question}\\nCypher query: \",\n",
-    "    input_variables=[\"question\", \"schema\"],\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "You are a Neo4j expert. Given an input question, create a syntactically correct Cypher query to run.\n",
-      "\n",
-      "Here is the schema information\n",
-      "foo.\n",
-      "\n",
-      "Below are a number of examples of questions and their corresponding Cypher queries.\n",
-      "\n",
-      "User input: How many artists are there?\n",
-      "Cypher query: MATCH (a:Person)-[:ACTED_IN]->(:Movie) RETURN count(DISTINCT a)\n",
-      "\n",
-      "User input: How many movies has Tom Hanks acted in?\n",
-      "Cypher query: MATCH (a:Person {name: 'Tom Hanks'})-[:ACTED_IN]->(m:Movie) RETURN count(m)\n",
-      "\n",
-      "User input: Which actors have worked in movies from both the comedy and action genres?\n",
-      "Cypher query: MATCH (a:Person)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g1:Genre), (a)-[:ACTED_IN]->(:Movie)-[:IN_GENRE]->(g2:Genre) WHERE g1.name = 'Comedy' AND g2.name = 'Action' RETURN DISTINCT a.name\n",
-      "\n",
-      "User input: Which directors have made movies with at least three different actors named 'John'?\n",
-      "Cypher query: MATCH (d:Person)-[:DIRECTED]->(m:Movie)<-[:ACTED_IN]-(a:Person) WHERE a.name STARTS WITH 'John' WITH d, COUNT(DISTINCT a) AS JohnsCount WHERE JohnsCount >= 3 RETURN d.name\n",
-      "\n",
-      "User input: Find the actor with the highest number of movies in the database.\n",
-      "Cypher query: MATCH (a:Actor)-[:ACTED_IN]->(m:Movie) RETURN a.name, COUNT(m) AS movieCount ORDER BY movieCount DESC LIMIT 1\n",
-      "\n",
-      "User input: how many artists are there?\n",
-      "Cypher query: \n"
-     ]
-    }
-   ],
-   "source": [
-    "print(prompt.format(question=\"how many artists are there?\", schema=\"foo\"))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n",
-    "chain = GraphCypherQAChain.from_llm(\n",
-    "    graph=graph,\n",
-    "    llm=llm,\n",
-    "    cypher_prompt=prompt,\n",
-    "    verbose=True,\n",
-    "    allow_dangerous_requests=True,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "\n",
-      "\u001b[1m> Entering new GraphCypherQAChain chain...\u001b[0m\n",
-      "Generated Cypher:\n",
-      "\u001b[32;1m\u001b[1;3mMATCH (a:Person)-[:ACTED_IN]->(:Movie) RETURN count(DISTINCT a)\u001b[0m\n",
-      "Full Context:\n",
-      "\u001b[32;1m\u001b[1;3m[{'count(DISTINCT a)': 967}]\u001b[0m\n",
-      "\n",
-      "\u001b[1m> Finished chain.\u001b[0m\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'query': 'How many actors are in the graph?',\n",
-       " 'result': 'There are 967 actors in the graph.'}"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "chain.invoke(\"How many actors are in the graph?\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.1"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}

docs/docs/how_to/index.mdx

@@ -159,6 +159,7 @@ See [supported integrations](/docs/integrations/text_embedding/) for details on
 
 - [How to: embed text data](/docs/how_to/embed_text)
 - [How to: cache embedding results](/docs/how_to/caching_embeddings)
+- [How to: create a custom embeddings class](/docs/how_to/custom_embeddings)
 
 ### Vector stores
 
@@ -244,6 +245,7 @@ All of LangChain components can easily be extended to support your own versions.
 
 - [How to: create a custom chat model class](/docs/how_to/custom_chat_model)
 - [How to: create a custom LLM class](/docs/how_to/custom_llm)
+- [How to: create a custom embeddings class](/docs/how_to/custom_embeddings)
 - [How to: write a custom retriever class](/docs/how_to/custom_retriever)
 - [How to: write a custom document loader](/docs/how_to/document_loader_custom)
 - [How to: write a custom output parser class](/docs/how_to/output_parser_custom)
@@ -316,9 +318,7 @@ For a high-level tutorial, check out [this guide](/docs/tutorials/sql_qa/).
 You can use an LLM to do question answering over graph databases.
 For a high-level tutorial, check out [this guide](/docs/tutorials/graph/).
 
-- [How to: map values to a database](/docs/how_to/graph_mapping)
 - [How to: add a semantic layer over the database](/docs/how_to/graph_semantic)
-- [How to: improve results with prompting](/docs/how_to/graph_prompting)
 - [How to: construct knowledge graphs](/docs/how_to/graph_constructing)
 
 ### Summarization

docs/docs/how_to/indexing.ipynb

@@ -39,19 +39,20 @@
     "| None        | ✅                    | ✅            | ❌                               | ❌                                                 | -                  |\n",
     "| Incremental | ✅                    | ✅            | ❌                               | ✅                                                 | Continuously       |\n",
     "| Full        | ✅                    | ❌            | ✅                               | ✅                                                 | At end of indexing |\n",
+    "| Scoped_Full | ✅                    | ✅            | ❌                               | ✅                                                 | At end of indexing |\n",
     "\n",
     "\n",
     "`None` does not do any automatic clean up, allowing the user to manually do clean up of old content. \n",
     "\n",
-    "`incremental` and `full` offer the following automated clean up:\n",
+    "`incremental`, `full` and `scoped_full` offer the following automated clean up:\n",
     "\n",
-    "* If the content of the source document or derived documents has **changed**, both `incremental` or `full` modes will clean up (delete) previous versions of the content.\n",
-    "* If the source document has been **deleted** (meaning it is not included in the documents currently being indexed), the `full` cleanup mode will delete it from the vector store correctly, but the `incremental` mode will not.\n",
+    "* If the content of the source document or derived documents has **changed**, all 3 modes will clean up (delete) previous versions of the content.\n",
+    "* If the source document has been **deleted** (meaning it is not included in the documents currently being indexed), the `full` cleanup mode will delete it from the vector store correctly, but the `incremental` and `scoped_full` mode will not.\n",
     "\n",
     "When content is mutated (e.g., the source PDF file was revised) there will be a period of time during indexing when both the new and old versions may be returned to the user. This happens after the new content was written, but before the old version was deleted.\n",
     "\n",
     "* `incremental` indexing minimizes this period of time as it is able to do clean up continuously, as it writes.\n",
-    "* `full` mode does the clean up after all batches have been written.\n",
+    "* `full` and `scoped_full` mode does the clean up after all batches have been written.\n",
     "\n",
     "## Requirements\n",
     "\n",
@@ -64,7 +65,7 @@
     "  \n",
     "## Caution\n",
     "\n",
-    "The record manager relies on a time-based mechanism to determine what content can be cleaned up (when using `full` or `incremental` cleanup modes).\n",
+    "The record manager relies on a time-based mechanism to determine what content can be cleaned up (when using `full` or `incremental` or `scoped_full` cleanup modes).\n",
     "\n",
     "If two tasks run back-to-back, and the first task finishes before the clock time changes, then the second task may not be able to clean up content.\n",
     "\n",

docs/docs/how_to/markdown_header_metadata_splitter.ipynb

@@ -162,6 +162,18 @@
     "md_header_splits"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "fb3f834a",
+   "metadata": {},
+   "source": [
+    ":::note\n",
+    "\n",
+    "The default `MarkdownHeaderTextSplitter` strips white spaces and new lines. To preserve the original formatting of your Markdown documents, check out [ExperimentalMarkdownSyntaxTextSplitter](https://python.langchain.com/api_reference/text_splitters/markdown/langchain_text_splitters.markdown.ExperimentalMarkdownSyntaxTextSplitter.html).\n",
+    "\n",
+    ":::"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "aa67e0cc-d721-4536-9c7a-9fa3a7a69cbe",

docs/docs/how_to/output_parser_custom.ipynb

@@ -12,7 +12,7 @@
     "There are two ways to implement a custom parser:\n",
     "\n",
     "1. Using `RunnableLambda` or `RunnableGenerator` in [LCEL](/docs/concepts/lcel/) -- we strongly recommend this for most use cases\n",
-    "2. By inherting from one of the base classes for out parsing -- this is the hard way of doing things\n",
+    "2. By inheriting from one of the base classes for out parsing -- this is the hard way of doing things\n",
     "\n",
     "The difference between the two approaches are mostly superficial and are mainly in terms of which callbacks are triggered (e.g., `on_chain_start` vs. `on_parser_start`), and how a runnable lambda vs. a parser might be visualized in a tracing platform like LangSmith."
    ]
@@ -200,7 +200,7 @@
    "id": "24067447-8a5a-4d6b-86a3-4b9cc4b4369b",
    "metadata": {},
    "source": [
-    "## Inherting from Parsing Base Classes"
+    "## Inheriting from Parsing Base Classes"
    ]
   },
   {
@@ -208,7 +208,7 @@
    "id": "9713f547-b2e4-48eb-807f-a0f6f6d0e7e0",
    "metadata": {},
    "source": [
-    "Another approach to implement a parser is by inherting from `BaseOutputParser`, `BaseGenerationOutputParser` or another one of the base parsers depending on what you need to do.\n",
+    "Another approach to implement a parser is by inheriting from `BaseOutputParser`, `BaseGenerationOutputParser` or another one of the base parsers depending on what you need to do.\n",
     "\n",
     "In general, we **do not** recommend this approach for most use cases as it results in more code to write without significant benefits.\n",
     "\n",

docs/docs/how_to/passthrough.ipynb

@@ -29,7 +29,7 @@
     ":::\n",
     "\n",
     "\n",
-    "When composing chains with several steps, sometimes you will want to pass data from previous steps unchanged for use as input to a later step. The [`RunnablePassthrough`](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.passthrough.RunnablePassthrough.html) class allows you to do just this, and is typically is used in conjuction with a [RunnableParallel](/docs/how_to/parallel/) to pass data through to a later step in your constructed chains.\n",
+    "When composing chains with several steps, sometimes you will want to pass data from previous steps unchanged for use as input to a later step. The [`RunnablePassthrough`](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.passthrough.RunnablePassthrough.html) class allows you to do just this, and is typically is used in conjunction with a [RunnableParallel](/docs/how_to/parallel/) to pass data through to a later step in your constructed chains.\n",
     "\n",
     "See the example below:"
    ]

docs/docs/how_to/semantic-chunker.ipynb

@@ -125,9 +125,11 @@
     "\n",
     "There are a few ways to determine what that threshold is, which are controlled by the `breakpoint_threshold_type` kwarg.\n",
     "\n",
+    "Note: if the resulting chunk sizes are too small/big, the additional kwargs `breakpoint_threshold_amount` and `min_chunk_size` can be used for adjustments.\n",
+    "\n",
     "### Percentile\n",
     "\n",
-    "The default way to split is based on percentile. In this method, all differences between sentences are calculated, and then any difference greater than the X percentile is split."
+    "The default way to split is based on percentile. In this method, all differences between sentences are calculated, and then any difference greater than the X percentile is split. The default value for X is 95.0 and can be adjusted by the keyword argument `breakpoint_threshold_amount` which expects a number between 0.0 and 100.0."
    ]
   },
   {
@@ -186,7 +188,7 @@
    "source": [
     "### Standard Deviation\n",
     "\n",
-    "In this method, any difference greater than X standard deviations is split."
+    "In this method, any difference greater than X standard deviations is split. The default value for X is 3.0 and can be adjusted by the keyword argument `breakpoint_threshold_amount`."
    ]
   },
   {
@@ -245,7 +247,7 @@
    "source": [
     "### Interquartile\n",
     "\n",
-    "In this method, the interquartile distance is used to split chunks."
+    "In this method, the interquartile distance is used to split chunks. The interquartile range can be scaled by the keyword argument `breakpoint_threshold_amount`, the default value is 1.5."
    ]
   },
   {
@@ -306,8 +308,8 @@
    "source": [
     "### Gradient\n",
     "\n",
-    "In this method, the gradient of distance is used to split chunks along with the percentile method.\n",
-    "This method is useful when chunks are highly correlated with each other or specific to a domain e.g. legal or medical. The idea is to apply anomaly detection on gradient array so that the distribution become wider and easy to identify boundaries in highly semantic data."
+    "In this method, the gradient of distance is used to split chunks along with the percentile method. This method is useful when chunks are highly correlated with each other or specific to a domain e.g. legal or medical. The idea is to apply anomaly detection on gradient array so that the distribution become wider and easy to identify boundaries in highly semantic data.\n",
+    "Similar to the percentile method, the split can be adjusted by the keyword argument `breakpoint_threshold_amount` which expects a number between 0.0 and 100.0, the default value is 95.0."
    ]
   },
   {

docs/docs/how_to/sql_large_db.ipynb

@@ -55,7 +55,7 @@
     "* Run `.read Chinook_Sqlite.sql`\n",
     "* Test `SELECT * FROM Artist LIMIT 10;`\n",
     "\n",
-    "Now, `Chinhook.db` is in our directory and we can interface with it using the SQLAlchemy-driven [SQLDatabase](https://python.langchain.com/api_reference/community/utilities/langchain_community.utilities.sql_database.SQLDatabase.html) class:"
+    "Now, `Chinook.db` is in our directory and we can interface with it using the SQLAlchemy-driven [SQLDatabase](https://python.langchain.com/api_reference/community/utilities/langchain_community.utilities.sql_database.SQLDatabase.html) class:"
    ]
   },
   {

docs/docs/how_to/sql_prompting.ipynb

@@ -51,7 +51,7 @@
     "* Run `.read Chinook_Sqlite.sql`\n",
     "* Test `SELECT * FROM Artist LIMIT 10;`\n",
     "\n",
-    "Now, `Chinhook.db` is in our directory and we can interface with it using the SQLAlchemy-driven `SQLDatabase` class:"
+    "Now, `Chinook.db` is in our directory and we can interface with it using the SQLAlchemy-driven `SQLDatabase` class:"
    ]
   },
   {

docs/docs/how_to/sql_query_checking.ipynb

@@ -54,7 +54,7 @@
     "* Run `.read Chinook_Sqlite.sql`\n",
     "* Test `SELECT * FROM Artist LIMIT 10;`\n",
     "\n",
-    "Now, `Chinhook.db` is in our directory and we can interface with it using the SQLAlchemy-driven `SQLDatabase` class:"
+    "Now, `Chinook.db` is in our directory and we can interface with it using the SQLAlchemy-driven `SQLDatabase` class:"
    ]
   },
   {

docs/docs/integrations/callbacks/uptrain.ipynb

@@ -336,7 +336,7 @@
     "\n",
     "The **MultiQueryRetriever** is used to tackle the problem that the RAG pipeline might not return the best set of documents based on the query. It generates multiple queries that mean the same as the original query and then fetches documents for each.\n",
     "\n",
-    "To evluate this retriever, UpTrain will run the following evaluation:\n",
+    "To evaluate this retriever, UpTrain will run the following evaluation:\n",
     "- **[Multi Query Accuracy](https://docs.uptrain.ai/predefined-evaluations/query-quality/multi-query-accuracy)**: Checks if the multi-queries generated mean the same as the original query."
    ]
   },

docs/docs/integrations/chat/cerebras.ipynb

@@ -139,7 +139,7 @@
     "from langchain_cerebras import ChatCerebras\n",
     "\n",
     "llm = ChatCerebras(\n",
-    "    model=\"llama3.1-70b\",\n",
+    "    model=\"llama-3.3-70b\",\n",
     "    # other params...\n",
     ")"
    ]
@@ -215,7 +215,7 @@
     "from langchain_core.prompts import ChatPromptTemplate\n",
     "\n",
     "llm = ChatCerebras(\n",
-    "    model=\"llama3.1-70b\",\n",
+    "    model=\"llama-3.3-70b\",\n",
     "    # other params...\n",
     ")\n",
     "\n",
@@ -280,7 +280,7 @@
     "from langchain_core.prompts import ChatPromptTemplate\n",
     "\n",
     "llm = ChatCerebras(\n",
-    "    model=\"llama3.1-70b\",\n",
+    "    model=\"llama-3.3-70b\",\n",
     "    # other params...\n",
     ")\n",
     "\n",
@@ -324,7 +324,7 @@
     "from langchain_core.prompts import ChatPromptTemplate\n",
     "\n",
     "llm = ChatCerebras(\n",
-    "    model=\"llama3.1-70b\",\n",
+    "    model=\"llama-3.3-70b\",\n",
     "    # other params...\n",
     ")\n",
     "\n",
@@ -371,7 +371,7 @@
     "from langchain_core.prompts import ChatPromptTemplate\n",
     "\n",
     "llm = ChatCerebras(\n",
-    "    model=\"llama3.1-70b\",\n",
+    "    model=\"llama-3.3-70b\",\n",
     "    # other params...\n",
     ")\n",
     "\n",

docs/docs/integrations/chat/ibm_watsonx.ipynb

@@ -36,7 +36,7 @@
     "### Integration details\n",
     "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/chat/ibm/) | Package downloads | Package latest |\n",
     "| :--- | :--- | :---: | :---: |  :---: | :---: | :---: |\n",
-    "| [ChatWatsonx](https://python.langchain.com/api_reference/ibm/chat_models/langchain_ibm.chat_models.ChatWatsonx.html#langchain_ibm.chat_models.ChatWatsonx) | [langchain-ibm](https://python.langchain.com/api_reference/ibm/index.html) | ❌ | ❌ | ✅ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-ibm?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-ibm?style=flat-square&label=%20) |\n",
+    "| [ChatWatsonx](https://python.langchain.com/api_reference/ibm/chat_models/langchain_ibm.chat_models.ChatWatsonx.html) | [langchain-ibm](https://python.langchain.com/api_reference/ibm/index.html) | ❌ | ❌ | ✅ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-ibm?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-ibm?style=flat-square&label=%20) |\n",
     "\n",
     "### Model features\n",
     "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | JSON mode | Image input | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",

docs/docs/integrations/chat/litellm_router.ipynb

@@ -63,9 +63,9 @@
     "        },\n",
     "    },\n",
     "    {\n",
-    "        \"model_name\": \"gpt-4\",\n",
+    "        \"model_name\": \"gpt-35-turbo\",\n",
     "        \"litellm_params\": {\n",
-    "            \"model\": \"azure/gpt-4-1106-preview\",\n",
+    "            \"model\": \"azure/gpt-35-turbo\",\n",
     "            \"api_key\": \"<your-api-key>\",\n",
     "            \"api_version\": \"2023-05-15\",\n",
     "            \"api_base\": \"https://<your-endpoint>.openai.azure.com/\",\n",
@@ -73,7 +73,7 @@
     "    },\n",
     "]\n",
     "litellm_router = Router(model_list=model_list)\n",
-    "chat = ChatLiteLLMRouter(router=litellm_router)"
+    "chat = ChatLiteLLMRouter(router=litellm_router, model_name=\"gpt-35-turbo\")"
    ]
   },
   {
@@ -177,6 +177,7 @@
    "source": [
     "chat = ChatLiteLLMRouter(\n",
     "    router=litellm_router,\n",
+    "    model_name=\"gpt-35-turbo\",\n",
     "    streaming=True,\n",
     "    verbose=True,\n",
     "    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),\n",
@@ -209,7 +210,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.13"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

docs/docs/integrations/chat/oci_generative_ai.ipynb

@@ -26,14 +26,14 @@
     "## Overview\n",
     "### Integration details\n",
     "\n",
-    "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/chat/oci_generative_ai) | Package downloads | Package latest |\n",
-    "| :--- | :--- | :---: | :---: |  :---: | :---: | :---: |\n",
-    "| [ChatOCIGenAI](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html) | [langchain-community](https://python.langchain.com/api_reference/community/index.html) | ❌ | ❌ | ❌ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-oci-generative-ai?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-oci-generative-ai?style=flat-square&label=%20) |\n",
+    "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/chat/oci_generative_ai) |\n",
+    "| :--- | :--- | :---: | :---: |  :---: |\n",
+    "| [ChatOCIGenAI](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html) | [langchain-community](https://python.langchain.com/api_reference/community/index.html) | ❌ | ❌ | ❌ |\n",
     "\n",
     "### Model features\n",
-    "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
+    "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | [JSON mode](/docs/how_to/structured_output/#advanced-specifying-the-method-for-structuring-outputs) | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
     "| :---: | :---: | :---: | :---: |  :---: | :---: | :---: | :---: | :---: | :---: |\n",
-    "| ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | \n",
+    "| ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | \n",
     "\n",
     "## Setup\n",
     "\n",

docs/docs/integrations/chat/ollama.ipynb

@@ -34,8 +34,8 @@
     "\n",
     "### Model features\n",
     "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
-    "| :---: | :---: | :---: | :---: |  :---: | :---: | :---: | :---: | :---: | :---: |\n",
-    "| ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | \n",
+    "| :---: |:----------------------------------------------------:| :---: | :---: |  :---: | :---: | :---: | :---: | :---: | :---: |\n",
+    "| ✅ |                          ✅                           | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ |\n",
     "\n",
     "## Setup\n",
     "\n",
@@ -96,6 +96,20 @@
     "%pip install -qU langchain-ollama"
    ]
   },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "Make sure you're using the latest Ollama version for structured outputs. Update by running:",
+   "id": "b18bd692076f7cf7"
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "outputs": [],
+   "execution_count": null,
+   "source": "%pip install -U ollama",
+   "id": "b7a05cba95644c2e"
+  },
   {
    "cell_type": "markdown",
    "id": "a38cde65-254d-4219-a441-068766c0d4b5",

docs/docs/integrations/chat/sambastudio.ipynb

@@ -34,7 +34,7 @@
     "\n",
     "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
     "| :---: | :---: | :---: | :---: |  :---: | :---: | :---: | :---: | :---: | :---: |\n",
-    "| ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | \n",
+    "| ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | \n",
     "\n",
     "## Setup\n",
     "\n",
@@ -119,20 +119,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "from langchain_community.chat_models.sambanova import ChatSambaStudio\n",
     "\n",
     "llm = ChatSambaStudio(\n",
-    "    model=\"Meta-Llama-3-70B-Instruct-4096\",  # set if using a CoE endpoint\n",
+    "    model=\"Meta-Llama-3-70B-Instruct-4096\",  # set if using a Bundle endpoint\n",
     "    max_tokens=1024,\n",
     "    temperature=0.7,\n",
     "    top_k=1,\n",
     "    top_p=0.01,\n",
     "    do_sample=True,\n",
-    "    process_prompt=\"True\",  # set if using a CoE endpoint\n",
+    "    process_prompt=\"True\",  # set if using a Bundle endpoint\n",
     ")"
    ]
   },
@@ -349,6 +349,134 @@
     "    print(chunk.content, end=\"\", flush=True)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tool calling"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datetime import datetime\n",
+    "\n",
+    "from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage\n",
+    "from langchain_core.tools import tool\n",
+    "\n",
+    "\n",
+    "@tool\n",
+    "def get_time(kind: str = \"both\") -> str:\n",
+    "    \"\"\"Returns current date, current time or both.\n",
+    "    Args:\n",
+    "        kind: date, time or both\n",
+    "    \"\"\"\n",
+    "    if kind == \"date\":\n",
+    "        date = datetime.now().strftime(\"%m/%d/%Y\")\n",
+    "        return f\"Current date: {date}\"\n",
+    "    elif kind == \"time\":\n",
+    "        time = datetime.now().strftime(\"%H:%M:%S\")\n",
+    "        return f\"Current time: {time}\"\n",
+    "    else:\n",
+    "        date = datetime.now().strftime(\"%m/%d/%Y\")\n",
+    "        time = datetime.now().strftime(\"%H:%M:%S\")\n",
+    "        return f\"Current date: {date}, Current time: {time}\"\n",
+    "\n",
+    "\n",
+    "tools = [get_time]\n",
+    "\n",
+    "\n",
+    "def invoke_tools(tool_calls, messages):\n",
+    "    available_functions = {tool.name: tool for tool in tools}\n",
+    "    for tool_call in tool_calls:\n",
+    "        selected_tool = available_functions[tool_call[\"name\"]]\n",
+    "        tool_output = selected_tool.invoke(tool_call[\"args\"])\n",
+    "        print(f\"Tool output: {tool_output}\")\n",
+    "        messages.append(ToolMessage(tool_output, tool_call_id=tool_call[\"id\"]))\n",
+    "    return messages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm_with_tools = llm.bind_tools(tools=tools)\n",
+    "messages = [\n",
+    "    HumanMessage(\n",
+    "        content=\"I need to schedule a meeting for two weeks from today. Can you tell me the exact date of the meeting?\"\n",
+    "    )\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Intermediate model response: [{'name': 'get_time', 'args': {'kind': 'date'}, 'id': 'call_4092d5dd21cd4eb494', 'type': 'tool_call'}]\n",
+      "Tool output: Current date: 11/07/2024\n",
+      "final response: The meeting will be exactly two weeks from today, which would be 25/07/2024.\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = llm_with_tools.invoke(messages)\n",
+    "while len(response.tool_calls) > 0:\n",
+    "    print(f\"Intermediate model response: {response.tool_calls}\")\n",
+    "    messages.append(response)\n",
+    "    messages = invoke_tools(response.tool_calls, messages)\n",
+    "response = llm_with_tools.invoke(messages)\n",
+    "\n",
+    "print(f\"final response: {response.content}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Structured Outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Joke(setup='Why did the cat join a band?', punchline='Because it wanted to be the purr-cussionist!')"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from pydantic import BaseModel, Field\n",
+    "\n",
+    "\n",
+    "class Joke(BaseModel):\n",
+    "    \"\"\"Joke to tell user.\"\"\"\n",
+    "\n",
+    "    setup: str = Field(description=\"The setup of the joke\")\n",
+    "    punchline: str = Field(description=\"The punchline to the joke\")\n",
+    "\n",
+    "\n",
+    "structured_llm = llm.with_structured_output(Joke)\n",
+    "\n",
+    "structured_llm.invoke(\"Tell me a joke about cats\")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

docs/docs/integrations/chat/snowflake.ipynb

@@ -22,24 +22,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Note: you may need to restart the kernel to use updated packages.\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "%pip install --upgrade --quiet snowflake-snowpark-python"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -73,14 +65,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "from langchain_community.chat_models import ChatSnowflakeCortex\n",
     "from langchain_core.messages import HumanMessage, SystemMessage\n",
     "\n",
-    "# By default, we'll be using the cortex provided model: `snowflake-arctic`, with function: `complete`\n",
+    "# By default, we'll be using the cortex provided model: `mistral-large`, with function: `complete`\n",
     "chat = ChatSnowflakeCortex()"
    ]
   },
@@ -92,16 +84,16 @@
     "\n",
     "```python\n",
     "chat = ChatSnowflakeCortex(\n",
-    "    # change default cortex model and function\n",
-    "    model=\"snowflake-arctic\",\n",
+    "    # Change the default cortex model and function\n",
+    "    model=\"mistral-large\",\n",
     "    cortex_function=\"complete\",\n",
     "\n",
-    "    # change default generation parameters\n",
+    "    # Change the default generation parameters\n",
     "    temperature=0,\n",
     "    max_tokens=10,\n",
     "    top_p=0.95,\n",
     "\n",
-    "    # specify snowflake credentials\n",
+    "    # Specify your Snowflake Credentials\n",
     "    account=\"YOUR_SNOWFLAKE_ACCOUNT\",\n",
     "    username=\"YOUR_SNOWFLAKE_USERNAME\",\n",
     "    password=\"YOUR_SNOWFLAKE_PASSWORD\",\n",
@@ -117,28 +109,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Calling the model\n",
-    "We can now call the model using the `invoke` or `generate` method.\n",
-    "\n",
-    "#### Generation"
+    "### Calling the chat model\n",
+    "We can now call the chat model using the `invoke` or `stream` methods."
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": 9,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "AIMessage(content=\" Large language models are artificial intelligence systems designed to understand, generate, and manipulate human language. These models are typically based on deep learning techniques and are trained on vast amounts of text data to learn patterns and structures in language. They can perform a wide range of language-related tasks, such as language translation, text generation, sentiment analysis, and answering questions. Some well-known large language models include Google's BERT, OpenAI's GPT series, and Facebook's RoBERTa. These models have shown remarkable performance in various natural language processing tasks, and their applications continue to expand as research in AI progresses.\", response_metadata={'completion_tokens': 131, 'prompt_tokens': 29, 'total_tokens': 160}, id='run-5435bd0a-83fd-4295-b237-66cbd1b5c0f3-0')"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
    "source": [
     "messages = [\n",
     "    SystemMessage(content=\"You are a friendly assistant.\"),\n",
@@ -151,14 +128,31 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Streaming\n",
-    "`ChatSnowflakeCortex` doesn't support streaming as of now. Support for streaming will be coming in the later versions!"
+    "### Stream"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Sample input prompt\n",
+    "messages = [\n",
+    "    SystemMessage(content=\"You are a friendly assistant.\"),\n",
+    "    HumanMessage(content=\"What are large language models?\"),\n",
+    "]\n",
+    "\n",
+    "# Invoke the stream method and print each chunk as it arrives\n",
+    "print(\"Stream Method Response:\")\n",
+    "for chunk in chat._stream(messages):\n",
+    "    print(chunk.message.content)"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "langchain",
    "language": "python",
    "name": "python3"
   },
@@ -172,7 +166,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.9"
+   "version": "3.9.20"
   }
  },
  "nbformat": 4,

docs/docs/integrations/document_loaders/confluence.ipynb

@@ -11,7 +11,7 @@
     "A loader for `Confluence` pages.\n",
     "\n",
     "\n",
-    "This currently supports `username/api_key`, `Oauth2 login`. Additionally, on-prem installations also support `token` authentication. \n",
+    "This currently supports `username/api_key`, `Oauth2 login`, `cookies`. Additionally, on-prem installations also support `token` authentication. \n",
     "\n",
     "\n",
     "Specify a list `page_id`-s and/or `space_key` to load in the corresponding pages into Document objects, if both are specified the union of both sets will be returned.\n",

docs/docs/integrations/document_loaders/yt_dlp.ipynb

@@ -0,0 +1,149 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: YoutubeLoaderDL\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# YoutubeLoaderDL\n",
+    "\n",
+    "Loader for Youtube leveraging the `yt-dlp` library.\n",
+    "\n",
+    "This package implements a [document loader](/docs/concepts/document_loaders/) for Youtube. In contrast to the [YoutubeLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.youtube.YoutubeLoader.html) of `langchain-community`, which relies on `pytube`, `YoutubeLoaderDL` is able to fetch YouTube metadata. `langchain-yt-dlp` leverages the robust `yt-dlp` library, providing a more reliable and feature-rich YouTube document loader.\n",
+    "\n",
+    "## Overview\n",
+    "### Integration details\n",
+    "\n",
+    "| Class | Package | Local | Serializable | JS Support |\n",
+    "| :--- | :--- | :---: | :---: | :---: |\n",
+    "| YoutubeLoader | langchain-yt-dlp | ✅ | ✅ | ❌ |\n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "### Installation\n",
+    "\n",
+    "```bash\n",
+    "pip install langchain-yt-dlp\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Initialization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_yt_dlp.youtube_loader import YoutubeLoaderDL\n",
+    "\n",
+    "# Basic transcript loading\n",
+    "loader = YoutubeLoaderDL.from_youtube_url(\n",
+    "    \"https://www.youtube.com/watch?v=dQw4w9WgXcQ\", add_video_info=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "documents = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'source': 'dQw4w9WgXcQ',\n",
+       " 'title': 'Rick Astley - Never Gonna Give You Up (Official Music Video)',\n",
+       " 'description': 'The official video for “Never Gonna Give You Up” by Rick Astley. \\n\\nNever: The Autobiography 📚 OUT NOW! \\nFollow this link to get your copy and listen to Rick’s ‘Never’ playlist ❤️ #RickAstleyNever\\nhttps://linktr.ee/rickastleynever\\n\\n“Never Gonna Give You Up” was a global smash on its release in July 1987, topping the charts in 25 countries including Rick’s native UK and the US Billboard Hot 100.  It also won the Brit Award for Best single in 1988. Stock Aitken and Waterman wrote and produced the track which was the lead-off single and lead track from Rick’s debut LP “Whenever You Need Somebody”.  The album was itself a UK number one and would go on to sell over 15 million copies worldwide.\\n\\nThe legendary video was directed by Simon West – who later went on to make Hollywood blockbusters such as Con Air, Lara Croft – Tomb Raider and The Expendables 2.  The video passed the 1bn YouTube views milestone on 28 July 2021.\\n\\nSubscribe to the official Rick Astley YouTube channel: https://RickAstley.lnk.to/YTSubID\\n\\nFollow Rick Astley:\\nFacebook: https://RickAstley.lnk.to/FBFollowID \\nTwitter: https://RickAstley.lnk.to/TwitterID \\nInstagram: https://RickAstley.lnk.to/InstagramID \\nWebsite: https://RickAstley.lnk.to/storeID \\nTikTok: https://RickAstley.lnk.to/TikTokID\\n\\nListen to Rick Astley:\\nSpotify: https://RickAstley.lnk.to/SpotifyID \\nApple Music: https://RickAstley.lnk.to/AppleMusicID \\nAmazon Music: https://RickAstley.lnk.to/AmazonMusicID \\nDeezer: https://RickAstley.lnk.to/DeezerID \\n\\nLyrics:\\nWe’re no strangers to love\\nYou know the rules and so do I\\nA full commitment’s what I’m thinking of\\nYou wouldn’t get this from any other guy\\n\\nI just wanna tell you how I’m feeling\\nGotta make you understand\\n\\nNever gonna give you up\\nNever gonna let you down\\nNever gonna run around and desert you\\nNever gonna make you cry\\nNever gonna say goodbye\\nNever gonna tell a lie and hurt you\\n\\nWe’ve known each other for so long\\nYour heart’s been aching but you’re too shy to say it\\nInside we both know what’s been going on\\nWe know the game and we’re gonna play it\\n\\nAnd if you ask me how I’m feeling\\nDon’t tell me you’re too blind to see\\n\\nNever gonna give you up\\nNever gonna let you down\\nNever gonna run around and desert you\\nNever gonna make you cry\\nNever gonna say goodbye\\nNever gonna tell a lie and hurt you\\n\\n#RickAstley #NeverGonnaGiveYouUp #WheneverYouNeedSomebody #OfficialMusicVideo',\n",
+       " 'view_count': 1603360806,\n",
+       " 'publish_date': datetime.datetime(2009, 10, 25, 0, 0),\n",
+       " 'length': 212,\n",
+       " 'author': 'Rick Astley',\n",
+       " 'channel_id': 'UCuAXFkgsw1L7xaCfnd5JJOw',\n",
+       " 'webpage_url': 'https://www.youtube.com/watch?v=dQw4w9WgXcQ'}"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "documents[0].metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lazy Load"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "- No lazy loading is implemented"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## API reference:\n",
+    "\n",
+    "- [Github](https://github.com/aqib0770/langchain-yt-dlp)\n",
+    "- [PyPi](https://pypi.org/project/langchain-yt-dlp/)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/docs/integrations/llm_caching.ipynb

@@ -12,6 +12,16 @@
     "First, let's install some dependencies"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bedbf252-4ea5-4eea-a3dc-d18ccc84aca3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -qU langchain-openai langchain-community"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -19,8 +29,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%pip install -qU langchain-openai langchain-community\n",
-    "\n",
     "import os\n",
     "from getpass import getpass\n",
     "\n",
@@ -55,7 +63,7 @@
     "tags": []
    },
    "source": [
-    "## `In Memory` Cache"
+    "## `In Memory` cache"
    ]
   },
   {
@@ -139,7 +147,7 @@
     "tags": []
    },
    "source": [
-    "## `SQLite` Cache"
+    "## `SQLite` cache"
    ]
   },
   {
@@ -234,7 +242,7 @@
    "id": "e71273ab",
    "metadata": {},
    "source": [
-    "## `Upstash Redis` Cache"
+    "## `Upstash Redis` caches"
    ]
   },
   {
@@ -242,7 +250,7 @@
    "id": "f10dabef",
    "metadata": {},
    "source": [
-    "### Standard Cache\n",
+    "### Standard cache\n",
     "Use [Upstash Redis](https://upstash.com) to cache prompts and responses with a serverless HTTP API."
    ]
   },
@@ -340,7 +348,8 @@
    "id": "b29dd776",
    "metadata": {},
    "source": [
-    "### Semantic Cache\n",
+    "### Semantic cache\n",
+    "\n",
     "Use [Upstash Vector](https://upstash.com/docs/vector/overall/whatisvector) to do a semantic similarity search and cache the most similar response in the database. The vectorization is automatically done by the selected embedding model while creating Upstash Vector database. "
    ]
   },
@@ -454,11 +463,10 @@
    "cell_type": "markdown",
    "id": "278ad7ae",
    "metadata": {
-    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
-    "## `Redis` Cache\n",
+    "## `Redis` caches\n",
     "\n",
     "See the main [Redis cache docs](/docs/integrations/caches/redis_llm_caching/) for detail."
    ]
@@ -468,7 +476,7 @@
    "id": "c5c9a4d5",
    "metadata": {},
    "source": [
-    "### Standard Cache\n",
+    "### Standard cache\n",
     "Use [Redis](/docs/integrations/providers/redis) to cache prompts and responses."
    ]
   },
@@ -564,7 +572,7 @@
    "id": "82be23f6",
    "metadata": {},
    "source": [
-    "### Semantic Cache\n",
+    "### Semantic cache\n",
     "Use [Redis](/docs/integrations/providers/redis) to cache prompts and responses and evaluate hits based on semantic similarity."
    ]
   },
@@ -660,7 +668,6 @@
    "cell_type": "markdown",
    "id": "684eab55",
    "metadata": {
-    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -905,7 +912,7 @@
    "id": "9b2b2777",
    "metadata": {},
    "source": [
-    "## `MongoDB Atlas` Cache\n",
+    "## `MongoDB Atlas` caches\n",
     "\n",
     "[MongoDB Atlas](https://www.mongodb.com/docs/atlas/) is a fully-managed cloud database available in AWS, Azure, and GCP. It has native support for \n",
     "Vector Search on the MongoDB document data.\n",
@@ -917,8 +924,9 @@
    "id": "ecdc2a0a",
    "metadata": {},
    "source": [
-    "### `MongoDBCache`\n",
-    "An abstraction to store a simple cache in MongoDB. This does not use Semantic Caching, nor does it require an index to be made on the collection before generation.\n",
+    "### Standard cache\n",
+    "\n",
+    "Standard cache is a simple cache in MongoDB. It does not use Semantic Caching, nor does it require an index to be made on the collection before generation.\n",
     "\n",
     "To import this cache, first install the required dependency:\n",
     "\n",
@@ -950,8 +958,9 @@
     "```\n",
     "\n",
     "\n",
-    "### `MongoDBAtlasSemanticCache`\n",
-    "Semantic caching allows users to retrieve cached prompts based on semantic similarity between the user input and previously cached results. Under the hood it blends MongoDBAtlas as both a cache and a vectorstore.\n",
+    "### Semantic cache\n",
+    "\n",
+    "Semantic caching allows retrieval of cached prompts based on semantic similarity between the user input and previously cached results. Under the hood, it blends MongoDBAtlas as both a cache and a vectorstore.\n",
     "The MongoDBAtlasSemanticCache inherits from `MongoDBAtlasVectorSearch` and needs an Atlas Vector Search Index defined to work. Please look at the [usage example](/docs/integrations/vectorstores/mongodb_atlas) on how to set up the index.\n",
     "\n",
     "To import this cache:\n",
@@ -985,14 +994,13 @@
    "cell_type": "markdown",
    "id": "726fe754",
    "metadata": {
-    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
-    "## `Momento` Cache\n",
+    "## `Momento` cache\n",
     "Use [Momento](/docs/integrations/providers/momento) to cache prompts and responses.\n",
     "\n",
-    "Requires momento to use, uncomment below to install:"
+    "Requires installing the `momento` package:"
    ]
   },
   {
@@ -1096,13 +1104,14 @@
    "cell_type": "markdown",
    "id": "934943dc",
    "metadata": {
-    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
-    "## `SQLAlchemy` Cache\n",
+    "## `SQLAlchemy` cache\n",
     "\n",
-    "You can use `SQLAlchemyCache` to cache with any SQL database supported by `SQLAlchemy`."
+    "You can use `SQLAlchemyCache` to cache with any SQL database supported by `SQLAlchemy`.\n",
+    "\n",
+    "### Standard cache"
    ]
   },
   {
@@ -1112,11 +1121,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# from langchain.cache import SQLAlchemyCache\n",
-    "# from sqlalchemy import create_engine\n",
+    "from langchain.cache import SQLAlchemyCache\n",
+    "from sqlalchemy import create_engine\n",
     "\n",
-    "# engine = create_engine(\"postgresql://postgres:postgres@localhost:5432/postgres\")\n",
-    "# set_llm_cache(SQLAlchemyCache(engine))"
+    "engine = create_engine(\"postgresql://postgres:postgres@localhost:5432/postgres\")\n",
+    "set_llm_cache(SQLAlchemyCache(engine))"
    ]
   },
   {
@@ -1124,7 +1133,9 @@
    "id": "0959d640",
    "metadata": {},
    "source": [
-    "### Custom SQLAlchemy Schemas"
+    "### Custom SQLAlchemy schemas\n",
+    "\n",
+    "You can define your own declarative `SQLAlchemyCache` child class to customize the schema used for caching. For example, to support high-speed fulltext prompt indexing with `Postgres`, use:"
    ]
   },
   {
@@ -1134,8 +1145,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# You can define your own declarative SQLAlchemyCache child class to customize the schema used for caching. For example, to support high-speed fulltext prompt indexing with Postgres, use:\n",
-    "\n",
     "from langchain_community.cache import SQLAlchemyCache\n",
     "from sqlalchemy import Column, Computed, Index, Integer, Sequence, String, create_engine\n",
     "from sqlalchemy.ext.declarative import declarative_base\n",
@@ -1185,7 +1194,7 @@
    "id": "6cf6acb4-1bc4-4c4b-9325-2420c17e5e2b",
    "metadata": {},
    "source": [
-    "### Required dependency"
+    "Required dependency:"
    ]
   },
   {
@@ -1203,7 +1212,7 @@
    "id": "a4a6725d",
    "metadata": {},
    "source": [
-    "### Connect to the DB\n",
+    "### Connecting to the DB\n",
     "\n",
     "The Cassandra caches shown in this page can be used with Cassandra as well as other derived databases, such as Astra DB, which use the CQL (Cassandra Query Language) protocol.\n",
     "\n",
@@ -1217,7 +1226,7 @@
    "id": "15735abe-2567-43ce-aa91-f253b33b5a88",
    "metadata": {},
    "source": [
-    "#### Connecting to a Cassandra cluster\n",
+    "#### to a Cassandra cluster\n",
     "\n",
     "You first need to create a `cassandra.cluster.Session` object, as described in the [Cassandra driver documentation](https://docs.datastax.com/en/developer/python-driver/latest/api/cassandra/cluster/#module-cassandra.cluster). The details vary (e.g. with network settings and authentication), but this might be something like:"
    ]
@@ -1270,7 +1279,7 @@
    "id": "2cc7ba29-8f84-4fbf-aaf7-3daa1be7e7b0",
    "metadata": {},
    "source": [
-    "#### Connecting to Astra DB through CQL\n",
+    "#### to Astra DB through CQL\n",
     "\n",
     "In this case you initialize CassIO with the following connection parameters:\n",
     "\n",
@@ -1329,7 +1338,7 @@
    "id": "8665664a",
    "metadata": {},
    "source": [
-    "### Cassandra: Exact cache\n",
+    "### Standard cache\n",
     "\n",
     "This will avoid invoking the LLM when the supplied prompt is _exactly_ the same as one encountered already:"
    ]
@@ -1400,7 +1409,7 @@
    "id": "8fc4d017",
    "metadata": {},
    "source": [
-    "### Cassandra: Semantic cache\n",
+    "### Semantic cache\n",
     "\n",
     "This cache will do a semantic similarity search and return a hit if it finds a cached entry that is similar enough, For this, you need to provide an `Embeddings` instance of your choice."
    ]
@@ -1488,9 +1497,9 @@
    "id": "55dc84b3-37cb-4f19-b175-40e18e06f83f",
    "metadata": {},
    "source": [
-    "#### Attribution statement\n",
+    "**Attribution statement:**\n",
     "\n",
-    ">Apache Cassandra, Cassandra and Apache are either registered trademarks or trademarks of the [Apache Software Foundation](http://www.apache.org/) in the United States and/or other countries."
+    ">`Apache Cassandra`, `Cassandra` and `Apache` are either registered trademarks or trademarks of the [Apache Software Foundation](http://www.apache.org/) in the United States and/or other countries."
    ]
   },
   {
@@ -1498,7 +1507,7 @@
    "id": "8712f8fc-bb89-4164-beb9-c672778bbd91",
    "metadata": {},
    "source": [
-    "## `Astra DB` Caches"
+    "## `Astra DB` caches"
    ]
   },
   {
@@ -1543,7 +1552,7 @@
    "id": "ee6d587f-4b7c-43f4-9e90-5129c842a143",
    "metadata": {},
    "source": [
-    "### Astra DB exact LLM cache\n",
+    "### Standard cache\n",
     "\n",
     "This will avoid invoking the LLM when the supplied prompt is _exactly_ the same as one encountered already:"
    ]
@@ -1619,7 +1628,7 @@
    "id": "524b94fa-6162-4880-884d-d008749d14e2",
    "metadata": {},
    "source": [
-    "### Astra DB Semantic cache\n",
+    "### Semantic cache\n",
     "\n",
     "This cache will do a semantic similarity search and return a hit if it finds a cached entry that is similar enough, For this, you need to provide an `Embeddings` instance of your choice."
    ]
@@ -1713,7 +1722,7 @@
     }
    },
    "source": [
-    "## Azure Cosmos DB Semantic Cache\n",
+    "## `Azure Cosmos DB` semantic cache\n",
     "\n",
     "You can use this integrated [vector database](https://learn.microsoft.com/en-us/azure/cosmos-db/vector-database) for caching."
    ]
@@ -1848,18 +1857,162 @@
      "output_type": "execute_result"
     }
    ],
+   "source": [
+    "%%time\n",
+    "# The second time it is, so it goes faster\n",
+    "llm.invoke(\"Tell me a joke\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "235ff73bf7143f13",
+   "metadata": {},
+   "source": [
+    "## `Azure Cosmos DB NoSql` semantic cache\n",
+    "\n",
+    "You can use this integrated [vector database](https://learn.microsoft.com/en-us/azure/cosmos-db/vector-database) for caching."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "41fea5aa7b2153ca",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-12-06T00:55:38.648972Z",
+     "start_time": "2024-12-06T00:55:38.290541Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from typing import Any, Dict\n",
+    "\n",
+    "from azure.cosmos import CosmosClient, PartitionKey\n",
+    "from langchain_community.cache import AzureCosmosDBNoSqlSemanticCache\n",
+    "from langchain_openai import OpenAIEmbeddings\n",
+    "\n",
+    "HOST = \"COSMOS_DB_URI\"\n",
+    "KEY = \"COSMOS_DB_KEY\"\n",
+    "\n",
+    "cosmos_client = CosmosClient(HOST, KEY)\n",
+    "\n",
+    "\n",
+    "def get_vector_indexing_policy() -> dict:\n",
+    "    return {\n",
+    "        \"indexingMode\": \"consistent\",\n",
+    "        \"includedPaths\": [{\"path\": \"/*\"}],\n",
+    "        \"excludedPaths\": [{\"path\": '/\"_etag\"/?'}],\n",
+    "        \"vectorIndexes\": [{\"path\": \"/embedding\", \"type\": \"diskANN\"}],\n",
+    "    }\n",
+    "\n",
+    "\n",
+    "def get_vector_embedding_policy() -> dict:\n",
+    "    return {\n",
+    "        \"vectorEmbeddings\": [\n",
+    "            {\n",
+    "                \"path\": \"/embedding\",\n",
+    "                \"dataType\": \"float32\",\n",
+    "                \"dimensions\": 1536,\n",
+    "                \"distanceFunction\": \"cosine\",\n",
+    "            }\n",
+    "        ]\n",
+    "    }\n",
+    "\n",
+    "\n",
+    "cosmos_container_properties_test = {\"partition_key\": PartitionKey(path=\"/id\")}\n",
+    "cosmos_database_properties_test: Dict[str, Any] = {}\n",
+    "\n",
+    "set_llm_cache(\n",
+    "    AzureCosmosDBNoSqlSemanticCache(\n",
+    "        cosmos_client=cosmos_client,\n",
+    "        embedding=OpenAIEmbeddings(),\n",
+    "        vector_embedding_policy=get_vector_embedding_policy(),\n",
+    "        indexing_policy=get_vector_indexing_policy(),\n",
+    "        cosmos_container_properties=cosmos_container_properties_test,\n",
+    "        cosmos_database_properties=cosmos_database_properties_test,\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "1e1cd93819921bf6",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-12-06T00:55:44.513080Z",
+     "start_time": "2024-12-06T00:55:41.353843Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "CPU times: user 374 ms, sys: 34.2 ms, total: 408 ms\n",
+      "Wall time: 3.15 s\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"\\n\\nWhy couldn't the bicycle stand up by itself? Because it was two-tired!\""
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "%%time\n",
     "# The first time, it is not yet in cache, so it should take longer\n",
     "llm.invoke(\"Tell me a joke\")"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "576ce24c1244812a",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-12-06T00:55:50.925865Z",
+     "start_time": "2024-12-06T00:55:50.548520Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "CPU times: user 17.7 ms, sys: 2.88 ms, total: 20.6 ms\n",
+      "Wall time: 373 ms\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "\"\\n\\nWhy couldn't the bicycle stand up by itself? Because it was two-tired!\""
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "%%time\n",
+    "# The second time it is, so it goes faster\n",
+    "llm.invoke(\"Tell me a joke\")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "306ff47b",
    "metadata": {},
    "source": [
-    "## `Elasticsearch` Cache\n",
+    "## `Elasticsearch` caches\n",
+    "\n",
     "A caching layer for LLMs that uses Elasticsearch.\n",
     "\n",
     "First install the LangChain integration with Elasticsearch."
@@ -1880,6 +2033,8 @@
    "id": "9e70b0a0",
    "metadata": {},
    "source": [
+    "### Standard cache\n",
+    "\n",
     "Use the class `ElasticsearchCache`.\n",
     "\n",
     "Simple example:"
@@ -1987,6 +2142,26 @@
     "please only make additive modifications, keeping the base mapping intact."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "3dc15b3c-8793-432e-98c3-d2726d497a5a",
+   "metadata": {},
+   "source": [
+    "### Embedding cache\n",
+    "\n",
+    "An Elasticsearch store for caching embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b0ae5bd3-517d-470f-9b44-14d9359e6940",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_elasticsearch import ElasticsearchEmbeddingsCache"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "0c69d84d",
@@ -1994,8 +2169,9 @@
     "tags": []
    },
    "source": [
-    "## Optional Caching\n",
-    "You can also turn off caching for specific LLMs should you choose. In the example below, even though global caching is enabled, we turn it off for a specific LLM"
+    "## LLM-specific optional caching\n",
+    "\n",
+    "You can also turn off caching for specific LLMs. In the example below, even though global caching is enabled, we turn it off for a specific LLM."
    ]
   },
   {
@@ -2075,7 +2251,8 @@
     "tags": []
    },
    "source": [
-    "## Optional Caching in Chains\n",
+    "## Optional caching in Chains\n",
+    "\n",
     "You can also turn off caching for particular nodes in chains. Note that because of certain interfaces, its often easier to construct the chain first, and then edit the LLM afterwards.\n",
     "\n",
     "As an example, we will load a summarizer map-reduce chain. We will cache results for the map-step, but then not freeze it for the combine step."
@@ -2242,7 +2419,7 @@
    "id": "9ecfa565038eff71",
    "metadata": {},
    "source": [
-    "## OpenSearch Semantic Cache\n",
+    "## `OpenSearch` semantic cache\n",
     "Use [OpenSearch](https://python.langchain.com/docs/integrations/vectorstores/opensearch/) as a semantic cache to cache prompts and responses and evaluate hits based on semantic similarity."
    ]
   },
@@ -2346,7 +2523,8 @@
    "id": "2ac1a8c7",
    "metadata": {},
    "source": [
-    "## SingleStoreDB Semantic Cache\n",
+    "## `SingleStoreDB` semantic cache\n",
+    "\n",
     "You can use [SingleStoreDB](https://python.langchain.com/docs/integrations/vectorstores/singlestoredb/) as a semantic cache to cache prompts and responses."
    ]
   },
@@ -2373,7 +2551,7 @@
    "id": "7e6b9b1a",
    "metadata": {},
    "source": [
-    "## `Memcached` Cache\n",
+    "## `Memcached` cache\n",
     "You can use [Memcached](https://www.memcached.org/) as a cache to cache prompts and responses through [pymemcache](https://github.com/pinterest/pymemcache).\n",
     "\n",
     "This cache requires the pymemcache dependency to be installed:"
@@ -2469,7 +2647,7 @@
    "id": "7019c991-0101-4f9c-b212-5729a5471293",
    "metadata": {},
    "source": [
-    "## Couchbase Caches\n",
+    "## `Couchbase` caches\n",
     "\n",
     "Use [Couchbase](https://couchbase.com/) as a cache for prompts and responses."
    ]
@@ -2479,7 +2657,7 @@
    "id": "d6aac680-ba32-4c19-8864-6471cf0e7d5a",
    "metadata": {},
    "source": [
-    "### Couchbase Cache\n",
+    "### Standard cache\n",
     "\n",
     "The standard cache that looks for an exact match of the user prompt."
    ]
@@ -2613,7 +2791,7 @@
    "id": "1dca39d8-233a-45ba-ad7d-0920dfbc4a50",
    "metadata": {},
    "source": [
-    "### Specifying a Time to Live (TTL) for the Cached entries\n",
+    "#### Time to Live (TTL) for the cached entries\n",
     "The Cached documents can be deleted after a specified time automatically by specifying a `ttl` parameter along with the initialization of the Cache."
    ]
   },
@@ -2642,7 +2820,7 @@
    "id": "43626f33-d184-4260-b641-c9341cef5842",
    "metadata": {},
    "source": [
-    "### Couchbase Semantic Cache\n",
+    "### Semantic cache\n",
     "Semantic caching allows users to retrieve cached prompts based on semantic similarity between the user input and previously cached inputs. Under the hood it uses Couchbase as both a cache and a vectorstore. This needs an appropriate Vector Search Index defined to work. Please look at the usage example on how to set up the index."
    ]
   },
@@ -2685,7 +2863,9 @@
     "- The search index for the semantic cache needs to be defined before using the semantic cache. \n",
     "- The optional parameter, `score_threshold` in the Semantic Cache that you can use to tune the results of the semantic search.\n",
     "\n",
-    "### How to Import an Index to the Full Text Search service?\n",
+    "#### Index to the Full Text Search service\n",
+    "\n",
+    "How to Import an Index to the Full Text Search service?\n",
     " - [Couchbase Server](https://docs.couchbase.com/server/current/search/import-search-index.html)\n",
     "     - Click on Search -> Add Index -> Import\n",
     "     - Copy the following Index definition in the Import screen\n",
@@ -2695,7 +2875,8 @@
     "     - Import the file in Capella using the instructions in the documentation.\n",
     "     - Click on Create Index to create the index.\n",
     "\n",
-    "#### Example index for the vector search. \n",
+    "**Example index for the vector search:**\n",
+    "\n",
     "  ```\n",
     "  {\n",
     "    \"type\": \"fulltext-index\",\n",
@@ -2855,7 +3036,8 @@
    "id": "f6f674fa-70b5-4cf9-a208-992aad2c3c89",
    "metadata": {},
    "source": [
-    "### Specifying a Time to Live (TTL) for the Cached entries\n",
+    "#### Time to Live (TTL) for the cached entries\n",
+    "\n",
     "The Cached documents can be deleted after a specified time automatically by specifying a `ttl` parameter along with the initialization of the Cache."
    ]
   },
@@ -2913,10 +3095,10 @@
    "source": [
     "**Cache** classes are implemented by inheriting the [BaseCache](https://python.langchain.com/api_reference/core/caches/langchain_core.caches.BaseCache.html) class.\n",
     "\n",
-    "This table lists all 21 derived classes with links to the API Reference.\n",
+    "This table lists all derived classes with links to the API Reference.\n",
     "\n",
     "\n",
-    "| Namespace 🔻 | Class |\n",
+    "| Namespace | Class 🔻 |\n",
     "|------------|---------|\n",
     "| langchain_astradb.cache | [AstraDBCache](https://python.langchain.com/api_reference/astradb/cache/langchain_astradb.cache.AstraDBCache.html) |\n",
     "| langchain_astradb.cache | [AstraDBSemanticCache](https://python.langchain.com/api_reference/astradb/cache/langchain_astradb.cache.AstraDBSemanticCache.html) |\n",
@@ -2925,22 +3107,31 @@
     "| langchain_community.cache | [AzureCosmosDBSemanticCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.AzureCosmosDBSemanticCache.html) |\n",
     "| langchain_community.cache | [CassandraCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.CassandraCache.html) |\n",
     "| langchain_community.cache | [CassandraSemanticCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.CassandraSemanticCache.html) |\n",
+    "| langchain_couchbase.cache | [CouchbaseCache](https://python.langchain.com/api_reference/couchbase/cache/langchain_couchbase.cache.CouchbaseCache.html) |\n",
+    "| langchain_couchbase.cache | [CouchbaseSemanticCache](https://python.langchain.com/api_reference/couchbase/cache/langchain_couchbase.cache.CouchbaseSemanticCache.html) |\n",
+    "| langchain_elasticsearch.cache | [ElasticsearchCache](https://python.langchain.com/api_reference/elasticsearch/cache/langchain_elasticsearch.cache.AsyncElasticsearchCache.html) |\n",
+    "| langchain_elasticsearch.cache | [ElasticsearchEmbeddingsCache](https://python.langchain.com/api_reference/elasticsearch/cache/langchain_elasticsearch.cache.AsyncElasticsearchEmbeddingsCache.html) |\n",
     "| langchain_community.cache | [GPTCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.GPTCache.html) |\n",
+    "| langchain_core.caches | [InMemoryCache](https://python.langchain.com/api_reference/core/caches/langchain_core.caches.InMemoryCache.html) |\n",
     "| langchain_community.cache | [InMemoryCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.InMemoryCache.html) |\n",
     "| langchain_community.cache | [MomentoCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.MomentoCache.html) |\n",
+    "| langchain_mongodb.cache | [MongoDBAtlasSemanticCache](https://python.langchain.com/api_reference/mongodb/cache/langchain_mongodb.cache.MongoDBAtlasSemanticCache.html) |\n",
+    "| langchain_mongodb.cache | [MongoDBCache](https://python.langchain.com/api_reference/mongodb/cache/langchain_mongodb.cache.MongoDBCache.html) |\n",
     "| langchain_community.cache | [OpenSearchSemanticCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.OpenSearchSemanticCache.html) |\n",
     "| langchain_community.cache | [RedisSemanticCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.RedisSemanticCache.html) |\n",
     "| langchain_community.cache | [SingleStoreDBSemanticCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.SingleStoreDBSemanticCache.html) |\n",
     "| langchain_community.cache | [SQLAlchemyCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.SQLAlchemyCache.html) |\n",
     "| langchain_community.cache | [SQLAlchemyMd5Cache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.SQLAlchemyMd5Cache.html) |\n",
-    "| langchain_community.cache | [UpstashRedisCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.UpstashRedisCache.html) |\n",
-    "| langchain_core.caches | [InMemoryCache](https://python.langchain.com/api_reference/core/caches/langchain_core.caches.InMemoryCache.html) |\n",
-    "| langchain_elasticsearch.cache | [ElasticsearchCache](https://python.langchain.com/api_reference/elasticsearch/cache/langchain_elasticsearch.cache.ElasticsearchCache.html) |\n",
-    "| langchain_mongodb.cache | [MongoDBAtlasSemanticCache](https://python.langchain.com/api_reference/mongodb/cache/langchain_mongodb.cache.MongoDBAtlasSemanticCache.html) |\n",
-    "| langchain_mongodb.cache | [MongoDBCache](https://python.langchain.com/api_reference/mongodb/cache/langchain_mongodb.cache.MongoDBCache.html) |\n",
-    "| langchain_couchbase.cache | [CouchbaseCache](https://python.langchain.com/api_reference/couchbase/cache/langchain_couchbase.cache.CouchbaseCache.html) |\n",
-    "| langchain_couchbase.cache | [CouchbaseSemanticCache](https://python.langchain.com/api_reference/couchbase/cache/langchain_couchbase.cache.CouchbaseSemanticCache.html) |\n"
+    "| langchain_community.cache | [UpstashRedisCache](https://python.langchain.com/api_reference/community/cache/langchain_community.cache.UpstashRedisCache.html) |\n"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1ef1a2d4-da2e-4fb1-aae4-ffc4aef6c3ad",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -2959,7 +3150,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.13"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

docs/docs/integrations/llms/openllm.ipynb

@@ -7,7 +7,14 @@
    "source": [
     "# OpenLLM\n",
     "\n",
-    "[🦾 OpenLLM](https://github.com/bentoml/OpenLLM) is an open platform for operating large language models (LLMs) in production. It enables developers to easily run inference with any open-source LLMs, deploy to the cloud or on-premises, and build powerful AI apps."
+    "[🦾 OpenLLM](https://github.com/bentoml/OpenLLM) lets developers run any **open-source LLMs** as **OpenAI-compatible API** endpoints with **a single command**.\n",
+    "\n",
+    "- 🔬 Build for fast and production usages\n",
+    "- 🚂 Support llama3, qwen2, gemma, etc, and many **quantized** versions [full list](https://github.com/bentoml/openllm-models)\n",
+    "- ⛓️ OpenAI-compatible API\n",
+    "- 💬 Built-in ChatGPT like UI\n",
+    "- 🔥 Accelerated LLM decoding with state-of-the-art inference backends\n",
+    "- 🌥️ Ready for enterprise-grade cloud deployment (Kubernetes, Docker and BentoCloud)"
    ]
   },
   {
@@ -37,10 +44,10 @@
    "source": [
     "## Launch OpenLLM server locally\n",
     "\n",
-    "To start an LLM server, use `openllm start` command. For example, to start a dolly-v2 server, run the following command from a terminal:\n",
+    "To start an LLM server, use `openllm hello` command:\n",
     "\n",
     "```bash\n",
-    "openllm start dolly-v2\n",
+    "openllm hello\n",
     "```\n",
     "\n",
     "\n",
@@ -57,74 +64,7 @@
     "from langchain_community.llms import OpenLLM\n",
     "\n",
     "server_url = \"http://localhost:3000\"  # Replace with remote host if you are running on a remote server\n",
-    "llm = OpenLLM(server_url=server_url)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "4f830f9d",
-   "metadata": {},
-   "source": [
-    "### Optional: Local LLM Inference\n",
-    "\n",
-    "You may also choose to initialize an LLM managed by OpenLLM locally from current process. This is useful for development purpose and allows developers to quickly try out different types of LLMs.\n",
-    "\n",
-    "When moving LLM applications to production, we recommend deploying the OpenLLM server separately and access via the `server_url` option demonstrated above.\n",
-    "\n",
-    "To load an LLM locally via the LangChain wrapper:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "82c392b6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_community.llms import OpenLLM\n",
-    "\n",
-    "llm = OpenLLM(\n",
-    "    model_name=\"dolly-v2\",\n",
-    "    model_id=\"databricks/dolly-v2-3b\",\n",
-    "    temperature=0.94,\n",
-    "    repetition_penalty=1.2,\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f15ebe0d",
-   "metadata": {},
-   "source": [
-    "### Integrate with a LLMChain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "8b02a97a",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "iLkb\n"
-     ]
-    }
-   ],
-   "source": [
-    "from langchain.chains import LLMChain\n",
-    "from langchain_core.prompts import PromptTemplate\n",
-    "\n",
-    "template = \"What is a good name for a company that makes {product}?\"\n",
-    "\n",
-    "prompt = PromptTemplate.from_template(template)\n",
-    "\n",
-    "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
-    "\n",
-    "generated = llm_chain.run(product=\"mechanical keyboard\")\n",
-    "print(generated)"
+    "llm = OpenLLM(base_url=server_url, api_key=\"na\")"
    ]
   },
   {
@@ -133,7 +73,9 @@
    "id": "56cb4bc0",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "llm(\"To build a LLM from scratch, the following are the steps:\")"
+   ]
   }
  ],
  "metadata": {
@@ -152,7 +94,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,

docs/docs/integrations/providers/aerospike.mdx

@@ -0,0 +1,24 @@
+# Aerospike
+
+>[Aerospike](https://aerospike.com/docs/vector) is a high-performance, distributed database known for its speed and scalability, now with support for vector storage and search, enabling retrieval and search of embedding vectors for machine learning and AI applications.
+> See the documentation for Aerospike Vector Search (AVS) [here](https://aerospike.com/docs/vector).
+
+## Installation and Setup
+
+Install the AVS Python SDK and AVS langchain vector store:
+
+```bash
+pip install aerospike-vector-search langchain-community
+
+See the documentation for the Ptyhon SDK [here](https://aerospike-vector-search-python-client.readthedocs.io/en/latest/index.html).
+The documentation for the AVS langchain vector store is [here](https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.aerospike.Aerospike.html).
+
+## Vector Store
+
+To import this vectorstore:
+
+```python
+from langchain_community.vectorstores import Aerospike
+
+See a usage example [here](https://python.langchain.com/docs/integrations/vectorstores/aerospike/).
+

docs/docs/integrations/providers/alibaba_cloud.mdx

@@ -89,3 +89,11 @@ See [installation instructions and a usage example](/docs/integrations/vectorsto
 ```python
 from langchain_community.vectorstores import Hologres
 ```
+
+### Tablestore
+
+See [installation instructions and a usage example](/docs/integrations/vectorstores/tablestore).
+
+```python
+from langchain_community.vectorstores import TablestoreVectorStore
+```

docs/docs/integrations/providers/couchbase.mdx

@@ -33,7 +33,7 @@ from langchain_community.document_loaders.couchbase import CouchbaseLoader
 ### CouchbaseCache
 Use Couchbase as a cache for prompts and responses.
 
-See a [usage example](/docs/integrations/llm_caching/#couchbase-cache).
+See a [usage example](/docs/integrations/llm_caching/#couchbase-caches).
 
 To import this cache:
 ```python
@@ -61,7 +61,7 @@ set_llm_cache(
 Semantic caching allows users to retrieve cached prompts based on the semantic similarity between the user input and previously cached inputs. Under the hood it uses Couchbase as both a cache and a vectorstore.
 The CouchbaseSemanticCache needs a Search Index defined to work. Please look at the [usage example](/docs/integrations/vectorstores/couchbase) on how to set up the index.
 
-See a [usage example](/docs/integrations/llm_caching/#couchbase-semantic-cache).
+See a [usage example](/docs/integrations/llm_caching/#couchbase-caches).
 
 To import this cache:
 ```python

docs/docs/integrations/providers/dataherald.mdx

@@ -41,7 +41,7 @@ tool = DataheraldTextToSQL(api_wrapper=api_wrapper)
 llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
 prompt = hub.pull("hwchase17/react")
 agent = create_react_agent(llm, tools, prompt)
-agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
+agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=True)
 agent_executor.invoke({"input":"Return the sql for this question: How many employees are in the company?"})
 ```
 

docs/docs/integrations/providers/elasticsearch.mdx

@@ -84,7 +84,7 @@ from langchain_elasticsearch import ElasticsearchChatMessageHistory
 
 ## LLM cache
 
-See a [usage example](/docs/integrations/llm_caching/#elasticsearch-cache).
+See a [usage example](/docs/integrations/llm_caching/#elasticsearch-caches).
 
 ```python
 from langchain_elasticsearch import ElasticsearchCache

docs/docs/integrations/providers/jina.mdx

@@ -2,6 +2,10 @@
 
 >[Jina AI](https://jina.ai/about-us) is a search AI company. `Jina` helps businesses and developers unlock multimodal data with a better search.
 
+:::caution
+For proper compatibility, please ensure you are using the `openai` SDK at version **0.x**.
+:::
+
 ## Installation and Setup
 - Get a Jina AI API token from [here](https://jina.ai/embeddings/) and set it as an environment variable (`JINA_API_TOKEN`)
 

docs/docs/integrations/providers/linkup.mdx

@@ -0,0 +1,39 @@
+# Linkup
+
+> [Linkup](https://www.linkup.so/) provides an API to connect LLMs to the web and the Linkup Premium Partner sources.
+
+## Installation and Setup
+
+To use the Linkup provider, you first need a valid API key, which you can find by signing-up [here](https://app.linkup.so/sign-up).
+You will also need the `langchain-linkup` package, which you can install using pip:
+
+```bash
+pip install langchain-linkup
+```
+
+## Retriever
+
+See a [usage example](/docs/integrations/retrievers/linkup_search).
+
+```python
+from langchain_linkup import LinkupSearchRetriever
+
+retriever = LinkupSearchRetriever(
+    depth="deep",  # "standard" or "deep"
+    linkup_api_key=None,  # API key can be passed here or set as the LINKUP_API_KEY environment variable
+)
+```
+
+## Tools
+
+See a [usage example](/docs/integrations/tools/linkup_search).
+
+```python
+from langchain_linkup import LinkupSearchTool
+
+tool = LinkupSearchTool(
+    depth="deep",  # "standard" or "deep"
+    output_type="searchResults",  # "searchResults", "sourcedAnswer" or "structured"
+    linkup_api_key=None,  # API key can be passed here or set as the LINKUP_API_KEY environment variable
+)
+```

docs/docs/integrations/providers/localai.mdx

@@ -6,6 +6,10 @@
 > audio (and not only) locally or on-prem with consumer grade hardware, 
 > supporting multiple model families and architectures.
 
+:::caution
+For proper compatibility, please ensure you are using the `openai` SDK at version **0.x**.
+:::
+
 ## Installation and Setup
 
 We have to install several python packages: 

docs/docs/integrations/providers/microsoft.mdx

@@ -343,6 +343,31 @@ See a [usage example](/docs/integrations/memory/postgres_chat_message_history/).
 
 Since Azure Database for PostgreSQL is open-source Postgres, you can use the [LangChain's Postgres support](/docs/integrations/vectorstores/pgvector/) to connect to Azure Database for PostgreSQL.
 
+### Azure SQL Database
+
+>[Azure SQL Database](https://learn.microsoft.com/azure/azure-sql/database/sql-database-paas-overview?view=azuresql) is a robust service that combines scalability, security, and high availability, providing all the benefits of a modern database solution.  It also provides a dedicated Vector data type & built-in functions that simplifies the storage and querying of vector embeddings directly within a relational database. This eliminates the need for separate vector databases and related integrations, increasing the security of your solutions while reducing the overall complexity.
+
+By leveraging your current SQL Server databases for vector search, you can enhance data capabilities while minimizing expenses and avoiding the challenges of transitioning to new systems.
+
+##### Installation and Setup
+
+See [detail configuration instructions](/docs/integrations/vectorstores/sqlserver).
+
+We need to install the `langchain-sqlserver` python package.
+
+```bash
+!pip install langchain-sqlserver==0.1.1
+```
+
+##### Deploy Azure SQL DB on Microsoft Azure
+
+[Sign Up](https://learn.microsoft.com/azure/azure-sql/database/free-offer?view=azuresql) for free to get started today.
+
+See a [usage example](/docs/integrations/vectorstores/sqlserver).
+
+```python
+from langchain_sqlserver import SQLServer_VectorStore
+```
 
 ### Azure AI Search
 

docs/docs/integrations/providers/oceanbase.mdx

@@ -0,0 +1,31 @@
+# OceanBase
+
+[OceanBase Database](https://github.com/oceanbase/oceanbase) is a distributed relational database. 
+It is developed entirely by Ant Group. The OceanBase Database is built on a common server cluster. 
+Based on the Paxos protocol and its distributed structure, the OceanBase Database provides high availability and linear scalability.
+
+OceanBase currently has the ability to store vectors. Users can easily perform the following operations with SQL:
+
+- Create a table containing vector type fields;
+- Create a vector index table based on the HNSW algorithm;
+- Perform vector approximate nearest neighbor queries;
+- ...
+
+## Installation
+
+```bash
+pip install -U langchain-oceanbase
+```
+
+We recommend using Docker to deploy OceanBase:
+
+```shell
+docker run --name=ob433 -e MODE=slim -p 2881:2881 -d oceanbase/oceanbase-ce:4.3.3.0-100000132024100711
+```
+
+[More methods to deploy OceanBase cluster](https://github.com/oceanbase/oceanbase-doc/blob/V4.3.1/en-US/400.deploy/500.deploy-oceanbase-database-community-edition/100.deployment-overview.md)
+
+### Usage
+
+For a more detailed walkthrough of the OceanBase Wrapper, see [this notebook](https://github.com/oceanbase/langchain-oceanbase/blob/main/docs/vectorstores.ipynb)
+

docs/docs/integrations/providers/openllm.mdx

@@ -1,11 +1,17 @@
+---
+keywords: [openllm]
+---
+
 # OpenLLM
 
-This page demonstrates how to use [OpenLLM](https://github.com/bentoml/OpenLLM)
-with LangChain.
+OpenLLM lets developers run any **open-source LLMs** as **OpenAI-compatible API** endpoints with **a single command**.
 
-`OpenLLM` is an open platform for operating large language models (LLMs) in
-production. It enables developers to easily run inference with any open-source
-LLMs, deploy to the cloud or on-premises, and build powerful AI apps.
+- 🔬 Build for fast and production usages
+- 🚂 Support llama3, qwen2, gemma, etc, and many **quantized** versions [full list](https://github.com/bentoml/openllm-models)
+- ⛓️ OpenAI-compatible API
+- 💬 Built-in ChatGPT like UI
+- 🔥 Accelerated LLM decoding with state-of-the-art inference backends
+- 🌥️ Ready for enterprise-grade cloud deployment (Kubernetes, Docker and BentoCloud)
 
 ## Installation and Setup
 
@@ -23,8 +29,7 @@ are pre-optimized for OpenLLM.
 
 ## Wrappers
 
-There is a OpenLLM Wrapper which supports loading LLM in-process or accessing a
-remote OpenLLM server:
+There is a OpenLLM Wrapper which supports interacting with running server with OpenLLM:
 
 ```python
 from langchain_community.llms import OpenLLM
@@ -32,13 +37,12 @@ from langchain_community.llms import OpenLLM
 
 ### Wrapper for OpenLLM server
 
-This wrapper supports connecting to an OpenLLM server via HTTP or gRPC. The
-OpenLLM server can run either locally or on the cloud.
+This wrapper supports interacting with OpenLLM's OpenAI-compatible endpoint.
 
-To try it out locally, start an OpenLLM server:
+To run a model, do:
 
 ```bash
-openllm start flan-t5
+openllm hello
 ```
 
 Wrapper usage:
@@ -46,20 +50,7 @@ Wrapper usage:
 ```python
 from langchain_community.llms import OpenLLM
 
-llm = OpenLLM(server_url='http://localhost:3000')
-
-llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
-```
-
-### Wrapper for Local Inference
-
-You can also use the OpenLLM wrapper to load LLM in current Python process for
-running inference.
-
-```python
-from langchain_community.llms import OpenLLM
-
-llm = OpenLLM(model_name="dolly-v2", model_id='databricks/dolly-v2-7b')
+llm = OpenLLM(base_url="http://localhost:3000/v1", api_key="na")
 
 llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
 ```

docs/docs/integrations/providers/robocorp.mdx

@@ -1,4 +1,4 @@
-# Robocorp
+# Sema4 (fka Robocorp)
 
 >[Robocorp](https://robocorp.com/) helps build and operate Python workers that run seamlessly anywhere at any scale
 

docs/docs/integrations/providers/scrapegraph.mdx

@@ -0,0 +1,41 @@
+# ScrapeGraph AI
+
+>[ScrapeGraph AI](https://scrapegraphai.com) is a service that provides AI-powered web scraping capabilities.
+>It offers tools for extracting structured data, converting webpages to markdown, and processing local HTML content
+>using natural language prompts.
+
+## Installation and Setup
+
+Install the required packages:
+
+```bash
+pip install langchain-scrapegraph
+```
+
+Set up your API key:
+
+```bash
+export SGAI_API_KEY="your-scrapegraph-api-key"
+```
+
+## Tools
+
+See a [usage example](/docs/integrations/tools/scrapegraph).
+
+There are four tools available:
+
+```python
+from langchain_scrapegraph.tools import (
+    SmartScraperTool,    # Extract structured data from websites
+    MarkdownifyTool,     # Convert webpages to markdown
+    LocalScraperTool,    # Process local HTML content
+    GetCreditsTool,      # Check remaining API credits
+)
+```
+
+Each tool serves a specific purpose:
+
+- `SmartScraperTool`: Extract structured data from websites given a URL, prompt and optional output schema
+- `MarkdownifyTool`: Convert any webpage to clean markdown format
+- `LocalScraperTool`: Extract structured data from a local HTML file given a prompt and optional output schema
+- `GetCreditsTool`: Check your remaining ScrapeGraph AI credits 

docs/docs/integrations/providers/unstructured.mdx

@@ -22,7 +22,7 @@ dependencies running.
 
 - To run everything locally, install the open-source python package with `pip install unstructured`
   along with `pip install langchain-community` and use the same `UnstructuredLoader` as mentioned above.
-    - You can install document specific dependencies with extras, e.g. `pip install "unstructured[docx]"`.
+    - You can install document specific dependencies with extras, e.g. `pip install "unstructured[docx]"`. Learn more about extras [here](https://docs.unstructured.io/open-source/installation/full-installation).
     - To install the dependencies for all document types, use `pip install "unstructured[all-docs]"`.
 - Install the following system dependencies if they are not already available on your system with e.g. `brew install` for Mac.
   Depending on what document types you're parsing, you may not need all of these.

docs/docs/integrations/providers/upstage.ipynb

@@ -8,7 +8,7 @@
     "\n",
     ">[Upstage](https://upstage.ai) is a leading artificial intelligence (AI) company specializing in delivering above-human-grade performance LLM components.\n",
     ">\n",
-    ">**Solar Mini Chat** is a fast yet powerful advanced large language model focusing on English and Korean. It has been specifically fine-tuned for multi-turn chat purposes, showing enhanced performance across a wide range of natural language processing tasks, like multi-turn conversation or tasks that require an understanding of long contexts, such as RAG (Retrieval-Augmented Generation), compared to other models of a similar size. This fine-tuning equips it with the ability to handle longer conversations more effectively, making it particularly adept for interactive applications.\n",
+    ">**Solar Pro** is an enterprise-grade LLM optimized for single-GPU deployment, excelling in instruction-following and processing structured formats like HTML and Markdown. It supports English, Korean, and Japanese with top multilingual performance and offers domain expertise in finance, healthcare, and legal.\n",
     "\n",
     ">Other than Solar, Upstage also offers features for real-world RAG (retrieval-augmented generation), such as **Document Parse** and **Groundedness Check**. \n"
    ]
@@ -21,12 +21,12 @@
     "\n",
     "| API | Description | Import | Example usage |\n",
     "| --- | --- | --- | --- |\n",
-    "| Chat | Build assistants using Solar Mini Chat | `from langchain_upstage import ChatUpstage` | [Go](../../chat/upstage) |\n",
+    "| Chat | Build assistants using Solar Chat | `from langchain_upstage import ChatUpstage` | [Go](../../chat/upstage) |\n",
     "| Text Embedding | Embed strings to vectors | `from langchain_upstage import UpstageEmbeddings` | [Go](../../text_embedding/upstage) |\n",
     "| Groundedness Check | Verify groundedness of assistant's response | `from langchain_upstage import UpstageGroundednessCheck` | [Go](../../tools/upstage_groundedness_check) |\n",
     "| Document Parse | Serialize documents with tables and figures | `from langchain_upstage import UpstageDocumentParseLoader` | [Go](../../document_loaders/upstage) |\n",
     "\n",
-    "See [documentations](https://developers.upstage.ai/) for more details about the features."
+    "See [documentations](https://console.upstage.ai/docs/getting-started/overview) for more details about the models and features."
    ]
   },
   {

docs/docs/integrations/providers/wandb.mdx

@@ -0,0 +1,42 @@
+# Weights & Biases
+
+>[Weights & Biases](https://wandb.ai/) is provider of the AI developer platform to train and 
+> fine-tune AI models and develop AI applications.
+ 
+`Weights & Biase` products can be used to log metrics and artifacts during training, 
+and to trace the execution of your code.
+
+There are several main ways to use `Weights & Biases` products within LangChain:
+- with `wandb_tracing_enabled`
+- with `Weave` lightweight toolkit
+- with `WandbCallbackHandler` (deprecated)
+
+
+## wandb_tracing_enabled
+
+See a [usage example](/docs/integrations/providers/wandb_tracing).
+
+See in the [W&B documentation](https://docs.wandb.ai/guides/integrations/langchain).
+
+```python
+from langchain_community.callbacks import wandb_tracing_enabled
+```
+
+## Weave
+
+See in the [W&B documentation](https://weave-docs.wandb.ai/guides/integrations/langchain).
+
+
+## WandbCallbackHandler
+
+**Note:** the `WandbCallbackHandler` is being deprecated in favour of the `wandb_tracing_enabled`.
+
+See a [usage example](/docs/integrations/providers/wandb_tracking).
+
+See in the [W&B documentation](https://docs.wandb.ai/guides/integrations/langchain).
+
+```python
+from langchain_community.callbacks import WandbCallbackHandler
+```
+
+

docs/docs/integrations/providers/wandb_tracing.ipynb

@@ -5,7 +5,7 @@
    "id": "5371a9bb",
    "metadata": {},
    "source": [
-    "# WandB Tracing\n",
+    "# Weights & Biases tracing\n",
     "\n",
     "There are two recommended ways to trace your LangChains:\n",
     "\n",
@@ -165,7 +165,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

docs/docs/integrations/providers/wandb_tracking.ipynb

@@ -6,19 +6,24 @@
    "id": "e43f4ea0",
    "metadata": {},
    "source": [
-    "# Weights & Biases\n",
+    "# Weights & Biases tracking\n",
     "\n",
-    "This notebook goes over how to track your LangChain experiments into one centralized Weights and Biases dashboard. To learn more about prompt engineering and the callback please refer to this Report which explains both alongside the resultant dashboards you can expect to see.\n",
+    "This notebook goes over how to track your LangChain experiments into one centralized `Weights and Biases` dashboard. \n",
     "\n",
+    "To learn more about prompt engineering and the callback please refer to this notebook which explains both alongside the resultant dashboards you can expect to see:\n",
     "\n",
     "<a href=\"https://colab.research.google.com/drive/1DXH4beT4HFaRKy_Vm4PoxhXVDRf7Ym8L?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
     "\n",
     "\n",
-    "[View Report](https://wandb.ai/a-sh0ts/langchain_callback_demo/reports/Prompt-Engineering-LLMs-with-LangChain-and-W-B--VmlldzozNjk1NTUw#👋-how-to-build-a-callback-in-langchain-for-better-prompt-engineering\n",
-    ") \n",
+    "View a detailed description and examples in the [W&B article](https://wandb.ai/a-sh0ts/langchain_callback_demo/reports/Prompt-Engineering-LLMs-with-LangChain-and-W-B--VmlldzozNjk1NTUw#👋-how-to-build-a-callback-in-langchain-for-better-prompt-engineering\n",
+    "). \n",
     "\n",
     "\n",
-    "**Note**: _the `WandbCallbackHandler` is being deprecated in favour of the `WandbTracer`_ . In future please use the `WandbTracer` as it is more flexible and allows for more granular logging. To know more about the `WandbTracer` refer to the [agent_with_wandb_tracing](/docs/integrations/providers/wandb_tracing) notebook or use the following [colab notebook](http://wandb.me/prompts-quickstart). To know more about Weights & Biases Prompts refer to the following [prompts documentation](https://docs.wandb.ai/guides/prompts)."
+    "**Note**: _the `WandbCallbackHandler` is being deprecated in favour of the `WandbTracer`_ . In future please use the `WandbTracer` as it is more flexible and allows for more granular logging. \n",
+    "\n",
+    "To know more about the `WandbTracer` refer to the [agent_with_wandb_tracing](/docs/integrations/providers/wandb_tracing) notebook or use the following [colab notebook](http://wandb.me/prompts-quickstart). \n",
+    "\n",
+    "To know more about Weights & Biases Prompts refer to the following [prompts documentation](https://docs.wandb.ai/guides/prompts)."
    ]
   },
   {
@@ -248,6 +253,38 @@
     "The `flush_tracker` function is used to log LangChain sessions to Weights & Biases. It takes in the LangChain module or agent, and logs at minimum the prompts and generations alongside the serialized form of the LangChain module to the specified Weights & Biases project. By default we reset the session as opposed to concluding the session outright."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "483eefd4-633e-4686-8730-944705fe8a80",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2024-11-12T18:20:31.003316Z",
+     "iopub.status.busy": "2024-11-12T18:20:31.003152Z",
+     "iopub.status.idle": "2024-11-12T18:20:31.006033Z",
+     "shell.execute_reply": "2024-11-12T18:20:31.005546Z",
+     "shell.execute_reply.started": "2024-11-12T18:20:31.003303Z"
+    }
+   },
+   "source": [
+    "## Usage Scenarios"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "066adc04-8180-4936-8d75-5c3660b61de7",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2024-11-12T18:20:45.826326Z",
+     "iopub.status.busy": "2024-11-12T18:20:45.825714Z",
+     "iopub.status.idle": "2024-11-12T18:20:45.830483Z",
+     "shell.execute_reply": "2024-11-12T18:20:45.830037Z",
+     "shell.execute_reply.started": "2024-11-12T18:20:45.826279Z"
+    }
+   },
+   "source": [
+    "### With LLM"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 4,
@@ -373,6 +410,14 @@
     "wandb_callback.flush_tracker(llm, name=\"simple_sequential\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "7dcfc24b-f0b0-48ec-89ce-beeb2fb763d5",
+   "metadata": {},
+   "source": [
+    "### Within Chains"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 5,
@@ -524,6 +569,14 @@
     "wandb_callback.flush_tracker(synopsis_chain, name=\"agent\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "533cd4c9-56e2-4ad9-a83e-4653bfcf322f",
+   "metadata": {},
+   "source": [
+    "### With Agents"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 7,
@@ -646,7 +699,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.1"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

docs/docs/integrations/retrievers/ibm_watsonx_ranker.ipynb

@@ -35,9 +35,9 @@
     "\n",
     "### Integration details\n",
     "\n",
-    "| Class | Package | JS support | Package downloads | Package latest |\n",
+    "| Class | Package | [JS support](https://js.langchain.com/docs/integrations/document_compressors/ibm/) | Package downloads | Package latest |\n",
     "| :--- | :--- | :---: | :---: | :---: |\n",
-    "| [WatsonxRerank](https://python.langchain.com/api_reference/ibm/chat_models/langchain_ibm.rerank.WatsonxRerank.html) | [langchain-ibm](https://python.langchain.com/api_reference/ibm/index.html) | ❌ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-ibm?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-ibm?style=flat-square&label=%20) |"
+    "| [WatsonxRerank](https://python.langchain.com/api_reference/ibm/rerank/langchain_ibm.rerank.WatsonxRerank.html) | [langchain-ibm](https://python.langchain.com/api_reference/ibm/index.html) | ✅ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-ibm?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-ibm?style=flat-square&label=%20) |"
    ]
   },
   {
@@ -445,7 +445,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "langchain_ibm",
    "language": "python",
    "name": "python3"
   },

docs/docs/integrations/retrievers/linkup_search.ipynb

@@ -0,0 +1,270 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "id": "afaf8039",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: LinkupSearchRetriever\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e49f1e0d",
+   "metadata": {},
+   "source": [
+    "# LinkupSearchRetriever\n",
+    "\n",
+    "> [Linkup](https://www.linkup.so/) provides an API to connect LLMs to the web and the Linkup Premium Partner sources.\n",
+    "\n",
+    "This will help you getting started with the LinkupSearchRetriever [retriever](/docs/concepts/retrievers/). For detailed documentation of all LinkupSearchRetriever features and configurations head to the [API reference](https://python.langchain.com/api_reference/linkup/retrievers/linkup_langchain.search_retriever.LinkupSearchRetriever.html).\n",
+    "\n",
+    "### Integration details\n",
+    "\n",
+    "| Retriever | Source | Package |\n",
+    "| :--- | :--- | :---: |\n",
+    "[LinkupSearchRetriever](https://python.langchain.com/api_reference/linkup/retrievers/linkup_langchain.search_retriever.LinkupSearchRetriever.html) | Web and partner sources | langchain-linkup |\n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "To use the Linkup provider, you need a valid API key, which you can find by signing-up [here](https://app.linkup.so/sign-up). You can then set it up as the `LINKUP_API_KEY` environment variable. For the chain example below, you also need to set an OpenAI API key as `OPENAI_API_KEY` environment variable, which you can also do here:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0c6cab32-8f55-473d-b5bc-72673ea4da61",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import os\n",
+    "# os.environ[\"LINKUP_API_KEY\"] = \"\"  # Fill with your API key\n",
+    "# os.environ[\"OPENAI_API_KEY\"] = \"\"  # Fill with your API key"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "72ee0c4b-9764-423a-9dbf-95129e185210",
+   "metadata": {},
+   "source": [
+    "If you want to get automated tracing from individual queries, you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a15d341e-3e26-4ca3-830b-5aab30ed66de",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",
+    "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0730d6a1-c893-4840-9817-5e5251676d5d",
+   "metadata": {},
+   "source": [
+    "### Installation\n",
+    "\n",
+    "This retriever lives in the `langchain-linkup` package:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "652d6238-1f87-422a-b135-f5abbb8652fc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -qU langchain-linkup"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a38cde65-254d-4219-a441-068766c0d4b5",
+   "metadata": {},
+   "source": [
+    "## Instantiation\n",
+    "\n",
+    "Now we can instantiate our retriever:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "70cc8e65-2a02-408a-bbc6-8ef649057d82",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_linkup import LinkupSearchRetriever\n",
+    "\n",
+    "retriever = LinkupSearchRetriever(\n",
+    "    depth=\"deep\",  # \"standard\" or \"deep\"\n",
+    "    linkup_api_key=None,  # API key can be passed here or set as the LINKUP_API_KEY environment variable\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5c5f2839-4020-424e-9fc9-07777eede442",
+   "metadata": {},
+   "source": [
+    "## Usage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "51a60dbe-9f2e-4e04-bb62-23968f17164a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(metadata={'name': 'US presidential election results 2024: Harris vs. Trump | Live maps ...', 'url': 'https://www.reuters.com/graphics/USA-ELECTION/RESULTS/zjpqnemxwvx/'}, page_content='Updated results from the 2024 election for the US president. Reuters live coverage of the 2024 US President, Senate, House and state governors races.'),\n",
+       " Document(metadata={'name': 'Election 2024: Presidential results - CNN', 'url': 'https://www.cnn.com/election/2024/results/president'}, page_content='View maps and real-time results for the 2024 US presidential election matchup between former President Donald Trump and Vice President Kamala Harris. For more ...'),\n",
+       " Document(metadata={'name': 'Presidential Election 2024 Live Results: Donald Trump wins - NBC News', 'url': 'https://www.nbcnews.com/politics/2024-elections/president-results'}, page_content='View live election results from the 2024 presidential race as Kamala Harris and Donald Trump face off. See the map of votes by state as results are tallied.'),\n",
+       " Document(metadata={'name': '2024 President Election - Live Results | RealClearPolitics', 'url': 'https://www.realclearpolitics.com/elections/live_results/2024/president/'}, page_content='Latest Election 2024 Results • President • United States • Tuesday November 3rd • Presidential Election Details'),\n",
+       " Document(metadata={'name': 'Live: Presidential Election Results 2024 : NPR', 'url': 'https://apps.npr.org/2024-election-results/'}, page_content='Presidential race ratings are based on NPR analysis. Maps do not shade in until 50% of the estimated vote is in for a given state, to mitigate flutuations in early returns . 2024 General Election Results'),\n",
+       " Document(metadata={'name': '2024 US Presidential Election Results: Live Map - Bloomberg.com', 'url': 'https://www.bloomberg.com/graphics/2024-us-election-results/'}, page_content='US Presidential Election Results November 5, 2024. Bloomberg News is reporting live election results in the presidential race between Democratic Vice President Kamala Harris and her Republican ...'),\n",
+       " Document(metadata={'name': 'Presidential Election Results 2024: Electoral Votes & Map by State ...', 'url': 'https://www.politico.com/2024-election/results/president/'}, page_content='Live 2024 Presidential election results, maps and electoral votes by state. POLITICO’s real-time coverage of 2024 races for President, Senate, House and Governor.'),\n",
+       " Document(metadata={'name': 'US Presidential Election Results 2024 - BBC News', 'url': 'https://www.bbc.com/news/election/2024/us/results'}, page_content='Kamala Harris of the Democrat party has 74,498,303 votes (48.3%) Donald Trump of the Republican party has 76,989,499 votes (49.9%) This map of the US states was filled in as presidential results ...'),\n",
+       " Document(metadata={'name': 'Election Results 2024: Live Map - Races by State - POLITICO', 'url': 'https://www.politico.com/2024-election/results/'}, page_content='Live 2024 election results and maps by state. POLITICO’s real-time coverage of 2024 races for President, Senate, House and Governor.'),\n",
+       " Document(metadata={'name': '2024 U.S. Presidential Election: Live Results and Maps - USA TODAY', 'url': 'https://www.usatoday.com/elections/results/2024-11-05/president'}, page_content='See who is winning in the Nov. 5, 2024 U.S. Presidential election nationwide with real-time results and state-by-state maps.'),\n",
+       " Document(metadata={'name': 'Presidential Election 2024 Live Results: Donald Trump winsNBC News LogoSearchSearchNBC News LogoMSNBC LogoToday Logo', 'url': 'https://www.nbcnews.com/politics/2024-elections/president-results'}, page_content=\"Profile\\n\\nSections\\n\\nLocal\\n\\ntv\\n\\nFeatured\\n\\nMore From NBC\\n\\nFollow NBC News\\n\\nnews Alerts\\n\\nThere are no new alerts at this time\\n\\n2024 President Results: Trump wins\\n==================================\\n\\nDonald Trump has secured more than the 270 Electoral College votes needed to secure the presidency, NBC News projects.\\n\\nRaces to watch\\n--------------\\n\\nAll Presidential races\\n----------------------\\n\\nElection Night Coverage\\n-----------------------\\n\\n### China competition should be top priority for Trump, Sullivan says, as Biden and Xi prepare for final meeting\\n\\n### Jim Himes says 'truth and analysis are not what drive’ Gabbard and Gaetz\\n\\n### Trump praises RFK Jr. in Mar-a-Lago remarks\\n\\n### Trump announces North Dakota Gov. Doug Burgum as his pick for interior secretary\\n\\n### House Ethics Committee cancels meeting at which Gaetz probe was on the agenda\\n\\n### Trump picks former Rep. Doug Collins for veterans affairs secretary\\n\\n### Trump to nominate his criminal defense lawyer for deputy attorney general\\n\\n### From ‘brilliant’ to ‘dangerous’: Mixed reactions roll in after Trump picks RFK Jr. for top health post\\n\\n### Donald Trump Jr. says he played key role in RFK Jr., Tulsi Gabbard picks\\n\\n### Jared Polis offers surprising words of support for RFK Jr. pick for HHS secretary\\n\\nNational early voting\\n---------------------\\n\\n### 88,233,886 mail-in and early in-person votes cast nationally\\n\\n### 65,676,748 mail-in and early in-person votes requested nationally\\n\\nPast Presidential Elections\\n---------------------------\\n\\n### Vote Margin by State in the 2020 Presidential Election\\n\\nCircle size represents the number electoral votes in that state.\\n\\nThe expected vote is the total number of votes that are expected in a given race once all votes are counted. This number is an estimate and is based on several different factors, including information on the number of votes cast early as well as information provided to our vote reporters on Election Day from county election officials. The figure can change as NBC News gathers new information.\\n\\n**Source**: [National Election Pool (NEP)](https://www.nbcnews.com/politics/2024-elections/how-election-data-is-collected )\\n\\n2024 election results\\n---------------------\\n\\nElection Night Coverage\\n-----------------------\\n\\n### China competition should be top priority for Trump, Sullivan says, as Biden and Xi prepare for final meeting\\n\\n### Jim Himes says 'truth and analysis are not what drive’ Gabbard and Gaetz\\n\\n### Trump praises RFK Jr. in Mar-a-Lago remarks\\n\\n©\\xa02024 NBCUniversal Media, LLC\")]"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "query = \"Who won the latest US presidential elections?\"\n",
+    "\n",
+    "retriever.invoke(query)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dfe8aad4-8626-4330-98a9-7ea1ca5d2e0e",
+   "metadata": {},
+   "source": [
+    "## Use within a chain\n",
+    "\n",
+    "Like other retrievers, LinkupSearchRetriever can be incorporated into LLM applications via [chains](/docs/how_to/sequence/).\n",
+    "\n",
+    "We will need a LLM or chat model:\n",
+    "\n",
+    "```{=mdx}\n",
+    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
+    "\n",
+    "<ChatModelTabs customVarName=\"llm\" />\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "25b647a3-f8f2-4541-a289-7a241e43f9df",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# | output: false\n",
+    "# | echo: false\n",
+    "\n",
+    "from langchain_openai import ChatOpenAI\n",
+    "\n",
+    "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "23e11cc9-abd6-4855-a7eb-799f45ca01ae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_core.output_parsers import StrOutputParser\n",
+    "from langchain_core.prompts import ChatPromptTemplate\n",
+    "from langchain_core.runnables import RunnablePassthrough\n",
+    "\n",
+    "prompt = ChatPromptTemplate.from_template(\n",
+    "    \"\"\"Answer the question based only on the context provided.\n",
+    "\n",
+    "Context: {context}\n",
+    "\n",
+    "Question: {question}\"\"\"\n",
+    ")\n",
+    "\n",
+    "\n",
+    "def format_docs(docs):\n",
+    "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
+    "\n",
+    "\n",
+    "chain = (\n",
+    "    {\"context\": retriever | format_docs, \"question\": RunnablePassthrough()}\n",
+    "    | prompt\n",
+    "    | llm\n",
+    "    | StrOutputParser()\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "d47c37dd-5c11-416c-a3b6-bec413cd70e8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The 3 latest US presidential elections were won by Joe Biden in 2020, Donald Trump in 2016, and Barack Obama in 2012.'"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.invoke(\"Who won the 3 latest US presidential elections?\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",
+   "metadata": {},
+   "source": [
+    "## API reference\n",
+    "\n",
+    "For detailed documentation of all LinkupSearchRetriever features and configurations head to the [API reference](https://python.langchain.com/api_reference/linkup/retrievers/linkup_langchain.search_retriever.LinkupSearchRetriever.html)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/docs/integrations/retrievers/self_query/activeloop_deeplake_self_query.ipynb

@@ -194,7 +194,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/astradb.ipynb

@@ -146,7 +146,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/chroma_self_query.ipynb

@@ -164,7 +164,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/dashvector.ipynb

@@ -185,7 +185,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_community.llms import Tongyi\n",
     "\n",

docs/docs/integrations/retrievers/self_query/databricks_vector_search.ipynb

@@ -282,7 +282,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/dingo.ipynb

@@ -196,7 +196,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/hanavector_self_query.ipynb

@@ -125,7 +125,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_community.query_constructors.hanavector import HanaTranslator\n",
     "from langchain_openai import ChatOpenAI\n",

docs/docs/integrations/retrievers/self_query/milvus_self_query.ipynb

@@ -119,7 +119,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/mongodb_atlas.ipynb

@@ -160,7 +160,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/myscale_self_query.ipynb

@@ -165,7 +165,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import ChatOpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/neo4j_self_query.ipynb

@@ -168,7 +168,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/opensearch_self_query.ipynb

@@ -135,7 +135,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/pgvector_self_query.ipynb

@@ -141,7 +141,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/pinecone.ipynb

@@ -190,7 +190,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/qdrant_self_query.ipynb

@@ -144,7 +144,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/redis_self_query.ipynb

@@ -194,7 +194,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/supabase_self_query.ipynb

@@ -308,7 +308,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/tencentvectordb.ipynb

@@ -218,7 +218,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import ChatOpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/timescalevector_self_query.ipynb

@@ -249,7 +249,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/self_query/vectara_self_query.ipynb

@@ -91,7 +91,7 @@
     "os.environ[\"VECTARA_CORPUS_ID\"] = \"<YOUR_VECTARA_CORPUS_ID>\"\n",
     "os.environ[\"VECTARA_CUSTOMER_ID\"] = \"<YOUR_VECTARA_CUSTOMER_ID>\"\n",
     "\n",
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_community.vectorstores import Vectara\n",
     "from langchain_openai.chat_models import ChatOpenAI"

docs/docs/integrations/retrievers/self_query/weaviate_self_query.ipynb

@@ -115,7 +115,7 @@
    },
    "outputs": [],
    "source": [
-    "from langchain.chains.query_constructor.base import AttributeInfo\n",
+    "from langchain.chains.query_constructor.schema import AttributeInfo\n",
     "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
     "from langchain_openai import OpenAI\n",
     "\n",

docs/docs/integrations/retrievers/weaviate-hybrid.ipynb

@@ -1,297 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "ce0f17b9",
-   "metadata": {},
-   "source": [
-    "# Weaviate Hybrid Search\n",
-    "\n",
-    ">[Weaviate](https://weaviate.io/developers/weaviate) is an open-source vector database.\n",
-    "\n",
-    ">[Hybrid search](https://weaviate.io/blog/hybrid-search-explained) is a technique that combines multiple search algorithms to improve the accuracy and relevance of search results. It uses the best features of both keyword-based search algorithms with vector search techniques.\n",
-    "\n",
-    ">The `Hybrid search in Weaviate` uses sparse and dense vectors to represent the meaning and context of search queries and documents.\n",
-    "\n",
-    "This notebook shows how to use `Weaviate hybrid search` as a LangChain retriever."
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "c307b082",
-   "metadata": {},
-   "source": [
-    "Set up the retriever:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "bba863a2-977c-4add-b5f4-bfc33a80eae5",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "%pip install --upgrade --quiet  weaviate-client"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "c10dd962",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "\n",
-    "import weaviate\n",
-    "\n",
-    "WEAVIATE_URL = os.getenv(\"WEAVIATE_URL\")\n",
-    "auth_client_secret = (weaviate.AuthApiKey(api_key=os.getenv(\"WEAVIATE_API_KEY\")),)\n",
-    "client = weaviate.Client(\n",
-    "    url=WEAVIATE_URL,\n",
-    "    additional_headers={\n",
-    "        \"X-Openai-Api-Key\": os.getenv(\"OPENAI_API_KEY\"),\n",
-    "    },\n",
-    ")\n",
-    "\n",
-    "# client.schema.delete_all()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "f47a2bfe",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from langchain_community.retrievers import (\n",
-    "    WeaviateHybridSearchRetriever,\n",
-    ")\n",
-    "from langchain_core.documents import Document"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "f2eff08e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "retriever = WeaviateHybridSearchRetriever(\n",
-    "    client=client,\n",
-    "    index_name=\"LangChain\",\n",
-    "    text_key=\"text\",\n",
-    "    attributes=[],\n",
-    "    create_schema_if_missing=True,\n",
-    ")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "b68debff",
-   "metadata": {},
-   "source": [
-    "Add some data:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "cd8a7b17",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "docs = [\n",
-    "    Document(\n",
-    "        metadata={\n",
-    "            \"title\": \"Embracing The Future: AI Unveiled\",\n",
-    "            \"author\": \"Dr. Rebecca Simmons\",\n",
-    "        },\n",
-    "        page_content=\"A comprehensive analysis of the evolution of artificial intelligence, from its inception to its future prospects. Dr. Simmons covers ethical considerations, potentials, and threats posed by AI.\",\n",
-    "    ),\n",
-    "    Document(\n",
-    "        metadata={\n",
-    "            \"title\": \"Symbiosis: Harmonizing Humans and AI\",\n",
-    "            \"author\": \"Prof. Jonathan K. Sterling\",\n",
-    "        },\n",
-    "        page_content=\"Prof. Sterling explores the potential for harmonious coexistence between humans and artificial intelligence. The book discusses how AI can be integrated into society in a beneficial and non-disruptive manner.\",\n",
-    "    ),\n",
-    "    Document(\n",
-    "        metadata={\"title\": \"AI: The Ethical Quandary\", \"author\": \"Dr. Rebecca Simmons\"},\n",
-    "        page_content=\"In her second book, Dr. Simmons delves deeper into the ethical considerations surrounding AI development and deployment. It is an eye-opening examination of the dilemmas faced by developers, policymakers, and society at large.\",\n",
-    "    ),\n",
-    "    Document(\n",
-    "        metadata={\n",
-    "            \"title\": \"Conscious Constructs: The Search for AI Sentience\",\n",
-    "            \"author\": \"Dr. Samuel Cortez\",\n",
-    "        },\n",
-    "        page_content=\"Dr. Cortez takes readers on a journey exploring the controversial topic of AI consciousness. The book provides compelling arguments for and against the possibility of true AI sentience.\",\n",
-    "    ),\n",
-    "    Document(\n",
-    "        metadata={\n",
-    "            \"title\": \"Invisible Routines: Hidden AI in Everyday Life\",\n",
-    "            \"author\": \"Prof. Jonathan K. Sterling\",\n",
-    "        },\n",
-    "        page_content=\"In his follow-up to 'Symbiosis', Prof. Sterling takes a look at the subtle, unnoticed presence and influence of AI in our everyday lives. It reveals how AI has become woven into our routines, often without our explicit realization.\",\n",
-    "    ),\n",
-    "]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "3c5970db",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['3a27b0a5-8dbb-4fee-9eba-8b6bc2c252be',\n",
-       " 'eeb9fd9b-a3ac-4d60-a55b-a63a25d3b907',\n",
-       " '7ebbdae7-1061-445f-a046-1989f2343d8f',\n",
-       " 'c2ab315b-3cab-467f-b23a-b26ed186318d',\n",
-       " 'b83765f2-e5d2-471f-8c02-c3350ade4c4f']"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever.add_documents(docs)"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "6e030694",
-   "metadata": {},
-   "source": [
-    "Do a hybrid search:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "bf7dbb98",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='In her second book, Dr. Simmons delves deeper into the ethical considerations surrounding AI development and deployment. It is an eye-opening examination of the dilemmas faced by developers, policymakers, and society at large.', metadata={}),\n",
-       " Document(page_content='A comprehensive analysis of the evolution of artificial intelligence, from its inception to its future prospects. Dr. Simmons covers ethical considerations, potentials, and threats posed by AI.', metadata={}),\n",
-       " Document(page_content=\"In his follow-up to 'Symbiosis', Prof. Sterling takes a look at the subtle, unnoticed presence and influence of AI in our everyday lives. It reveals how AI has become woven into our routines, often without our explicit realization.\", metadata={}),\n",
-       " Document(page_content='Prof. Sterling explores the potential for harmonious coexistence between humans and artificial intelligence. The book discusses how AI can be integrated into society in a beneficial and non-disruptive manner.', metadata={})]"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever.invoke(\"the ethical implications of AI\")"
-   ]
-  },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "d0c5bb4d",
-   "metadata": {},
-   "source": [
-    "Do a hybrid search with where filter:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "b2bc87c1",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='Prof. Sterling explores the potential for harmonious coexistence between humans and artificial intelligence. The book discusses how AI can be integrated into society in a beneficial and non-disruptive manner.', metadata={}),\n",
-       " Document(page_content=\"In his follow-up to 'Symbiosis', Prof. Sterling takes a look at the subtle, unnoticed presence and influence of AI in our everyday lives. It reveals how AI has become woven into our routines, often without our explicit realization.\", metadata={})]"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever.invoke(\n",
-    "    \"AI integration in society\",\n",
-    "    where_filter={\n",
-    "        \"path\": [\"author\"],\n",
-    "        \"operator\": \"Equal\",\n",
-    "        \"valueString\": \"Prof. Jonathan K. Sterling\",\n",
-    "    },\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "5ae2899e",
-   "metadata": {},
-   "source": [
-    "Do a hybrid search with scores:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "4fffd0af",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[Document(page_content='Prof. Sterling explores the potential for harmonious coexistence between humans and artificial intelligence. The book discusses how AI can be integrated into society in a beneficial and non-disruptive manner.', metadata={'_additional': {'explainScore': '(bm25)\\n(hybrid) Document eeb9fd9b-a3ac-4d60-a55b-a63a25d3b907 contributed 0.00819672131147541 to the score\\n(hybrid) Document eeb9fd9b-a3ac-4d60-a55b-a63a25d3b907 contributed 0.00819672131147541 to the score', 'score': '0.016393442'}}),\n",
-       " Document(page_content=\"In his follow-up to 'Symbiosis', Prof. Sterling takes a look at the subtle, unnoticed presence and influence of AI in our everyday lives. It reveals how AI has become woven into our routines, often without our explicit realization.\", metadata={'_additional': {'explainScore': '(bm25)\\n(hybrid) Document b83765f2-e5d2-471f-8c02-c3350ade4c4f contributed 0.0078125 to the score\\n(hybrid) Document b83765f2-e5d2-471f-8c02-c3350ade4c4f contributed 0.008064516129032258 to the score', 'score': '0.015877016'}}),\n",
-       " Document(page_content='In her second book, Dr. Simmons delves deeper into the ethical considerations surrounding AI development and deployment. It is an eye-opening examination of the dilemmas faced by developers, policymakers, and society at large.', metadata={'_additional': {'explainScore': '(bm25)\\n(hybrid) Document 7ebbdae7-1061-445f-a046-1989f2343d8f contributed 0.008064516129032258 to the score\\n(hybrid) Document 7ebbdae7-1061-445f-a046-1989f2343d8f contributed 0.0078125 to the score', 'score': '0.015877016'}}),\n",
-       " Document(page_content='A comprehensive analysis of the evolution of artificial intelligence, from its inception to its future prospects. Dr. Simmons covers ethical considerations, potentials, and threats posed by AI.', metadata={'_additional': {'explainScore': '(vector) [-0.0071824766 -0.0006682752 0.001723625 -0.01897258 -0.0045127636 0.0024410256 -0.020503938 0.013768672 0.009520169 -0.037972264]...  \\n(hybrid) Document 3a27b0a5-8dbb-4fee-9eba-8b6bc2c252be contributed 0.007936507936507936 to the score', 'score': '0.007936508'}})]"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "retriever.invoke(\n",
-    "    \"AI integration in society\",\n",
-    "    score=True,\n",
-    ")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/docs/integrations/text_embedding/ibm_watsonx.ipynb

@@ -327,7 +327,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "langchain",
+   "display_name": "langchain_ibm",
    "language": "python",
    "name": "python3"
   },

docs/docs/integrations/text_embedding/localai.ipynb

@@ -8,22 +8,40 @@
    "source": [
     "# LocalAI\n",
     "\n",
+    ":::info\n",
+    "\n",
+    "`langchain-localai` is a 3rd party integration package for LocalAI. It provides a simple way to use LocalAI services in Langchain.\n",
+    "\n",
+    "The source code is available on [Github](https://github.com/mkhludnev/langchain-localai)\n",
+    "\n",
+    ":::\n",
+    "\n",
     "Let's load the LocalAI Embedding class. In order to use the LocalAI Embedding class, you need to have the LocalAI service hosted somewhere and configure the embedding models. See the documentation at https://localai.io/basics/getting_started/index.html and https://localai.io/features/embeddings/index.html."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "id": "0be1af71",
+   "execution_count": null,
+   "id": "799d1f77",
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.embeddings import LocalAIEmbeddings"
+    "%pip install -U langchain-localai"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 2,
+   "id": "0be1af71",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_localai import LocalAIEmbeddings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
    "id": "2c66e5da",
    "metadata": {},
    "outputs": [],
@@ -35,7 +53,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "id": "01370375",
    "metadata": {},
    "outputs": [],
@@ -45,7 +63,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "bfb6142c",
    "metadata": {},
    "outputs": [],
@@ -140,7 +158,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": ".venv",
    "language": "python",
    "name": "python3"
   },
@@ -154,12 +172,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "e971737741ff4ec9aff7dc6155a1060a59a8a6d52c757dbbe66bf8ee389494b1"
-   }
+   "version": "3.11.4"
   }
  },
  "nbformat": 4,

docs/docs/integrations/text_embedding/model2vec.ipynb

@@ -0,0 +1,201 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e8712110",
+   "metadata": {},
+   "source": [
+    "## Overview\n",
+    "\n",
+    "Model2Vec is a technique to turn any sentence transformer into a really small static model\n",
+    "[model2vec](https://github.com/MinishLab/model2vec) can be used to generate embeddings."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "266dd424",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "```bash\n",
+    "pip install -U langchain-community\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "78ab91a6",
+   "metadata": {},
+   "source": [
+    "## Instantiation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d06e7719",
+   "metadata": {},
+   "source": [
+    "Ensure that `model2vec` is installed\n",
+    "\n",
+    "```bash\n",
+    "pip install -U model2vec\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f8ea1ed5",
+   "metadata": {},
+   "source": [
+    "## Indexing and Retrieval"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "d25dc22d-b656-46c6-a42d-eace958590cd",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2023-05-24T15:13:17.176956Z",
+     "start_time": "2023-05-24T15:13:15.399076Z"
+    },
+    "execution": {
+     "iopub.execute_input": "2024-03-29T15:39:19.252281Z",
+     "iopub.status.busy": "2024-03-29T15:39:19.252101Z",
+     "iopub.status.idle": "2024-03-29T15:39:19.339106Z",
+     "shell.execute_reply": "2024-03-29T15:39:19.338614Z",
+     "shell.execute_reply.started": "2024-03-29T15:39:19.252260Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from langchain_community.embeddings import Model2vecEmbeddings"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "8397b91f-a1f9-4be6-a699-fedaada7c37a",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2023-05-24T15:13:17.193751Z",
+     "start_time": "2023-05-24T15:13:17.182053Z"
+    },
+    "execution": {
+     "iopub.execute_input": "2024-03-29T15:39:19.901573Z",
+     "iopub.status.busy": "2024-03-29T15:39:19.900935Z",
+     "iopub.status.idle": "2024-03-29T15:39:19.906540Z",
+     "shell.execute_reply": "2024-03-29T15:39:19.905345Z",
+     "shell.execute_reply.started": "2024-03-29T15:39:19.901529Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "embeddings = Model2vecEmbeddings(\"minishlab/potion-base-8M\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "abcf98b7-424c-4691-a1cd-862c3d53be11",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2023-05-24T15:13:17.844903Z",
+     "start_time": "2023-05-24T15:13:17.198751Z"
+    },
+    "execution": {
+     "iopub.execute_input": "2024-03-29T15:39:20.434581Z",
+     "iopub.status.busy": "2024-03-29T15:39:20.433117Z",
+     "iopub.status.idle": "2024-03-29T15:39:22.178650Z",
+     "shell.execute_reply": "2024-03-29T15:39:22.176058Z",
+     "shell.execute_reply.started": "2024-03-29T15:39:20.434501Z"
+    },
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "query_text = \"This is a test query.\"\n",
+    "query_result = embeddings.embed_query(query_text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "98897454-b280-4ee1-bbb9-2c6c15342f87",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2023-05-24T15:13:18.605339Z",
+     "start_time": "2023-05-24T15:13:17.845906Z"
+    },
+    "execution": {
+     "iopub.execute_input": "2024-03-29T15:39:28.164009Z",
+     "iopub.status.busy": "2024-03-29T15:39:28.161759Z",
+     "iopub.status.idle": "2024-03-29T15:39:30.217232Z",
+     "shell.execute_reply": "2024-03-29T15:39:30.215348Z",
+     "shell.execute_reply.started": "2024-03-29T15:39:28.163876Z"
+    },
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "document_text = \"This is a test document.\"\n",
+    "document_result = embeddings.embed_documents([document_text])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11bac134",
+   "metadata": {},
+   "source": [
+    "## Direct Usage\n",
+    "\n",
+    "Here's how you would directly make use of `model2vec`\n",
+    "\n",
+    "```python\n",
+    "from model2vec import StaticModel\n",
+    "\n",
+    "# Load a model from the HuggingFace hub (in this case the potion-base-8M model)\n",
+    "model = StaticModel.from_pretrained(\"minishlab/potion-base-8M\")\n",
+    "\n",
+    "# Make embeddings\n",
+    "embeddings = model.encode([\"It's dangerous to go alone!\", \"It's a secret to everybody.\"])\n",
+    "\n",
+    "# Make sequences of token embeddings\n",
+    "token_embeddings = model.encode_as_sequence([\"It's dangerous to go alone!\", \"It's a secret to everybody.\"])\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d81e21aa",
+   "metadata": {},
+   "source": [
+    "## API Reference\n",
+    "\n",
+    "For more information check out the model2vec github [repo](https://github.com/MinishLab/model2vec)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/docs/integrations/tools/scrapegraph.ipynb

@@ -0,0 +1,380 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "id": "10238e62-3465-4973-9279-606cbb7ccf16",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: ScrapeGraph\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a6f91f20",
+   "metadata": {},
+   "source": [
+    "# ScrapeGraph\n",
+    "\n",
+    "This notebook provides a quick overview for getting started with ScrapeGraph [tools](/docs/integrations/tools/). For detailed documentation of all ScrapeGraph features and configurations head to the [API reference](https://python.langchain.com/docs/integrations/tools/scrapegraph).\n",
+    "\n",
+    "For more information about ScrapeGraph AI:\n",
+    "- [ScrapeGraph AI Website](https://scrapegraphai.com)\n",
+    "- [Open Source Project](https://github.com/ScrapeGraphAI/Scrapegraph-ai)\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "### Integration details\n",
+    "\n",
+    "| Class | Package | Serializable | JS support | Package latest |\n",
+    "| :--- | :--- | :---: | :---: | :---: |\n",
+    "| [SmartScraperTool](https://python.langchain.com/docs/integrations/tools/scrapegraph) | langchain-scrapegraph | ✅ | ❌ | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-scrapegraph?style=flat-square&label=%20) |\n",
+    "| [MarkdownifyTool](https://python.langchain.com/docs/integrations/tools/scrapegraph) | langchain-scrapegraph | ✅ | ❌ | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-scrapegraph?style=flat-square&label=%20) |\n",
+    "| [LocalScraperTool](https://python.langchain.com/docs/integrations/tools/scrapegraph) | langchain-scrapegraph | ✅ | ❌ | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-scrapegraph?style=flat-square&label=%20) |\n",
+    "| [GetCreditsTool](https://python.langchain.com/docs/integrations/tools/scrapegraph) | langchain-scrapegraph | ✅ | ❌ | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-scrapegraph?style=flat-square&label=%20) |\n",
+    "\n",
+    "### Tool features\n",
+    "\n",
+    "| Tool | Purpose | Input | Output |\n",
+    "| :--- | :--- | :--- | :--- |\n",
+    "| SmartScraperTool | Extract structured data from websites | URL + prompt | JSON |\n",
+    "| MarkdownifyTool | Convert webpages to markdown | URL | Markdown text |\n",
+    "| LocalScraperTool | Extract data from HTML content | HTML + prompt | JSON |\n",
+    "| GetCreditsTool | Check API credits | None | Credit info |\n",
+    "\n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "The integration requires the following packages:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "f85b4089",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install --quiet -U langchain-scrapegraph"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b15e9266",
+   "metadata": {},
+   "source": [
+    "### Credentials\n",
+    "\n",
+    "You'll need a ScrapeGraph AI API key to use these tools. Get one at [scrapegraphai.com](https://scrapegraphai.com)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "e0b178a2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import getpass\n",
+    "import os\n",
+    "\n",
+    "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+    "    os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"ScrapeGraph AI API key:\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bc5ab717",
+   "metadata": {},
+   "source": [
+    "It's also helpful (but not needed) to set up [LangSmith](https://smith.langchain.com/) for best-in-class observability:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a6c2f136",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
+    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c97218f",
+   "metadata": {},
+   "source": [
+    "## Instantiation\n",
+    "\n",
+    "Here we show how to instantiate instances of the ScrapeGraph tools:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "8b3ddfe9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_scrapegraph.tools import (\n",
+    "    GetCreditsTool,\n",
+    "    LocalScraperTool,\n",
+    "    MarkdownifyTool,\n",
+    "    SmartScraperTool,\n",
+    ")\n",
+    "\n",
+    "smartscraper = SmartScraperTool()\n",
+    "markdownify = MarkdownifyTool()\n",
+    "localscraper = LocalScraperTool()\n",
+    "credits = GetCreditsTool()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "74147a1a",
+   "metadata": {},
+   "source": [
+    "## Invocation\n",
+    "\n",
+    "### [Invoke directly with args](/docs/concepts/tools)\n",
+    "\n",
+    "Let's try each tool individually:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "65310a8b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "SmartScraper Result: {'company_name': 'ScrapeGraphAI', 'description': \"ScrapeGraphAI is a powerful AI web scraping tool that turns entire websites into clean, structured data through a simple API. It's designed to help developers and AI companies extract valuable data from websites efficiently and transform it into formats that are ready for use in LLM applications and data analysis.\"}\n",
+      "\n",
+      "Markdownify Result (first 200 chars): [![ScrapeGraphAI Logo](https://scrapegraphai.com/images/scrapegraphai_logo.svg)ScrapeGraphAI](https://scrapegraphai.com/)\n",
+      "\n",
+      "PartnersPricingFAQ[Blog](https://scrapegraphai.com/blog)DocsLog inSign up\n",
+      "\n",
+      "Op\n",
+      "LocalScraper Result: {'company_name': 'Company Name', 'description': 'We are a technology company focused on AI solutions.', 'contact': {'email': 'contact@example.com', 'phone': '(555) 123-4567'}}\n",
+      "\n",
+      "Credits Info: {'remaining_credits': 49679, 'total_credits_used': 914}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# SmartScraper\n",
+    "result = smartscraper.invoke(\n",
+    "    {\n",
+    "        \"user_prompt\": \"Extract the company name and description\",\n",
+    "        \"website_url\": \"https://scrapegraphai.com\",\n",
+    "    }\n",
+    ")\n",
+    "print(\"SmartScraper Result:\", result)\n",
+    "\n",
+    "# Markdownify\n",
+    "markdown = markdownify.invoke({\"website_url\": \"https://scrapegraphai.com\"})\n",
+    "print(\"\\nMarkdownify Result (first 200 chars):\", markdown[:200])\n",
+    "\n",
+    "local_html = \"\"\"\n",
+    "<html>\n",
+    "    <body>\n",
+    "        <h1>Company Name</h1>\n",
+    "        <p>We are a technology company focused on AI solutions.</p>\n",
+    "        <div class=\"contact\">\n",
+    "            <p>Email: contact@example.com</p>\n",
+    "            <p>Phone: (555) 123-4567</p>\n",
+    "        </div>\n",
+    "    </body>\n",
+    "</html>\n",
+    "\"\"\"\n",
+    "\n",
+    "# LocalScraper\n",
+    "result_local = localscraper.invoke(\n",
+    "    {\n",
+    "        \"user_prompt\": \"Make a summary of the webpage and extract the email and phone number\",\n",
+    "        \"website_html\": local_html,\n",
+    "    }\n",
+    ")\n",
+    "print(\"LocalScraper Result:\", result_local)\n",
+    "\n",
+    "# Check credits\n",
+    "credits_info = credits.invoke({})\n",
+    "print(\"\\nCredits Info:\", credits_info)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d6e73897",
+   "metadata": {},
+   "source": [
+    "### [Invoke with ToolCall](/docs/concepts/tools)\n",
+    "\n",
+    "We can also invoke the tool with a model-generated ToolCall:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "f90e33a7",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "ToolMessage(content='{\"main_heading\": \"Get the data you need from any website\", \"description\": \"Easily extract and gather information with just a few lines of code with a simple api. Turn websites into clean and usable structured data.\"}', name='SmartScraper', tool_call_id='1')"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model_generated_tool_call = {\n",
+    "    \"args\": {\n",
+    "        \"user_prompt\": \"Extract the main heading and description\",\n",
+    "        \"website_url\": \"https://scrapegraphai.com\",\n",
+    "    },\n",
+    "    \"id\": \"1\",\n",
+    "    \"name\": smartscraper.name,\n",
+    "    \"type\": \"tool_call\",\n",
+    "}\n",
+    "smartscraper.invoke(model_generated_tool_call)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "659f9fbd",
+   "metadata": {},
+   "source": [
+    "## Chaining\n",
+    "\n",
+    "Let's use our tools with an LLM to analyze a website:\n",
+    "\n",
+    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
+    "\n",
+    "<ChatModelTabs customVarName=\"llm\" />"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "af3123ad",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# | output: false\n",
+    "# | echo: false\n",
+    "\n",
+    "# %pip install -qU langchain langchain-openai\n",
+    "from langchain.chat_models import init_chat_model\n",
+    "\n",
+    "llm = init_chat_model(model=\"gpt-4o\", model_provider=\"openai\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "fdbf35b5",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content='ScrapeGraph AI is an AI-powered web scraping tool that efficiently extracts and converts website data into structured formats via a simple API. It caters to developers, data scientists, and AI researchers, offering features like easy integration, support for dynamic content, and scalability for large projects. It supports various website types, including business, e-commerce, and educational sites. Contact: contact@scrapegraphai.com.', additional_kwargs={'tool_calls': [{'id': 'call_shkRPyjyAtfjH9ffG5rSy9xj', 'function': {'arguments': '{\"user_prompt\":\"Extract details about the products, services, and key features offered by ScrapeGraph AI, as well as any unique selling points or innovations mentioned on the website.\",\"website_url\":\"https://scrapegraphai.com\"}', 'name': 'SmartScraper'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 47, 'prompt_tokens': 480, 'total_tokens': 527, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_c7ca0ebaca', 'finish_reason': 'stop', 'logprobs': None}, id='run-45a12c86-d499-4273-8c59-0db926799bc7-0', tool_calls=[{'name': 'SmartScraper', 'args': {'user_prompt': 'Extract details about the products, services, and key features offered by ScrapeGraph AI, as well as any unique selling points or innovations mentioned on the website.', 'website_url': 'https://scrapegraphai.com'}, 'id': 'call_shkRPyjyAtfjH9ffG5rSy9xj', 'type': 'tool_call'}], usage_metadata={'input_tokens': 480, 'output_tokens': 47, 'total_tokens': 527, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain_core.prompts import ChatPromptTemplate\n",
+    "from langchain_core.runnables import RunnableConfig, chain\n",
+    "\n",
+    "prompt = ChatPromptTemplate(\n",
+    "    [\n",
+    "        (\n",
+    "            \"system\",\n",
+    "            \"You are a helpful assistant that can use tools to extract structured information from websites.\",\n",
+    "        ),\n",
+    "        (\"human\", \"{user_input}\"),\n",
+    "        (\"placeholder\", \"{messages}\"),\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "llm_with_tools = llm.bind_tools([smartscraper], tool_choice=smartscraper.name)\n",
+    "llm_chain = prompt | llm_with_tools\n",
+    "\n",
+    "\n",
+    "@chain\n",
+    "def tool_chain(user_input: str, config: RunnableConfig):\n",
+    "    input_ = {\"user_input\": user_input}\n",
+    "    ai_msg = llm_chain.invoke(input_, config=config)\n",
+    "    tool_msgs = smartscraper.batch(ai_msg.tool_calls, config=config)\n",
+    "    return llm_chain.invoke({**input_, \"messages\": [ai_msg, *tool_msgs]}, config=config)\n",
+    "\n",
+    "\n",
+    "tool_chain.invoke(\n",
+    "    \"What does ScrapeGraph AI do? Extract this information from their website https://scrapegraphai.com\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ac8146c",
+   "metadata": {},
+   "source": [
+    "## API reference\n",
+    "\n",
+    "For detailed documentation of all ScrapeGraph features and configurations head to the Langchain API reference: https://python.langchain.com/docs/integrations/tools/scrapegraph\n",
+    "\n",
+    "Or to the official SDK repo: https://github.com/ScrapeGraphAI/langchain-scrapegraph"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/docs/integrations/tools/sql_database.ipynb

@@ -192,10 +192,10 @@
     {
      "data": {
       "text/plain": [
-       "[QuerySQLDataBaseTool(description=\"Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields.\", db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x105e02860>),\n",
-       " InfoSQLDatabaseTool(description='Input to this tool is a comma-separated list of tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist by calling sql_db_list_tables first! Example Input: table1, table2, table3', db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x105e02860>),\n",
-       " ListSQLDatabaseTool(db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x105e02860>),\n",
-       " QuerySQLCheckerTool(description='Use this tool to double check if your query is correct before executing it. Always use this tool before executing a query with sql_db_query!', db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x105e02860>, llm=ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x1148a97b0>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x1148aaec0>, temperature=0.0, openai_api_key=SecretStr('**********'), openai_proxy=''), llm_chain=LLMChain(prompt=PromptTemplate(input_variables=['dialect', 'query'], template='\\n{query}\\nDouble check the {dialect} query above for common mistakes, including:\\n- Using NOT IN with NULL values\\n- Using UNION when UNION ALL should have been used\\n- Using BETWEEN for exclusive ranges\\n- Data type mismatch in predicates\\n- Properly quoting identifiers\\n- Using the correct number of arguments for functions\\n- Casting to the correct data type\\n- Using the proper columns for joins\\n\\nIf there are any of the above mistakes, rewrite the query. If there are no mistakes, just reproduce the original query.\\n\\nOutput the final SQL query only.\\n\\nSQL Query: '), llm=ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x1148a97b0>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x1148aaec0>, temperature=0.0, openai_api_key=SecretStr('**********'), openai_proxy='')))]"
+       "[QuerySQLDatabaseTool(description=\"Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields.\", db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x103d5fa60>),\n",
+       " InfoSQLDatabaseTool(description='Input to this tool is a comma-separated list of tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist by calling sql_db_list_tables first! Example Input: table1, table2, table3', db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x103d5fa60>),\n",
+       " ListSQLDatabaseTool(db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x103d5fa60>),\n",
+       " QuerySQLCheckerTool(description='Use this tool to double check if your query is correct before executing it. Always use this tool before executing a query with sql_db_query!', db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x103d5fa60>, llm=ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x10742d720>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x10742f7f0>, root_client=<openai.OpenAI object at 0x103d5fac0>, root_async_client=<openai.AsyncOpenAI object at 0x10742d780>, temperature=0.0, model_kwargs={}, openai_api_key=SecretStr('**********')), llm_chain=LLMChain(verbose=False, prompt=PromptTemplate(input_variables=['dialect', 'query'], input_types={}, partial_variables={}, template='\\n{query}\\nDouble check the {dialect} query above for common mistakes, including:\\n- Using NOT IN with NULL values\\n- Using UNION when UNION ALL should have been used\\n- Using BETWEEN for exclusive ranges\\n- Data type mismatch in predicates\\n- Properly quoting identifiers\\n- Using the correct number of arguments for functions\\n- Casting to the correct data type\\n- Using the proper columns for joins\\n\\nIf there are any of the above mistakes, rewrite the query. If there are no mistakes, just reproduce the original query.\\n\\nOutput the final SQL query only.\\n\\nSQL Query: '), llm=ChatOpenAI(client=<openai.resources.chat.completions.Completions object at 0x10742d720>, async_client=<openai.resources.chat.completions.AsyncCompletions object at 0x10742f7f0>, root_client=<openai.OpenAI object at 0x103d5fac0>, root_async_client=<openai.AsyncOpenAI object at 0x10742d780>, temperature=0.0, model_kwargs={}, openai_api_key=SecretStr('**********')), output_parser=StrOutputParser(), llm_kwargs={}))]"
       ]
      },
      "execution_count": 4,
@@ -226,7 +226,7 @@
     "    InfoSQLDatabaseTool,\n",
     "    ListSQLDatabaseTool,\n",
     "    QuerySQLCheckerTool,\n",
-    "    QuerySQLDataBaseTool,\n",
+    "    QuerySQLDatabaseTool,\n",
     ")"
    ]
   },
@@ -242,7 +242,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "id": "eda12f8b-be90-4697-ac84-2ece9e2d1708",
    "metadata": {},
    "outputs": [
@@ -265,7 +265,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "id": "3470ae96-e5e5-4717-a6d6-d7d28c7b7347",
    "metadata": {},
    "outputs": [],
@@ -283,7 +283,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 8,
    "id": "48bca92c-9b4b-4d5c-bcce-1b239c9e901c",
    "metadata": {},
    "outputs": [],
@@ -305,7 +305,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 9,
    "id": "39e6d2bf-3194-4aba-854b-63faf919157b",
    "metadata": {},
    "outputs": [
@@ -318,8 +318,8 @@
       "Which country's customers spent the most?\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_list_tables (call_eiheSxiL0s90KE50XyBnBtJY)\n",
-      " Call ID: call_eiheSxiL0s90KE50XyBnBtJY\n",
+      "  sql_db_list_tables (call_EBPjyfzqXzFutDn8BklYACLj)\n",
+      " Call ID: call_EBPjyfzqXzFutDn8BklYACLj\n",
       "  Args:\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
       "Name: sql_db_list_tables\n",
@@ -327,8 +327,8 @@
       "Album, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_schema (call_YKwGWt4UUVmxxY7vjjBDzFLJ)\n",
-      " Call ID: call_YKwGWt4UUVmxxY7vjjBDzFLJ\n",
+      "  sql_db_schema (call_kGcnKpxRVFIY8dPjYIJbRoVU)\n",
+      " Call ID: call_kGcnKpxRVFIY8dPjYIJbRoVU\n",
       "  Args:\n",
       "    table_names: Customer, Invoice, InvoiceLine\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
@@ -405,14 +405,14 @@
       "*/\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_query (call_7WBDcMxl1h7MnI05njx1q8V9)\n",
-      " Call ID: call_7WBDcMxl1h7MnI05njx1q8V9\n",
+      "  sql_db_query (call_cTfI7OrY64FzJaDd49ILFWw7)\n",
+      " Call ID: call_cTfI7OrY64FzJaDd49ILFWw7\n",
       "  Args:\n",
       "    query: SELECT c.Country, SUM(i.Total) AS TotalSpent FROM Customer c JOIN Invoice i ON c.CustomerId = i.CustomerId GROUP BY c.Country ORDER BY TotalSpent DESC LIMIT 1\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
       "Name: sql_db_query\n",
       "\n",
-      "[('USA', 523.0600000000003)]\n",
+      "[('USA', 523.06)]\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "\n",
       "Customers from the USA spent the most, with a total amount spent of $523.06.\n"
@@ -440,7 +440,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 10,
    "id": "23c1235c-6d18-43e4-98ab-85b426b53d94",
    "metadata": {},
    "outputs": [
@@ -453,8 +453,8 @@
       "Who are the top 3 best selling artists?\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_query (call_9F6Bp2vwsDkeLW6FsJFqLiet)\n",
-      " Call ID: call_9F6Bp2vwsDkeLW6FsJFqLiet\n",
+      "  sql_db_query (call_xAkvYiRFM7nCMKXsDNvk1OMx)\n",
+      " Call ID: call_xAkvYiRFM7nCMKXsDNvk1OMx\n",
       "  Args:\n",
       "    query: SELECT artist_name, SUM(quantity) AS total_sold FROM sales GROUP BY artist_name ORDER BY total_sold DESC LIMIT 3\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
@@ -465,8 +465,8 @@
       "(Background on this error at: https://sqlalche.me/e/20/e3q8)\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_list_tables (call_Gx5adzWnrBDIIxzUDzsn83zO)\n",
-      " Call ID: call_Gx5adzWnrBDIIxzUDzsn83zO\n",
+      "  sql_db_list_tables (call_K4Zvbowsq7XPgGFepbvc5G7i)\n",
+      " Call ID: call_K4Zvbowsq7XPgGFepbvc5G7i\n",
       "  Args:\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
       "Name: sql_db_list_tables\n",
@@ -474,8 +474,8 @@
       "Album, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_schema (call_ftywrZgEgGWLrnk9dYC0xtZv)\n",
-      " Call ID: call_ftywrZgEgGWLrnk9dYC0xtZv\n",
+      "  sql_db_schema (call_tUztueSK7VO2klZ99xT4ZVhM)\n",
+      " Call ID: call_tUztueSK7VO2klZ99xT4ZVhM\n",
       "  Args:\n",
       "    table_names: Artist, Album, InvoiceLine\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
@@ -534,8 +534,8 @@
       "*/\n",
       "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
       "Tool Calls:\n",
-      "  sql_db_query (call_i6n3lmS7E2ZivN758VOayTiy)\n",
-      " Call ID: call_i6n3lmS7E2ZivN758VOayTiy\n",
+      "  sql_db_query (call_tVtLQIRPmCM6pukgpHFfq86A)\n",
+      " Call ID: call_tVtLQIRPmCM6pukgpHFfq86A\n",
       "  Args:\n",
       "    query: SELECT Artist.Name AS artist_name, SUM(InvoiceLine.Quantity) AS total_sold FROM Artist JOIN Album ON Artist.ArtistId = Album.ArtistId JOIN Track ON Album.AlbumId = Track.AlbumId JOIN InvoiceLine ON Track.TrackId = InvoiceLine.TrackId GROUP BY Artist.Name ORDER BY total_sold DESC LIMIT 3\n",
       "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
@@ -614,7 +614,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.10.4"
   }
  },
  "nbformat": 4,

docs/docs/integrations/tools/wikidata.ipynb

@@ -15,7 +15,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3d9195d4",
+   "id": "d622c581",
    "metadata": {
     "vscode": {
      "languageId": "shellscript"
@@ -23,7 +23,7 @@
    },
    "outputs": [],
    "source": [
-    "%pip install --upgrade --quiet \"wikibase-rest-api-client<0.2\" mediawikiapi"
+    "%pip install --upgrade --quiet wikibase-rest-api-client mediawikiapi"
    ]
   },
   {
@@ -110,7 +110,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.5"
+   "version": "3.12.5"
   }
  },
  "nbformat": 4,

docs/docs/integrations/vectorstores/aerospike.ipynb

@@ -29,8 +29,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "PROXIMUS_HOST = \"<avs-ip>\"\n",
-    "PROXIMUS_PORT = 5000"
+    "AVS_HOST = \"<avs-ip>\"\n",
+    "AVS_PORT = 5000"
    ]
   },
   {
@@ -51,7 +51,7 @@
    },
    "outputs": [],
    "source": [
-    "!pip install --upgrade --quiet aerospike-vector-search==0.6.1 langchain-community sentence-transformers langchain"
+    "!pip install --upgrade --quiet aerospike-vector-search==3.0.1 langchain-community sentence-transformers langchain"
    ]
   },
   {
@@ -369,7 +369,7 @@
     "from langchain_community.vectorstores import Aerospike\n",
     "\n",
     "# Here we are using the AVS host and port you configured earlier\n",
-    "seed = HostPort(host=PROXIMUS_HOST, port=PROXIMUS_PORT)\n",
+    "seed = HostPort(host=AVS_HOST, port=AVS_PORT)\n",
     "\n",
     "# The namespace of where to place our vectors. This should match the vector configured in your docstore.conf file.\n",
     "NAMESPACE = \"test\"\n",
@@ -401,7 +401,7 @@
     "        vector_field=VECTOR_KEY,\n",
     "        vector_distance_metric=MODEL_DISTANCE_CALC,\n",
     "        dimensions=MODEL_DIM,\n",
-    "        index_meta_data={\n",
+    "        index_labels={\n",
     "            \"model\": \"miniLM-L6-v2\",\n",
     "            \"date\": \"05/04/2024\",\n",
     "            \"dim\": str(MODEL_DIM),\n",

docs/docs/integrations/vectorstores/azure_cosmos_db.ipynb

@@ -38,9 +38,6 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "\r\n",
-      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\r\n",
-      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n",
       "Note: you may need to restart the kernel to use updated packages.\n"
      ]
     }
@@ -74,7 +71,7 @@
    "id": "f2e66b097c6ce2e3",
    "metadata": {},
    "source": [
-    "We want to use `OpenAIEmbeddings` so we need to set up our Azure OpenAI API Key alongside other environment variables. "
+    "We want to use `AzureOpenAIEmbeddings` so we need to set up our Azure OpenAI API Key alongside other environment variables. "
    ]
   },
   {
@@ -90,15 +87,10 @@
    "outputs": [],
    "source": [
     "# Set up the OpenAI Environment Variables\n",
-    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n",
-    "os.environ[\"OPENAI_API_VERSION\"] = \"2023-05-15\"\n",
-    "os.environ[\"OPENAI_API_BASE\"] = (\n",
-    "    \"YOUR_OPEN_AI_ENDPOINT\"  # https://example.openai.azure.com/\n",
-    ")\n",
-    "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_OPENAI_API_KEY\"\n",
-    "os.environ[\"OPENAI_EMBEDDINGS_DEPLOYMENT\"] = (\n",
-    "    \"smart-agent-embedding-ada\"  # the deployment name for the embedding model\n",
-    ")\n",
+    "\n",
+    "os.environ[\"AZURE_OPENAI_API_KEY\"] = \"YOUR_AZURE_OPENAI_API_KEY\"\n",
+    "os.environ[\"AZURE_OPENAI_ENDPOINT\"] = \"YOUR_AZURE_OPENAI_ENDPOINT\"\n",
+    "os.environ[\"AZURE_OPENAI_API_VERSION\"] = \"2023-05-15\"\n",
     "os.environ[\"OPENAI_EMBEDDINGS_MODEL_NAME\"] = \"text-embedding-ada-002\"  # the model name"
    ]
   },
@@ -130,7 +122,7 @@
     "    CosmosDBSimilarityType,\n",
     "    CosmosDBVectorSearchType,\n",
     ")\n",
-    "from langchain_openai import OpenAIEmbeddings\n",
+    "from langchain_openai import AzureOpenAIEmbeddings\n",
     "from langchain_text_splitters import CharacterTextSplitter\n",
     "\n",
     "SOURCE_FILE_NAME = \"../../how_to/state_of_the_union.txt\"\n",
@@ -147,14 +139,35 @@
     "model_name = os.getenv(\"OPENAI_EMBEDDINGS_MODEL_NAME\", \"text-embedding-ada-002\")\n",
     "\n",
     "\n",
-    "openai_embeddings: OpenAIEmbeddings = OpenAIEmbeddings(\n",
-    "    deployment=model_deployment, model=model_name, chunk_size=1\n",
+    "openai_embeddings: AzureOpenAIEmbeddings = AzureOpenAIEmbeddings(\n",
+    "    model=model_name, chunk_size=1\n",
     ")"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 5,
+   "id": "f6c6ed80-7b91-4833-bab5-c9b2b5edcdec",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Document(metadata={'source': '../../how_to/state_of_the_union.txt'}, page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\nWith a duty to one another to the American people to the Constitution. \\n\\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \\n\\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \\n\\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \\n\\nHe met the Ukrainian people. \\n\\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.')"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "docs[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
    "id": "39ae6058c2f7fdf1",
    "metadata": {
     "ExecuteTime": {
@@ -166,14 +179,10 @@
     {
      "data": {
       "text/plain": [
-       "{'raw': {'defaultShard': {'numIndexesBefore': 1,\n",
-       "   'numIndexesAfter': 2,\n",
-       "   'createdCollectionAutomatically': False,\n",
-       "   'ok': 1}},\n",
-       " 'ok': 1}"
+       "'\\n# DiskANN vectorstore\\nmaxDegree = 40\\ndimensions = 1536\\nsimilarity_algorithm = CosmosDBSimilarityType.COS\\nkind = CosmosDBVectorSearchType.VECTOR_DISKANN\\nlBuild = 20\\n\\nvectorstore.create_index(\\n            dimensions=dimensions,\\n            similarity=similarity_algorithm,\\n            kind=kind ,\\n            max_degree=maxDegree,\\n            l_build=lBuild,\\n        )\\n\\n# -----------------------------------------------------------\\n\\n# HNSW vectorstore\\ndimensions = 1536\\nsimilarity_algorithm = CosmosDBSimilarityType.COS\\nkind = CosmosDBVectorSearchType.VECTOR_HNSW\\nm = 16\\nef_construction = 64\\n\\nvectorstore.create_index(\\n            dimensions=dimensions,\\n            similarity=similarity_algorithm,\\n            kind=kind ,\\n            m=m,\\n            ef_construction=ef_construction,\\n        )\\n'"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -212,12 +221,46 @@
     "\n",
     "vectorstore.create_index(\n",
     "    num_lists, dimensions, similarity_algorithm, kind, m, ef_construction\n",
-    ")"
+    ")\n",
+    "\n",
+    "\"\"\"\n",
+    "# DiskANN vectorstore\n",
+    "maxDegree = 40\n",
+    "dimensions = 1536\n",
+    "similarity_algorithm = CosmosDBSimilarityType.COS\n",
+    "kind = CosmosDBVectorSearchType.VECTOR_DISKANN\n",
+    "lBuild = 20\n",
+    "\n",
+    "vectorstore.create_index(\n",
+    "            dimensions=dimensions,\n",
+    "            similarity=similarity_algorithm,\n",
+    "            kind=kind ,\n",
+    "            max_degree=maxDegree,\n",
+    "            l_build=lBuild,\n",
+    "        )\n",
+    "\n",
+    "# -----------------------------------------------------------\n",
+    "\n",
+    "# HNSW vectorstore\n",
+    "dimensions = 1536\n",
+    "similarity_algorithm = CosmosDBSimilarityType.COS\n",
+    "kind = CosmosDBVectorSearchType.VECTOR_HNSW\n",
+    "m = 16\n",
+    "ef_construction = 64\n",
+    "\n",
+    "vectorstore.create_index(\n",
+    "            dimensions=dimensions,\n",
+    "            similarity=similarity_algorithm,\n",
+    "            kind=kind ,\n",
+    "            m=m,\n",
+    "            ef_construction=ef_construction,\n",
+    "        )\n",
+    "\"\"\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "id": "32c68d3246adc21f",
    "metadata": {
     "ExecuteTime": {
@@ -234,7 +277,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 8,
    "id": "8feeeb4364efb204",
    "metadata": {
     "ExecuteTime": {
@@ -271,7 +314,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 9,
    "id": "3c218ab6f59301f7",
    "metadata": {
     "ExecuteTime": {
@@ -308,7 +351,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 10,
    "id": "fd67e4d92c9ab32f",
    "metadata": {
     "ExecuteTime": {
@@ -352,10 +395,106 @@
     "Azure Cosmos DB for MongoDB supports pre-filtering with $lt, $lte, $eq, $neq, $gte, $gt, $in, $nin, and $regex. To use this feature, enable \"filtering vector search\" in the \"Preview Features\" tab of your Azure Subscription. Learn more about preview features [here](https://learn.microsoft.com/azure/cosmos-db/mongodb/vcore/vector-search#filtered-vector-search-preview)."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "19c43de6-47f9-45f0-a422-8d852a5d191f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'raw': {'defaultShard': {'numIndexesBefore': 3,\n",
+       "   'numIndexesAfter': 4,\n",
+       "   'createdCollectionAutomatically': False,\n",
+       "   'ok': 1}},\n",
+       " 'ok': 1}"
+      ]
+     },
+     "execution_count": 29,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# create a filter index\n",
+    "vectorstore.create_filter_index(\n",
+    "    property_to_filter=\"metadata.source\", index_name=\"filter_index\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "c7031279-dfb8-43f2-a7a8-d10a3786023b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
+    "docs = vectorstore.similarity_search(\n",
+    "    query, pre_filter={\"metadata.source\": {\"$ne\": \"filter content\"}}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "3860be72-d293-43b9-a727-425f166ff6c6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "4"
+      ]
+     },
+     "execution_count": 31,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(docs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "id": "b7fb9800-b1cf-4315-af9d-e8c572d3e05f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = vectorstore.similarity_search(\n",
+    "    query,\n",
+    "    pre_filter={\"metadata.source\": {\"$ne\": \"../../how_to/state_of_the_union.txt\"}},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "id": "dba9d39e-6220-4fad-84fa-e123aa7ca6e4",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0"
+      ]
+     },
+     "execution_count": 33,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(docs)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "50bb4346",
+   "id": "25ea7250-6e8f-48e6-aac9-196effbdc8d8",
    "metadata": {},
    "outputs": [],
    "source": []