poetry

docs/docs/integrations/retrievers/Activeloop DeepMemory+LangChain.ipynb

@@ -0,0 +1,614 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Activeloop DeepLake's DeepMemory + LangChain + ragas"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Recently, RAGs (Retrieval-Augmented Generators) have surged in popularity. Advanced RAG techniques and agents are elevating the potential of what can be achieved with RAGs. However, a significant challenge remains: domain shift. If your Language Model (LLM) was pretrained on a dataset significantly different from your production dataset's distribution, you might face reduced recall. This can hinder the performance of RAGs in production settings. Traditionally, this issue can be addressed by either incorporating a new dataset during the LLM's pretraining phase or through extensive fine-tuning. Both approaches come with drawbacks. Adding data during the pretraining phase can be expensive, and fine-tuning might degrade the model's overall performance, as it could lose previously learned information. [Activeloop](https://activeloop.ai/) has tackled this challenge by introducing an additional layer to the LLM. This layer transforms corpus data from the embedding space to a latent space better aligned with user queries. This is achieved with [Deep Memory](https://www.activeloop.ai/resources/use-deep-memory-to-boost-rag-apps-accuracy-by-up-to-22/), a feature available to Activeloop Deep Lake users, where you can train a deep memory network that learns to align user questions with the model's embeddings.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For this tutorial we will parse deeplake documentation, and create a RAG system that could answer the question from the docs. \n",
+    "\n",
+    "The tutorial can be divided into several parts:\n",
+    "1. [Dataset creation and uploading](#1-dataset-creation)\n",
+    "2. [Generating synthetic queries and training deep_memory](#2-generating-synthetic-queries-and-training-deep_memory)\n",
+    "3. [Evaluating deep memory performance](#3-evaluating-deep-memory-performance)\n",
+    "    - 3.1 [using deepmemory recall@10 metric](#31-using-deepmemory-recall10-metric)\n",
+    "    - 3.2 [using ragas](#32-deepmemory--ragas)\n",
+    "    - 3.3 [deep_memory inference](#33-deepmemory-inference)\n",
+    "    - 3.4 [deep_memory cost savings](#34-cost-savings)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"dataset-creation\"></a>\n",
+    "## 1. Dataset Creation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We will parse activeloop's docs for this tutorial using `BeautifulSoup` library and LangChain's document parsers like `Html2TextTransformer`, `AsyncHtmlLoader`. So we will need to install the following libraries:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install tiktoken openai python-dotenv datasets langchain deeplake beautifulsoup4 html2text ragas"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Also you'll need to create a [Activeloop]((https://activeloop.ai/)) account."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import getpass\n",
+    "\n",
+    "from langchain.vectorstores.deeplake import DeepLake\n",
+    "\n",
+    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
+    "from langchain.chains import RetrievalQA\n",
+    "from langchain.llms import OpenAIChat\n",
+    "\n",
+    "\n",
+    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter your OpenAI API token: \")\n",
+    "# # activeloop token is needed if you are not signed in using CLI: `activeloop login -u <USERNAME> -p <PASSWORD>`\n",
+    "os.environ[\"ACTIVELOOP_TOKEN\"] = getpass.getpass(\"Enter your ActiveLoop API token: \") # Get your API token from https://app.activeloop.ai, click on your profile picture in the top right corner, and select \"API Tokens\"\n",
+    "\n",
+    "token = os.getenv(\"ACTIVELOOP_TOKEN\")\n",
+    "openai_embeddings = OpenAIEmbeddings()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db = DeepLake(\n",
+    "    dataset_path=f\"hub://{ORG_ID}/deeplake-docs-deepmemory\", # org_id stands for your username or organization from activeloop\n",
+    "    embedding=openai_embeddings,\n",
+    "    runtime={\"tensor_db\": True},\n",
+    "    token=token,\n",
+    "    # overwrite=True, # user overwrite flag if you want to overwrite the full dataset\n",
+    "    read_only=False,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "parsing all links in the webpage using `BeautifulSoup`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "from bs4 import BeautifulSoup\n",
+    "from urllib.parse import urljoin\n",
+    "\n",
+    "def get_all_links(url):\n",
+    "    response = requests.get(url)\n",
+    "    if response.status_code != 200:\n",
+    "        print(f'Failed to retrieve the page: {url}')\n",
+    "        return []\n",
+    "\n",
+    "    soup = BeautifulSoup(response.content, 'html.parser')\n",
+    "    \n",
+    "    # Finding all 'a' tags which typically contain href attribute for links\n",
+    "    links = [urljoin(url, a['href']) for a in soup.find_all('a', href=True) if a['href']]\n",
+    "\n",
+    "    return links\n",
+    "\n",
+    "base_url = \"https://docs.deeplake.ai/en/latest/\"\n",
+    "all_links = get_all_links(base_url)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Loading data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_loaders import AsyncHtmlLoader\n",
+    "\n",
+    "loader = AsyncHtmlLoader(all_links)\n",
+    "docs = loader.load()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Converting data into user readable format:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.document_transformers import Html2TextTransformer\n",
+    "\n",
+    "html2text = Html2TextTransformer()\n",
+    "docs_transformed = html2text.transform_documents(docs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let us chunk further the documents as some of the contain too much text:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "\n",
+    "\n",
+    "chunk_size = 4096\n",
+    "docs_new = []\n",
+    "\n",
+    "text_splitter = RecursiveCharacterTextSplitter(\n",
+    "    chunk_size = chunk_size,\n",
+    ")\n",
+    "\n",
+    "for doc in docs_transformed:\n",
+    "    if len(doc.page_content) < chunk_size:\n",
+    "        docs_new.append(doc)\n",
+    "    else:\n",
+    "        docs = text_splitter.create_documents([doc.page_content])\n",
+    "        docs_new.extend(docs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Populating VectorStore:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "docs = db.add_documents(docs_new)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"training\"></a>\n",
+    "## 2. Generating synthetic queries and training deep_memory "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next step would be to train a deep_memory model that will align your users queries with the dataset that you already have. If you don't have any user queries yet, no worries, we will generate them using LLM!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here above we showed the overall schema how deep_memory works. So as you can see, in order to train it you need relevence, queries together with corpus data (data that we want to query). Corpus data was already populated in the previous section, here we will be generating questions and relevance. \n",
+    "\n",
+    "1. `questions` - is a text of strings, where each string represents a query\n",
+    "2. `relevence` - contains links to the ground truth for each question. There might be several docs that contain answer to the given question. Because of this relevenve is `List[List[tuple[str, float]]]`, where outer list represents queries and inner list relevent documents. Tuple contains str, float pair where string represent the id of the source doc (corresponds to the `id` tensor in the dataset), while float corresponds to how much current document is related to the question.  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let us generate synthetic questions and relevance:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from typing import Optional, List\n",
+    "\n",
+    "from langchain.chains.openai_functions import (\n",
+    "    create_openai_fn_chain,\n",
+    "    create_structured_output_chain,\n",
+    ")\n",
+    "from langchain.chat_models import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate, HumanMessagePromptTemplate\n",
+    "from langchain.schema import HumanMessage, SystemMessage\n",
+    "\n",
+    "from pydantic import BaseModel, Field"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# fetch dataset docs and ids if they exist (optional you can also ingest)\n",
+    "docs = db.vectorstore.dataset.text.data(fetch_chunks=True, aslist=True)['value']\n",
+    "ids = db.vectorstore.dataset.id.data(fetch_chunks=True, aslist=True)['value']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# If we pass in a model explicitly, we need to make sure it supports the OpenAI function-calling API.\n",
+    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n",
+    "\n",
+    "class Questions(BaseModel):\n",
+    "    \"\"\"Identifying information about a person.\"\"\"\n",
+    "    question: str = Field(..., description=\"Questions about text\")\n",
+    "\n",
+    "prompt_msgs = [\n",
+    "    SystemMessage(\n",
+    "        content=\"You are a world class expert for generating questions based on provided context. \\\n",
+    "                You make sure the question can be answered by the text.\"\n",
+    "    ),\n",
+    "    HumanMessagePromptTemplate.from_template(\n",
+    "        \"Use the given text to generate a question from the following input: {input}\"\n",
+    "    ),\n",
+    "    HumanMessage(content=\"Tips: Make sure to answer in the correct format\"),\n",
+    "]\n",
+    "prompt = ChatPromptTemplate(messages=prompt_msgs)\n",
+    "chain = create_structured_output_chain(Questions, llm, prompt, verbose=True)\n",
+    "\n",
+    "text = \"# Understanding Hallucinations and Bias ## **Introduction** In this lesson, we'll cover the concept of **hallucinations** in LLMs, highlighting their influence on AI applications and demonstrating how to mitigate them using techniques like the retriever's architectures. We'll also explore **bias** within LLMs with examples.\"\n",
+    "questions = chain.run(input=text)\n",
+    "print(questions)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "from tqdm import tqdm\n",
+    "from langchain.embeddings import OpenAIEmbeddings\n",
+    "\n",
+    "def generate_queries(docs: List[str], ids: List[str], n: int=100):\n",
+    "\n",
+    "    questions = []\n",
+    "    relevances = []\n",
+    "    pbar = tqdm(total=n)\n",
+    "    while len(questions) < n:\n",
+    "      # 1. randomly draw a piece of text and relevance id\n",
+    "      r = random.randint(0, len(docs)-1)\n",
+    "      text, label = docs[r], ids[r]\n",
+    "\n",
+    "      # 2. generate queries and assign and relevance id\n",
+    "      generated_qs = [chain.run(input=text).question]\n",
+    "      questions.extend(generated_qs)\n",
+    "      relevances.extend([[(label, 1)] for _ in generated_qs])\n",
+    "      pbar.update(len(generated_qs))\n",
+    "      if len(questions) % 10 == 0:\n",
+    "        print(f\"q: {len(questions)}\")\n",
+    "    return questions[:n], relevances[:n]\n",
+    "\n",
+    "chain = create_structured_output_chain(Questions, llm, prompt, verbose=False)\n",
+    "questions, relevances = generate_queries(docs, ids, n=200)\n",
+    "\n",
+    "train_questions, train_relevances = questions[:100], relevances[:100]\n",
+    "test_questions, test_relevances = questions[100:], relevances[100:]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we created 100 training queries as well as 100 queries for testing. Now let us train the deep_memory:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "job_id = db.vectorstore.deep_memory.train(\n",
+    "    queries=train_questions,\n",
+    "    relevance=train_relevances,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let us track the training progress:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "db.vectorstore.deep_memory.status('6538939ca0b69a9ca45c528c')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"evaluation\"></a>\n",
+    "## 3. Evaluating deep memory performance"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Great we've trained the model! It's showing some substantial improvement in recall, but how can we use it now and evaluate on unseen new data? In this section we will delve into model evaluation and inference part and see how it can be used with LangChain in order to increase retrieval accuracy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"recall@10\"></a>\n",
+    "### 3.1 using deepmemory recall@10 metric"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For the beginning we can use deep_memory's builtin evaluation method. it can be done easily in a few lines of code:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "recall = db.vectorstore.deep_memory.evaluate(\n",
+    "    queries=test_questions,\n",
+    "    relevance=test_relevances,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It is showing quite substatntial improvement on an unseen test dataset too!!!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"ragas\"></a>\n",
+    "### 3.2 DeepMemory + ragas"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ragas.metrics import (\n",
+    "    context_recall,\n",
+    ")\n",
+    "from ragas.langchain import RagasEvaluatorChain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let us convert recall into ground truths:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def convert_relevance_to_ground_truth(docs, relevance):\n",
+    "    ground_truths = []\n",
+    "    \n",
+    "    for rel in relevance:\n",
+    "        ground_truth = []\n",
+    "        for doc_id, _ in rel:\n",
+    "            ground_truth.append(docs[doc_id])\n",
+    "        ground_truths.append(ground_truth)\n",
+    "    return ground_truths"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ground_truths = convert_relevance_to_ground_truth(docs, test_relevances)\n",
+    "\n",
+    "for deep_memory in [False, True]:\n",
+    "    print(\"\\nEvaluating with deep_memory =\", deep_memory)\n",
+    "    print(\"===================================\")\n",
+    "    \n",
+    "    retriever = db.as_retriever()\n",
+    "    retriever.search_kwargs[\"deep_memory\"] = deep_memory\n",
+    "\n",
+    "    qa_chain = RetrievalQA.from_chain_type(\n",
+    "        llm=OpenAIChat(model=\"gpt-3.5-turbo\"),\n",
+    "        chain_type=\"stuff\",\n",
+    "        retriever=retriever,\n",
+    "        return_source_documents=True,\n",
+    "    )\n",
+    "    \n",
+    "    metrics = {\n",
+    "        \"context_recall_score\": 0,\n",
+    "    }\n",
+    "    \n",
+    "    eval_chains = {\n",
+    "        m.name: RagasEvaluatorChain(metric=m)\n",
+    "        for m in [context_recall]\n",
+    "    }\n",
+    "    \n",
+    "    for question, ground_truth in zip(test_questions, ground_truths):\n",
+    "        result = qa_chain({\"query\": question})\n",
+    "        result[\"ground_truths\"] = ground_truth\n",
+    "        for name, eval_chain in eval_chains.items():\n",
+    "            score_name = f\"{name}_score\"\n",
+    "            metrics[score_name] += eval_chain(result)[score_name]\n",
+    "    \n",
+    "    for metric in metrics:\n",
+    "        metrics[metric] /= len(test_questions)\n",
+    "        print(f\"{metric}: {metrics[metric]}\")\n",
+    "    print(\"===================================\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"inference\"></a>\n",
+    "### 3.3 DeepMemory Inference"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "with deep_memory"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "retriver = db.as_retriever()\n",
+    "retriver.search_kwargs[\"deep_memory\"] = True\n",
+    "retriver.search_kwargs[\"k\"] = 10\n",
+    "\n",
+    "query=\"Deamination of cytidine to uridine on the minus strand of viral DNA results in catastrophic G-to-A mutations in the viral genome.\"\n",
+    "qa = RetrievalQA.from_chain_type(llm=OpenAIChat(model='gpt-4'), chain_type='stuff', retriever=retriver)\n",
+    "print(qa.run(query))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "without deep_memory"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "retriver = db.as_retriever()\n",
+    "retriver.search_kwargs[\"deep_memory\"] = False\n",
+    "retriver.search_kwargs[\"k\"] = 10\n",
+    "\n",
+    "query=\"Deamination of cytidine to uridine on the minus strand of viral DNA results in catastrophic G-to-A mutations in the viral genome.\"\n",
+    "qa = RetrievalQA.from_chain_type(llm=OpenAIChat(model='gpt-4'), chain_type='stuff', retriever=retriver)\n",
+    "qa.run(query)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name=\"cost\"></a>\n",
+    "### 3.4 Cost savings"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Deep Memory increases retrieval accuracy without altering your existing workflow. Additionally, by reducing the top_k input into the LLM, you can significantly cut inference costs via lower token usage."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

libs/langchain/langchain/vectorstores/deeplake.py

@@ -7,8 +7,8 @@ import numpy as np
 
 try:
     import deeplake
+    from deeplake import VectorStore as DeepLakeVectorStore
     from deeplake.core.fast_forwarding import version_compare
-    from deeplake.core.vectorstore import DeepLakeVectorStore
 
     _DEEPLAKE_INSTALLED = True
 except ImportError:
@@ -63,6 +63,7 @@ class DeepLake(VectorStore):
         verbose: bool = True,
         exec_option: Optional[str] = None,
         runtime: Optional[Dict] = None,
+        index_params: Optional[Dict[str, Union[int, str]]] = None,
         **kwargs: Any,
     ) -> None:
         """Creates an empty DeepLakeVectorStore or loads an existing one.
@@ -119,6 +120,23 @@ class DeepLake(VectorStore):
                 Deep Lake's Managed Tensor Database. Not applicable when loading an
                 existing Vector Store. To create a Vector Store in the Managed Tensor
                 Database, set `runtime = {"tensor_db": True}`.
+            index_params (Optional[Dict[str, Union[int, str]]], optional): Dictionary
+                containing information about vector index that will be created. Defaults
+                to None, which will utilize ``DEFAULT_VECTORSTORE_INDEX_PARAMS`` from
+                ``deeplake.constants``. The specified key-values override the default
+                ones.
+                - threshold: The threshold for the dataset size above which an index
+                    will be created for the embedding tensor. When the threshold value
+                    is set to -1, index creation is turned off. Defaults to -1, which
+                    turns off the index.
+                - distance_metric: This key specifies the method of calculating the
+                    distance between vectors when creating the vector database (VDB)
+                    index. It can either be a string that corresponds to a member of
+                    the DistanceType enumeration, or the string value itself.
+                    - If no value is provided, it defaults to "L2".
+                    - "L2" corresponds to DistanceType.L2_NORM.
+                    - "COS" corresponds to DistanceType.COSINE_SIMILARITY.
+                - additional_params: Additional parameters for fine-tuning the index.
             **kwargs: Other optional keyword arguments.
 
         Raises:
@@ -161,6 +179,7 @@ class DeepLake(VectorStore):
             exec_option=exec_option,
             verbose=verbose,
             runtime=runtime,
+            index_params=index_params,
             **kwargs,
         )
 
@@ -295,12 +314,13 @@ class DeepLake(VectorStore):
         embedding: Optional[Union[List[float], np.ndarray]] = None,
         embedding_function: Optional[Callable] = None,
         k: int = 4,
-        distance_metric: str = "L2",
+        distance_metric: Optional[str] = None,
         use_maximal_marginal_relevance: bool = False,
         fetch_k: Optional[int] = 20,
         filter: Optional[Union[Dict, Callable]] = None,
         return_score: bool = False,
         exec_option: Optional[str] = None,
+        deep_memory: bool = False,
         **kwargs: Any,
     ) -> Any[List[Document], List[Tuple[Document, float]]]:
         """
@@ -312,9 +332,9 @@ class DeepLake(VectorStore):
             embedding_function (Callable, optional): Function to convert `query`
                 into embedding.
             k (int): Number of Documents to return.
-            distance_metric (str): `L2` for Euclidean, `L1` for Nuclear, `max`
-                for L-infinity distance, `cos` for cosine similarity, 'dot' for dot
-                product.
+            distance_metric (Optional[str], optional): `L2` for Euclidean, `L1` for
+                Nuclear, `max` for L-infinity distance, `cos` for cosine similarity,
+                'dot' for dot product.
             filter (Union[Dict, Callable], optional): Additional filter prior
                 to the embedding search.
                 - ``Dict`` - Key-value search on tensors of htype json, on an
@@ -334,6 +354,13 @@ class DeepLake(VectorStore):
                 - ``tensor_db`` - Hosted Managed Tensor Database for storage
                     and query execution. Only for data in Deep Lake Managed Database.
                     Use runtime = {"db_engine": True} during dataset creation.
+            deep_memory (bool): Whether to use the Deep Memory model for improving
+                search results. Defaults to False if deep_memory is not specified in
+                the Vector Store initialization. If True, the distance metric is set
+                to "deepmemory_distance", which represents the metric with which the
+                model was trained. The search is performed using the Deep Memory model.
+                If False, the distance metric is set to "COS" or whatever distance
+                metric user specifies.
             **kwargs: Additional keyword arguments.
 
         Returns:
@@ -386,7 +413,8 @@ class DeepLake(VectorStore):
             distance_metric=distance_metric,
             filter=filter,
             exec_option=exec_option,
-            return_tensors=["embedding", "metadata", "text"],
+            return_tensors=["embedding", "metadata", "text", "id"],
+            deep_memory=deep_memory,
         )
 
         scores = result["score"]
@@ -467,6 +495,13 @@ class DeepLake(VectorStore):
                     - 'tensor_db': Managed Tensor Database for storage and query.
                         Only for data in Deep Lake Managed Database.
                         Use `runtime = {"db_engine": True}` during dataset creation.
+                deep_memory (bool): Whether to use the Deep Memory model for improving
+                    search results. Defaults to False if deep_memory is not specified
+                    in the Vector Store initialization. If True, the distance metric
+                    is set to "deepmemory_distance", which represents the metric with
+                    which the model was trained. The search is performed using the Deep
+                    Memory model. If False, the distance metric is set to "COS" or
+                    whatever distance metric user specifies.
 
         Returns:
             List[Document]: List of Documents most similar to the query vector.
@@ -530,6 +565,13 @@ class DeepLake(VectorStore):
                 distance_metric (str): `L2` for Euclidean, `L1` for Nuclear,
                     `max` for L-infinity distance, `cos` for cosine similarity,
                     'dot' for dot product. Defaults to `L2`.
+                deep_memory (bool): Whether to use the Deep Memory model for improving
+                    search results. Defaults to False if deep_memory is not specified
+                    in the Vector Store initialization. If True, the distance metric
+                    is set to "deepmemory_distance", which represents the metric with
+                    which the model was trained. The search is performed using the Deep
+                    Memory model. If False, the distance metric is set to "COS" or
+                    whatever distance metric user specifies.
 
         Returns:
             List[Document]: List of Documents most similar to the query vector.
@@ -586,6 +628,13 @@ class DeepLake(VectorStore):
                         data stored in the Deep Lake Managed Database. To store datasets
                         in this database, specify `runtime = {"db_engine": True}`
                         during dataset creation.
+                deep_memory (bool): Whether to use the Deep Memory model for improving
+                    search results. Defaults to False if deep_memory is not specified
+                    in the Vector Store initialization. If True, the distance metric
+                    is set to "deepmemory_distance", which represents the metric with
+                    which the model was trained. The search is performed using the Deep
+                    Memory model. If False, the distance metric is set to "COS" or
+                    whatever distance metric user specifies.
 
         Returns:
             List[Tuple[Document, float]]: List of documents most similar to the query
@@ -641,6 +690,13 @@ class DeepLake(VectorStore):
                     data stored in the Deep Lake Managed Database. To store datasets
                     in this database, specify `runtime = {"db_engine": True}`
                     during dataset creation.
+            deep_memory (bool): Whether to use the Deep Memory model for improving
+                search results. Defaults to False if deep_memory is not specified
+                in the Vector Store initialization. If True, the distance metric
+                is set to "deepmemory_distance", which represents the metric with
+                which the model was trained. The search is performed using the Deep
+                Memory model. If False, the distance metric is set to "COS" or
+                whatever distance metric user specifies.
             **kwargs: Additional keyword arguments.
 
         Returns:
@@ -701,6 +757,13 @@ class DeepLake(VectorStore):
                         for data stored in the Deep Lake Managed Database. To store
                         datasets in this database, specify
                         `runtime = {"db_engine": True}` during dataset creation.
+            deep_memory (bool): Whether to use the Deep Memory model for improving
+                search results. Defaults to False if deep_memory is not specified
+                in the Vector Store initialization. If True, the distance metric
+                is set to "deepmemory_distance", which represents the metric with
+                which the model was trained. The search is performed using the Deep
+                Memory model. If False, the distance metric is set to "COS" or
+                whatever distance metric user specifies.
             **kwargs: Additional keyword arguments
 
         Returns:

libs/langchain/poetry.lock

@@ -1902,12 +1902,12 @@ files = [
 
 [[package]]
 name = "deeplake"
-version = "3.7.1"
+version = "3.8.3"
 description = "Activeloop Deep Lake"
 optional = true
 python-versions = "*"
 files = [
-    {file = "deeplake-3.7.1.tar.gz", hash = "sha256:11ef339ac58d307f42ecb5549d77af11673a7251b4034a692ab206d977006c7a"},
+    {file = "deeplake-3.8.3.tar.gz", hash = "sha256:db6ea8b50549bab35579c2a6d70888356162d9aa89c18b55c16a9f5aeaf4a5fe"},
 ]
 
 [package.dependencies]
@@ -1915,8 +1915,9 @@ aioboto3 = {version = ">=10.4.0", markers = "python_version >= \"3.7\" and sys_p
 boto3 = "*"
 click = "*"
 humbug = ">=0.3.1"
+libdeeplake = "0.0.84"
+lz4 = "*"
 nest_asyncio = {version = "*", markers = "python_version >= \"3.7\" and sys_platform != \"win32\""}
-numcodecs = "*"
 numpy = "*"
 pathos = "*"
 pillow = "*"
@@ -1924,12 +1925,12 @@ pyjwt = "*"
 tqdm = "*"
 
 [package.extras]
-all = ["IPython", "av (>=8.1.0)", "azure-cli", "azure-identity", "azure-storage-blob", "flask", "google-api-python-client (>=2.31.0,<2.32.0)", "google-auth (>=2.0.1,<2.1.0)", "google-auth-oauthlib (>=0.4.5,<0.5.0)", "google-cloud-storage (>=1.42.0,<1.43.0)", "laspy", "libdeeplake (==0.0.78)", "nibabel", "oauth2client (>=4.1.3,<4.2.0)", "pydicom"]
+all = ["IPython", "av (>=8.1.0)", "azure-cli", "azure-identity", "azure-storage-blob", "flask", "google-api-python-client (>=2.31.0,<2.32.0)", "google-auth (>=2.0.1,<2.1.0)", "google-auth-oauthlib (>=0.4.5,<0.5.0)", "google-cloud-storage (>=1.42.0,<1.43.0)", "laspy", "libdeeplake (==0.0.84)", "nibabel", "oauth2client (>=4.1.3,<4.2.0)", "pydicom"]
 audio = ["av (>=8.1.0)"]
 av = ["av (>=8.1.0)"]
 azure = ["azure-cli", "azure-identity", "azure-storage-blob"]
 dicom = ["nibabel", "pydicom"]
-enterprise = ["libdeeplake (==0.0.78)", "pyjwt"]
+enterprise = ["libdeeplake (==0.0.84)", "pyjwt"]
 gcp = ["google-auth (>=2.0.1,<2.1.0)", "google-auth-oauthlib (>=0.4.5,<0.5.0)", "google-cloud-storage (>=1.42.0,<1.43.0)"]
 gdrive = ["google-api-python-client (>=2.31.0,<2.32.0)", "google-auth (>=2.0.1,<2.1.0)", "google-auth-oauthlib (>=0.4.5,<0.5.0)", "oauth2client (>=4.1.3,<4.2.0)"]
 medical = ["nibabel", "pydicom"]
@@ -3790,6 +3791,7 @@ optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*"
 files = [
     {file = "jsonpointer-2.4-py2.py3-none-any.whl", hash = "sha256:15d51bba20eea3165644553647711d150376234112651b4f1811022aecad7d7a"},
+    {file = "jsonpointer-2.4.tar.gz", hash = "sha256:585cee82b70211fa9e6043b7bb89db6e1aa49524340dde8ad6b63206ea689d88"},
 ]
 
 [[package]]
@@ -4225,34 +4227,33 @@ files = [
 
 [[package]]
 name = "libdeeplake"
-version = "0.0.60"
+version = "0.0.84"
 description = "C++ backend for Deep Lake"
 optional = true
 python-versions = "*"
 files = [
-    {file = "libdeeplake-0.0.60-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:077c663370834831343b512cb4d2b166634f1aaea31e31f86721e257ce0ed1dd"},
-    {file = "libdeeplake-0.0.60-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c6b152970a9719f10fb4656409a19cab1b3721a4c2ff9d36b8acb883992dbe7c"},
-    {file = "libdeeplake-0.0.60-cp310-cp310-manylinux2010_x86_64.whl", hash = "sha256:e54901b4ffd6b606ad5829354788626972b7a7aa7a9dea055bb987cd06544ea3"},
-    {file = "libdeeplake-0.0.60-cp310-cp310-manylinux2014_aarch64.whl", hash = "sha256:010f172e87027fb81b8eaaec35bda898783fe618632f44dd716166aa6d3bf3f6"},
-    {file = "libdeeplake-0.0.60-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:2d18b4b77939af29d927afa5e9d4652603821b5a48c7959ad648caf5b2509efd"},
-    {file = "libdeeplake-0.0.60-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:1c424719292da9abf4f531b21181b8dcb6a77c10db6d8d559da708b6053745c9"},
-    {file = "libdeeplake-0.0.60-cp311-cp311-manylinux2010_x86_64.whl", hash = "sha256:27ebe8638429dcdfb650f942128e9aa9944ad68332b5a5dad8d943eeb9905d24"},
-    {file = "libdeeplake-0.0.60-cp311-cp311-manylinux2014_aarch64.whl", hash = "sha256:9f3841343806372905c42ea30c89b553f79530cdaa24d5e7753cbd45fa8444f8"},
-    {file = "libdeeplake-0.0.60-cp37-cp37m-macosx_10_12_x86_64.whl", hash = "sha256:afdbf256de09b362c3fb198e33547dff157583762814f7973d4c01357506cfd7"},
-    {file = "libdeeplake-0.0.60-cp37-cp37m-macosx_11_0_arm64.whl", hash = "sha256:aee8a0f1d9521cb964efc39bee6e4cfbc18935501e0a0969dd43fe90ccd3e440"},
-    {file = "libdeeplake-0.0.60-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:7bf53919c17318eff53ab5f46514fe7bf86d32bfd0e7b73ab0e47e57279fb58f"},
-    {file = "libdeeplake-0.0.60-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:ad570eacbdc79533cc9fbc005903a7af2e037a7a743e60ed5d1486c7daade737"},
-    {file = "libdeeplake-0.0.60-cp38-cp38-macosx_10_12_x86_64.whl", hash = "sha256:957123ca007ce552f08646ceaf386e60703d9aa1f299e76f71d4b922c8ccbf4f"},
-    {file = "libdeeplake-0.0.60-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:1ea962a884eea30d092ed379dd83f3c6a162268180b55d6afa8778970ea99863"},
-    {file = "libdeeplake-0.0.60-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:c330c1ca89c057cdabca9b6261ecc051ebad5f658d17c140a211a8842bd34ce5"},
-    {file = "libdeeplake-0.0.60-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:397a4da31581472fc9c7444b91ee4619f29d9b18371554514a2de857f5bad8a8"},
-    {file = "libdeeplake-0.0.60-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:cf33e027abed9d123e6fccb822fd128ea9d9d5833600aac82a7e0280df0c190e"},
-    {file = "libdeeplake-0.0.60-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:5844475365631984453b563e3e7c69d65d96d91e44e8e0762f1877befcac58b5"},
-    {file = "libdeeplake-0.0.60-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:753ac3840852b6009056ad725afd48c460a431470b3cccac3f717e423e2c7c1c"},
-    {file = "libdeeplake-0.0.60-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:6f50fa21ca263084783b313a71ffb26ea7d614b9ca5e16398be1f9a1dbe0a775"},
+    {file = "libdeeplake-0.0.84-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:b24e7d025ce29b0e1505c8b952f2e7eda454461fefc9bad61ddc804d1b081c80"},
+    {file = "libdeeplake-0.0.84-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:dc120ec1cb4fe3f49bfbf02e0a5e8e15129329083ab74f1d618cfe71867eeb61"},
+    {file = "libdeeplake-0.0.84-cp310-cp310-manylinux2014_aarch64.whl", hash = "sha256:d94c419063d6e96c4defc5fcdce2a9e143a668bb9292105a2abb8040fbef519e"},
+    {file = "libdeeplake-0.0.84-cp310-cp310-manylinux2014_x86_64.whl", hash = "sha256:65e6eab3b1920acef9fbfd5ecb11c4521c38e66bff0b3f02a0760c7d5722e242"},
+    {file = "libdeeplake-0.0.84-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:3a3a2acf2deef3a72bcde1856a025b13bd594267e84ada91bc79a8881606f29a"},
+    {file = "libdeeplake-0.0.84-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:69cea8d6d5e736a1a655c191d9192d47c93543f68fc08703a3660b327b2ea8af"},
+    {file = "libdeeplake-0.0.84-cp311-cp311-manylinux2014_aarch64.whl", hash = "sha256:d4be0dc25d01d07fb3e3cb61d8ff287520fa21f7eeb5190cb8ac38bc32e24a13"},
+    {file = "libdeeplake-0.0.84-cp311-cp311-manylinux2014_x86_64.whl", hash = "sha256:74798f43084827025e783cedee6e480699cf8c00b89730fc98d39fae8a48bba7"},
+    {file = "libdeeplake-0.0.84-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:6b9b489254ae2ed18a59eb3ebb9b0229dbd68dfbff695ed61a8b7707cbeae188"},
+    {file = "libdeeplake-0.0.84-cp37-cp37m-manylinux2014_x86_64.whl", hash = "sha256:f6dfd25181d80fedcb57f93a3e14ba15918b19e30e6758c8868ec452ec4c3d31"},
+    {file = "libdeeplake-0.0.84-cp38-cp38-macosx_10_12_x86_64.whl", hash = "sha256:bb2f2a0b9c292b342e33e87c9f08f8eed8bc4c0a1b2d16d44a48fc1553a77860"},
+    {file = "libdeeplake-0.0.84-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:c61696a92b0e07c2ecec5faebb5f78a3f1aae54b12bd84ca553a2d8b9dfdc539"},
+    {file = "libdeeplake-0.0.84-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:7983b44e2efb541ecaa0960abe2f15eabf1f2567ff2b6c6042b261c5b3399a65"},
+    {file = "libdeeplake-0.0.84-cp38-cp38-manylinux2014_x86_64.whl", hash = "sha256:6cd70f8955dd63247cbc56cf7128d948184a0ebf080b46476142bc701a1fc090"},
+    {file = "libdeeplake-0.0.84-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:553a22bc3c887fdfa5a73a049f72594170e3e0098c0801f7112e64861f25faf2"},
+    {file = "libdeeplake-0.0.84-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:f5b8bc4243c26132bfc87c5dec9c64d42bb9e330a0055a2d8c965daaa0b19927"},
+    {file = "libdeeplake-0.0.84-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:8bded8b6b3d9055bcb21ba34a2dc8384d84ba61042f8f030ab765f93709029ae"},
+    {file = "libdeeplake-0.0.84-cp39-cp39-manylinux2014_x86_64.whl", hash = "sha256:301dab8e7894bc02431173a4d084271a1e451758f3e99b7029d48b5333c0d849"},
 ]
 
 [package.dependencies]
+dill = "*"
 numpy = "*"
 
 [[package]]
@@ -4597,6 +4598,16 @@ files = [
     {file = "MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:5bbe06f8eeafd38e5d0a4894ffec89378b6c6a625ff57e3028921f8ff59318ac"},
     {file = "MarkupSafe-2.1.3-cp311-cp311-win32.whl", hash = "sha256:dd15ff04ffd7e05ffcb7fe79f1b98041b8ea30ae9234aed2a9168b5797c3effb"},
     {file = "MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl", hash = "sha256:134da1eca9ec0ae528110ccc9e48041e0828d79f24121a1a146161103c76e686"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:f698de3fd0c4e6972b92290a45bd9b1536bffe8c6759c62471efaa8acb4c37bc"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:aa57bd9cf8ae831a362185ee444e15a93ecb2e344c8e52e4d721ea3ab6ef1823"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffcc3f7c66b5f5b7931a5aa68fc9cecc51e685ef90282f4a82f0f5e9b704ad11"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47d4f1c5f80fc62fdd7777d0d40a2e9dda0a05883ab11374334f6c4de38adffd"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1f67c7038d560d92149c060157d623c542173016c4babc0c1913cca0564b9939"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:9aad3c1755095ce347e26488214ef77e0485a3c34a50c5a5e2471dff60b9dd9c"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:14ff806850827afd6b07a5f32bd917fb7f45b046ba40c57abdb636674a8b559c"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8f9293864fe09b8149f0cc42ce56e3f0e54de883a9de90cd427f191c346eb2e1"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-win32.whl", hash = "sha256:715d3562f79d540f251b99ebd6d8baa547118974341db04f5ad06d5ea3eb8007"},
+    {file = "MarkupSafe-2.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:1b8dd8c3fd14349433c79fa8abeb573a55fc0fdd769133baac1f5e07abf54aeb"},
     {file = "MarkupSafe-2.1.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:8e254ae696c88d98da6555f5ace2279cf7cd5b3f52be2b5cf97feafe883b58d2"},
     {file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cb0932dc158471523c9637e807d9bfb93e06a95cbf010f1a38b98623b929ef2b"},
     {file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9402b03f1a1b4dc4c19845e5c749e3ab82d5078d16a2a4c2cd2df62d57bb0707"},
@@ -5533,41 +5544,6 @@ importlib-metadata = {version = "*", markers = "python_version < \"3.9\""}
 llvmlite = "==0.41.*"
 numpy = ">=1.21,<1.26"
 
-[[package]]
-name = "numcodecs"
-version = "0.12.0"
-description = "A Python package providing buffer compression and transformation codecs for use in data storage and communication applications."
-optional = true
-python-versions = ">=3.8"
-files = [
-    {file = "numcodecs-0.12.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e03960dd507e00bc102ff4ca2f14fa40b0cfc2ba7279752d31558d0787431a53"},
-    {file = "numcodecs-0.12.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:68b3a18a93a96cba0a1d367ae76c02a74f29f93790e1c8b0423eacc4ce5d421a"},
-    {file = "numcodecs-0.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3a265db9177bd4a19939651b68722b72044bc92bb0b646e2a0d55835c0acb9d5"},
-    {file = "numcodecs-0.12.0-cp310-cp310-win_amd64.whl", hash = "sha256:3b5a0be940093d81eb49b0adba62615d3b973174d8167dbd63cc6d392e157bf6"},
-    {file = "numcodecs-0.12.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f16787a674d1badd55f827b01bbc62b3ef2adecbed59a7db7139a328f0744e4a"},
-    {file = "numcodecs-0.12.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:98dbc61366e2974a1bdc28e08ed790c74d39c9cb40ce3f487ae6e6a76da843dd"},
-    {file = "numcodecs-0.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bd746cd6e7af4925bd2d3e902b5027147d71590cdc8e9e2ad999014fc2405c3b"},
-    {file = "numcodecs-0.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:eae479f65b75af0e75a20049bf83beff154c4662a233695b4f7848d5eee0ef2d"},
-    {file = "numcodecs-0.12.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0c1f679b148bfdc9341686814485d03ad652ea551a90debadbbf9da3fb4cc003"},
-    {file = "numcodecs-0.12.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:aa78ffc79a94aa78234821639c253219d8a26455f020c760ad1b331144363849"},
-    {file = "numcodecs-0.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:01f5457c0c81a556812240a2318c6022ca5c6f66fe2a51f619bdf8b0c855b5f2"},
-    {file = "numcodecs-0.12.0-cp38-cp38-win_amd64.whl", hash = "sha256:e9fc2f2abcb09c301c8e1db16e4d5dc9faf93be8c46d88ac3974e023f0a3533b"},
-    {file = "numcodecs-0.12.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:98c2cedb3d6dd1238b033657da0b710689a9600813bfece28fd7c158328c0d4d"},
-    {file = "numcodecs-0.12.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:182458355f5cad297575f9a16e804fe345c22c7a1b796ee9a0a8bce5a9f66c60"},
-    {file = "numcodecs-0.12.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1fe66c7a2b016e772a60dc8d68479958ae8c9ce306bcc318ee3d2ca883930e94"},
-    {file = "numcodecs-0.12.0-cp39-cp39-win_amd64.whl", hash = "sha256:488ba767d956f8dbf794c4c30df1983a385f048a4f1bc670dd0761b8fe7fd7a3"},
-    {file = "numcodecs-0.12.0.tar.gz", hash = "sha256:6388e5f4e94d18a7165fbd1c9d3637673b74157cff8bc644005f9e2a4c717d6e"},
-]
-
-[package.dependencies]
-numpy = ">=1.7"
-
-[package.extras]
-docs = ["mock", "numpydoc", "sphinx (<7.0.0)", "sphinx-issues"]
-msgpack = ["msgpack"]
-test = ["coverage", "flake8", "pytest", "pytest-cov"]
-zfpy = ["zfpy (>=1.0.0)"]
-
 [[package]]
 name = "numexpr"
 version = "2.8.6"
@@ -7717,6 +7693,7 @@ files = [
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938"},
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d"},
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515"},
+    {file = "PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:326c013efe8048858a6d312ddd31d56e468118ad4cdeda36c719bf5bb6192290"},
     {file = "PyYAML-6.0.1-cp310-cp310-win32.whl", hash = "sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924"},
     {file = "PyYAML-6.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d"},
     {file = "PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007"},
@@ -7724,8 +7701,15 @@ files = [
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d"},
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc"},
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673"},
+    {file = "PyYAML-6.0.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:e7d73685e87afe9f3b36c799222440d6cf362062f78be1013661b00c5c6f678b"},
     {file = "PyYAML-6.0.1-cp311-cp311-win32.whl", hash = "sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741"},
     {file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9"},
+    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0"},
+    {file = "PyYAML-6.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win32.whl", hash = "sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:0d3304d8c0adc42be59c5f8a4d9e3d7379e6955ad754aa9d6ab7a398b59dd1df"},
     {file = "PyYAML-6.0.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47"},
     {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98"},
     {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c"},
@@ -7742,6 +7726,7 @@ files = [
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5"},
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696"},
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735"},
+    {file = "PyYAML-6.0.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:49a183be227561de579b4a36efbb21b3eab9651dd81b1858589f796549873dd6"},
     {file = "PyYAML-6.0.1-cp38-cp38-win32.whl", hash = "sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206"},
     {file = "PyYAML-6.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62"},
     {file = "PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8"},
@@ -7749,6 +7734,7 @@ files = [
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6"},
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0"},
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c"},
+    {file = "PyYAML-6.0.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:04ac92ad1925b2cff1db0cfebffb6ffc43457495c9b3c39d3fcae417d7125dc5"},
     {file = "PyYAML-6.0.1-cp39-cp39-win32.whl", hash = "sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c"},
     {file = "PyYAML-6.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486"},
     {file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"},
@@ -8712,6 +8698,11 @@ files = [
     {file = "scikit_learn-1.3.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f66eddfda9d45dd6cadcd706b65669ce1df84b8549875691b1f403730bdef217"},
     {file = "scikit_learn-1.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c6448c37741145b241eeac617028ba6ec2119e1339b1385c9720dae31367f2be"},
     {file = "scikit_learn-1.3.1-cp311-cp311-win_amd64.whl", hash = "sha256:c413c2c850241998168bbb3bd1bb59ff03b1195a53864f0b80ab092071af6028"},
+    {file = "scikit_learn-1.3.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:ef540e09873e31569bc8b02c8a9f745ee04d8e1263255a15c9969f6f5caa627f"},
+    {file = "scikit_learn-1.3.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:9147a3a4df4d401e618713880be023e36109c85d8569b3bf5377e6cd3fecdeac"},
+    {file = "scikit_learn-1.3.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d2cd3634695ad192bf71645702b3df498bd1e246fc2d529effdb45a06ab028b4"},
+    {file = "scikit_learn-1.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c275a06c5190c5ce00af0acbb61c06374087949f643ef32d355ece12c4db043"},
+    {file = "scikit_learn-1.3.1-cp312-cp312-win_amd64.whl", hash = "sha256:0e1aa8f206d0de814b81b41d60c1ce31f7f2c7354597af38fae46d9c47c45122"},
     {file = "scikit_learn-1.3.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:52b77cc08bd555969ec5150788ed50276f5ef83abb72e6f469c5b91a0009bbca"},
     {file = "scikit_learn-1.3.1-cp38-cp38-macosx_12_0_arm64.whl", hash = "sha256:a683394bc3f80b7c312c27f9b14ebea7766b1f0a34faf1a2e9158d80e860ec26"},
     {file = "scikit_learn-1.3.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a15d964d9eb181c79c190d3dbc2fff7338786bf017e9039571418a1d53dab236"},
@@ -11008,7 +10999,7 @@ cffi = {version = ">=1.11", markers = "platform_python_implementation == \"PyPy\
 cffi = ["cffi (>=1.11)"]
 
 [extras]
-all = ["O365", "aleph-alpha-client", "amadeus", "arxiv", "atlassian-python-api", "awadb", "azure-ai-formrecognizer", "azure-ai-vision", "azure-cognitiveservices-speech", "azure-cosmos", "azure-identity", "beautifulsoup4", "clarifai", "clickhouse-connect", "cohere", "deeplake", "docarray", "duckduckgo-search", "elasticsearch", "esprima", "faiss-cpu", "google-api-python-client", "google-auth", "google-search-results", "gptcache", "html2text", "huggingface_hub", "jinja2", "jq", "lancedb", "langkit", "lark", "libdeeplake", "librosa", "lxml", "manifest-ml", "marqo", "momento", "nebula3-python", "neo4j", "networkx", "nlpcloud", "nltk", "nomic", "openai", "openlm", "opensearch-py", "pdfminer-six", "pexpect", "pgvector", "pinecone-client", "pinecone-text", "psycopg2-binary", "pymongo", "pyowm", "pypdf", "pytesseract", "python-arango", "pyvespa", "qdrant-client", "rdflib", "redis", "requests-toolbelt", "sentence-transformers", "singlestoredb", "tensorflow-text", "tigrisdb", "tiktoken", "torch", "transformers", "weaviate-client", "wikipedia", "wolframalpha"]
+all = ["O365", "aleph-alpha-client", "amadeus", "arxiv", "atlassian-python-api", "awadb", "azure-ai-formrecognizer", "azure-ai-vision", "azure-cognitiveservices-speech", "azure-cosmos", "azure-identity", "beautifulsoup4", "clarifai", "clickhouse-connect", "cohere", "deeplake", "docarray", "duckduckgo-search", "elasticsearch", "esprima", "faiss-cpu", "google-api-python-client", "google-auth", "google-search-results", "gptcache", "html2text", "huggingface_hub", "jinja2", "jq", "lancedb", "langkit", "lark", "librosa", "lxml", "manifest-ml", "marqo", "momento", "nebula3-python", "neo4j", "networkx", "nlpcloud", "nltk", "nomic", "openai", "openlm", "opensearch-py", "pdfminer-six", "pexpect", "pgvector", "pinecone-client", "pinecone-text", "psycopg2-binary", "pymongo", "pyowm", "pypdf", "pytesseract", "python-arango", "pyvespa", "qdrant-client", "rdflib", "redis", "requests-toolbelt", "sentence-transformers", "singlestoredb", "tensorflow-text", "tigrisdb", "tiktoken", "torch", "transformers", "weaviate-client", "wikipedia", "wolframalpha"]
 azure = ["azure-ai-formrecognizer", "azure-ai-vision", "azure-cognitiveservices-speech", "azure-core", "azure-cosmos", "azure-identity", "azure-search-documents", "openai"]
 clarifai = ["clarifai"]
 cli = ["typer"]
@@ -11025,4 +11016,4 @@ text-helpers = ["chardet"]
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.8.1,<4.0"
-content-hash = "19dcb9abd0bda24034e36b571e7ac04d432f47281a80fdc5d4a9810add60966b"
+content-hash = "5bbf0e82483bddabf6ce1ec954786d3a70fe6078f4aa3fc78abefcedb33037b4"

libs/langchain/pyproject.toml

@@ -57,8 +57,7 @@ arxiv = {version = "^1.4", optional = true}
 pypdf = {version = "^3.4.0", optional = true}
 networkx = {version=">=2.6.3, <4", optional = true}
 aleph-alpha-client = {version="^2.15.0", optional = true}
-deeplake = {version = "^3.6.8", optional = true}
-libdeeplake = {version = "^0.0.60", optional = true}
+deeplake = {version = "^3.8.3", optional = true}
 pgvector = {version = "^0.1.6", optional = true}
 psycopg2-binary = {version = "^2.9.5", optional = true}
 pyowm = {version = "^3.3.0", optional = true}
@@ -268,7 +267,6 @@ all = [
     "nomic",
     "aleph-alpha-client",
     "deeplake",
-    "libdeeplake",
     "pgvector",
     "psycopg2-binary",
     "pyowm",