From 289b3422dc242da75de7976cd324ffd7b3d301dc Mon Sep 17 00:00:00 2001 From: Mark Perfect <58151606+markperfect@users.noreply.github.com> Date: Wed, 26 Feb 2025 16:31:40 -0500 Subject: [PATCH] docs: Add Milvus Standalone to documentation (#29650) - [x] **PR title**: - [x] **PR message**: - Added a new section for how to set up and use Milvus with Docker, and added an example of how to instantiate Milvus for hybrid retrieval - Fixed the documentation setup to run `make lint` and `make format` - [x] **Add tests and docs**: If you're adding a new integration, please include N/A - [x] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. --------- Co-authored-by: Mark Perfect Co-authored-by: Chester Curme --- .../how_to/documentation/setup.mdx | 13 - .../integrations/vectorstores/milvus.ipynb | 1341 +++++++++-------- 2 files changed, 731 insertions(+), 623 deletions(-) diff --git a/docs/docs/contributing/how_to/documentation/setup.mdx b/docs/docs/contributing/how_to/documentation/setup.mdx index edbcb3ca1b0..d7ad896d74a 100644 --- a/docs/docs/contributing/how_to/documentation/setup.mdx +++ b/docs/docs/contributing/how_to/documentation/setup.mdx @@ -50,11 +50,6 @@ locally to ensure that it looks good and is free of errors. If you're unable to build it locally that's okay as well, as you will be able to see a preview of the documentation on the pull request page. -From the **monorepo root**, run the following command to install the dependencies: - -```bash -poetry install --with lint,docs --no-root -```` ### Building @@ -158,14 +153,6 @@ the working directory to the `langchain-community` directory: cd [root]/libs/langchain-community ``` -Set up a virtual environment for the package if you haven't done so already. - -Install the dependencies for the package. - -```bash -poetry install --with lint -``` - Then you can run the following commands to lint and format the in-code documentation: ```bash diff --git a/docs/docs/integrations/vectorstores/milvus.ipynb b/docs/docs/integrations/vectorstores/milvus.ipynb index 77aad50e555..e147541a04a 100644 --- a/docs/docs/integrations/vectorstores/milvus.ipynb +++ b/docs/docs/integrations/vectorstores/milvus.ipynb @@ -1,613 +1,734 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "683953b3", - "metadata": { - "id": "683953b3" - }, - "source": [ - "# Milvus\n", - "\n", - ">[Milvus](https://milvus.io/docs/overview.md) is a database that stores, indexes, and manages massive embedding vectors generated by deep neural networks and other machine learning (ML) models.\n", - "\n", - "This notebook shows how to use functionality related to the Milvus vector database.\n", - "\n", - "## Setup\n", - "\n", - "You'll need to install `langchain-milvus` with `pip install -qU langchain-milvus` to use this integration.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a62cff8a-bcf7-4e33-bbbc-76999c2e3e20", - "metadata": { - "id": "a62cff8a-bcf7-4e33-bbbc-76999c2e3e20", - "tags": [] - }, - "outputs": [], - "source": [ - "%pip install -qU langchain_milvus" - ] - }, - { - "cell_type": "markdown", - "id": "633addc3", - "metadata": { - "id": "633addc3" - }, - "source": [ - "The latest version of `pymilvus` comes with a local vector database called Milvus Lite, which is good for prototyping. If you have a large amount of data (e.g., more than a million vectors), we recommend setting up a more performant Milvus server on [Docker](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus) or [Kubernetes](https://milvus.io/docs/install_cluster-milvusoperator.md).\n", - "\n", - "### Credentials\n", - "\n", - "No credentials are needed to use the `Milvus` vector store.\n", - "\n", - "## Initialization\n", - "\n", - "import EmbeddingTabs from \"@theme/EmbeddingTabs\";\n", - "\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a7dd253f", - "metadata": { - "id": "a7dd253f" - }, - "outputs": [], - "source": [ - "# | output: false\n", - "# | echo: false\n", - "from langchain_openai import OpenAIEmbeddings\n", - "\n", - "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "dcf88bdf", - "metadata": { - "id": "dcf88bdf", - "tags": [] - }, - "outputs": [], - "source": [ - "from langchain_milvus import Milvus\n", - "\n", - "# The easiest way is to use Milvus Lite where everything is stored in a local file.\n", - "# If you have a Milvus server you can use the server URI such as \"http://localhost:19530\".\n", - "URI = \"./milvus_example.db\"\n", - "\n", - "vector_store = Milvus(\n", - " embedding_function=embeddings,\n", - " connection_args={\"uri\": URI},\n", - " # Set index_params if needed\n", - " index_params={\"index_type\": \"FLAT\", \"metric_type\": \"L2\"},\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "cae1a7d5", - "metadata": { - "id": "cae1a7d5" - }, - "source": [ - "### Compartmentalize the data with Milvus Collections\n", - "\n", - "You can store unrelated documents in different collections within the same Milvus instance." - ] - }, - { - "cell_type": "markdown", - "id": "c07cd24b", - "metadata": { - "id": "c07cd24b" - }, - "source": [ - "Here's how you can create a new collection:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c6f4973d", - "metadata": { - "id": "c6f4973d" - }, - "outputs": [], - "source": [ - "from langchain_core.documents import Document\n", - "\n", - "vector_store_saved = Milvus.from_documents(\n", - " [Document(page_content=\"foo!\")],\n", - " embeddings,\n", - " collection_name=\"langchain_example\",\n", - " connection_args={\"uri\": URI},\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "3b12df8c", - "metadata": { - "id": "3b12df8c" - }, - "source": [ - "And here is how you retrieve that stored collection:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "12817d16", - "metadata": { - "id": "12817d16" - }, - "outputs": [], - "source": [ - "vector_store_loaded = Milvus(\n", - " embeddings,\n", - " connection_args={\"uri\": URI},\n", - " collection_name=\"langchain_example\",\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "f1fc3818", - "metadata": { - "id": "f1fc3818" - }, - "source": [ - "## Manage vector store\n", - "\n", - "Once you have created your vector store, we can interact with it by adding and deleting different items.\n", - "\n", - "### Add items to vector store\n", - "\n", - "We can add items to our vector store by using the `add_documents` function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3ced24f6", - "metadata": { - "id": "3ced24f6", - "outputId": "9c57a6bb-86eb-456c-f007-6cabd6865299" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "['b0248595-2a41-4f6b-9c25-3a24c1278bb3',\n", - " 'fa642726-5329-4495-a072-187e948dd71f',\n", - " '9905001c-a4a3-455e-ab94-72d0ed11b476',\n", - " 'eacc7256-d7fa-4036-b1f7-83d7a4bee0c5',\n", - " '7508f7ff-c0c9-49ea-8189-634f8a0244d8',\n", - " '2e179609-3ff7-4c6a-9e05-08978903fe26',\n", - " 'fab1f2ac-43e1-45f9-b81b-fc5d334c6508',\n", - " '1206d237-ee3a-484f-baf2-b5ac38eeb314',\n", - " 'd43cbf9a-a772-4c40-993b-9439065fec01',\n", - " '25e667bb-6f09-4574-a368-661069301906']" - ] - }, - "execution_count": 31, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from uuid import uuid4\n", - "\n", - "from langchain_core.documents import Document\n", - "\n", - "document_1 = Document(\n", - " page_content=\"I had chocalate chip pancakes and scrambled eggs for breakfast this morning.\",\n", - " metadata={\"source\": \"tweet\"},\n", - ")\n", - "\n", - "document_2 = Document(\n", - " page_content=\"The weather forecast for tomorrow is cloudy and overcast, with a high of 62 degrees.\",\n", - " metadata={\"source\": \"news\"},\n", - ")\n", - "\n", - "document_3 = Document(\n", - " page_content=\"Building an exciting new project with LangChain - come check it out!\",\n", - " metadata={\"source\": \"tweet\"},\n", - ")\n", - "\n", - "document_4 = Document(\n", - " page_content=\"Robbers broke into the city bank and stole $1 million in cash.\",\n", - " metadata={\"source\": \"news\"},\n", - ")\n", - "\n", - "document_5 = Document(\n", - " page_content=\"Wow! That was an amazing movie. I can't wait to see it again.\",\n", - " metadata={\"source\": \"tweet\"},\n", - ")\n", - "\n", - "document_6 = Document(\n", - " page_content=\"Is the new iPhone worth the price? Read this review to find out.\",\n", - " metadata={\"source\": \"website\"},\n", - ")\n", - "\n", - "document_7 = Document(\n", - " page_content=\"The top 10 soccer players in the world right now.\",\n", - " metadata={\"source\": \"website\"},\n", - ")\n", - "\n", - "document_8 = Document(\n", - " page_content=\"LangGraph is the best framework for building stateful, agentic applications!\",\n", - " metadata={\"source\": \"tweet\"},\n", - ")\n", - "\n", - "document_9 = Document(\n", - " page_content=\"The stock market is down 500 points today due to fears of a recession.\",\n", - " metadata={\"source\": \"news\"},\n", - ")\n", - "\n", - "document_10 = Document(\n", - " page_content=\"I have a bad feeling I am going to get deleted :(\",\n", - " metadata={\"source\": \"tweet\"},\n", - ")\n", - "\n", - "documents = [\n", - " document_1,\n", - " document_2,\n", - " document_3,\n", - " document_4,\n", - " document_5,\n", - " document_6,\n", - " document_7,\n", - " document_8,\n", - " document_9,\n", - " document_10,\n", - "]\n", - "uuids = [str(uuid4()) for _ in range(len(documents))]\n", - "\n", - "vector_store.add_documents(documents=documents, ids=uuids)" - ] - }, - { - "cell_type": "markdown", - "id": "e23c22d8", - "metadata": { - "id": "e23c22d8" - }, - "source": [ - "### Delete items from vector store" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1f387fa8", - "metadata": { - "id": "1f387fa8", - "outputId": "62fee30d-92c9-4efd-df8a-453545ff61d0" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "(insert count: 0, delete count: 1, upsert count: 0, timestamp: 0, success count: 0, err count: 0, cost: 0)" - ] - }, - "execution_count": 32, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "vector_store.delete(ids=[uuids[-1]])" - ] - }, - { - "cell_type": "markdown", - "id": "fb12fa75", - "metadata": { - "id": "fb12fa75" - }, - "source": [ - "## Query vector store\n", - "\n", - "Once your vector store has been created and the relevant documents have been added, you will most likely wish to query it during the running of your chain or agent.\n", - "\n", - "### Query directly\n", - "\n", - "#### Similarity search\n", - "\n", - "Performing a simple similarity search with filtering on metadata can be done as follows:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "35801a55", - "metadata": { - "id": "35801a55", - "outputId": "13865abb-11a2-41ae-9ad7-44e8586fd099" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "* Building an exciting new project with LangChain - come check it out! [{'pk': '9905001c-a4a3-455e-ab94-72d0ed11b476', 'source': 'tweet'}]\n", - "* LangGraph is the best framework for building stateful, agentic applications! [{'pk': '1206d237-ee3a-484f-baf2-b5ac38eeb314', 'source': 'tweet'}]\n" - ] - } - ], - "source": [ - "results = vector_store.similarity_search(\n", - " \"LangChain provides abstractions to make working with LLMs easy\",\n", - " k=2,\n", - " expr='source == \"tweet\"',\n", - ")\n", - "for res in results:\n", - " print(f\"* {res.page_content} [{res.metadata}]\")" - ] - }, - { - "cell_type": "markdown", - "id": "35574409", - "metadata": { - "id": "35574409" - }, - "source": [ - "#### Similarity search with score\n", - "\n", - "You can also search with score:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c360af3d", - "metadata": { - "id": "c360af3d", - "outputId": "16cb1961-9f4a-494a-9500-27b98a1158d8" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "* [SIM=21192.628906] bar [{'pk': '2', 'source': 'https://example.com'}]\n" - ] - } - ], - "source": [ - "results = vector_store.similarity_search_with_score(\n", - " \"Will it be hot tomorrow?\", k=1, expr='source == \"news\"'\n", - ")\n", - "for res, score in results:\n", - " print(f\"* [SIM={score:3f}] {res.page_content} [{res.metadata}]\")" - ] - }, - { - "cell_type": "markdown", - "id": "14db337f", - "metadata": { - "id": "14db337f" - }, - "source": [ - "For a full list of all the search options available when using the `Milvus` vector store, you can visit the [API reference](https://python.langchain.com/api_reference/milvus/vectorstores/langchain_milvus.vectorstores.milvus.Milvus.html).\n", - "\n", - "### Query by turning into retriever\n", - "\n", - "You can also transform the vector store into a retriever for easier usage in your chains." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f6d9357c", - "metadata": { - "id": "f6d9357c", - "outputId": "bcaa7620-a1c0-418f-9f54-684a472b0b55" - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[Document(metadata={'pk': 'eacc7256-d7fa-4036-b1f7-83d7a4bee0c5', 'source': 'news'}, page_content='Robbers broke into the city bank and stole $1 million in cash.')]" - ] - }, - "execution_count": 34, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "retriever = vector_store.as_retriever(search_type=\"mmr\", search_kwargs={\"k\": 1})\n", - "retriever.invoke(\"Stealing from the bank is a crime\", filter={\"source\": \"news\"})" - ] - }, - { - "cell_type": "markdown", - "id": "8ac953f1", - "metadata": { - "id": "8ac953f1" - }, - "source": [ - "## Usage for retrieval-augmented generation\n", - "\n", - "For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:\n", - "\n", - "- [Tutorials](/docs/tutorials/)\n", - "- [How-to: Question and answer with RAG](https://python.langchain.com/docs/how_to/#qa-with-rag)\n", - "- [Retrieval conceptual docs](https://python.langchain.com/docs/concepts/retrieval)" - ] - }, - { - "cell_type": "markdown", - "id": "7fb27b941602401d91542211134fc71a", - "metadata": { - "id": "7fb27b941602401d91542211134fc71a", - "pycharm": { - "name": "#%% md\n" - } - }, - "source": [ - "### Per-User Retrieval\n", - "\n", - "When building a retrieval app, you often have to build it with multiple users in mind. This means that you may be storing data not just for one user, but for many different users, and they should not be able to see each other’s data.\n", - "\n", - "Milvus recommends using [partition_key](https://milvus.io/docs/multi_tenancy.md#Partition-key-based-multi-tenancy) to implement multi-tenancy. Here is an example:\n", - "> The feature of Partition key is now not available in Milvus Lite, if you want to use it, you need to start Milvus server, as mentioned above." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "acae54e37e7d407bbb7b55eff062a284", - "metadata": { - "id": "acae54e37e7d407bbb7b55eff062a284", - "pycharm": { - "name": "#%%\n" - } - }, - "outputs": [], - "source": [ - "from langchain_core.documents import Document\n", - "\n", - "docs = [\n", - " Document(page_content=\"i worked at kensho\", metadata={\"namespace\": \"harrison\"}),\n", - " Document(page_content=\"i worked at facebook\", metadata={\"namespace\": \"ankush\"}),\n", - "]\n", - "vectorstore = Milvus.from_documents(\n", - " docs,\n", - " embeddings,\n", - " connection_args={\"uri\": URI},\n", - " drop_old=True,\n", - " partition_key_field=\"namespace\", # Use the \"namespace\" field as the partition key\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "9a63283cbaf04dbcab1f6479b197f3a8", - "metadata": { - "id": "9a63283cbaf04dbcab1f6479b197f3a8", - "pycharm": { - "name": "#%% md\n" - } - }, - "source": [ - "To conduct a search using the partition key, you should include either of the following in the boolean expression of the search request:\n", - "\n", - "`search_kwargs={\"expr\": ' == \"xxxx\"'}`\n", - "\n", - "`search_kwargs={\"expr\": ' == in [\"xxx\", \"xxx\"]'}`\n", - "\n", - "Do replace `` with the name of the field that is designated as the partition key.\n", - "\n", - "Milvus changes to a partition based on the specified partition key, filters entities according to the partition key, and searches among the filtered entities.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8dd0d8092fe74a7c96281538738b07e2", - "metadata": { - "id": "8dd0d8092fe74a7c96281538738b07e2", - "outputId": "e38ff0ea-1425-4f12-cfb5-7767d040397b", - "pycharm": { - "name": "#%%\n" - } - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[Document(page_content='i worked at facebook', metadata={'namespace': 'ankush'})]" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# This will only get documents for Ankush\n", - "vectorstore.as_retriever(search_kwargs={\"expr\": 'namespace == \"ankush\"'}).invoke(\n", - " \"where did i work?\"\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "72eea5119410473aa328ad9291626812", - "metadata": { - "id": "72eea5119410473aa328ad9291626812", - "outputId": "9d3ad63e-fcb9-4f9a-bdf1-1bc263ce832b", - "pycharm": { - "name": "#%%\n" - } - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[Document(page_content='i worked at kensho', metadata={'namespace': 'harrison'})]" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# This will only get documents for Harrison\n", - "vectorstore.as_retriever(search_kwargs={\"expr\": 'namespace == \"harrison\"'}).invoke(\n", - " \"where did i work?\"\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "f1a873c5", - "metadata": { - "id": "f1a873c5" - }, - "source": [ - "## API reference\n", - "\n", - "For detailed documentation of all __ModuleName__VectorStore features and configurations head to the API reference: https://python.langchain.com/api_reference/milvus/vectorstores/langchain_milvus.vectorstores.milvus.Milvus.html" - ] - } - ], - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.9" - } + "cells": [ + { + "cell_type": "markdown", + "id": "683953b3", + "metadata": { + "id": "683953b3" + }, + "source": [ + "# Milvus\n", + "\n", + ">[Milvus](https://milvus.io/docs/overview.md) is a database that stores, indexes, and manages massive embedding vectors generated by deep neural networks and other machine learning (ML) models.\n", + "\n", + "This notebook shows how to use functionality related to the Milvus vector database.\n", + "\n", + "## Setup\n", + "\n", + "You'll need to install `langchain-milvus` with `pip install -qU langchain-milvus` to use this integration.\n" + ] }, - "nbformat": 4, - "nbformat_minor": 5 + { + "cell_type": "code", + "execution_count": 1, + "id": "a62cff8a-bcf7-4e33-bbbc-76999c2e3e20", + "metadata": { + "id": "a62cff8a-bcf7-4e33-bbbc-76999c2e3e20", + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -qU langchain_milvus" + ] + }, + { + "cell_type": "markdown", + "id": "dfd17253", + "metadata": {}, + "source": [ + "### Credentials\n", + "\n", + "No credentials are needed to use the `Milvus` vector store." + ] + }, + { + "cell_type": "markdown", + "id": "633addc3", + "metadata": { + "id": "633addc3" + }, + "source": [ + "## Initialization\n", + "\n", + "import EmbeddingTabs from \"@theme/EmbeddingTabs\";\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "a7dd253f", + "metadata": { + "id": "a7dd253f" + }, + "outputs": [], + "source": [ + "# | output: false\n", + "# | echo: false\n", + "from langchain_openai import OpenAIEmbeddings\n", + "\n", + "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")" + ] + }, + { + "cell_type": "markdown", + "id": "50e55d42", + "metadata": {}, + "source": [ + "### Milvus Lite\n", + "\n", + "The easiest way to prototype is to use Milvus Lite, where everything is stored in a local vector database file. Only the Flat index can be used." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "dcf88bdf", + "metadata": { + "id": "dcf88bdf", + "tags": [] + }, + "outputs": [], + "source": [ + "from langchain_milvus import Milvus\n", + "\n", + "URI = \"./milvus_example.db\"\n", + "\n", + "vector_store = Milvus(\n", + " embedding_function=embeddings,\n", + " connection_args={\"uri\": URI},\n", + " index_params={\"index_type\": \"FLAT\", \"metric_type\": \"L2\"},\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "df34e8f4", + "metadata": {}, + "source": [ + "### Milvus Standalone\n", + "\n", + "If you have a large amount of data (e.g., more than a million vectors), we recommend setting up a more performant Milvus server on [Docker](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus) or [Kubernetes](https://milvus.io/docs/install_cluster-milvusoperator.md).\n", + "\n", + "Milvus Standalone also supports different [indexes](https://milvus.io/docs/index.md?tab=floating), if you want to improve retrieval functionality.\n", + "\n", + "To launch the Docker container, run:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0fddc9a6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Password:" + ] + } + ], + "source": [ + "!curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh\n", + "\n", + "!bash standalone_embed.sh start" + ] + }, + { + "cell_type": "markdown", + "id": "9045ef4a", + "metadata": {}, + "source": [ + "Here we create a Milvus database:" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "fcff1834", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Database 'milvus_demo' does not exist.\n", + "Database 'milvus_demo' created successfully.\n" + ] + } + ], + "source": [ + "from pymilvus import Collection, MilvusException, connections, db, utility\n", + "\n", + "conn = connections.connect(host=\"127.0.0.1\", port=19530)\n", + "\n", + "# Check if the database exists\n", + "db_name = \"milvus_demo\"\n", + "try:\n", + " existing_databases = db.list_database()\n", + " if db_name in existing_databases:\n", + " print(f\"Database '{db_name}' already exists.\")\n", + "\n", + " # Use the database context\n", + " db.using_database(db_name)\n", + "\n", + " # Drop all collections in the database\n", + " collections = utility.list_collections()\n", + " for collection_name in collections:\n", + " collection = Collection(name=collection_name)\n", + " collection.drop()\n", + " print(f\"Collection '{collection_name}' has been dropped.\")\n", + "\n", + " db.drop_database(db_name)\n", + " print(f\"Database '{db_name}' has been deleted.\")\n", + " else:\n", + " print(f\"Database '{db_name}' does not exist.\")\n", + " database = db.create_database(db_name)\n", + " print(f\"Database '{db_name}' created successfully.\")\n", + "except MilvusException as e:\n", + " print(f\"An error occurred: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b50e3ff7", + "metadata": {}, + "source": [ + "Note the change in the URI below. Once the instance is initialized, navigate to http://127.0.0.1:9091/webui to view the local web UI.\n", + "\n", + "Here is an example of how you would use a dense embedding + the Milvus BM25 built-in function to assemble a hybrid retrieval vector store instance:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "07460732", + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_milvus import BM25BuiltInFunction, Milvus\n", + "\n", + "dense_index_param = {\n", + " \"metric_type\": \"COSINE\",\n", + " \"index_type\": \"HNSW\",\n", + "}\n", + "sparse_index_param = {\n", + " \"metric_type\": \"BM25\",\n", + " \"index_type\": \"AUTOINDEX\",\n", + "}\n", + "\n", + "URI = \"http://localhost:19530\"\n", + "\n", + "vectorstore = Milvus(\n", + " embedding_function=embeddings,\n", + " builtin_function=BM25BuiltInFunction(output_field_names=\"sparse\"),\n", + " index_params=[dense_index_param, sparse_index_param],\n", + " vector_field=[\"dense\", \"sparse\"],\n", + " connection_args={\"uri\": URI, \"token\": \"root:Milvus\", \"db_name\": \"milvus_demo\"},\n", + " consistency_level=\"Strong\",\n", + " drop_old=False, # set to True if seeking to drop the collection with that name if it exists\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "cae1a7d5", + "metadata": { + "id": "cae1a7d5" + }, + "source": [ + "### Compartmentalize the data with Milvus Collections\n", + "\n", + "You can store unrelated documents in different collections within the same Milvus instance.\n", + "\n", + "Here's how you can create a new collection:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c6f4973d", + "metadata": { + "id": "c6f4973d" + }, + "outputs": [], + "source": [ + "from langchain_core.documents import Document\n", + "\n", + "vector_store_saved = Milvus.from_documents(\n", + " [Document(page_content=\"foo!\")],\n", + " embeddings,\n", + " collection_name=\"langchain_example\",\n", + " connection_args={\"uri\": URI},\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "3b12df8c", + "metadata": { + "id": "3b12df8c" + }, + "source": [ + "And here is how you retrieve that stored collection:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12817d16", + "metadata": { + "id": "12817d16" + }, + "outputs": [], + "source": [ + "vector_store_loaded = Milvus(\n", + " embeddings,\n", + " connection_args={\"uri\": URI},\n", + " collection_name=\"langchain_example\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "f1fc3818", + "metadata": { + "id": "f1fc3818" + }, + "source": [ + "## Manage vector store\n", + "\n", + "Once you have created your vector store, we can interact with it by adding and deleting different items.\n", + "\n", + "### Add items to vector store\n", + "\n", + "We can add items to our vector store by using the `add_documents` function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3ced24f6", + "metadata": { + "id": "3ced24f6", + "outputId": "9c57a6bb-86eb-456c-f007-6cabd6865299" + }, + "outputs": [], + "source": [ + "from uuid import uuid4\n", + "\n", + "from langchain_core.documents import Document\n", + "\n", + "document_1 = Document(\n", + " page_content=\"I had chocolate chip pancakes and scrambled eggs for breakfast this morning.\",\n", + " metadata={\"source\": \"tweet\"},\n", + ")\n", + "\n", + "document_2 = Document(\n", + " page_content=\"The weather forecast for tomorrow is cloudy and overcast, with a high of 62 degrees.\",\n", + " metadata={\"source\": \"news\"},\n", + ")\n", + "\n", + "document_3 = Document(\n", + " page_content=\"Building an exciting new project with LangChain - come check it out!\",\n", + " metadata={\"source\": \"tweet\"},\n", + ")\n", + "\n", + "document_4 = Document(\n", + " page_content=\"Robbers broke into the city bank and stole $1 million in cash.\",\n", + " metadata={\"source\": \"news\"},\n", + ")\n", + "\n", + "document_5 = Document(\n", + " page_content=\"Wow! That was an amazing movie. I can't wait to see it again.\",\n", + " metadata={\"source\": \"tweet\"},\n", + ")\n", + "\n", + "document_6 = Document(\n", + " page_content=\"Is the new iPhone worth the price? Read this review to find out.\",\n", + " metadata={\"source\": \"website\"},\n", + ")\n", + "\n", + "document_7 = Document(\n", + " page_content=\"The top 10 soccer players in the world right now.\",\n", + " metadata={\"source\": \"website\"},\n", + ")\n", + "\n", + "document_8 = Document(\n", + " page_content=\"LangGraph is the best framework for building stateful, agentic applications!\",\n", + " metadata={\"source\": \"tweet\"},\n", + ")\n", + "\n", + "document_9 = Document(\n", + " page_content=\"The stock market is down 500 points today due to fears of a recession.\",\n", + " metadata={\"source\": \"news\"},\n", + ")\n", + "\n", + "document_10 = Document(\n", + " page_content=\"I have a bad feeling I am going to get deleted :(\",\n", + " metadata={\"source\": \"tweet\"},\n", + ")\n", + "\n", + "documents = [\n", + " document_1,\n", + " document_2,\n", + " document_3,\n", + " document_4,\n", + " document_5,\n", + " document_6,\n", + " document_7,\n", + " document_8,\n", + " document_9,\n", + " document_10,\n", + "]\n", + "uuids = [str(uuid4()) for _ in range(len(documents))]\n", + "\n", + "vector_store.add_documents(documents=documents, ids=uuids)" + ] + }, + { + "cell_type": "markdown", + "id": "e23c22d8", + "metadata": { + "id": "e23c22d8" + }, + "source": [ + "### Delete items from vector store" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1f387fa8", + "metadata": { + "id": "1f387fa8", + "outputId": "62fee30d-92c9-4efd-df8a-453545ff61d0" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "(insert count: 0, delete count: 1, upsert count: 0, timestamp: 0, success count: 0, err count: 0, cost: 0)" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "vector_store.delete(ids=[uuids[-1]])" + ] + }, + { + "cell_type": "markdown", + "id": "fb12fa75", + "metadata": { + "id": "fb12fa75" + }, + "source": [ + "## Query vector store\n", + "\n", + "Once your vector store has been created and the relevant documents have been added, you will most likely wish to query it during the running of your chain or agent.\n", + "\n", + "### Query directly\n", + "\n", + "#### Similarity search\n", + "\n", + "Performing a simple similarity search with filtering on metadata can be done as follows:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "35801a55", + "metadata": { + "id": "35801a55", + "outputId": "13865abb-11a2-41ae-9ad7-44e8586fd099" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "* Building an exciting new project with LangChain - come check it out! [{'pk': '9905001c-a4a3-455e-ab94-72d0ed11b476', 'source': 'tweet'}]\n", + "* LangGraph is the best framework for building stateful, agentic applications! [{'pk': '1206d237-ee3a-484f-baf2-b5ac38eeb314', 'source': 'tweet'}]\n" + ] + } + ], + "source": [ + "results = vector_store.similarity_search(\n", + " \"LangChain provides abstractions to make working with LLMs easy\",\n", + " k=2,\n", + " expr='source == \"tweet\"',\n", + ")\n", + "for res in results:\n", + " print(f\"* {res.page_content} [{res.metadata}]\")" + ] + }, + { + "cell_type": "markdown", + "id": "35574409", + "metadata": { + "id": "35574409" + }, + "source": [ + "#### Similarity search with score\n", + "\n", + "You can also search with score:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c360af3d", + "metadata": { + "id": "c360af3d", + "outputId": "16cb1961-9f4a-494a-9500-27b98a1158d8" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "* [SIM=21192.628906] bar [{'pk': '2', 'source': 'https://example.com'}]\n" + ] + } + ], + "source": [ + "results = vector_store.similarity_search_with_score(\n", + " \"Will it be hot tomorrow?\", k=1, expr='source == \"news\"'\n", + ")\n", + "for res, score in results:\n", + " print(f\"* [SIM={score:3f}] {res.page_content} [{res.metadata}]\")" + ] + }, + { + "cell_type": "markdown", + "id": "14db337f", + "metadata": { + "id": "14db337f" + }, + "source": [ + "For a full list of all the search options available when using the `Milvus` vector store, you can visit the [API reference](https://python.langchain.com/api_reference/milvus/vectorstores/langchain_milvus.vectorstores.milvus.Milvus.html).\n", + "\n", + "### Query by turning into retriever\n", + "\n", + "You can also transform the vector store into a retriever for easier usage in your chains." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6d9357c", + "metadata": { + "id": "f6d9357c", + "outputId": "bcaa7620-a1c0-418f-9f54-684a472b0b55" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(metadata={'pk': 'eacc7256-d7fa-4036-b1f7-83d7a4bee0c5', 'source': 'news'}, page_content='Robbers broke into the city bank and stole $1 million in cash.')]" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "retriever = vector_store.as_retriever(search_type=\"mmr\", search_kwargs={\"k\": 1})\n", + "retriever.invoke(\"Stealing from the bank is a crime\", filter={\"source\": \"news\"})" + ] + }, + { + "cell_type": "markdown", + "id": "8ac953f1", + "metadata": { + "id": "8ac953f1" + }, + "source": [ + "## Usage for retrieval-augmented generation\n", + "\n", + "For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:\n", + "\n", + "- [Tutorials](/docs/tutorials/)\n", + "- [How-to: Question and answer with RAG](https://python.langchain.com/docs/how_to/#qa-with-rag)\n", + "- [Retrieval conceptual docs](https://python.langchain.com/docs/concepts/retrieval)" + ] + }, + { + "cell_type": "markdown", + "id": "7fb27b941602401d91542211134fc71a", + "metadata": { + "id": "7fb27b941602401d91542211134fc71a", + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "### Per-User Retrieval\n", + "\n", + "When building a retrieval app, you often have to build it with multiple users in mind. This means that you may be storing data not just for one user, but for many different users, and they should not be able to see each other’s data.\n", + "\n", + "Milvus recommends using [partition_key](https://milvus.io/docs/multi_tenancy.md#Partition-key-based-multi-tenancy) to implement multi-tenancy. Here is an example:\n", + "> The Partition key feature is not available in Milvus Lite, if you want to use it, you need to start Milvus server, as mentioned above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acae54e37e7d407bbb7b55eff062a284", + "metadata": { + "id": "acae54e37e7d407bbb7b55eff062a284", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "from langchain_core.documents import Document\n", + "\n", + "docs = [\n", + " Document(page_content=\"i worked at kensho\", metadata={\"namespace\": \"harrison\"}),\n", + " Document(page_content=\"i worked at facebook\", metadata={\"namespace\": \"ankush\"}),\n", + "]\n", + "vectorstore = Milvus.from_documents(\n", + " docs,\n", + " embeddings,\n", + " connection_args={\"uri\": URI},\n", + " drop_old=True,\n", + " partition_key_field=\"namespace\", # Use the \"namespace\" field as the partition key\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "metadata": { + "id": "9a63283cbaf04dbcab1f6479b197f3a8", + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "To conduct a search using the partition key, you should include either of the following in the boolean expression of the search request:\n", + "\n", + "`search_kwargs={\"expr\": ' == \"xxxx\"'}`\n", + "\n", + "`search_kwargs={\"expr\": ' == in [\"xxx\", \"xxx\"]'}`\n", + "\n", + "Do replace `` with the name of the field that is designated as the partition key.\n", + "\n", + "Milvus changes to a partition based on the specified partition key, filters entities according to the partition key, and searches among the filtered entities.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8dd0d8092fe74a7c96281538738b07e2", + "metadata": { + "id": "8dd0d8092fe74a7c96281538738b07e2", + "outputId": "e38ff0ea-1425-4f12-cfb5-7767d040397b", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(page_content='i worked at facebook', metadata={'namespace': 'ankush'})]" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# This will only get documents for Ankush\n", + "vectorstore.as_retriever(search_kwargs={\"expr\": 'namespace == \"ankush\"'}).invoke(\n", + " \"where did i work?\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "72eea5119410473aa328ad9291626812", + "metadata": { + "id": "72eea5119410473aa328ad9291626812", + "outputId": "9d3ad63e-fcb9-4f9a-bdf1-1bc263ce832b", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(page_content='i worked at kensho', metadata={'namespace': 'harrison'})]" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# This will only get documents for Harrison\n", + "vectorstore.as_retriever(search_kwargs={\"expr\": 'namespace == \"harrison\"'}).invoke(\n", + " \"where did i work?\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "f1a873c5", + "metadata": { + "id": "f1a873c5" + }, + "source": [ + "## API reference\n", + "\n", + "For detailed documentation of all __ModuleName__VectorStore features and configurations head to the API reference: https://python.langchain.com/api_reference/milvus/vectorstores/langchain_milvus.vectorstores.milvus.Milvus.html" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 }