mirror of
https://github.com/hwchase17/langchain.git
synced 2025-06-05 06:33:20 +00:00
d0603c86b6
This PR adds support for the Azure Cosmos DB MongoDB vCore Vector Store https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/ https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search Summary: - **Description:** added vector store integration for Azure Cosmos DB MongoDB vCore Vector Store, - **Issue:** the issue # it fixes #11627, - **Dependencies:** pymongo dependency, - **Tag maintainer:** @hwchase17, - **Twitter handle:** @izzyacademy --------- Co-authored-by: Israel Ekpo <israel.ekpo@gmail.com> Co-authored-by: Israel Ekpo <44282278+izzyacademy@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>
363 lines
13 KiB
Plaintext
363 lines
13 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"# Azure Cosmos DB\n",
|
||
"\n",
|
||
">[Azure Cosmos DB for MongoDB vCore](https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/) makes it easy to create a database with full native MongoDB support.\n",
|
||
"> You can apply your MongoDB experience and continue to use your favorite MongoDB drivers, SDKs, and tools by pointing your application to the API for MongoDB vCore account's connection string.\n",
|
||
"> Use vector search in Azure Cosmos DB for MongoDB vCore to seamlessly integrate your AI-based applications with your data that's stored in Azure Cosmos DB.\n",
|
||
"\n",
|
||
"This notebook shows you how to leverage the [Vector Search](https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search) capabilities within Azure Cosmos DB for Mongo vCore to store documents in collections, create indicies and perform vector search queries using approximate nearest neighbor algorithms such as COS (cosine distance), L2 (Euclidean distance), and IP (inner product) to locate documents close to the query vectors. \n",
|
||
" \n",
|
||
"Azure Cosmos DB for MongoDB vCore provides developers with a fully managed MongoDB-compatible database service for building modern applications with a familiar architecture.\n",
|
||
"\n",
|
||
"With Cosmos DB for MongoDB vCore, developers can enjoy the benefits of native Azure integrations, low total cost of ownership (TCO), and the familiar vCore architecture when migrating existing applications or building new ones.\n",
|
||
"\n",
|
||
"[Sign Up](https://azure.microsoft.com/en-us/free/) for free to get started today.\n",
|
||
" "
|
||
],
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"id": "245c0aa70db77606"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Requirement already satisfied: pymongo in /Users/iekpo/Langchain/langchain-python/.venv/lib/python3.10/site-packages (4.5.0)\r\n",
|
||
"Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in /Users/iekpo/Langchain/langchain-python/.venv/lib/python3.10/site-packages (from pymongo) (2.4.2)\r\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!pip install pymongo"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:20:00.721985Z",
|
||
"start_time": "2023-10-10T17:19:57.996265Z"
|
||
}
|
||
},
|
||
"id": "ab8e45f5bd435ade"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 24,
|
||
"outputs": [],
|
||
"source": [
|
||
"import os\n",
|
||
"import getpass\n",
|
||
"\n",
|
||
"CONNECTION_STRING = \"AZURE COSMOS DB MONGO vCORE connection string\"\n",
|
||
"INDEX_NAME = \"izzy-test-index\"\n",
|
||
"NAMESPACE = \"izzy_test_db.izzy_test_collection\"\n",
|
||
"DB_NAME, COLLECTION_NAME = NAMESPACE.split(\".\")"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:50:03.615234Z",
|
||
"start_time": "2023-10-10T17:50:03.604289Z"
|
||
}
|
||
},
|
||
"id": "9c7ce9e7b26efbb0"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"We want to use `OpenAIEmbeddings` so we need to set up our Azure OpenAI API Key alongside other environment variables. "
|
||
],
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"id": "f2e66b097c6ce2e3"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 25,
|
||
"outputs": [],
|
||
"source": [
|
||
"# Set up the OpenAI Environment Variables\n",
|
||
"os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n",
|
||
"os.environ[\"OPENAI_API_VERSION\"] = \"2023-05-15\"\n",
|
||
"os.environ[\"OPENAI_API_BASE\"] = \"YOUR_OPEN_AI_ENDPOINT\" # https://example.openai.azure.com/\n",
|
||
"os.environ[\"OPENAI_API_KEY\"] = \"YOUR_OPEN_AI_KEY\"\n",
|
||
"os.environ[\"OPENAI_EMBEDDINGS_DEPLOYMENT\"] = \"smart-agent-embedding-ada\" # the deployment name for the embedding model\n",
|
||
"os.environ[\"OPENAI_EMBEDDINGS_MODEL_NAME\"] = \"text-embedding-ada-002\" # the model name\n"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:50:11.712929Z",
|
||
"start_time": "2023-10-10T17:50:11.703871Z"
|
||
}
|
||
},
|
||
"id": "4a052d99c6b8a2a7"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"Now, we need to load the documents into the collection, create the index and then run our queries against the index to retrieve matches.\n",
|
||
"\n",
|
||
"Please refer to the [documentation](https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search) if you have questions about certain parameters"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"id": "ebaa28c6e2b35063"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain.docstore.document import Document\n",
|
||
"from langchain.embeddings import OpenAIEmbeddings\n",
|
||
"from langchain.schema.embeddings import Embeddings\n",
|
||
"from langchain.vectorstores.azure_cosmos_db_vector_search import AzureCosmosDBVectorSearch, CosmosDBSimilarityType\n",
|
||
"from langchain.text_splitter import CharacterTextSplitter\n",
|
||
"from langchain.document_loaders import TextLoader\n",
|
||
"\n",
|
||
"SOURCE_FILE_NAME = \"../../modules/state_of_the_union.txt\"\n",
|
||
"\n",
|
||
"loader = TextLoader(SOURCE_FILE_NAME)\n",
|
||
"documents = loader.load()\n",
|
||
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
|
||
"docs = text_splitter.split_documents(documents)\n",
|
||
"\n",
|
||
"# OpenAI Settings\n",
|
||
"model_deployment = os.getenv(\"OPENAI_EMBEDDINGS_DEPLOYMENT\", \"smart-agent-embedding-ada\")\n",
|
||
"model_name = os.getenv(\"OPENAI_EMBEDDINGS_MODEL_NAME\", \"text-embedding-ada-002\")\n",
|
||
"\n",
|
||
"\n",
|
||
"openai_embeddings: OpenAIEmbeddings = OpenAIEmbeddings(deployment=model_deployment, model=model_name, chunk_size=1)"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:50:16.732718Z",
|
||
"start_time": "2023-10-10T17:50:16.716642Z"
|
||
}
|
||
},
|
||
"id": "183741cf8f4c7c53"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 28,
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": "{'raw': {'defaultShard': {'numIndexesBefore': 2,\n 'numIndexesAfter': 3,\n 'createdCollectionAutomatically': False,\n 'ok': 1}},\n 'ok': 1}"
|
||
},
|
||
"execution_count": 28,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from pymongo import MongoClient\n",
|
||
"\n",
|
||
"INDEX_NAME = \"izzy-test-index-2\"\n",
|
||
"NAMESPACE = \"izzy_test_db.izzy_test_collection\"\n",
|
||
"DB_NAME, COLLECTION_NAME = NAMESPACE.split(\".\")\n",
|
||
"\n",
|
||
"client: MongoClient = MongoClient(CONNECTION_STRING)\n",
|
||
"collection = client[DB_NAME][COLLECTION_NAME]\n",
|
||
"\n",
|
||
"model_deployment = os.getenv(\"OPENAI_EMBEDDINGS_DEPLOYMENT\", \"smart-agent-embedding-ada\")\n",
|
||
"model_name = os.getenv(\"OPENAI_EMBEDDINGS_MODEL_NAME\", \"text-embedding-ada-002\")\n",
|
||
"\n",
|
||
"vectorstore = AzureCosmosDBVectorSearch.from_documents(\n",
|
||
" docs,\n",
|
||
" openai_embeddings,\n",
|
||
" collection=collection,\n",
|
||
" index_name=INDEX_NAME,\n",
|
||
")\n",
|
||
"\n",
|
||
"num_lists = 100\n",
|
||
"dimensions = 1536\n",
|
||
"similarity_algorithm = CosmosDBSimilarityType.COS\n",
|
||
"\n",
|
||
"vectorstore.create_index(num_lists, dimensions, similarity_algorithm)\n"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:51:17.980698Z",
|
||
"start_time": "2023-10-10T17:51:11.786336Z"
|
||
}
|
||
},
|
||
"id": "39ae6058c2f7fdf1"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 29,
|
||
"outputs": [],
|
||
"source": [
|
||
"# perform a similarity search between the embedding of the query and the embeddings of the documents\n",
|
||
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
|
||
"docs = vectorstore.similarity_search(query)"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:51:44.840121Z",
|
||
"start_time": "2023-10-10T17:51:44.498639Z"
|
||
}
|
||
},
|
||
"id": "32c68d3246adc21f"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 31,
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
|
||
"\n",
|
||
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
|
||
"\n",
|
||
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
|
||
"\n",
|
||
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(docs[0].page_content)"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:52:08.049294Z",
|
||
"start_time": "2023-10-10T17:52:08.038511Z"
|
||
}
|
||
},
|
||
"id": "8feeeb4364efb204"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"Once the documents have been loaded and the index has been created, you can now instantiate the vector store directly and run queries against the index"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"id": "37e4df8c7d7db851"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 32,
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
|
||
"\n",
|
||
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
|
||
"\n",
|
||
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
|
||
"\n",
|
||
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"vectorstore = AzureCosmosDBVectorSearch.from_connection_string(CONNECTION_STRING, NAMESPACE, openai_embeddings, index_name=INDEX_NAME)\n",
|
||
"\n",
|
||
"# perform a similarity search between a query and the ingested documents\n",
|
||
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
|
||
"docs = vectorstore.similarity_search(query)\n",
|
||
"\n",
|
||
"print(docs[0].page_content)"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:52:14.994861Z",
|
||
"start_time": "2023-10-10T17:52:13.986379Z"
|
||
}
|
||
},
|
||
"id": "3c218ab6f59301f7"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 33,
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
|
||
"\n",
|
||
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
|
||
"\n",
|
||
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
|
||
"\n",
|
||
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"vectorstore = AzureCosmosDBVectorSearch(collection, openai_embeddings, index_name=INDEX_NAME)\n",
|
||
"\n",
|
||
"# perform a similarity search between a query and the ingested documents\n",
|
||
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
|
||
"docs = vectorstore.similarity_search(query)\n",
|
||
"\n",
|
||
"print(docs[0].page_content)"
|
||
],
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"ExecuteTime": {
|
||
"end_time": "2023-10-10T17:53:21.145431Z",
|
||
"start_time": "2023-10-10T17:53:20.884531Z"
|
||
}
|
||
},
|
||
"id": "fd67e4d92c9ab32f"
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"outputs": [],
|
||
"source": [],
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"id": "b63c73c7e905001c"
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 2
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython2",
|
||
"version": "2.7.6"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|