mirror of
https://github.com/hwchase17/langchain.git
synced 2025-05-11 18:16:12 +00:00
78ff51ce83
**Description:** This updates the langchain_community > huggingface > default bge embeddings ([the current default recommends this change](https://huggingface.co/BAAI/bge-large-en)) **Issue:** None **Dependencies:** None **Twitter handle:** @jonzeolla --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
303 lines
10 KiB
Plaintext
303 lines
10 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a3afefb0-7e99-4912-a222-c6b186da11af",
|
||
"metadata": {},
|
||
"source": [
|
||
"# SurrealDB\n",
|
||
"\n",
|
||
">[SurrealDB](https://surrealdb.com/) is an end-to-end cloud-native database designed for modern applications, including web, mobile, serverless, Jamstack, backend, and traditional applications. With SurrealDB, you can simplify your database and API infrastructure, reduce development time, and build secure, performant apps quickly and cost-effectively.\n",
|
||
">\n",
|
||
">**Key features of SurrealDB include:**\n",
|
||
">\n",
|
||
">* **Reduces development time:** SurrealDB simplifies your database and API stack by removing the need for most server-side components, allowing you to build secure, performant apps faster and cheaper.\n",
|
||
">* **Real-time collaborative API backend service:** SurrealDB functions as both a database and an API backend service, enabling real-time collaboration.\n",
|
||
">* **Support for multiple querying languages:** SurrealDB supports SQL querying from client devices, GraphQL, ACID transactions, WebSocket connections, structured and unstructured data, graph querying, full-text indexing, and geospatial querying.\n",
|
||
">* **Granular access control:** SurrealDB provides row-level permissions-based access control, giving you the ability to manage data access with precision.\n",
|
||
">\n",
|
||
">View the [features](https://surrealdb.com/features), the latest [releases](https://surrealdb.com/releases), and [documentation](https://surrealdb.com/docs).\n",
|
||
"\n",
|
||
"This notebook shows how to use functionality related to the `SurrealDBStore`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5031a3ec",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Setup\n",
|
||
"\n",
|
||
"Uncomment the below cells to install surrealdb."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7cd7391f-7759-4a21-952a-2ec972d818c6",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# %pip install --upgrade --quiet surrealdb langchain langchain-community"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6e57a389-f637-4b8f-9ab2-759ae7485f78",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Using SurrealDBStore"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "1c2d942d-5d90-4f9f-af96-dff976e4510f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# add this import for running in jupyter notebook\n",
|
||
"import nest_asyncio\n",
|
||
"\n",
|
||
"nest_asyncio.apply()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "e49be085-ddf1-4028-8c0c-97836ce4a873",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_community.document_loaders import TextLoader\n",
|
||
"from langchain_community.vectorstores import SurrealDBStore\n",
|
||
"from langchain_huggingface import HuggingFaceEmbeddings\n",
|
||
"from langchain_text_splitters import CharacterTextSplitter"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "38222aee-adc5-44c2-913c-97977b394cf5",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"documents = TextLoader(\"../../how_to/state_of_the_union.txt\").load()\n",
|
||
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
|
||
"docs = text_splitter.split_documents(documents)\n",
|
||
"\n",
|
||
"model_name = \"sentence-transformers/all-mpnet-base-v2\"\n",
|
||
"embeddings = HuggingFaceEmbeddings(model_name=model_name)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8e240306-803c-4c1a-b036-b9fc69eb6cba",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Creating a SurrealDBStore object"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "ff9d0304-1e11-4db2-9454-1350db7907e6",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"['documents:38hz49bv1p58f5lrvrdc',\n",
|
||
" 'documents:niayw63vzwm2vcbh6w2s',\n",
|
||
" 'documents:it1fa3ktplbuye43n0ch',\n",
|
||
" 'documents:il8f7vgbbp9tywmsn98c',\n",
|
||
" 'documents:vza4c6cqje0avqd58gal']"
|
||
]
|
||
},
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"db = SurrealDBStore(\n",
|
||
" dburl=\"ws://localhost:8000/rpc\", # url for the hosted SurrealDB database\n",
|
||
" embedding_function=embeddings,\n",
|
||
" db_user=\"root\", # SurrealDB credentials if needed: db username\n",
|
||
" db_pass=\"root\", # SurrealDB credentials if needed: db password\n",
|
||
" # ns=\"langchain\", # namespace to use for vectorstore\n",
|
||
" # db=\"database\", # database to use for vectorstore\n",
|
||
" # collection=\"documents\", #collection to use for vectorstore\n",
|
||
")\n",
|
||
"\n",
|
||
"# this is needed to initialize the underlying async library for SurrealDB\n",
|
||
"await db.initialize()\n",
|
||
"\n",
|
||
"# delete all existing documents from the vectorstore collection\n",
|
||
"await db.adelete()\n",
|
||
"\n",
|
||
"# add documents to the vectorstore\n",
|
||
"ids = await db.aadd_documents(docs)\n",
|
||
"\n",
|
||
"# document ids of the added documents\n",
|
||
"ids[:5]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "94a742a9-9507-4076-9cc3-616a4ed6866f",
|
||
"metadata": {},
|
||
"source": [
|
||
"### (alternatively) Create a SurrealDBStore object and add documents"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "73d66563-4e1f-4edf-9e95-5fc9adcfa2cb",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"await db.adelete()\n",
|
||
"\n",
|
||
"db = await SurrealDBStore.afrom_documents(\n",
|
||
" dburl=\"ws://localhost:8000/rpc\", # url for the hosted SurrealDB database\n",
|
||
" embedding=embeddings,\n",
|
||
" documents=docs,\n",
|
||
" db_user=\"root\", # SurrealDB credentials if needed: db username\n",
|
||
" db_pass=\"root\", # SurrealDB credentials if needed: db password\n",
|
||
" # ns=\"langchain\", # namespace to use for vectorstore\n",
|
||
" # db=\"database\", # database to use for vectorstore\n",
|
||
" # collection=\"documents\", #collection to use for vectorstore\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "efbb6684-3846-4332-a624-ddd4d75844c1",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Similarity search"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "aa28a7f8-41d0-4299-84eb-91d1576e8a63",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
|
||
"docs = await db.asimilarity_search(query)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "1eb16d2a-b466-456a-b412-5e74bb8523dd",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
|
||
"\n",
|
||
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
|
||
"\n",
|
||
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
|
||
"\n",
|
||
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(docs[0].page_content)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "43896697-f99e-47b6-9117-47a25e9afa9c",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Similarity search with score"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "414a9bc9",
|
||
"metadata": {},
|
||
"source": [
|
||
"The returned distance score is cosine distance. Therefore, a lower score is better."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "8e9eef05-1516-469a-ad36-880c69aef7a9",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"docs = await db.asimilarity_search_with_score(query)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "bd5fb0e4-2a94-4bb4-af8a-27327ecb1a7f",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', metadata={'id': 'documents:slgdlhjkfknhqo15xz0w', 'source': '../../how_to/state_of_the_union.txt'}),\n",
|
||
" 0.39839531721941895)"
|
||
]
|
||
},
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"docs[0]"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.10.12"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|