mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-06 05:25:04 +00:00
community: Added integrations for ThirdAI's NeuralDB with Retriever and VectorStore frameworks (#15280)
**Description:** Adds ThirdAI NeuralDB retriever and vectorstore integration. NeuralDB is a CPU-friendly and fine-tunable text retrieval engine.
This commit is contained in:
Benito Geordie
GitHub
160
docs/docs/integrations/vectorstores/thirdai_neuraldb.ipynb
Normal file
160
docs/docs/integrations/vectorstores/thirdai_neuraldb.ipynb
Normal file
@@ -0,0 +1,160 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# **NeuralDB**\n",
|
||||
"NeuralDB is a CPU-friendly and fine-tunable vector store developed by ThirdAI.\n",
|
||||
"\n",
|
||||
"### **Initialization**\n",
|
||||
"There are three initialization methods:\n",
|
||||
"- From Scratch: Basic model\n",
|
||||
"- From Bazaar: Download a pretrained base model from our model bazaar for better performance\n",
|
||||
"- From Checkpoint: Load a model that was previously saved\n",
|
||||
"\n",
|
||||
"For all of the following initialization methods, the `thirdai_key` parameter can be ommitted if the `THIRDAI_KEY` environment variable is set.\n",
|
||||
"\n",
|
||||
"ThirdAI API keys can be obtained at https://www.thirdai.com/try-bolt/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.vectorstores import NeuralDBVectorStore\n",
|
||||
"\n",
|
||||
"# From scratch\n",
|
||||
"vectorstore = NeuralDBVectorStore.from_scratch(thirdai_key=\"your-thirdai-key\")\n",
|
||||
"\n",
|
||||
"# From bazaar\n",
|
||||
"vectorstore = NeuralDBVectorStore.from_bazaar(\n",
|
||||
" # Name of base model to be downloaded from model bazaar.\n",
|
||||
" # \"General QnA\" gives better performance on question-answering.\n",
|
||||
" base=\"General QnA\",\n",
|
||||
" # Path to a directory that caches models to prevent repeated downloading.\n",
|
||||
" # Defaults to {CWD}/model_bazaar\n",
|
||||
" bazaar_cache=\"/path/to/bazaar_cache\",\n",
|
||||
" thirdai_key=\"your-thirdai-key\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# From checkpoint\n",
|
||||
"vectorstore = NeuralDBVectorStore.from_checkpoint(\n",
|
||||
" # Path to a NeuralDB checkpoint. For example, if you call\n",
|
||||
" # vectorstore.save(\"/path/to/checkpoint.ndb\") in one script, then you can\n",
|
||||
" # call NeuralDBVectorStore.from_checkpoint(\"/path/to/checkpoint.ndb\") in\n",
|
||||
" # another script to load the saved model.\n",
|
||||
" checkpoint=\"/path/to/checkpoint.ndb\",\n",
|
||||
" thirdai_key=\"your-thirdai-key\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### **Inserting document sources**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectorstore.insert(\n",
|
||||
" # If you have PDF, DOCX, or CSV files, you can directly pass the paths to the documents\n",
|
||||
" sources=[\"/path/to/doc.pdf\", \"/path/to/doc.docx\", \"/path/to/doc.csv\"],\n",
|
||||
" # When True this means that the underlying model in the NeuralDB will\n",
|
||||
" # undergo unsupervised pretraining on the inserted files. Defaults to True.\n",
|
||||
" train=True,\n",
|
||||
" # Much faster insertion with a slight drop in performance. Defaults to True.\n",
|
||||
" fast_mode=True,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"from thirdai import neural_db as ndb\n",
|
||||
"\n",
|
||||
"vectorstore.insert(\n",
|
||||
" # If you have files in other formats, or prefer to configure how\n",
|
||||
" # your files are parsed, then you can pass in NeuralDB document objects\n",
|
||||
" # like this.\n",
|
||||
" sources=[\n",
|
||||
" ndb.PDF(\n",
|
||||
" \"/path/to/doc.pdf\",\n",
|
||||
" version=\"v2\",\n",
|
||||
" chunk_size=100,\n",
|
||||
" metadata={\"published\": 2022},\n",
|
||||
" ),\n",
|
||||
" ndb.Unstructured(\"/path/to/deck.pptx\"),\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### **Similarity search**\n",
|
||||
"To query the vectorstore, you can use the standard LangChain vectorstore method `similarity_search`, which returns a list of LangChain Document objects. Each document object represents a chunk of text from the indexed files. For example, it may contain a paragraph from one of the indexed PDF files. In addition to the text, the document's metadata field contains information such as the document's ID, the source of this document (which file it came from), and the score of the document."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# This returns a list of LangChain Document objects\n",
|
||||
"documents = vectorstore.similarity_search(\"query\", k=10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### **Fine tuning**\n",
|
||||
"NeuralDBVectorStore can be fine-tuned to user behavior and domain-specific knowledge. It can be fine-tuned in two ways:\n",
|
||||
"1. Association: the vectorstore associates a source phrase with a target phrase. When the vectorstore sees the source phrase, it will also consider results that are relevant to the target phrase.\n",
|
||||
"2. Upvoting: the vectorstore upweights the score of a document for a specific query. This is useful when you want to fine-tune the vectorstore to user behavior. For example, if a user searches \"how is a car manufactured\" and likes the returned document with id 52, then we can upvote the document with id 52 for the query \"how is a car manufactured\"."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"vectorstore.associate(source=\"source phrase\", target=\"target phrase\")\n",
|
||||
"vectorstore.associate_batch(\n",
|
||||
" [\n",
|
||||
" (\"source phrase 1\", \"target phrase 1\"),\n",
|
||||
" (\"source phrase 2\", \"target phrase 2\"),\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"vectorstore.upvote(query=\"how is a car manufactured\", document_id=52)\n",
|
||||
"vectorstore.upvote_batch(\n",
|
||||
" [\n",
|
||||
" (\"query 1\", 52),\n",
|
||||
" (\"query 2\", 20),\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "langchain",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python",
|
||||
"version": "3.10.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Reference in New Issue
Block a user