github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-06 13:33:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: misc retrievers fixes (#9791)

Browse Source 
Various miscellaneous fixes to most pages in the 'Retrievers' section of
the documentation:
- "VectorStore" and "vectorstore" changed to "vector store" for
consistency
- Various spelling, grammar, and formatting improvements for readability

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
This commit is contained in:
seamusp
2023-09-03 20:26:49 -07:00
committed by GitHub
parent 8bc452a466
commit 16945c9922
39 changed files with 148 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/extras/modules/data_connection/document_transformers/post_retrieval/long_context_reorder.ipynb
View File

@@ -9,7 +9,7 @@
"# Lost in the middle: The problem with long contexts\n",
"\n",
"No matter the architecture of your model, there is a substantial performance degradation when you include 10+ retrieved documents.\n",
"In brief: When models must access relevant information in the middle of long contexts, then tend to ignore the provided documents.\n",
"In brief: When models must access relevant information in the middle of long contexts, they tend to ignore the provided documents.\n",
"See: https://arxiv.org/abs/2307.03172\n",
"\n",
"To avoid this issue you can re-order documents after retrieval to avoid performance degradation."

12
docs/extras/modules/data_connection/document_transformers/text_splitters/markdown_header_metadata.ipynb
View File

@@ -17,17 +17,7 @@
"When a full paragraph or document is embedded, the embedding process considers both the overall context and the relationships between the sentences and phrases within the text. This can result in a more comprehensive vector representation that captures the broader meaning and themes of the text.\n",
"```\n",
" \n",
"As mentioned, chunking often aims to keep text with common context together.\n",
"\n",
"With this in mind, we might want to specifically honor the structure of the document itself.\n",
"\n",
"For example, a markdown file is organized by headers.\n",
"\n",
"Creating chunks within specific header groups is an intuitive idea.\n",
"\n",
"To address this challenge, we can use `MarkdownHeaderTextSplitter`.\n",
"\n",
"This will split a markdown file by a specified set of headers. \n",
"As mentioned, chunking often aims to keep text with common context together. With this in mind, we might want to specifically honor the structure of the document itself. For example, a markdown file is organized by headers. Creating chunks within specific header groups is an intuitive idea. To address this challenge, we can use `MarkdownHeaderTextSplitter`. This will split a markdown file by a specified set of headers. \n",
"\n",
"For example, if we want to split this markdown:\n",
"```\n",

14
docs/extras/modules/data_connection/document_transformers/text_splitters/split_by_token.ipynb
View File

@@ -22,8 +22,8 @@
"\n",
"We can use it to estimate tokens used. It will probably be more accurate for the OpenAI models.\n",
"\n",
"1. How the text is split: by character passed in\n",
"2. How the chunk size is measured: by `tiktoken` tokenizer"
"1. How the text is split: by character passed in.\n",
"2. How the chunk size is measured: by `tiktoken` tokenizer."
]
},
{
@@ -122,8 +122,8 @@
"\n",
"Another alternative to `NLTK` is to use [spaCy tokenizer](https://spacy.io/api/tokenizer).\n",
"\n",
"1. How the text is split: by `spaCy` tokenizer\n",
"2. How the chunk size is measured: by number of characters"
"1. How the text is split: by `spaCy` tokenizer.\n",
"2. How the chunk size is measured: by number of characters."
]
},
{
@@ -331,7 +331,7 @@
"Rather than just splitting on \"\\n\\n\", we can use `NLTK` to split based on [NLTK tokenizers](https://www.nltk.org/api/nltk.tokenize.html).\n",
"\n",
"1. How the text is split: by `NLTK` tokenizer.\n",
"2. How the chunk size is measured:by number of characters"
"2. How the chunk size is measured: by number of characters."
]
},
{
@@ -430,8 +430,8 @@
"\n",
"We use Hugging Face tokenizer, the [GPT2TokenizerFast](https://huggingface.co/Ransaka/gpt2-tokenizer-fast) to count the text length in tokens.\n",
"\n",
"1. How the text is split: by character passed in\n",
"2. How the chunk size is measured: by number of tokens calculated by the `Hugging Face` tokenizer\n"
"1. How the text is split: by character passed in.\n",
"2. How the chunk size is measured: by number of tokens calculated by the `Hugging Face` tokenizer.\n"
]
},
{

34
docs/extras/modules/data_connection/indexing.ipynb
View File

@@ -45,7 +45,7 @@
"\n",
"`incremental` and `full` offer the following automated clean up:\n",
"\n",
"* If the content of source document or derived documents has **changed**, both `incremental` or `full` modes will clean up (delete) previous versions of the content.\n",
"* If the content of the source document or derived documents has **changed**, both `incremental` or `full` modes will clean up (delete) previous versions of the content.\n",
"* If the source document has been **deleted** (meaning it is not included in the documents currently being indexed), the `full` cleanup mode will delete it from the vector store correctly, but the `incremental` mode will not.\n",
"\n",
"When content is mutated (e.g., the source PDF file was revised) there will be a period of time during indexing when both the new and old versions may be returned to the user. This happens after the new content was written, but before the old version was deleted.\n",
@@ -56,7 +56,7 @@
"## Requirements\n",
"\n",
"1. Do not use with a store that has been pre-populated with content independently of the indexing API, as the record manager will not know that records have been inserted previously.\n",
"2. Only works with LangChain ``VectorStore``'s that support:\n",
"2. Only works with LangChain `vectorstore`'s that support:\n",
" * document addition by id (`add_documents` method with `ids` argument)\n",
" * delete by id (`delete` method with)\n",
" \n",
@@ -64,11 +64,11 @@
"\n",
"The record manager relies on a time-based mechanism to determine what content can be cleaned up (when using `full` or `incremental` cleanup modes).\n",
"\n",
"If two tasks run back to back, and the first task finishes before the the clock time changes, then the second task may not be able to clean up content.\n",
"If two tasks run back-to-back, and the first task finishes before the the clock time changes, then the second task may not be able to clean up content.\n",
"\n",
"This is unlikely to be an issue in actual settings for the following reasons:\n",
"\n",
"1. The RecordManager uses higher resolutino timestamps.\n",
"1. The RecordManager uses higher resolution timestamps.\n",
"2. The data would need to change between the first and the second tasks runs, which becomes unlikely if the time interval between the tasks is small.\n",
"3. Indexing tasks typically take more than a few ms."
]
@@ -99,7 +99,7 @@
"id": "f81201ab-d997-433c-9f18-ceea70e61cbd",
"metadata": {},
"source": [
"Initialize a vector store and set up the embeddings"
"Initialize a vector store and set up the embeddings:"
]
},
{
@@ -125,7 +125,7 @@
"source": [
"Initialize a record manager with an appropriate namespace.\n",
"\n",
"**Suggestion** Use a namespace that takes into account both the vectostore and the collection name in the vectorstore; e.g., 'redis/my_docs', 'chromadb/my_docs' or 'postgres/my_docs'"
"**Suggestion:** Use a namespace that takes into account both the vector store and the collection name in the vector store; e.g., 'redis/my_docs', 'chromadb/my_docs' or 'postgres/my_docs'."
]
},
{
@@ -148,7 +148,7 @@
"id": "835c2c19-68ec-4086-9066-f7ba40877fd5",
"metadata": {},
"source": [
"Create a schema before using the record manager"
"Create a schema before using the record manager."
]
},
{
@@ -166,7 +166,7 @@
"id": "7f07c6bd-6ada-4b17-a8c5-fe5e4a5278fd",
"metadata": {},
"source": [
"Let's index some test documents"
"Let's index some test documents:"
]
},
{
@@ -185,7 +185,7 @@
"id": "c7d572be-a913-4511-ab64-2864a252458a",
"metadata": {},
"source": [
"Indexing into an empty vectorstore"
"Indexing into an empty vector store:"
]
},
{
@@ -285,7 +285,7 @@
"id": "7be3e55a-5fe9-4f40-beff-577c2aa5e76a",
"metadata": {},
"source": [
"Second time around all content will be skipped"
"Second time around all content will be skipped:"
]
},
{
@@ -396,7 +396,7 @@
"id": "b205c1ba-f069-4a4e-af93-dc98afd5c9e6",
"metadata": {},
"source": [
"If we provide no documents with incremental indexing mode, nothing will change"
"If we provide no documents with incremental indexing mode, nothing will change."
]
},
{
@@ -476,7 +476,7 @@
"\n",
"In `full` mode the user should pass the `full` universe of content that should be indexed into the indexing function.\n",
"\n",
"Any documents that are not passed into the indexing functino and are present in the vectorstore will be deleted!\n",
"Any documents that are not passed into the indexing function and are present in the vectorstore will be deleted!\n",
"\n",
"This behavior is useful to handle deletions of source documents."
]
@@ -527,7 +527,7 @@
"id": "887c45c6-4363-4389-ac56-9cdad682b4c8",
"metadata": {},
"source": [
"Say someone deleted the first doc"
"Say someone deleted the first doc:"
]
},
{
@@ -566,7 +566,7 @@
"id": "d940bcb4-cf6d-4c21-a565-e7f53f6dacf1",
"metadata": {},
"source": [
"Using full mode will clean up the deleted content as well"
"Using full mode will clean up the deleted content as well."
]
},
{
@@ -716,7 +716,7 @@
"id": "ab1c0915-3f9e-42ac-bdb5-3017935c6e7f",
"metadata": {},
"source": [
"This should delete the old versions of documents associated with `doggy.txt` source and replace them with the new versions"
"This should delete the old versions of documents associated with `doggy.txt` source and replace them with the new versions."
]
},
{
@@ -774,11 +774,11 @@
"id": "c0af4d24-d735-4e5d-ad9b-a2e8b281f9f1",
"metadata": {},
"source": [
"## Using with Loaders\n",
"## Using with loaders\n",
"\n",
"Indexing can accept either an iterable of documents or else any loader.\n",
"\n",
"**Attention** The loader **MUST** set source keys correctly."
"**Attention:** The loader **must** set source keys correctly."
]
},
{

6
docs/extras/modules/data_connection/retrievers/MultiQueryRetriever.ipynb
View File

@@ -7,7 +7,7 @@
"source": [
"# MultiQueryRetriever\n",
"\n",
"Distance-based vector database retrieval embeds (represents) queries in high-dimensional space and finds similar embedded documents based on \"distance\". But, retrieval may produce difference results with subtle changes in query wording or if the embeddings do not capture the semantics of the data well. Prompt engineering / tuning is sometimes done to manually address these problems, but can be tedious.\n",
"Distance-based vector database retrieval embeds (represents) queries in high-dimensional space and finds similar embedded documents based on \"distance\". But, retrieval may produce different results with subtle changes in query wording or if the embeddings do not capture the semantics of the data well. Prompt engineering / tuning is sometimes done to manually address these problems, but can be tedious.\n",
"\n",
"The `MultiQueryRetriever` automates the process of prompt tuning by using an LLM to generate multiple queries from different perspectives for a given user input query. For each query, it retrieves a set of relevant documents and takes the unique union across all queries to get a larger set of potentially relevant documents. By generating multiple perspectives on the same question, the `MultiQueryRetriever` might be able to overcome some of the limitations of the distance-based retrieval and get a richer set of results."
]
@@ -43,7 +43,7 @@
"id": "cca8f56c",
"metadata": {},
"source": [
"`Simple usage`\n",
"#### Simple usage\n",
"\n",
"Specify the LLM to use for query generation, and the retriver will do the rest."
]
@@ -113,7 +113,7 @@
"id": "c54a282f",
"metadata": {},
"source": [
"`Supplying your own prompt`\n",
"#### Supplying your own prompt\n",
"\n",
"You can also supply a prompt along with an output parser to split the results into a list of queries."
]

4
docs/extras/modules/data_connection/retrievers/ensemble.ipynb
View File

@@ -6,11 +6,11 @@
"source": [
"# Ensemble Retriever\n",
"\n",
"The `EnsembleRetriever` takes a list of retrievers as input and ensemble the results of their get_relevant_documents() methods and rerank the results based on the [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) algorithm.\n",
"The `EnsembleRetriever` takes a list of retrievers as input and ensemble the results of their `get_relevant_documents()` methods and rerank the results based on the [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) algorithm.\n",
"\n",
"By leveraging the strengths of different algorithms, the `EnsembleRetriever` can achieve better performance than any single algorithm. \n",
"\n",
"The most common pattern is to combine a sparse retriever(like BM25) with a dense retriever(like Embedding similarity), because their strengths are complementary. It is also known as \"hybrid search\".The sparse retriever is good at finding relevant documents based on keywords, while the dense retriever is good at finding relevant documents based on semantic similarity."
"The most common pattern is to combine a sparse retriever (like BM25) with a dense retriever (like embedding similarity), because their strengths are complementary. It is also known as \"hybrid search\". The sparse retriever is good at finding relevant documents based on keywords, while the dense retriever is good at finding relevant documents based on semantic similarity."
]
},
{

10
docs/extras/modules/data_connection/retrievers/multi_vector.ipynb
View File

@@ -11,12 +11,12 @@
"\n",
"The methods to create multiple vectors per document include:\n",
"\n",
"- smaller chunks: split a document into smaller chunks, and embed those (this is ParentDocumentRetriever)\n",
"- summary: create a summary for each document, embed that along with (or instead of) the document\n",
"- hypothetical questions: create hypothetical questions that each document would be appropriate to answer, embed those along with (or instead of) the document\n",
"- Smaller chunks: split a document into smaller chunks, and embed those (this is ParentDocumentRetriever).\n",
"- Summary: create a summary for each document, embed that along with (or instead of) the document.\n",
"- Hypothetical questions: create hypothetical questions that each document would be appropriate to answer, embed those along with (or instead of) the document.\n",
"\n",
"\n",
"Note that this also enables another method of adding embeddings - manually. This is great because you can explicitly add questions or queries that should lead to a document being recovered, giving you more control"
"Note that this also enables another method of adding embeddings - manually. This is great because you can explicitly add questions or queries that should lead to a document being recovered, giving you more control."
]
},
{
@@ -68,7 +68,7 @@
"source": [
"## Smaller chunks\n",
"\n",
"Often times it can be useful to retrieve larger chunks of information, but embed smaller chunks. This allows for embeddings to capture the semantic meaning as closely as possible, but for as much context as possible to be passed downstream. NOTE: this is what the ParentDocumentRetriever does. Here we show what is going on under the hood."
"Often times it can be useful to retrieve larger chunks of information, but embed smaller chunks. This allows for embeddings to capture the semantic meaning as closely as possible, but for as much context as possible to be passed downstream. Note that this is what the `ParentDocumentRetriever` does. Here we show what is going on under the hood."
]
},
{

12
docs/extras/modules/data_connection/retrievers/parent_document_retriever.ipynb
View File

@@ -15,7 +15,7 @@
"2. You want to have long enough documents that the context of each chunk is\n",
" retained.\n",
"\n",
"The ParentDocumentRetriever strikes that balance by splitting and storing\n",
"The `ParentDocumentRetriever` strikes that balance by splitting and storing\n",
"small chunks of data. During retrieval, it first fetches the small chunks\n",
"but then looks up the parent ids for those chunks and returns those larger\n",
"documents.\n",
@@ -70,7 +70,7 @@
"id": "d3943f72",
"metadata": {},
"source": [
"## Retrieving Full Documents\n",
"## Retrieving full documents\n",
"\n",
"In this mode, we want to retrieve the full documents. Therefore, we only specify a child splitter."
]
@@ -143,7 +143,7 @@
"id": "f895d62b",
"metadata": {},
"source": [
"Let's now call the vectorstore search functionality - we should see that it returns small chunks (since we're storing the small chunks)."
"Let's now call the vector store search functionality - we should see that it returns small chunks (since we're storing the small chunks)."
]
},
{
@@ -220,7 +220,7 @@
"id": "14f813a5",
"metadata": {},
"source": [
"## Retrieving Larger Chunks\n",
"## Retrieving larger chunks\n",
"\n",
"Sometimes, the full documents can be too big to want to retrieve them as is. In that case, what we really want to do is to first split the raw documents into larger chunks, and then split it into smaller chunks. We then index the smaller chunks, but on retrieval we retrieve the larger chunks (but still not the full documents)."
]
@@ -273,7 +273,7 @@
"id": "64ad3c8c",
"metadata": {},
"source": [
"We can see that there are much more than two documents now - these are the larger chunks"
"We can see that there are much more than two documents now - these are the larger chunks."
]
},
{
@@ -302,7 +302,7 @@
"id": "baaef673",
"metadata": {},
"source": [
"Let's make sure the underlying vectorstore still retrieves the small chunks."
"Let's make sure the underlying vector store still retrieves the small chunks."
]
},
{

10
docs/extras/modules/data_connection/retrievers/self_query/activeloop_deeplake_self_query.ipynb
View File

@@ -8,9 +8,9 @@
"source": [
"# Deep Lake self-querying \n",
"\n",
">[DeepLake](https://www.activeloop.ai) is a multimodal database for building AI applications.\n",
">[Deep Lake](https://www.activeloop.ai) is a multimodal database for building AI applications.\n",
"\n",
"In the notebook we'll demo the `SelfQueryRetriever` wrapped around a DeepLake vector store. "
"In the notebook we'll demo the `SelfQueryRetriever` wrapped around a Deep Lake vector store. "
]
},
{
@@ -19,10 +19,10 @@
"id": "68e75fb9",
"metadata": {},
"source": [
"## Creating a Deep Lake vectorstore\n",
"First we'll want to create a DeepLake VectorStore and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"## Creating a Deep Lake vector store\n",
"First we'll want to create a Deep Lake vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"\n",
"NOTE: The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `deeplake` package."
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `deeplake` package."
]
},
{

8
docs/extras/modules/data_connection/retrievers/self_query/chroma_self_query.ipynb
View File

@@ -17,10 +17,10 @@
"id": "68e75fb9",
"metadata": {},
"source": [
"## Creating a Chroma vectorstore\n",
"First we'll want to create a Chroma VectorStore and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"## Creating a Chroma vector store\n",
"First we'll want to create a Chroma vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"\n",
"NOTE: The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `chromadb` package."
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `chromadb` package."
]
},
{
@@ -64,7 +64,7 @@
},
"outputs": [
{
"name": "stdin",
"name": "stdout",
"output_type": "stream",
"text": [
"OpenAI API Key: ········\n"

6
docs/extras/modules/data_connection/retrievers/self_query/elasticsearch_self_query.ipynb
View File

@@ -13,10 +13,10 @@
"id": "68e75fb9",
"metadata": {},
"source": [
"## Creating a Elasticsearch vectorstore\n",
"First we'll want to create a Elasticsearch VectorStore and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"## Creating a Elasticsearch vector store\n",
"First we'll want to create a Elasticsearch vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"\n",
"NOTE: The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `elasticsearch` package."
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `elasticsearch` package."
]
},
{

16
docs/extras/modules/data_connection/retrievers/self_query/myscale_self_query.ipynb
View File

@@ -9,7 +9,7 @@
"\n",
">[MyScale](https://docs.myscale.com/en/) is an integrated vector database. You can access your database in SQL and also from here, LangChain. MyScale can make a use of [various data types and functions for filters](https://blog.myscale.com/2023/06/06/why-integrated-database-solution-can-boost-your-llm-apps/#filter-on-anything-without-constraints). It will boost up your LLM app no matter if you are scaling up your data or expand your system to broader application.\n",
"\n",
"In the notebook we'll demo the `SelfQueryRetriever` wrapped around a MyScale vector store with some extra piece we contributed to LangChain. In short, it can be concluded into 4 points:\n",
"In the notebook we'll demo the `SelfQueryRetriever` wrapped around a MyScale vector store with some extra pieces we contributed to LangChain. In short, it can be condensed into 4 points:\n",
"1. Add `contain` comparator to match list of any if there is more than one element matched\n",
"2. Add `timestamp` data type for datetime match (ISO-format, or YYYY-MM-DD)\n",
"3. Add `like` comparator for string pattern search\n",
@@ -21,10 +21,10 @@
"id": "68e75fb9",
"metadata": {},
"source": [
"## Creating a MyScale vectorstore\n",
"MyScale has already been integrated to LangChain for a while. So you can follow [this notebook](/docs/integrations/vectorstores/myscale.ipynb) to create your own vectorstore for a self-query retriever.\n",
"## Creating a MyScale vector store\n",
"MyScale has already been integrated to LangChain for a while. So you can follow [this notebook](/docs/integrations/vectorstores/myscale) to create your own vectorstore for a self-query retriever.\n",
"\n",
"NOTE: All self-query retrievers requires you to have `lark` installed (`pip install lark`). We use `lark` for grammar definition. Before you proceed to the next step, we also want to remind you that `clickhouse-connect` is also needed to interact with your MyScale backend."
"**Note:** All self-query retrievers requires you to have `lark` installed (`pip install lark`). We use `lark` for grammar definition. Before you proceed to the next step, we also want to remind you that `clickhouse-connect` is also needed to interact with your MyScale backend."
]
},
{
@@ -44,7 +44,7 @@
"id": "83811610-7df3-4ede-b268-68a6a83ba9e2",
"metadata": {},
"source": [
"In this tutorial we follow other example's setting and use `OpenAIEmbeddings`. Remember to get a OpenAI API Key for valid accesss to LLMs."
"In this tutorial we follow other example's setting and use `OpenAIEmbeddings`. Remember to get an OpenAI API Key for valid accesss to LLMs."
]
},
{
@@ -88,7 +88,7 @@
"metadata": {},
"source": [
"## Create some sample data\n",
"As you can see, the data we created has some difference to other self-query retrievers. We replaced keyword `year` to `date` which gives you a finer control on timestamps. We also altered the type of keyword `gerne` to list of strings, where LLM can use a new `contain` comparator to construct filters. We also provides comparator `like` and arbitrary function support to filters, which will be introduced in next few cells.\n",
"As you can see, the data we created has some differences compared to other self-query retrievers. We replaced the keyword `year` with `date` which gives you finer control on timestamps. We also changed the type of the keyword `gerne` to a list of strings, where an LLM can use a new `contain` comparator to construct filters. We also provide the `like` comparator and arbitrary function support to filters, which will be introduced in next few cells.\n",
"\n",
"Now let's look at the data first."
]
@@ -146,7 +146,7 @@
"metadata": {},
"source": [
"## Creating our self-querying retriever\n",
"Just like other retrievers... Simple and nice."
"Just like other retrievers... simple and nice."
]
},
{
@@ -272,7 +272,7 @@
"id": "86371ac8",
"metadata": {},
"source": [
"# Wait a second... What else?\n",
"# Wait a second... what else?\n",
"\n",
"Self-query retriever with MyScale can do more! Let's find out."
]

6
docs/extras/modules/data_connection/retrievers/self_query/pinecone.ipynb
View File

@@ -16,11 +16,11 @@
"metadata": {},
"source": [
"## Creating a Pinecone index\n",
"First we'll want to create a `Pinecone` VectorStore and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"First we'll want to create a `Pinecone` vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"\n",
"To use Pinecone, you have to have `pinecone` package installed and you must have an API key and an Environment. Here are the [installation instructions](https://docs.pinecone.io/docs/quickstart).\n",
"To use Pinecone, you have to have `pinecone` package installed and you must have an API key and an environment. Here are the [installation instructions](https://docs.pinecone.io/docs/quickstart).\n",
"\n",
"NOTE: The self-query retriever requires you to have `lark` package installed."
"**Note:** The self-query retriever requires you to have `lark` package installed."
]
},
{

8
docs/extras/modules/data_connection/retrievers/self_query/qdrant_self_query.ipynb
View File

@@ -8,7 +8,7 @@
"source": [
"# Qdrant self-querying \n",
"\n",
">[Qdrant](https://qdrant.tech/documentation/) (read: quadrant ) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage points - vectors with an additional payload. `Qdrant` is tailored to extended filtering support. It makes it useful \n",
">[Qdrant](https://qdrant.tech/documentation/) (read: quadrant) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage points - vectors with an additional payload. `Qdrant` is tailored to extended filtering support.\n",
"\n",
"In the notebook we'll demo the `SelfQueryRetriever` wrapped around a Qdrant vector store. "
]
@@ -19,10 +19,10 @@
"id": "68e75fb9",
"metadata": {},
"source": [
"## Creating a Qdrant vectorstore\n",
"First we'll want to create a Qdrant VectorStore and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"## Creating a Qdrant vector store\n",
"First we'll want to create a Qdrant vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"\n",
"NOTE: The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `qdrant-client` package."
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `qdrant-client` package."
]
},
{

6
docs/extras/modules/data_connection/retrievers/self_query/weaviate_self_query.ipynb
View File

@@ -13,10 +13,10 @@
"id": "68e75fb9",
"metadata": {},
"source": [
"## Creating a Weaviate vectorstore\n",
"First we'll want to create a Weaviate VectorStore and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"## Creating a Weaviate vector store\n",
"First we'll want to create a Weaviate vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
"\n",
"NOTE: The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `weaviate-client` package."
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `weaviate-client` package."
]
},
{

14
docs/extras/modules/data_connection/retrievers/web_research.ipynb
View File

@@ -80,9 +80,9 @@
"id": "39114da4",
"metadata": {},
"source": [
"`Run with citations`\n",
"#### Run with citations\n",
"\n",
"We can use `RetrievalQAWithSourcesChain` to retrieve docs and provide citations"
"We can use `RetrievalQAWithSourcesChain` to retrieve docs and provide citations."
]
},
{
@@ -126,7 +126,7 @@
"id": "357559fd",
"metadata": {},
"source": [
"`Run with logging`\n",
"#### Run with logging\n",
"\n",
"Here, we use `get_relevant_documents` method to return docs."
]
@@ -171,9 +171,9 @@
"id": "b681a846",
"metadata": {},
"source": [
"`Generate answer using retrieved docs`\n",
"#### Generate answer using retrieved docs\n",
"\n",
"We can use `load_qa_chain` for QA using the retrieved docs"
"We can use `load_qa_chain` for QA using the retrieved docs."
]
},
{
@@ -207,7 +207,7 @@
"source": [
"### More flexibility\n",
"\n",
"Pass an LLM chain with custom prompt and output parsing"
"Pass an LLM chain with custom prompt and output parsing."
]
},
{
@@ -326,7 +326,7 @@
"source": [
"### Run locally\n",
"\n",
"Specify LLM and embeddings that will run locally (e.g., on your laptop)"
"Specify LLM and embeddings that will run locally (e.g., on your laptop)."
]
},
{

19
docs/extras/modules/data_connection/text_embedding/caching_embeddings.ipynb
View File

@@ -9,13 +9,8 @@
"\n",
"Embeddings can be stored or temporarily cached to avoid needing to recompute them.\n",
"\n",
"Caching embeddings can be done using a `CacheBackedEmbeddings`.\n",
"\n",
"The cache backed embedder is a wrapper around an embedder that caches\n",
"embeddings in a key-value store. \n",
"\n",
"The text is hashed and the hash is used as the key in the cache.\n",
"\n",
"Caching embeddings can be done using a `CacheBackedEmbeddings`. The cache backed embedder is a wrapper around an embedder that caches\n",
"embeddings in a key-value store. The text is hashed and the hash is used as the key in the cache.\n",
"\n",
"The main supported way to initialized a `CacheBackedEmbeddings` is `from_bytes_store`. This takes in the following parameters:\n",
"\n",
@@ -44,9 +39,9 @@
"id": "9ddf07dd-3e72-41de-99d4-78e9521e272f",
"metadata": {},
"source": [
"## Using with a vectorstore\n",
"## Using with a vector store\n",
"\n",
"First, let's see an example that uses the local file system for storing embeddings and uses FAISS vectorstore for retrieval."
"First, let's see an example that uses the local file system for storing embeddings and uses FAISS vector store for retrieval."
]
},
{
@@ -91,7 +86,7 @@
"id": "f8cdf33c-321d-4d2c-b76b-d6f5f8b42a92",
"metadata": {},
"source": [
"The cache is empty prior to embedding"
"The cache is empty prior to embedding:"
]
},
{
@@ -140,7 +135,7 @@
"id": "f526444b-93f8-423f-b6d1-dab539450921",
"metadata": {},
"source": [
"create the vectorstore"
"Create the vector store:"
]
},
{
@@ -168,7 +163,7 @@
"id": "64fc53f5-d559-467f-bf62-5daef32ffbc0",
"metadata": {},
"source": [
"If we try to create the vectostore again, it'll be much faster since it does not need to re-compute any embeddings."
"If we try to create the vector store again, it'll be much faster since it does not need to re-compute any embeddings."
]
},
{