Compare commits

...

2 Commits

Author SHA1 Message Date
Bagatur
41d6dbe428 gpt-4 2023-09-08 13:18:40 -07:00
Bagatur
9b1d012d73 rewrite 2023-09-08 11:26:07 -07:00

View File

@@ -4,58 +4,32 @@ sidebar_position: 1
# Retrieval
Many LLM applications require user-specific data that is not part of the model's training set.
The primary way of accomplishing this is through Retrieval Augmented Generation (RAG).
In this process, external data is *retrieved* and then passed to the LLM when doing the *generation* step.
Imagine needing to feed a Language Learning Model (LLM) with user-specific data that isn't part of its training set. Retrieval Augmented Generation (RAG) makes this possible, serving as a bridge to fetch and supply this external data to the LLM during the generation phase.
LangChain provides all the building blocks for RAG applications - from simple to complex.
This section of the documentation covers everything related to the *retrieval* step - e.g. the fetching of the data.
Although this sounds simple, it can be subtly complex.
This encompasses several key modules.
LangChain provides all the necessary tools to implement RAG applications, catering to all levels of complexity. This section delves into the intricacies of the *retrieval* step. While it may seem straightforward, this process can involve sophisticated complexities and revolves around several key modules.
![data_connection_diagram](/img/data_connection.jpg)
**[Document loaders](/docs/modules/data_connection/document_loaders/)**
**[Document Loaders](/docs/modules/data_connection/document_loaders/)**
Load documents from many different sources.
LangChain provides over 100 different document loaders as well as integrations with other major providers in the space,
like AirByte and Unstructured.
We provide integrations to load all types of documents (HTML, PDF, code) from all types of locations (private s3 buckets, public websites).
Document loaders act as the 'gatherers', fetching documents from a multitude of sources. For instance, LangChain's loaders can gather HTML files from private s3 buckets or PDFs from public websites. We offer over 100 different loaders and seamlessly integrate with major providers like AirByte and Unstructured.
**[Document transformers](/docs/modules/data_connection/document_transformers/)**
**[Document Transformers](/docs/modules/data_connection/document_transformers/)**
A key part of retrieval is fetching only the relevant parts of documents.
This involves several transformation steps in order to best prepare the documents for retrieval.
One of the primary ones here is splitting (or chunking) a large document into smaller chunks.
LangChain provides several different algorithms for doing this, as well as logic optimized for specific document types (code, markdown, etc).
Transformers work as the 'selectors', fetching only the relevant parts of documents. For example, they can split a large code document into smaller, manageable chunks for easier processing. LangChain provides specialized algorithms and logic for this purpose.
**[Text embedding models](/docs/modules/data_connection/text_embedding/)**
**[Text Embedding Models](/docs/modules/data_connection/text_embedding/)**
Another key part of retrieval has become creating embeddings for documents.
Embeddings capture the semantic meaning of the text, allowing you to quickly and
efficiently find other pieces of text that are similar.
LangChain provides integrations with over 25 different embedding providers and methods,
from open-source to proprietary API,
allowing you to choose the one best suited for your needs.
LangChain provides a standard interface, allowing you to easily swap between models.
Text embeddings serve as 'capsules' capturing the semantic meaning of text, helping in finding similar text pieces efficiently. For example, they can help find documents related to 'climate change' even if the exact phrase isn't used. LangChain integrates with over 25 embedding providers, offering a standard interface for easy model swapping.
**[Vector stores](/docs/modules/data_connection/vectorstores/)**
**[Vector Stores](/docs/modules/data_connection/vectorstores/)**
With the rise of embeddings, there has emerged a need for databases to support efficient storage and searching of these embeddings.
LangChain provides integrations with over 50 different vectorstores, from open-source local ones to cloud-hosted proprietary ones,
allowing you to choose the one best suited for your needs.
LangChain exposes a standard interface, allowing you to easily swap between vector stores.
Vector stores function as the 'libraries' that efficiently store and search text embeddings. LangChain integrates with over 50 vector stores, from local open-source ones to cloud-hosted proprietary ones, providing a standard interface for easy store swapping.
**[Retrievers](/docs/modules/data_connection/retrievers/)**
Once the data is in the database, you still need to retrieve it.
LangChain supports many different retrieval algorithms and is one of the places where we add the most value.
We support basic methods that are easy to get started - namely simple semantic search.
However, we have also added a collection of algorithms on top of this to increase performance.
These include:
Retrievers act as the 'fetchers' that extract the stored data when needed. LangChain supports various retrieval algorithms, such as:
- [Parent Document Retriever](/docs/modules/data_connection/retrievers/parent_document_retriever): This allows you to create multiple embeddings per parent document, allowing you to look up smaller chunks but return larger context.
- [Self Query Retriever](/docs/modules/data_connection/retrievers/self_query): User questions often contain a reference to something that isn't just semantic but rather expresses some logic that can best be represented as a metadata filter. Self-query allows you to parse out the *semantic* part of a query from other *metadata filters* present in the query.
- [Ensemble Retriever](/docs/modules/data_connection/retrievers/ensemble): Sometimes you may want to retrieve documents from multiple different sources, or using multiple different algorithms. The ensemble retriever allows you to easily do this.
- And more!
- And more!