mirror of
https://github.com/imartinez/privateGPT.git
synced 2025-07-04 02:56:41 +00:00
Merge pull request #224 from imartinez/feature/sentence-transformers-embeddings
Feature/sentence transformers embeddings
This commit is contained in:
commit
355b4be7c0
17
README.md
17
README.md
@ -1,7 +1,7 @@
|
|||||||
# privateGPT
|
# privateGPT
|
||||||
Ask questions to your documents without an internet connection, using the power of LLMs. 100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!
|
Ask questions to your documents without an internet connection, using the power of LLMs. 100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!
|
||||||
|
|
||||||
Built with [LangChain](https://github.com/hwchase17/langchain) and [GPT4All](https://github.com/nomic-ai/gpt4all) and [LlamaCpp](https://github.com/ggerganov/llama.cpp)
|
Built with [LangChain](https://github.com/hwchase17/langchain), [GPT4All](https://github.com/nomic-ai/gpt4all), [LlamaCpp](https://github.com/ggerganov/llama.cpp), [Chroma](https://www.trychroma.com/) and [SentenceTransformers](https://www.sbert.net/).
|
||||||
|
|
||||||
<img width="902" alt="demo" src="https://user-images.githubusercontent.com/721666/236942256-985801c9-25b9-48ef-80be-3acbb4575164.png">
|
<img width="902" alt="demo" src="https://user-images.githubusercontent.com/721666/236942256-985801c9-25b9-48ef-80be-3acbb4575164.png">
|
||||||
|
|
||||||
@ -12,20 +12,19 @@ In order to set your environment up to run the code here, first install all requ
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
Then, download the 2 models and place them in a directory of your choice.
|
Then, download the LLM model and place it in a directory of your choice:
|
||||||
- LLM: default to [ggml-gpt4all-j-v1.3-groovy.bin](https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin). If you prefer a different GPT4All-J compatible model, just download it and reference it in your `.env` file.
|
- LLM: default to [ggml-gpt4all-j-v1.3-groovy.bin](https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin). If you prefer a different GPT4All-J compatible model, just download it and reference it in your `.env` file.
|
||||||
- Embedding: default to [ggml-model-q4_0.bin](https://huggingface.co/Pi3141/alpaca-native-7B-ggml/resolve/397e872bf4c83f4c642317a5bf65ce84a105786e/ggml-model-q4_0.bin). If you prefer a different compatible Embeddings model, just download it and reference it in your `.env` file.
|
|
||||||
|
|
||||||
Rename `example.env` to `.env` and edit the variables appropriately.
|
Rename `example.env` to `.env` and edit the variables appropriately.
|
||||||
```
|
```
|
||||||
MODEL_TYPE: supports LlamaCpp or GPT4All
|
MODEL_TYPE: supports LlamaCpp or GPT4All
|
||||||
PERSIST_DIRECTORY: is the folder you want your vectorstore in
|
PERSIST_DIRECTORY: is the folder you want your vectorstore in
|
||||||
LLAMA_EMBEDDINGS_MODEL: (absolute) Path to your LlamaCpp supported embeddings model
|
|
||||||
MODEL_PATH: Path to your GPT4All or LlamaCpp supported LLM
|
MODEL_PATH: Path to your GPT4All or LlamaCpp supported LLM
|
||||||
MODEL_N_CTX: Maximum token limit for both embeddings and LLM models
|
MODEL_N_CTX: Maximum token limit for the LLM model
|
||||||
|
EMBEDDINGS_MODEL_NAME: SentenceTransformers embeddings model name (see https://www.sbert.net/docs/pretrained_models.html)
|
||||||
```
|
```
|
||||||
|
|
||||||
Note: because of the way `langchain` loads the `LLAMA` embeddings, you need to specify the absolute path of your embeddings model binary. This means it will not work if you use a home directory shortcut (eg. `~/` or `$HOME/`).
|
Note: because of the way `langchain` loads the `SentenceTransformers` embeddings, the first time you run the script it will require internet connection to download the embeddings model itself.
|
||||||
|
|
||||||
## Test dataset
|
## Test dataset
|
||||||
This repo uses a [state of the union transcript](https://github.com/imartinez/privateGPT/blob/main/source_documents/state_of_the_union.txt) as an example.
|
This repo uses a [state of the union transcript](https://github.com/imartinez/privateGPT/blob/main/source_documents/state_of_the_union.txt) as an example.
|
||||||
@ -55,11 +54,11 @@ Run the following command to ingest all the data.
|
|||||||
python ingest.py
|
python ingest.py
|
||||||
```
|
```
|
||||||
|
|
||||||
It will create a `db` folder containing the local vectorstore. Will take time, depending on the size of your documents.
|
It will create a `db` folder containing the local vectorstore. Will take 20-30 seconds per document, depending on the size of the document.
|
||||||
You can ingest as many documents as you want, and all will be accumulated in the local embeddings database.
|
You can ingest as many documents as you want, and all will be accumulated in the local embeddings database.
|
||||||
If you want to start from an empty database, delete the `db` folder.
|
If you want to start from an empty database, delete the `db` folder.
|
||||||
|
|
||||||
Note: during the ingest process no data leaves your local environment. You could ingest without an internet connection.
|
Note: during the ingest process no data leaves your local environment. You could ingest without an internet connection, except for the first time you run the ingest script, when the embeddings model is downloaded.
|
||||||
|
|
||||||
## Ask questions to your documents, locally!
|
## Ask questions to your documents, locally!
|
||||||
In order to ask a question, run a command like:
|
In order to ask a question, run a command like:
|
||||||
@ -83,7 +82,7 @@ Type `exit` to finish the script.
|
|||||||
# How does it work?
|
# How does it work?
|
||||||
Selecting the right local models and the power of `LangChain` you can run the entire pipeline locally, without any data leaving your environment, and with reasonable performance.
|
Selecting the right local models and the power of `LangChain` you can run the entire pipeline locally, without any data leaving your environment, and with reasonable performance.
|
||||||
|
|
||||||
- `ingest.py` uses `LangChain` tools to parse the document and create embeddings locally using `LlamaCppEmbeddings`. It then stores the result in a local vector database using `Chroma` vector store.
|
- `ingest.py` uses `LangChain` tools to parse the document and create embeddings locally using `HuggingFaceEmbeddings` (`SentenceTransformers`). It then stores the result in a local vector database using `Chroma` vector store.
|
||||||
- `privateGPT.py` uses a local LLM based on `GPT4All-J` or `LlamaCpp` to understand questions and create answers. The context for the answers is extracted from the local vector store using a similarity search to locate the right piece of context from the docs.
|
- `privateGPT.py` uses a local LLM based on `GPT4All-J` or `LlamaCpp` to understand questions and create answers. The context for the answers is extracted from the local vector store using a similarity search to locate the right piece of context from the docs.
|
||||||
- `GPT4All-J` wrapper was introduced in LangChain 0.0.162.
|
- `GPT4All-J` wrapper was introduced in LangChain 0.0.162.
|
||||||
|
|
||||||
|
@ -1,5 +1,5 @@
|
|||||||
PERSIST_DIRECTORY=db
|
PERSIST_DIRECTORY=db
|
||||||
LLAMA_EMBEDDINGS_MODEL=models/ggml-model-q4_0.bin
|
|
||||||
MODEL_TYPE=GPT4All
|
MODEL_TYPE=GPT4All
|
||||||
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
|
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
|
||||||
|
EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2
|
||||||
MODEL_N_CTX=1000
|
MODEL_N_CTX=1000
|
15
ingest.py
15
ingest.py
@ -19,7 +19,7 @@ from langchain.document_loaders import (
|
|||||||
|
|
||||||
from langchain.text_splitter import RecursiveCharacterTextSplitter
|
from langchain.text_splitter import RecursiveCharacterTextSplitter
|
||||||
from langchain.vectorstores import Chroma
|
from langchain.vectorstores import Chroma
|
||||||
from langchain.embeddings import LlamaCppEmbeddings
|
from langchain.embeddings import HuggingFaceEmbeddings
|
||||||
from langchain.docstore.document import Document
|
from langchain.docstore.document import Document
|
||||||
from constants import CHROMA_SETTINGS
|
from constants import CHROMA_SETTINGS
|
||||||
|
|
||||||
@ -72,22 +72,23 @@ def main():
|
|||||||
# Load environment variables
|
# Load environment variables
|
||||||
persist_directory = os.environ.get('PERSIST_DIRECTORY')
|
persist_directory = os.environ.get('PERSIST_DIRECTORY')
|
||||||
source_directory = os.environ.get('SOURCE_DIRECTORY', 'source_documents')
|
source_directory = os.environ.get('SOURCE_DIRECTORY', 'source_documents')
|
||||||
llama_embeddings_model = os.environ.get('LLAMA_EMBEDDINGS_MODEL')
|
embeddings_model_name = os.environ.get('EMBEDDINGS_MODEL_NAME')
|
||||||
model_n_ctx = os.environ.get('MODEL_N_CTX')
|
|
||||||
|
|
||||||
# Load documents and split in chunks
|
# Load documents and split in chunks
|
||||||
print(f"Loading documents from {source_directory}")
|
print(f"Loading documents from {source_directory}")
|
||||||
|
chunk_size = 500
|
||||||
|
chunk_overlap = 50
|
||||||
documents = load_documents(source_directory)
|
documents = load_documents(source_directory)
|
||||||
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
|
text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
|
||||||
texts = text_splitter.split_documents(documents)
|
texts = text_splitter.split_documents(documents)
|
||||||
print(f"Loaded {len(documents)} documents from {source_directory}")
|
print(f"Loaded {len(documents)} documents from {source_directory}")
|
||||||
print(f"Split into {len(texts)} chunks of text (max. 500 tokens each)")
|
print(f"Split into {len(texts)} chunks of text (max. {chunk_size} characters each)")
|
||||||
|
|
||||||
# Create embeddings
|
# Create embeddings
|
||||||
llama = LlamaCppEmbeddings(model_path=llama_embeddings_model, n_ctx=model_n_ctx)
|
embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name)
|
||||||
|
|
||||||
# Create and store locally vectorstore
|
# Create and store locally vectorstore
|
||||||
db = Chroma.from_documents(texts, llama, persist_directory=persist_directory, client_settings=CHROMA_SETTINGS)
|
db = Chroma.from_documents(texts, embeddings, persist_directory=persist_directory, client_settings=CHROMA_SETTINGS)
|
||||||
db.persist()
|
db.persist()
|
||||||
db = None
|
db = None
|
||||||
|
|
||||||
|
@ -1,6 +1,6 @@
|
|||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
from langchain.chains import RetrievalQA
|
from langchain.chains import RetrievalQA
|
||||||
from langchain.embeddings import LlamaCppEmbeddings
|
from langchain.embeddings import HuggingFaceEmbeddings
|
||||||
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
|
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
|
||||||
from langchain.vectorstores import Chroma
|
from langchain.vectorstores import Chroma
|
||||||
from langchain.llms import GPT4All, LlamaCpp
|
from langchain.llms import GPT4All, LlamaCpp
|
||||||
@ -8,7 +8,7 @@ import os
|
|||||||
|
|
||||||
load_dotenv()
|
load_dotenv()
|
||||||
|
|
||||||
llama_embeddings_model = os.environ.get("LLAMA_EMBEDDINGS_MODEL")
|
embeddings_model_name = os.environ.get("EMBEDDINGS_MODEL_NAME")
|
||||||
persist_directory = os.environ.get('PERSIST_DIRECTORY')
|
persist_directory = os.environ.get('PERSIST_DIRECTORY')
|
||||||
|
|
||||||
model_type = os.environ.get('MODEL_TYPE')
|
model_type = os.environ.get('MODEL_TYPE')
|
||||||
@ -18,8 +18,8 @@ model_n_ctx = os.environ.get('MODEL_N_CTX')
|
|||||||
from constants import CHROMA_SETTINGS
|
from constants import CHROMA_SETTINGS
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
llama = LlamaCppEmbeddings(model_path=llama_embeddings_model, n_ctx=model_n_ctx)
|
embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name)
|
||||||
db = Chroma(persist_directory=persist_directory, embedding_function=llama, client_settings=CHROMA_SETTINGS)
|
db = Chroma(persist_directory=persist_directory, embedding_function=embeddings, client_settings=CHROMA_SETTINGS)
|
||||||
retriever = db.as_retriever()
|
retriever = db.as_retriever()
|
||||||
# Prepare the LLM
|
# Prepare the LLM
|
||||||
callbacks = [StreamingStdOutCallbackHandler()]
|
callbacks = [StreamingStdOutCallbackHandler()]
|
||||||
|
Loading…
Reference in New Issue
Block a user