mirror of
https://github.com/imartinez/privateGPT.git
synced 2025-06-24 14:33:23 +00:00
Merged with dev
This commit is contained in:
Saurab-Shrestha
commit
3f99b0996f
35
CHANGELOG.md
35
CHANGELOG.md
@ -1,5 +1,40 @@
|
||||
# Changelog
|
||||
|
||||
## [0.5.0](https://github.com/zylon-ai/private-gpt/compare/v0.4.0...v0.5.0) (2024-04-02)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **code:** improve concat of strings in ui ([#1785](https://github.com/zylon-ai/private-gpt/issues/1785)) ([bac818a](https://github.com/zylon-ai/private-gpt/commit/bac818add51b104cda925b8f1f7b51448e935ca1))
|
||||
* **docker:** set default Docker to use Ollama ([#1812](https://github.com/zylon-ai/private-gpt/issues/1812)) ([f83abff](https://github.com/zylon-ai/private-gpt/commit/f83abff8bc955a6952c92cc7bcb8985fcec93afa))
|
||||
* **docs:** Add guide Llama-CPP Linux AMD GPU support ([#1782](https://github.com/zylon-ai/private-gpt/issues/1782)) ([8a836e4](https://github.com/zylon-ai/private-gpt/commit/8a836e4651543f099c59e2bf497ab8c55a7cd2e5))
|
||||
* **docs:** Feature/upgrade docs ([#1741](https://github.com/zylon-ai/private-gpt/issues/1741)) ([5725181](https://github.com/zylon-ai/private-gpt/commit/572518143ac46532382db70bed6f73b5082302c1))
|
||||
* **docs:** upgrade fern ([#1596](https://github.com/zylon-ai/private-gpt/issues/1596)) ([84ad16a](https://github.com/zylon-ai/private-gpt/commit/84ad16af80191597a953248ce66e963180e8ddec))
|
||||
* **ingest:** Created a faster ingestion mode - pipeline ([#1750](https://github.com/zylon-ai/private-gpt/issues/1750)) ([134fc54](https://github.com/zylon-ai/private-gpt/commit/134fc54d7d636be91680dc531f5cbe2c5892ac56))
|
||||
* **llm - embed:** Add support for Azure OpenAI ([#1698](https://github.com/zylon-ai/private-gpt/issues/1698)) ([1efac6a](https://github.com/zylon-ai/private-gpt/commit/1efac6a3fe19e4d62325e2c2915cd84ea277f04f))
|
||||
* **llm:** adds serveral settings for llamacpp and ollama ([#1703](https://github.com/zylon-ai/private-gpt/issues/1703)) ([02dc83e](https://github.com/zylon-ai/private-gpt/commit/02dc83e8e9f7ada181ff813f25051bbdff7b7c6b))
|
||||
* **llm:** Ollama LLM-Embeddings decouple + longer keep_alive settings ([#1800](https://github.com/zylon-ai/private-gpt/issues/1800)) ([b3b0140](https://github.com/zylon-ai/private-gpt/commit/b3b0140e244e7a313bfaf4ef10eb0f7e4192710e))
|
||||
* **llm:** Ollama timeout setting ([#1773](https://github.com/zylon-ai/private-gpt/issues/1773)) ([6f6c785](https://github.com/zylon-ai/private-gpt/commit/6f6c785dac2bbad37d0b67fda215784298514d39))
|
||||
* **local:** tiktoken cache within repo for offline ([#1467](https://github.com/zylon-ai/private-gpt/issues/1467)) ([821bca3](https://github.com/zylon-ai/private-gpt/commit/821bca32e9ee7c909fd6488445ff6a04463bf91b))
|
||||
* **nodestore:** add Postgres for the doc and index store ([#1706](https://github.com/zylon-ai/private-gpt/issues/1706)) ([68b3a34](https://github.com/zylon-ai/private-gpt/commit/68b3a34b032a08ca073a687d2058f926032495b3))
|
||||
* **rag:** expose similarity_top_k and similarity_score to settings ([#1771](https://github.com/zylon-ai/private-gpt/issues/1771)) ([087cb0b](https://github.com/zylon-ai/private-gpt/commit/087cb0b7b74c3eb80f4f60b47b3a021c81272ae1))
|
||||
* **RAG:** Introduce SentenceTransformer Reranker ([#1810](https://github.com/zylon-ai/private-gpt/issues/1810)) ([83adc12](https://github.com/zylon-ai/private-gpt/commit/83adc12a8ef0fa0c13a0dec084fa596445fc9075))
|
||||
* **scripts:** Wipe qdrant and obtain db Stats command ([#1783](https://github.com/zylon-ai/private-gpt/issues/1783)) ([ea153fb](https://github.com/zylon-ai/private-gpt/commit/ea153fb92f1f61f64c0d04fff0048d4d00b6f8d0))
|
||||
* **ui:** Add Model Information to ChatInterface label ([f0b174c](https://github.com/zylon-ai/private-gpt/commit/f0b174c097c2d5e52deae8ef88de30a0d9013a38))
|
||||
* **ui:** add sources check to not repeat identical sources ([#1705](https://github.com/zylon-ai/private-gpt/issues/1705)) ([290b9fb](https://github.com/zylon-ai/private-gpt/commit/290b9fb084632216300e89bdadbfeb0380724b12))
|
||||
* **UI:** Faster startup and document listing ([#1763](https://github.com/zylon-ai/private-gpt/issues/1763)) ([348df78](https://github.com/zylon-ai/private-gpt/commit/348df781b51606b2f9810bcd46f850e54192fd16))
|
||||
* **ui:** maintain score order when curating sources ([#1643](https://github.com/zylon-ai/private-gpt/issues/1643)) ([410bf7a](https://github.com/zylon-ai/private-gpt/commit/410bf7a71f17e77c4aec723ab80c233b53765964))
|
||||
* unify settings for vector and nodestore connections to PostgreSQL ([#1730](https://github.com/zylon-ai/private-gpt/issues/1730)) ([63de7e4](https://github.com/zylon-ai/private-gpt/commit/63de7e4930ac90dd87620225112a22ffcbbb31ee))
|
||||
* wipe per storage type ([#1772](https://github.com/zylon-ai/private-gpt/issues/1772)) ([c2d6948](https://github.com/zylon-ai/private-gpt/commit/c2d694852b4696834962a42fde047b728722ad74))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **docs:** Minor documentation amendment ([#1739](https://github.com/zylon-ai/private-gpt/issues/1739)) ([258d02d](https://github.com/zylon-ai/private-gpt/commit/258d02d87c5cb81d6c3a6f06aa69339b670dffa9))
|
||||
* Fixed docker-compose ([#1758](https://github.com/zylon-ai/private-gpt/issues/1758)) ([774e256](https://github.com/zylon-ai/private-gpt/commit/774e2560520dc31146561d09a2eb464c68593871))
|
||||
* **ingest:** update script label ([#1770](https://github.com/zylon-ai/private-gpt/issues/1770)) ([7d2de5c](https://github.com/zylon-ai/private-gpt/commit/7d2de5c96fd42e339b26269b3155791311ef1d08))
|
||||
* **settings:** set default tokenizer to avoid running make setup fail ([#1709](https://github.com/zylon-ai/private-gpt/issues/1709)) ([d17c34e](https://github.com/zylon-ai/private-gpt/commit/d17c34e81a84518086b93605b15032e2482377f7))
|
||||
|
||||
## [0.4.0](https://github.com/imartinez/privateGPT/compare/v0.3.0...v0.4.0) (2024-03-06)
|
||||
|
||||
|
||||
|
||||
@ -14,7 +14,7 @@ FROM base as dependencies
|
||||
WORKDIR /home/worker/app
|
||||
COPY pyproject.toml poetry.lock ./
|
||||
|
||||
RUN poetry install --extras "ui vector-stores-qdrant"
|
||||
RUN poetry install --extras "ui vector-stores-qdrant llms-ollama embeddings-ollama"
|
||||
|
||||
FROM base as app
|
||||
|
||||
|
||||
3
Makefile
3
Makefile
@ -51,6 +51,9 @@ api-docs:
|
||||
ingest:
|
||||
@poetry run python scripts/ingest_folder.py $(call args)
|
||||
|
||||
stats:
|
||||
poetry run python scripts/utils.py stats
|
||||
|
||||
wipe:
|
||||
poetry run python scripts/utils.py wipe
|
||||
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
services:
|
||||
private-gpt:
|
||||
build:
|
||||
dockerfile: Dockerfile.local
|
||||
dockerfile: Dockerfile.external
|
||||
volumes:
|
||||
- ./local_data/:/home/worker/app/local_data
|
||||
- ./models/:/home/worker/app/models
|
||||
ports:
|
||||
<<<<<<< HEAD
|
||||
- 80:80
|
||||
environment:
|
||||
PORT: 80
|
||||
@ -15,3 +15,14 @@ services:
|
||||
env_file:
|
||||
- .env
|
||||
|
||||
=======
|
||||
- 8001:8080
|
||||
environment:
|
||||
PORT: 8080
|
||||
PGPT_PROFILES: docker
|
||||
PGPT_MODE: ollama
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
volumes:
|
||||
- ./models:/root/.ollama
|
||||
>>>>>>> dev
|
||||
|
||||
@ -58,10 +58,14 @@ navigation:
|
||||
contents:
|
||||
- page: Vector Stores
|
||||
path: ./docs/pages/manual/vectordb.mdx
|
||||
- page: Node Stores
|
||||
path: ./docs/pages/manual/nodestore.mdx
|
||||
- section: Advanced Setup
|
||||
contents:
|
||||
- page: LLM Backends
|
||||
path: ./docs/pages/manual/llms.mdx
|
||||
- page: Reranking
|
||||
path: ./docs/pages/manual/reranker.mdx
|
||||
- section: User Interface
|
||||
contents:
|
||||
- page: User interface (Gradio) Manual
|
||||
|
||||
@ -8,14 +8,14 @@ The clients are kept up to date automatically, so we encourage you to use the la
|
||||
|
||||
<Cards>
|
||||
<Card
|
||||
title="Node.js/TypeScript"
|
||||
title="Node.js/TypeScript - WIP"
|
||||
icon="fa-brands fa-node"
|
||||
href="https://github.com/imartinez/privateGPT-typescript"
|
||||
/>
|
||||
<Card
|
||||
title="Python"
|
||||
title="Python - Ready!"
|
||||
icon="fa-brands fa-python"
|
||||
href="https://github.com/imartinez/privateGPT-python"
|
||||
href="https://github.com/imartinez/pgpt_python"
|
||||
/>
|
||||
<br />
|
||||
</Cards>
|
||||
@ -24,12 +24,12 @@ The clients are kept up to date automatically, so we encourage you to use the la
|
||||
|
||||
<Cards>
|
||||
<Card
|
||||
title="Java"
|
||||
title="Java - WIP"
|
||||
icon="fa-brands fa-java"
|
||||
href="https://github.com/imartinez/privateGPT-java"
|
||||
/>
|
||||
<Card
|
||||
title="Go"
|
||||
title="Go - WIP"
|
||||
icon="fa-brands fa-golang"
|
||||
href="https://github.com/imartinez/privateGPT-go"
|
||||
/>
|
||||
|
||||
@ -30,8 +30,8 @@ pyenv local 3.11
|
||||
PrivateGPT allows to customize the setup -from fully local to cloud based- by deciding the modules to use.
|
||||
Here are the different options available:
|
||||
|
||||
- LLM: "llama-cpp", "ollama", "sagemaker", "openai", "openailike"
|
||||
- Embeddings: "huggingface", "openai", "sagemaker"
|
||||
- LLM: "llama-cpp", "ollama", "sagemaker", "openai", "openailike", "azopenai"
|
||||
- Embeddings: "huggingface", "openai", "sagemaker", "azopenai"
|
||||
- Vector stores: "qdrant", "chroma", "postgres"
|
||||
- UI: whether or not to enable UI (Gradio) or just go with the API
|
||||
|
||||
@ -49,10 +49,12 @@ Where `<extra>` can be any of the following:
|
||||
- llms-sagemaker: adds support for Amazon Sagemaker LLM, requires Sagemaker inference endpoints
|
||||
- llms-openai: adds support for OpenAI LLM, requires OpenAI API key
|
||||
- llms-openai-like: adds support for 3rd party LLM providers that are compatible with OpenAI's API
|
||||
- llms-azopenai: adds support for Azure OpenAI LLM, requires Azure OpenAI inference endpoints
|
||||
- embeddings-ollama: adds support for Ollama Embeddings, requires Ollama running locally
|
||||
- embeddings-huggingface: adds support for local Embeddings using HuggingFace
|
||||
- embeddings-sagemaker: adds support for Amazon Sagemaker Embeddings, requires Sagemaker inference endpoints
|
||||
- embeddings-openai = adds support for OpenAI Embeddings, requires OpenAI API key
|
||||
- embeddings-azopenai = adds support for Azure OpenAI Embeddings, requires Azure OpenAI inference endpoints
|
||||
- vector-stores-qdrant: adds support for Qdrant vector store
|
||||
- vector-stores-chroma: adds support for Chroma DB vector store
|
||||
- vector-stores-postgres: adds support for Postgres vector store
|
||||
@ -160,6 +162,29 @@ PrivateGPT will use the already existing `settings-openai.yaml` settings file, w
|
||||
|
||||
The UI will be available at http://localhost:8001
|
||||
|
||||
### Non-Private, Azure OpenAI-powered test setup
|
||||
|
||||
If you want to test PrivateGPT with Azure OpenAI's LLM and Embeddings -taking into account your data is going to Azure OpenAI!- you can run the following command:
|
||||
|
||||
You need to have access to Azure OpenAI inference endpoints for the LLM and / or the embeddings, and have Azure OpenAI credentials properly configured.
|
||||
|
||||
Edit the `settings-azopenai.yaml` file to include the correct Azure OpenAI endpoints.
|
||||
|
||||
Then, install PrivateGPT with the following command:
|
||||
```bash
|
||||
poetry install --extras "ui llms-azopenai embeddings-azopenai vector-stores-qdrant"
|
||||
```
|
||||
|
||||
Once installed, you can run PrivateGPT.
|
||||
|
||||
```bash
|
||||
PGPT_PROFILES=azopenai make run
|
||||
```
|
||||
|
||||
PrivateGPT will use the already existing `settings-azopenai.yaml` settings file, which is already configured to use Azure OpenAI LLM and Embeddings endpoints, and Qdrant.
|
||||
|
||||
The UI will be available at http://localhost:8001
|
||||
|
||||
### Local, Llama-CPP powered setup
|
||||
|
||||
If you want to run PrivateGPT fully locally without relying on Ollama, you can run the following command:
|
||||
@ -275,6 +300,40 @@ llama_new_context_with_model: total VRAM used: 4857.93 MB (model: 4095.05 MB, co
|
||||
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
|
||||
```
|
||||
|
||||
##### Llama-CPP Linux AMD GPU support
|
||||
|
||||
Linux GPU support is done through ROCm.
|
||||
Some tips:
|
||||
* Install ROCm from [quick-start install guide](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html)
|
||||
* [Install PyTorch for ROCm](https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/install-pytorch.html)
|
||||
```bash
|
||||
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-6.0/torch-2.1.1%2Brocm6.0-cp311-cp311-linux_x86_64.whl
|
||||
poetry run pip install --force-reinstall --no-cache-dir torch-2.1.1+rocm6.0-cp311-cp311-linux_x86_64.whl
|
||||
```
|
||||
* Install bitsandbytes for ROCm
|
||||
```bash
|
||||
PYTORCH_ROCM_ARCH=gfx900,gfx906,gfx908,gfx90a,gfx1030,gfx1100,gfx1101,gfx940,gfx941,gfx942
|
||||
BITSANDBYTES_VERSION=62353b0200b8557026c176e74ac48b84b953a854
|
||||
git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6
|
||||
cd bitsandbytes-rocm-5.6
|
||||
git checkout ${BITSANDBYTES_VERSION}
|
||||
make hip ROCM_TARGET=${PYTORCH_ROCM_ARCH} ROCM_HOME=/opt/rocm/
|
||||
pip install . --extra-index-url https://download.pytorch.org/whl/nightly
|
||||
```
|
||||
|
||||
After that running the following command in the repository will install llama.cpp with GPU support:
|
||||
```bash
|
||||
LLAMA_CPP_PYTHON_VERSION=0.2.56
|
||||
DAMDGPU_TARGETS=gfx900;gfx906;gfx908;gfx90a;gfx1030;gfx1100;gfx1101;gfx940;gfx941;gfx942
|
||||
CMAKE_ARGS="-DLLAMA_HIPBLAS=ON -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DAMDGPU_TARGETS=${DAMDGPU_TARGETS}" poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python==${LLAMA_CPP_PYTHON_VERSION}
|
||||
```
|
||||
|
||||
If your installation was correct, you should see a message similar to the following next time you start the server `BLAS = 1`.
|
||||
|
||||
```
|
||||
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
|
||||
```
|
||||
|
||||
##### Llama-CPP Known issues and Troubleshooting
|
||||
|
||||
Execution of LLMs locally still has a lot of sharp edges, specially when running on non Linux platforms.
|
||||
|
||||
@ -62,6 +62,7 @@ The following ingestion mode exist:
|
||||
* `simple`: historic behavior, ingest one document at a time, sequentially
|
||||
* `batch`: read, parse, and embed multiple documents using batches (batch read, and then batch parse, and then batch embed)
|
||||
* `parallel`: read, parse, and embed multiple documents in parallel. This is the fastest ingestion mode for local setup.
|
||||
* `pipeline`: Alternative to parallel.
|
||||
To change the ingestion mode, you can use the `embedding.ingest_mode` configuration value. The default value is `simple`.
|
||||
|
||||
To configure the number of workers used for parallel or batched ingestion, you can use
|
||||
|
||||
@ -98,6 +98,43 @@ to run an OpenAI compatible server. Then, you can run PrivateGPT using the `sett
|
||||
|
||||
`PGPT_PROFILES=vllm make run`
|
||||
|
||||
### Using Azure OpenAI
|
||||
|
||||
If you cannot run a local model (because you don't have a GPU, for example) or for testing purposes, you may
|
||||
decide to run PrivateGPT using Azure OpenAI as the LLM and Embeddings model.
|
||||
|
||||
In order to do so, create a profile `settings-azopenai.yaml` with the following contents:
|
||||
|
||||
```yaml
|
||||
llm:
|
||||
mode: azopenai
|
||||
|
||||
embedding:
|
||||
mode: azopenai
|
||||
|
||||
azopenai:
|
||||
api_key: <your_azopenai_api_key> # You could skip this configuration and use the AZ_OPENAI_API_KEY env var instead
|
||||
azure_endpoint: <your_azopenai_endpoint> # You could skip this configuration and use the AZ_OPENAI_ENDPOINT env var instead
|
||||
api_version: <api_version> # The API version to use. Default is "2023_05_15"
|
||||
embedding_deployment_name: <your_embedding_deployment_name> # You could skip this configuration and use the AZ_OPENAI_EMBEDDING_DEPLOYMENT_NAME env var instead
|
||||
embedding_model: <openai_embeddings_to_use> # Optional model to use. Default is "text-embedding-ada-002"
|
||||
llm_deployment_name: <your_model_deployment_name> # You could skip this configuration and use the AZ_OPENAI_LLM_DEPLOYMENT_NAME env var instead
|
||||
llm_model: <openai_model_to_use> # Optional model to use. Default is "gpt-35-turbo"
|
||||
```
|
||||
|
||||
And run PrivateGPT loading that profile you just created:
|
||||
|
||||
`PGPT_PROFILES=azopenai make run`
|
||||
|
||||
or
|
||||
|
||||
`PGPT_PROFILES=azopenai poetry run python -m private_gpt`
|
||||
|
||||
When the server is started it will print a log *Application startup complete*.
|
||||
Navigate to http://localhost:8001 to use the Gradio UI or to http://localhost:8001/docs (API section) to try the API.
|
||||
You'll notice the speed and quality of response is higher, given you are using Azure OpenAI's servers for the heavy
|
||||
computations.
|
||||
|
||||
### Using AWS Sagemaker
|
||||
|
||||
For a fully private & performant setup, you can choose to have both your LLM and Embeddings model deployed using Sagemaker.
|
||||
|
||||
66
fern/docs/pages/manual/nodestore.mdx
Normal file
66
fern/docs/pages/manual/nodestore.mdx
Normal file
@ -0,0 +1,66 @@
|
||||
## NodeStores
|
||||
PrivateGPT supports **Simple** and [Postgres](https://www.postgresql.org/) providers. Simple being the default.
|
||||
|
||||
In order to select one or the other, set the `nodestore.database` property in the `settings.yaml` file to `simple` or `postgres`.
|
||||
|
||||
```yaml
|
||||
nodestore:
|
||||
database: simple
|
||||
```
|
||||
|
||||
### Simple Document Store
|
||||
|
||||
Setting up simple document store: Persist data with in-memory and disk storage.
|
||||
|
||||
Enabling the simple document store is an excellent choice for small projects or proofs of concept where you need to persist data while maintaining minimal setup complexity. To get started, set the nodestore.database property in your settings.yaml file as follows:
|
||||
|
||||
```yaml
|
||||
nodestore:
|
||||
database: simple
|
||||
```
|
||||
The beauty of the simple document store is its flexibility and ease of implementation. It provides a solid foundation for managing and retrieving data without the need for complex setup or configuration. The combination of in-memory processing and disk persistence ensures that you can efficiently handle small to medium-sized datasets while maintaining data consistency across runs.
|
||||
|
||||
### Postgres Document Store
|
||||
|
||||
To enable Postgres, set the `nodestore.database` property in the `settings.yaml` file to `postgres` and install the `storage-nodestore-postgres` extra. Note: Vector Embeddings Storage in Postgres is configured separately
|
||||
|
||||
```bash
|
||||
poetry install --extras storage-nodestore-postgres
|
||||
```
|
||||
|
||||
The available configuration options are:
|
||||
| Field | Description |
|
||||
|---------------|-----------------------------------------------------------|
|
||||
| **host** | The server hosting the Postgres database. Default is `localhost` |
|
||||
| **port** | The port on which the Postgres database is accessible. Default is `5432` |
|
||||
| **database** | The specific database to connect to. Default is `postgres` |
|
||||
| **user** | The username for database access. Default is `postgres` |
|
||||
| **password** | The password for database access. (Required) |
|
||||
| **schema_name** | The database schema to use. Default is `private_gpt` |
|
||||
|
||||
For example:
|
||||
```yaml
|
||||
nodestore:
|
||||
database: postgres
|
||||
|
||||
postgres:
|
||||
host: localhost
|
||||
port: 5432
|
||||
database: postgres
|
||||
user: postgres
|
||||
password: <PASSWORD>
|
||||
schema_name: private_gpt
|
||||
```
|
||||
|
||||
Given the above configuration, Two PostgreSQL tables will be created upon successful connection: one for storing metadata related to the index and another for document data itself.
|
||||
|
||||
```
|
||||
postgres=# \dt private_gpt.*
|
||||
List of relations
|
||||
Schema | Name | Type | Owner
|
||||
-------------+-----------------+-------+--------------
|
||||
private_gpt | data_docstore | table | postgres
|
||||
private_gpt | data_indexstore | table | postgres
|
||||
|
||||
postgres=#
|
||||
```
|
||||
36
fern/docs/pages/manual/reranker.mdx
Normal file
36
fern/docs/pages/manual/reranker.mdx
Normal file
@ -0,0 +1,36 @@
|
||||
## Enhancing Response Quality with Reranking
|
||||
|
||||
PrivateGPT offers a reranking feature aimed at optimizing response generation by filtering out irrelevant documents, potentially leading to faster response times and enhanced relevance of answers generated by the LLM.
|
||||
|
||||
### Enabling Reranking
|
||||
|
||||
Document reranking can significantly improve the efficiency and quality of the responses by pre-selecting the most relevant documents before generating an answer. To leverage this feature, ensure that it is enabled in the RAG settings and consider adjusting the parameters to best fit your use case.
|
||||
|
||||
#### Additional Requirements
|
||||
|
||||
Before enabling reranking, you must install additional dependencies:
|
||||
|
||||
```bash
|
||||
poetry install --extras rerank-sentence-transformers
|
||||
```
|
||||
|
||||
This command installs dependencies for the cross-encoder reranker from sentence-transformers, which is currently the only supported method by PrivateGPT for document reranking.
|
||||
|
||||
#### Configuration
|
||||
|
||||
To enable and configure reranking, adjust the `rag` section within the `settings.yaml` file. Here are the key settings to consider:
|
||||
|
||||
- `similarity_top_k`: Determines the number of documents to initially retrieve and consider for reranking. This value should be larger than `top_n`.
|
||||
- `rerank`:
|
||||
- `enabled`: Set to `true` to activate the reranking feature.
|
||||
- `top_n`: Specifies the number of documents to use in the final answer generation process, chosen from the top-ranked documents provided by `similarity_top_k`.
|
||||
|
||||
Example configuration snippet:
|
||||
|
||||
```yaml
|
||||
rag:
|
||||
similarity_top_k: 10 # Number of documents to retrieve and consider for reranking
|
||||
rerank:
|
||||
enabled: true
|
||||
top_n: 3 # Number of top-ranked documents to use for generating the answer
|
||||
```
|
||||
@ -1,7 +1,7 @@
|
||||
## Vectorstores
|
||||
PrivateGPT supports [Qdrant](https://qdrant.tech/), [Chroma](https://www.trychroma.com/) and [PGVector](https://github.com/pgvector/pgvector) as vectorstore providers. Qdrant being the default.
|
||||
|
||||
In order to select one or the other, set the `vectorstore.database` property in the `settings.yaml` file to `qdrant`, `chroma` or `pgvector`.
|
||||
In order to select one or the other, set the `vectorstore.database` property in the `settings.yaml` file to `qdrant`, `chroma` or `postgres`.
|
||||
|
||||
```yaml
|
||||
vectorstore:
|
||||
@ -50,14 +50,15 @@ poetry install --extras chroma
|
||||
By default `chroma` will use a disk-based database stored in local_data_path / "chroma_db" (being local_data_path defined in settings.yaml)
|
||||
|
||||
### PGVector
|
||||
To use the PGVector store a [postgreSQL](https://www.postgresql.org/) database with the PGVector extension must be used.
|
||||
|
||||
To enable PGVector, set the `vectorstore.database` property in the `settings.yaml` file to `pgvector` and install the `pgvector` extra.
|
||||
To enable PGVector, set the `vectorstore.database` property in the `settings.yaml` file to `postgres` and install the `vector-stores-postgres` extra.
|
||||
|
||||
```bash
|
||||
poetry install --extras pgvector
|
||||
poetry install --extras vector-stores-postgres
|
||||
```
|
||||
|
||||
PGVector settings can be configured by setting values to the `pgvector` property in the `settings.yaml` file.
|
||||
PGVector settings can be configured by setting values to the `postgres` property in the `settings.yaml` file.
|
||||
|
||||
The available configuration options are:
|
||||
| Field | Description |
|
||||
@ -67,19 +68,36 @@ The available configuration options are:
|
||||
| **database** | The specific database to connect to. Default is `postgres` |
|
||||
| **user** | The username for database access. Default is `postgres` |
|
||||
| **password** | The password for database access. (Required) |
|
||||
| **embed_dim** | The dimensionality of the embedding model (Required) |
|
||||
| **schema_name** | The database schema to use. Default is `private_gpt` |
|
||||
| **table_name** | The database table to use. Default is `embeddings` |
|
||||
|
||||
For example:
|
||||
```yaml
|
||||
pgvector:
|
||||
vectorstore:
|
||||
database: postgres
|
||||
|
||||
postgres:
|
||||
host: localhost
|
||||
port: 5432
|
||||
database: postgres
|
||||
user: postgres
|
||||
password: <PASSWORD>
|
||||
embed_dim: 384 # 384 is for BAAI/bge-small-en-v1.5
|
||||
schema_name: private_gpt
|
||||
table_name: embeddings
|
||||
```
|
||||
|
||||
The following table will be created in the database
|
||||
```
|
||||
postgres=# \d private_gpt.data_embeddings
|
||||
Table "private_gpt.data_embeddings"
|
||||
Column | Type | Collation | Nullable | Default
|
||||
-----------+-------------------+-----------+----------+---------------------------------------------------------
|
||||
id | bigint | | not null | nextval('private_gpt.data_embeddings_id_seq'::regclass)
|
||||
text | character varying | | not null |
|
||||
metadata_ | json | | |
|
||||
node_id | character varying | | |
|
||||
embedding | vector(768) | | |
|
||||
Indexes:
|
||||
"data_embeddings_pkey" PRIMARY KEY, btree (id)
|
||||
|
||||
postgres=#
|
||||
```
|
||||
The dimensions of the embeddings columns will be set based on the `embedding.embed_dim` value. If the embedding model changes this table may need to be dropped and recreated to avoid a dimension mismatch.
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
{
|
||||
"organization": "privategpt",
|
||||
"version": "0.17.2"
|
||||
"version": "0.19.10"
|
||||
}
|
||||
4485
poetry.lock
generated
4485
poetry.lock
generated
File diff suppressed because it is too large
Load Diff
@ -69,7 +69,25 @@ class EmbeddingComponent:
|
||||
ollama_settings = settings.ollama
|
||||
self.embedding_model = OllamaEmbedding(
|
||||
model_name=ollama_settings.embedding_model,
|
||||
base_url=ollama_settings.api_base,
|
||||
base_url=ollama_settings.embedding_api_base,
|
||||
)
|
||||
case "azopenai":
|
||||
try:
|
||||
from llama_index.embeddings.azure_openai import ( # type: ignore
|
||||
AzureOpenAIEmbedding,
|
||||
)
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
"Azure OpenAI dependencies not found, install with `poetry install --extras embeddings-azopenai`"
|
||||
) from e
|
||||
|
||||
azopenai_settings = settings.azopenai
|
||||
self.embedding_model = AzureOpenAIEmbedding(
|
||||
model=azopenai_settings.embedding_model,
|
||||
deployment_name=azopenai_settings.embedding_deployment_name,
|
||||
api_key=azopenai_settings.api_key,
|
||||
azure_endpoint=azopenai_settings.azure_endpoint,
|
||||
api_version=azopenai_settings.api_version,
|
||||
)
|
||||
case "mock":
|
||||
# Not a random number, is the dimensionality used by
|
||||
|
||||
@ -6,6 +6,7 @@ import multiprocessing.pool
|
||||
import os
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from queue import Queue
|
||||
from typing import Any
|
||||
|
||||
from llama_index.core.data_structs import IndexDict
|
||||
@ -13,12 +14,13 @@ from llama_index.core.embeddings.utils import EmbedType
|
||||
from llama_index.core.indices import VectorStoreIndex, load_index_from_storage
|
||||
from llama_index.core.indices.base import BaseIndex
|
||||
from llama_index.core.ingestion import run_transformations
|
||||
from llama_index.core.schema import Document, TransformComponent
|
||||
from llama_index.core.schema import BaseNode, Document, TransformComponent
|
||||
from llama_index.core.storage import StorageContext
|
||||
|
||||
from private_gpt.components.ingest.ingest_helper import IngestionHelper
|
||||
from private_gpt.paths import local_data_path
|
||||
from private_gpt.settings.settings import Settings
|
||||
from private_gpt.utils.eta import eta
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -314,6 +316,170 @@ class ParallelizedIngestComponent(BaseIngestComponentWithIndex):
|
||||
self._file_to_documents_work_pool.terminate()
|
||||
|
||||
|
||||
class PipelineIngestComponent(BaseIngestComponentWithIndex):
|
||||
"""Pipeline ingestion - keeping the embedding worker pool as busy as possible.
|
||||
|
||||
This class implements a threaded ingestion pipeline, which comprises two threads
|
||||
and two queues. The primary thread is responsible for reading and parsing files
|
||||
into documents. These documents are then placed into a queue, which is
|
||||
distributed to a pool of worker processes for embedding computation. After
|
||||
embedding, the documents are transferred to another queue where they are
|
||||
accumulated until a threshold is reached. Upon reaching this threshold, the
|
||||
accumulated documents are flushed to the document store, index, and vector
|
||||
store.
|
||||
|
||||
Exception handling ensures robustness against erroneous files. However, in the
|
||||
pipelined design, one error can lead to the discarding of multiple files. Any
|
||||
discarded files will be reported.
|
||||
"""
|
||||
|
||||
NODE_FLUSH_COUNT = 5000 # Save the index every # nodes.
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_context: StorageContext,
|
||||
embed_model: EmbedType,
|
||||
transformations: list[TransformComponent],
|
||||
count_workers: int,
|
||||
*args: Any,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
super().__init__(storage_context, embed_model, transformations, *args, **kwargs)
|
||||
self.count_workers = count_workers
|
||||
assert (
|
||||
len(self.transformations) >= 2
|
||||
), "Embeddings must be in the transformations"
|
||||
assert count_workers > 0, "count_workers must be > 0"
|
||||
self.count_workers = count_workers
|
||||
# We are doing our own multiprocessing
|
||||
# To do not collide with the multiprocessing of huggingface, we disable it
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
# doc_q stores parsed files as Document chunks.
|
||||
# Using a shallow queue causes the filesystem parser to block
|
||||
# when it reaches capacity. This ensures it doesn't outpace the
|
||||
# computationally intensive embeddings phase, avoiding unnecessary
|
||||
# memory consumption. The semaphore is used to bound the async worker
|
||||
# embedding computations to cause the doc Q to fill and block.
|
||||
self.doc_semaphore = multiprocessing.Semaphore(
|
||||
self.count_workers
|
||||
) # limit the doc queue to # items.
|
||||
self.doc_q: Queue[tuple[str, str | None, list[Document] | None]] = Queue(20)
|
||||
# node_q stores documents parsed into nodes (embeddings).
|
||||
# Larger queue size so we don't block the embedding workers during a slow
|
||||
# index update.
|
||||
self.node_q: Queue[
|
||||
tuple[str, str | None, list[Document] | None, list[BaseNode] | None]
|
||||
] = Queue(40)
|
||||
threading.Thread(target=self._doc_to_node, daemon=True).start()
|
||||
threading.Thread(target=self._write_nodes, daemon=True).start()
|
||||
|
||||
def _doc_to_node(self) -> None:
|
||||
# Parse documents into nodes
|
||||
with multiprocessing.pool.ThreadPool(processes=self.count_workers) as pool:
|
||||
while True:
|
||||
try:
|
||||
cmd, file_name, documents = self.doc_q.get(
|
||||
block=True
|
||||
) # Documents for a file
|
||||
if cmd == "process":
|
||||
# Push CPU/GPU embedding work to the worker pool
|
||||
# Acquire semaphore to control access to worker pool
|
||||
self.doc_semaphore.acquire()
|
||||
pool.apply_async(
|
||||
self._doc_to_node_worker, (file_name, documents)
|
||||
)
|
||||
elif cmd == "quit":
|
||||
break
|
||||
finally:
|
||||
if cmd != "process":
|
||||
self.doc_q.task_done() # unblock Q joins
|
||||
|
||||
def _doc_to_node_worker(self, file_name: str, documents: list[Document]) -> None:
|
||||
# CPU/GPU intensive work in its own process
|
||||
try:
|
||||
nodes = run_transformations(
|
||||
documents, # type: ignore[arg-type]
|
||||
self.transformations,
|
||||
show_progress=self.show_progress,
|
||||
)
|
||||
self.node_q.put(("process", file_name, documents, nodes))
|
||||
finally:
|
||||
self.doc_semaphore.release()
|
||||
self.doc_q.task_done() # unblock Q joins
|
||||
|
||||
def _save_docs(
|
||||
self, files: list[str], documents: list[Document], nodes: list[BaseNode]
|
||||
) -> None:
|
||||
try:
|
||||
logger.info(
|
||||
f"Saving {len(files)} files ({len(documents)} documents / {len(nodes)} nodes)"
|
||||
)
|
||||
self._index.insert_nodes(nodes)
|
||||
for document in documents:
|
||||
self._index.docstore.set_document_hash(
|
||||
document.get_doc_id(), document.hash
|
||||
)
|
||||
self._save_index()
|
||||
except Exception:
|
||||
# Tell the user so they can investigate these files
|
||||
logger.exception(f"Processing files {files}")
|
||||
finally:
|
||||
# Clearing work, even on exception, maintains a clean state.
|
||||
nodes.clear()
|
||||
documents.clear()
|
||||
files.clear()
|
||||
|
||||
def _write_nodes(self) -> None:
|
||||
# Save nodes to index. I/O intensive.
|
||||
node_stack: list[BaseNode] = []
|
||||
doc_stack: list[Document] = []
|
||||
file_stack: list[str] = []
|
||||
while True:
|
||||
try:
|
||||
cmd, file_name, documents, nodes = self.node_q.get(block=True)
|
||||
if cmd in ("flush", "quit"):
|
||||
if file_stack:
|
||||
self._save_docs(file_stack, doc_stack, node_stack)
|
||||
if cmd == "quit":
|
||||
break
|
||||
elif cmd == "process":
|
||||
node_stack.extend(nodes) # type: ignore[arg-type]
|
||||
doc_stack.extend(documents) # type: ignore[arg-type]
|
||||
file_stack.append(file_name) # type: ignore[arg-type]
|
||||
# Constant saving is heavy on I/O - accumulate to a threshold
|
||||
if len(node_stack) >= self.NODE_FLUSH_COUNT:
|
||||
self._save_docs(file_stack, doc_stack, node_stack)
|
||||
finally:
|
||||
self.node_q.task_done()
|
||||
|
||||
def _flush(self) -> None:
|
||||
self.doc_q.put(("flush", None, None))
|
||||
self.doc_q.join()
|
||||
self.node_q.put(("flush", None, None, None))
|
||||
self.node_q.join()
|
||||
|
||||
def ingest(self, file_name: str, file_data: Path) -> list[Document]:
|
||||
documents = IngestionHelper.transform_file_into_documents(file_name, file_data)
|
||||
self.doc_q.put(("process", file_name, documents))
|
||||
self._flush()
|
||||
return documents
|
||||
|
||||
def bulk_ingest(self, files: list[tuple[str, Path]]) -> list[Document]:
|
||||
docs = []
|
||||
for file_name, file_data in eta(files):
|
||||
try:
|
||||
documents = IngestionHelper.transform_file_into_documents(
|
||||
file_name, file_data
|
||||
)
|
||||
self.doc_q.put(("process", file_name, documents))
|
||||
docs.extend(documents)
|
||||
except Exception:
|
||||
logger.exception(f"Skipping {file_data.name}")
|
||||
self._flush()
|
||||
return docs
|
||||
|
||||
|
||||
def get_ingestion_component(
|
||||
storage_context: StorageContext,
|
||||
embed_model: EmbedType,
|
||||
@ -336,6 +502,13 @@ def get_ingestion_component(
|
||||
transformations=transformations,
|
||||
count_workers=settings.embedding.count_workers,
|
||||
)
|
||||
elif ingest_mode == "pipeline":
|
||||
return PipelineIngestComponent(
|
||||
storage_context=storage_context,
|
||||
embed_model=embed_model,
|
||||
transformations=transformations,
|
||||
count_workers=settings.embedding.count_workers,
|
||||
)
|
||||
else:
|
||||
return SimpleIngestComponent(
|
||||
storage_context=storage_context,
|
||||
|
||||
@ -1,4 +1,6 @@
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
from injector import inject, singleton
|
||||
from llama_index.core.llms import LLM, MockLLM
|
||||
@ -139,6 +141,43 @@ class LLMComponent:
|
||||
temperature=settings.llm.temperature,
|
||||
context_window=settings.llm.context_window,
|
||||
additional_kwargs=settings_kwargs,
|
||||
request_timeout=ollama_settings.request_timeout,
|
||||
)
|
||||
|
||||
if (
|
||||
ollama_settings.keep_alive
|
||||
!= ollama_settings.model_fields["keep_alive"].default
|
||||
):
|
||||
# Modify Ollama methods to use the "keep_alive" field.
|
||||
def add_keep_alive(func: Callable[..., Any]) -> Callable[..., Any]:
|
||||
def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
kwargs["keep_alive"] = ollama_settings.keep_alive
|
||||
return func(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
Ollama.chat = add_keep_alive(Ollama.chat)
|
||||
Ollama.stream_chat = add_keep_alive(Ollama.stream_chat)
|
||||
Ollama.complete = add_keep_alive(Ollama.complete)
|
||||
Ollama.stream_complete = add_keep_alive(Ollama.stream_complete)
|
||||
|
||||
case "azopenai":
|
||||
try:
|
||||
from llama_index.llms.azure_openai import ( # type: ignore
|
||||
AzureOpenAI,
|
||||
)
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
"Azure OpenAI dependencies not found, install with `poetry install --extras llms-azopenai`"
|
||||
) from e
|
||||
|
||||
azopenai_settings = settings.azopenai
|
||||
self.llm = AzureOpenAI(
|
||||
model=azopenai_settings.llm_model,
|
||||
deployment_name=azopenai_settings.llm_deployment_name,
|
||||
api_key=azopenai_settings.api_key,
|
||||
azure_endpoint=azopenai_settings.azure_endpoint,
|
||||
api_version=azopenai_settings.api_version,
|
||||
)
|
||||
case "mock":
|
||||
self.llm = MockLLM()
|
||||
|
||||
@ -6,6 +6,7 @@ from llama_index.core.storage.index_store import SimpleIndexStore
|
||||
from llama_index.core.storage.index_store.types import BaseIndexStore
|
||||
|
||||
from private_gpt.paths import local_data_path
|
||||
from private_gpt.settings.settings import Settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -16,19 +17,51 @@ class NodeStoreComponent:
|
||||
doc_store: BaseDocumentStore
|
||||
|
||||
@inject
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
self.index_store = SimpleIndexStore.from_persist_dir(
|
||||
persist_dir=str(local_data_path)
|
||||
)
|
||||
except FileNotFoundError:
|
||||
logger.debug("Local index store not found, creating a new one")
|
||||
self.index_store = SimpleIndexStore()
|
||||
def __init__(self, settings: Settings) -> None:
|
||||
match settings.nodestore.database:
|
||||
case "simple":
|
||||
try:
|
||||
self.index_store = SimpleIndexStore.from_persist_dir(
|
||||
persist_dir=str(local_data_path)
|
||||
)
|
||||
except FileNotFoundError:
|
||||
logger.debug("Local index store not found, creating a new one")
|
||||
self.index_store = SimpleIndexStore()
|
||||
|
||||
try:
|
||||
self.doc_store = SimpleDocumentStore.from_persist_dir(
|
||||
persist_dir=str(local_data_path)
|
||||
)
|
||||
except FileNotFoundError:
|
||||
logger.debug("Local document store not found, creating a new one")
|
||||
self.doc_store = SimpleDocumentStore()
|
||||
try:
|
||||
self.doc_store = SimpleDocumentStore.from_persist_dir(
|
||||
persist_dir=str(local_data_path)
|
||||
)
|
||||
except FileNotFoundError:
|
||||
logger.debug("Local document store not found, creating a new one")
|
||||
self.doc_store = SimpleDocumentStore()
|
||||
|
||||
case "postgres":
|
||||
try:
|
||||
from llama_index.core.storage.docstore.postgres_docstore import (
|
||||
PostgresDocumentStore,
|
||||
)
|
||||
from llama_index.core.storage.index_store.postgres_index_store import (
|
||||
PostgresIndexStore,
|
||||
)
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"Postgres dependencies not found, install with `poetry install --extras storage-nodestore-postgres`"
|
||||
) from None
|
||||
|
||||
if settings.postgres is None:
|
||||
raise ValueError("Postgres index/doc store settings not found.")
|
||||
|
||||
self.index_store = PostgresIndexStore.from_params(
|
||||
**settings.postgres.model_dump(exclude_none=True)
|
||||
)
|
||||
self.doc_store = PostgresDocumentStore.from_params(
|
||||
**settings.postgres.model_dump(exclude_none=True)
|
||||
)
|
||||
|
||||
case _:
|
||||
# Should be unreachable
|
||||
# The settings validator should have caught this
|
||||
raise ValueError(
|
||||
f"Database {settings.nodestore.database} not supported"
|
||||
)
|
||||
|
||||
@ -38,7 +38,7 @@ class VectorStoreComponent:
|
||||
def __init__(self, settings: Settings) -> None:
|
||||
self.settings = settings
|
||||
match settings.vectorstore.database:
|
||||
case "pgvector":
|
||||
case "postgres":
|
||||
try:
|
||||
from llama_index.vector_stores.postgres import ( # type: ignore
|
||||
PGVectorStore,
|
||||
@ -48,15 +48,17 @@ class VectorStoreComponent:
|
||||
"Postgres dependencies not found, install with `poetry install --extras vector-stores-postgres`"
|
||||
) from e
|
||||
|
||||
if settings.pgvector is None:
|
||||
if settings.postgres is None:
|
||||
raise ValueError(
|
||||
"PGVectorStore settings not found. Please provide settings."
|
||||
"Postgres settings not found. Please provide settings."
|
||||
)
|
||||
|
||||
self.vector_store = typing.cast(
|
||||
VectorStore,
|
||||
PGVectorStore.from_params(
|
||||
**settings.pgvector.model_dump(exclude_none=True)
|
||||
**settings.postgres.model_dump(exclude_none=True),
|
||||
table_name="embeddings",
|
||||
embed_dim=settings.embedding.embed_dim,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@ -8,6 +8,10 @@ from llama_index.core.chat_engine.types import (
|
||||
from llama_index.core.indices import VectorStoreIndex
|
||||
from llama_index.core.indices.postprocessor import MetadataReplacementPostProcessor
|
||||
from llama_index.core.llms import ChatMessage, MessageRole
|
||||
from llama_index.core.postprocessor import (
|
||||
SentenceTransformerRerank,
|
||||
SimilarityPostprocessor,
|
||||
)
|
||||
from llama_index.core.storage import StorageContext
|
||||
from llama_index.core.types import TokenGen
|
||||
from pydantic import BaseModel
|
||||
@ -20,6 +24,7 @@ from private_gpt.components.vector_store.vector_store_component import (
|
||||
)
|
||||
from private_gpt.open_ai.extensions.context_filter import ContextFilter
|
||||
from private_gpt.server.chunks.chunks_service import Chunk
|
||||
from private_gpt.settings.settings import Settings
|
||||
|
||||
|
||||
class Completion(BaseModel):
|
||||
@ -68,14 +73,18 @@ class ChatEngineInput:
|
||||
|
||||
@singleton
|
||||
class ChatService:
|
||||
settings: Settings
|
||||
|
||||
@inject
|
||||
def __init__(
|
||||
self,
|
||||
settings: Settings,
|
||||
llm_component: LLMComponent,
|
||||
vector_store_component: VectorStoreComponent,
|
||||
embedding_component: EmbeddingComponent,
|
||||
node_store_component: NodeStoreComponent,
|
||||
) -> None:
|
||||
self.settings = settings
|
||||
self.llm_component = llm_component
|
||||
self.embedding_component = embedding_component
|
||||
self.vector_store_component = vector_store_component
|
||||
@ -98,17 +107,31 @@ class ChatService:
|
||||
use_context: bool = False,
|
||||
context_filter: ContextFilter | None = None,
|
||||
) -> BaseChatEngine:
|
||||
settings = self.settings
|
||||
if use_context:
|
||||
vector_index_retriever = self.vector_store_component.get_retriever(
|
||||
index=self.index, context_filter=context_filter
|
||||
index=self.index,
|
||||
context_filter=context_filter,
|
||||
similarity_top_k=self.settings.rag.similarity_top_k,
|
||||
)
|
||||
node_postprocessors = [
|
||||
MetadataReplacementPostProcessor(target_metadata_key="window"),
|
||||
SimilarityPostprocessor(
|
||||
similarity_cutoff=settings.rag.similarity_value
|
||||
),
|
||||
]
|
||||
|
||||
if settings.rag.rerank.enabled:
|
||||
rerank_postprocessor = SentenceTransformerRerank(
|
||||
model=settings.rag.rerank.model, top_n=settings.rag.rerank.top_n
|
||||
)
|
||||
node_postprocessors.append(rerank_postprocessor)
|
||||
|
||||
return ContextChatEngine.from_defaults(
|
||||
system_prompt=system_prompt,
|
||||
retriever=vector_index_retriever,
|
||||
llm=self.llm_component.llm, # Takes no effect at the moment
|
||||
node_postprocessors=[
|
||||
MetadataReplacementPostProcessor(target_metadata_key="window"),
|
||||
],
|
||||
node_postprocessors=node_postprocessors,
|
||||
)
|
||||
else:
|
||||
return SimpleChatEngine.from_defaults(
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
import logging
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import AnyStr, BinaryIO
|
||||
from typing import TYPE_CHECKING, AnyStr, BinaryIO
|
||||
|
||||
from injector import inject, singleton
|
||||
from llama_index.core.node_parser import SentenceWindowNodeParser
|
||||
@ -17,6 +17,9 @@ from private_gpt.components.vector_store.vector_store_component import (
|
||||
from private_gpt.server.ingest.model import IngestedDoc
|
||||
from private_gpt.settings.settings import settings
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from llama_index.core.storage.docstore.types import RefDocInfo
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@ -86,17 +89,15 @@ class IngestService:
|
||||
return [IngestedDoc.from_document(document) for document in documents]
|
||||
|
||||
def list_ingested(self) -> list[IngestedDoc]:
|
||||
ingested_docs = []
|
||||
ingested_docs: list[IngestedDoc] = []
|
||||
try:
|
||||
docstore = self.storage_context.docstore
|
||||
ingested_docs_ids: set[str] = set()
|
||||
ref_docs: dict[str, RefDocInfo] | None = docstore.get_all_ref_doc_info()
|
||||
|
||||
for node in docstore.docs.values():
|
||||
if node.ref_doc_id is not None:
|
||||
ingested_docs_ids.add(node.ref_doc_id)
|
||||
if not ref_docs:
|
||||
return ingested_docs
|
||||
|
||||
for doc_id in ingested_docs_ids:
|
||||
ref_doc_info = docstore.get_ref_doc_info(ref_doc_id=doc_id)
|
||||
for doc_id, ref_doc_info in ref_docs.items():
|
||||
doc_metadata = None
|
||||
if ref_doc_info is not None and ref_doc_info.metadata is not None:
|
||||
doc_metadata = IngestedDoc.curate_metadata(ref_doc_info.metadata)
|
||||
|
||||
@ -81,7 +81,9 @@ class DataSettings(BaseModel):
|
||||
|
||||
|
||||
class LLMSettings(BaseModel):
|
||||
mode: Literal["llamacpp", "openai", "openailike", "sagemaker", "mock", "ollama"]
|
||||
mode: Literal[
|
||||
"llamacpp", "openai", "openailike", "azopenai", "sagemaker", "mock", "ollama"
|
||||
]
|
||||
max_new_tokens: int = Field(
|
||||
256,
|
||||
description="The maximum number of token that the LLM is authorized to generate in one completion.",
|
||||
@ -105,7 +107,11 @@ class LLMSettings(BaseModel):
|
||||
|
||||
|
||||
class VectorstoreSettings(BaseModel):
|
||||
database: Literal["chroma", "qdrant", "pgvector"]
|
||||
database: Literal["chroma", "qdrant", "postgres"]
|
||||
|
||||
|
||||
class NodeStoreSettings(BaseModel):
|
||||
database: Literal["simple", "postgres"]
|
||||
|
||||
|
||||
class LlamaCPPSettings(BaseModel):
|
||||
@ -148,14 +154,15 @@ class HuggingFaceSettings(BaseModel):
|
||||
|
||||
|
||||
class EmbeddingSettings(BaseModel):
|
||||
mode: Literal["huggingface", "openai", "sagemaker", "ollama", "mock"]
|
||||
ingest_mode: Literal["simple", "batch", "parallel"] = Field(
|
||||
mode: Literal["huggingface", "openai", "azopenai", "sagemaker", "ollama", "mock"]
|
||||
ingest_mode: Literal["simple", "batch", "parallel", "pipeline"] = Field(
|
||||
"simple",
|
||||
description=(
|
||||
"The ingest mode to use for the embedding engine:\n"
|
||||
"If `simple` - ingest files sequentially and one by one. It is the historic behaviour.\n"
|
||||
"If `batch` - if multiple files, parse all the files in parallel, "
|
||||
"and send them in batch to the embedding model.\n"
|
||||
"In `pipeline` - The Embedding engine is kept as busy as possible\n"
|
||||
"If `parallel` - parse the files in parallel using multiple cores, and embedd them in parallel.\n"
|
||||
"`parallel` is the fastest mode for local setup, as it parallelize IO RW in the index.\n"
|
||||
"For modes that leverage parallelization, you can specify the number of "
|
||||
@ -168,11 +175,16 @@ class EmbeddingSettings(BaseModel):
|
||||
"The number of workers to use for file ingestion.\n"
|
||||
"In `batch` mode, this is the number of workers used to parse the files.\n"
|
||||
"In `parallel` mode, this is the number of workers used to parse the files and embed them.\n"
|
||||
"In `pipeline` mode, this is the number of workers that can perform embeddings.\n"
|
||||
"This is only used if `ingest_mode` is not `simple`.\n"
|
||||
"Do not go too high with this number, as it might cause memory issues. (especially in `parallel` mode)\n"
|
||||
"Do not set it higher than your number of threads of your CPU."
|
||||
),
|
||||
)
|
||||
embed_dim: int = Field(
|
||||
384,
|
||||
description="The dimension of the embeddings stored in the Postgres database",
|
||||
)
|
||||
|
||||
|
||||
class SagemakerSettings(BaseModel):
|
||||
@ -197,6 +209,10 @@ class OllamaSettings(BaseModel):
|
||||
"http://localhost:11434",
|
||||
description="Base URL of Ollama API. Example: 'https://localhost:11434'.",
|
||||
)
|
||||
embedding_api_base: str = Field(
|
||||
"http://localhost:11434",
|
||||
description="Base URL of Ollama embedding API. Example: 'https://localhost:11434'.",
|
||||
)
|
||||
llm_model: str = Field(
|
||||
None,
|
||||
description="Model to use. Example: 'llama2-uncensored'.",
|
||||
@ -205,6 +221,10 @@ class OllamaSettings(BaseModel):
|
||||
None,
|
||||
description="Model to use. Example: 'nomic-embed-text'.",
|
||||
)
|
||||
keep_alive: str = Field(
|
||||
"5m",
|
||||
description="Time the model will stay loaded in memory after a request. examples: 5m, 5h, '-1' ",
|
||||
)
|
||||
tfs_z: float = Field(
|
||||
1.0,
|
||||
description="Tail free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting.",
|
||||
@ -229,6 +249,29 @@ class OllamaSettings(BaseModel):
|
||||
1.1,
|
||||
description="Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.1)",
|
||||
)
|
||||
request_timeout: float = Field(
|
||||
120.0,
|
||||
description="Time elapsed until ollama times out the request. Default is 120s. Format is float. ",
|
||||
)
|
||||
|
||||
|
||||
class AzureOpenAISettings(BaseModel):
|
||||
api_key: str
|
||||
azure_endpoint: str
|
||||
api_version: str = Field(
|
||||
"2023_05_15",
|
||||
description="The API version to use for this operation. This follows the YYYY-MM-DD format.",
|
||||
)
|
||||
embedding_deployment_name: str
|
||||
embedding_model: str = Field(
|
||||
"text-embedding-ada-002",
|
||||
description="OpenAI Model to use. Example: 'text-embedding-ada-002'.",
|
||||
)
|
||||
llm_deployment_name: str
|
||||
llm_model: str = Field(
|
||||
"gpt-35-turbo",
|
||||
description="OpenAI Model to use. Example: 'gpt-4'.",
|
||||
)
|
||||
|
||||
|
||||
class UISettings(BaseModel):
|
||||
@ -249,7 +292,34 @@ class UISettings(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
class PGVectorSettings(BaseModel):
|
||||
class RerankSettings(BaseModel):
|
||||
enabled: bool = Field(
|
||||
False,
|
||||
description="This value controls whether a reranker should be included in the RAG pipeline.",
|
||||
)
|
||||
model: str = Field(
|
||||
"cross-encoder/ms-marco-MiniLM-L-2-v2",
|
||||
description="Rerank model to use. Limited to SentenceTransformer cross-encoder models.",
|
||||
)
|
||||
top_n: int = Field(
|
||||
2,
|
||||
description="This value controls the number of documents returned by the RAG pipeline.",
|
||||
)
|
||||
|
||||
|
||||
class RagSettings(BaseModel):
|
||||
similarity_top_k: int = Field(
|
||||
2,
|
||||
description="This value controls the number of documents returned by the RAG pipeline or considered for reranking if enabled.",
|
||||
)
|
||||
similarity_value: float = Field(
|
||||
None,
|
||||
description="If set, any documents retrieved from the RAG must meet a certain match score. Acceptable values are between 0 and 1.",
|
||||
)
|
||||
rerank: RerankSettings
|
||||
|
||||
|
||||
class PostgresSettings(BaseModel):
|
||||
host: str = Field(
|
||||
"localhost",
|
||||
description="The server hosting the Postgres database",
|
||||
@ -270,17 +340,9 @@ class PGVectorSettings(BaseModel):
|
||||
"postgres",
|
||||
description="The database to use to connect to the Postgres database",
|
||||
)
|
||||
embed_dim: int = Field(
|
||||
384,
|
||||
description="The dimension of the embeddings stored in the Postgres database",
|
||||
)
|
||||
schema_name: str = Field(
|
||||
"public",
|
||||
description="The name of the schema in the Postgres database where the embeddings are stored",
|
||||
)
|
||||
table_name: str = Field(
|
||||
"embeddings",
|
||||
description="The name of the table in the Postgres database where the embeddings are stored",
|
||||
description="The name of the schema in the Postgres database to use",
|
||||
)
|
||||
|
||||
|
||||
@ -349,9 +411,12 @@ class Settings(BaseModel):
|
||||
sagemaker: SagemakerSettings
|
||||
openai: OpenAISettings
|
||||
ollama: OllamaSettings
|
||||
azopenai: AzureOpenAISettings
|
||||
vectorstore: VectorstoreSettings
|
||||
nodestore: NodeStoreSettings
|
||||
rag: RagSettings
|
||||
qdrant: QdrantSettings | None = None
|
||||
pgvector: PGVectorSettings | None = None
|
||||
postgres: PostgresSettings | None = None
|
||||
|
||||
|
||||
"""
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
"""This file should be imported only and only if you want to run the UI locally."""
|
||||
"""This file should be imported if and only if you want to run the UI locally."""
|
||||
|
||||
import itertools
|
||||
import logging
|
||||
@ -104,12 +104,12 @@ class PrivateGptUi:
|
||||
sources_text = "\n\n\n"
|
||||
used_files = set()
|
||||
for index, source in enumerate(cur_sources, start=1):
|
||||
if (source.file + "-" + source.page) not in used_files:
|
||||
if f"{source.file}-{source.page}" not in used_files:
|
||||
sources_text = (
|
||||
sources_text
|
||||
+ f"{index}. {source.file} (page {source.page}) \n\n"
|
||||
)
|
||||
used_files.add(source.file + "-" + source.page)
|
||||
used_files.add(f"{source.file}-{source.page}")
|
||||
full_response += sources_text
|
||||
yield full_response
|
||||
|
||||
@ -409,11 +409,54 @@ class PrivateGptUi:
|
||||
inputs=system_prompt_input,
|
||||
)
|
||||
|
||||
def get_model_label() -> str | None:
|
||||
"""Get model label from llm mode setting YAML.
|
||||
|
||||
Raises:
|
||||
ValueError: If an invalid 'llm_mode' is encountered.
|
||||
|
||||
Returns:
|
||||
str: The corresponding model label.
|
||||
"""
|
||||
# Get model label from llm mode setting YAML
|
||||
# Labels: local, openai, openailike, sagemaker, mock, ollama
|
||||
config_settings = settings()
|
||||
if config_settings is None:
|
||||
raise ValueError("Settings are not configured.")
|
||||
|
||||
# Get llm_mode from settings
|
||||
llm_mode = config_settings.llm.mode
|
||||
|
||||
# Mapping of 'llm_mode' to corresponding model labels
|
||||
model_mapping = {
|
||||
"llamacpp": config_settings.llamacpp.llm_hf_model_file,
|
||||
"openai": config_settings.openai.model,
|
||||
"openailike": config_settings.openai.model,
|
||||
"sagemaker": config_settings.sagemaker.llm_endpoint_name,
|
||||
"mock": llm_mode,
|
||||
"ollama": config_settings.ollama.llm_model,
|
||||
}
|
||||
|
||||
if llm_mode not in model_mapping:
|
||||
print(f"Invalid 'llm mode': {llm_mode}")
|
||||
return None
|
||||
|
||||
return model_mapping[llm_mode]
|
||||
|
||||
with gr.Column(scale=7, elem_id="col"):
|
||||
# Determine the model label based on the value of PGPT_PROFILES
|
||||
model_label = get_model_label()
|
||||
if model_label is not None:
|
||||
label_text = (
|
||||
f"LLM: {settings().llm.mode} | Model: {model_label}"
|
||||
)
|
||||
else:
|
||||
label_text = f"LLM: {settings().llm.mode}"
|
||||
|
||||
_ = gr.ChatInterface(
|
||||
self._chat,
|
||||
chatbot=gr.Chatbot(
|
||||
label=f"LLM: {settings().llm.mode}",
|
||||
label=label_text,
|
||||
show_copy_button=True,
|
||||
elem_id="chatbot",
|
||||
render=False,
|
||||
|
||||
122
private_gpt/utils/eta.py
Normal file
122
private_gpt/utils/eta.py
Normal file
@ -0,0 +1,122 @@
|
||||
import datetime
|
||||
import logging
|
||||
import math
|
||||
import time
|
||||
from collections import deque
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def human_time(*args: Any, **kwargs: Any) -> str:
|
||||
def timedelta_total_seconds(timedelta: datetime.timedelta) -> float:
|
||||
return (
|
||||
timedelta.microseconds
|
||||
+ 0.0
|
||||
+ (timedelta.seconds + timedelta.days * 24 * 3600) * 10**6
|
||||
) / 10**6
|
||||
|
||||
secs = float(timedelta_total_seconds(datetime.timedelta(*args, **kwargs)))
|
||||
# We want (ms) precision below 2 seconds
|
||||
if secs < 2:
|
||||
return f"{secs * 1000}ms"
|
||||
units = [("y", 86400 * 365), ("d", 86400), ("h", 3600), ("m", 60), ("s", 1)]
|
||||
parts = []
|
||||
for unit, mul in units:
|
||||
if secs / mul >= 1 or mul == 1:
|
||||
if mul > 1:
|
||||
n = int(math.floor(secs / mul))
|
||||
secs -= n * mul
|
||||
else:
|
||||
# >2s we drop the (ms) component.
|
||||
n = int(secs)
|
||||
if n:
|
||||
parts.append(f"{n}{unit}")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def eta(iterator: list[Any]) -> Any:
|
||||
"""Report an ETA after 30s and every 60s thereafter."""
|
||||
total = len(iterator)
|
||||
_eta = ETA(total)
|
||||
_eta.needReport(30)
|
||||
for processed, data in enumerate(iterator, start=1):
|
||||
yield data
|
||||
_eta.update(processed)
|
||||
if _eta.needReport(60):
|
||||
logger.info(f"{processed}/{total} - ETA {_eta.human_time()}")
|
||||
|
||||
|
||||
class ETA:
|
||||
"""Predict how long something will take to complete."""
|
||||
|
||||
def __init__(self, total: int):
|
||||
self.total: int = total # Total expected records.
|
||||
self.rate: float = 0.0 # per second
|
||||
self._timing_data: deque[tuple[float, int]] = deque(maxlen=100)
|
||||
self.secondsLeft: float = 0.0
|
||||
self.nexttime: float = 0.0
|
||||
|
||||
def human_time(self) -> str:
|
||||
if self._calc():
|
||||
return f"{human_time(seconds=self.secondsLeft)} @ {int(self.rate * 60)}/min"
|
||||
return "(computing)"
|
||||
|
||||
def update(self, count: int) -> None:
|
||||
# count should be in the range 0 to self.total
|
||||
assert count > 0
|
||||
assert count <= self.total
|
||||
self._timing_data.append((time.time(), count)) # (X,Y) for pearson
|
||||
|
||||
def needReport(self, whenSecs: int) -> bool:
|
||||
now = time.time()
|
||||
if now > self.nexttime:
|
||||
self.nexttime = now + whenSecs
|
||||
return True
|
||||
return False
|
||||
|
||||
def _calc(self) -> bool:
|
||||
# A sample before a prediction. Need two points to compute slope!
|
||||
if len(self._timing_data) < 3:
|
||||
return False
|
||||
|
||||
# http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
|
||||
# Calculate means and standard deviations.
|
||||
samples = len(self._timing_data)
|
||||
# column wise sum of the timing tuples to compute their mean.
|
||||
mean_x, mean_y = (
|
||||
sum(i) / samples for i in zip(*self._timing_data, strict=False)
|
||||
)
|
||||
std_x = math.sqrt(
|
||||
sum(pow(i[0] - mean_x, 2) for i in self._timing_data) / (samples - 1)
|
||||
)
|
||||
std_y = math.sqrt(
|
||||
sum(pow(i[1] - mean_y, 2) for i in self._timing_data) / (samples - 1)
|
||||
)
|
||||
|
||||
# Calculate coefficient.
|
||||
sum_xy, sum_sq_v_x, sum_sq_v_y = 0.0, 0.0, 0
|
||||
for x, y in self._timing_data:
|
||||
x -= mean_x
|
||||
y -= mean_y
|
||||
sum_xy += x * y
|
||||
sum_sq_v_x += pow(x, 2)
|
||||
sum_sq_v_y += pow(y, 2)
|
||||
pearson_r = sum_xy / math.sqrt(sum_sq_v_x * sum_sq_v_y)
|
||||
|
||||
# Calculate regression line.
|
||||
# y = mx + b where m is the slope and b is the y-intercept.
|
||||
m = self.rate = pearson_r * (std_y / std_x)
|
||||
y = self.total
|
||||
b = mean_y - m * mean_x
|
||||
x = (y - b) / m
|
||||
|
||||
# Calculate fitted line (transformed/shifted regression line horizontally).
|
||||
fitted_b = self._timing_data[-1][1] - (m * self._timing_data[-1][0])
|
||||
fitted_x = (y - fitted_b) / m
|
||||
_, count = self._timing_data[-1] # adjust last data point progress count
|
||||
adjusted_x = ((fitted_x - x) * (count / self.total)) + x
|
||||
eta_epoch = adjusted_x
|
||||
|
||||
self.secondsLeft = max([eta_epoch - time.time(), 0])
|
||||
return True
|
||||
@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "private-gpt"
|
||||
version = "0.4.0"
|
||||
version = "0.5.0"
|
||||
description = "Private GPT"
|
||||
authors = ["Zylon <hi@zylon.ai>"]
|
||||
|
||||
@ -16,7 +16,7 @@ alembic = "^1.13.1"
|
||||
sqlalchemy = "^2.0.28"
|
||||
bcrypt = "4.0.1"
|
||||
python-jose = "^3.3.0"
|
||||
psycopg2-binary = "^2.9.9"
|
||||
#psycopg2-binary = "^2.9.9"
|
||||
passlib = "^1.7.4"
|
||||
docx2txt = "^0.8"
|
||||
ldap3 = "^2.9.1"
|
||||
@ -25,7 +25,7 @@ python-doctr = "^0.8.1"
|
||||
python-docx = "^1.1.0"
|
||||
watchdog = "^4.0.0"
|
||||
# https://stackoverflow.com/questions/76327419/valueerror-libcublas-so-0-9-not-found-in-the-system-path
|
||||
torch = ">=2.0.0, !=2.0.1, !=2.1.0"
|
||||
#torch = ">=2.0.0, !=2.0.1, !=2.1.0"
|
||||
torchvision = "^0.17.1"
|
||||
transformers = "^4.38.2"
|
||||
|
||||
@ -38,16 +38,28 @@ llama-index-llms-llama-cpp = {version = "^0.1.3", optional = true}
|
||||
llama-index-llms-openai = {version = "^0.1.6", optional = true}
|
||||
llama-index-llms-openai-like = {version ="^0.1.3", optional = true}
|
||||
llama-index-llms-ollama = {version ="^0.1.2", optional = true}
|
||||
llama-index-llms-azure-openai = {version ="^0.1.5", optional = true}
|
||||
llama-index-embeddings-ollama = {version ="^0.1.2", optional = true}
|
||||
llama-index-embeddings-huggingface = {version ="^0.1.4", optional = true}
|
||||
llama-index-embeddings-openai = {version ="^0.1.6", optional = true}
|
||||
llama-index-embeddings-azure-openai = {version ="^0.1.6", optional = true}
|
||||
llama-index-vector-stores-qdrant = {version ="^0.1.3", optional = true}
|
||||
llama-index-vector-stores-chroma = {version ="^0.1.4", optional = true}
|
||||
llama-index-vector-stores-postgres = {version ="^0.1.2", optional = true}
|
||||
llama-index-storage-docstore-postgres = {version ="^0.1.2", optional = true}
|
||||
llama-index-storage-index-store-postgres = {version ="^0.1.2", optional = true}
|
||||
|
||||
# Postgres
|
||||
psycopg2-binary = {version ="^2.9.9", optional = false}
|
||||
asyncpg = {version="^0.29.0", optional = true}
|
||||
|
||||
# Optional Sagemaker dependency
|
||||
boto3 = {version ="^1.34.51", optional = true}
|
||||
|
||||
# Optional Reranker dependencies
|
||||
torch = {version ="^2.1.2", optional = false}
|
||||
sentence-transformers = {version ="^2.6.1", optional = true}
|
||||
|
||||
# Optional UI
|
||||
gradio = {version ="^4.19.2", optional = true}
|
||||
aiofiles = "^23.2.1"
|
||||
@ -66,14 +78,17 @@ llms-openai = ["llama-index-llms-openai"]
|
||||
llms-openai-like = ["llama-index-llms-openai-like"]
|
||||
llms-ollama = ["llama-index-llms-ollama"]
|
||||
llms-sagemaker = ["boto3"]
|
||||
llms-azopenai = ["llama-index-llms-azure-openai"]
|
||||
embeddings-ollama = ["llama-index-embeddings-ollama"]
|
||||
embeddings-huggingface = ["llama-index-embeddings-huggingface"]
|
||||
embeddings-openai = ["llama-index-embeddings-openai"]
|
||||
embeddings-sagemaker = ["boto3"]
|
||||
embeddings-azopenai = ["llama-index-embeddings-azure-openai"]
|
||||
vector-stores-qdrant = ["llama-index-vector-stores-qdrant"]
|
||||
vector-stores-chroma = ["llama-index-vector-stores-chroma"]
|
||||
vector-stores-postgres = ["llama-index-vector-stores-postgres"]
|
||||
|
||||
storage-nodestore-postgres = ["llama-index-storage-docstore-postgres","llama-index-storage-index-store-postgres","psycopg2-binary","asyncpg"]
|
||||
rerank-sentence-transformers = ["torch", "sentence-transformers"]
|
||||
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
black = "^22"
|
||||
|
||||
165
scripts/utils.py
165
scripts/utils.py
@ -1,10 +1,22 @@
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from private_gpt.paths import local_data_path
|
||||
from private_gpt.settings.settings import settings
|
||||
|
||||
|
||||
def wipe():
|
||||
path = "local_data"
|
||||
def wipe_file(file: str) -> None:
|
||||
if os.path.isfile(file):
|
||||
os.remove(file)
|
||||
print(f" - Deleted {file}")
|
||||
|
||||
|
||||
def wipe_tree(path: str) -> None:
|
||||
if not os.path.exists(path):
|
||||
print(f"Warning: Path not found {path}")
|
||||
return
|
||||
print(f"Wiping {path}...")
|
||||
all_files = os.listdir(path)
|
||||
|
||||
@ -24,14 +36,149 @@ def wipe():
|
||||
continue
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
commands = {
|
||||
"wipe": wipe,
|
||||
class Postgres:
|
||||
tables: ClassVar[dict[str, list[str]]] = {
|
||||
"nodestore": ["data_docstore", "data_indexstore"],
|
||||
"vectorstore": ["data_embeddings"],
|
||||
}
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"mode", help="select a mode to run", choices=list(commands.keys())
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
import psycopg2
|
||||
except ModuleNotFoundError:
|
||||
raise ModuleNotFoundError("Postgres dependencies not found") from None
|
||||
|
||||
connection = settings().postgres.model_dump(exclude_none=True)
|
||||
self.schema = connection.pop("schema_name")
|
||||
self.conn = psycopg2.connect(**connection)
|
||||
|
||||
def wipe(self, storetype: str) -> None:
|
||||
cur = self.conn.cursor()
|
||||
try:
|
||||
for table in self.tables[storetype]:
|
||||
sql = f"DROP TABLE IF EXISTS {self.schema}.{table}"
|
||||
cur.execute(sql)
|
||||
print(f"Table {self.schema}.{table} dropped.")
|
||||
self.conn.commit()
|
||||
finally:
|
||||
cur.close()
|
||||
|
||||
def stats(self, store_type: str) -> None:
|
||||
template = "SELECT '{table}', COUNT(*), pg_size_pretty(pg_total_relation_size('{table}')) FROM {table}"
|
||||
sql = " UNION ALL ".join(
|
||||
template.format(table=tbl) for tbl in self.tables[store_type]
|
||||
)
|
||||
|
||||
cur = self.conn.cursor()
|
||||
try:
|
||||
print(f"Storage for Postgres {store_type}.")
|
||||
print("{:<15} | {:>15} | {:>9}".format("Table", "Rows", "Size"))
|
||||
print("-" * 45) # Print a line separator
|
||||
|
||||
cur.execute(sql)
|
||||
for row in cur.fetchall():
|
||||
formatted_row_count = f"{row[1]:,}"
|
||||
print(f"{row[0]:<15} | {formatted_row_count:>15} | {row[2]:>9}")
|
||||
|
||||
print()
|
||||
finally:
|
||||
cur.close()
|
||||
|
||||
def __del__(self):
|
||||
if hasattr(self, "conn") and self.conn:
|
||||
self.conn.close()
|
||||
|
||||
|
||||
class Simple:
|
||||
def wipe(self, store_type: str) -> None:
|
||||
assert store_type == "nodestore"
|
||||
from llama_index.core.storage.docstore.types import (
|
||||
DEFAULT_PERSIST_FNAME as DOCSTORE,
|
||||
)
|
||||
from llama_index.core.storage.index_store.types import (
|
||||
DEFAULT_PERSIST_FNAME as INDEXSTORE,
|
||||
)
|
||||
|
||||
for store in (DOCSTORE, INDEXSTORE):
|
||||
wipe_file(str((local_data_path / store).absolute()))
|
||||
|
||||
|
||||
class Chroma:
|
||||
def wipe(self, store_type: str) -> None:
|
||||
assert store_type == "vectorstore"
|
||||
wipe_tree(str((local_data_path / "chroma_db").absolute()))
|
||||
|
||||
|
||||
class Qdrant:
|
||||
COLLECTION = (
|
||||
"make_this_parameterizable_per_api_call" # ?! see vector_store_component.py
|
||||
)
|
||||
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
from qdrant_client import QdrantClient # type: ignore
|
||||
except ImportError:
|
||||
raise ImportError("Qdrant dependencies not found") from None
|
||||
self.client = QdrantClient(**settings().qdrant.model_dump(exclude_none=True))
|
||||
|
||||
def wipe(self, store_type: str) -> None:
|
||||
assert store_type == "vectorstore"
|
||||
try:
|
||||
self.client.delete_collection(self.COLLECTION)
|
||||
print("Collection dropped successfully.")
|
||||
except Exception as e:
|
||||
print("Error dropping collection:", e)
|
||||
|
||||
def stats(self, store_type: str) -> None:
|
||||
print(f"Storage for Qdrant {store_type}.")
|
||||
try:
|
||||
collection_data = self.client.get_collection(self.COLLECTION)
|
||||
if collection_data:
|
||||
# Collection Info
|
||||
# https://qdrant.tech/documentation/concepts/collections/
|
||||
print(f"\tPoints: {collection_data.points_count:,}")
|
||||
print(f"\tVectors: {collection_data.vectors_count:,}")
|
||||
print(f"\tIndex Vectors: {collection_data.indexed_vectors_count:,}")
|
||||
return
|
||||
except ValueError:
|
||||
pass
|
||||
print("\t- Qdrant collection not found or empty")
|
||||
|
||||
|
||||
class Command:
|
||||
DB_HANDLERS: ClassVar[dict[str, Any]] = {
|
||||
"simple": Simple, # node store
|
||||
"chroma": Chroma, # vector store
|
||||
"postgres": Postgres, # node, index and vector store
|
||||
"qdrant": Qdrant, # vector store
|
||||
}
|
||||
|
||||
def for_each_store(self, cmd: str):
|
||||
for store_type in ("nodestore", "vectorstore"):
|
||||
database = getattr(settings(), store_type).database
|
||||
handler_class = self.DB_HANDLERS.get(database)
|
||||
if handler_class is None:
|
||||
print(f"No handler found for database '{database}'")
|
||||
continue
|
||||
handler_instance = handler_class() # Instantiate the class
|
||||
# If the DB can handle this cmd dispatch it.
|
||||
if hasattr(handler_instance, cmd) and callable(
|
||||
func := getattr(handler_instance, cmd)
|
||||
):
|
||||
func(store_type)
|
||||
else:
|
||||
print(
|
||||
f"Unable to execute command '{cmd}' on '{store_type}' in database '{database}'"
|
||||
)
|
||||
|
||||
def execute(self, cmd: str) -> None:
|
||||
if cmd in ("wipe", "stats"):
|
||||
self.for_each_store(cmd)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("mode", help="select a mode to run", choices=["wipe", "stats"])
|
||||
args = parser.parse_args()
|
||||
commands[args.mode.lower()]()
|
||||
|
||||
Command().execute(args.mode.lower())
|
||||
|
||||
17
settings-azopenai.yaml
Normal file
17
settings-azopenai.yaml
Normal file
@ -0,0 +1,17 @@
|
||||
server:
|
||||
env_name: ${APP_ENV:azopenai}
|
||||
|
||||
llm:
|
||||
mode: azopenai
|
||||
|
||||
embedding:
|
||||
mode: azopenai
|
||||
|
||||
azopenai:
|
||||
api_key: ${AZ_OPENAI_API_KEY:}
|
||||
azure_endpoint: ${AZ_OPENAI_ENDPOINT:}
|
||||
embedding_deployment_name: ${AZ_OPENAI_EMBEDDING_DEPLOYMENT_NAME:}
|
||||
llm_deployment_name: ${AZ_OPENAI_LLM_DEPLOYMENT_NAME:}
|
||||
api_version: "2023-05-15"
|
||||
embedding_model: text-embedding-ada-002
|
||||
llm_model: gpt-35-turbo
|
||||
@ -23,7 +23,6 @@ ollama:
|
||||
llm_model: ${PGPT_OLLAMA_LLM_MODEL:mistral}
|
||||
embedding_model: ${PGPT_OLLAMA_EMBEDDING_MODEL:nomic-embed-text}
|
||||
api_base: ${PGPT_OLLAMA_API_BASE:http://ollama:11434}
|
||||
embedding_api_base: ${PGPT_OLLAMA_EMBEDDING_API_BASE:http://ollama:11434}
|
||||
tfs_z: ${PGPT_OLLAMA_TFS_Z:1.0}
|
||||
top_k: ${PGPT_OLLAMA_TOP_K:40}
|
||||
top_p: ${PGPT_OLLAMA_TOP_P:0.9}
|
||||
|
||||
@ -1,3 +1,4 @@
|
||||
# poetry install --extras "ui llms-llama-cpp vector-stores-qdrant embeddings-huggingface"
|
||||
server:
|
||||
env_name: ${APP_ENV:local}
|
||||
|
||||
|
||||
34
settings-ollama-pg.yaml
Normal file
34
settings-ollama-pg.yaml
Normal file
@ -0,0 +1,34 @@
|
||||
# Using ollama and postgres for the vector, doc and index store. Ollama is also used for embeddings.
|
||||
# To use install these extras:
|
||||
# poetry install --extras "llms-ollama ui vector-stores-postgres embeddings-ollama storage-nodestore-postgres"
|
||||
server:
|
||||
env_name: ${APP_ENV:ollama}
|
||||
|
||||
llm:
|
||||
mode: ollama
|
||||
max_new_tokens: 512
|
||||
context_window: 3900
|
||||
|
||||
embedding:
|
||||
mode: ollama
|
||||
embed_dim: 768
|
||||
|
||||
ollama:
|
||||
llm_model: mistral
|
||||
embedding_model: nomic-embed-text
|
||||
api_base: http://localhost:11434
|
||||
|
||||
nodestore:
|
||||
database: postgres
|
||||
|
||||
vectorstore:
|
||||
database: postgres
|
||||
|
||||
postgres:
|
||||
host: localhost
|
||||
port: 5432
|
||||
database: postgres
|
||||
user: postgres
|
||||
password: admin
|
||||
schema_name: private_gpt
|
||||
|
||||
@ -14,11 +14,14 @@ ollama:
|
||||
llm_model: mistral
|
||||
embedding_model: nomic-embed-text
|
||||
api_base: http://localhost:11434
|
||||
tfs_z: 1.0 # Tail free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting.
|
||||
top_k: 40 # Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative. (Default: 40)
|
||||
top_p: 0.9 # Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)
|
||||
repeat_last_n: 64 # Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)
|
||||
repeat_penalty: 1.2 # Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.1)
|
||||
embedding_api_base: http://localhost:11434 # change if your embedding model runs on another ollama
|
||||
keep_alive: 5m
|
||||
tfs_z: 1.0 # Tail free sampling is used to reduce the impact of less probable tokens from the output. A higher value (e.g., 2.0) will reduce the impact more, while a value of 1.0 disables this setting.
|
||||
top_k: 40 # Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative. (Default: 40)
|
||||
top_p: 0.9 # Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)
|
||||
repeat_last_n: 64 # Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)
|
||||
repeat_penalty: 1.2 # Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.1)
|
||||
request_timeout: 120.0 # Time elapsed until ollama times out the request. Default is 120s. Format is float.
|
||||
|
||||
vectorstore:
|
||||
database: qdrant
|
||||
|
||||
@ -40,9 +40,19 @@ llm:
|
||||
# Should be matching the selected model
|
||||
max_new_tokens: 512
|
||||
context_window: 3900
|
||||
temperature: 0.1 # The temperature of the model. Increasing the temperature will make the model answer more creatively. A value of 0.1 would be more factual. (Default: 0.1)
|
||||
tokenizer: mistralai/Mistral-7B-Instruct-v0.2
|
||||
|
||||
temperature: 0.1 # The temperature of the model. Increasing the temperature will make the model answer more creatively. A value of 0.1 would be more factual. (Default: 0.1)
|
||||
|
||||
rag:
|
||||
similarity_top_k: 10
|
||||
#This value controls how many "top" documents the RAG returns to use in the context.
|
||||
#similarity_value: 0.45
|
||||
#This value is disabled by default. If you enable this settings, the RAG will only use articles that meet a certain percentage score.
|
||||
rerank:
|
||||
enabled: true
|
||||
model: cross-encoder/ms-marco-MiniLM-L-2-v2
|
||||
top_n: 3
|
||||
|
||||
llamacpp:
|
||||
prompt_style: "chatml"
|
||||
llm_hf_repo_id: TheBloke/OpenHermes-2.5-Mistral-7B-GGUF
|
||||
@ -57,6 +67,7 @@ embedding:
|
||||
# Should be matching the value above in most cases
|
||||
mode: huggingface
|
||||
ingest_mode: simple
|
||||
embed_dim: 384 # 384 is for BAAI/bge-small-en-v1.5
|
||||
|
||||
huggingface:
|
||||
embedding_hf_model_name: mixedbread-ai/mxbai-embed-large-v1
|
||||
@ -64,10 +75,13 @@ huggingface:
|
||||
vectorstore:
|
||||
database: qdrant
|
||||
|
||||
nodestore:
|
||||
database: simple
|
||||
|
||||
qdrant:
|
||||
path: local_data/private_gpt/qdrant
|
||||
|
||||
pgvector:
|
||||
postgres:
|
||||
host: localhost
|
||||
port: 5432
|
||||
database: postgres
|
||||
@ -75,7 +89,6 @@ pgvector:
|
||||
password: postgres
|
||||
embed_dim: 768 # 384 is for BAAI/bge-small-en-v1.5
|
||||
schema_name: private_gpt
|
||||
table_name: embeddings
|
||||
|
||||
sagemaker:
|
||||
llm_endpoint_name: huggingface-pytorch-tgi-inference-2023-09-25-19-53-32-140
|
||||
@ -89,3 +102,15 @@ ollama:
|
||||
llm_model: llama2
|
||||
embedding_model: nomic-embed-text
|
||||
api_base: http://localhost:11434
|
||||
embedding_api_base: http://localhost:11434 # change if your embedding model runs on another ollama
|
||||
keep_alive: 5m
|
||||
request_timeout: 120.0
|
||||
|
||||
azopenai:
|
||||
api_key: ${AZ_OPENAI_API_KEY:}
|
||||
azure_endpoint: ${AZ_OPENAI_ENDPOINT:}
|
||||
embedding_deployment_name: ${AZ_OPENAI_EMBEDDING_DEPLOYMENT_NAME:}
|
||||
llm_deployment_name: ${AZ_OPENAI_LLM_DEPLOYMENT_NAME:}
|
||||
api_version: "2023-05-15"
|
||||
embedding_model: text-embedding-ada-002
|
||||
llm_model: gpt-35-turbo
|
||||
|
||||
@ -1 +1 @@
|
||||
0.4.0
|
||||
0.5.0
|
||||
|
||||
Loading…
Reference in New Issue
Block a user