mirror of
https://github.com/imartinez/privateGPT.git
synced 2025-10-10 16:53:33 +00:00
Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
e8ac51bba4 | ||
|
145f3ec9f4 | ||
|
|
a072a40a7c | ||
|
a3ed14c58f | ||
|
f235c50be9 | ||
|
9302620eac | ||
|
9cf972563e |
@@ -1,5 +1,13 @@
|
||||
# Changelog
|
||||
|
||||
## [0.2.0](https://github.com/imartinez/privateGPT/compare/v0.1.0...v0.2.0) (2023-12-10)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **llm:** drop default_system_prompt ([#1385](https://github.com/imartinez/privateGPT/issues/1385)) ([a3ed14c](https://github.com/imartinez/privateGPT/commit/a3ed14c58f77351dbd5f8f2d7868d1642a44f017))
|
||||
* **ui:** Allows User to Set System Prompt via "Additional Options" in Chat Interface ([#1353](https://github.com/imartinez/privateGPT/issues/1353)) ([145f3ec](https://github.com/imartinez/privateGPT/commit/145f3ec9f41c4def5abf4065a06fb0786e2d992a))
|
||||
|
||||
## [0.1.0](https://github.com/imartinez/privateGPT/compare/v0.0.2...v0.1.0) (2023-11-30)
|
||||
|
||||
|
||||
|
||||
5
Makefile
5
Makefile
@@ -52,4 +52,7 @@ ingest:
|
||||
@poetry run python scripts/ingest_folder.py $(call args)
|
||||
|
||||
wipe:
|
||||
poetry run python scripts/utils.py wipe
|
||||
poetry run python scripts/utils.py wipe
|
||||
|
||||
setup:
|
||||
poetry run python scripts/setup
|
||||
|
||||
@@ -1,474 +0,0 @@
|
||||
## Introduction
|
||||
|
||||
PrivateGPT provides an **API** containing all the building blocks required to build
|
||||
**private, context-aware AI applications**. The API follows and extends OpenAI API standard, and supports
|
||||
both normal and streaming responses.
|
||||
|
||||
The API is divided in two logical blocks:
|
||||
|
||||
- High-level API, abstracting all the complexity of a RAG (Retrieval Augmented Generation) pipeline implementation:
|
||||
- Ingestion of documents: internally managing document parsing, splitting, metadata extraction,
|
||||
embedding generation and storage.
|
||||
- Chat & Completions using context from ingested documents: abstracting the retrieval of context, the prompt
|
||||
engineering and the response generation.
|
||||
- Low-level API, allowing advanced users to implement their own complex pipelines:
|
||||
- Embeddings generation: based on a piece of text.
|
||||
- Contextual chunks retrieval: given a query, returns the most relevant chunks of text from the ingested
|
||||
documents.
|
||||
|
||||
> A working **Gradio UI client** is provided to test the API, together with a set of
|
||||
> useful tools such as bulk model download script, ingestion script, documents folder
|
||||
> watch, etc.
|
||||
|
||||
## Quick Local Installation steps
|
||||
|
||||
The steps in `Installation and Settings` section are better explained and cover more
|
||||
setup scenarios. But if you are looking for a quick setup guide, here it is:
|
||||
|
||||
```
|
||||
# Clone the repo
|
||||
git clone https://github.com/imartinez/privateGPT
|
||||
cd privateGPT
|
||||
|
||||
# Install Python 3.11
|
||||
pyenv install 3.11
|
||||
pyenv local 3.11
|
||||
|
||||
# Install dependencies
|
||||
poetry install --with ui,local
|
||||
|
||||
# Download Embedding and LLM models
|
||||
poetry run python scripts/setup
|
||||
|
||||
# (Optional) For Mac with Metal GPU, enable it. Check Installation and Settings section
|
||||
to know how to enable GPU on other platforms
|
||||
CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python
|
||||
|
||||
# Run the local server
|
||||
PGPT_PROFILES=local make run
|
||||
|
||||
# Note: on Mac with Metal you should see a ggml_metal_add_buffer log, stating GPU is
|
||||
being used
|
||||
|
||||
# Navigate to the UI and try it out!
|
||||
http://localhost:8001/
|
||||
```
|
||||
|
||||
## Installation and Settings
|
||||
|
||||
### Base requirements to run PrivateGPT
|
||||
|
||||
* Git clone PrivateGPT repository, and navigate to it:
|
||||
|
||||
```
|
||||
git clone https://github.com/imartinez/privateGPT
|
||||
cd privateGPT
|
||||
```
|
||||
|
||||
* Install Python 3.11. Ideally through a python version manager like `pyenv`.
|
||||
Python 3.12
|
||||
should work too. Earlier python versions are not supported.
|
||||
* osx/linux: [pyenv](https://github.com/pyenv/pyenv)
|
||||
* windows: [pyenv-win](https://github.com/pyenv-win/pyenv-win)
|
||||
|
||||
```
|
||||
pyenv install 3.11
|
||||
pyenv local 3.11
|
||||
```
|
||||
|
||||
* Install [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer) for dependency management:
|
||||
|
||||
* Have a valid C++ compiler like gcc. See [Troubleshooting: C++ Compiler](#troubleshooting-c-compiler) for more details.
|
||||
|
||||
* Install `make` for scripts:
|
||||
* osx: (Using homebrew): `brew install make`
|
||||
* windows: (Using chocolatey) `choco install make`
|
||||
|
||||
### Install dependencies
|
||||
|
||||
Install the dependencies:
|
||||
|
||||
```bash
|
||||
poetry install --with ui
|
||||
```
|
||||
|
||||
Verify everything is working by running `make run` (or `poetry run python -m private_gpt`) and navigate to
|
||||
http://localhost:8001. You should see a [Gradio UI](https://gradio.app/) **configured with a mock LLM** that will
|
||||
echo back the input. Later we'll see how to configure a real LLM.
|
||||
|
||||
### Settings
|
||||
|
||||
> Note: the default settings of PrivateGPT work out-of-the-box for a 100% local setup. Skip this section if you just
|
||||
> want to test PrivateGPT locally, and come back later to learn about more configuration options.
|
||||
|
||||
PrivateGPT is configured through *profiles* that are defined using yaml files, and selected through env variables.
|
||||
The full list of properties configurable can be found in `settings.yaml`
|
||||
|
||||
#### env var `PGPT_SETTINGS_FOLDER`
|
||||
|
||||
The location of the settings folder. Defaults to the root of the project.
|
||||
Should contain the default `settings.yaml` and any other `settings-{profile}.yaml`.
|
||||
|
||||
#### env var `PGPT_PROFILES`
|
||||
|
||||
By default, the profile definition in `settings.yaml` is loaded.
|
||||
Using this env var you can load additional profiles; format is a comma separated list of profile names.
|
||||
This will merge `settings-{profile}.yaml` on top of the base settings file.
|
||||
|
||||
For example:
|
||||
`PGPT_PROFILES=local,cuda` will load `settings-local.yaml`
|
||||
and `settings-cuda.yaml`, their contents will be merged with
|
||||
later profiles properties overriding values of earlier ones like `settings.yaml`.
|
||||
|
||||
During testing, the `test` profile will be active along with the default, therefore `settings-test.yaml`
|
||||
file is required.
|
||||
|
||||
#### Environment variables expansion
|
||||
|
||||
Configuration files can contain environment variables,
|
||||
they will be expanded at runtime.
|
||||
|
||||
Expansion must follow the pattern `${VARIABLE_NAME:default_value}`.
|
||||
|
||||
For example, the following configuration will use the value of the `PORT`
|
||||
environment variable or `8001` if it's not set.
|
||||
Missing variables with no default will produce an error.
|
||||
|
||||
```yaml
|
||||
server:
|
||||
port: ${PORT:8001}
|
||||
```
|
||||
|
||||
### Local LLM requirements
|
||||
|
||||
Install extra dependencies for local execution:
|
||||
|
||||
```bash
|
||||
poetry install --with local
|
||||
```
|
||||
|
||||
For PrivateGPT to run fully locally GPU acceleration is required
|
||||
(CPU execution is possible, but very slow), however,
|
||||
typical Macbook laptops or window desktops with mid-range GPUs lack VRAM to run
|
||||
even the smallest LLMs. For that reason
|
||||
**local execution is only supported for models compatible with [llama.cpp](https://github.com/ggerganov/llama.cpp)**
|
||||
|
||||
These two models are known to work well:
|
||||
|
||||
* https://huggingface.co/TheBloke/Llama-2-7B-chat-GGUF
|
||||
* https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF (recommended)
|
||||
|
||||
To ease the installation process, use the `setup` script that will download both
|
||||
the embedding and the LLM model and place them in the correct location (under `models` folder):
|
||||
|
||||
```bash
|
||||
poetry run python scripts/setup
|
||||
```
|
||||
|
||||
If you are ok with CPU execution, you can skip the rest of this section.
|
||||
|
||||
As stated before, llama.cpp is required and in
|
||||
particular [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
|
||||
is used.
|
||||
|
||||
> It's highly encouraged that you fully read llama-cpp and llama-cpp-python documentation relevant to your platform.
|
||||
> Running into installation issues is very likely, and you'll need to troubleshoot them yourself.
|
||||
|
||||
#### Customizing low level parameters
|
||||
|
||||
Currently not all the parameters of llama-cpp and llama-cpp-python are available at PrivateGPT's `settings.yaml` file. In case you need to customize parameters such as the number of layers loaded into the GPU, you might change these at the `llm_component.py` file under the `private_gpt/components/llm/llm_component.py`. If you are getting an out of memory error, you might also try a smaller model or stick to the proposed recommended models, instead of custom tuning the parameters.
|
||||
|
||||
#### OSX GPU support
|
||||
|
||||
You will need to build [llama.cpp](https://github.com/ggerganov/llama.cpp) with
|
||||
metal support. To do that run:
|
||||
|
||||
```bash
|
||||
CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python
|
||||
```
|
||||
|
||||
#### Windows NVIDIA GPU support
|
||||
|
||||
Windows GPU support is done through CUDA.
|
||||
Follow the instructions on the original [llama.cpp](https://github.com/ggerganov/llama.cpp) repo to install the required
|
||||
dependencies.
|
||||
|
||||
Some tips to get it working with an NVIDIA card and CUDA (Tested on Windows 10 with CUDA 11.5 RTX 3070):
|
||||
|
||||
* Install latest VS2022 (and build tools) https://visualstudio.microsoft.com/vs/community/
|
||||
* Install CUDA toolkit https://developer.nvidia.com/cuda-downloads
|
||||
* Verify your installation is correct by running `nvcc --version` and `nvidia-smi`, ensure your CUDA version is up to
|
||||
date and your GPU is detected.
|
||||
* [Optional] Install CMake to troubleshoot building issues by compiling llama.cpp directly https://cmake.org/download/
|
||||
|
||||
If you have all required dependencies properly configured running the
|
||||
following powershell command should succeed.
|
||||
|
||||
```powershell
|
||||
$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
|
||||
```
|
||||
|
||||
If your installation was correct, you should see a message similar to the following next
|
||||
time you start the server `BLAS = 1`.
|
||||
|
||||
```
|
||||
llama_new_context_with_model: total VRAM used: 4857.93 MB (model: 4095.05 MB, context: 762.87 MB)
|
||||
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
|
||||
```
|
||||
|
||||
Note that llama.cpp offloads matrix calculations to the GPU but the performance is
|
||||
still hit heavily due to latency between CPU and GPU communication. You might need to tweak
|
||||
batch sizes and other parameters to get the best performance for your particular system.
|
||||
|
||||
#### Linux NVIDIA GPU support and Windows-WSL
|
||||
|
||||
Linux GPU support is done through CUDA.
|
||||
Follow the instructions on the original [llama.cpp](https://github.com/ggerganov/llama.cpp) repo to install the required
|
||||
external
|
||||
dependencies.
|
||||
|
||||
Some tips:
|
||||
|
||||
* Make sure you have an up-to-date C++ compiler
|
||||
* Install CUDA toolkit https://developer.nvidia.com/cuda-downloads
|
||||
* Verify your installation is correct by running `nvcc --version` and `nvidia-smi`, ensure your CUDA version is up to
|
||||
date and your GPU is detected.
|
||||
|
||||
After that running the following command in the repository will install llama.cpp with GPU support:
|
||||
|
||||
`
|
||||
CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
|
||||
`
|
||||
|
||||
If your installation was correct, you should see a message similar to the following next
|
||||
time you start the server `BLAS = 1`.
|
||||
|
||||
```
|
||||
llama_new_context_with_model: total VRAM used: 4857.93 MB (model: 4095.05 MB, context: 762.87 MB)
|
||||
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
|
||||
```
|
||||
|
||||
#### Vectorstores
|
||||
PrivateGPT supports [Chroma](https://www.trychroma.com/), [Qdrant](https://qdrant.tech/) as vectorstore providers. Chroma being the default.
|
||||
|
||||
To enable Qdrant, set the `vectorstore.database` property in the `settings.yaml` file to `qdrant` and install the `qdrant` extra.
|
||||
|
||||
```bash
|
||||
poetry install --extras qdrant
|
||||
```
|
||||
|
||||
By default Qdrant tries to connect to an instance at `http://localhost:3000`.
|
||||
|
||||
Qdrant settings can be configured by setting values to the `qdrant` property in the `settings.yaml` file.
|
||||
|
||||
The available configuration options are:
|
||||
| Field | Description |
|
||||
|--------------|-------------|
|
||||
| location | If `:memory:` - use in-memory Qdrant instance.<br>If `str` - use it as a `url` parameter.|
|
||||
| url | Either host or str of 'Optional[scheme], host, Optional[port], Optional[prefix]'.<br> Eg. `http://localhost:6333` |
|
||||
| port | Port of the REST API interface. Default: `6333` |
|
||||
| grpc_port | Port of the gRPC interface. Default: `6334` |
|
||||
| prefer_grpc | If `true` - use gRPC interface whenever possible in custom methods. |
|
||||
| https | If `true` - use HTTPS(SSL) protocol.|
|
||||
| api_key | API key for authentication in Qdrant Cloud.|
|
||||
| prefix | If set, add `prefix` to the REST URL path.<br>Example: `service/v1` will result in `http://localhost:6333/service/v1/{qdrant-endpoint}` for REST API.|
|
||||
| timeout | Timeout for REST and gRPC API requests.<br>Default: 5.0 seconds for REST and unlimited for gRPC |
|
||||
| host | Host name of Qdrant service. If url and host are not set, defaults to 'localhost'.|
|
||||
| path | Persistence path for QdrantLocal. Eg. `local_data/private_gpt/qdrant`|
|
||||
| force_disable_check_same_thread | Force disable check_same_thread for QdrantLocal sqlite connection.|
|
||||
|
||||
#### Known issues and Troubleshooting
|
||||
|
||||
Execution of LLMs locally still has a lot of sharp edges, specially when running on non Linux platforms.
|
||||
You might encounter several issues:
|
||||
|
||||
* Performance: RAM or VRAM usage is very high, your computer might experience slowdowns or even crashes.
|
||||
* GPU Virtualization on Windows and OSX: Simply not possible with docker desktop, you have to run the server directly on
|
||||
the host.
|
||||
* Building errors: Some of PrivateGPT dependencies need to build native code, and they might fail on some platforms.
|
||||
Most likely you are missing some dev tools in your machine (updated C++ compiler, CUDA is not on PATH, etc.).
|
||||
If you encounter any of these issues, please open an issue and we'll try to help.
|
||||
|
||||
#### Troubleshooting: C++ Compiler
|
||||
|
||||
If you encounter an error while building a wheel during the `pip install` process, you may need to install a C++
|
||||
compiler on your computer.
|
||||
|
||||
**For Windows 10/11**
|
||||
|
||||
To install a C++ compiler on Windows 10/11, follow these steps:
|
||||
|
||||
1. Install Visual Studio 2022.
|
||||
2. Make sure the following components are selected:
|
||||
* Universal Windows Platform development
|
||||
* C++ CMake tools for Windows
|
||||
3. Download the MinGW installer from the [MinGW website](https://sourceforge.net/projects/mingw/).
|
||||
4. Run the installer and select the `gcc` component.
|
||||
|
||||
** For OSX **
|
||||
|
||||
1. Check if you have a C++ compiler installed, Xcode might have done it for you. for example running `gcc`.
|
||||
2. If not, you can install clang or gcc with homebrew `brew install gcc`
|
||||
|
||||
#### Troubleshooting: Mac Running Intel
|
||||
|
||||
When running a Mac with Intel hardware (not M1), you may run into _clang: error: the clang compiler does not support '
|
||||
-march=native'_ during pip install.
|
||||
|
||||
If so set your archflags during pip install. eg: _ARCHFLAGS="-arch x86_64" pip3 install -r requirements.txt_
|
||||
|
||||
## Running the Server
|
||||
|
||||
After following the installation steps you should be ready to go. Here are some common run setups:
|
||||
|
||||
### Running 100% locally
|
||||
|
||||
Make sure you have followed the *Local LLM requirements* section before moving on.
|
||||
|
||||
This command will start PrivateGPT using the `settings.yaml` (default profile) together with the `settings-local.yaml`
|
||||
configuration files. By default, it will enable both the API and the Gradio UI. Run:
|
||||
|
||||
```
|
||||
PGPT_PROFILES=local make run
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
PGPT_PROFILES=local poetry run python -m private_gpt
|
||||
```
|
||||
|
||||
When the server is started it will print a log *Application startup complete*.
|
||||
Navigate to http://localhost:8001 to use the Gradio UI or to http://localhost:8001/docs (API section) to try the API
|
||||
using Swagger UI.
|
||||
|
||||
### Local server using OpenAI as LLM
|
||||
|
||||
If you cannot run a local model (because you don't have a GPU, for example) or for testing purposes, you may
|
||||
decide to run PrivateGPT using OpenAI as the LLM.
|
||||
|
||||
In order to do so, create a profile `settings-openai.yaml` with the following contents:
|
||||
|
||||
```yaml
|
||||
llm:
|
||||
mode: openai
|
||||
|
||||
openai:
|
||||
api_key: <your_openai_api_key> # You could skip this configuration and use the OPENAI_API_KEY env var instead
|
||||
```
|
||||
|
||||
And run PrivateGPT loading that profile you just created:
|
||||
|
||||
```PGPT_PROFILES=openai make run```
|
||||
|
||||
or
|
||||
|
||||
```PGPT_PROFILES=openai poetry run python -m private_gpt```
|
||||
|
||||
> Note this will still use the local Embeddings model, as it is ok to use it on a CPU.
|
||||
> We'll support using OpenAI embeddings in a future release.
|
||||
|
||||
When the server is started it will print a log *Application startup complete*.
|
||||
Navigate to http://localhost:8001 to use the Gradio UI or to http://localhost:8001/docs (API section) to try the API.
|
||||
You'll notice the speed and quality of response is higher, given you are using OpenAI's servers for the heavy
|
||||
computations.
|
||||
|
||||
### Use AWS's Sagemaker
|
||||
|
||||
🚧 Under construction 🚧
|
||||
|
||||
## Gradio UI user manual
|
||||
|
||||
Gradio UI is a ready to use way of testing most of PrivateGPT API functionalities.
|
||||
|
||||
|
||||
|
||||
### Execution Modes
|
||||
|
||||
It has 3 modes of execution (you can select in the top-left):
|
||||
|
||||
* Query Docs: uses the context from the
|
||||
ingested documents to answer the questions posted in the chat. It also takes
|
||||
into account previous chat messages as context.
|
||||
* Makes use of `/chat/completions` API with `use_context=true` and no
|
||||
`context_filter`.
|
||||
* Search in Docs: fast search that returns the 4 most related text
|
||||
chunks, together with their source document and page.
|
||||
* Makes use of `/chunks` API with no `context_filter`, `limit=4` and
|
||||
`prev_next_chunks=0`.
|
||||
* LLM Chat: simple, non-contextual chat with the LLM. The ingested documents won't
|
||||
be taken into account, only the previous messages.
|
||||
* Makes use of `/chat/completions` API with `use_context=false`.
|
||||
|
||||
### Document Ingestion
|
||||
|
||||
Ingest documents by using the `Upload a File` button. You can check the progress of
|
||||
the ingestion in the console logs of the server.
|
||||
|
||||
The list of ingested files is shown below the button.
|
||||
|
||||
If you want to delete the ingested documents, refer to *Reset Local documents
|
||||
database* section in the documentation.
|
||||
|
||||
### Chat
|
||||
|
||||
Normal chat interface, self-explanatory ;)
|
||||
|
||||
You can check the actual prompt being passed to the LLM by looking at the logs of
|
||||
the server. We'll add better observability in future releases.
|
||||
|
||||
## Deployment options
|
||||
|
||||
🚧 We are working on Dockerized deployment guidelines 🚧
|
||||
|
||||
## Observability
|
||||
|
||||
Basic logs are enabled using LlamaIndex
|
||||
basic logging (for example ingestion progress or LLM prompts and answers).
|
||||
|
||||
🚧 We are working on improved Observability. 🚧
|
||||
|
||||
## Ingesting & Managing Documents
|
||||
|
||||
🚧 Document Update and Delete are still WIP. 🚧
|
||||
|
||||
The ingestion of documents can be done in different ways:
|
||||
|
||||
* Using the `/ingest` API
|
||||
* Using the Gradio UI
|
||||
* Using the Bulk Local Ingestion functionality (check next section)
|
||||
|
||||
### Bulk Local Ingestion
|
||||
|
||||
When you are running PrivateGPT in a fully local setup, you can ingest a complete folder for convenience (containing
|
||||
pdf, text files, etc.)
|
||||
and optionally watch changes on it with the command:
|
||||
|
||||
```bash
|
||||
make ingest /path/to/folder -- --watch
|
||||
```
|
||||
|
||||
To log the processed and failed files to an additional file, use:
|
||||
|
||||
```bash
|
||||
make ingest /path/to/folder -- --watch --log-file /path/to/log/file.log
|
||||
```
|
||||
|
||||
After ingestion is complete, you should be able to chat with your documents
|
||||
by navigating to http://localhost:8001 and using the option `Query documents`,
|
||||
or using the completions / chat API.
|
||||
|
||||
### Reset Local documents database
|
||||
|
||||
When running in a local setup, you can remove all ingested documents by simply
|
||||
deleting all contents of `local_data` folder (except .gitignore).
|
||||
|
||||
To simplify this process, you can use the command:
|
||||
```bash
|
||||
make wipe
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
As explained in the introduction, the API contains high level APIs (ingestion and chat/completions) and low level APIs
|
||||
(embeddings and chunk retrieval). In this section the different specific API calls are explained.
|
||||
@@ -1,22 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>PrivateGPT Docs</title>
|
||||
<!-- needed for adaptive design -->
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link href="https://fonts.googleapis.com/css?family=Montserrat:300,400,700|Roboto:300,400,700" rel="stylesheet">
|
||||
<link rel="shortcut icon" href="https://fastapi.tiangolo.com/img/favicon.png">
|
||||
<!-- ReDoc doesn't change outer page styles -->
|
||||
<style>
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<noscript> ReDoc requires Javascript to function. Please enable it to browse the documentation. </noscript>
|
||||
<redoc spec-url="/openapi.json"></redoc>
|
||||
<script src="https://cdn.jsdelivr.net/npm/redoc@next/bundles/redoc.standalone.js"></script>
|
||||
</body>
|
||||
BIN
docs/logo.png
BIN
docs/logo.png
Binary file not shown.
|
Before Width: | Height: | Size: 2.6 KiB |
File diff suppressed because one or more lines are too long
@@ -38,6 +38,8 @@ llm:
|
||||
|
||||
openai:
|
||||
api_key: <your_openai_api_key> # You could skip this configuration and use the OPENAI_API_KEY env var instead
|
||||
model: <openai_model_to_use> # Optional model to use. Default is "gpt-3.5-turbo"
|
||||
# Note: Open AI Models are listed here: https://platform.openai.com/docs/models
|
||||
```
|
||||
|
||||
And run PrivateGPT loading that profile you just created:
|
||||
|
||||
@@ -35,5 +35,32 @@ database* section in the documentation.
|
||||
|
||||
Normal chat interface, self-explanatory ;)
|
||||
|
||||
You can check the actual prompt being passed to the LLM by looking at the logs of
|
||||
the server. We'll add better observability in future releases.
|
||||
#### System Prompt
|
||||
You can view and change the system prompt being passed to the LLM by clicking "Additional Inputs"
|
||||
in the chat interface. The system prompt is also logged on the server.
|
||||
|
||||
By default, the `Query Docs` mode uses the setting value `ui.default_query_system_prompt`.
|
||||
|
||||
The `LLM Chat` mode attempts to use the optional settings value `ui.default_chat_system_prompt`.
|
||||
|
||||
If no system prompt is entered, the UI will display the default system prompt being used
|
||||
for the active mode.
|
||||
|
||||
##### System Prompt Examples:
|
||||
|
||||
The system prompt can effectively provide your chat bot specialized roles, and results tailored to the prompt
|
||||
you have given the model. Examples of system prompts can be be found
|
||||
[here](https://www.w3schools.com/gen_ai/chatgpt-3-5/chatgpt-3-5_roles.php).
|
||||
|
||||
Some interesting examples to try include:
|
||||
|
||||
* You are -X-. You have all the knowledge and personality of -X-. Answer as if you were -X- using
|
||||
their manner of speaking and vocabulary.
|
||||
* Example: You are Shakespeare. You have all the knowledge and personality of Shakespeare.
|
||||
Answer as if you were Shakespeare using their manner of speaking and vocabulary.
|
||||
* You are an expert (at) -role-. Answer all questions using your expertise on -specific domain topic-.
|
||||
* Example: You are an expert software engineer. Answer all questions using your expertise on Python.
|
||||
* You are a -role- bot, respond with -response criteria needed-. If no -response criteria- is needed,
|
||||
respond with -alternate response-.
|
||||
* Example: You are a grammar checking bot, respond with any grammatical corrections needed. If no corrections
|
||||
are needed, respond with "verified".
|
||||
@@ -92,4 +92,14 @@ local:
|
||||
llm_hf_model_file: godzilla2-70b.Q4_K_M.gguf
|
||||
embedding_hf_model_name: BAAI/bge-large-en
|
||||
prompt_style: "llama2"
|
||||
```
|
||||
```
|
||||
### German speaking model
|
||||
`settings-de.yaml`:
|
||||
```yml
|
||||
local:
|
||||
llm_hf_repo_id: TheBloke/em_german_leo_mistral-GGUF
|
||||
llm_hf_model_file: em_german_leo_mistral.Q4_K_M.gguf
|
||||
embedding_hf_model_name: T-Systems-onsite/german-roberta-sentence-transformer-v2
|
||||
#llama, default or tag
|
||||
prompt_style: "default"
|
||||
```
|
||||
|
||||
@@ -1,20 +1,8 @@
|
||||
{
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "PrivateGPT",
|
||||
"summary": "PrivateGPT is a production-ready AI project that allows you to ask questions to your documents using the power of Large Language Models (LLMs), even in scenarios without Internet connection. 100% private, no data leaves your execution environment at any point.",
|
||||
"description": "",
|
||||
"contact": {
|
||||
"url": "https://github.com/imartinez/privateGPT"
|
||||
},
|
||||
"license": {
|
||||
"name": "Apache 2.0",
|
||||
"url": "https://www.apache.org/licenses/LICENSE-2.0.html"
|
||||
},
|
||||
"version": "0.1.0",
|
||||
"x-logo": {
|
||||
"url": "https://lh3.googleusercontent.com/drive-viewer/AK7aPaD_iNlMoTquOBsw4boh4tIYxyEuhz6EtEs8nzq3yNkNAK00xGjE1KUCmPJSk3TYOjcs6tReG6w_cLu1S7L_gPgT9z52iw=s2560"
|
||||
}
|
||||
"title": "FastAPI",
|
||||
"version": "0.1.0"
|
||||
},
|
||||
"paths": {
|
||||
"/v1/completions": {
|
||||
@@ -65,7 +53,7 @@
|
||||
"Contextual Completions"
|
||||
],
|
||||
"summary": "Chat Completion",
|
||||
"description": "Given a list of messages comprising a conversation, return a response.\n\nOptionally include a `system_prompt` to influence the way the LLM answers.\n\nIf `use_context` is set to `true`, the model will use context coming\nfrom the ingested documents to create the response. The documents being used can\nbe filtered using the `context_filter` and passing the document IDs to be used.\nIngested documents IDs can be found using `/ingest/list` endpoint. If you want\nall ingested documents to be used, remove `context_filter` altogether.\n\nWhen using `'include_sources': true`, the API will return the source Chunks used\nto create the response, which come from the context provided.\n\nWhen using `'stream': true`, the API will return data chunks following [OpenAI's\nstreaming model](https://platform.openai.com/docs/api-reference/chat/streaming):\n```\n{\"id\":\"12345\",\"object\":\"completion.chunk\",\"created\":1694268190,\n\"model\":\"private-gpt\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\n\"finish_reason\":null}]}\n```",
|
||||
"description": "Given a list of messages comprising a conversation, return a response.\n\nOptionally include an initial `role: system` message to influence the way\nthe LLM answers.\n\nIf `use_context` is set to `true`, the model will use context coming\nfrom the ingested documents to create the response. The documents being used can\nbe filtered using the `context_filter` and passing the document IDs to be used.\nIngested documents IDs can be found using `/ingest/list` endpoint. If you want\nall ingested documents to be used, remove `context_filter` altogether.\n\nWhen using `'include_sources': true`, the API will return the source Chunks used\nto create the response, which come from the context provided.\n\nWhen using `'stream': true`, the API will return data chunks following [OpenAI's\nstreaming model](https://platform.openai.com/docs/api-reference/chat/streaming):\n```\n{\"id\":\"12345\",\"object\":\"completion.chunk\",\"created\":1694268190,\n\"model\":\"private-gpt\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\n\"finish_reason\":null}]}\n```",
|
||||
"operationId": "chat_completion_v1_chat_completions_post",
|
||||
"requestBody": {
|
||||
"content": {
|
||||
@@ -338,17 +326,6 @@
|
||||
"type": "array",
|
||||
"title": "Messages"
|
||||
},
|
||||
"system_prompt": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "System Prompt"
|
||||
},
|
||||
"use_context": {
|
||||
"type": "boolean",
|
||||
"title": "Use Context",
|
||||
@@ -389,13 +366,16 @@
|
||||
},
|
||||
"include_sources": true,
|
||||
"messages": [
|
||||
{
|
||||
"content": "You are a rapper. Always answer with a rap.",
|
||||
"role": "system"
|
||||
},
|
||||
{
|
||||
"content": "How do you fry an egg?",
|
||||
"role": "user"
|
||||
}
|
||||
],
|
||||
"stream": false,
|
||||
"system_prompt": "You are a rapper. Always answer with a rap.",
|
||||
"use_context": true
|
||||
}
|
||||
]
|
||||
@@ -591,6 +571,7 @@
|
||||
"include_sources": false,
|
||||
"prompt": "How do you fry an egg?",
|
||||
"stream": false,
|
||||
"system_prompt": "You are a rapper. Always answer with a rap.",
|
||||
"use_context": false
|
||||
}
|
||||
]
|
||||
@@ -986,27 +967,5 @@
|
||||
"title": "ValidationError"
|
||||
}
|
||||
}
|
||||
},
|
||||
"tags": [
|
||||
{
|
||||
"name": "Ingestion",
|
||||
"description": "High-level APIs covering document ingestion -internally managing document parsing, splitting,metadata extraction, embedding generation and storage- and ingested documents CRUD.Each ingested document is identified by an ID that can be used to filter the contextused in *Contextual Completions* and *Context Chunks* APIs."
|
||||
},
|
||||
{
|
||||
"name": "Contextual Completions",
|
||||
"description": "High-level APIs covering contextual Chat and Completions. They follow OpenAI's format, extending it to allow using the context coming from ingested documents to create the response. Internallymanage context retrieval, prompt engineering and the response generation."
|
||||
},
|
||||
{
|
||||
"name": "Context Chunks",
|
||||
"description": "Low-level API that given a query return relevant chunks of text coming from the ingesteddocuments."
|
||||
},
|
||||
{
|
||||
"name": "Embeddings",
|
||||
"description": "Low-level API to obtain the vector representation of a given text, using an Embeddings model.Follows OpenAI's embeddings API format."
|
||||
},
|
||||
{
|
||||
"name": "Health",
|
||||
"description": "Simple health API to make sure the server is up and running."
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -23,10 +23,7 @@ class LLMComponent:
|
||||
case "local":
|
||||
from llama_index.llms import LlamaCPP
|
||||
|
||||
prompt_style_cls = get_prompt_style(settings.local.prompt_style)
|
||||
prompt_style = prompt_style_cls(
|
||||
default_system_prompt=settings.local.default_system_prompt
|
||||
)
|
||||
prompt_style = get_prompt_style(settings.local.prompt_style)
|
||||
|
||||
self.llm = LlamaCPP(
|
||||
model_path=str(models_path / settings.local.llm_hf_model_file),
|
||||
@@ -53,7 +50,9 @@ class LLMComponent:
|
||||
case "openai":
|
||||
from llama_index.llms import OpenAI
|
||||
|
||||
openai_settings = settings.openai.api_key
|
||||
self.llm = OpenAI(api_key=openai_settings)
|
||||
openai_settings = settings.openai
|
||||
self.llm = OpenAI(
|
||||
api_key=openai_settings.api_key, model=openai_settings.model
|
||||
)
|
||||
case "mock":
|
||||
self.llm = MockLLM()
|
||||
|
||||
@@ -5,7 +5,6 @@ from typing import Any, Literal
|
||||
|
||||
from llama_index.llms import ChatMessage, MessageRole
|
||||
from llama_index.llms.llama_utils import (
|
||||
DEFAULT_SYSTEM_PROMPT,
|
||||
completion_to_prompt,
|
||||
messages_to_prompt,
|
||||
)
|
||||
@@ -29,7 +28,6 @@ class AbstractPromptStyle(abc.ABC):
|
||||
series of messages into a prompt.
|
||||
"""
|
||||
|
||||
@abc.abstractmethod
|
||||
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||
logger.debug("Initializing prompt_style=%s", self.__class__.__name__)
|
||||
|
||||
@@ -52,15 +50,6 @@ class AbstractPromptStyle(abc.ABC):
|
||||
return prompt
|
||||
|
||||
|
||||
class AbstractPromptStyleWithSystemPrompt(AbstractPromptStyle, abc.ABC):
|
||||
_DEFAULT_SYSTEM_PROMPT = DEFAULT_SYSTEM_PROMPT
|
||||
|
||||
def __init__(self, default_system_prompt: str | None) -> None:
|
||||
super().__init__()
|
||||
logger.debug("Got default_system_prompt='%s'", default_system_prompt)
|
||||
self.default_system_prompt = default_system_prompt
|
||||
|
||||
|
||||
class DefaultPromptStyle(AbstractPromptStyle):
|
||||
"""Default prompt style that uses the defaults from llama_utils.
|
||||
|
||||
@@ -83,7 +72,7 @@ class DefaultPromptStyle(AbstractPromptStyle):
|
||||
return ""
|
||||
|
||||
|
||||
class Llama2PromptStyle(AbstractPromptStyleWithSystemPrompt):
|
||||
class Llama2PromptStyle(AbstractPromptStyle):
|
||||
"""Simple prompt style that just uses the default llama_utils functions.
|
||||
|
||||
It transforms the sequence of messages into a prompt that should look like:
|
||||
@@ -94,18 +83,14 @@ class Llama2PromptStyle(AbstractPromptStyleWithSystemPrompt):
|
||||
```
|
||||
"""
|
||||
|
||||
def __init__(self, default_system_prompt: str | None = None) -> None:
|
||||
# If no system prompt is given, the default one of the implementation is used.
|
||||
super().__init__(default_system_prompt=default_system_prompt)
|
||||
|
||||
def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
|
||||
return messages_to_prompt(messages, self.default_system_prompt)
|
||||
return messages_to_prompt(messages)
|
||||
|
||||
def _completion_to_prompt(self, completion: str) -> str:
|
||||
return completion_to_prompt(completion, self.default_system_prompt)
|
||||
return completion_to_prompt(completion)
|
||||
|
||||
|
||||
class TagPromptStyle(AbstractPromptStyleWithSystemPrompt):
|
||||
class TagPromptStyle(AbstractPromptStyle):
|
||||
"""Tag prompt style (used by Vigogne) that uses the prompt style `<|ROLE|>`.
|
||||
|
||||
It transforms the sequence of messages into a prompt that should look like:
|
||||
@@ -119,37 +104,8 @@ class TagPromptStyle(AbstractPromptStyleWithSystemPrompt):
|
||||
FIXME: should we add surrounding `<s>` and `</s>` tags, like in llama2?
|
||||
"""
|
||||
|
||||
def __init__(self, default_system_prompt: str | None = None) -> None:
|
||||
# We have to define a default system prompt here as the LLM will not
|
||||
# use the default llama_utils functions.
|
||||
default_system_prompt = default_system_prompt or self._DEFAULT_SYSTEM_PROMPT
|
||||
super().__init__(default_system_prompt)
|
||||
self.system_prompt: str = default_system_prompt
|
||||
|
||||
def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
|
||||
messages = list(messages)
|
||||
if messages[0].role != MessageRole.SYSTEM:
|
||||
logger.info(
|
||||
"Adding system_promt='%s' to the given messages as there are none given in the session",
|
||||
self.system_prompt,
|
||||
)
|
||||
messages = [
|
||||
ChatMessage(content=self.system_prompt, role=MessageRole.SYSTEM),
|
||||
*messages,
|
||||
]
|
||||
return self._format_messages_to_prompt(messages)
|
||||
|
||||
def _completion_to_prompt(self, completion: str) -> str:
|
||||
return (
|
||||
f"<|system|>: {self.system_prompt.strip()}\n"
|
||||
f"<|user|>: {completion.strip()}\n"
|
||||
"<|assistant|>: "
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _format_messages_to_prompt(messages: list[ChatMessage]) -> str:
|
||||
"""Format message to prompt with `<|ROLE|>: MSG` style."""
|
||||
assert messages[0].role == MessageRole.SYSTEM
|
||||
prompt = ""
|
||||
for message in messages:
|
||||
role = message.role
|
||||
@@ -161,19 +117,24 @@ class TagPromptStyle(AbstractPromptStyleWithSystemPrompt):
|
||||
prompt += "<|assistant|>: "
|
||||
return prompt
|
||||
|
||||
def _completion_to_prompt(self, completion: str) -> str:
|
||||
return self._messages_to_prompt(
|
||||
[ChatMessage(content=completion, role=MessageRole.USER)]
|
||||
)
|
||||
|
||||
|
||||
def get_prompt_style(
|
||||
prompt_style: Literal["default", "llama2", "tag"] | None
|
||||
) -> type[AbstractPromptStyle]:
|
||||
) -> AbstractPromptStyle:
|
||||
"""Get the prompt style to use from the given string.
|
||||
|
||||
:param prompt_style: The prompt style to use.
|
||||
:return: The prompt style to use.
|
||||
"""
|
||||
if prompt_style is None or prompt_style == "default":
|
||||
return DefaultPromptStyle
|
||||
return DefaultPromptStyle()
|
||||
elif prompt_style == "llama2":
|
||||
return Llama2PromptStyle
|
||||
return Llama2PromptStyle()
|
||||
elif prompt_style == "tag":
|
||||
return TagPromptStyle
|
||||
return TagPromptStyle()
|
||||
raise ValueError(f"Unknown prompt_style='{prompt_style}'")
|
||||
|
||||
@@ -1,13 +1,10 @@
|
||||
"""FastAPI app creation, logger configuration and main API routes."""
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import Depends, FastAPI, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.openapi.utils import get_openapi
|
||||
from injector import Injector
|
||||
|
||||
from private_gpt.paths import docs_path
|
||||
from private_gpt.server.chat.chat_router import chat_router
|
||||
from private_gpt.server.chunks.chunks_router import chunks_router
|
||||
from private_gpt.server.completions.completions_router import completions_router
|
||||
@@ -22,107 +19,35 @@ logger = logging.getLogger(__name__)
|
||||
def create_app(root_injector: Injector) -> FastAPI:
|
||||
|
||||
# Start the API
|
||||
with open(docs_path / "description.md") as description_file:
|
||||
description = description_file.read()
|
||||
async def bind_injector_to_request(request: Request) -> None:
|
||||
request.state.injector = root_injector
|
||||
|
||||
tags_metadata = [
|
||||
{
|
||||
"name": "Ingestion",
|
||||
"description": "High-level APIs covering document ingestion -internally "
|
||||
"managing document parsing, splitting,"
|
||||
"metadata extraction, embedding generation and storage- and ingested "
|
||||
"documents CRUD."
|
||||
"Each ingested document is identified by an ID that can be used to filter the "
|
||||
"context"
|
||||
"used in *Contextual Completions* and *Context Chunks* APIs.",
|
||||
},
|
||||
{
|
||||
"name": "Contextual Completions",
|
||||
"description": "High-level APIs covering contextual Chat and Completions. They "
|
||||
"follow OpenAI's format, extending it to "
|
||||
"allow using the context coming from ingested documents to create the "
|
||||
"response. Internally"
|
||||
"manage context retrieval, prompt engineering and the response generation.",
|
||||
},
|
||||
{
|
||||
"name": "Context Chunks",
|
||||
"description": "Low-level API that given a query return relevant chunks of "
|
||||
"text coming from the ingested"
|
||||
"documents.",
|
||||
},
|
||||
{
|
||||
"name": "Embeddings",
|
||||
"description": "Low-level API to obtain the vector representation of a given "
|
||||
"text, using an Embeddings model."
|
||||
"Follows OpenAI's embeddings API format.",
|
||||
},
|
||||
{
|
||||
"name": "Health",
|
||||
"description": "Simple health API to make sure the server is up and running.",
|
||||
},
|
||||
]
|
||||
app = FastAPI(dependencies=[Depends(bind_injector_to_request)])
|
||||
|
||||
async def bind_injector_to_request(request: Request) -> None:
|
||||
request.state.injector = root_injector
|
||||
app.include_router(completions_router)
|
||||
app.include_router(chat_router)
|
||||
app.include_router(chunks_router)
|
||||
app.include_router(ingest_router)
|
||||
app.include_router(embeddings_router)
|
||||
app.include_router(health_router)
|
||||
|
||||
app = FastAPI(dependencies=[Depends(bind_injector_to_request)])
|
||||
settings = root_injector.get(Settings)
|
||||
if settings.server.cors.enabled:
|
||||
logger.debug("Setting up CORS middleware")
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_credentials=settings.server.cors.allow_credentials,
|
||||
allow_origins=settings.server.cors.allow_origins,
|
||||
allow_origin_regex=settings.server.cors.allow_origin_regex,
|
||||
allow_methods=settings.server.cors.allow_methods,
|
||||
allow_headers=settings.server.cors.allow_headers,
|
||||
)
|
||||
|
||||
def custom_openapi() -> dict[str, Any]:
|
||||
if app.openapi_schema:
|
||||
return app.openapi_schema
|
||||
openapi_schema = get_openapi(
|
||||
title="PrivateGPT",
|
||||
description=description,
|
||||
version="0.1.0",
|
||||
summary="PrivateGPT is a production-ready AI project that allows you to "
|
||||
"ask questions to your documents using the power of Large Language "
|
||||
"Models (LLMs), even in scenarios without Internet connection. "
|
||||
"100% private, no data leaves your execution environment at any point.",
|
||||
contact={
|
||||
"url": "https://github.com/imartinez/privateGPT",
|
||||
},
|
||||
license_info={
|
||||
"name": "Apache 2.0",
|
||||
"url": "https://www.apache.org/licenses/LICENSE-2.0.html",
|
||||
},
|
||||
routes=app.routes,
|
||||
tags=tags_metadata,
|
||||
)
|
||||
openapi_schema["info"]["x-logo"] = {
|
||||
"url": "https://lh3.googleusercontent.com/drive-viewer"
|
||||
"/AK7aPaD_iNlMoTquOBsw4boh4tIYxyEuhz6EtEs8nzq3yNkNAK00xGj"
|
||||
"E1KUCmPJSk3TYOjcs6tReG6w_cLu1S7L_gPgT9z52iw=s2560"
|
||||
}
|
||||
if settings.ui.enabled:
|
||||
logger.debug("Importing the UI module")
|
||||
from private_gpt.ui.ui import PrivateGptUi
|
||||
|
||||
app.openapi_schema = openapi_schema
|
||||
return app.openapi_schema
|
||||
ui = root_injector.get(PrivateGptUi)
|
||||
ui.mount_in_app(app, settings.ui.path)
|
||||
|
||||
app.openapi = custom_openapi # type: ignore[method-assign]
|
||||
|
||||
app.include_router(completions_router)
|
||||
app.include_router(chat_router)
|
||||
app.include_router(chunks_router)
|
||||
app.include_router(ingest_router)
|
||||
app.include_router(embeddings_router)
|
||||
app.include_router(health_router)
|
||||
|
||||
settings = root_injector.get(Settings)
|
||||
if settings.server.cors.enabled:
|
||||
logger.debug("Setting up CORS middleware")
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_credentials=settings.server.cors.allow_credentials,
|
||||
allow_origins=settings.server.cors.allow_origins,
|
||||
allow_origin_regex=settings.server.cors.allow_origin_regex,
|
||||
allow_methods=settings.server.cors.allow_methods,
|
||||
allow_headers=settings.server.cors.allow_headers,
|
||||
)
|
||||
|
||||
if settings.ui.enabled:
|
||||
logger.debug("Importing the UI module")
|
||||
from private_gpt.ui.ui import PrivateGptUi
|
||||
|
||||
ui = root_injector.get(PrivateGptUi)
|
||||
ui.mount_in_app(app, settings.ui.path)
|
||||
|
||||
return app
|
||||
return app
|
||||
|
||||
@@ -108,15 +108,6 @@ class LocalSettings(BaseModel):
|
||||
"`llama2` is the historic behaviour. `default` might work better with your custom models."
|
||||
),
|
||||
)
|
||||
default_system_prompt: str | None = Field(
|
||||
None,
|
||||
description=(
|
||||
"The default system prompt to use for the chat engine. "
|
||||
"If none is given - use the default system prompt (from the llama_index). "
|
||||
"Please note that the default prompt might not be the same for all prompt styles. "
|
||||
"Also note that this is only used if the first message is not a system message. "
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class EmbeddingSettings(BaseModel):
|
||||
@@ -154,11 +145,22 @@ class SagemakerSettings(BaseModel):
|
||||
|
||||
class OpenAISettings(BaseModel):
|
||||
api_key: str
|
||||
model: str = Field(
|
||||
"gpt-3.5-turbo",
|
||||
description="OpenAI Model to use. Example: 'gpt-4'.",
|
||||
)
|
||||
|
||||
|
||||
class UISettings(BaseModel):
|
||||
enabled: bool
|
||||
path: str
|
||||
default_chat_system_prompt: str = Field(
|
||||
None,
|
||||
description="The default system prompt to use for the chat mode.",
|
||||
)
|
||||
default_query_system_prompt: str = Field(
|
||||
None, description="The default system prompt to use for the query mode."
|
||||
)
|
||||
|
||||
|
||||
class QdrantSettings(BaseModel):
|
||||
|
||||
@@ -30,6 +30,8 @@ UI_TAB_TITLE = "My Private GPT"
|
||||
|
||||
SOURCES_SEPARATOR = "\n\n Sources: \n"
|
||||
|
||||
MODES = ["Query Docs", "Search in Docs", "LLM Chat"]
|
||||
|
||||
|
||||
class Source(BaseModel):
|
||||
file: str
|
||||
@@ -71,6 +73,10 @@ class PrivateGptUi:
|
||||
# Cache the UI blocks
|
||||
self._ui_block = None
|
||||
|
||||
# Initialize system prompt based on default mode
|
||||
self.mode = MODES[0]
|
||||
self._system_prompt = self._get_default_system_prompt(self.mode)
|
||||
|
||||
def _chat(self, message: str, history: list[list[str]], mode: str, *_: Any) -> Any:
|
||||
def yield_deltas(completion_gen: CompletionGen) -> Iterable[str]:
|
||||
full_response: str = ""
|
||||
@@ -114,25 +120,22 @@ class PrivateGptUi:
|
||||
|
||||
new_message = ChatMessage(content=message, role=MessageRole.USER)
|
||||
all_messages = [*build_history(), new_message]
|
||||
# If a system prompt is set, add it as a system message
|
||||
if self._system_prompt:
|
||||
all_messages.insert(
|
||||
0,
|
||||
ChatMessage(
|
||||
content=self._system_prompt,
|
||||
role=MessageRole.SYSTEM,
|
||||
),
|
||||
)
|
||||
match mode:
|
||||
case "Query Docs":
|
||||
# Add a system message to force the behaviour of the LLM
|
||||
# to answer only questions about the provided context.
|
||||
all_messages.insert(
|
||||
0,
|
||||
ChatMessage(
|
||||
content="You can only answer questions about the provided context. If you know the answer "
|
||||
"but it is not based in the provided context, don't provide the answer, just state "
|
||||
"the answer is not in the context provided.",
|
||||
role=MessageRole.SYSTEM,
|
||||
),
|
||||
)
|
||||
query_stream = self._chat_service.stream_chat(
|
||||
messages=all_messages,
|
||||
use_context=True,
|
||||
)
|
||||
yield from yield_deltas(query_stream)
|
||||
|
||||
case "LLM Chat":
|
||||
llm_stream = self._chat_service.stream_chat(
|
||||
messages=all_messages,
|
||||
@@ -154,6 +157,37 @@ class PrivateGptUi:
|
||||
for index, source in enumerate(sources, start=1)
|
||||
)
|
||||
|
||||
# On initialization and on mode change, this function set the system prompt
|
||||
# to the default prompt based on the mode (and user settings).
|
||||
@staticmethod
|
||||
def _get_default_system_prompt(mode: str) -> str:
|
||||
p = ""
|
||||
match mode:
|
||||
# For query chat mode, obtain default system prompt from settings
|
||||
case "Query Docs":
|
||||
p = settings().ui.default_query_system_prompt
|
||||
# For chat mode, obtain default system prompt from settings
|
||||
case "LLM Chat":
|
||||
p = settings().ui.default_chat_system_prompt
|
||||
# For any other mode, clear the system prompt
|
||||
case _:
|
||||
p = ""
|
||||
return p
|
||||
|
||||
def _set_system_prompt(self, system_prompt_input: str) -> None:
|
||||
logger.info(f"Setting system prompt to: {system_prompt_input}")
|
||||
self._system_prompt = system_prompt_input
|
||||
|
||||
def _set_current_mode(self, mode: str) -> Any:
|
||||
self.mode = mode
|
||||
self._set_system_prompt(self._get_default_system_prompt(mode))
|
||||
# Update placeholder and allow interaction if default system prompt is set
|
||||
if self._system_prompt:
|
||||
return gr.update(placeholder=self._system_prompt, interactive=True)
|
||||
# Update placeholder and disable interaction if no default system prompt is set
|
||||
else:
|
||||
return gr.update(placeholder=self._system_prompt, interactive=False)
|
||||
|
||||
def _list_ingested_files(self) -> list[list[str]]:
|
||||
files = set()
|
||||
for ingested_document in self._ingest_service.list_ingested():
|
||||
@@ -193,7 +227,7 @@ class PrivateGptUi:
|
||||
with gr.Row():
|
||||
with gr.Column(scale=3, variant="compact"):
|
||||
mode = gr.Radio(
|
||||
["Query Docs", "Search in Docs", "LLM Chat"],
|
||||
MODES,
|
||||
label="Mode",
|
||||
value="Query Docs",
|
||||
)
|
||||
@@ -220,6 +254,23 @@ class PrivateGptUi:
|
||||
outputs=ingested_dataset,
|
||||
)
|
||||
ingested_dataset.render()
|
||||
system_prompt_input = gr.Textbox(
|
||||
placeholder=self._system_prompt,
|
||||
label="System Prompt",
|
||||
lines=2,
|
||||
interactive=True,
|
||||
render=False,
|
||||
)
|
||||
# When mode changes, set default system prompt
|
||||
mode.change(
|
||||
self._set_current_mode, inputs=mode, outputs=system_prompt_input
|
||||
)
|
||||
# On blur, set system prompt to use in queries
|
||||
system_prompt_input.blur(
|
||||
self._set_system_prompt,
|
||||
inputs=system_prompt_input,
|
||||
)
|
||||
|
||||
with gr.Column(scale=7):
|
||||
_ = gr.ChatInterface(
|
||||
self._chat,
|
||||
@@ -232,7 +283,7 @@ class PrivateGptUi:
|
||||
AVATAR_BOT,
|
||||
),
|
||||
),
|
||||
additional_inputs=[mode, upload_button],
|
||||
additional_inputs=[mode, upload_button, system_prompt_input],
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
@@ -22,6 +22,13 @@ data:
|
||||
ui:
|
||||
enabled: true
|
||||
path: /
|
||||
default_chat_system_prompt: "You are a helpful, respectful and honest assistant.
|
||||
Always answer as helpfully as possible and follow ALL given instructions.
|
||||
Do not speculate or make up information.
|
||||
Do not reference any given instructions or context."
|
||||
default_query_system_prompt: "You can only answer questions about the provided context.
|
||||
If you know the answer but it is not based in the provided context, don't provide
|
||||
the answer, just state the answer is not in the context provided."
|
||||
|
||||
llm:
|
||||
mode: local
|
||||
@@ -49,3 +56,4 @@ sagemaker:
|
||||
|
||||
openai:
|
||||
api_key: ${OPENAI_API_KEY:}
|
||||
model: gpt-3.5-turbo
|
||||
@@ -18,7 +18,7 @@ from private_gpt.components.llm.prompt_helper import (
|
||||
],
|
||||
)
|
||||
def test_get_prompt_style_success(prompt_style, expected_prompt_style):
|
||||
assert get_prompt_style(prompt_style) == expected_prompt_style
|
||||
assert isinstance(get_prompt_style(prompt_style), expected_prompt_style)
|
||||
|
||||
|
||||
def test_get_prompt_style_failure():
|
||||
@@ -45,20 +45,7 @@ def test_tag_prompt_style_format():
|
||||
|
||||
|
||||
def test_tag_prompt_style_format_with_system_prompt():
|
||||
system_prompt = "This is a system prompt from configuration."
|
||||
prompt_style = TagPromptStyle(default_system_prompt=system_prompt)
|
||||
messages = [
|
||||
ChatMessage(content="Hello, how are you doing?", role=MessageRole.USER),
|
||||
]
|
||||
|
||||
expected_prompt = (
|
||||
f"<|system|>: {system_prompt}\n"
|
||||
"<|user|>: Hello, how are you doing?\n"
|
||||
"<|assistant|>: "
|
||||
)
|
||||
|
||||
assert prompt_style.messages_to_prompt(messages) == expected_prompt
|
||||
|
||||
prompt_style = TagPromptStyle()
|
||||
messages = [
|
||||
ChatMessage(
|
||||
content="FOO BAR Custom sys prompt from messages.", role=MessageRole.SYSTEM
|
||||
@@ -94,22 +81,7 @@ def test_llama2_prompt_style_format():
|
||||
|
||||
|
||||
def test_llama2_prompt_style_with_system_prompt():
|
||||
system_prompt = "This is a system prompt from configuration."
|
||||
prompt_style = Llama2PromptStyle(default_system_prompt=system_prompt)
|
||||
messages = [
|
||||
ChatMessage(content="Hello, how are you doing?", role=MessageRole.USER),
|
||||
]
|
||||
|
||||
expected_prompt = (
|
||||
"<s> [INST] <<SYS>>\n"
|
||||
f" {system_prompt} \n"
|
||||
"<</SYS>>\n"
|
||||
"\n"
|
||||
" Hello, how are you doing? [/INST]"
|
||||
)
|
||||
|
||||
assert prompt_style.messages_to_prompt(messages) == expected_prompt
|
||||
|
||||
prompt_style = Llama2PromptStyle()
|
||||
messages = [
|
||||
ChatMessage(
|
||||
content="FOO BAR Custom sys prompt from messages.", role=MessageRole.SYSTEM
|
||||
|
||||
@@ -1 +1 @@
|
||||
0.1.0
|
||||
0.2.0
|
||||
|
||||
Reference in New Issue
Block a user