From 117a8e7faad881fa997797786882d0f6e56ee604 Mon Sep 17 00:00:00 2001 From: Max Cembalest <70534565+mcembalest@users.noreply.github.com> Date: Thu, 26 Sep 2024 11:07:49 -0400 Subject: [PATCH] Docs section & page for the GPT4All API server (#2990) Signed-off-by: Max Cembalest --- .../python/docs/gpt4all_api_server/home.md | 86 +++++++++++++++++++ .../python/docs/gpt4all_desktop/settings.md | 1 - gpt4all-bindings/python/mkdocs.yml | 2 + 3 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 gpt4all-bindings/python/docs/gpt4all_api_server/home.md diff --git a/gpt4all-bindings/python/docs/gpt4all_api_server/home.md b/gpt4all-bindings/python/docs/gpt4all_api_server/home.md new file mode 100644 index 00000000..d13740ec --- /dev/null +++ b/gpt4all-bindings/python/docs/gpt4all_api_server/home.md @@ -0,0 +1,86 @@ +# GPT4All API Server + +GPT4All provides a local API server that allows you to run LLMs over an HTTP API. + +## Key Features + +- **Local Execution**: Run models on your own hardware for privacy and offline use. +- **LocalDocs Integration**: Run the API with relevant text snippets provided to your LLM from a [LocalDocs collection](../gpt4all_desktop/localdocs.md). +- **OpenAI API Compatibility**: Use existing OpenAI-compatible clients and tools with your local models. + +## Activating the API Server + +1. Open the GPT4All Chat Desktop Application. +2. Go to `Settings` > `Application` and scroll down to `Advanced`. +3. Check the box for the `"Enable Local API Server"` setting. +4. The server listens on port 4891 by default. You can choose another port number in the `"API Server Port"` setting. + +## Connecting to the API Server + +The base URL used for the API server is `http://localhost:4891/v1` (or `http://localhost:/v1` if you are using a different port number). + +The server only accepts HTTP connections (not HTTPS) and only listens on localhost (127.0.0.1) (e.g. not to the IPv6 localhost address `::1`.) + +## Examples + +!!! note "Example GPT4All API calls" + + === "cURL" + + ```bash + curl -X POST http://localhost:4891/v1/chat/completions -d '{ + "model": "Phi-3 Mini Instruct", + "messages": [{"role":"user","content":"Who is Lionel Messi?"}], + "max_tokens": 50, + "temperature": 0.28 + }' + ``` + + === "PowerShell" + + ```powershell + Invoke-WebRequest -URI http://localhost:4891/v1/chat/completions -Method POST -ContentType application/json -Body '{ + "model": "Phi-3 Mini Instruct", + "messages": [{"role":"user","content":"Who is Lionel Messi?"}], + "max_tokens": 50, + "temperature": 0.28 + }' + ``` + +## API Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/v1/models` | List available models | +| GET | `/v1/models/` | Get details of a specific model | +| POST | `/v1/completions` | Generate text completions | +| POST | `/v1/chat/completions` | Generate chat completions | + +## LocalDocs Integration + +You can use LocalDocs with the API server: + +1. Open the Chats view in the GPT4All application. +2. Scroll to the bottom of the chat history sidebar. +3. Select the server chat (it has a different background color). +4. Activate LocalDocs collections in the right sidebar. + +(Note: LocalDocs can currently only be activated through the GPT4All UI, not via the API itself). + +Now, your API calls to your local LLM will have relevant references from your LocalDocs collection retrieved and placed in the input message for the LLM to respond to. + +The references retrieved for your API call can be accessed in the API response object at + +`response["choices"][0]["references"]` + +The data included in the `references` are: + +- `text`: the actual text content from the snippet that was extracted from the reference document + +- `author`: the author of the reference document (if available) + +- `date`: the date of creation of the reference document (if available) + +- `page`: the page number the snippet is from (only available for PDF documents for now) + +- `title`: the title of the reference document (if available) diff --git a/gpt4all-bindings/python/docs/gpt4all_desktop/settings.md b/gpt4all-bindings/python/docs/gpt4all_desktop/settings.md index 2b7017e2..87e8aecc 100644 --- a/gpt4all-bindings/python/docs/gpt4all_desktop/settings.md +++ b/gpt4all-bindings/python/docs/gpt4all_desktop/settings.md @@ -11,7 +11,6 @@ | **Device** | Device that will run your models. Options are `Auto` (GPT4All chooses), `Metal` (Apple Silicon M1+), `CPU`, and `GPU` | `Auto` | | **Default Model** | Choose your preferred LLM to load by default on startup| Auto | | **Download Path** | Select a destination on your device to save downloaded models | Windows: `C:\Users\{username}\AppData\Local\nomic.ai\GPT4All`

Mac: `/Users/{username}/Library/Application Support/nomic.ai/GPT4All/`

Linux: `/home/{username}/.local/share/nomic.ai/GPT4All` | - | **Enable Datalake** | Opt-in to sharing interactions with GPT4All community (**anonymous** and **optional**) | Off | !!! note "Advanced Application Settings" diff --git a/gpt4all-bindings/python/mkdocs.yml b/gpt4all-bindings/python/mkdocs.yml index ea5ecc1c..4e7d73ab 100644 --- a/gpt4all-bindings/python/mkdocs.yml +++ b/gpt4all-bindings/python/mkdocs.yml @@ -18,6 +18,8 @@ nav: - 'Local AI Chat with your Google Drive': 'gpt4all_desktop/cookbook/use-local-ai-models-to-privately-chat-with-google-drive.md' - 'Local AI Chat with your Obsidian Vault': 'gpt4all_desktop/cookbook/use-local-ai-models-to-privately-chat-with-Obsidian.md' - 'Local AI Chat with your OneDrive': 'gpt4all_desktop/cookbook/use-local-ai-models-to-privately-chat-with-One-Drive.md' + - 'API Server': + - 'gpt4all_api_server/home.md' - 'Python SDK': - 'gpt4all_python/home.md' - 'Monitoring': 'gpt4all_python/monitoring.md'