WIP

Signed-off-by: Jared Van Bortel <jared@nomic.ai>

.circleci/config.yml

@@ -8,11 +8,13 @@ workflows:
   generate-config:
     jobs:
       - path-filtering/filter:
+          filters:
+            tags:
+              only:
+                - /.*/
           base-revision: main
           config-path: .circleci/continue_config.yml
           mapping: |
             .circleci/.* run-all-workflows true
             gpt4all-backend/.* run-all-workflows true
-            gpt4all-bindings/python/.* run-python-workflow true
-            gpt4all-bindings/typescript/.* run-ts-workflow true
             gpt4all-chat/.* run-chat-workflow true

.github/ISSUE_TEMPLATE/bindings-bug.md

@@ -1,35 +0,0 @@
----
-name: "\U0001F6E0 Bindings Bug Report"
-about: A bug report for the GPT4All Bindings
-labels: ["bindings", "bug-unconfirmed"]
----
-
-<!-- Before creating a new issue, please make sure to take a few moments to check the issue tracker for existing issues about the bug. -->
-
-### Bug Report
-
-<!-- A clear and concise description of what the bug is. -->
-
-### Example Code
-
-<!-- Please provide a minimal code example that can be used to experience this issue. Delete this section if it does not apply. -->
-
-### Steps to Reproduce
-
-<!-- List the steps that should be taken to experience this issue. -->
-
-1.
-2.
-3.
-
-### Expected Behavior
-
-<!-- In a few words, what did you expect to happen? -->
-
-### Your Environment
-
-- Bindings version (e.g. "Version" from `pip show gpt4all`):
-- Operating System:
-- Chat model used (if applicable):
-
-<!-- You can freely edit this text, please remove all the lines you believe are unnecessary. -->

.gitmodules

@@ -1,7 +1,7 @@
-[submodule "llama.cpp-mainline"]
-	path = gpt4all-backend/deps/llama.cpp-mainline
+[submodule "gpt4all-backend-old/deps/llama.cpp-mainline"]
+	path = gpt4all-backend-old/deps/llama.cpp-mainline
 	url = https://github.com/nomic-ai/llama.cpp.git
-	branch = master
+    branch = master
 [submodule "gpt4all-chat/usearch"]
 	path = gpt4all-chat/deps/usearch
 	url = https://github.com/nomic-ai/usearch.git
@@ -9,17 +9,20 @@
 	path = gpt4all-chat/deps/SingleApplication
 	url = https://github.com/nomic-ai/SingleApplication.git
 [submodule "gpt4all-chat/deps/fmt"]
-	path = gpt4all-chat/deps/fmt
-	url = https://github.com/fmtlib/fmt.git
+	path = deps/fmt
+	url = https://github.com/nomic-ai/fmt.git
 [submodule "gpt4all-chat/deps/DuckX"]
 	path = gpt4all-chat/deps/DuckX
 	url = https://github.com/nomic-ai/DuckX.git
 [submodule "gpt4all-chat/deps/QXlsx"]
 	path = gpt4all-chat/deps/QXlsx
 	url = https://github.com/nomic-ai/QXlsx.git
-[submodule "gpt4all-chat/deps/Jinja2Cpp"]
-	path = gpt4all-chat/deps/Jinja2Cpp
-	url = https://github.com/nomic-ai/jinja2cpp.git
-[submodule "gpt4all-chat/deps/rapidjson"]
-	path = gpt4all-chat/deps/rapidjson
-	url = https://github.com/nomic-ai/rapidjson.git
+[submodule "gpt4all-chat/deps/minja"]
+	path = gpt4all-chat/deps/minja
+	url = https://github.com/nomic-ai/minja.git
+[submodule "gpt4all-chat/deps/json"]
+	path = gpt4all-chat/deps/json
+	url = https://github.com/nlohmann/json.git
+[submodule "gpt4all-backend/deps/qcoro"]
+	path = deps/qcoro
+	url = https://github.com/qcoro/qcoro.git

MAINTAINERS.md

@@ -29,13 +29,6 @@ Jared Van Bortel ([@cebtenzzre](https://github.com/cebtenzzre))<br/>
 E-mail: jared@nomic.ai<br/>
 Discord: `@cebtenzzre`
 - gpt4all-backend
-- Python binding
-- Python CLI app
-
-Jacob Nguyen ([@jacoobes](https://github.com/jacoobes))<br/>
-Discord: `@jacoobes`<br/>
-E-mail: `jacoobes@sern.dev`
-- TypeScript binding
 
 Dominik ([@cosmic-snow](https://github.com/cosmic-snow))<br/>
 E-mail: cosmic-snow@mailfence.com<br/>
@@ -45,17 +38,12 @@ Discord: `@cosmic__snow`
 Max Cembalest ([@mcembalest](https://github.com/mcembalest))<br/>
 E-mail: max@nomic.ai<br/>
 Discord: `@maxcembalest.`
-- Official documentation (gpt4all-bindings/python/docs -> https://docs.gpt4all.io/)
+- Official documentation (docs -> https://docs.gpt4all.io/)
 
 Thiago Ramos ([@thiagojramos](https://github.com/thiagojramos))<br/>
 E-mail: thiagojramos@outlook.com<br/>
 - pt\_BR translation
 
-Victor Emanuel ([@SINAPSA-IC](https://github.com/SINAPSA-IC))<br/>
-E-mail: contact@sinapsaro.ro<br/>
-Discord: `@sinapsa_ic_56124_99632`
-- ro\_RO translation
-
 不知火 Shiranui ([@supersonictw](https://github.com/supersonictw))<br/>
 E-mail: supersonic@livemail.tw<br/>
 Discord: `@supersonictw`
@@ -77,6 +65,6 @@ Discord: `@Tim453`
 - Flatpak
 
 Jack ([@wuodoo](https://github.com/wuodoo))<br/>
-E-mail: 2296103047@qq.com><br/>
+E-mail: 2296103047@qq.com<br/>
 Discord: `@mikage`
 - zh\_CN translation

README.md

@@ -1,5 +1,9 @@
 <h1 align="center">GPT4All</h1>
 
+<p align="center">
+  Now with support for DeepSeek R1 Distillations
+</p>
+
 <p align="center">
   <a href="https://www.nomic.ai/gpt4all">Website</a> &bull; <a href="https://docs.gpt4all.io">Documentation</a> &bull; <a href="https://discord.gg/mGZE39AS3e">Discord</a> &bull; <a href="https://www.youtube.com/watch?v=gQcZDXRVJok">YouTube Tutorial</a>
 </p>
@@ -23,25 +27,22 @@ https://github.com/nomic-ai/gpt4all/assets/70534565/513a0f15-4964-4109-89e4-4f9a
 <p align="center">
 GPT4All is made possible by our compute partner <a href="https://www.paperspace.com/">Paperspace</a>.
 </p>
-<p align="center">
- <a href="https://www.phorm.ai/query?projectId=755eecd3-24ad-49cc-abf4-0ab84caacf63"><img src="https://img.shields.io/badge/Phorm-Ask_AI-%23F2777A.svg" alt="phorm.ai"></a>
-</p>
 
 ## Download Links
 
 <p>
   &mdash; <a href="https://gpt4all.io/installers/gpt4all-installer-win64.exe">
-    <img src="gpt4all-bindings/python/docs/assets/windows.png" style="height: 1em; width: auto" /> Windows Installer
+    <img src="docs/assets/windows.png" style="height: 1em; width: auto" /> Windows Installer
   </a> &mdash;
 </p>
 <p>
   &mdash; <a href="https://gpt4all.io/installers/gpt4all-installer-darwin.dmg">
-    <img src="gpt4all-bindings/python/docs/assets/mac.png" style="height: 1em; width: auto" /> macOS Installer
+    <img src="docs/assets/mac.png" style="height: 1em; width: auto" /> macOS Installer
   </a> &mdash;
 </p>
 <p>
   &mdash; <a href="https://gpt4all.io/installers/gpt4all-installer-linux.run">
-    <img src="gpt4all-bindings/python/docs/assets/ubuntu.svg" style="height: 1em; width: auto" /> Ubuntu Installer
+    <img src="docs/assets/ubuntu.svg" style="height: 1em; width: auto" /> Ubuntu Installer
   </a> &mdash;
 </p>
 <p>
@@ -62,24 +63,6 @@ See the full [System Requirements](gpt4all-chat/system_requirements.md) for more
   </a>
 </p>
 
-## Install GPT4All Python
-
-`gpt4all` gives you access to LLMs with our Python client around [`llama.cpp`](https://github.com/ggerganov/llama.cpp) implementations. 
-
-Nomic contributes to open source software like [`llama.cpp`](https://github.com/ggerganov/llama.cpp) to make LLMs accessible and efficient **for all**.
-
-```bash
-pip install gpt4all
-```
-
-```python
-from gpt4all import GPT4All
-model = GPT4All("Meta-Llama-3-8B-Instruct.Q4_0.gguf") # downloads / loads a 4.66GB LLM
-with model.chat_session():
-    print(model.generate("How can I run LLMs efficiently on my laptop?", max_tokens=1024))
-```
-
-
 ## Integrations
 
 :parrot::link: [Langchain](https://python.langchain.com/v0.2/docs/integrations/providers/gpt4all/)
@@ -107,7 +90,7 @@ Please see CONTRIBUTING.md and follow the issues, bug reports, and PR markdown t
 
 Check project discord, with project owners, or through existing issues/PRs to avoid duplicate work.
 Please make sure to tag all of the above with relevant project identifiers or your contribution could potentially get lost.
-Example tags: `backend`, `bindings`, `python-bindings`, `documentation`, etc.
+Example tags: `backend`, `documentation`, etc.
 
 ## Citation
 

common/common.cmake

@@ -1,3 +1,8 @@
+# enable color diagnostics with ninja
+if (CMAKE_GENERATOR STREQUAL Ninja AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
+    add_compile_options(-fdiagnostics-color=always)
+endif()
+
 function(gpt4all_add_warning_options target)
     if (MSVC)
         return()
@@ -11,7 +16,6 @@ function(gpt4all_add_warning_options target)
         -Wextra-semi
         -Wformat=2
         -Wmissing-include-dirs
-        -Wstrict-overflow=2
         -Wsuggest-override
         -Wvla
         # errors

deps/CMakeLists.txt

@@ -0,0 +1,14 @@
+set(BUILD_SHARED_LIBS OFF)
+
+set(FMT_INSTALL OFF)
+set(FMT_MODULE ON)
+add_subdirectory(fmt)
+
+set(BUILD_TESTING           OFF)
+set(QCORO_BUILD_EXAMPLES    OFF)
+set(QCORO_WITH_QTDBUS       OFF)
+set(QCORO_WITH_QTWEBSOCKETS OFF)
+set(QCORO_WITH_QTQUICK      OFF)
+set(QCORO_WITH_QML          OFF)
+set(QCORO_WITH_QTTEST       OFF)
+add_subdirectory(qcoro)

docs/gpt4all_desktop/cookbook/use-local-ai-models-to-privately-chat-with-Obsidian.md

@@ -46,7 +46,7 @@ Obsidian for Desktop is a powerful management and note-taking software designed
             <tr>
                <td>
                   <!-- Screenshot of adding collection in LocalDocs -->
-                  <img width="1348" alt="Screenshot of adding collection" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/124ef867a9d9afd9e14d3858cd77bce858f79773/gpt4all-bindings/python/docs/assets/obsidian_adding_collection.png">
+                  <img width="1348" alt="Screenshot of adding collection" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/main/docs/assets/obsidian_adding_collection.png">
                </td>
             </tr>
          </table>
@@ -65,7 +65,7 @@ Obsidian for Desktop is a powerful management and note-taking software designed
             <tr>
                <td>
                   <!-- Screenshot of accessing LocalDocs in chats -->
-                  <img width="1447" alt="Accessing LocalDocs in chats" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/124ef867a9d9afd9e14d3858cd77bce858f79773/gpt4all-bindings/python/docs/assets/obsidian_docs.png">
+                  <img width="1447" alt="Accessing LocalDocs in chats" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/main/docs/assets/obsidian_docs.png">
                </td>
             </tr>
          </table>
@@ -76,7 +76,7 @@ Obsidian for Desktop is a powerful management and note-taking software designed
             <tr>
                <td>
                   <!-- Screenshot of interacting sources -->
-                  <img width="662" alt="osbsidian user interaction" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/124ef867a9d9afd9e14d3858cd77bce858f79773/gpt4all-bindings/python/docs/assets/osbsidian_user_interaction.png">
+                  <img width="662" alt="osbsidian user interaction" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/main/docs/assets/osbsidian_user_interaction.png">
                </td>
             </tr>
          </table>
@@ -84,7 +84,7 @@ Obsidian for Desktop is a powerful management and note-taking software designed
             <tr>
                <td>
                   <!-- Screenshot of viewing sources -->
-                  <img width="662" alt="osbsidian GPT4ALL response" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/124ef867a9d9afd9e14d3858cd77bce858f79773/gpt4all-bindings/python/docs/assets/obsidian_response.png">
+                  <img width="662" alt="osbsidian GPT4ALL response" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/main/docs/assets/obsidian_response.png">
                </td>
             </tr>
          </table>
@@ -96,7 +96,7 @@ Obsidian for Desktop is a powerful management and note-taking software designed
             <tr>
                <td>
                   <!-- Referenced Files  -->
-                  <img width="643" alt="Referenced Files" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/124ef867a9d9afd9e14d3858cd77bce858f79773/gpt4all-bindings/python/docs/assets/obsidian_sources.png">
+                  <img width="643" alt="Referenced Files" src="https://raw.githubusercontent.com/nomic-ai/gpt4all/main/docs/assets/obsidian_sources.png">
                </td>
             </tr>
          </table>
@@ -104,6 +104,3 @@ Obsidian for Desktop is a powerful management and note-taking software designed
 ## How It Works
 
 Obsidian for Desktop syncs your Obsidian notes to your computer, while LocalDocs integrates these files into your LLM chats using embedding models. These models find semantically similar snippets from your files to enhance the context of your interactions.
-
-To learn more about embedding models and explore further, refer to the [Nomic Python SDK documentation](https://docs.nomic.ai/atlas/capabilities/embeddings).
-

docs/gpt4all_desktop/localdocs.md

@@ -44,5 +44,3 @@ LocalDocs brings the information you have from files on-device into your LLM cha
 ## How It Works
 
 A LocalDocs collection uses Nomic AI's free and fast on-device embedding models to index your folder into text snippets that each get an **embedding vector**. These vectors allow us to find snippets from your files that are semantically similar to the questions and prompts you enter in your chats. We then include those semantically similar snippets in the prompt to the LLM.
-
-To try the embedding models yourself, we recommend using the [Nomic Python SDK](https://docs.nomic.ai/atlas/capabilities/embeddings)

docs/gpt4all_desktop/settings.md

@@ -8,8 +8,10 @@
     | --- | --- | --- |
     | **Theme** | Color theme for the application. Options are `Light`, `Dark`, and `LegacyDark` | `Light` |
     | **Font Size** | Font size setting for text throughout the application. Options are Small, Medium, and Large | Small |
+    | **Language and Locale** | The language and locale of that language you wish to use | System Locale |
     | **Device** | Device that will run your models. Options are `Auto` (GPT4All chooses), `Metal` (Apple Silicon M1+), `CPU`, and `GPU` | `Auto` |
     | **Default Model** | Choose your preferred LLM to load by default on startup| Auto |
+    | **Suggestion Mode** | Generate suggested follow up questions at the end of responses | When chatting with LocalDocs | 
     | **Download Path** | Select a destination on your device to save downloaded models | Windows: `C:\Users\{username}\AppData\Local\nomic.ai\GPT4All`<br><br>Mac: `/Users/{username}/Library/Application Support/nomic.ai/GPT4All/`<br><br>Linux: `/home/{username}/.local/share/nomic.ai/GPT4All` |
     | **Enable Datalake** | Opt-in to sharing interactions with GPT4All community (**anonymous** and **optional**) | Off |
 
@@ -18,7 +20,7 @@
     | Setting | Description | Default Value |
     | --- | --- | --- |
     | **CPU Threads** | Number of concurrently running CPU threads (more can speed up responses) | 4 |
-    | **Save Chat Context** | Save chat context to disk to pick up exactly where a model left off. | Off |
+    | **Enable System Tray** | The application will minimize to the system tray / taskbar when the window is closed | Off |
     | **Enable Local Server** | Allow any application on your device to use GPT4All via an OpenAI-compatible GPT4All API | Off |
     | **API Server Port** | Local HTTP port for the local API server | 4891 |
 
@@ -29,8 +31,11 @@
     | Setting | Description | Default Value |
     | --- | --- | --- |
     | **Name** | Unique name of this model / character| set by model uploader |
-    | **System Prompt** | General instructions for the chats this model will be used for | set by model uploader |
-    | **Prompt Template** | Format of user <-> assistant interactions for the chats this model will be used for | set by model uploader |
+    | **Model File** | Filename (.gguf) of the model | set by model uploader |
+    | **System Message** | General instructions for the chats this model will be used for | set by model uploader |
+    | **Chat Template** | Format of user <-> assistant interactions for the chats this model will be used for | set by model uploader |
+    | **Chat Name Prompt** | Prompt used to automatically generate chat names | Describe the above conversation in seven words or less. |
+    | **Suggested FollowUp Prompt** | Prompt used to automatically generate follow up questions after a chat response | Suggest three very short factual follow-up questions that have not been answered yet or cannot be found inspired by the previous conversation and excerpts. |
 
 ### Clone
 

docs/gpt4all_help/faq.md

@@ -6,32 +6,16 @@
 
 We support models with a `llama.cpp` implementation which have been uploaded to [HuggingFace](https://huggingface.co/).
 
-### Which embedding models are supported?
-
-We support SBert and Nomic Embed Text v1 & v1.5.
-
 ## Software
 
 ### What software do I need?
 
 All you need is to [install GPT4all](../index.md) onto you Windows, Mac, or Linux computer.
 
-### Which SDK languages are supported?
-
-Our SDK is in Python for usability, but these are light bindings around [`llama.cpp`](https://github.com/ggerganov/llama.cpp) implementations that we contribute to for efficiency and accessibility on everyday computers.
-
 ### Is there an API?
 
 Yes, you can run your model in server-mode with our [OpenAI-compatible API](https://platform.openai.com/docs/api-reference/completions), which you can configure in [settings](../gpt4all_desktop/settings.md#application-settings)
 
-### Can I monitor a GPT4All deployment?
-
-Yes, GPT4All [integrates](../gpt4all_python/monitoring.md) with [OpenLIT](https://github.com/openlit/openlit) so you can deploy LLMs with user interactions and hardware usage automatically monitored for full observability.
-
-### Is there a command line interface (CLI)?
-
-[Yes](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-bindings/cli), we have a lightweight use of the Python client as a CLI. We welcome further contributions!
-
 ## Hardware
 
 ### What hardware do I need?

docs/gpt4all_help/troubleshooting.md

@@ -2,7 +2,7 @@
 
 ## Error Loading Models
 
-It is possible you are trying to load a model from HuggingFace whose weights are not compatible with our [backend](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-bindings).
+It is possible you are trying to load a model from HuggingFace whose weights are not compatible with our [backend](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-backend).
 
 Try downloading one of the officially supported models listed on the main models page in the application. If the problem persists, please share your experience on our [Discord](https://discord.com/channels/1076964370942267462).
 

docs/index.md

@@ -12,17 +12,3 @@ No API calls or GPUs required - you can just download the application and [get s
         [Download for Mac](https://gpt4all.io/installers/gpt4all-installer-darwin.dmg)     
         [Download for Linux](https://gpt4all.io/installers/gpt4all-installer-linux.run)
     </div>
-
-!!! note "Python SDK"
-    Use GPT4All in Python to program with LLMs implemented with the [`llama.cpp`](https://github.com/ggerganov/llama.cpp) backend and [Nomic's C backend](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-backend). Nomic contributes to open source software like [`llama.cpp`](https://github.com/ggerganov/llama.cpp) to make LLMs accessible and efficient **for all**.
-
-    ```bash
-    pip install gpt4all
-    ```
-
-    ```python
-    from gpt4all import GPT4All
-    model = GPT4All("Meta-Llama-3-8B-Instruct.Q4_0.gguf") # downloads / loads a 4.66GB LLM
-    with model.chat_session():
-        print(model.generate("How can I run LLMs efficiently on my laptop?", max_tokens=1024))
-    ```

gpt4all-backend-old/CMakeLists.txt

@@ -0,0 +1,189 @@
+cmake_minimum_required(VERSION 3.23)  # for FILE_SET
+
+include(../common/common.cmake)
+
+set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+if (APPLE)
+    option(BUILD_UNIVERSAL "Build a Universal binary on macOS" ON)
+else()
+    option(LLMODEL_KOMPUTE "llmodel: use Kompute"              ON)
+    option(LLMODEL_VULKAN  "llmodel: use Vulkan"               OFF)
+    option(LLMODEL_CUDA    "llmodel: use CUDA"                 ON)
+    option(LLMODEL_ROCM    "llmodel: use ROCm"                 OFF)
+endif()
+
+if (APPLE)
+  if (BUILD_UNIVERSAL)
+    # Build a Universal binary on macOS
+    # This requires that the found Qt library is compiled as Universal binaries.
+    set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64" CACHE STRING "" FORCE)
+  else()
+    # Build for the host architecture on macOS
+    if (NOT CMAKE_OSX_ARCHITECTURES)
+      set(CMAKE_OSX_ARCHITECTURES "${CMAKE_HOST_SYSTEM_PROCESSOR}" CACHE STRING "" FORCE)
+    endif()
+  endif()
+endif()
+
+# Include the binary directory for the generated header file
+include_directories("${CMAKE_CURRENT_BINARY_DIR}")
+
+set(LLMODEL_VERSION_MAJOR 0)
+set(LLMODEL_VERSION_MINOR 5)
+set(LLMODEL_VERSION_PATCH 0)
+set(LLMODEL_VERSION "${LLMODEL_VERSION_MAJOR}.${LLMODEL_VERSION_MINOR}.${LLMODEL_VERSION_PATCH}")
+project(llmodel VERSION ${LLMODEL_VERSION} LANGUAGES CXX C)
+
+set(CMAKE_CXX_STANDARD 23)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})
+set(BUILD_SHARED_LIBS ON)
+
+# Check for IPO support
+include(CheckIPOSupported)
+check_ipo_supported(RESULT IPO_SUPPORTED OUTPUT IPO_ERROR)
+if (NOT IPO_SUPPORTED)
+    message(WARNING "Interprocedural optimization is not supported by your toolchain! This will lead to bigger file sizes and worse performance: ${IPO_ERROR}")
+else()
+    message(STATUS "Interprocedural optimization support detected")
+endif()
+
+set(DIRECTORY deps/llama.cpp-mainline)
+include(llama.cpp.cmake)
+
+set(BUILD_VARIANTS)
+if (APPLE)
+    list(APPEND BUILD_VARIANTS metal)
+endif()
+if (LLMODEL_KOMPUTE)
+    list(APPEND BUILD_VARIANTS kompute kompute-avxonly)
+else()
+    list(PREPEND BUILD_VARIANTS cpu cpu-avxonly)
+endif()
+if (LLMODEL_VULKAN)
+    list(APPEND BUILD_VARIANTS vulkan vulkan-avxonly)
+endif()
+if (LLMODEL_CUDA)
+    cmake_minimum_required(VERSION 3.18)  # for CMAKE_CUDA_ARCHITECTURES
+
+    # Defaults must be set before enable_language(CUDA).
+    # Keep this in sync with the arch list in ggml/src/CMakeLists.txt.
+    if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+        # 52 == lowest CUDA 12 standard
+        # 60 == f16 CUDA intrinsics
+        # 61 == integer CUDA intrinsics
+        # 70 == compute capability at which unrolling a loop in mul_mat_q kernels is faster
+        if (GGML_CUDA_F16 OR GGML_CUDA_DMMV_F16)
+            set(CMAKE_CUDA_ARCHITECTURES "60;61;70;75") # needed for f16 CUDA intrinsics
+        else()
+            set(CMAKE_CUDA_ARCHITECTURES "52;61;70;75") # lowest CUDA 12 standard + lowest for integer intrinsics
+            #set(CMAKE_CUDA_ARCHITECTURES "OFF") # use this to compile much faster, but only F16 models work
+        endif()
+    endif()
+    message(STATUS "Using CUDA architectures: ${CMAKE_CUDA_ARCHITECTURES}")
+
+    include(CheckLanguage)
+    check_language(CUDA)
+    if (NOT CMAKE_CUDA_COMPILER)
+        message(WARNING "CUDA Toolkit not found. To build without CUDA, use -DLLMODEL_CUDA=OFF.")
+    endif()
+    enable_language(CUDA)
+    list(APPEND BUILD_VARIANTS cuda cuda-avxonly)
+endif()
+if (LLMODEL_ROCM)
+    enable_language(HIP)
+    list(APPEND BUILD_VARIANTS rocm rocm-avxonly)
+endif()
+
+# Go through each build variant
+foreach(BUILD_VARIANT IN LISTS BUILD_VARIANTS)
+    # Determine flags
+    if (BUILD_VARIANT MATCHES avxonly)
+        set(GPT4ALL_ALLOW_NON_AVX OFF)
+    else()
+        set(GPT4ALL_ALLOW_NON_AVX ON)
+    endif()
+    set(GGML_AVX2 ${GPT4ALL_ALLOW_NON_AVX})
+    set(GGML_F16C ${GPT4ALL_ALLOW_NON_AVX})
+    set(GGML_FMA  ${GPT4ALL_ALLOW_NON_AVX})
+
+    set(GGML_METAL   OFF)
+    set(GGML_KOMPUTE OFF)
+    set(GGML_VULKAN  OFF)
+    set(GGML_CUDA    OFF)
+    set(GGML_ROCM    OFF)
+    if (BUILD_VARIANT MATCHES metal)
+        set(GGML_METAL   ON)
+    elseif (BUILD_VARIANT MATCHES kompute)
+        set(GGML_KOMPUTE ON)
+    elseif (BUILD_VARIANT MATCHES vulkan)
+        set(GGML_VULKAN  ON)
+    elseif (BUILD_VARIANT MATCHES cuda)
+        set(GGML_CUDA    ON)
+    elseif (BUILD_VARIANT MATCHES rocm)
+        set(GGML_HIPBLAS ON)
+    endif()
+
+    # Include GGML
+    include_ggml(-mainline-${BUILD_VARIANT})
+
+    if (BUILD_VARIANT MATCHES metal)
+        set(GGML_METALLIB "${GGML_METALLIB}" PARENT_SCOPE)
+    endif()
+
+    # Function for preparing individual implementations
+    function(prepare_target TARGET_NAME BASE_LIB)
+        set(TARGET_NAME ${TARGET_NAME}-${BUILD_VARIANT})
+        message(STATUS "Configuring model implementation target ${TARGET_NAME}")
+        # Link to ggml/llama
+        target_link_libraries(${TARGET_NAME}
+            PRIVATE ${BASE_LIB}-${BUILD_VARIANT})
+        # Let it know about its build variant
+        target_compile_definitions(${TARGET_NAME}
+            PRIVATE GGML_BUILD_VARIANT="${BUILD_VARIANT}")
+        # Enable IPO if possible
+# FIXME: Doesn't work with msvc reliably. See https://github.com/nomic-ai/gpt4all/issues/841
+#        set_property(TARGET ${TARGET_NAME}
+#                     PROPERTY INTERPROCEDURAL_OPTIMIZATION ${IPO_SUPPORTED})
+    endfunction()
+
+    # Add each individual implementations
+    add_library(llamamodel-mainline-${BUILD_VARIANT} SHARED
+        src/llamamodel.cpp src/llmodel_shared.cpp)
+    gpt4all_add_warning_options(llamamodel-mainline-${BUILD_VARIANT})
+    target_compile_definitions(llamamodel-mainline-${BUILD_VARIANT} PRIVATE
+        LLAMA_VERSIONS=>=3 LLAMA_DATE=999999)
+    target_include_directories(llamamodel-mainline-${BUILD_VARIANT} PRIVATE
+        src include/gpt4all-backend
+    )
+    prepare_target(llamamodel-mainline llama-mainline)
+
+    if (NOT PROJECT_IS_TOP_LEVEL AND BUILD_VARIANT STREQUAL cuda)
+        set(CUDAToolkit_BIN_DIR ${CUDAToolkit_BIN_DIR} PARENT_SCOPE)
+    endif()
+endforeach()
+
+add_library(llmodel
+    src/dlhandle.cpp
+    src/llmodel.cpp
+    src/llmodel_c.cpp
+    src/llmodel_shared.cpp
+)
+gpt4all_add_warning_options(llmodel)
+target_sources(llmodel PUBLIC
+    FILE_SET public_headers TYPE HEADERS BASE_DIRS include
+    FILES include/gpt4all-backend/llmodel.h
+          include/gpt4all-backend/llmodel_c.h
+          include/gpt4all-backend/sysinfo.h
+)
+target_compile_definitions(llmodel PRIVATE LIB_FILE_EXT="${CMAKE_SHARED_LIBRARY_SUFFIX}")
+target_include_directories(llmodel PRIVATE src include/gpt4all-backend)
+
+set_target_properties(llmodel PROPERTIES
+                              VERSION ${PROJECT_VERSION}
+                              SOVERSION ${PROJECT_VERSION_MAJOR})
+
+set(COMPONENT_NAME_MAIN ${PROJECT_NAME})
+set(CMAKE_INSTALL_PREFIX ${CMAKE_BINARY_DIR}/install)

gpt4all-backend-old/src/llamamodel.cpp

@@ -53,6 +53,8 @@ static const std::vector<const char *> KNOWN_ARCHES {
     "gpt2",
     // "gptj", -- no inference code
     "gptneox",
+    "granite",
+    "granitemoe",
     "mpt",
     "baichuan",
     "starcoder",
@@ -80,6 +82,7 @@ static const std::vector<const char *> KNOWN_ARCHES {
     "command-r",
     // "dbrx", -- 16x12B parameters
     "olmo",
+    "olmoe",
     "openelm",
     // "arctic", -- 10B+128x3.66B parameters
     "deepseek2",

gpt4all-backend-old/src/llmodel.cpp

@@ -140,9 +140,14 @@ const std::vector<LLModel::Implementation> &LLModel::Implementation::implementat
             std::string path;
             // Split the paths string by the delimiter and process each path.
             while (std::getline(ss, path, ';')) {
-                std::u8string u8_path(path.begin(), path.end());
+                fs::directory_iterator iter;
+                try {
+                    iter = fs::directory_iterator(std::u8string(path.begin(), path.end()));
+                } catch (const fs::filesystem_error &) {
+                    continue; // skip nonexistent path
+                }
                 // Iterate over all libraries
-                for (const auto &f : fs::directory_iterator(u8_path)) {
+                for (const auto &f : iter) {
                     const fs::path &p = f.path();
 
                     if (p.extension() != LIB_FILE_EXT) continue;

gpt4all-backend-test/CMakeLists.txt

@@ -0,0 +1,8 @@
+cmake_minimum_required(VERSION 3.29)
+project(gpt4all-backend-test VERSION 0.1 LANGUAGES CXX)
+
+set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+include(../common/common.cmake)
+
+add_subdirectory(../gpt4all-backend "${CMAKE_CURRENT_BINARY_DIR}/gpt4all-backend")
+add_subdirectory(src)

gpt4all-backend-test/src/CMakeLists.txt

@@ -0,0 +1,21 @@
+set(TARGET test-backend)
+
+configure_file(config.cppm.in "${CMAKE_CURRENT_BINARY_DIR}/config.cppm")
+
+add_executable(${TARGET}
+    main.cpp
+)
+target_compile_features(${TARGET} PUBLIC cxx_std_23)
+if (CMAKE_COMPILER_IS_GNUCXX)
+    target_compile_options(${TARGET} PUBLIC -fmodules-ts)
+endif()
+target_sources(${TARGET} PRIVATE
+    FILE_SET gpt4all_backend TYPE CXX_MODULES BASE_DIRS
+        "${CMAKE_CURRENT_BINARY_DIR}"
+    FILES
+        "${CMAKE_CURRENT_BINARY_DIR}/config.cppm"
+)
+gpt4all_add_warning_options(${TARGET})
+target_link_libraries(${TARGET} PRIVATE
+    gpt4all-backend
+)

gpt4all-backend-test/src/config.cppm.in

@@ -0,0 +1,10 @@
+module;
+
+#include <QString>
+
+export module gpt4all.test.config;
+
+using namespace Qt::Literals::StringLiterals;
+
+
+export inline QString OLLAMA_URL = u"@G4A_TEST_OLLAMA_URL@"_s;

gpt4all-backend-test/src/main.cpp

@@ -0,0 +1,14 @@
+#include <QLatin1StringView>
+
+import fmt;
+import gpt4all.backend.main;
+import gpt4all.test.config;
+
+using gpt4all::backend::LLMProvider;
+
+
+int main()
+{
+    LLMProvider provider(OLLAMA_URL);
+    fmt::print("Server version: {}", provider.getVersion());
+}

gpt4all-backend/CMakeLists.txt

@@ -1,189 +1,11 @@
-cmake_minimum_required(VERSION 3.23)  # for FILE_SET
+cmake_minimum_required(VERSION 3.29)
+project(gpt4all-backend VERSION 0.1 LANGUAGES CXX)
 
+set(CMAKE_CXX_STANDARD 23)  # make sure fmt module is compiled with the same C++ version as us
 include(../common/common.cmake)
 
-set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)
-set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+find_package(Qt6 6.8 COMPONENTS Concurrent Core Network REQUIRED)
 
-if (APPLE)
-    option(BUILD_UNIVERSAL "Build a Universal binary on macOS" ON)
-else()
-    option(LLMODEL_KOMPUTE "llmodel: use Kompute"              ON)
-    option(LLMODEL_VULKAN  "llmodel: use Vulkan"               OFF)
-    option(LLMODEL_CUDA    "llmodel: use CUDA"                 ON)
-    option(LLMODEL_ROCM    "llmodel: use ROCm"                 OFF)
-endif()
-
-if (APPLE)
-  if (BUILD_UNIVERSAL)
-    # Build a Universal binary on macOS
-    # This requires that the found Qt library is compiled as Universal binaries.
-    set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64" CACHE STRING "" FORCE)
-  else()
-    # Build for the host architecture on macOS
-    if (NOT CMAKE_OSX_ARCHITECTURES)
-      set(CMAKE_OSX_ARCHITECTURES "${CMAKE_HOST_SYSTEM_PROCESSOR}" CACHE STRING "" FORCE)
-    endif()
-  endif()
-endif()
-
-# Include the binary directory for the generated header file
-include_directories("${CMAKE_CURRENT_BINARY_DIR}")
-
-set(LLMODEL_VERSION_MAJOR 0)
-set(LLMODEL_VERSION_MINOR 5)
-set(LLMODEL_VERSION_PATCH 0)
-set(LLMODEL_VERSION "${LLMODEL_VERSION_MAJOR}.${LLMODEL_VERSION_MINOR}.${LLMODEL_VERSION_PATCH}")
-project(llmodel VERSION ${LLMODEL_VERSION} LANGUAGES CXX C)
-
-set(CMAKE_CXX_STANDARD 23)
-set(CMAKE_CXX_STANDARD_REQUIRED ON)
-set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})
-set(BUILD_SHARED_LIBS ON)
-
-# Check for IPO support
-include(CheckIPOSupported)
-check_ipo_supported(RESULT IPO_SUPPORTED OUTPUT IPO_ERROR)
-if (NOT IPO_SUPPORTED)
-    message(WARNING "Interprocedural optimization is not supported by your toolchain! This will lead to bigger file sizes and worse performance: ${IPO_ERROR}")
-else()
-    message(STATUS "Interprocedural optimization support detected")
-endif()
-
-set(DIRECTORY deps/llama.cpp-mainline)
-include(llama.cpp.cmake)
-
-set(BUILD_VARIANTS)
-if (APPLE)
-    list(APPEND BUILD_VARIANTS metal)
-endif()
-if (LLMODEL_KOMPUTE)
-    list(APPEND BUILD_VARIANTS kompute kompute-avxonly)
-else()
-    list(PREPEND BUILD_VARIANTS cpu cpu-avxonly)
-endif()
-if (LLMODEL_VULKAN)
-    list(APPEND BUILD_VARIANTS vulkan vulkan-avxonly)
-endif()
-if (LLMODEL_CUDA)
-    cmake_minimum_required(VERSION 3.18)  # for CMAKE_CUDA_ARCHITECTURES
-
-    # Defaults must be set before enable_language(CUDA).
-    # Keep this in sync with the arch list in ggml/src/CMakeLists.txt.
-    if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
-        # 52 == lowest CUDA 12 standard
-        # 60 == f16 CUDA intrinsics
-        # 61 == integer CUDA intrinsics
-        # 70 == compute capability at which unrolling a loop in mul_mat_q kernels is faster
-        if (GGML_CUDA_F16 OR GGML_CUDA_DMMV_F16)
-            set(CMAKE_CUDA_ARCHITECTURES "60;61;70;75") # needed for f16 CUDA intrinsics
-        else()
-            set(CMAKE_CUDA_ARCHITECTURES "52;61;70;75") # lowest CUDA 12 standard + lowest for integer intrinsics
-            #set(CMAKE_CUDA_ARCHITECTURES "OFF") # use this to compile much faster, but only F16 models work
-        endif()
-    endif()
-    message(STATUS "Using CUDA architectures: ${CMAKE_CUDA_ARCHITECTURES}")
-
-    include(CheckLanguage)
-    check_language(CUDA)
-    if (NOT CMAKE_CUDA_COMPILER)
-        message(WARNING "CUDA Toolkit not found. To build without CUDA, use -DLLMODEL_CUDA=OFF.")
-    endif()
-    enable_language(CUDA)
-    list(APPEND BUILD_VARIANTS cuda cuda-avxonly)
-endif()
-if (LLMODEL_ROCM)
-    enable_language(HIP)
-    list(APPEND BUILD_VARIANTS rocm rocm-avxonly)
-endif()
-
-# Go through each build variant
-foreach(BUILD_VARIANT IN LISTS BUILD_VARIANTS)
-    # Determine flags
-    if (BUILD_VARIANT MATCHES avxonly)
-        set(GPT4ALL_ALLOW_NON_AVX OFF)
-    else()
-        set(GPT4ALL_ALLOW_NON_AVX ON)
-    endif()
-    set(GGML_AVX2 ${GPT4ALL_ALLOW_NON_AVX})
-    set(GGML_F16C ${GPT4ALL_ALLOW_NON_AVX})
-    set(GGML_FMA  ${GPT4ALL_ALLOW_NON_AVX})
-
-    set(GGML_METAL   OFF)
-    set(GGML_KOMPUTE OFF)
-    set(GGML_VULKAN  OFF)
-    set(GGML_CUDA    OFF)
-    set(GGML_ROCM    OFF)
-    if (BUILD_VARIANT MATCHES metal)
-        set(GGML_METAL   ON)
-    elseif (BUILD_VARIANT MATCHES kompute)
-        set(GGML_KOMPUTE ON)
-    elseif (BUILD_VARIANT MATCHES vulkan)
-        set(GGML_VULKAN  ON)
-    elseif (BUILD_VARIANT MATCHES cuda)
-        set(GGML_CUDA    ON)
-    elseif (BUILD_VARIANT MATCHES rocm)
-        set(GGML_HIPBLAS ON)
-    endif()
-
-    # Include GGML
-    include_ggml(-mainline-${BUILD_VARIANT})
-
-    if (BUILD_VARIANT MATCHES metal)
-        set(GGML_METALLIB "${GGML_METALLIB}" PARENT_SCOPE)
-    endif()
-
-    # Function for preparing individual implementations
-    function(prepare_target TARGET_NAME BASE_LIB)
-        set(TARGET_NAME ${TARGET_NAME}-${BUILD_VARIANT})
-        message(STATUS "Configuring model implementation target ${TARGET_NAME}")
-        # Link to ggml/llama
-        target_link_libraries(${TARGET_NAME}
-            PRIVATE ${BASE_LIB}-${BUILD_VARIANT})
-        # Let it know about its build variant
-        target_compile_definitions(${TARGET_NAME}
-            PRIVATE GGML_BUILD_VARIANT="${BUILD_VARIANT}")
-        # Enable IPO if possible
-# FIXME: Doesn't work with msvc reliably. See https://github.com/nomic-ai/gpt4all/issues/841
-#        set_property(TARGET ${TARGET_NAME}
-#                     PROPERTY INTERPROCEDURAL_OPTIMIZATION ${IPO_SUPPORTED})
-    endfunction()
-
-    # Add each individual implementations
-    add_library(llamamodel-mainline-${BUILD_VARIANT} SHARED
-        src/llamamodel.cpp src/llmodel_shared.cpp)
-    gpt4all_add_warning_options(llamamodel-mainline-${BUILD_VARIANT})
-    target_compile_definitions(llamamodel-mainline-${BUILD_VARIANT} PRIVATE
-        LLAMA_VERSIONS=>=3 LLAMA_DATE=999999)
-    target_include_directories(llamamodel-mainline-${BUILD_VARIANT} PRIVATE
-        src include/gpt4all-backend
-    )
-    prepare_target(llamamodel-mainline llama-mainline)
-
-    if (NOT PROJECT_IS_TOP_LEVEL AND BUILD_VARIANT STREQUAL cuda)
-        set(CUDAToolkit_BIN_DIR ${CUDAToolkit_BIN_DIR} PARENT_SCOPE)
-    endif()
-endforeach()
-
-add_library(llmodel
-    src/dlhandle.cpp
-    src/llmodel.cpp
-    src/llmodel_c.cpp
-    src/llmodel_shared.cpp
-)
-gpt4all_add_warning_options(llmodel)
-target_sources(llmodel PUBLIC
-    FILE_SET public_headers TYPE HEADERS BASE_DIRS include
-    FILES include/gpt4all-backend/llmodel.h
-          include/gpt4all-backend/llmodel_c.h
-          include/gpt4all-backend/sysinfo.h
-)
-target_compile_definitions(llmodel PRIVATE LIB_FILE_EXT="${CMAKE_SHARED_LIBRARY_SUFFIX}")
-target_include_directories(llmodel PRIVATE src include/gpt4all-backend)
-
-set_target_properties(llmodel PROPERTIES
-                              VERSION ${PROJECT_VERSION}
-                              SOVERSION ${PROJECT_VERSION_MAJOR})
-
-set(COMPONENT_NAME_MAIN ${PROJECT_NAME})
-set(CMAKE_INSTALL_PREFIX ${CMAKE_BINARY_DIR}/install)
+set(FMT_MODULE ON)
+add_subdirectory(../deps "${CMAKE_CURRENT_BINARY_DIR}/common_deps")
+add_subdirectory(src)

gpt4all-backend/src/CMakeLists.txt

@@ -0,0 +1,20 @@
+set(TARGET gpt4all-backend)
+
+add_library(${TARGET} STATIC
+    main.cpp
+)
+target_compile_features(${TARGET} PUBLIC cxx_std_23)
+if (CMAKE_COMPILER_IS_GNUCXX)
+    target_compile_options(${TARGET} PUBLIC -fmodules-ts)
+endif()
+target_sources(${TARGET} PUBLIC
+    FILE_SET gpt4all_backend TYPE CXX_MODULES FILES
+    main.cppm
+)
+gpt4all_add_warning_options(${TARGET})
+target_link_libraries(${TARGET} PUBLIC
+    QCoro6::Coro Qt6::Core Qt6::Network
+)
+target_link_libraries(${TARGET} PRIVATE
+    QCoro6::Network fmt::fmt
+)

gpt4all-backend/src/main.cpp

@@ -0,0 +1,33 @@
+module;
+
+#include <expected>
+#include <memory>
+
+#include <QFuture>
+#include <QNetworkAccessManager>
+#include <QNetworkReply>
+#include <QNetworkRequest>
+#include <QByteArray>
+#include <QString>
+
+#include <QCoro/QCoroNetworkReply>
+
+using namespace Qt::Literals::StringLiterals;
+
+module gpt4all.backend.main;
+
+
+namespace gpt4all::backend {
+
+auto LLMProvider::getVersion() const -> QCoro::Task<DataOrNetErr<QString>>
+{
+    QNetworkAccessManager nam;
+    std::unique_ptr<QNetworkReply> reply(co_await nam.get(QNetworkRequest(m_baseUrl.resolved(u"/api/version"_s))));
+    if (auto err = reply->error())
+        co_return std::unexpected(err);
+
+    // TODO(jared): parse JSON here instead of just returning the data
+    co_return QString::fromUtf8(reply->readAll());
+}
+
+} // namespace gpt4all::backend

gpt4all-backend/src/main.cppm

@@ -0,0 +1,38 @@
+module;
+
+#include <expected>
+
+#include <QNetworkReply>
+#include <QUrl>
+
+#include <QCoro/QCoroTask>
+
+class QString;
+template <typename T> class QFuture;
+
+export module gpt4all.backend.main;
+
+
+export namespace gpt4all::backend {
+
+template <typename T>
+using DataOrNetErr = std::expected<T, QNetworkReply::NetworkError>;
+
+
+class LLMProvider {
+public:
+    LLMProvider(QUrl baseUrl)
+        : m_baseUrl(baseUrl)
+        {}
+
+    const QUrl &baseUrl() const { return m_baseUrl; }
+    void getBaseUrl(QUrl value) { m_baseUrl = std::move(value); }
+
+    /// Retrieve the Ollama version, e.g. "0.5.1"
+    auto getVersion() const -> QCoro::Task<DataOrNetErr<QString>>;
+
+private:
+    QUrl m_baseUrl;
+};
+
+} // namespace gpt4all::backend

gpt4all-bindings/README.md

@@ -1,21 +0,0 @@
-# GPT4All Language Bindings
-These are the language bindings for the GPT4All backend. They provide functionality to load GPT4All models (and other llama.cpp models), generate text, and (in the case of the Python bindings) embed text as a vector representation.
-
-See their respective folders for language-specific documentation.
-
-### Languages
-- [Python](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-bindings/python) (Nomic official, maintained by [@cebtenzzre](https://github.com/cebtenzzre))
-- [Node.js/Typescript](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-bindings/typescript) (community, maintained by [@jacoobes](https://github.com/jacoobes) and [@iimez](https://github.com/iimez))
-
-<br/>
-<br/>
-
-<details><summary><b>Archived Bindings</b></summary>
-<br/>
-
-The following bindings have been removed from this repository due to lack of maintenance. If adopted, they can be brought back&mdash;feel free to message a developer on Dicsord if you are interested in maintaining one of them. Below are links to their last available version (not necessarily the last working version).
-- C#: [41c9013f](https://github.com/nomic-ai/gpt4all/tree/41c9013fa46a194b3e4fee6ced1b9d1b65e177ac/gpt4all-bindings/csharp)
-- Java: [41c9013f](https://github.com/nomic-ai/gpt4all/tree/41c9013fa46a194b3e4fee6ced1b9d1b65e177ac/gpt4all-bindings/java)
-- Go: [41c9013f](https://github.com/nomic-ai/gpt4all/tree/41c9013fa46a194b3e4fee6ced1b9d1b65e177ac/gpt4all-bindings/golang)
-
-</details>

gpt4all-bindings/cli/README.md

@@ -1,43 +0,0 @@
-# GPT4All Command-Line Interface (CLI)
-
-GPT4All on the command-line.
-
-More details on the [wiki](https://github.com/nomic-ai/gpt4all/wiki/Python-CLI).
-
-## Quickstart
-
-The CLI is based on the `gpt4all` Python bindings and the `typer` package.
-
-The following shows one way to get started with the CLI, the documentation has more information.
-Typically, you will want to replace `python` with `python3` on _Unix-like_ systems and `py -3` on
-_Windows_. Also, it's assumed you have all the necessary Python components already installed.
-
-The CLI is a self-contained Python script named [app.py] ([download][app.py-download]). As long as
-its package dependencies are present, you can download and run it from wherever you like.
-
-[app.py]: https://github.com/nomic-ai/gpt4all/blob/main/gpt4all-bindings/cli/app.py
-[app.py-download]: https://raw.githubusercontent.com/nomic-ai/gpt4all/main/gpt4all-bindings/cli/app.py
-
-```shell
-# optional but recommended: create and use a virtual environment
-python -m venv gpt4all-cli
-```
-_Windows_ and _Unix-like_ systems differ slightly in how you activate a _virtual environment_:
-- _Unix-like_, typically: `. gpt4all-cli/bin/activate`
-- _Windows_: `gpt4all-cli\Scripts\activate`
-
-Then:
-```shell
-# pip-install the necessary packages; omit '--user' if using a virtual environment
-python -m pip install --user --upgrade gpt4all typer
-# run the CLI
-python app.py repl
-```
-By default, it will automatically download the `Mistral Instruct` model to `.cache/gpt4all/` in your
-user directory, if necessary.
-
-If you have already saved a model beforehand, specify its path with the `-m`/`--model` argument,
-for example:
-```shell
-python app.py repl --model /home/user/my-gpt4all-models/mistral-7b-instruct-v0.1.Q4_0.gguf
-```

gpt4all-bindings/cli/app.py

@@ -1,184 +0,0 @@
-#!/usr/bin/env python3
-"""GPT4All CLI
-
-The GPT4All CLI is a self-contained script based on the `gpt4all` and `typer` packages. It offers a
-REPL to communicate with a language model similar to the chat GUI application, but more basic.
-"""
-
-import importlib.metadata
-import io
-import sys
-from collections import namedtuple
-from typing_extensions import Annotated
-
-import typer
-from gpt4all import GPT4All
-
-
-MESSAGES = [
-    {"role": "system", "content": "You are a helpful assistant."},
-    {"role": "user", "content": "Hello there."},
-    {"role": "assistant", "content": "Hi, how can I help you?"},
-]
-
-SPECIAL_COMMANDS = {
-    "/reset": lambda messages: messages.clear(),
-    "/exit": lambda _: sys.exit(),
-    "/clear": lambda _: print("\n" * 100),
-    "/help": lambda _: print("Special commands: /reset, /exit, /help and /clear"),
-}
-
-VersionInfo = namedtuple('VersionInfo', ['major', 'minor', 'micro'])
-VERSION_INFO = VersionInfo(1, 0, 2)
-VERSION = '.'.join(map(str, VERSION_INFO))  # convert to string form, like: '1.2.3'
-
-CLI_START_MESSAGE = f"""
-    
- ██████  ██████  ████████ ██   ██  █████  ██      ██      
-██       ██   ██    ██    ██   ██ ██   ██ ██      ██      
-██   ███ ██████     ██    ███████ ███████ ██      ██      
-██    ██ ██         ██         ██ ██   ██ ██      ██      
- ██████  ██         ██         ██ ██   ██ ███████ ███████ 
-                                                          
-
-Welcome to the GPT4All CLI! Version {VERSION}
-Type /help for special commands.
-                                                    
-"""
-
-# create typer app
-app = typer.Typer()
-
-@app.command()
-def repl(
-    model: Annotated[
-        str,
-        typer.Option("--model", "-m", help="Model to use for chatbot"),
-    ] = "mistral-7b-instruct-v0.1.Q4_0.gguf",
-    n_threads: Annotated[
-        int,
-        typer.Option("--n-threads", "-t", help="Number of threads to use for chatbot"),
-    ] = None,
-    device: Annotated[
-        str,
-        typer.Option("--device", "-d", help="Device to use for chatbot, e.g. gpu, amd, nvidia, intel. Defaults to CPU."),
-    ] = None,
-):
-    """The CLI read-eval-print loop."""
-    gpt4all_instance = GPT4All(model, device=device)
-
-    # if threads are passed, set them
-    if n_threads is not None:
-        num_threads = gpt4all_instance.model.thread_count()
-        print(f"\nAdjusted: {num_threads} →", end="")
-
-        # set number of threads
-        gpt4all_instance.model.set_thread_count(n_threads)
-
-        num_threads = gpt4all_instance.model.thread_count()
-        print(f" {num_threads} threads", end="", flush=True)
-    else:
-        print(f"\nUsing {gpt4all_instance.model.thread_count()} threads", end="")
-
-    print(CLI_START_MESSAGE)
-
-    use_new_loop = False
-    try:
-        version = importlib.metadata.version('gpt4all')
-        version_major = int(version.split('.')[0])
-        if version_major >= 1:
-            use_new_loop = True
-    except:
-        pass  # fall back to old loop
-    if use_new_loop:
-        _new_loop(gpt4all_instance)
-    else:
-        _old_loop(gpt4all_instance)
-
-
-def _old_loop(gpt4all_instance):
-    while True:
-        message = input(" ⇢  ")
-
-        # Check if special command and take action
-        if message in SPECIAL_COMMANDS:
-            SPECIAL_COMMANDS[message](MESSAGES)
-            continue
-
-        # if regular message, append to messages
-        MESSAGES.append({"role": "user", "content": message})
-
-        # execute chat completion and ignore the full response since 
-        # we are outputting it incrementally
-        full_response = gpt4all_instance.chat_completion(
-            MESSAGES,
-            # preferential kwargs for chat ux
-            n_past=0,
-            n_predict=200,
-            top_k=40,
-            top_p=0.9,
-            min_p=0.0,
-            temp=0.9,
-            n_batch=9,
-            repeat_penalty=1.1,
-            repeat_last_n=64,
-            context_erase=0.0,
-            # required kwargs for cli ux (incremental response)
-            verbose=False,
-            streaming=True,
-        )
-        # record assistant's response to messages
-        MESSAGES.append(full_response.get("choices")[0].get("message"))
-        print() # newline before next prompt
-
-
-def _new_loop(gpt4all_instance):
-    with gpt4all_instance.chat_session():
-        while True:
-            message = input(" ⇢  ")
-
-            # Check if special command and take action
-            if message in SPECIAL_COMMANDS:
-                SPECIAL_COMMANDS[message](MESSAGES)
-                continue
-
-            # if regular message, append to messages
-            MESSAGES.append({"role": "user", "content": message})
-
-            # execute chat completion and ignore the full response since 
-            # we are outputting it incrementally
-            response_generator = gpt4all_instance.generate(
-                message,
-                # preferential kwargs for chat ux
-                max_tokens=200,
-                temp=0.9,
-                top_k=40,
-                top_p=0.9,
-                min_p=0.0,
-                repeat_penalty=1.1,
-                repeat_last_n=64,
-                n_batch=9,
-                # required kwargs for cli ux (incremental response)
-                streaming=True,
-            )
-            response = io.StringIO()
-            for token in response_generator:
-                print(token, end='', flush=True)
-                response.write(token)
-
-            # record assistant's response to messages
-            response_message = {'role': 'assistant', 'content': response.getvalue()}
-            response.close()
-            gpt4all_instance.current_chat_session.append(response_message)
-            MESSAGES.append(response_message)
-            print() # newline before next prompt
-
-
-@app.command()
-def version():
-    """The CLI version command."""
-    print(f"gpt4all-cli v{VERSION}")
-
-
-if __name__ == "__main__":
-    app()

gpt4all-bindings/cli/developer_notes.md

@@ -1,25 +0,0 @@
-# Developing the CLI
-## Documentation
-Documentation can be found in three places:
-- `app.py` docstrings & comments
-- a Readme: `gpt4all-bindings/cli/README.md`
-- the actual CLI documentation: `gpt4all-bindings/python/docs/gpt4all_cli.md`
-
-The _docstrings_ are meant for programmatic use. Since the CLI is primarily geared towards users and
-not to build on top, they're kept terse.
-
-The _Readme_ is mostly meant for users and includes:
-- a link to the _CLI documentation_ (on the [website])
-- a Quickstart section with some guidance on how to get started with a sane setup
-
-The _CLI documentation_ and other documentation are located in the above mentioned `docs/` folder.
-They're in Markdown format and built for the [website]. Of the three, they should be the most
-detailed.
-
-[website]: https://docs.gpt4all.io/gpt4all_cli.html
-
-
-## Versioning
-The version number should now follow the `gpt4all` PyPI package, so compatibility is more clear.
-
-The one place to change it is the `namedtuple` called `VERSION_INFO`.