typescript: fix final bugs and polishing, circle ci documentation (#960)

* fix: esm and cjs compatibility Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * Update prebuild.js Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * fix gpt4all.js Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * Fix compile for windows and linux again. PLEASE DON'T REVERT THISgit gui! * version bump * polish up spec and build scripts * lock file refresh * fix: proper resource closing and error handling * check make sure libPath not null * add msvc build script and update readme requirements * python workflows in circleci * dummy python change * no need for main * second hold for pypi deploy * let me deploy pls * bring back when condition * Typo, ignore list (#967) Fix typo in javadoc, Add word to ignore list for codespellrc --------- Co-authored-by: felix <felix@zaslavskiy.net> * llmodel: change tokenToString to not use string_view (#968) fixes a definite use-after-free and likely avoids some other potential ones - std::string will convert to a std::string_view automatically but as soon as the std::string in question goes out of scope it is already freed and the string_view is pointing at freed memory - this is *mostly* fine if its returning a reference to the tokenizer's internal vocab table but it's, imo, too easy to return a reference to a dynamically constructed string with this as replit is doing (and unfortunately needs to do to convert the internal whitespace replacement symbol back to a space) * Initial Library Loader for .NET Bindings / Update bindings to support newest changes (#763) * Initial Library Loader * Load library as part of Model factory * Dynamically search and find the dlls * Update tests to use locally built runtimes * Fix dylib loading, add macos runtime support for sample/tests * Bypass automatic loading by default. * Only set CMAKE_OSX_ARCHITECTURES if not already set, allow cross-compile * Switch Loading again * Update build scripts for mac/linux * Update bindings to support newest breaking changes * Fix build * Use llmodel for Windows * Actually, it does need to be libllmodel * Name * Remove TFMs, bypass loading by default * Fix script * Delete mac script --------- Co-authored-by: Tim Miller <innerlogic4321@ghmail.com> * bump llama.cpp mainline to latest (#964) * fix prompt context so it's preserved in class * update setup.py * metal replit (#931) metal+replit makes replit work with Metal and removes its use of `mem_per_token` in favor of fixed size scratch buffers (closer to llama.cpp) * update documentation scripts and generation to include readme.md * update readme and documentation for source * begin tests, import jest, fix listModels export * fix typo * chore: update spec * fix: finally, reduced potential of empty string * chore: add stub for createTokenSream * refactor: protecting resources properly * add basic jest tests * update * update readme * refactor: namespace the res variable * circleci integration to automatically build docs * add starter docs * typo * more circle ci typo * forgot to add nodejs circle ci orb * fix circle ci * feat: @iimez verify download and fix prebuild script * fix: oops, option name wrong * fix: gpt4all utils not emitting docs * chore: fix up scripts * fix: update docs and typings for md5 sum * fix: macos compilation * some refactoring * Update index.cc Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * update readme and enable exceptions on mac * circle ci progress * basic embedding with sbert (not tested & cpp side only) * fix circle ci * fix circle ci * update circle ci script * bruh * fix again * fix * fixed required workflows * fix ci * fix pwd * fix pwd * update ci * revert * fix * prevent rebuild * revmove noop * Update continue_config.yml Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * Update binding.gyp Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * fix fs not found * remove cpp 20 standard * fix warnings, safer way to calculate arrsize * readd build backend * basic embeddings and yarn test" * fix circle ci Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> Update continue_config.yml Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> fix macos paths update readme and roadmap split up spec update readme check for url in modelsjson update docs and inline stuff update yarn configuration and readme update readme readd npm publish script add exceptions bruh one space broke the yaml codespell oops forgot to add runtimes folder bump version try code snippet https://support.circleci.com/hc/en-us/articles/8325075309339-How-to-install-NPM-on-Windows-images add fallback for unknown architectures attached to wrong workspace hopefuly fix moving everything under backend to persist should work now * update circle ci script * prevent rebuild * revmove noop * Update continue_config.yml Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * Update binding.gyp Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> * fix fs not found * remove cpp 20 standard * fix warnings, safer way to calculate arrsize * readd build backend * basic embeddings and yarn test" * fix circle ci Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> Update continue_config.yml Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> fix macos paths update readme and roadmap split up spec update readme check for url in modelsjson update docs and inline stuff update yarn configuration and readme update readme readd npm publish script add exceptions bruh one space broke the yaml codespell oops forgot to add runtimes folder bump version try code snippet https://support.circleci.com/hc/en-us/articles/8325075309339-How-to-install-NPM-on-Windows-images add fallback for unknown architectures attached to wrong workspace hopefuly fix moving everything under backend to persist should work now * Update README.md Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> --------- Signed-off-by: Jacob Nguyen <76754747+jacoobes@users.noreply.github.com> Co-authored-by: Adam Treat <treat.adam@gmail.com> Co-authored-by: Richard Guo <richardg7890@gmail.com> Co-authored-by: Felix Zaslavskiy <felix.zaslavskiy@gmail.com> Co-authored-by: felix <felix@zaslavskiy.net> Co-authored-by: Aaron Miller <apage43@ninjawhale.com> Co-authored-by: Tim Miller <drasticactions@users.noreply.github.com> Co-authored-by: Tim Miller <innerlogic4321@ghmail.com>
2025-11-03 15:37:43 +00:00 · 2023-07-25 10:46:40 -05:00
parent b3f84c56e7
commit 545c23b4bd
30 changed files with 8091 additions and 3205 deletions
--- a/gpt4all-bindings/typescript/src/gpt4all.d.ts
+++ b/gpt4all-bindings/typescript/src/gpt4all.d.ts
@@ -1,7 +1,6 @@
 /// <reference types="node" />
 declare module "gpt4all";

-export * from "./util.d.ts";

 /** Type of the model */
 type ModelType = "gptj" | "llama" | "mpt" | "replit";
@@ -61,7 +60,7 @@ declare class LLModel {
    type(): ModelType | undefined;

    /** The name of the model. */
-    name(): ModelFile;
+    name(): string;

    /**
     * Get the size of the internal state of the model.
@@ -85,7 +84,7 @@ declare class LLModel {

    /**
     * Prompt the model with a given input and optional parameters.
-     * This is the raw output from std out.
+     * This is the raw output from model.
     * Use the prompt function exported for a value
     * @param q The prompt input.
     * @param params Optional parameters for the prompt context.
@@ -93,6 +92,15 @@ declare class LLModel {
     */
    raw_prompt(q: string, params: Partial<LLModelPromptContext>, callback: (res: string) => void): void; // TODO work on return type

+    /**
+     * Embed text with the model. Keep in mind that 
+     * not all models can embed text, (only bert can embed as of 07/16/2023 (mm/dd/yyyy))
+     * Use the prompt function exported for a value
+     * @param q The prompt input.
+     * @param params Optional parameters for the prompt context.
+     * @returns The result of the model prompt.
+     */
+    embed(text: string) : Float32Array
    /**
     * Whether the model is loaded or not.
     */
@@ -115,11 +123,20 @@ interface LoadModelOptions {
    verbose?: boolean;
 }

+/**
+ * Loads a machine learning model with the specified name. The defacto way to create a model.
+ * By default this will download a model from the official GPT4ALL website, if a model is not present at given path.
+ *
+ * @param {string} modelName - The name of the model to load.
+ * @param {LoadModelOptions|undefined} [options] - (Optional) Additional options for loading the model.
+ * @returns {Promise<LLModel>} A promise that resolves to an instance of the loaded LLModel.
+ */
 declare function loadModel(
    modelName: string,
    options?: LoadModelOptions
 ): Promise<LLModel>;

+
 /**
 * The nodejs equivalent to python binding's chat_completion
 * @param {LLModel} llmodel - The language model object.
@@ -144,6 +161,19 @@ declare function createCompletion(
    options?: CompletionOptions
 ): Promise<CompletionReturn>;

+
+/**
+ * The nodejs moral equivalent to python binding's Embed4All().embed()
+ * meow
+ * @param {LLModel} llmodel - The language model object.
+ * @param {string} text - text to embed
+ * @returns {Float32Array} The completion result.
+ */
+declare function createEmbedding(
+    llmodel: LLModel,
+    text: string,
+): Float32Array
+
 /**
 * The options for creating the completion.
 */
@@ -294,6 +324,77 @@ interface PromptMessage {
    role: "system" | "assistant" | "user";
    content: string;
 }
+
+/**
+ * Initiates the download of a model file of a specific model type.
+ * By default this downloads without waiting. use the controller returned to alter this behavior.
+ * @param {ModelFile} modelName - The model file to be downloaded.
+ * @param {DownloadOptions} options - to pass into the downloader. Default is { location: (cwd), debug: false }.
+ * @returns {DownloadController} object that allows controlling the download process.
+ *
+ * @throws {Error} If the model already exists in the specified location.
+ * @throws {Error} If the model cannot be found at the specified url.
+ *
+ * @example
+ * const controller = download('ggml-gpt4all-j-v1.3-groovy.bin')
+ * controller.promise().then(() => console.log('Downloaded!'))
+ */
+declare function downloadModel(
+    modelName: string,
+    options?: DownloadModelOptions
+): DownloadController;
+
+/**
+ * Options for the model download process.
+ */
+interface DownloadModelOptions {
+    /**
+     * location to download the model.
+     * Default is process.cwd(), or the current working directory
+     */
+    modelPath?: string;
+
+    /**
+     * Debug mode -- check how long it took to download in seconds
+     * @default false
+     */
+    debug?: boolean;
+
+    /**
+     * Remote download url. Defaults to `https://gpt4all.io/models`
+     * @default https://gpt4all.io/models
+     */
+    url?: string;
+    /**
+     * Whether to verify the hash of the download to ensure a proper download occurred.
+     * @default true
+     */
+    md5sum?: boolean;
+}
+
+declare function listModels(): Promise<Record<string, string>[]>;
+
+interface RetrieveModelOptions {
+    allowDownload?: boolean;
+    verbose?: boolean;
+    modelPath?: string;
+}
+
+declare function retrieveModel(
+    model: string,
+    options?: RetrieveModelOptions
+): Promise<string>;
+
+/**
+ * Model download controller.
+ */
+interface DownloadController {
+    /** Cancel the request to download from gpt4all website if this is called. */
+    cancel: () => void;
+    /** Convert the downloader into a promise, allowing people to await and manage its lifetime */
+    promise: () => Promise<void>;
+}
+
 export {
    ModelType,
    ModelFile,
@@ -304,7 +405,14 @@ export {
    LoadModelOptions,
    loadModel,
    createCompletion,
+    createEmbedding,
    createTokenStream,
    DEFAULT_DIRECTORY,
    DEFAULT_LIBRARIES_DIRECTORY,
+    downloadModel,
+    retrieveModel,
+    listModels,
+    DownloadController,
+    RetrieveModelOptions,
+    DownloadModelOptions
 };
--- a/gpt4all-bindings/typescript/src/gpt4all.js
+++ b/gpt4all-bindings/typescript/src/gpt4all.js
@@ -10,12 +10,12 @@ const {
    downloadModel,
    appendBinSuffixIfMissing,
 } = require("./util.js");
-const config = require("./config.js");
+const { DEFAULT_DIRECTORY, DEFAULT_LIBRARIES_DIRECTORY } = require("./config.js");

 async function loadModel(modelName, options = {}) {
    const loadOptions = {
-        modelPath: config.DEFAULT_DIRECTORY,
-        librariesPath: config.DEFAULT_LIBRARIES_DIRECTORY,
+        modelPath: DEFAULT_DIRECTORY,
+        librariesPath: DEFAULT_LIBRARIES_DIRECTORY,
        allowDownload: true,
        verbose: true,
        ...options,
@@ -37,7 +37,9 @@ async function loadModel(modelName, options = {}) {
            break;
        }
    }
-
+    if(!libPath) {
+        throw Error("Could not find a valid path from " + libSearchPaths);
+    }
    const llmOptions = {
        model_name: appendBinSuffixIfMissing(modelName),
        model_path: loadOptions.modelPath,
@@ -53,38 +55,40 @@ async function loadModel(modelName, options = {}) {
 }

 function createPrompt(messages, hasDefaultHeader, hasDefaultFooter) {
-    let fullPrompt = "";
+    let fullPrompt = [];

    for (const message of messages) {
        if (message.role === "system") {
-            const systemMessage = message.content + "\n";
-            fullPrompt += systemMessage;
+            const systemMessage = message.content;
+            fullPrompt.push(systemMessage);
        }
    }
    if (hasDefaultHeader) {
-        fullPrompt += `### Instruction: 
-        The prompt below is a question to answer, a task to complete, or a conversation 
-        to respond to; decide which and write an appropriate response.
-        \n### Prompt: 
-        `;
+        fullPrompt.push(`### Instruction: The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.`);
    }
+    let prompt = "### Prompt:";
    for (const message of messages) {
        if (message.role === "user") {
-            const user_message = "\n" + message["content"];
-            fullPrompt += user_message;
+            const user_message = message["content"];
+            prompt += user_message;
        }
        if (message["role"] == "assistant") {
-            const assistant_message = "\nResponse: " + message["content"];
-            fullPrompt += assistant_message;
+            const assistant_message = "Response:" + message["content"];
+            prompt += assistant_message;
        }
    }
+    fullPrompt.push(prompt);
    if (hasDefaultFooter) {
-        fullPrompt += "\n### Response:";
+        fullPrompt.push("### Response:");
    }

-    return fullPrompt;
+    return fullPrompt.join('\n');
 }

+
+function createEmbedding(llmodel, text) {
+    return llmodel.embed(text)
+}
 async function createCompletion(
    llmodel,
    messages,
@@ -98,16 +102,12 @@ async function createCompletion(
    const fullPrompt = createPrompt(
        messages,
        options.hasDefaultHeader ?? true,
-        options.hasDefaultFooter
+        options.hasDefaultFooter ?? true
    );
    if (options.verbose) {
        console.log("Sent: " + fullPrompt);
    }
-    const promisifiedRawPrompt = new Promise((resolve, rej) => {
-        llmodel.raw_prompt(fullPrompt, options, (s) => {
-            resolve(s);
-        });
-    });
+    const promisifiedRawPrompt = llmodel.raw_prompt(fullPrompt, options, (s) => {});
    return promisifiedRawPrompt.then((response) => {
        return {
            llmodel: llmodel.name(),
@@ -128,11 +128,18 @@ async function createCompletion(
    });
 }

+function createTokenStream() {
+    throw Error("This API has not been completed yet!")
+}
+
 module.exports = {
-    ...config,
+    DEFAULT_LIBRARIES_DIRECTORY,
+    DEFAULT_DIRECTORY,
    LLModel,
    createCompletion,
+    createEmbedding,
    downloadModel,
    retrieveModel,
    loadModel,
+    createTokenStream
 };
--- a/gpt4all-bindings/typescript/src/util.d.ts
+++ b/gpt4all-bindings/typescript/src/util.d.ts
@@ -1,69 +0,0 @@
-/// <reference types="node" />
-declare module "gpt4all";
-
-/**
- * Initiates the download of a model file of a specific model type.
- * By default this downloads without waiting. use the controller returned to alter this behavior.
- * @param {ModelFile} model - The model file to be downloaded.
- * @param {DownloadOptions} options - to pass into the downloader. Default is { location: (cwd), debug: false }.
- * @returns {DownloadController} object that allows controlling the download process.
- *
- * @throws {Error} If the model already exists in the specified location.
- * @throws {Error} If the model cannot be found at the specified url.
- *
- * @example
- * const controller = download('ggml-gpt4all-j-v1.3-groovy.bin')
- * controller.promise().then(() => console.log('Downloaded!'))
- */
-declare function downloadModel(
-    modelName: string,
-    options?: DownloadModelOptions
-): DownloadController;
-
-/**
- * Options for the model download process.
- */
-export interface DownloadModelOptions {
-    /**
-     * location to download the model.
-     * Default is process.cwd(), or the current working directory
-     */
-    modelPath?: string;
-
-    /**
-     * Debug mode -- check how long it took to download in seconds
-     * @default false
-     */
-    debug?: boolean;
-
-    /**
-     * Remote download url. Defaults to `https://gpt4all.io/models`
-     * @default https://gpt4all.io/models
-     */
-    url?: string;
-}
-
-declare function listModels(): Promise<Record<string, string>[]>;
-
-interface RetrieveModelOptions {
-    allowDownload?: boolean;
-    verbose?: boolean;
-    modelPath?: string;
-}
-
-declare async function retrieveModel(
-    model: string,
-    options?: RetrieveModelOptions
-): Promise<string>;
-
-/**
- * Model download controller.
- */
-interface DownloadController {
-    /** Cancel the request to download from gpt4all website if this is called. */
-    cancel: () => void;
-    /** Convert the downloader into a promise, allowing people to await and manage its lifetime */
-    promise: () => Promise<void>;
-}
-
-export { downloadModel, DownloadModelOptions, DownloadController, listModels, retrieveModel, RetrieveModelOptions };
--- a/gpt4all-bindings/typescript/src/util.js
+++ b/gpt4all-bindings/typescript/src/util.js
@@ -1,9 +1,10 @@
-const { createWriteStream, existsSync } = require("fs");
+const { createWriteStream, existsSync, statSync } = require("node:fs");
+const fsp = require('node:fs/promises')
 const { performance } = require("node:perf_hooks");
 const path = require("node:path");
 const {mkdirp} = require("mkdirp");
 const { DEFAULT_DIRECTORY, DEFAULT_LIBRARIES_DIRECTORY } = require("./config.js");
-
+const md5File = require('md5-file');
 async function listModels() {
    const res = await fetch("https://gpt4all.io/models/models.json");
    const modelList = await res.json();
@@ -31,62 +32,108 @@ function readChunks(reader) {
    };
 }

-function downloadModel(
-    modelName,
-    options = {}
-) {
+function downloadModel(modelName, options = {}) {
    const downloadOptions = {
        modelPath: DEFAULT_DIRECTORY,
        debug: false,
-        url: "https://gpt4all.io/models",
+        md5sum: true,
        ...options,
    };

    const modelFileName = appendBinSuffixIfMissing(modelName);
-    const fullModelPath = path.join(downloadOptions.modelPath, modelFileName);
-    const modelUrl = `${downloadOptions.url}/${modelFileName}`
+    const partialModelPath = path.join(
+        downloadOptions.modelPath,
+        modelName + ".part"
+    );
+    const finalModelPath = path.join(downloadOptions.modelPath, modelFileName);
+    const modelUrl = downloadOptions.url ?? `https://gpt4all.io/models/${modelFileName}`;

-    if (existsSync(fullModelPath)) {
-        throw Error(`Model already exists at ${fullModelPath}`);
+    if (existsSync(finalModelPath)) {
+        throw Error(`Model already exists at ${finalModelPath}`);
+    }
+
+    const headers = {
+        "Accept-Ranges": "arraybuffer",
+        "Response-Type": "arraybuffer",
+    };
+
+    const writeStreamOpts = {};
+
+    if (existsSync(partialModelPath)) {
+        console.log("Partial model exists, resuming download...");
+        const startRange = statSync(partialModelPath).size;
+        headers["Range"] = `bytes=${startRange}-`;
+        writeStreamOpts.flags = "a";
    }

    const abortController = new AbortController();
    const signal = abortController.signal;

-    //wrapper function to get the readable stream from request
-    // const baseUrl = options.url ?? "https://gpt4all.io/models";
-    const fetchModel = () =>
+    // wrapper function to get the readable stream from request
+    const fetchModel = (fetchOpts = {}) =>
        fetch(modelUrl, {
            signal,
+            ...fetchOpts,
        }).then((res) => {
            if (!res.ok) {
-                throw Error(`Failed to download model from ${modelUrl} - ${res.statusText}`);
+                throw Error(
+                    `Failed to download model from ${modelUrl} - ${res.statusText}`
+                );
            }
            return res.body.getReader();
        });

-    //a promise that executes and writes to a stream. Resolves when done writing.
+    // a promise that executes and writes to a stream. Resolves when done writing.
    const res = new Promise((resolve, reject) => {
-        fetchModel()
-            //Resolves an array of a reader and writestream.
-            .then((reader) => [reader, createWriteStream(fullModelPath)])
+        fetchModel({ headers })
+            // Resolves an array of a reader and writestream.
+            .then((reader) => [
+                reader,
+                createWriteStream(partialModelPath, writeStreamOpts),
+            ])
            .then(async ([readable, wstream]) => {
-                console.log("Downloading @ ", fullModelPath);
+                console.log("Downloading @ ", partialModelPath);
                let perf;
+
                if (options.debug) {
                    perf = performance.now();
                }
+
+                wstream.on("finish", () => {
+                    if (options.debug) {
+                        console.log(
+                            "Time taken: ",
+                            (performance.now() - perf).toFixed(2),
+                            " ms"
+                        );
+                    }
+                    wstream.close();
+                });
+
+                wstream.on("error", (e) => {
+                    wstream.close();
+                    reject(e);
+                });
+
                for await (const chunk of readChunks(readable)) {
                    wstream.write(chunk);
                }
-                if (options.debug) {
-                    console.log(
-                        "Time taken: ",
-                        (performance.now() - perf).toFixed(2),
-                        " ms"
-                    );
+
+                if (options.md5sum) {
+                    const fileHash = await md5File(partialModelPath);
+                    if (fileHash !== options.md5sum) {
+                        await fsp.unlink(partialModelPath);
+                        return reject(
+                            Error(`Model "${modelName}" failed verification: Hashes mismatch`)
+                        );
+                    }
+                    if (options.debug) {
+                        console.log("MD5 hash verified: ", fileHash);
+                    }
                }
-                resolve(fullModelPath);
+
+                await fsp.rename(partialModelPath, finalModelPath);
+                resolve(finalModelPath);
            })
            .catch(reject);
    });
@@ -95,7 +142,7 @@ function downloadModel(
        cancel: () => abortController.abort(),
        promise: () => res,
    };
-};
+}

 async function retrieveModel (
    modelName,
@@ -123,12 +170,13 @@ async function retrieveModel (
    }

    const availableModels = await listModels();
+    
    const foundModel = availableModels.find((model) => model.filename === modelFileName);

    if (!foundModel) {
        throw Error(`Model "${modelName}" is not available.`);
    }
-
+    //todo  
    if (retrieveOptions.verbose) {
        console.log(`Downloading ${modelName}...`);
    }
@@ -136,6 +184,7 @@ async function retrieveModel (
    const downloadController = downloadModel(modelName, {
        modelPath: retrieveOptions.modelPath,
        debug: retrieveOptions.verbose,
+        url: foundModel.url
    });

    const downloadPath = await downloadController.promise();
@@ -153,4 +202,5 @@ module.exports = {
    appendBinSuffixIfMissing,
    downloadModel,
    retrieveModel,
-};
+    listModels
+};