github/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2025-09-06 02:50:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

chatllm: do not attempt to serialize incompatible state (#1742)

Browse Source
This commit is contained in:
Jared Van Bortel
2023-12-12 11:45:03 -05:00
committed by GitHub
parent 9481762802
commit 0600f551b3
3 changed files with 18 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
gpt4all-backend/llamamodel.cpp
View File

@@ -163,6 +163,10 @@ bool LLamaModel::loadModel(const std::string &modelPath)
d_ptr->ctx_params.seed = params.seed;
d_ptr->ctx_params.f16_kv = params.memory_f16;
// The new batch API provides space for n_vocab*n_tokens logits. Tell llama.cpp early
// that we want this many logits so the state serializes consistently.
d_ptr->ctx_params.logits_all = true;
d_ptr->n_threads = std::min(4, (int32_t) std::thread::hardware_concurrency());
d_ptr->ctx_params.n_threads = d_ptr->n_threads;
d_ptr->ctx_params.n_threads_batch = d_ptr->n_threads;