github/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2025-08-07 19:13:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Explicitly clear the kv cache each time we eval tokens to match n_past. (#1808)

Browse Source
This commit is contained in:
AT 2024-01-03 13:06:08 -06:00 committed by GitHub
parent 2d566710e5
commit 96cee4f9ac
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
gpt4all-backend/llamamodel.cpp
View File

@ -298,6 +298,8 @@ LLModel::Token LLamaModel::sampleToken(PromptContext &promptCtx) const
bool LLamaModel::evalTokens(PromptContext &ctx, const std::vector<int32_t> &tokens) const bool LLamaModel::evalTokens(PromptContext &ctx, const std::vector<int32_t> &tokens) const
{ {
llama_kv_cache_seq_rm(d_ptr->ctx, 0, ctx.n_past, -1);
llama_batch batch = llama_batch_init(tokens.size(), 0, 1); llama_batch batch = llama_batch_init(tokens.size(), 0, 1);
batch.n_tokens = tokens.size(); batch.n_tokens = tokens.size();