github/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2025-09-02 00:57:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

llmodel: change tokenToString to not use string_view (#968)

Browse Source 
fixes a definite use-after-free and likely avoids some other
potential ones - std::string will convert to a std::string_view
automatically but as soon as the std::string in question goes out of
scope it is already freed and the string_view is pointing at freed
memory - this is *mostly* fine if its returning a reference to the
tokenizer's internal vocab table but it's, imo, too easy to return a
reference to a dynamically constructed string with this as replit is
doing (and unfortunately needs to do to convert the internal whitespace
replacement symbol back to a space)
This commit is contained in:
Aaron Miller
2023-06-13 04:14:02 -07:00
committed by GitHub
parent 726dcbd43d
commit 88616fde7f
11 changed files with 11 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
gpt4all-chat/chatgpt.h
View File

@@ -39,7 +39,7 @@ protected:
// them as they are only called from the default implementation of 'prompt' which we override and
// completely replace
std::vector<Token> tokenize(PromptContext &, const std::string&) const override { return std::vector<Token>(); }
std::string_view tokenToString(Token) const override { return std::string_view(); }
std::string tokenToString(Token) const override { return std::string(); }
Token sampleToken(PromptContext &ctx) const override { return -1; }
bool evalTokens(PromptContext &/*ctx*/, const std::vector<int32_t>& /*tokens*/) const override { return false; }
int32_t contextLength() const override { return -1; }