github/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2025-09-06 19:10:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

backend: dedupe tokenizing code in gptj/mpt

Browse Source
This commit is contained in:
Aaron Miller
2023-05-15 17:30:45 -07:00
committed by AT
parent 4cd8bdf9a1
commit 6182026c70
3 changed files with 57 additions and 121 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
gpt4all-backend/utils.h
View File

@@ -44,6 +44,11 @@ struct gpt_vocab {
std::map<token, id> token_to_id;
std::map<id, token> id_to_token;
std::vector<std::string> special_tokens;
void add_special_token(const std::string &token) {
special_tokens.push_back(token);
}
};
void replace(std::string & str, const std::string & needle, const std::string & replacement);