github/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2025-12-22 19:45:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
899 Commits 42 Branches 71 Tags
fb09f412ffa705d42b4923e40c968ee2dfabd892
Commit Graph

4 Commits

Author SHA1 Message Date
Adam Treat
4a317eeb33 Revert "New tokenizer implementation for MPT and GPT-J" 
This reverts commit ee3469ba6c.
 2023-05-30 12:59:00 -04:00
Aaron Miller
ee3469ba6c New tokenizer implementation for MPT and GPT-J 
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
 2023-05-30 12:05:57 -04:00
Zach Nussbaum
53730c5f7f fix: use right conversion script 2023-05-11 11:20:43 -04:00
Adam Treat
8e7b96bd92 Move the llmodel C API to new top-level directory and version it. 2023-05-10 11:46:40 -04:00