github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-06-23 15:19:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix huggingface tokenizer default length function (#30185)

Browse Source 
#30184
This commit is contained in:
keshavshrikant 2025-03-31 21:24:30 +05:30 committed by GitHub
parent 4419340039
commit e8be3cca5c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
libs/text-splitters/langchain_text_splitters/base.py
View File

@ -158,7 +158,7 @@ class TextSplitter(BaseDocumentTransformer, ABC):
) )
def _huggingface_tokenizer_length(text: str) -> int: def _huggingface_tokenizer_length(text: str) -> int:
return len(tokenizer.encode(text)) return len(tokenizer.tokenize(text))
except ImportError: except ImportError:
raise ValueError( raise ValueError(