github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-07-16 17:53:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Use tiktoken for Python 3.8 (#2709)

Browse Source 
Fixes issue #2677

`tiktoken` is supported for Python 3.8, so there is no need to use the
fallback GPT-2 tokenizer.
This commit is contained in:
Joshua Snyder 2023-04-11 20:02:28 +02:00 committed by GitHub
parent 186ca9d3e4
commit f435f2267c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
langchain/chat_models/openai.py
View File

@ -327,8 +327,8 @@ class ChatOpenAI(BaseChatModel):
def get_num_tokens(self, text: str) -> int:
"""Calculate num tokens with tiktoken package."""
# tiktoken NOT supported for Python 3.8 or below
if sys.version_info[1] <= 8:
# tiktoken NOT supported for Python 3.7 or below
if sys.version_info[1] <= 7:
return super().get_num_tokens(text)
try:
import tiktoken