github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-07-18 10:43:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix wrong token counts from get_num_tokens from openai llms (#2952)

Browse Source 
The encoding fetch was out of date. Luckily OpenAI has a nice[
`encoding_for_model`](46287bfa49/tiktoken/model.py)
function in `tiktoken` we can use now.
This commit is contained in:
Tim Asp 2023-04-15 16:09:17 -07:00 committed by GitHub
parent fea5619ce9
commit b9db20481f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
langchain/llms/openai.py
View File

@ -446,15 +446,9 @@ class BaseOpenAI(BaseLLM):
"This is needed in order to calculate get_num_tokens. " "This is needed in order to calculate get_num_tokens. "
"Please install it with `pip install tiktoken`." "Please install it with `pip install tiktoken`."
) )
encoder = "gpt2"
if self.model_name in ("text-davinci-003", "text-davinci-002"):
encoder = "p50k_base"
if self.model_name.startswith("code"):
encoder = "p50k_base"
# create a GPT-3 encoder instance
enc = tiktoken.get_encoding(encoder)
# encode the text using the GPT-3 encoder enc = tiktoken.encoding_for_model(self.model_name)
tokenized_text = enc.encode(text) tokenized_text = enc.encode(text)
# calculate the number of tokens in the encoded text # calculate the number of tokens in the encoded text