openai[patch]: upgrade tiktoken and fix test (#30621)

Related to https://github.com/langchain-ai/langchain/issues/30344

https://github.com/langchain-ai/langchain/pull/30542 introduced an
erroneous test for token counts for o-series models. tiktoken==0.8 does
not support o-series models in
`tiktoken.encoding_for_model(model_name)`, and this is the version of
tiktoken we had in the lock file. So we would default to `cl100k_base`
for o-series, which is the wrong encoding model. The test tested against
this wrong encoding (so it passed with tiktoken 0.8).

Here we update tiktoken to 0.9 in the lock file, and fix the expected
counts in the test. Verified that we are pulling
[o200k_base](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py#L8),
as expected.
This commit is contained in:
ccurme
2025-04-02 10:44:48 -04:00
committed by GitHub
parent 38807871ec
commit fe0fd9dd70
2 changed files with 34 additions and 34 deletions

View File

@@ -10,8 +10,8 @@ _EXPECTED_NUM_TOKENS = {
"gpt-4": 12,
"gpt-4-32k": 12,
"gpt-3.5-turbo": 12,
"o1": 12,
"o3": 12,
"o1": 11,
"o3": 11,
"gpt-4o": 11,
}