mirror of
https://github.com/hwchase17/langchain.git
synced 2025-04-27 19:46:55 +00:00
fe0fd9dd70
Related to https://github.com/langchain-ai/langchain/issues/30344 https://github.com/langchain-ai/langchain/pull/30542 introduced an erroneous test for token counts for o-series models. tiktoken==0.8 does not support o-series models in `tiktoken.encoding_for_model(model_name)`, and this is the version of tiktoken we had in the lock file. So we would default to `cl100k_base` for o-series, which is the wrong encoding model. The test tested against this wrong encoding (so it passed with tiktoken 0.8). Here we update tiktoken to 0.9 in the lock file, and fix the expected counts in the test. Verified that we are pulling [o200k_base](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py#L8), as expected. |
||
|---|---|---|
| .. | ||
| cli | ||
| community | ||
| core | ||
| experimental | ||
| langchain | ||
| partners | ||
| standard-tests | ||
| text-splitters | ||
| packages.yml | ||