mirror of
https://github.com/hwchase17/langchain.git
synced 2026-07-13 20:47:15 +00:00
Now returns (`_iter`, `tokens`, `indices`, token_counts`). The `token_counts` are calculated directly during tokenization, which is more accurate and efficient than splitting strings later.