mirror of
https://github.com/hwchase17/langchain.git
synced 2026-07-11 18:18:51 +00:00
a2e53fda73
Switches type checking for `langchain-text-splitters` from `mypy` to [`ty`](https://docs.astral.sh/ty/), which is much faster. The `ollama` package already [switched to `ty`](https://github.com/langchain-ai/langchain/pull/36571). ## What changed The core of this PR is the config swap (`[tool.mypy]` → `[tool.ty.rules]`/`[tool.ty.analysis]`, `Makefile`, and the `typing` dependency group). Because `ty` runs with `all = "error"`, a few modules also needed source-level adjustments to satisfy the stricter analysis. These are **behavior-preserving refactors** except for one intentional fix, called out below so reviewers know where to look. ### Behavioral change (intentional fix) - `SentenceTransformersTokenTextSplitter` now raises a clear `ValueError` when the underlying model reports no maximum sequence length **and** no `tokens_per_chunk` was provided. Previously this combination reached a `None > None` comparison and surfaced as an opaque `TypeError`. As a consequence, the public `maximum_tokens_per_chunk` attribute is now honestly typed as `int | None` — it can remain `None` when the caller supplies `tokens_per_chunk` explicitly for a model without a limit. ### Behavior-preserving refactors (no user-visible change) - `TokenTextSplitter.from_tiktoken_encoder` is now an explicit override rather than the base method dispatching on `issubclass(cls, TokenTextSplitter)`. The shared length-function logic moved into a private helper. Public signatures and return types are unchanged. - `NLTKTextSplitter` builds its tokenizer once at construction, so `_tokenizer` is now always a `Callable[[str], list[str]]`. The private attributes `_language` and `_use_span_tokenize` are no longer stored — flagging in case any downstream code read those (they are underscore-private). Tokenization output is unchanged. - `HTMLSemanticPreservingSplitter` text extraction was rewritten from a `cast`-based check to `isinstance(element, Tag)` narrowing; output is equivalent for tags, text nodes, and comments. --------- Co-authored-by: Mason Daugherty <github@mdrxy.com>