## Summary
- Moves `nltk`, `spacy`, `sentence-transformers`, and `konlpy` imports
back inside class constructors/functions so they are only loaded when
the respective splitter is actually instantiated
- Adds a subprocess-based regression test to verify no heavy packages
are imported at `langchain_text_splitters` load time
## Why
PR #32325 moved these optional dependency imports to module-level
`try/except` blocks (to satisfy ruff's `PLC0415` rule). Since
`__init__.py` imports all four splitter modules, this caused `import
langchain_text_splitters` to eagerly load all optional heavy packages,
resulting in:
- A PyTorch NVML warning (`UserWarning: Can't initialize NVML`) on
non-GPU machines
- A ~650MB memory spike on import (74MB → 736MB), vs ~50MB in 0.3.x
The fix restores the lazy import pattern with `# noqa: PLC0415` to
suppress the linter rule, which is the correct trade-off when a
dependency has high instantiation cost.
## Review notes
- The `PLC0415` suppressions are intentional — these are optional heavy
dependencies that should never be loaded unless the user explicitly
instantiates the splitter class
- The regression test uses a subprocess for proper isolation (the test
file itself imports `langchain_text_splitters` at the top, so
`sys.modules` checks within the same process would not reflect a clean
import state)
Fixes#35437.
> **AI disclaimer:** This PR was developed with assistance from Claude
Code (Anthropic AI).
---------
Co-authored-by: AshwathB-debug <ashwathbalaji04@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Switches type checking for `langchain-text-splitters` from `mypy` to
[`ty`](https://docs.astral.sh/ty/), which is much faster. The `ollama`
package already [switched to
`ty`](https://github.com/langchain-ai/langchain/pull/36571).
## What changed
The core of this PR is the config swap (`[tool.mypy]` →
`[tool.ty.rules]`/`[tool.ty.analysis]`, `Makefile`, and the `typing`
dependency group). Because `ty` runs with `all = "error"`, a few modules
also needed source-level adjustments to satisfy the stricter analysis.
These are **behavior-preserving refactors** except for one intentional
fix, called out below so reviewers know where to look.
### Behavioral change (intentional fix)
- `SentenceTransformersTokenTextSplitter` now raises a clear
`ValueError` when the underlying model reports no maximum sequence
length **and** no `tokens_per_chunk` was provided. Previously this
combination reached a `None > None` comparison and surfaced as an opaque
`TypeError`. As a consequence, the public `maximum_tokens_per_chunk`
attribute is now honestly typed as `int | None` — it can remain `None`
when the caller supplies `tokens_per_chunk` explicitly for a model
without a limit.
### Behavior-preserving refactors (no user-visible change)
- `TokenTextSplitter.from_tiktoken_encoder` is now an explicit override
rather than the base method dispatching on `issubclass(cls,
TokenTextSplitter)`. The shared length-function logic moved into a
private helper. Public signatures and return types are unchanged.
- `NLTKTextSplitter` builds its tokenizer once at construction, so
`_tokenizer` is now always a `Callable[[str], list[str]]`. The private
attributes `_language` and `_use_span_tokenize` are no longer stored —
flagging in case any downstream code read those (they are
underscore-private). Tokenization output is unchanged.
- `HTMLSemanticPreservingSplitter` text extraction was rewritten from a
`cast`-based check to `isinstance(element, Tag)` narrowing; output is
equivalent for tags, text nodes, and comments.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
This PR closes#27781
# Problem
The current implementation of `NLTKTextSplitter` is using
`sent_tokenize`. However, this `sent_tokenize` doesn't handle chars
between 2 tokenized sentences... hence, this behavior throws errors when
we are using `add_start_index=True`, as described in issue #27781. In
particular:
```python
from nltk.tokenize import sent_tokenize
output1 = sent_tokenize("Innovation drives our success. Collaboration fosters creative solutions. Efficiency enhances data management.", language="english")
print(output1)
output2 = sent_tokenize("Innovation drives our success. Collaboration fosters creative solutions. Efficiency enhances data management.", language="english")
print(output2)
>>> ['Innovation drives our success.', 'Collaboration fosters creative solutions.', 'Efficiency enhances data management.']
>>> ['Innovation drives our success.', 'Collaboration fosters creative solutions.', 'Efficiency enhances data management.']
```
# Solution
With this new `use_span_tokenize` parameter, we can use NLTK to create
sentences (with `span_tokenize`), but also add extra chars to be sure
that we still can map the chunks to the original text.
---------
Co-authored-by: Erick Friis <erick@langchain.dev>
Co-authored-by: Erick Friis <erickfriis@gmail.com>
