## Summary
- Moves `nltk`, `spacy`, `sentence-transformers`, and `konlpy` imports
back inside class constructors/functions so they are only loaded when
the respective splitter is actually instantiated
- Adds a subprocess-based regression test to verify no heavy packages
are imported at `langchain_text_splitters` load time
## Why
PR #32325 moved these optional dependency imports to module-level
`try/except` blocks (to satisfy ruff's `PLC0415` rule). Since
`__init__.py` imports all four splitter modules, this caused `import
langchain_text_splitters` to eagerly load all optional heavy packages,
resulting in:
- A PyTorch NVML warning (`UserWarning: Can't initialize NVML`) on
non-GPU machines
- A ~650MB memory spike on import (74MB → 736MB), vs ~50MB in 0.3.x
The fix restores the lazy import pattern with `# noqa: PLC0415` to
suppress the linter rule, which is the correct trade-off when a
dependency has high instantiation cost.
## Review notes
- The `PLC0415` suppressions are intentional — these are optional heavy
dependencies that should never be loaded unless the user explicitly
instantiates the splitter class
- The regression test uses a subprocess for proper isolation (the test
file itself imports `langchain_text_splitters` at the top, so
`sys.modules` checks within the same process would not reflect a clean
import state)
Fixes#35437.
> **AI disclaimer:** This PR was developed with assistance from Claude
Code (Anthropic AI).
---------
Co-authored-by: AshwathB-debug <ashwathbalaji04@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Switches type checking for `langchain-text-splitters` from `mypy` to
[`ty`](https://docs.astral.sh/ty/), which is much faster. The `ollama`
package already [switched to
`ty`](https://github.com/langchain-ai/langchain/pull/36571).
## What changed
The core of this PR is the config swap (`[tool.mypy]` →
`[tool.ty.rules]`/`[tool.ty.analysis]`, `Makefile`, and the `typing`
dependency group). Because `ty` runs with `all = "error"`, a few modules
also needed source-level adjustments to satisfy the stricter analysis.
These are **behavior-preserving refactors** except for one intentional
fix, called out below so reviewers know where to look.
### Behavioral change (intentional fix)
- `SentenceTransformersTokenTextSplitter` now raises a clear
`ValueError` when the underlying model reports no maximum sequence
length **and** no `tokens_per_chunk` was provided. Previously this
combination reached a `None > None` comparison and surfaced as an opaque
`TypeError`. As a consequence, the public `maximum_tokens_per_chunk`
attribute is now honestly typed as `int | None` — it can remain `None`
when the caller supplies `tokens_per_chunk` explicitly for a model
without a limit.
### Behavior-preserving refactors (no user-visible change)
- `TokenTextSplitter.from_tiktoken_encoder` is now an explicit override
rather than the base method dispatching on `issubclass(cls,
TokenTextSplitter)`. The shared length-function logic moved into a
private helper. Public signatures and return types are unchanged.
- `NLTKTextSplitter` builds its tokenizer once at construction, so
`_tokenizer` is now always a `Callable[[str], list[str]]`. The private
attributes `_language` and `_use_span_tokenize` are no longer stored —
flagging in case any downstream code read those (they are
underscore-private). Tokenization output is unchanged.
- `HTMLSemanticPreservingSplitter` text extraction was rewritten from a
`cast`-based check to `isinstance(element, Tag)` narrowing; output is
equivalent for tags, text nodes, and comments.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
As seen in #23188, turned on Google-style docstrings by enabling
`pydocstyle` linting in the `text-splitters` package. Each resulting
linting error was addressed differently: ignored, resolved, suppressed,
and missing docstrings were added.
Fixes one of the checklist items from #25154, similar to #25939 in
`core` package. Ran `make format`, `make lint` and `make test` from the
root of the package `text-splitters` to ensure no issues were found.
---------
Co-authored-by: Erick Friis <erick@langchain.dev>
Previously, regardless of whether or not strip_whitespace was set to
true or false, the strip text method in the SpacyTextSplitter class used
`sent.text` to get the sentence. I modified this to include a ternary
such that if strip_whitespace is false, it uses `sent.text_with_ws`
I also modified the project.toml to include the spacy pipeline package
and to lock the numpy version, as higher versions break spacy.
- **Issue:** N/a
- **Dependencies:** None
