## Summary
- Moves `nltk`, `spacy`, `sentence-transformers`, and `konlpy` imports
back inside class constructors/functions so they are only loaded when
the respective splitter is actually instantiated
- Adds a subprocess-based regression test to verify no heavy packages
are imported at `langchain_text_splitters` load time
## Why
PR #32325 moved these optional dependency imports to module-level
`try/except` blocks (to satisfy ruff's `PLC0415` rule). Since
`__init__.py` imports all four splitter modules, this caused `import
langchain_text_splitters` to eagerly load all optional heavy packages,
resulting in:
- A PyTorch NVML warning (`UserWarning: Can't initialize NVML`) on
non-GPU machines
- A ~650MB memory spike on import (74MB → 736MB), vs ~50MB in 0.3.x
The fix restores the lazy import pattern with `# noqa: PLC0415` to
suppress the linter rule, which is the correct trade-off when a
dependency has high instantiation cost.
## Review notes
- The `PLC0415` suppressions are intentional — these are optional heavy
dependencies that should never be loaded unless the user explicitly
instantiates the splitter class
- The regression test uses a subprocess for proper isolation (the test
file itself imports `langchain_text_splitters` at the top, so
`sys.modules` checks within the same process would not reflect a clean
import state)
Fixes#35437.
> **AI disclaimer:** This PR was developed with assistance from Claude
Code (Anthropic AI).
---------
Co-authored-by: AshwathB-debug <ashwathbalaji04@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Switches type checking for `langchain-text-splitters` from `mypy` to
[`ty`](https://docs.astral.sh/ty/), which is much faster. The `ollama`
package already [switched to
`ty`](https://github.com/langchain-ai/langchain/pull/36571).
## What changed
The core of this PR is the config swap (`[tool.mypy]` →
`[tool.ty.rules]`/`[tool.ty.analysis]`, `Makefile`, and the `typing`
dependency group). Because `ty` runs with `all = "error"`, a few modules
also needed source-level adjustments to satisfy the stricter analysis.
These are **behavior-preserving refactors** except for one intentional
fix, called out below so reviewers know where to look.
### Behavioral change (intentional fix)
- `SentenceTransformersTokenTextSplitter` now raises a clear
`ValueError` when the underlying model reports no maximum sequence
length **and** no `tokens_per_chunk` was provided. Previously this
combination reached a `None > None` comparison and surfaced as an opaque
`TypeError`. As a consequence, the public `maximum_tokens_per_chunk`
attribute is now honestly typed as `int | None` — it can remain `None`
when the caller supplies `tokens_per_chunk` explicitly for a model
without a limit.
### Behavior-preserving refactors (no user-visible change)
- `TokenTextSplitter.from_tiktoken_encoder` is now an explicit override
rather than the base method dispatching on `issubclass(cls,
TokenTextSplitter)`. The shared length-function logic moved into a
private helper. Public signatures and return types are unchanged.
- `NLTKTextSplitter` builds its tokenizer once at construction, so
`_tokenizer` is now always a `Callable[[str], list[str]]`. The private
attributes `_language` and `_use_span_tokenize` are no longer stored —
flagging in case any downstream code read those (they are
underscore-private). Tokenization output is unchanged.
- `HTMLSemanticPreservingSplitter` text extraction was rewritten from a
`cast`-based check to `isinstance(element, Tag)` narrowing; output is
equivalent for tags, text nodes, and comments.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
CI lint jobs use `uv run --all-groups` for all tools, but ruff doesn't
need dependency resolution — only mypy does. By splitting into
`UV_RUN_LINT` (ruff) and `UV_RUN_TYPE` (mypy), the CI-facing targets run
ruff with `--group lint` only, giving fast-fail feedback before mypy
triggers the full environment sync.
For packages where source code only conditionally imports heavy deps
(text-splitters, huggingface), `lint_package` also overrides
`UV_RUN_TYPE` to `--group lint --group typing`, skipping the ~3.5GB
`test_integration` download entirely. `lint_tests` keeps `--all-groups`
since test code legitimately imports those deps.
Additionally, `lint_imports.sh` was inconsistently wired — most packages
had the script but weren't calling it.
## Changes
**Makefile optimization**
- Introduce `UV_RUN_LINT` and `UV_RUN_TYPE` Make variables, both
defaulting to `uv run --all-groups`. For `lint_package` and
`lint_tests`, `UV_RUN_LINT` is overridden to `uv run --group lint` so
ruff runs instantly without syncing heavy deps
- For `text-splitters` and `huggingface`, override `UV_RUN_TYPE` on
`lint_package` to `uv run --group lint --group typing` — mypy runs
without downloading torch, CUDA, spacy, etc.
**mypy config for lean groups**
- Add `transformers` and `transformers.*` to `ignore_missing_imports` in
`text-splitters` pyproject.toml (conditional `try/except` import, same
treatment as existing `konlpy`/`nltk` entries)
- Add `torch`, `torch.*`, `langchain_community`, `langchain_community.*`
to `ignore_missing_imports` in `huggingface` pyproject.toml
- Add dual `# type: ignore[unreachable, unused-ignore]` in
`text-splitters/base.py` to handle the `PreTrainedTokenizerBase`
isinstance check that behaves differently depending on whether
transformers is installed
**lint_imports.sh consistency**
- Add `./scripts/lint_imports.sh` to the lint recipe in every package
that wasn't calling it (standard-tests, model-profiles, all 15
partners), and create the script for the two packages missing it
entirely (`model-profiles`, `openrouter`)
- Update all `lint_imports.sh` scripts to allow `from langchain.agents`
and `from langchain.tools` imports (legitimate v1 middleware
dependencies used by `langchain-anthropic` and `langchain-openai`)
During an automated code review of .github/scripts/get_min_versions.py,
the following issue was identified. Set a timeout on get min versions
HTTP calls. Network calls without a timeout can hang a worker
indefinitely. I kept the patch small and re-ran syntax checks after
applying it.
Thank you for contributing to LangChain!
- [x] **PR title**: "package: description"
- Where "package" is whichever of langchain, core, etc. is being
modified. Use "docs: ..." for purely docs changes, "infra: ..." for CI
changes.
- Example: "core: add foobar LLM"
- [x] **PR message**: ***Delete this entire checklist*** and replace
with
- **Description:** a description of the change
- **Issue:** the issue # it fixes, if applicable
- **Dependencies:** any dependencies required for this change
- **Twitter handle:** if your PR gets announced, and you'd like a
mention, we'll gladly shout you out!
- [x] **Add tests and docs**: If you're adding a new integration, please
include
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in
`docs/docs/integrations` directory.
- [x] **Lint and test**: Run `make format`, `make lint` and `make test`
from the root of the package(s) you've modified. See contribution
guidelines for more: https://python.langchain.com/docs/contributing/
Additional guidelines:
- Make sure optional dependencies are imported within a function.
- Please do not add dependencies to pyproject.toml files (even optional
ones) unless they are required for unit tests.
- Most PRs should not touch more than one package.
- Changes should be backwards compatible.
If no one reviews your PR within a few days, please @-mention one of
baskaryan, eyurtsev, ccurme, vbarda, hwchase17.
As seen in #23188, turned on Google-style docstrings by enabling
`pydocstyle` linting in the `text-splitters` package. Each resulting
linting error was addressed differently: ignored, resolved, suppressed,
and missing docstrings were added.
Fixes one of the checklist items from #25154, similar to #25939 in
`core` package. Ran `make format`, `make lint` and `make test` from the
root of the package `text-splitters` to ensure no issues were found.
---------
Co-authored-by: Erick Friis <erick@langchain.dev>
Hi 👋
First off, thanks a ton for your work on this 💚 Really appreciate what
you're providing here for the community.
## Description
This PR adds a basic language parser for the
[Elixir](https://elixir-lang.org/) programming language. The parser code
is based upon the approach outlined in
https://github.com/langchain-ai/langchain/pull/13318: it's using
`tree-sitter` under the hood and aligns with all the other `tree-sitter`
based parses added that PR.
The `CHUNK_QUERY` I'm using here is probably not the most sophisticated
one, but it worked for my application. It's a starting point to provide
"core" parsing support for Elixir in LangChain. It enables people to use
the language parser out in real world applications which may then lead
to further tweaking of the queries. I consider this PR just the ground
work.
- **Dependencies:** requires `tree-sitter` and `tree-sitter-languages`
from the extended dependencies
- **Twitter handle:**`@bitcrowd`
## Checklist
- [x] **PR title**: "package: description"
- [x] **Add tests and docs**
- [x] **Lint and test**: Run `make format`, `make lint` and `make test`
from the root of the package(s) you've modified.
<!-- If no one reviews your PR within a few days, please @-mention one
of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. -->
**Description:** Added extra functionality to `CharacterTextSplitter`,
`TextSplitter` classes.
The user can select whether to append the separator to the previous
chunk with `keep_separator='end' ` or else prepend to the next chunk.
Previous functionality prepended by default to next chunk.
**Issue:** Fixes#20908
---------
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
- **Description:** Haskell language support added in text_splitter
module
- **Dependencies:** No
- **Twitter handle:** @nisargtr
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
