Switches type checking for `langchain-text-splitters` from `mypy` to
[`ty`](https://docs.astral.sh/ty/), which is much faster. The `ollama`
package already [switched to
`ty`](https://github.com/langchain-ai/langchain/pull/36571).
## What changed
The core of this PR is the config swap (`[tool.mypy]` →
`[tool.ty.rules]`/`[tool.ty.analysis]`, `Makefile`, and the `typing`
dependency group). Because `ty` runs with `all = "error"`, a few modules
also needed source-level adjustments to satisfy the stricter analysis.
These are **behavior-preserving refactors** except for one intentional
fix, called out below so reviewers know where to look.
### Behavioral change (intentional fix)
- `SentenceTransformersTokenTextSplitter` now raises a clear
`ValueError` when the underlying model reports no maximum sequence
length **and** no `tokens_per_chunk` was provided. Previously this
combination reached a `None > None` comparison and surfaced as an opaque
`TypeError`. As a consequence, the public `maximum_tokens_per_chunk`
attribute is now honestly typed as `int | None` — it can remain `None`
when the caller supplies `tokens_per_chunk` explicitly for a model
without a limit.
### Behavior-preserving refactors (no user-visible change)
- `TokenTextSplitter.from_tiktoken_encoder` is now an explicit override
rather than the base method dispatching on `issubclass(cls,
TokenTextSplitter)`. The shared length-function logic moved into a
private helper. Public signatures and return types are unchanged.
- `NLTKTextSplitter` builds its tokenizer once at construction, so
`_tokenizer` is now always a `Callable[[str], list[str]]`. The private
attributes `_language` and `_use_span_tokenize` are no longer stored —
flagging in case any downstream code read those (they are
underscore-private). Tokenization output is unchanged.
- `HTMLSemanticPreservingSplitter` text extraction was rewritten from a
`cast`-based check to `isinstance(element, Tag)` narrowing; output is
equivalent for tags, text nodes, and comments.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
## Summary
Fixes four issues in `get_separators_for_language()` in `character.py`:
- **Kotlin**: removed `"\ncase "` — `case` is not a Kotlin keyword.
Kotlin uses `when` expressions (already present in the list). This was
copied from Java/Swift.
- **Rust**: removed duplicate `"\nconst "` — appeared twice, once under
function definitions and again under control flow statements.
- **Haskell**: removed duplicate `"\n:: "` — appeared under function
definitions and again under type declarations.
- **Haskell**: removed duplicate `"\ndata "` — appeared under type
declarations and again under record field declarations.
All four are dead separators that never match or produce redundant
splits.
## Issue
Closes#37038
## Types of changes
- [x] Bug fix
## Checklist
- [x] I have read the CONTRIBUTING doc
- [x] Lint and unit tests pass locally with my changes
## Summary
Removes two incorrect separators from `get_separators_for_language()` in
`RecursiveCharacterTextSplitter`:
- **C#**: `"\nimplements "` is a Java keyword. C# uses `:` for interface
implementation. This separator never matches valid C# source code.
- **Elixir**: `"\nwhile "` does not exist in Elixir. The language uses
recursion and `Enum.reduce_while/3` instead of while loops.
Both are dead separators that silently degrade chunking quality by
occupying positions in the separator priority list without contributing
useful split points.
## Tests
Added two targeted tests:
- `test_csharp_separators_no_java_keywords`: verifies `"\nimplements "`
is not in the C# separator list
- `test_elixir_separators_no_while`: verifies `"\nwhile "` is not in the
Elixir separator list
Existing `test_csharp_code_splitter` continues to pass (no change to
expected output since `implements` never matched valid C# code).
Full suite: 129 passed, 0 failed.
Fixes#37030
**Description:**
Fix the merge logic in `CharacterTextSplitter.split_text` so that when
using a regex lookahead separator (`is_separator_regex=True`) with
`keep_separator=False`, the raw pattern is not re-inserted between
chunks.
**Issue:**
Fixes#31136
**Dependencies:**
None
**Twitter handle:**
None
Since this is my first open-source PR, please feel free to point out any
mistakes, and I'll be eager to make corrections.
As seen in #23188, turned on Google-style docstrings by enabling
`pydocstyle` linting in the `text-splitters` package. Each resulting
linting error was addressed differently: ignored, resolved, suppressed,
and missing docstrings were added.
Fixes one of the checklist items from #25154, similar to #25939 in
`core` package. Ran `make format`, `make lint` and `make test` from the
root of the package `text-splitters` to ensure no issues were found.
---------
Co-authored-by: Erick Friis <erick@langchain.dev>
**Description:** Spell check fixes for docs, comments, and a couple of
strings. No code change e.g. variable names.
**Issue:** none
**Dependencies:** none
**Twitter handle:** hmartin
Hi 👋
First off, thanks a ton for your work on this 💚 Really appreciate what
you're providing here for the community.
## Description
This PR adds a basic language parser for the
[Elixir](https://elixir-lang.org/) programming language. The parser code
is based upon the approach outlined in
https://github.com/langchain-ai/langchain/pull/13318: it's using
`tree-sitter` under the hood and aligns with all the other `tree-sitter`
based parses added that PR.
The `CHUNK_QUERY` I'm using here is probably not the most sophisticated
one, but it worked for my application. It's a starting point to provide
"core" parsing support for Elixir in LangChain. It enables people to use
the language parser out in real world applications which may then lead
to further tweaking of the queries. I consider this PR just the ground
work.
- **Dependencies:** requires `tree-sitter` and `tree-sitter-languages`
from the extended dependencies
- **Twitter handle:**`@bitcrowd`
## Checklist
- [x] **PR title**: "package: description"
- [x] **Add tests and docs**
- [x] **Lint and test**: Run `make format`, `make lint` and `make test`
from the root of the package(s) you've modified.
<!-- If no one reviews your PR within a few days, please @-mention one
of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. -->
**Description:** Added extra functionality to `CharacterTextSplitter`,
`TextSplitter` classes.
The user can select whether to append the separator to the previous
chunk with `keep_separator='end' ` or else prepend to the next chunk.
Previous functionality prepended by default to next chunk.
**Issue:** Fixes#20908
---------
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
- **Description:** Complete the support for Lua code in
langchain.text_splitter module.
- **Dependencies:** No
- **Twitter handle:** @saberuster
If no one reviews your PR within a few days, please @-mention one of
baskaryan, efriis, eyurtsev, hwchase17.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
- **Description:** Haskell language support added in text_splitter
module
- **Dependencies:** No
- **Twitter handle:** @nisargtr
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
