github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-10-23 19:44:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
9cc85387d1e38cea3a25c2f00f12b7b54bd0df3c
langchain/libs/text-splitters
History
Hyunjoon Jeong 9cc85387d1 fix(text-splitters): add validation to prevent infinite loop and prevent empty token splitter (#32205) 
### Description
1) Add validation to prevent infinite loop condition when
```tokenizer.tokens_per_chunk > tokenizer.chunk_overlap```
2) Avoid empty decoded chunk when splitter appends tokens

---------

Co-authored-by: Eugene Yurtsev <eugene@langchain.dev>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2025-09-11 16:55:32 -04:00
..
langchain_text_splitters
fix(text-splitters): add validation to prevent infinite loop and prevent empty token splitter (#32205)
2025-09-11 16:55:32 -04:00
scripts
chore(text-splitters): select ALL rules with exclusions (#32325)
2025-09-08 14:46:09 +00:00
tests
fix(text-splitters): add validation to prevent infinite loop and prevent empty token splitter (#32205)
2025-09-11 16:55:32 -04:00
extended_testing_deps.txt
…
Makefile
feat(docs): improve devx, fix Makefile targets (#32237)
2025-07-25 14:49:03 -04:00
pyproject.toml
revert: "chore: remove ruff target-version" (#32895)
2025-09-10 20:56:48 -04:00
README.md
chore(docs): update package READMEs (#32869)
2025-09-09 14:50:32 +00:00
uv.lock
chore(text-splitters): select ALL rules with exclusions (#32325)
2025-09-08 14:46:09 +00:00

README.md

🦜✂️ LangChain Text Splitters

PyPI - License PyPI - Downloads

Quick Install

pip install langchain-text-splitters

What is it?

LangChain Text Splitters contains utilities for splitting into chunks a wide variety of text documents.

For full documentation see the API reference and the Text Splitters module in the main docs.

📕 Releases & Versioning

Minor version increases will occur for:

  • Breaking changes for any public interfaces NOT marked beta

Patch version increases will occur for:

  • Bug fixes
  • New features
  • Any changes to private interfaces
  • Any changes to beta features

💁 Contributing

As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.

For detailed information on how to contribute, see the Contributing Guide.