mirror of https://github.com/hwchase17/langchain.git synced 2026-01-18 02:45:33 +00:00

Files

Tom-Trumper 532e6455e9 text-splitters: Add keep_separator arg to HTMLSemanticPreservingSplitter (#31588 )

### Description
Add keep_separator arg to HTMLSemanticPreservingSplitter and pass value
to instance of RecursiveCharacterTextSplitter used under the hood.
### Issue
Documents returned by `HTMLSemanticPreservingSplitter.split_text(text)`
are defaulted to use separators at beginning of page_content. [See third
and fourth document in example output from how-to
guide](https://python.langchain.com/docs/how_to/split_html/#using-htmlsemanticpreservingsplitter):
```
[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),
 Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3"),
 Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video: ![image:example_image_link.mp4](example_image_link.mp4) ![video:example_video_link.mp4](example_video_link.mp4)'),
 Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: <code:html> <div> <p>This is a paragraph inside a div.</p> </div> </code>'),
 Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]
```
### Dependencies
None

@ttrumper3

2025-06-14 17:56:14 -04:00

langchain_text_splitters

text-splitters: Add keep_separator arg to HTMLSemanticPreservingSplitter (#31588 )

2025-06-14 17:56:14 -04:00

scripts

multiple: pydantic 2 compatibility, v0.3 (#26443 )

2024-09-13 14:38:45 -07:00

tests

text-splitters: Add keep_separator arg to HTMLSemanticPreservingSplitter (#31588 )

2025-06-14 17:56:14 -04:00

extended_testing_deps.txt

multiple: get rid of pyproject extras (#22581 )

2024-06-06 15:45:22 -07:00

Makefile

text-splitters: Set strict mypy rules (#30900 )

2025-04-22 20:41:24 -07:00

pyproject.toml

text-splitters[patch]: fix some import-untyped errors (#31030 )

2025-05-15 11:34:22 -04:00

README.md

docs: more api ref links, add linting step to prevent more (#28495 )

2024-12-04 04:19:42 +00:00

uv.lock

text-splitters[patch]: fix some import-untyped errors (#31030 )

2025-05-15 11:34:22 -04:00

README.md

🦜✂️ LangChain Text Splitters

Quick Install

pip install langchain-text-splitters

What is it?

LangChain Text Splitters contains utilities for splitting into chunks a wide variety of text documents.

For full documentation see the API reference and the Text Splitters module in the main docs.

📕 Releases & Versioning

langchain-text-splitters is currently on version 0.0.x.

Minor version increases will occur for:

Breaking changes for any public interfaces NOT marked beta

Patch version increases will occur for:

Bug fixes
New features
Any changes to private interfaces
Any changes to beta features

💁 Contributing

As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.

For detailed information on how to contribute, see the Contributing Guide.