mirror of
https://github.com/hwchase17/langchain.git
synced 2025-05-23 07:57:16 +00:00
f69695069d
**Description:** With current HTML splitters, they rely on secondary use of the `RecursiveCharacterSplitter` to further chunk the document into manageable chunks. The issue with this is it fails to maintain important structures such as tables, lists, etc within HTML. This Implementation of a HTML splitter, allows the user to define a maximum chunk size, HTML elements to preserve in full, options to preserve `<a>` href links in the output and custom handlers. The core splitting begins with headers, similar to `HTMLHeaderSplitter`. If these sections exceed the length of the `max_chunk_size` further recursive splitting is triggered. During this splitting, elements listed to preserve, will be excluded from the splitting process. This can cause chunks to be slightly larger then the max size, depending on preserved length. However, all contextual relevance of the preserved item remains intact. **Custom Handlers**: Sometimes, companies such as Atlassian have custom HTML elements, that are not parsed by default with `BeautifulSoup`. Custom handlers allows a user to provide a function to be ran whenever a specific html tag is encountered. This allows the user to preserve and gather information within custom html tags that `bs4` will potentially miss during extraction. **Dependencies:** User will need to install `bs4` in their project to utilise this class I have also added in `how_to` and unit tests, which require `bs4` to run, otherwise they will be skipped. Flowchart of process:  --------- Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Chester Curme <chester.curme@gmail.com>
80 lines
2.2 KiB
Python
80 lines
2.2 KiB
Python
"""**Text Splitters** are classes for splitting text.
|
|
|
|
**Class hierarchy:**
|
|
|
|
.. code-block::
|
|
|
|
BaseDocumentTransformer --> TextSplitter --> <name>TextSplitter # Example: CharacterTextSplitter
|
|
RecursiveCharacterTextSplitter --> <name>TextSplitter
|
|
|
|
Note: **MarkdownHeaderTextSplitter** and **HTMLHeaderTextSplitter do not derive from TextSplitter.
|
|
|
|
|
|
**Main helpers:**
|
|
|
|
.. code-block::
|
|
|
|
Document, Tokenizer, Language, LineType, HeaderType
|
|
|
|
""" # noqa: E501
|
|
|
|
from langchain_text_splitters.base import (
|
|
Language,
|
|
TextSplitter,
|
|
Tokenizer,
|
|
TokenTextSplitter,
|
|
split_text_on_tokens,
|
|
)
|
|
from langchain_text_splitters.character import (
|
|
CharacterTextSplitter,
|
|
RecursiveCharacterTextSplitter,
|
|
)
|
|
from langchain_text_splitters.html import (
|
|
ElementType,
|
|
HTMLHeaderTextSplitter,
|
|
HTMLSectionSplitter,
|
|
HTMLSemanticPreservingSplitter,
|
|
)
|
|
from langchain_text_splitters.json import RecursiveJsonSplitter
|
|
from langchain_text_splitters.konlpy import KonlpyTextSplitter
|
|
from langchain_text_splitters.latex import LatexTextSplitter
|
|
from langchain_text_splitters.markdown import (
|
|
ExperimentalMarkdownSyntaxTextSplitter,
|
|
HeaderType,
|
|
LineType,
|
|
MarkdownHeaderTextSplitter,
|
|
MarkdownTextSplitter,
|
|
)
|
|
from langchain_text_splitters.nltk import NLTKTextSplitter
|
|
from langchain_text_splitters.python import PythonCodeTextSplitter
|
|
from langchain_text_splitters.sentence_transformers import (
|
|
SentenceTransformersTokenTextSplitter,
|
|
)
|
|
from langchain_text_splitters.spacy import SpacyTextSplitter
|
|
|
|
__all__ = [
|
|
"TokenTextSplitter",
|
|
"TextSplitter",
|
|
"Tokenizer",
|
|
"Language",
|
|
"RecursiveCharacterTextSplitter",
|
|
"RecursiveJsonSplitter",
|
|
"LatexTextSplitter",
|
|
"PythonCodeTextSplitter",
|
|
"KonlpyTextSplitter",
|
|
"SpacyTextSplitter",
|
|
"NLTKTextSplitter",
|
|
"split_text_on_tokens",
|
|
"SentenceTransformersTokenTextSplitter",
|
|
"ElementType",
|
|
"HeaderType",
|
|
"LineType",
|
|
"HTMLHeaderTextSplitter",
|
|
"HTMLSectionSplitter",
|
|
"HTMLSemanticPreservingSplitter",
|
|
"MarkdownHeaderTextSplitter",
|
|
"MarkdownTextSplitter",
|
|
"CharacterTextSplitter",
|
|
"ExperimentalMarkdownSyntaxTextSplitter",
|
|
]
|