github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-02 19:47:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

text_splitters: Add HTMLSemanticPreservingSplitter (#25911)

Browse Source 
**Description:** 

With current HTML splitters, they rely on secondary use of the
`RecursiveCharacterSplitter` to further chunk the document into
manageable chunks. The issue with this is it fails to maintain important
structures such as tables, lists, etc within HTML.

This Implementation of a HTML splitter, allows the user to define a
maximum chunk size, HTML elements to preserve in full, options to
preserve `<a>` href links in the output and custom handlers.

The core splitting begins with headers, similar to `HTMLHeaderSplitter`.
If these sections exceed the length of the `max_chunk_size` further
recursive splitting is triggered. During this splitting, elements listed
to preserve, will be excluded from the splitting process. This can cause
chunks to be slightly larger then the max size, depending on preserved
length. However, all contextual relevance of the preserved item remains
intact.

**Custom Handlers**: Sometimes, companies such as Atlassian have custom
HTML elements, that are not parsed by default with `BeautifulSoup`.
Custom handlers allows a user to provide a function to be ran whenever a
specific html tag is encountered. This allows the user to preserve and
gather information within custom html tags that `bs4` will potentially
miss during extraction.

**Dependencies:** User will need to install `bs4` in their project to
utilise this class

I have also added in `how_to` and unit tests, which require `bs4` to
run, otherwise they will be skipped.

Flowchart of process:


![HTMLSemanticPreservingSplitter](https://github.com/user-attachments/assets/20873c36-22ed-4c80-884b-d3c6f433f5a7)

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Chester Curme <chester.curme@gmail.com>
This commit is contained in:
Luke
2024-12-20 04:09:22 +11:00
committed by GitHub
parent 24bfa062bf
commit f69695069d
13 changed files with 1835 additions and 574 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/cassettes/HTML_header_metadata_splitter_5a5ec1482171b119.msgpack.zlib
View File

File diff suppressed because one or more lines are too long

1
docs/cassettes/HTML_header_metadata_splitter_6ecb9fb2-32ff-4249-a4b4-d5e5e191f013.msgpack.zlib
View File

File diff suppressed because one or more lines are too long

1
docs/cassettes/split_html_332043f6-ec39-4fcd-aa57-4dec5252684b.msgpack.zlib Normal file
View File

File diff suppressed because one or more lines are too long

1
docs/cassettes/split_html_581eeddf-7e88-48a7-999d-da56304e3522.msgpack.zlib Normal file
View File

File diff suppressed because one or more lines are too long

2
docs/docs/concepts/text_splitters.mdx
View File

@@ -110,7 +110,7 @@ Examples of structure-based splitting:
* See the how-to guide for [Markdown splitting](/docs/how_to/markdown_header_metadata_splitter/).
* See the how-to guide for [Recursive JSON splitting](/docs/how_to/recursive_json_splitter/).
* See the how-to guide for [Code splitting](/docs/how_to/code_splitter/).
* See the how-to guide for [HTML splitting](/docs/how_to/HTML_header_metadata_splitter/).
* See the how-to guide for [HTML splitting](/docs/how_to/split_html/).
:::

359
docs/docs/how_to/HTML_header_metadata_splitter.ipynb
View File

@@ -1,359 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c95fcd15cd52c944",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"# How to split by HTML header \n",
"## Description and motivation\n",
"\n",
"[HTMLHeaderTextSplitter](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html) is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the HTML element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n",
"\n",
"It is analogous to the [MarkdownHeaderTextSplitter](/docs/how_to/markdown_header_metadata_splitter) for markdown files.\n",
"\n",
"To specify what headers to split on, specify `headers_to_split_on` when instantiating `HTMLHeaderTextSplitter` as shown below.\n",
"\n",
"## Usage examples\n",
"### 1) How to split HTML strings:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2e55d44c-1fff-449a-bf52-0d6df488323f",
"metadata": {},
"outputs": [],
"source": [
"%pip install -qU langchain-text-splitters"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "initial_id",
"metadata": {
"ExecuteTime": {
"end_time": "2023-10-02T18:57:49.208965400Z",
"start_time": "2023-10-02T18:57:48.899756Z"
},
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='Foo'),\n",
" Document(page_content='Some intro text about Foo. \\nBar main section Bar subsection 1 Bar subsection 2', metadata={'Header 1': 'Foo'}),\n",
" Document(page_content='Some intro text about Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section'}),\n",
" Document(page_content='Some text about the first subtopic of Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 1'}),\n",
" Document(page_content='Some text about the second subtopic of Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 2'}),\n",
" Document(page_content='Baz', metadata={'Header 1': 'Foo'}),\n",
" Document(page_content='Some text about Baz', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}),\n",
" Document(page_content='Some concluding text about Foo', metadata={'Header 1': 'Foo'})]"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import HTMLHeaderTextSplitter\n",
"\n",
"html_string = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
"<body>\n",
" <div>\n",
" <h1>Foo</h1>\n",
" <p>Some intro text about Foo.</p>\n",
" <div>\n",
" <h2>Bar main section</h2>\n",
" <p>Some intro text about Bar.</p>\n",
" <h3>Bar subsection 1</h3>\n",
" <p>Some text about the first subtopic of Bar.</p>\n",
" <h3>Bar subsection 2</h3>\n",
" <p>Some text about the second subtopic of Bar.</p>\n",
" </div>\n",
" <div>\n",
" <h2>Baz</h2>\n",
" <p>Some text about Baz</p>\n",
" </div>\n",
" <br>\n",
" <p>Some concluding text about Foo</p>\n",
" </div>\n",
"</body>\n",
"</html>\n",
"\"\"\"\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
" (\"h3\", \"Header 3\"),\n",
"]\n",
"\n",
"html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
"html_header_splits = html_splitter.split_text(html_string)\n",
"html_header_splits"
]
},
{
"cell_type": "markdown",
"id": "7126f179-f4d0-4b5d-8bef-44e83b59262c",
"metadata": {},
"source": [
"To return each element together with their associated headers, specify `return_each_element=True` when instantiating `HTMLHeaderTextSplitter`:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "90c23088-804c-4c89-bd09-b820587ceeef",
"metadata": {},
"outputs": [],
"source": [
"html_splitter = HTMLHeaderTextSplitter(\n",
" headers_to_split_on,\n",
" return_each_element=True,\n",
")\n",
"html_header_splits_elements = html_splitter.split_text(html_string)"
]
},
{
"cell_type": "markdown",
"id": "b776c54e-9159-4d88-9d6c-3a1d0b639dfe",
"metadata": {},
"source": [
"Comparing with the above, where elements are aggregated by their headers:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "711abc74-a7b0-4dc5-a4bb-af3cafe4e0f4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"page_content='Foo'\n",
"page_content='Some intro text about Foo. \\nBar main section Bar subsection 1 Bar subsection 2' metadata={'Header 1': 'Foo'}\n"
]
}
],
"source": [
"for element in html_header_splits[:2]:\n",
" print(element)"
]
},
{
"cell_type": "markdown",
"id": "fe5528db-187c-418a-9480-fc0267645d42",
"metadata": {},
"source": [
"Now each element is returned as a distinct `Document`:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "24722d8e-d073-46a8-a821-6b722412f1be",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"page_content='Foo'\n",
"page_content='Some intro text about Foo.' metadata={'Header 1': 'Foo'}\n",
"page_content='Bar main section Bar subsection 1 Bar subsection 2' metadata={'Header 1': 'Foo'}\n"
]
}
],
"source": [
"for element in html_header_splits_elements[:3]:\n",
" print(element)"
]
},
{
"cell_type": "markdown",
"id": "e29b4aade2a0070c",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"#### 2) How to split from a URL or HTML file:\n",
"\n",
"To read directly from a URL, pass the URL string into the `split_text_from_url` method.\n",
"\n",
"Similarly, a local HTML file can be passed to the `split_text_from_file` method."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "6ecb9fb2-32ff-4249-a4b4-d5e5e191f013",
"metadata": {},
"outputs": [],
"source": [
"url = \"https://plato.stanford.edu/entries/goedel/\"\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
" (\"h3\", \"Header 3\"),\n",
" (\"h4\", \"Header 4\"),\n",
"]\n",
"\n",
"html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
"\n",
"# for local file use html_splitter.split_text_from_file(<path_to_file>)\n",
"html_header_splits = html_splitter.split_text_from_url(url)"
]
},
{
"cell_type": "markdown",
"id": "c6e3dd41-0c57-472a-a3d4-4e7e8ea6914f",
"metadata": {},
"source": [
"### 2) How to constrain chunk sizes:\n",
"\n",
"`HTMLHeaderTextSplitter`, which splits based on HTML headers, can be composed with another splitter which constrains splits based on character lengths, such as `RecursiveCharacterTextSplitter`.\n",
"\n",
"This can be done using the `.split_documents` method of the second splitter:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "6ada8ea093ea0475",
"metadata": {
"ExecuteTime": {
"end_time": "2023-10-02T18:57:51.016141300Z",
"start_time": "2023-10-02T18:57:50.647495400Z"
},
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth', metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}),\n",
" Document(page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.', metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}),\n",
" Document(page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians', metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}),\n",
" Document(page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.', metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}),\n",
" Document(page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.', metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'})]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
"\n",
"chunk_size = 500\n",
"chunk_overlap = 30\n",
"text_splitter = RecursiveCharacterTextSplitter(\n",
" chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
")\n",
"\n",
"# Split\n",
"splits = text_splitter.split_documents(html_header_splits)\n",
"splits[80:85]"
]
},
{
"cell_type": "markdown",
"id": "ac0930371d79554a",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"## Limitations\n",
"\n",
"There can be quite a bit of structural variation from one HTML document to another, and while `HTMLHeaderTextSplitter` will attempt to attach all \"relevant\" headers to any given chunk, it can sometimes miss certain headers. For example, the algorithm assumes an informational hierarchy in which headers are always at nodes \"above\" associated text, i.e. prior siblings, ancestors, and combinations thereof. In the following news article (as of the writing of this document), the document is structured such that the text of the top-level headline, while tagged \"h1\", is in a *distinct* subtree from the text elements that we'd expect it to be *\"above\"*&mdash;so we can observe that the \"h1\" element and its associated text do not show up in the chunk metadata (but, where applicable, we do see \"h2\" and its associated text): \n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "5a5ec1482171b119",
"metadata": {
"ExecuteTime": {
"end_time": "2023-10-02T19:03:25.943524300Z",
"start_time": "2023-10-02T19:03:25.691641Z"
},
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"No two El Niño winters are the same, but many have temperature and precipitation trends in common. \n",
"Average conditions during an El Niño winter across the continental US. \n",
"One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA. \n",
"Because the jet stream is essentially a river of air that storms flow through, they c\n"
]
}
],
"source": [
"url = \"https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html\"\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
"]\n",
"\n",
"html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
"html_header_splits = html_splitter.split_text_from_url(url)\n",
"print(html_header_splits[1].page_content[:500])"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

207
docs/docs/how_to/HTML_section_aware_splitter.ipynb
View File

@@ -1,207 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c95fcd15cd52c944",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"# How to split by HTML sections\n",
"## Description and motivation\n",
"Similar in concept to the [HTMLHeaderTextSplitter](/docs/how_to/HTML_header_metadata_splitter), the `HTMLSectionSplitter` is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk.\n",
"\n",
"It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures.\n",
"\n",
"Use `xslt_path` to provide an absolute path to transform the HTML so that it can detect sections based on provided tags. The default is to use the `converting_to_header.xslt` file in the `data_connection/document_transformers` directory. This is for converting the html to a format/layout that is easier to detect sections. For example, `span` based on their font size can be converted to header tags to be detected as a section.\n",
"\n",
"## Usage examples\n",
"### 1) How to split HTML strings:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "initial_id",
"metadata": {
"ExecuteTime": {
"end_time": "2023-10-02T18:57:49.208965400Z",
"start_time": "2023-10-02T18:57:48.899756Z"
},
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='Foo \\n Some intro text about Foo.', metadata={'Header 1': 'Foo'}),\n",
" Document(page_content='Bar main section \\n Some intro text about Bar. \\n Bar subsection 1 \\n Some text about the first subtopic of Bar. \\n Bar subsection 2 \\n Some text about the second subtopic of Bar.', metadata={'Header 2': 'Bar main section'}),\n",
" Document(page_content='Baz \\n Some text about Baz \\n \\n \\n Some concluding text about Foo', metadata={'Header 2': 'Baz'})]"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import HTMLSectionSplitter\n",
"\n",
"html_string = \"\"\"\n",
" <!DOCTYPE html>\n",
" <html>\n",
" <body>\n",
" <div>\n",
" <h1>Foo</h1>\n",
" <p>Some intro text about Foo.</p>\n",
" <div>\n",
" <h2>Bar main section</h2>\n",
" <p>Some intro text about Bar.</p>\n",
" <h3>Bar subsection 1</h3>\n",
" <p>Some text about the first subtopic of Bar.</p>\n",
" <h3>Bar subsection 2</h3>\n",
" <p>Some text about the second subtopic of Bar.</p>\n",
" </div>\n",
" <div>\n",
" <h2>Baz</h2>\n",
" <p>Some text about Baz</p>\n",
" </div>\n",
" <br>\n",
" <p>Some concluding text about Foo</p>\n",
" </div>\n",
" </body>\n",
" </html>\n",
"\"\"\"\n",
"\n",
"headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n",
"\n",
"html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
"html_header_splits = html_splitter.split_text(html_string)\n",
"html_header_splits"
]
},
{
"cell_type": "markdown",
"id": "e29b4aade2a0070c",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"### 2) How to constrain chunk sizes:\n",
"\n",
"`HTMLSectionSplitter` can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "6ada8ea093ea0475",
"metadata": {
"ExecuteTime": {
"end_time": "2023-10-02T18:57:51.016141300Z",
"start_time": "2023-10-02T18:57:50.647495400Z"
},
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='Foo \\n Some intro text about Foo.', metadata={'Header 1': 'Foo'}),\n",
" Document(page_content='Bar main section \\n Some intro text about Bar.', metadata={'Header 2': 'Bar main section'}),\n",
" Document(page_content='Bar subsection 1 \\n Some text about the first subtopic of Bar.', metadata={'Header 3': 'Bar subsection 1'}),\n",
" Document(page_content='Bar subsection 2 \\n Some text about the second subtopic of Bar.', metadata={'Header 3': 'Bar subsection 2'}),\n",
" Document(page_content='Baz \\n Some text about Baz \\n \\n \\n Some concluding text about Foo', metadata={'Header 2': 'Baz'})]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
"\n",
"html_string = \"\"\"\n",
" <!DOCTYPE html>\n",
" <html>\n",
" <body>\n",
" <div>\n",
" <h1>Foo</h1>\n",
" <p>Some intro text about Foo.</p>\n",
" <div>\n",
" <h2>Bar main section</h2>\n",
" <p>Some intro text about Bar.</p>\n",
" <h3>Bar subsection 1</h3>\n",
" <p>Some text about the first subtopic of Bar.</p>\n",
" <h3>Bar subsection 2</h3>\n",
" <p>Some text about the second subtopic of Bar.</p>\n",
" </div>\n",
" <div>\n",
" <h2>Baz</h2>\n",
" <p>Some text about Baz</p>\n",
" </div>\n",
" <br>\n",
" <p>Some concluding text about Foo</p>\n",
" </div>\n",
" </body>\n",
" </html>\n",
"\"\"\"\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
" (\"h3\", \"Header 3\"),\n",
" (\"h4\", \"Header 4\"),\n",
"]\n",
"\n",
"html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
"\n",
"html_header_splits = html_splitter.split_text(html_string)\n",
"\n",
"chunk_size = 500\n",
"chunk_overlap = 30\n",
"text_splitter = RecursiveCharacterTextSplitter(\n",
" chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
")\n",
"\n",
"# Split\n",
"splits = text_splitter.split_documents(html_header_splits)\n",
"splits"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

3
docs/docs/how_to/index.mdx
View File

@@ -143,8 +143,7 @@ What LangChain calls [LLMs](/docs/concepts/text_llms) are older forms of languag
[Text Splitters](/docs/concepts/text_splitters) take a document and split into chunks that can be used for retrieval.
- [How to: recursively split text](/docs/how_to/recursive_text_splitter)
- [How to: split by HTML headers](/docs/how_to/HTML_header_metadata_splitter)
- [How to: split by HTML sections](/docs/how_to/HTML_section_aware_splitter)
- [How to: split HTML](/docs/how_to/split_html)
- [How to: split by character](/docs/how_to/character_text_splitter)
- [How to: split code](/docs/how_to/code_splitter)
- [How to: split Markdown by headers](/docs/how_to/markdown_header_metadata_splitter)

963
docs/docs/how_to/split_html.ipynb Normal file
View File

@@ -0,0 +1,963 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "7fb27b941602401d91542211134fc71a",
"metadata": {},
"source": [
"# How to split HTML"
]
},
{
"cell_type": "markdown",
"id": "acae54e37e7d407bbb7b55eff062a284",
"metadata": {},
"source": [
"Splitting HTML documents into manageable chunks is essential for various text processing tasks such as natural language processing, search indexing, and more. In this guide, we will explore three different text splitters provided by LangChain that you can use to split HTML content effectively:\n",
"\n",
"- [**HTMLHeaderTextSplitter**](#using-htmlheadertextsplitter)\n",
"- [**HTMLSectionSplitter**](#using-htmlsectionsplitter)\n",
"- [**HTMLSemanticPreservingSplitter**](#using-htmlsemanticpreservingsplitter)\n",
"\n",
"Each of these splitters has unique features and use cases. This guide will help you understand the differences between them, why you might choose one over the others, and how to use them effectively."
]
},
{
"cell_type": "markdown",
"id": "e48a7480-5ec3-47e6-9e6a-f36c74d16f4f",
"metadata": {},
"source": [
"```\n",
"%pip install -qU langchain-text-splitters\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "9a63283cbaf04dbcab1f6479b197f3a8",
"metadata": {},
"source": [
"## Overview of the Splitters"
]
},
{
"cell_type": "markdown",
"id": "8dd0d8092fe74a7c96281538738b07e2",
"metadata": {},
"source": [
"### [HTMLHeaderTextSplitter](#using-htmlheadertextsplitter)\n",
"\n",
":::info\n",
"Useful when you want to preserve the hierarchical structure of a document based on its headings.\n",
":::\n",
"\n",
"**Description**: Splits HTML text based on header tags (e.g., `<h1>`, `<h2>`, `<h3>`, etc.), and adds metadata for each header relevant to any given chunk.\n",
"\n",
"**Capabilities**:\n",
"- Splits text at the HTML element level.\n",
"- Preserves context-rich information encoded in document structures.\n",
"- Can return chunks element by element or combine elements with the same metadata.\n",
"\n",
"___\n"
]
},
{
"cell_type": "markdown",
"id": "72eea5119410473aa328ad9291626812",
"metadata": {},
"source": [
"### [HTMLSectionSplitter](#using-htmlsectionsplitter)\n",
"\n",
":::info \n",
"Useful when you want to split HTML documents into larger sections, such as `<section>`, `<div>`, or custom-defined sections. \n",
":::\n",
"\n",
"**Description**: Similar to HTMLHeaderTextSplitter but focuses on splitting HTML into sections based on specified tags.\n",
"\n",
"**Capabilities**:\n",
"- Uses XSLT transformations to detect and split sections.\n",
"- Internally uses `RecursiveCharacterTextSplitter` for large sections.\n",
"- Considers font sizes to determine sections.\n",
"___"
]
},
{
"cell_type": "markdown",
"id": "8edb47106e1a46a883d545849b8ab81b",
"metadata": {},
"source": [
"### [HTMLSemanticPreservingSplitter](#using-htmlsemanticpreservingsplitter)\n",
"\n",
":::info \n",
"Ideal when you need to ensure that structured elements are not split across chunks, preserving contextual relevancy. \n",
":::\n",
"\n",
"**Description**: Splits HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components.\n",
"\n",
"**Capabilities**:\n",
"- Preserves tables, lists, and other specified HTML elements.\n",
"- Allows custom handlers for specific HTML tags.\n",
"- Ensures that the semantic meaning of the document is maintained.\n",
"- Built in normalization & stopword removal\n",
"\n",
"___"
]
},
{
"cell_type": "markdown",
"id": "10185d26023b46108eb7d9f57d49d2b3",
"metadata": {},
"source": [
"### Choosing the Right Splitter\n",
"\n",
"- **Use `HTMLHeaderTextSplitter` when**: You need to split an HTML document based on its header hierarchy and maintain metadata about the headers.\n",
"- **Use `HTMLSectionSplitter` when**: You need to split the document into larger, more general sections, possibly based on custom tags or font sizes.\n",
"- **Use `HTMLSemanticPreservingSplitter` when**: You need to split the document into chunks while preserving semantic elements like tables and lists, ensuring that they are not split and that their context is maintained."
]
},
{
"cell_type": "markdown",
"id": "19d42e40-015c-4bfa-b9ce-783e8377af2b",
"metadata": {},
"source": [
"| Feature | HTMLHeaderTextSplitter | HTMLSectionSplitter | HTMLSemanticPreservingSplitter |\n",
"|--------------------------------------------|------------------------|---------------------|-------------------------------|\n",
"| Splits based on headers | Yes | Yes | Yes |\n",
"| Preserves semantic elements (tables, lists) | No | No | Yes |\n",
"| Adds metadata for headers | Yes | Yes | Yes |\n",
"| Custom handlers for HTML tags | No | No | Yes |\n",
"| Preserves media (images, videos) | No | No | Yes |\n",
"| Considers font sizes | No | Yes | No |\n",
"| Uses XSLT transformations | No | Yes | No |"
]
},
{
"cell_type": "markdown",
"id": "8763a12b2bbd4a93a75aff182afb95dc",
"metadata": {},
"source": [
"## Example HTML Document\n",
"\n",
"Let's use the following HTML document as an example:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c9ca2682",
"metadata": {},
"outputs": [],
"source": [
"html_string = \"\"\"\n",
"<!DOCTYPE html>\n",
" <html lang='en'>\n",
" <head>\n",
" <meta charset='UTF-8'>\n",
" <meta name='viewport' content='width=device-width, initial-scale=1.0'>\n",
" <title>Fancy Example HTML Page</title>\n",
" </head>\n",
" <body>\n",
" <h1>Main Title</h1>\n",
" <p>This is an introductory paragraph with some basic content.</p>\n",
" \n",
" <h2>Section 1: Introduction</h2>\n",
" <p>This section introduces the topic. Below is a list:</p>\n",
" <ul>\n",
" <li>First item</li>\n",
" <li>Second item</li>\n",
" <li>Third item with <strong>bold text</strong> and <a href='#'>a link</a></li>\n",
" </ul>\n",
" \n",
" <h3>Subsection 1.1: Details</h3>\n",
" <p>This subsection provides additional details. Here's a table:</p>\n",
" <table border='1'>\n",
" <thead>\n",
" <tr>\n",
" <th>Header 1</th>\n",
" <th>Header 2</th>\n",
" <th>Header 3</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>Row 1, Cell 1</td>\n",
" <td>Row 1, Cell 2</td>\n",
" <td>Row 1, Cell 3</td>\n",
" </tr>\n",
" <tr>\n",
" <td>Row 2, Cell 1</td>\n",
" <td>Row 2, Cell 2</td>\n",
" <td>Row 2, Cell 3</td>\n",
" </tr>\n",
" </tbody>\n",
" </table>\n",
" \n",
" <h2>Section 2: Media Content</h2>\n",
" <p>This section contains an image and a video:</p>\n",
" <img src='example_image_link.mp4' alt='Example Image'>\n",
" <video controls width='250' src='example_video_link.mp4' type='video/mp4'>\n",
" Your browser does not support the video tag.\n",
" </video>\n",
"\n",
" <h2>Section 3: Code Example</h2>\n",
" <p>This section contains a code block:</p>\n",
" <pre><code data-lang=\"html\">\n",
" &lt;div&gt;\n",
" &lt;p&gt;This is a paragraph inside a div.&lt;/p&gt;\n",
" &lt;/div&gt;\n",
" </code></pre>\n",
"\n",
" <h2>Conclusion</h2>\n",
" <p>This is the conclusion of the document.</p>\n",
" </body>\n",
" </html>\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"id": "86c25b86",
"metadata": {},
"source": [
"## Using HTMLHeaderTextSplitter\n",
"\n",
"[HTMLHeaderTextSplitter](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html) is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the HTML element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n",
"\n",
"It is analogous to the [MarkdownHeaderTextSplitter](/docs/how_to/markdown_header_metadata_splitter) for markdown files.\n",
"\n",
"To specify what headers to split on, specify `headers_to_split_on` when instantiating `HTMLHeaderTextSplitter` as shown below."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "23361e55",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
" Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a list: \\nFirst item Second item Third item with bold text and a link'),\n",
" Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction', 'Header 3': 'Subsection 1.1: Details'}, page_content=\"This subsection provides additional details. Here's a table:\"),\n",
" Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
" Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block:'),\n",
" Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import HTMLHeaderTextSplitter\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
" (\"h3\", \"Header 3\"),\n",
"]\n",
"\n",
"html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
"html_header_splits = html_splitter.split_text(html_string)\n",
"html_header_splits"
]
},
{
"cell_type": "markdown",
"id": "f7e9cfef-5387-4ffe-b1a8-4b9214b9debd",
"metadata": {},
"source": [
"To return each element together with their associated headers, specify `return_each_element=True` when instantiating `HTMLHeaderTextSplitter`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "adb0da17-d1d5-4913-aacd-dc5d70db66cf",
"metadata": {},
"outputs": [],
"source": [
"html_splitter = HTMLHeaderTextSplitter(\n",
" headers_to_split_on,\n",
" return_each_element=True,\n",
")\n",
"html_header_splits_elements = html_splitter.split_text(html_string)"
]
},
{
"cell_type": "markdown",
"id": "fa781f86-d04f-4c09-a4a1-26aac88d4fc3",
"metadata": {},
"source": [
"Comparing with the above, where elements are aggregated by their headers:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "d9d7b61f-f927-49fc-a592-1b0f049d1a2f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n",
"page_content='This section introduces the topic. Below is a list: \n",
"First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n"
]
}
],
"source": [
"for element in html_header_splits[:2]:\n",
" print(element)"
]
},
{
"cell_type": "markdown",
"id": "4b5869e0-3a9b-4fa2-82ec-dbde33464a52",
"metadata": {},
"source": [
"Now each element is returned as a distinct `Document`:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "e2677a27-875a-4455-8e3f-6a6c7706be20",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n",
"page_content='This section introduces the topic. Below is a list:' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n",
"page_content='First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n"
]
}
],
"source": [
"for element in html_header_splits_elements[:3]:\n",
" print(element)"
]
},
{
"cell_type": "markdown",
"id": "24200c53-6a52-436e-ba4d-8d0514cfa87c",
"metadata": {},
"source": [
"### How to split from a URL or HTML file:\n",
"\n",
"To read directly from a URL, pass the URL string into the `split_text_from_url` method.\n",
"\n",
"Similarly, a local HTML file can be passed to the `split_text_from_file` method."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "581eeddf-7e88-48a7-999d-da56304e3522",
"metadata": {},
"outputs": [],
"source": [
"url = \"https://plato.stanford.edu/entries/goedel/\"\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
" (\"h3\", \"Header 3\"),\n",
" (\"h4\", \"Header 4\"),\n",
"]\n",
"\n",
"html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
"\n",
"# for local file use html_splitter.split_text_from_file(<path_to_file>)\n",
"html_header_splits = html_splitter.split_text_from_url(url)"
]
},
{
"cell_type": "markdown",
"id": "88ba9878-068e-434f-bdac-17972b2aa9ad",
"metadata": {},
"source": [
"### How to constrain chunk sizes:\n",
"\n",
"`HTMLHeaderTextSplitter`, which splits based on HTML headers, can be composed with another splitter which constrains splits based on character lengths, such as `RecursiveCharacterTextSplitter`.\n",
"\n",
"This can be done using the `.split_documents` method of the second splitter:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "3efc1fb8-f264-4ae6-883d-694a4d252f86",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),\n",
" Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),\n",
" Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),\n",
" Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),\n",
" Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
"\n",
"chunk_size = 500\n",
"chunk_overlap = 30\n",
"text_splitter = RecursiveCharacterTextSplitter(\n",
" chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
")\n",
"\n",
"# Split\n",
"splits = text_splitter.split_documents(html_header_splits)\n",
"splits[80:85]"
]
},
{
"cell_type": "markdown",
"id": "27363092-c6b2-4290-9e12-f6aece503bfb",
"metadata": {},
"source": [
"### Limitations\n",
"\n",
"There can be quite a bit of structural variation from one HTML document to another, and while `HTMLHeaderTextSplitter` will attempt to attach all \"relevant\" headers to any given chunk, it can sometimes miss certain headers. For example, the algorithm assumes an informational hierarchy in which headers are always at nodes \"above\" associated text, i.e. prior siblings, ancestors, and combinations thereof. In the following news article (as of the writing of this document), the document is structured such that the text of the top-level headline, while tagged \"h1\", is in a *distinct* subtree from the text elements that we'd expect it to be *\"above\"*&mdash;so we can observe that the \"h1\" element and its associated text do not show up in the chunk metadata (but, where applicable, we do see \"h2\" and its associated text): \n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "332043f6-ec39-4fcd-aa57-4dec5252684b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"No two El Niño winters are the same, but many have temperature and precipitation trends in common. \n",
"Average conditions during an El Niño winter across the continental US. \n",
"One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA. \n",
"Because the jet stream is essentially a river of air that storms flow through, they c\n"
]
}
],
"source": [
"url = \"https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html\"\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
"]\n",
"\n",
"html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
"html_header_splits = html_splitter.split_text_from_url(url)\n",
"print(html_header_splits[1].page_content[:500])"
]
},
{
"cell_type": "markdown",
"id": "aa8ba422",
"metadata": {},
"source": [
"## Using HTMLSectionSplitter\n",
"\n",
"Similar in concept to the [HTMLHeaderTextSplitter](#using-htmlheadertextsplitter), the `HTMLSectionSplitter` is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk. It lets you split HTML by sections.\n",
"\n",
"It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures.\n",
"\n",
"Use `xslt_path` to provide an absolute path to transform the HTML so that it can detect sections based on provided tags. The default is to use the `converting_to_header.xslt` file in the `data_connection/document_transformers` directory. This is for converting the html to a format/layout that is easier to detect sections. For example, `span` based on their font size can be converted to header tags to be detected as a section.\n",
"\n",
"### How to split HTML strings:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "65376c86",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title \\n This is an introductory paragraph with some basic content.'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\"Section 1: Introduction \\n This section introduces the topic. Below is a list: \\n \\n First item \\n Second item \\n Third item with bold text and a link \\n \\n \\n Subsection 1.1: Details \\n This subsection provides additional details. Here's a table: \\n \\n \\n \\n Header 1 \\n Header 2 \\n Header 3 \\n \\n \\n \\n \\n Row 1, Cell 1 \\n Row 1, Cell 2 \\n Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1 \\n Row 2, Cell 2 \\n Row 2, Cell 3\"),\n",
" Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content \\n This section contains an image and a video: \\n \\n \\n Your browser does not support the video tag.'),\n",
" Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example \\n This section contains a code block: \\n \\n <div>\\n <p>This is a paragraph inside a div.</p>\\n </div>'),\n",
" Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion \\n This is the conclusion of the document.')]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import HTMLSectionSplitter\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
"]\n",
"\n",
"html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
"html_header_splits = html_splitter.split_text(html_string)\n",
"html_header_splits"
]
},
{
"cell_type": "markdown",
"id": "5aa8627f-0af3-48c2-b9ed-bfbc46f8030d",
"metadata": {},
"source": [
"### How to constrain chunk sizes:\n",
"\n",
"`HTMLSectionSplitter` can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "5a9759b2-f6ff-413e-b42d-9d8967f3f3e6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title'),\n",
" Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some'),\n",
" Document(metadata={'Header 1': 'Main Title'}, page_content='some basic content.'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Section 1: Introduction'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='is a list:'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='First item \\n Second item'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Third item with bold text and a link'),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Subsection 1.1: Details'),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='This subsection provides additional details.'),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content=\"Here's a table:\"),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Header 1 \\n Header 2 \\n Header 3'),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 1 \\n Row 1, Cell 2'),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1'),\n",
" Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 2, Cell 2 \\n Row 2, Cell 3'),\n",
" Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content'),\n",
" Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
" Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Your browser does not support the video'),\n",
" Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='tag.'),\n",
" Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example'),\n",
" Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: \\n \\n <div>'),\n",
" Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='<p>This is a paragraph inside a div.</p>'),\n",
" Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='</div>'),\n",
" Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion'),\n",
" Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
" (\"h3\", \"Header 3\"),\n",
"]\n",
"\n",
"html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
"\n",
"html_header_splits = html_splitter.split_text(html_string)\n",
"\n",
"chunk_size = 50\n",
"chunk_overlap = 5\n",
"text_splitter = RecursiveCharacterTextSplitter(\n",
" chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
")\n",
"\n",
"# Split\n",
"splits = text_splitter.split_documents(html_header_splits)\n",
"splits"
]
},
{
"cell_type": "markdown",
"id": "6fb9f81a",
"metadata": {},
"source": [
"## Using HTMLSemanticPreservingSplitter\n",
"\n",
"The `HTMLSemanticPreservingSplitter` is designed to split HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components. This ensures that such elements are not split across chunks, causing loss of contextual relevancy such as table headers, list headers etc.\n",
"\n",
"This splitter is designed at its heart, to create contextually relevant chunks. General Recursive splitting with `HTMLHeaderTextSplitter` can cause tables, lists and other structered elements to be split in the middle, losing signifcant context and creating bad chunks.\n",
"\n",
"The `HTMLSemanticPreservingSplitter` is essential for splitting HTML content that includes structured elements like tables and lists, especially when it's critical to preserve these elements intact. Additionally, its ability to define custom handlers for specific HTML tags makes it a versatile tool for processing complex HTML documents.\n",
"\n",
"**IMPORTANT**: `max_chunk_size` is not a definite maximum size of a chunk, the calculation of max size, occurs when the preserved content is not apart of the chunk, to ensure it is not split. When we add the preserved data back in to the chunk, there is a chance the chunk size will exceed the `max_chunk_size`. This is crucial to ensure we maintain the structure of the original document\n",
"\n",
":::info \n",
"\n",
"Notes:\n",
"\n",
"1. We have defined a custom handler to re-format the contents of code blocks\n",
"2. We defined a deny list for specific html elements, to decompose them and their contents pre-processing\n",
"3. We have intentionally set a small chunk size to demonstrate the non-splitting of elements\n",
"\n",
":::"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "6cd119a8-c3d1-48a8-b569-469cd1607169",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),\n",
" Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3\"),\n",
" Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video: ![image:example_image_link.mp4](example_image_link.mp4) ![video:example_video_link.mp4](example_video_link.mp4)'),\n",
" Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: <code:html> <div> <p>This is a paragraph inside a div.</p> </div> </code>'),\n",
" Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# BeautifulSoup is required to use the custom handlers\n",
"from bs4 import Tag\n",
"from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
"\n",
"headers_to_split_on = [\n",
" (\"h1\", \"Header 1\"),\n",
" (\"h2\", \"Header 2\"),\n",
"]\n",
"\n",
"\n",
"def code_handler(element: Tag) -> str:\n",
" data_lang = element.get(\"data-lang\")\n",
" code_format = f\"<code:{data_lang}>{element.get_text()}</code>\"\n",
"\n",
" return code_format\n",
"\n",
"\n",
"splitter = HTMLSemanticPreservingSplitter(\n",
" headers_to_split_on=headers_to_split_on,\n",
" separators=[\"\\n\\n\", \"\\n\", \". \", \"! \", \"? \"],\n",
" max_chunk_size=50,\n",
" preserve_images=True,\n",
" preserve_videos=True,\n",
" elements_to_preserve=[\"table\", \"ul\", \"ol\", \"code\"],\n",
" denylist_tags=[\"script\", \"style\", \"head\"],\n",
" custom_handlers={\"code\": code_handler},\n",
")\n",
"\n",
"documents = splitter.split_text(html_string)\n",
"documents"
]
},
{
"cell_type": "markdown",
"id": "8c5413e2-3c50-435a-bda7-11e574a3fbab",
"metadata": {},
"source": [
"### Preserving Tables and Lists\n",
"In this example, we will demonstrate how the `HTMLSemanticPreservingSplitter` can preserve a table and a large list within an HTML document. The chunk size will be set to 50 characters to illustrate how the splitter ensures that these elements are not split, even when they exceed the maximum defined chunk size."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "8d97a2a6-9c73-4396-a922-f7a4eebe47f8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Document(metadata={'Header 1': 'Section 1'}, page_content='This section contains an important table and list'), Document(metadata={'Header 1': 'Section 1'}, page_content='that should not be split across chunks.'), Document(metadata={'Header 1': 'Section 1'}, page_content='Item Quantity Price Apples 10 $1.00 Oranges 5 $0.50 Bananas 50 $1.50'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='Additional text in subsection 1.1 that is'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='separated from the table and list. Here is a'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content=\"detailed list: Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.\")]\n"
]
}
],
"source": [
"from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
"\n",
"html_string = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <body>\n",
" <div>\n",
" <h1>Section 1</h1>\n",
" <p>This section contains an important table and list that should not be split across chunks.</p>\n",
" <table>\n",
" <tr>\n",
" <th>Item</th>\n",
" <th>Quantity</th>\n",
" <th>Price</th>\n",
" </tr>\n",
" <tr>\n",
" <td>Apples</td>\n",
" <td>10</td>\n",
" <td>$1.00</td>\n",
" </tr>\n",
" <tr>\n",
" <td>Oranges</td>\n",
" <td>5</td>\n",
" <td>$0.50</td>\n",
" </tr>\n",
" <tr>\n",
" <td>Bananas</td>\n",
" <td>50</td>\n",
" <td>$1.50</td>\n",
" </tr>\n",
" </table>\n",
" <h2>Subsection 1.1</h2>\n",
" <p>Additional text in subsection 1.1 that is separated from the table and list.</p>\n",
" <p>Here is a detailed list:</p>\n",
" <ul>\n",
" <li>Item 1: Description of item 1, which is quite detailed and important.</li>\n",
" <li>Item 2: Description of item 2, which also contains significant information.</li>\n",
" <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>\n",
" </ul>\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\"\n",
"\n",
"headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n",
"\n",
"splitter = HTMLSemanticPreservingSplitter(\n",
" headers_to_split_on=headers_to_split_on,\n",
" max_chunk_size=50,\n",
" elements_to_preserve=[\"table\", \"ul\"],\n",
")\n",
"\n",
"documents = splitter.split_text(html_string)\n",
"print(documents)"
]
},
{
"cell_type": "markdown",
"id": "e44bab7f-5a81-4ec4-a7d8-0413c8e15b33",
"metadata": {},
"source": [
"#### Explanation\n",
"In this example, the `HTMLSemanticPreservingSplitter` ensures that the entire table and the unordered list (`<ul>`) are preserved within their respective chunks. Even though the chunk size is set to 50 characters, the splitter recognizes that these elements should not be split and keeps them intact.\n",
"\n",
"This is particularly important when dealing with data tables or lists, where splitting the content could lead to loss of context or confusion. The resulting `Document` objects retain the full structure of these elements, ensuring that the contextual relevance of the information is maintained."
]
},
{
"cell_type": "markdown",
"id": "0c0a83c9-1177-4f64-9f94-688b5ec4fafd",
"metadata": {},
"source": [
"### Using a Custom Handler\n",
"The `HTMLSemanticPreservingSplitter` allows you to define custom handlers for specific HTML elements. Some platforms, have custom HTML tags that are not natively parsed by `BeautifulSoup`, when this occurs, you can utilize custom handlers to add the formatting logic easily. \n",
"\n",
"This can be particularly useful for elements that require special processing, such as `<iframe>` tags or specific 'data-' elements. In this example, we'll create a custom handler for `iframe` tags that converts them into Markdown-like links."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "7179e5f4-cb00-4ec9-8b87-46e0061544b9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Document(metadata={'Header 1': 'Section with Iframe'}, page_content='[iframe:https://example.com/embed](https://example.com/embed) Some text after the iframe'), Document(metadata={'Header 1': 'Section with Iframe'}, page_content=\". Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.\")]\n"
]
}
],
"source": [
"def custom_iframe_extractor(iframe_tag):\n",
" iframe_src = iframe_tag.get(\"src\", \"\")\n",
" return f\"[iframe:{iframe_src}]({iframe_src})\"\n",
"\n",
"\n",
"splitter = HTMLSemanticPreservingSplitter(\n",
" headers_to_split_on=headers_to_split_on,\n",
" max_chunk_size=50,\n",
" separators=[\"\\n\\n\", \"\\n\", \". \"],\n",
" elements_to_preserve=[\"table\", \"ul\", \"ol\"],\n",
" custom_handlers={\"iframe\": custom_iframe_extractor},\n",
")\n",
"\n",
"html_string = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <body>\n",
" <div>\n",
" <h1>Section with Iframe</h1>\n",
" <iframe src=\"https://example.com/embed\"></iframe>\n",
" <p>Some text after the iframe.</p>\n",
" <ul>\n",
" <li>Item 1: Description of item 1, which is quite detailed and important.</li>\n",
" <li>Item 2: Description of item 2, which also contains significant information.</li>\n",
" <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>\n",
" </ul>\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\"\n",
"\n",
"documents = splitter.split_text(html_string)\n",
"print(documents)"
]
},
{
"cell_type": "markdown",
"id": "f2f62dc1-89ff-4bac-8f79-97cff3d1af3a",
"metadata": {},
"source": [
"#### Explanation\n",
"In this example, we defined a custom handler for `iframe` tags that converts them into Markdown-like links. When the splitter processes the HTML content, it uses this custom handler to transform the `iframe` tags while preserving other elements like tables and lists. The resulting `Document` objects show how the iframe is handled according to the custom logic you provided.\n",
"\n",
"**Important**: When presvering items such as links, you should be mindful not to include `.` in your seperators, or leave seperators blank. `RecursiveCharacterTextSplitter` splits on full stop, which will cut links in half. Ensure you provide a seperator list with `. ` instead."
]
},
{
"cell_type": "markdown",
"id": "138ee7a2-200e-41a7-9e45-e15658a7b2e9",
"metadata": {},
"source": [
"### Using a custom handler to analyze an image with an LLM\n",
"\n",
"With custom handler's, we can also override the default processing for any element. A great example of this, is inserting semantic analysis of an image within a document, directly in the chunking flow.\n",
"\n",
"Since our function is called when the tag is discovered, we can override the `<img>` tag and turn off `preserve_images` to insert any content we would like to embed in our chunks."
]
},
{
"cell_type": "markdown",
"id": "18f8bd11-770c-4b88-a2a7-a7cf42e2f481",
"metadata": {},
"source": [
"```python\n",
"\"\"\"This example assumes you have helper methods `load_image_from_url` and an LLM agent `llm` that can process image data.\"\"\"\n",
"\n",
"from langchain.agents import AgentExecutor\n",
"\n",
"# This example needs to be replaced with your own agent\n",
"llm = AgentExecutor(...)\n",
"\n",
"\n",
"# This method is a placeholder for loading image data from a URL and is not implemented here\n",
"def load_image_from_url(image_url: str) -> bytes:\n",
" # Assuming this method fetches the image data from the URL\n",
" return b\"image_data\"\n",
"\n",
"\n",
"html_string = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <body>\n",
" <div>\n",
" <h1>Section with Image and Link</h1>\n",
" <p>\n",
" <img src=\"https://example.com/image.jpg\" alt=\"An example image\" />\n",
" Some text after the image.\n",
" </p>\n",
" <ul>\n",
" <li>Item 1: Description of item 1, which is quite detailed and important.</li>\n",
" <li>Item 2: Description of item 2, which also contains significant information.</li>\n",
" <li>Item 3: Description of item 3, another item that we don't want to split across chunks.</li>\n",
" </ul>\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\"\n",
"\n",
"\n",
"def custom_image_handler(img_tag) -> str:\n",
" img_src = img_tag.get(\"src\", \"\")\n",
" img_alt = img_tag.get(\"alt\", \"No alt text provided\")\n",
"\n",
" image_data = load_image_from_url(img_src)\n",
" semantic_meaning = llm.invoke(image_data)\n",
"\n",
" markdown_text = f\"[Image Alt Text: {img_alt} | Image Source: {img_src} | Image Semantic Meaning: {semantic_meaning}]\"\n",
"\n",
" return markdown_text\n",
"\n",
"\n",
"splitter = HTMLSemanticPreservingSplitter(\n",
" headers_to_split_on=headers_to_split_on,\n",
" max_chunk_size=50,\n",
" separators=[\"\\n\\n\", \"\\n\", \". \"],\n",
" elements_to_preserve=[\"ul\"],\n",
" preserve_images=False,\n",
" custom_handlers={\"img\": custom_image_handler},\n",
")\n",
"\n",
"documents = splitter.split_text(html_string)\n",
"\n",
"print(documents)\n",
"```\n",
"\n",
"```\n",
"[Document(metadata={'Header 1': 'Section with Image and Link'}, page_content='[Image Alt Text: An example image | Image Source: https://example.com/image.jpg | Image Semantic Meaning: semantic-meaning] Some text after the image'), \n",
"Document(metadata={'Header 1': 'Section with Image and Link'}, page_content=\". Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.\")]\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "f07de062-29da-4f30-82c2-ec48242f2a6e",
"metadata": {},
"source": [
"#### Explanation:\n",
"\n",
"With our custom handler written to extract the specific fields from a `<img>` element in HTML, we can further process the data with our agent, and insert the result directly into our chunk. It is important to ensure `preserve_images` is set to `False` otherwise the default processing of `<img>` fields will take place. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "728e0cfe-d7fd-4cc0-9c46-b2c02d335953",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

8
docs/vercel.json
View File

@@ -70,6 +70,14 @@
"source": "/docs/how_to/graph_prompting(/?)",
"destination": "/docs/tutorials/graph#few-shot-prompting"
},
{
"source": "/docs/how_to/HTML_header_metadata_splitter(/?)",
"destination": "/docs/how_to/split_html#using-htmlheadertextsplitter"
},
{
"source": "/docs/how_to/HTML_section_aware_splitter(/?)",
"destination": "/docs/how_to/split_html#using-htmlsectionsplitter"
},
{
"source": "/docs/tutorials/data_generation",
"destination": "https://python.langchain.com/v0.2/docs/tutorials/data_generation/"