mirror of
https://github.com/hwchase17/langchain.git
synced 2026-01-29 21:30:18 +00:00
2c8b8114fa
The former link led to a site that explains that the docs have moved, but did not redirect the user to the actual site automatically. I just copied the provided url, checked that it works and updated the link to the current version. **Description:** Updated the link to Unstructured Docs at https://docs.unstructured.io **Issue:** #30315 **Dependencies:** None **Twitter handle:** @lahoramaker
120 lines
3.7 KiB
Plaintext
120 lines
3.7 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0c6c50fc-15e1-4767-925a-53a37c430b9b",
|
|
"metadata": {},
|
|
"source": [
|
|
"# How to load HTML\n",
|
|
"\n",
|
|
"The HyperText Markup Language or [HTML](https://en.wikipedia.org/wiki/HTML) is the standard markup language for documents designed to be displayed in a web browser.\n",
|
|
"\n",
|
|
"This covers how to load `HTML` documents into a LangChain [Document](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html#langchain_core.documents.base.Document) objects that we can use downstream.\n",
|
|
"\n",
|
|
"Parsing HTML files often requires specialized tools. Here we demonstrate parsing via [Unstructured](https://docs.unstructured.io) and [BeautifulSoup4](https://beautiful-soup-4.readthedocs.io/en/latest/), which can be installed via pip. Head over to the integrations page to find integrations with additional services, such as [Azure AI Document Intelligence](/docs/integrations/document_loaders/azure_document_intelligence) or [FireCrawl](/docs/integrations/document_loaders/firecrawl).\n",
|
|
"\n",
|
|
"## Loading HTML with Unstructured"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "617a5e2b-1e92-4bdd-bd04-95a4d2379410",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install unstructured"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "7d167ca3-c7c7-4ef0-b509-080629f0f482",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"[Document(page_content='My First Heading\\n\\nMy first paragraph.', metadata={'source': '../../docs/integrations/document_loaders/example_data/fake-content.html'})]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_community.document_loaders import UnstructuredHTMLLoader\n",
|
|
"\n",
|
|
"file_path = \"../../docs/integrations/document_loaders/example_data/fake-content.html\"\n",
|
|
"\n",
|
|
"loader = UnstructuredHTMLLoader(file_path)\n",
|
|
"data = loader.load()\n",
|
|
"\n",
|
|
"print(data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cc85f7e8-f62e-49bc-910e-d0b151c9d651",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Loading HTML with BeautifulSoup4\n",
|
|
"\n",
|
|
"We can also use `BeautifulSoup4` to load HTML documents using the `BSHTMLLoader`. This will extract the text from the HTML into `page_content`, and the page title as `title` into `metadata`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "06a5e555-8e1f-44a7-b921-4dd8aedd3bca",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install bs4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "0a2050a8-6df6-4696-9889-ba367d6f9caa",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"[Document(page_content='\\nTest Title\\n\\n\\nMy First Heading\\nMy first paragraph.\\n\\n\\n', metadata={'source': '../../docs/integrations/document_loaders/example_data/fake-content.html', 'title': 'Test Title'})]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_community.document_loaders import BSHTMLLoader\n",
|
|
"\n",
|
|
"loader = BSHTMLLoader(file_path)\n",
|
|
"data = loader.load()\n",
|
|
"\n",
|
|
"print(data)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.5"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|