Introduces SeleniumURLLoader for JavaScript-Dependent Web Page Data Retrieval (#2291)

### Summary
This PR introduces a `SeleniumURLLoader` which, similar to
`UnstructuredURLLoader`, loads data from URLs. However, it utilizes
`selenium` to fetch page content, enabling it to work with
JavaScript-rendered pages. The `unstructured` library is also employed
for loading the HTML content.

### Testing
```bash
pip install selenium
pip install unstructured
```

```python
from langchain.document_loaders import SeleniumURLLoader

urls = [
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "https://goo.gl/maps/NDSHwePEyaHMFGwh8"
]

loader = SeleniumURLLoader(urls=urls)
data = loader.load()
```
This commit is contained in:
Kevin Huang
2023-04-02 17:05:00 -04:00
committed by GitHub
parent 00d3ec5ed8
commit e4cfaa5680
3 changed files with 180 additions and 0 deletions

View File

@@ -52,6 +52,66 @@
"source": [
"data = loader.load()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f3afa135",
"metadata": {},
"source": [
"# Selenium URL Loader\n",
"\n",
"This covers how to load HTML documents from a list of URLs using the `SeleniumURLLoader`.\n",
"\n",
"Using selenium allows us to load pages that require JavaScript to render.\n",
"\n",
"## Setup\n",
"\n",
"To use the `SeleniumURLLoader`, you will need to install `selenium` and `unstructured`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5fc50835",
"metadata": {},
"outputs": [],
"source": [
"from langchain.document_loaders import SeleniumURLLoader"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "24e896ce",
"metadata": {},
"outputs": [],
"source": [
"urls = [\n",
" \"https://www.youtube.com/watch?v=dQw4w9WgXcQ\",\n",
" \"https://goo.gl/maps/NDSHwePEyaHMFGwh8\"\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "60a29397",
"metadata": {},
"outputs": [],
"source": [
"loader = SeleniumURLLoader(urls=urls)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0090cd57",
"metadata": {},
"outputs": [],
"source": [
"data = loader.load()"
]
}
],
"metadata": {