mirror of
https://github.com/hwchase17/langchain.git
synced 2025-05-14 11:33:00 +00:00
63aa28e2a6
### Summary
Allows users to pass in `**unstructured_kwargs` to Unstructured document
loaders. Implemented with the `strategy` kwargs in mind, but will pass
in other kwargs like `include_page_breaks` as well. The two currently
supported strategies are `"hi_res"`, which is more accurate but takes
longer, and `"fast"`, which processes faster but with lower accuracy.
The `"hi_res"` strategy is the default. For PDFs, if `detectron2` is not
available and the user selects `"hi_res"`, the loader will fallback to
using the `"fast"` strategy.
### Testing
#### Make sure the `strategy` kwarg works
Run the following in iPython to verify that the `"fast"` strategy is
indeed faster.
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
%timeit loader.load()
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
%timeit loader.load()
```
On my system I get:
```python
In [3]: from langchain.document_loaders import UnstructuredFileLoader
In [4]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
In [5]: %timeit loader.load()
247 ms ± 369 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [6]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
In [7]: %timeit loader.load()
2.45 s ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
#### Make sure older versions of `unstructured` still work
Run `pip install unstructured==0.5.3` and then verify the following runs
without error:
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
loader.load()
```
|
||
|---|---|---|
| .. | ||
| example_data | ||
| airbyte_json.ipynb | ||
| azlyrics.ipynb | ||
| college_confidential.ipynb | ||
| CoNLL-U.ipynb | ||
| copypaste.ipynb | ||
| csv.ipynb | ||
| directory_loader.ipynb | ||
| email.ipynb | ||
| evernote.ipynb | ||
| facebook_chat.ipynb | ||
| gcs_directory.ipynb | ||
| gcs_file.ipynb | ||
| gitbook.ipynb | ||
| googledrive.ipynb | ||
| gutenberg.ipynb | ||
| hn.ipynb | ||
| html.ipynb | ||
| ifixit.ipynb | ||
| image.ipynb | ||
| imsdb.ipynb | ||
| markdown.ipynb | ||
| microsoft_word.ipynb | ||
| notebook.ipynb | ||
| notion.ipynb | ||
| obsidian.ipynb | ||
| pdf.ipynb | ||
| powerpoint.ipynb | ||
| readthedocs_documentation.ipynb | ||
| roam.ipynb | ||
| s3_directory.ipynb | ||
| s3_file.ipynb | ||
| srt.ipynb | ||
| telegram.ipynb | ||
| unstructured_file.ipynb | ||
| url.ipynb | ||
| web_base.ipynb | ||
| word_document.ipynb | ||
| youtube.ipynb | ||