github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-04 20:46:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add Unstructured API loaders (#3906)

Browse Source 
### Summary

Adds `UnstructuredAPIFileLoaders` and `UnstructuredAPIFIleIOLoaders`
that partition documents through the Unstructured API. Defaults to the
URL for hosted Unstructured API, but can switch to a self hosted or
locally running API using the `url` kwarg. Currently, the Unstructured
API is open and does not require an API, but it will soon. A note was
added about that to the Unstructured ecosystem page.

### Testing


```python
from langchain.document_loaders import UnstructuredAPIFileIOLoader

filename = "fake-email.eml"

with open(filename, "rb") as f:
    loader = UnstructuredAPIFileIOLoader(file=f, file_filename=filename)
    docs = loader.load()

docs[0]
```

```python
from langchain.document_loaders import UnstructuredAPIFileLoader

filename = "fake-email.eml"
loader = UnstructuredAPIFileLoader(file_path=filename, mode="elements")
docs = loader.load()

docs[0]
```
This commit is contained in:
Matt Robinson
2023-05-01 23:37:35 -04:00
committed by GitHub
parent 13269fb583
commit c51dec5101
3 changed files with 88 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/ecosystem/unstructured.md
View File

@@ -10,6 +10,10 @@ This page is broken into two parts: installation and setup, and then references
`unstructured` wrappers.
## Installation and Setup
If you are using a loader that runs locally, use the following steps to get `unstructured` and
its dependencies running locally.
- Install the Python SDK with `pip install "unstructured[local-inference]"`
- Install the following system dependencies if they are not already available on your system.
Depending on what document types you're parsing, you may not need all of these.
@@ -25,6 +29,15 @@ This page is broken into two parts: installation and setup, and then references
using the `"fast"` strategy, which uses `pdfminer` directly and doesn't require
`detectron2`.
If you want to get up and running with less set up, you can
simply run `pip install unstructured` and use `UnstructuredAPIFileLoader` or
`UnstructuredAPIFileIOLoader`. That will process your document using the hosted Unstructured API.
Note that currently (as of 1 May 2023) the Unstructured API is open, but it will soon require
an API. The [Unstructured documentation page](https://unstructured-io.github.io/) will have
instructions on how to generate an API key once they're available. Check out the instructions
[here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image)
if you'd like to self-host the Unstructured API or run it locally.
## Wrappers
### Data Loaders