mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-08 14:31:55 +00:00
docs: compound ecosystem and integrations (#4870)
# Docs: compound ecosystem and integrations **Problem statement:** We have a big overlap between the References/Integrations and Ecosystem/LongChain Ecosystem pages. It confuses users. It creates a situation when new integration is added only on one of these pages, which creates even more confusion. - removed References/Integrations page (but move all its information into the individual integration pages - in the next PR). - renamed Ecosystem/LongChain Ecosystem into Integrations/Integrations. I like the Ecosystem term. It is more generic and semantically richer than the Integration term. But it mentally overloads users. The `integration` term is more concrete. UPDATE: after discussion, the Ecosystem is the term. Ecosystem/Integrations is the page (in place of Ecosystem/LongChain Ecosystem). As a result, a user gets a single place to start with the individual integration.
This commit is contained in:
58
docs/integrations/unstructured.md
Normal file
58
docs/integrations/unstructured.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Unstructured
|
||||
|
||||
This page covers how to use the [`unstructured`](https://github.com/Unstructured-IO/unstructured)
|
||||
ecosystem within LangChain. The `unstructured` package from
|
||||
[Unstructured.IO](https://www.unstructured.io/) extracts clean text from raw source documents like
|
||||
PDFs and Word documents.
|
||||
|
||||
|
||||
This page is broken into two parts: installation and setup, and then references to specific
|
||||
`unstructured` wrappers.
|
||||
|
||||
## Installation and Setup
|
||||
|
||||
If you are using a loader that runs locally, use the following steps to get `unstructured` and
|
||||
its dependencies running locally.
|
||||
|
||||
- Install the Python SDK with `pip install "unstructured[local-inference]"`
|
||||
- Install the following system dependencies if they are not already available on your system.
|
||||
Depending on what document types you're parsing, you may not need all of these.
|
||||
- `libmagic-dev` (filetype detection)
|
||||
- `poppler-utils` (images and PDFs)
|
||||
- `tesseract-ocr`(images and PDFs)
|
||||
- `libreoffice` (MS Office docs)
|
||||
- `pandoc` (EPUBs)
|
||||
- If you are parsing PDFs using the `"hi_res"` strategy, run the following to install the `detectron2` model, which
|
||||
`unstructured` uses for layout detection:
|
||||
- `pip install "detectron2@git+https://github.com/facebookresearch/detectron2.git@e2ce8dc#egg=detectron2"`
|
||||
- If `detectron2` is not installed, `unstructured` will fallback to processing PDFs
|
||||
using the `"fast"` strategy, which uses `pdfminer` directly and doesn't require
|
||||
`detectron2`.
|
||||
|
||||
If you want to get up and running with less set up, you can
|
||||
simply run `pip install unstructured` and use `UnstructuredAPIFileLoader` or
|
||||
`UnstructuredAPIFileIOLoader`. That will process your document using the hosted Unstructured API.
|
||||
Note that currently (as of 1 May 2023) the Unstructured API is open, but it will soon require
|
||||
an API. The [Unstructured documentation page](https://unstructured-io.github.io/) will have
|
||||
instructions on how to generate an API key once they're available. Check out the instructions
|
||||
[here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image)
|
||||
if you'd like to self-host the Unstructured API or run it locally.
|
||||
|
||||
## Wrappers
|
||||
|
||||
### Data Loaders
|
||||
|
||||
The primary `unstructured` wrappers within `langchain` are data loaders. The following
|
||||
shows how to use the most basic unstructured data loader. There are other file-specific
|
||||
data loaders available in the `langchain.document_loaders` module.
|
||||
|
||||
```python
|
||||
from langchain.document_loaders import UnstructuredFileLoader
|
||||
|
||||
loader = UnstructuredFileLoader("state_of_the_union.txt")
|
||||
loader.load()
|
||||
```
|
||||
|
||||
If you instantiate the loader with `UnstructuredFileLoader(mode="elements")`, the loader
|
||||
will track additional metadata like the page number and text type (i.e. title, narrative text)
|
||||
when that information is available.
|
Reference in New Issue
Block a user