Matt Robinson 3c489be773 feat: optional post-processing for Unstructured loaders (#7850) ...

### Summary

Adds a post-processing method for Unstructured loaders that allows users
to optionally modify or clean extracted elements.

### Testing

```python
from langchain.document_loaders import UnstructuredFileLoader
from unstructured.cleaners.core import clean_extra_whitespace

loader = UnstructuredFileLoader(
    "./example_data/layout-parser-paper.pdf",
    mode="elements",
    post_processors=[clean_extra_whitespace],
)

docs = loader.load()
docs[:5]
```


### Reviewrs
  - @rlancemartin
  - @eyurtsev
  - @hwchase17

2023-07-17 12:13:05 -07:00

..

parsers

feat (documents): add a source code loader based on AST manipulation (#6486)

2023-06-27 15:58:47 -07:00

__init__.py

Add new iFixit document loader (#1333)

2023-02-27 20:40:20 -08:00

test_arxiv.py

Arxiv document loader (#3627)

2023-04-26 21:04:56 -07:00

test_bigquery.py

Harrison/big query (#2100)

2023-03-28 08:17:22 -07:00

test_bilibili.py

Remove unnecessary spaces from document object’s page_content of BiliBiliLoader (#4619)

2023-05-16 13:13:57 -04:00

test_blockchain.py

codespell: workflow, config + some (quite a few) typos fixed (#6785)

2023-07-12 16:20:08 -04:00

test_confluence.py

Several confluence loader improvements (#3300)

2023-04-23 15:06:10 -07:00

test_csv_loader.py

feat: Add UnstructuredCSVLoader for CSV files (#5844)

2023-06-07 19:18:01 -07:00

test_dataframe.py

rm pandas dependency (#2102)

2023-03-28 08:38:19 -07:00

test_duckdb.py

Harrison/duckdb (#2064)

2023-03-27 19:51:34 -07:00

test_email.py

feat: enable UnstructuredEmailLoader to process attachments (#6977)

2023-07-01 06:09:26 -07:00

test_embaas.py

Add embaas document extraction api endpoints (#6048)

2023-06-12 19:13:52 -07:00

test_excel.py

feat: add UnstructuredExcelLoader for .xlsx and .xls files (#5617)

2023-06-03 12:44:12 -07:00

test_facebook_chat.py

Refactor TelegramChatLoader and FacebookChatLoader classes and add tests (#3863)

2023-05-03 15:59:19 -07:00

test_fauna.py

Harrison/fauna loader (#5864)

2023-06-07 21:32:23 -07:00

test_figma.py

Harrison/figma doc loader (#1908)

2023-03-22 19:57:46 -07:00

test_gitbook.py

Harrison/gitbook (#2044)

2023-03-28 15:28:33 -07:00

test_github.py

DocumentLoader for GitHub (#5408)

2023-05-29 20:11:21 -07:00

test_ifixit.py

Add new iFixit document loader (#1333)

2023-02-27 20:40:20 -08:00

test_joplin.py

Add Joplin document loader (#5153)

2023-05-24 12:31:55 -07:00

test_json_loader.py

JSON loader (#4067)

2023-05-05 14:48:13 -07:00

test_language.py

feat (documents): add a source code loader based on AST manipulation (#6486)

2023-06-27 15:58:47 -07:00

test_larksuite.py

feat (documents): add LarkSuite document loader (#6420)

2023-06-27 23:08:05 -07:00

test_mastodon.py

Add Mastodon toots loader (#5036)

2023-05-22 16:43:07 -07:00

test_max_compute.py

add maxcompute (#5533)

2023-06-01 00:54:42 -07:00

test_modern_treasury.py

Dev2049/add modern treasury (#3924)

2023-05-01 20:28:02 -07:00

test_odt.py

feat: add loader for open office odt files (#4405)

2023-05-10 01:37:17 -07:00

test_org_mode.py

feat: Add UnstructuredOrgModeLoader (#6842)

2023-06-27 16:34:17 -07:00

test_pdf.py

Harrison/unstructured page number (#6464)

2023-06-19 22:31:43 -07:00

test_pyspark_dataframe_loader.py

Harrison/spark reader (#5405)

2023-05-29 20:23:17 -07:00

test_python.py

Add PythonLoader which auto-detects encoding of Python files (#3311)

2023-04-21 10:47:57 -07:00

test_rocksetdb.py

Integrate Rockset as a document loader (#7681)

2023-07-14 07:58:13 -07:00

test_rst.py

feat: Add UnstructuredRSTLoader (#6594)

2023-06-25 12:41:57 -07:00

test_sitemap.py

Harrison/sitemap local (#4704)

2023-05-14 22:04:38 -07:00

test_slack.py

Add Slack Directory Loader (#2841)

2023-04-13 21:31:59 -07:00

test_spreedly.py

Harrison/spreedly (#3937)

2023-05-01 20:56:56 -07:00

test_stripe.py

Dev2049/add modern treasury (#3924)

2023-05-01 20:28:02 -07:00

test_tsv.py

feat: Add UnstructuredTSVLoader (#7367)

2023-07-10 03:07:10 -04:00

test_unstructured.py

feat: optional post-processing for Unstructured loaders (#7850)

2023-07-17 12:13:05 -07:00

test_url_playwright.py

Added matching async load func to PlaywrightURLLoader (#5938)

2023-07-13 17:51:38 -04:00

test_url.py

add continue to fix 'continue_on_failure' parameter for URL doc loader (#2735)

2023-04-11 21:12:39 -07:00

test_whatsapp_chat.py

Fix WhatsAppChatLoader : Enable parsing additional formats (#6663)

2023-06-25 12:08:43 -07:00

test_xml.py

feat: Add UnstructuredXMLLoader for .xml files (#5955)

2023-06-10 16:24:42 -07:00

test_xorbits.py

Add Xorbits Dataframe as a Document Loader (#7319)

2023-07-10 04:24:47 -04:00