Commit Graph

11 Commits

Author SHA1 Message Date
Harrison Chase
05d125ac23 cr 2023-02-09 23:44:14 -08:00
Andrew White
9011f690c6
Added PyPDF Loader and Splitter [Ready for Review] (#958)
Per discussion on Discord. This adds a PDF reader that uses `PyPDF` - a
simple PDF reader. It also tracks page numbers in a per split metadata.
Here's an example:

```python
from langchain.document_loaders import PagedPDFSplitter
from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings

loader = PagedPDFSplitter(chunk_size=250)
splits, metadatas = loader.load_and_split("examples/example_data/layout-parser-paper.pdf")

faiss_index = FAISS.from_texts(splits, OpenAIEmbeddings(), metadatas=metadatas)
docs = faiss_index.similarity_search("How will the community be engaged?", k=2)
for doc in docs:
    print(doc.metadata["pages"] + ":", doc.page_content)
```

## TODO

- [x] Learn where to add `pypdf` as dependency for building docs
- [x] Add unit test?

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-02-09 23:33:18 -08:00
Harrison Chase
01fa2d8117
Harrison/youtube fixes (#955)
Co-authored-by: Ji <jizhang.work@gmail.com>
Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>
2023-02-09 08:12:22 -08:00
zanderchase
8e126bc9bd
adding webpage loading logic (#942) 2023-02-09 07:52:50 -08:00
Harrison Chase
3e1901e1aa
gutenberg books (#946)
Co-authored-by: zanderchase <zander@unfold.ag>
Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>
2023-02-08 12:00:47 -08:00
Harrison Chase
44ecec3896
Harrison/add roam loader (#939) 2023-02-08 00:35:33 -08:00
Harrison Chase
637c0d6508
Harrison/obsidian (#920) 2023-02-06 22:21:16 -08:00
Ankush Gola
6bd1529cb7
add GoogleDriveLoader (#914)
only deal with docs files for now
2023-02-06 21:44:35 -08:00
Harrison Chase
2ec25ddd4c
add unstructured examples (#913) 2023-02-06 18:13:46 -08:00
Harrison Chase
71e662e88d
update docs (#905) 2023-02-06 00:26:20 -08:00
Harrison Chase
53d56d7650
Harrison/unstructured support (#903) 2023-02-05 23:02:07 -08:00