community: better support of pathlib paths in document loaders (#18396)

So this arose from the
https://github.com/langchain-ai/langchain/pull/18397 problem of document
loaders not supporting `pathlib.Path`.

This pull request provides more uniform support for Path as an argument.
The core ideas for this upgrade: 
- if there is a local file path used as an argument, it should be
supported as `pathlib.Path`
- if there are some external calls that may or may not support Pathlib,
the argument is immidiately converted to `str`
- if there `self.file_path` is used in a way that it allows for it to
stay pathlib without conversion, is is only converted for the metadata.

Twitter handle: https://twitter.com/mwmajewsk
This commit is contained in:
mwmajewsk
2024-03-26 16:51:52 +01:00
committed by GitHub
parent 94b869a974
commit f7a1fd91b8
32 changed files with 147 additions and 80 deletions

View File

@@ -1,4 +1,5 @@
import logging
from pathlib import Path
from typing import Dict, Iterator, Union
from langchain_core.documents import Document
@@ -13,7 +14,7 @@ class BSHTMLLoader(BaseLoader):
def __init__(
self,
file_path: str,
file_path: Union[str, Path],
open_encoding: Union[str, None] = None,
bs_kwargs: Union[dict, None] = None,
get_text_separator: str = "",
@@ -57,7 +58,7 @@ class BSHTMLLoader(BaseLoader):
title = ""
metadata: Dict[str, Union[str, None]] = {
"source": self.file_path,
"source": str(self.file_path),
"title": title,
}
yield Document(page_content=text, metadata=metadata)