github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-06-20 22:03:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

text_splitters: add request parameters for function HTMLHeaderTextSplitter.split_text… (#24178)

Browse Source 
**Description:**

The `split_text_from_url` method of `HTMLHeaderTextSplitter` does not
include parameters like `timeout` when using `requests` to send a
request. Therefore, I suggest adding a `kwargs` parameter to the
function, which can be passed as arguments to `requests.get()`
internally, allowing control over the `get` request.

---------

Co-authored-by: Chester Curme <chester.curme@gmail.com>
This commit is contained in:
blueoom 2024-07-16 00:43:56 +08:00 committed by GitHub
parent 9d0c1d2dc9
commit d895614d19
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
libs/text-splitters/langchain_text_splitters/html.py
View File

@ -71,13 +71,15 @@ class HTMLHeaderTextSplitter:
for chunk in aggregated_chunks for chunk in aggregated_chunks
] ]
def split_text_from_url(self, url: str) -> List[Document]: def split_text_from_url(self, url: str, **kwargs: Any) -> List[Document]:
"""Split HTML from web URL """Split HTML from web URL
Args: Args:
url: web URL url: web URL
**kwargs: Arbitrary additional keyword arguments. These are usually passed
to the fetch url content request.
""" """
r = requests.get(url) r = requests.get(url, **kwargs)
return self.split_text_from_file(BytesIO(r.content)) return self.split_text_from_file(BytesIO(r.content))
def split_text(self, text: str) -> List[Document]: def split_text(self, text: str) -> List[Document]: