[Community]: requests_kwargs not being used in _fetch (#28646)

- **Description:** `requests_kwargs` is not being passed to `_fetch`
which is fetching pages asynchronously. In this PR, making sure that we
are passing `requests_kwargs` to `_fetch` just like `_scrape`.
- **Issue:** #28634

---------

Co-authored-by: Erick Friis <erick@langchain.dev>
This commit is contained in:
Mohammad Mohtashim 2024-12-12 04:46:54 +05:00 committed by GitHub
parent 8c37808d47
commit fa155a422f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -208,7 +208,10 @@ class WebBaseLoader(BaseLoader):
)
if not self.session.verify:
kwargs["ssl"] = False
async with session.get(url, **kwargs) as response:
async with session.get(
url, **(self.requests_kwargs | kwargs)
) as response:
if self.raise_for_status:
response.raise_for_status()
return await response.text()