github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-07-18 02:33:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix RecursiveUrlLoader (#8582)

Browse Source 
Description: the recursive url loader does not fully crawl for all urls
under base url
Maintainer: @baskaryan
This commit is contained in:
Ryan Sloan 2023-08-03 19:51:57 -04:00 committed by GitHub
parent f81e613086
commit b786335dd1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
libs/langchain/langchain/document_loaders/recursive_url_loader.py
View File

@ -92,9 +92,7 @@ class RecursiveUrlLoader(BaseLoader):
yield from loaded_link yield from loaded_link
else: else:
yield loaded_link yield loaded_link
# If the link is a directory (w/ children) then visit it yield from self.get_child_links_recursive(link, visited)
if link.endswith("/"):
yield from self.get_child_links_recursive(link, visited)
return visited return visited