github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-07-05 12:48:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: ReadTheDocs loader main content filter (#2609)

Browse Source 
It seems the main element wrapper changed in ReadTheDocs website or for
some reason it's different for me ?

This adds an extra filter for the main content wrapper if the first one
returns no text.


![2023-04-09-043315_1178x873_scrot](https://user-images.githubusercontent.com/210457/230751369-24b69cb9-1601-4540-b5f3-d115165f55f6.jpg)

Co-authored-by: blob42 <spike@w530>
This commit is contained in:
blob42 2023-04-09 19:51:56 +00:00 committed by GitHub
parent aaac7071a3
commit 54b1645d13
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
langchain/document_loaders/readthedocs.py
View File

@ -45,6 +45,10 @@ class ReadTheDocsLoader(BaseLoader):
def _clean_data(data: str) -> str: def _clean_data(data: str) -> str:
soup = BeautifulSoup(data, **self.bs_kwargs) soup = BeautifulSoup(data, **self.bs_kwargs)
text = soup.find_all("main", {"id": "main-content"}) text = soup.find_all("main", {"id": "main-content"})
if len(text) == 0:
text = soup.find_all("div", {"role": "main"})
if len(text) != 0: if len(text) != 0:
text = text[0].get_text() text = text[0].get_text()
else: else: