github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-06-14 19:09:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix encoding issue in WebBaseLoader (#3602)

Browse Source 
The character code mismatches occurred when character information was
not included in the response header (In my case, a Japanese web page).
I solved this issue by changing the encoding setting to
apparent_encoding.
This commit is contained in:
Kohei Kumazaki 2023-04-29 12:56:33 +09:00 committed by GitHub
parent be7a8e0824
commit fa4c35e9e5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
langchain/document_loaders/web_base.py
View File

@ -169,6 +169,7 @@ class WebBaseLoader(BaseLoader):
self._check_parser(parser) self._check_parser(parser)
html_doc = self.session.get(url) html_doc = self.session.get(url)
html_doc.encoding = html_doc.apparent_encoding
return BeautifulSoup(html_doc.text, parser) return BeautifulSoup(html_doc.text, parser)
def scrape(self, parser: Union[str, None] = None) -> Any: def scrape(self, parser: Union[str, None] = None) -> Any: