github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-08 14:31:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

community[patch]: Add remove_comments option (default True): do not extract html comments (#13259)

Browse Source 
- **Description:** add `remove_comments` option (default: True): do not
extract html _comments_,
  - **Issue:** None,
  - **Dependencies:** None,
  - **Tag maintainer:** @nfcampos ,
  - **Twitter handle:** peter_v

I ran `make format`, `make lint` and `make test`.

Discussion: I my use case, I prefer to not have the comments in the
extracted text:
* e.g. from a Google tag that is added in the html as comment
* e.g. content that the authors have temporarily hidden to make it non
visible to the regular reader

Removing the comments makes the extracted text more alike the intended
text to be seen by the reader.


**Choice to make:** do we prefer to make the default for this
`remove_comments` option to be True or False?
I have changed it to True in a second commit, since that is how I would
prefer to use it by default. Have the
cleaned text (without technical Google tags etc.) and also closer to the
actually visible and intended content.
I am not sure what is best aligned with the conventions of langchain in
general ...


INITIAL VERSION (new version above):
~**Choice to make:** do we prefer to make the default for this
`ignore_comments` option to be True or False?
I have set it to False now to be backwards compatible. On the other
hand, I would use it mostly with True.
I am not sure what is best aligned with the conventions of langchain in
general ...~

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
This commit is contained in:
Peter Vandenabeele
2024-04-02 02:19:12 +02:00
committed by GitHub
parent 4f70bc119d
commit e830a4e731
2 changed files with 53 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
libs/community/tests/unit_tests/document_transformers/test_beautiful_soup_transformer.py
View File

@@ -230,3 +230,36 @@ def test_invalid_html() -> None:
)
assert docs_transformed[0].page_content == "First heading."
assert docs_transformed[1].page_content == ""
@pytest.mark.requires("bs4")
def test_remove_comments() -> None:
bs_transformer = BeautifulSoupTransformer()
html_with_comments = (
"<html><!-- Google tag (gtag.js) --><p>First paragraph.</p</html>"
)
documents = [
Document(page_content=html_with_comments),
]
docs_transformed = bs_transformer.transform_documents(
documents, tags_to_extract=["html"], remove_comments=True
)
assert docs_transformed[0].page_content == "First paragraph."
@pytest.mark.requires("bs4")
def test_do_not_remove_comments() -> None:
bs_transformer = BeautifulSoupTransformer()
html_with_comments = (
"<html><!-- Google tag (gtag.js) --><p>First paragraph.</p</html>"
)
documents = [
Document(page_content=html_with_comments),
]
docs_transformed = bs_transformer.transform_documents(
documents,
tags_to_extract=["html"],
)
assert docs_transformed[0].page_content == "Google tag (gtag.js) First paragraph."