github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-01 11:02:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Minor] Remove tagger from spacy sentencizer (#7534)

Browse Source 
@svlandeg gave me a tip for how to improve a bit on
https://github.com/hwchase17/langchain/pull/7442 for some extra speed
and memory gains. The tagger isn't needed for sentencization, so can be
disabled too.
This commit is contained in:
Jona Sassenhagen
2023-07-11 22:43:46 +02:00
committed by GitHub
parent 1f83b5f47e
commit 0ea7224535
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
langchain/text_splitter.py
View File

@@ -47,7 +47,7 @@ def _make_spacy_pipeline_for_splitting(pipeline: str) -> Any: # avoid importing
sentencizer = English() sentencizer = English()
sentencizer.add_pipe("sentencizer") sentencizer.add_pipe("sentencizer")
else: else:
sentencizer = spacy.load(pipeline, disable=["ner"]) sentencizer = spacy.load(pipeline, exclude=["ner", "tagger"])
return sentencizer return sentencizer