mirror of
https://github.com/hwchase17/langchain.git
synced 2025-08-31 10:23:18 +00:00
[Minor] Remove tagger from spacy sentencizer (#7534)
@svlandeg gave me a tip for how to improve a bit on https://github.com/hwchase17/langchain/pull/7442 for some extra speed and memory gains. The tagger isn't needed for sentencization, so can be disabled too.
This commit is contained in:
@@ -47,7 +47,7 @@ def _make_spacy_pipeline_for_splitting(pipeline: str) -> Any: # avoid importing
|
||||
sentencizer = English()
|
||||
sentencizer.add_pipe("sentencizer")
|
||||
else:
|
||||
sentencizer = spacy.load(pipeline, disable=["ner"])
|
||||
sentencizer = spacy.load(pipeline, exclude=["ner", "tagger"])
|
||||
return sentencizer
|
||||
|
||||
|
||||
|
Reference in New Issue
Block a user