Jens Madsen 2c91f0d750 chore: spedd up integration test by using smaller model (#6044) ...

Adds a new parameter `relative_chunk_overlap` for the
`SentenceTransformersTokenTextSplitter` constructor. The parameter sets
the chunk overlap using a relative factor, e.g. for a model where the
token limit is 100, a `relative_chunk_overlap=0.5` implies that
`chunk_overlap=50`

Tag maintainers/contributors who might be interested:

 @hwchase17, @dev2049

2023-06-12 13:27:10 -07:00

3.5 KiB

Raw Permalink Blame History

View Raw