Jens Madsen 8d9e9e013c refactor: extract token text splitter function (#5179) ...

# Token text splitter for sentence transformers

The current TokenTextSplitter only works with OpenAi models via the
`tiktoken` package. This is not clear from the name `TokenTextSplitter`.
In this (first PR) a token based text splitter for sentence transformer
models is added. In the future I think we should work towards injecting
a tokenizer into the TokenTextSplitter to make ti more flexible.
Could perhaps be reviewed by @dev2049

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>

2023-06-04 14:41:44 -07:00

..

_static

docs: Big Mendable Improvements (#4964)

2023-05-19 15:31:48 -07:00

additional_resources

docs: Deployments page moved into Ecosystem/ (#4949)

2023-05-21 21:18:22 -07:00

ecosystem

Add Chainlit to deployment options (#5314)

2023-05-27 21:12:53 -07:00

getting_started

added DeepLearing.AI course link (#5518)

2023-05-31 14:53:14 -07:00

integrations

Retitles Bedrock doc to appear in correct alphabetical order in site nav (#5639)

2023-06-04 14:39:25 -07:00

modules

refactor: extract token text splitter function (#5179)

2023-06-04 14:41:44 -07:00

reference

Documentation fixes (linting and broken links) (#5563)

2023-06-01 13:06:17 -07:00

templates

docs ecosystem/integrations update 3 (#5470)

2023-05-31 17:54:05 -07:00

tracing

py tracer fixes (#5377)

2023-05-30 18:47:06 -07:00

use_cases

minor refactor GenerativeAgentMemory (#5315)

2023-06-03 14:53:14 -07:00

conf.py

…

dependents.md

docs: added ecosystem/dependents page (#4941)

2023-05-18 13:11:08 -07:00

index.rst

docs: Deployments page moved into Ecosystem/ (#4949)

2023-05-21 21:18:22 -07:00

integrations.rst

docs: ecosystem/integrations update 1 (#5219)

2023-05-29 07:25:17 -07:00

make.bat

…

Makefile

…

reference.rst

docs: Deployments page moved into Ecosystem/ (#4949)

2023-05-21 21:18:22 -07:00

requirements.txt

…