Eugene Yurtsev d56313acba Improve effeciency of TextSplitter.split_documents, iterate once (#5111) ...

# Improve TextSplitter.split_documents, collect page_content and
metadata in one iteration

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:

@eyurtsev In the case where documents is a generator that can only be
iterated once making this change is a huge help. Otherwise a silent
issue happens where metadata is empty for all documents when documents
is a generator. So we expand the argument from `List[Document]` to
`Union[Iterable[Document], Sequence[Document]]`

---------

Co-authored-by: Steven Tartakovsky <tartakovsky.developer@gmail.com>

2023-05-22 23:00:24 -04:00

..

agents

Improving Resilience of MRKL Agent (#5014)

2023-05-22 11:08:08 -07:00

callbacks

[Breaking] Refactor Base Tracer(#4549)

2023-05-13 17:23:56 +00:00

chains

Callbacks Refactor [base] (#3256)

2023-04-30 11:14:09 -07:00

chat_models

Add ChatModel, LLM, and Embeddings for Google's PaLM APIs (#3575)

2023-05-01 15:23:16 -07:00

client

Separate Runner Functions from Client (#5079)

2023-05-22 05:28:47 +00:00

data

…

docstore

Add DocstoreFn - lookup doc via arbitrary function (#3760)

2023-04-28 19:50:32 -07:00

document_loaders

Harrison/psychic (#5063)

2023-05-21 09:13:20 -07:00

evaluation

Adding an in-context QA evaluation chain + chain of thought reasoning chain for improved accuracy (#2444)

2023-04-06 22:32:41 -07:00

examples

feat #4479: TextLoader auto detect encoding and improved exceptions (#4927)

2023-05-18 09:55:14 -04:00

llms

Add Invocation Params (#4509)

2023-05-11 15:34:06 -07:00

memory

Zep memory (#4898)

2023-05-17 20:01:01 -07:00

output_parsers

Harrison/json new line (#4646)

2023-05-13 21:46:33 -07:00

prompts

fix prompt saving (#4987)

2023-05-20 08:21:52 -07:00

retrievers

Zep Retriever - Vector Search Over Chat History (#4533)

2023-05-18 16:27:18 -07:00

tools

PowerBI major refinement in working of tool and tweaks in the rest (#5090)

2023-05-22 11:58:28 -07:00

utilities

Fix graphql tool (#4984)

2023-05-19 15:27:50 -07:00

vectorstores

fix #3884 (#3475)

2023-04-24 19:54:15 -07:00

__init__.py

…

conftest.py

Add pytest --only-extended and --only-core options (#4494)

2023-05-12 11:35:22 -04:00

test_bash.py

Add Mastodon toots loader (#5036)

2023-05-22 16:43:07 -07:00

test_depedencies.py

Catch changes to test group (#4802)

2023-05-16 14:48:56 -04:00

test_document_transformers.py

Contextual compression retriever (#2915)

2023-04-20 17:01:14 -07:00

test_formatting.py

…

test_math_utils.py

add get_top_k_cosine_similarity method to get max top k score and index (#5059)

2023-05-22 11:55:48 -07:00

test_pytest_config.py

Block sockets for unit-tests (#4803)

2023-05-16 14:41:24 -04:00

test_python.py

option for csv agent to not include df in prompt (#4610)

2023-05-12 21:55:22 -07:00

test_schema.py

[simple][test] Added test case for schema.py (#3692)

2023-04-28 20:42:24 -07:00

test_sql_database_schema.py

Suppress duckdb warning in unit tests explicitly (#3653)

2023-04-27 14:29:41 -04:00

test_sql_database.py

…

test_text_splitter.py

Improve effeciency of TextSplitter.split_documents, iterate once (#5111)

2023-05-22 23:00:24 -04:00