- Refactor task execution to improve concurrency control
- Implement batch processing for vector deletion in Chroma store
# Description
1 ,When chunk a knowledge base file, if there are too many chunks ,
using asyncio.gather(*tasks) will cause the CPU to freeze
2 , When using the Chroma vector database, if the one-time deletion of
embedding documents ids exceeds 5461, it will fail
# How Has This Been Tested?
# Snapshots:
Include snapshots for easier review.
# Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Any dependent changes have been merged and published in downstream
modules
- Add reranking support to improve the quality of retrieved chunks
- Implement RerankEmbeddingFactory and RerankEmbeddingsRanker
- Update RetrieverResource to use reranking when configured
- Modify retrieve method to apply reranking if needed
Close#2595
# Description
fix hybrid memory milvus create collection bug
# How Has This Been Tested?
switch milvus for hybrid storage type.
# Snapshots:
Include snapshots for easier review.
# Checklist:
- [ ] My code follows the style guidelines of this project
- [ ] I have already rebased the commits and make the commit message
conform to the project standard.
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Any dependent changes have been merged and published in downstream
modules
