mirror of https://github.com/csunny/DB-GPT.git synced 2025-09-28 21:12:13 +00:00

Files

alanchen d83704d233 fix(VectorStore): fix task concurrency and batch processing issue (#2671 )

- Refactor task execution to improve concurrency control
- Implement batch processing for vector deletion in Chroma store

# Description

1 ,When chunk a knowledge base file, if there are too many chunks ,
using asyncio.gather(*tasks) will cause the CPU to freeze
2 , When using the Chroma vector database, if the one-time deletion of
embedding documents ids exceeds 5461, it will fail

![0e64098babaa448ec139933fecf5776](https://github.com/user-attachments/assets/f9fa81c1-2914-44e6-8d31-a16d7717c6b3)


# How Has This Been Tested?

# Snapshots:

![ff0ca0d2b35be11dd19aa1983a196ba](https://github.com/user-attachments/assets/4db4202e-5c71-4c97-b075-465ea5419657)

![c8821b068b0c8b73757700d2ecaa23a](https://github.com/user-attachments/assets/b6bc28d0-9406-4b6b-92cf-6a9765288fd2)

Include snapshots for easier review.

# Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Any dependent changes have been merged and published in downstream
modules

2025-05-12 16:25:06 +08:00