- Refactor task execution to improve concurrency control
- Implement batch processing for vector deletion in Chroma store
# Description
1 ,When chunk a knowledge base file, if there are too many chunks ,
using asyncio.gather(*tasks) will cause the CPU to freeze
2 , When using the Chroma vector database, if the one-time deletion of
embedding documents ids exceeds 5461, it will fail
# How Has This Been Tested?
# Snapshots:
Include snapshots for easier review.
# Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Any dependent changes have been merged and published in downstream
modules
# Description
Closes#2642Closes#2637
# How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration
# Snapshots:
Include snapshots for easier review.
# Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] Any dependent changes have been merged and published in downstream
modules
fix https://github.com/eosphoros-ai/DB-GPT/issues/2636
# Description
Currently, the reranker model is not used for knowledge base recalls, In
the recall test function, the CFG.RERANK_MODEL is always none
In chat scenarios, knowledge base recalls are also not rerankered
# How Has This Been Tested?
# Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Any dependent changes have been merged and published in downstream
modules
- Add reranking support to improve the quality of retrieved chunks
- Implement RerankEmbeddingFactory and RerankEmbeddingsRanker
- Update RetrieverResource to use reranking when configured
- Modify retrieve method to apply reranking if needed
The logic for determining whether to use reranking and setting the `top_k` value now depends on the `app_config` instead of directly using `CFG.RERANK_MODEL` and `CFG.RERANK_TOP_K`. This change ensures that the application's specific configuration settings are respected.
Close#2595
# Description
fix hybrid memory milvus create collection bug
# How Has This Been Tested?
switch milvus for hybrid storage type.
# Snapshots:
Include snapshots for easier review.
# Checklist:
- [ ] My code follows the style guidelines of this project
- [ ] I have already rebased the commits and make the commit message
conform to the project standard.
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Any dependent changes have been merged and published in downstream
modules
# Description
Closes#2607
# How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration
# Snapshots:
Include snapshots for easier review.
# Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] Any dependent changes have been merged and published in downstream
modules
# Description
Support MCP authentication(SSE).
# How Has This Been Tested?
```python
from dbgpt.agent.resource import MCPToolPack
tools = MCPToolPack(
"http://127.0.0.1:8000/sse"
default_headers={"Authorization": "Bearer your_token"}
)
# Set the default headers for each server
tools2 = MCPToolPack(
"http://127.0.0.1:8000/sse"
headers = {
"http://127.0.0.1:8000/sse": {
"Authorization": "Bearer your_token"
}
}
)
```
# Snapshots:
Website:
# Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have already rebased the commits and make the commit message
conform to the project standard.
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] Any dependent changes have been merged and published in downstream
modules
