github/DB-GPT
1
0
Fork 0
mirror of https://github.com/csunny/DB-GPT.git synced 2025-09-04 10:34:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat:Add Knowledge Process Workflow (#2210)

Browse Source
This commit is contained in:
Aries-ckt
2024-12-18 11:16:30 +08:00
committed by GitHub
parent 3745d7411d
commit b05febbf77
23 changed files with 7217 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
docs/docs/awel/awel_tutorial/templates/Embedding.md Normal file
View File

@@ -0,0 +1,55 @@
# Embedding Process Workflow
# Introduction
the traditional knowledge extraction preparation process of Native RAG aims at the process of turning documents into databases, including reading unstructured documents-> knowledge slices-> document slices turning-> import vector databases.
# Applicable Scenarios
+ supports simple intelligent question and answer scenarios and recalls context information through semantic similarity.
+ Users can cut and add existing embedded processing processes according to their own business scenarios.
# How to use
+ enter the AWEL interface and add a workflow
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734354927468-feed0ac7-e0fe-45e8-b85c-aba170084f82.png)
+ import Knowledge Processing Template
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734358060884-672d3157-a2ee-498b-887e-ea51f1caddae.png)
+ adjust parameters and save
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734358170081-32d38282-7765-4bbf-9bf7-c068550907d1.png)
- `document knowledge loader operator `: Knowledge loading factory, by loading the specified document type, find the corresponding document processor for document content parsing.
- `Document Chunk Manager operator `: Slice the loaded document content according to the specified slicing parameters.
- `Vector storage machining operator `: You can connect different vector databases for vector storage, and you can also connect different Embedding models and services for vector extraction.
+ Register Post as http request
```bash
curl --location --request POST 'http://localhost:5670/api/v1/awel/trigger/rag/knowledge/embedding/process' \
--header 'Content-Type: application/json' \
--data-raw '{}'
```
```bash
[
{
"content": "\"What is AWEL?\": Agentic Workflow Expression Language(AWEL) is a set of intelligent agent workflow expression language specially designed for large model application\ndevelopment. It provides great functionality and flexibility. Through the AWEL API, you can focus on the development of business logic for LLMs applications\nwithout paying attention to cumbersome model and environment details. \nAWEL adopts a layered API design. AWEL's layered API design architecture is shown in the figure below. \n<p align=\"left\">\n<img src={'/img/awel.png'} width=\"480px\"/>\n</p>",
"metadata": {
"Header1": "What is AWEL?",
"source": "../../docs/docs/awel/awel.md"
},
"chunk_id": "c1ffa671-76d0-4c7a-b2dd-0b08dfd37712",
"chunk_name": "",
"score": 0.0,
"summary": "",
"separator": "\n",
"retriever": null
},...
]
```

46
docs/docs/awel/awel_tutorial/templates/Hybrid_Workflow.md Normal file
View File

@@ -0,0 +1,46 @@
# Hybrid Knowledge Process Workflow
# Introduction
At present, the DB-GPT knowledge base provides knowledge processing capabilities such as `document uploading` ->` parsing` ->` chunking` ->` Embedding` -> `Knowledge Graph triple extraction `-> `vector database storage` -> `graph database storage`, but it does not have the ability to extract complex information from documents, including vector extraction and Knowledge Graph extraction from document blocks at the same time. The hybrid knowledge processing template defines complex knowledge processing workflow, it also supports document vector extraction, Keyword extraction and Knowledge Graph extraction.
# Applicable Scenarios
+ It is not limited to the traditional, single knowledge processing process (only Embedding processing or knowledge graph extraction processing), knowledge processing workflow implements Embedding and Knowledge Graph extraction at the same time, as a mixed knowledge recall retrieval data storage.
+ Users can tailor and add existing knowledge processing processes based on their own business scenarios.
# How to use
+ Enter the AWEL interface and add a workflow
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734354927468-feed0ac7-e0fe-45e8-b85c-aba170084f82.png)
+ Import Knowledge Processing Template
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734357236704-5a15be65-3d11-4406-98d7-efb82e5142dc.png)
+ Adjust parameters and save
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734355123947-3e252e59-2b2a-4bca-adef-13a93ee6cdf3.png)
- `Document knowledge loading operator `: Knowledge loading factory, by loading the specified document type, find the corresponding document processor for document content parsing.
- `Document Chunk slicing operator `: Slice the loaded document content according to the specified slicing parameters.
- `Knowledge Processing branch operator `: You can connect different knowledge processing processes, including knowledge map processing processes, vector processing processes, and keyword processing processes.
- `Vector storage machining operator `: You can connect different vector databases for vector storage, and you can also connect different Embedding models and services for vector extraction.
- `Knowledge Graph processing operator `: You can connect different knowledge graph processing operators, including native knowledge graph processing operators and community summary Knowledge Graph processing operators. You can also specify different graph databases for storage. Currently, TuGraph databases are supported.
- `Result aggregation operator `: Summarize the results of vector extraction and Knowledge Graph extraction.
+ Register Post as http request
```bash
curl --location --request POST 'http://localhost:5670/api/v1/awel/trigger/rag/knowledge/hybrid/process' \
--header 'Content-Type: application/json' \
--data-raw '{}'
```
```bash
[
"async persist vector store success 1 chunks.",
"async persist graph store success 1 chunks."
]
```

58
docs/docs/awel/awel_tutorial/templates/Knowledge_Graph.md Normal file
View File

@@ -0,0 +1,58 @@
# Knowledge Graph Process Workflow
# Introduction
Unlike traditional Native RAG, which requires vectors as data carriers, GraphRAG requires triple extraction (entity -> relationship -> entity) to build a knowledge graph, so the entire knowledge processing can also be regarded as the process of building a knowledge graph.
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734357331126-a3a96fd7-c8fb-4208-8e3b-be798d1b73b4.png)
# Applicable Scenarios
+ It is necessary to use GraphRAG ability to mine the relationship between knowledge for multi-step reasoning.
+ Make up for the lack of integrity of Naive RAG in the recall context.
# How to use
+ Enter the AWEL interface and add a workflow
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734354927468-feed0ac7-e0fe-45e8-b85c-aba170084f82.png)
+ Import Knowledge Processing Template
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734356276305-a6e03aff-ba89-40c4-be2d-f88dff29d0f5.png)
+ Adjust parameters and save
![](https://intranetproxy.alipay.com/skylark/lark/0/2024/png/26456775/1734356745373-4e449611-d0bc-4735-b142-0aebafaa34d6.png)
- `document knowledge loading operator `: Knowledge loading factory, by loading the specified document type, find the corresponding document processor for document content parsing.
- `Document Chunk slicing operator `: Slice the loaded document content according to the specified slicing parameters.
- `Knowledge Graph processing operator `: You can connect different knowledge graph processing operators, including native knowledge graph processing operators and community summary Knowledge Graph processing operators. You can also specify different graph databases for storage. Currently, TuGraph databases are supported.
+ Register Post as http request
```bash
curl --location --request POST 'http://localhost:5670/api/v1/awel/trigger/rag/knowledge/kg/process' \
--header 'Content-Type: application/json' \
--data-raw '{}'
```
```bash
[
{
"content": "\"What is AWEL?\": Agentic Workflow Expression Language(AWEL) is a set of intelligent agent workflow expression language specially designed for large model application\ndevelopment. It provides great functionality and flexibility. Through the AWEL API, you can focus on the development of business logic for LLMs applications\nwithout paying attention to cumbersome model and environment details. \nAWEL adopts a layered API design. AWEL's layered API design architecture is shown in the figure below. \n<p align=\"left\">\n<img src={'/img/awel.png'} width=\"480px\"/>\n</p>",
"metadata": {
"Header1": "What is AWEL?",
"source": "../../docs/docs/awel/awel.md"
},
"chunk_id": "c1ffa671-76d0-4c7a-b2dd-0b08dfd37712",
"chunk_name": "",
"score": 0.0,
"summary": "",
"separator": "\n",
"retriever": null
},...
]
```

18
docs/sidebars.js
View File

@@ -186,6 +186,24 @@ const sidebars = {
id: "awel/awel_tutorial/advanced_guide/4.1_lifecycle"
}
]
},,
{
type: "category",
label: "5. AWEL Template",
collapsed: false,
collapsible: false,
items: [
{
type: "doc",
id: "awel/awel_tutorial/templates/Embedding"
},{
type: "doc",
id: "awel/awel_tutorial/templates/Knowledge_Graph"
},{
type: "doc",
id: "awel/awel_tutorial/templates/Hybrid_Workflow"
}
]
},
],
link: {