diff --git a/README.md b/README.md index 510486db7..30c140f7d 100644 --- a/README.md +++ b/README.md @@ -68,18 +68,6 @@ Run on an RTX 4090 GPU. ##### Chat Knowledge ![kbqa_new](https://github.com/eosphoros-ai/DB-GPT/assets/13723926/72266a48-edef-4c6d-88c6-fbb1a24a6c3e) -#### Chat with data, and figure charts. - -![db plugins demonstration](https://github.com/eosphoros-ai/DB-GPT/assets/13723926/4113ac15-83c2-4350-86c0-5fc795677abd) - -

- -

- -

- -

- ## Install ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white) ![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) @@ -109,66 +97,54 @@ Run on an RTX 4090 GPU. ## Features Currently, we have released multiple key features, which are listed below to demonstrate our current capabilities: -- SQL language capabilities - - SQL generation - - SQL diagnosis -- Private domain Q&A and data processing - - Knowledge Management(We currently support many document formats: txt, pdf, md, html, doc, ppt, and url.) -- ChatDB -- ChatExcel -- ChatDashboard -- Multi-Agents&Plugins -- Unified vector storage/indexing of knowledge base - - Support for unstructured data - - PDF - - TXT - - Markdown - - CSV - - DOC - - PPT - - WebURL -- Multi LLMs Support, Supports multiple large language models, currently supporting - - [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) - - [baichuan2-7b/baichuan2-13b](https://huggingface.co/baichuan-inc) - - [internlm/internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) - - [Qwen/Qwen-7B-Chat/Qwen-14B-Chat](https://huggingface.co/Qwen/) - - [Vicuna](https://huggingface.co/Tribbiani/vicuna-13b) - - [BlinkDL/RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven) - - [camel-ai/CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data) - - [databricks/dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b) - - [FreedomIntelligence/phoenix-inst-chat-7b](https://huggingface.co/FreedomIntelligence/phoenix-inst-chat-7b) - - [h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b) - - [lcw99/polyglot-ko-12.8b-chang-instruct-chat](https://huggingface.co/lcw99/polyglot-ko-12.8b-chang-instruct-chat) - - [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5) - - [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) - - [Neutralzz/BiLLa-7B-SFT](https://huggingface.co/Neutralzz/BiLLa-7B-SFT) - - [nomic-ai/gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy) - - [NousResearch/Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b) - - [openaccess-ai-collective/manticore-13b-chat-pyg](https://huggingface.co/openaccess-ai-collective/manticore-13b-chat-pyg) - - [OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5) - - [project-baize/baize-v2-7b](https://huggingface.co/project-baize/baize-v2-7b) - - [Salesforce/codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b) - - [StabilityAI/stablelm-tuned-alpha-7b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b) - - [THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b) - - [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) - - [tiiuae/falcon-40b](https://huggingface.co/tiiuae/falcon-40b) - - [timdettmers/guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged) - - [togethercomputer/RedPajama-INCITE-7B-Chat](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat) - - [WizardLM/WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0) - - [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) - - [baichuan-inc/baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B) - - [HuggingFaceH4/starchat-beta](https://huggingface.co/HuggingFaceH4/starchat-beta) - - [FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) - - [BAAI/AquilaChat-7B](https://huggingface.co/BAAI/AquilaChat-7B) - - [all models of OpenOrca](https://huggingface.co/Open-Orca) - - [Spicyboros](https://huggingface.co/jondurbin/spicyboros-7b-2.2?not-for-all-audiences=true) + [airoboros 2.2](https://huggingface.co/jondurbin/airoboros-l2-13b-2.2) - - [VMware's OpenLLaMa OpenInstruct](https://huggingface.co/VMware/open-llama-7b-open-instruct) +- Private KBQA & data processing + The DB-GPT project offers a range of features to enhance knowledge base construction and enable efficient storage and retrieval of both structured and unstructured data. These include built-in support for uploading multiple file formats, the ability to integrate plug-ins for custom data extraction, and unified vector storage and retrieval capabilities for managing large volumes of information. +- Multiple data sources & visualization + The DB-GPT project enables seamless natural language interaction with various data sources, including Excel, databases, and data warehouses. It facilitates effortless querying and retrieval of information from these sources, allowing users to engage in intuitive conversations and obtain insights. Additionally, DB-GPT supports the generation of analysis reports, providing users with valuable summaries and interpretations of the data. -- Support API Proxy LLMs - - [x] [ChatGPT](https://api.openai.com/) - - [x] [Tongyi](https://www.aliyun.com/product/dashscope) - - [x] [Wenxin](https://cloud.baidu.com/product/wenxinworkshop?track=dingbutonglan) - - [x] [ChatGLM](http://open.bigmodel.cn/) +- Multi-Agents&Plugins + Supports custom plug-ins to perform tasks, natively supports the Auto-GPT plug-in model, and the Agents protocol adopts the Agent Protocol standard + +- Fine-tuning text2SQL + An automated fine-tuning lightweight framework built around large language models, Text2SQL data sets, LoRA/QLoRA/Pturning and other fine-tuning methods, making TextSQL fine-tuning as convenient as an assembly line. [DB-GPT-Hub](https://github.com/eosphoros-ai/DB-GPT-Hub) + +- Multi LLMs Support, Supports multiple large language models, currently supporting + Massive model support, including dozens of large language models such as open source and API agents. Such as LLaMA/LLaMA2, Baichuan, ChatGLM, Wenxin, Tongyi, Zhipu, etc. + - [Vicuna](https://huggingface.co/Tribbiani/vicuna-13b) + - [vicuna-13b-v1.5](https://huggingface.co/lmsys/vicuna-13b-v1.5) + - [LLama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) + - [baichuan2-13b](https://huggingface.co/baichuan-inc) + - [baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B) + - [chatglm-6b](https://huggingface.co/THUDM/chatglm-6b) + - [chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) + - [falcon-40b](https://huggingface.co/tiiuae/falcon-40b) + - [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) + - [Qwen-7B-Chat/Qwen-14B-Chat](https://huggingface.co/Qwen/) + - [RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven) + - [CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data) + - [dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b) + - [h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b) + - [fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5) + - [mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) + - [gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy) + - [Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b) + - [codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b) + - [guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged) + - [WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0) + - [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) + - [Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) + - [OpenLLaMa OpenInstruct](https://huggingface.co/VMware/open-llama-7b-open-instruct) + + etc. + + - Support API Proxy LLMs + - [x] [ChatGPT](https://api.openai.com/) + - [x] [Tongyi](https://www.aliyun.com/product/dashscope) + - [x] [Wenxin](https://cloud.baidu.com/product/wenxinworkshop?track=dingbutonglan) + - [x] [ChatGLM](http://open.bigmodel.cn/) + +- Privacy and security + The privacy and security of data are ensured through various technologies such as privatized large models and proxy desensitization. - Support Datasources diff --git a/README.zh.md b/README.zh.md index fa3f54aca..31d36b3fe 100644 --- a/README.zh.md +++ b/README.zh.md @@ -72,8 +72,6 @@ DB-GPT 是一个开源的以数据库为基础的GPT实验项目,使用本地 #### 根据自然语言对话生成分析图表 -![db plugins demonstration](https://github.com/eosphoros-ai/DB-GPT/assets/13723926/4113ac15-83c2-4350-86c0-5fc795677abd) -

@@ -86,32 +84,6 @@ DB-GPT 是一个开源的以数据库为基础的GPT实验项目,使用本地

-#### 根据自然语言对话生成SQL -

- -

- -#### 与数据库元数据信息进行对话, 生成准确SQL语句 -

- -

- - -#### 与数据对话, 直接查看执行结果 -

- -

- -#### 知识库管理 -

- -

- -#### 根据知识库对话, 比如pdf、csv、txt、words等等. -

- -

- ## 安装 ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white) @@ -142,61 +114,46 @@ DB-GPT 是一个开源的以数据库为基础的GPT实验项目,使用本地 目前我们已经发布了多种关键的特性,这里一一列举展示一下当前发布的能力。 -- SQL 语言能力 - - SQL生成 - - SQL诊断 -- 私域问答与数据处理 - - 知识库管理(目前支持 txt, pdf, md, html, doc, ppt, and url) - - 数据库知识问答 - - 数据处理 -- 数据库对话 -- Chat2Dashboard -- 插件模型 -- 知识库统一向量存储/索引 - - 非结构化数据支持包括PDF、MarkDown、CSV、WebURL +- 私域问答&数据处理 + 支持内置、多文件格式上传、插件自抓取等方式自定义构建知识库,对海量结构化,非结构化数据做统一向量存储与检索 +- 多数据源&可视化 + 支持自然语言与Excel、数据库、数仓等多种数据源交互,并支持分析报告。 +- 自动化微调 + 围绕大语言模型、Text2SQL数据集、LoRA/QLoRA/Pturning等微调方法构建的自动化微调轻量框架, 让TextSQL微调像流水线一样方便。详见: [DB-GPT-Hub](https://github.com/eosphoros-ai/DB-GPT-Hub) +- Multi-Agents&Plugins + 支持自定义插件执行任务,原生支持Auto-GPT插件模型,Agents协议采用Agent Protocol标准 - 多模型支持与管理 + 海量模型支持,包括开源、API代理等几十种大语言模型。如LLaMA/LLaMA2、Baichuan、ChatGLM、文心、通义、智谱等。 - 支持多种大语言模型, 当前已支持如下模型: - - [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) - - [baichuan2-7b/baichuan2-13b](https://huggingface.co/baichuan-inc) - - [internlm/internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) - - [Qwen/Qwen-7B-Chat/Qwen-14B-Chat](https://huggingface.co/Qwen/) - [Vicuna](https://huggingface.co/Tribbiani/vicuna-13b) - - [BlinkDL/RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven) - - [camel-ai/CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data) - - [databricks/dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b) - - [FreedomIntelligence/phoenix-inst-chat-7b](https://huggingface.co/FreedomIntelligence/phoenix-inst-chat-7b) - - [h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b) - - [lcw99/polyglot-ko-12.8b-chang-instruct-chat](https://huggingface.co/lcw99/polyglot-ko-12.8b-chang-instruct-chat) - - [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5) - - [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) - - [Neutralzz/BiLLa-7B-SFT](https://huggingface.co/Neutralzz/BiLLa-7B-SFT) - - [nomic-ai/gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy) - - [NousResearch/Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b) - - [openaccess-ai-collective/manticore-13b-chat-pyg](https://huggingface.co/openaccess-ai-collective/manticore-13b-chat-pyg) - - [OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5) - - [project-baize/baize-v2-7b](https://huggingface.co/project-baize/baize-v2-7b) - - [Salesforce/codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b) - - [StabilityAI/stablelm-tuned-alpha-7b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b) - - [THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b) - - [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) - - [tiiuae/falcon-40b](https://huggingface.co/tiiuae/falcon-40b) - - [timdettmers/guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged) - - [togethercomputer/RedPajama-INCITE-7B-Chat](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat) - - [WizardLM/WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0) + - [vicuna-13b-v1.5](https://huggingface.co/lmsys/vicuna-13b-v1.5) + - [LLama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) + - [baichuan2-13b](https://huggingface.co/baichuan-inc) + - [baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B) + - [chatglm-6b](https://huggingface.co/THUDM/chatglm-6b) + - [chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b) + - [falcon-40b](https://huggingface.co/tiiuae/falcon-40b) + - [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) + - [Qwen-7B-Chat/Qwen-14B-Chat](https://huggingface.co/Qwen/) + - [RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven) + - [CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data) + - [dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b) + - [h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b) + - [fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5) + - [mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat) + - [gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy) + - [Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b) + - [codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b) + - [guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged) + - [WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0) - [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) - - [baichuan-inc/baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B) - - [HuggingFaceH4/starchat-beta](https://huggingface.co/HuggingFaceH4/starchat-beta) - - [FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) - - [BAAI/AquilaChat-7B](https://huggingface.co/BAAI/AquilaChat-7B) - - [all models of OpenOrca](https://huggingface.co/Open-Orca) - - [Spicyboros](https://huggingface.co/jondurbin/spicyboros-7b-2.2?not-for-all-audiences=true) + [airoboros 2.2](https://huggingface.co/jondurbin/airoboros-l2-13b-2.2) - - [VMware's OpenLLaMa OpenInstruct](https://huggingface.co/VMware/open-llama-7b-open-instruct) - -- 支持在线代理模型 - - [x] [ChatGPT](https://api.openai.com/) - - [x] [Tongyi](https://www.aliyun.com/product/dashscope) - - [x] [Wenxin](https://cloud.baidu.com/product/wenxinworkshop?track=dingbutonglan) - - [x] [ChatGLM](http://open.bigmodel.cn/) + - [Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) + - [OpenLLaMa OpenInstruct](https://huggingface.co/VMware/open-llama-7b-open-instruct) + - 支持在线代理模型 + - [x] [ChatGPT](https://api.openai.com/) + - [x] [Tongyi](https://www.aliyun.com/product/dashscope) + - [x] [Wenxin](https://cloud.baidu.com/product/wenxinworkshop?track=dingbutonglan) + - [x] [ChatGLM](http://open.bigmodel.cn/) - 支持数据源 @@ -227,7 +184,7 @@ DB-GPT基于 [FastChat](https://github.com/lm-sys/FastChat) 构建大模型运 整个DB-GPT的架构,如下图所示

- +

核心能力主要有以下几个部分。 diff --git a/assets/DB-GPT.png b/assets/DB-GPT.png index b136a1e57..b8f012ea6 100644 Binary files a/assets/DB-GPT.png and b/assets/DB-GPT.png differ diff --git a/assets/DB-GPT_zh.png b/assets/DB-GPT_zh.png new file mode 100644 index 000000000..ef6a3d40b Binary files /dev/null and b/assets/DB-GPT_zh.png differ