Merge branch 'dev' into ty_test

2025-09-17 15:10:14 +00:00 · 2023-06-08 20:55:19 +08:00
parent 8851ab9d45 38d7a37ef8
commit 58295a415a
14 changed files with 119 additions and 152 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -141,6 +141,7 @@ logs
 nltk_data
 .vectordb
 pilot/data/
+pilot/nltk_data

 logswebserver.log.*
 .history/*
--- a/README.md
+++ b/README.md
@@ -1,4 +1,5 @@
-# DB-GPT: A LLM Tool for Multi Databases
+# DB-GPT: Revolutionizing Database Interactions with Private LLM Technology
+ 
 <div align="center">
  <p>
    <a href="https://github.com/csunny/DB-GPT">
@@ -12,8 +13,6 @@
 [**简体中文**](README.zh.md)|[**Discord**](https://discord.gg/ea6BnZkY) 
 </div>

-[![Star History Chart](https://api.star-history.com/svg?repos=csunny/DB-GPT)](https://star-history.com/#csunny/DB-GPT)
-
 ## What is DB-GPT?

 As large models are released and iterated upon, they are becoming increasingly intelligent. However, in the process of using large models, we face significant challenges in data security and privacy. We need to ensure that our sensitive data and environments remain completely controlled and avoid any data privacy leaks or security risks. Based on this, we have launched the DB-GPT project to build a complete private large model solution for all database-based scenarios. This solution supports local deployment, allowing it to be applied not only in independent private environments but also to be independently deployed and isolated according to business modules, ensuring that the ability of large models is absolutely private, secure, and controllable.
@@ -53,7 +52,18 @@ Currently, we have released multiple key features, which are listed below to dem

 ## Demo

-Run on an RTX 4090 GPU. [YouTube](https://www.youtube.com/watch?v=1PWI6F89LPo)
+Run on an RTX 4090 GPU.
+<p align="center">
+  <img src="./assets/auto_sql_en.gif" width="680px" />
+</p>
+
+<p align="center">
+  <img src="./assets/chaturl_en.gif" width="680px" />
+</p>
+
+<p align="center">
+  <img src="./assets/new_knownledge_en.gif" width="680px" />
+</p>

 ## Introduction 
 DB-GPT creates a vast model operating system using [FastChat](https://github.com/lm-sys/FastChat) and offers a large language model powered by [Vicuna](https://huggingface.co/Tribbiani/vicuna-7b). In addition, we provide private domain knowledge base question-answering capability through LangChain. Furthermore, we also provide support for additional plugins, and our design natively supports the Auto-GPT plugin.
@@ -61,7 +71,7 @@ DB-GPT creates a vast model operating system using [FastChat](https://github.com
 Is the architecture of the entire DB-GPT shown in the following figure:

 <p align="center">
-  <img src="./assets/DB-GPT.png" width="600px" />
+  <img src="./assets/DB-GPT.png" width="680px" />
 </p>

 The core capabilities mainly consist of the following parts:
@@ -216,3 +226,5 @@ The MIT License (MIT)

 ## Contact Information
 We are working on building a community, if you have any ideas about building the community, feel free to contact us. [Discord](https://discord.gg/kMFf77FH)
+
+[![Star History Chart](https://api.star-history.com/svg?repos=csunny/DB-GPT)](https://star-history.com/#csunny/DB-GPT)
--- a/README.zh.md
+++ b/README.zh.md
@@ -1,4 +1,4 @@
-# DB-GPT: 数据库的 LLM 工具
+# DB-GPT: 用私有化LLM技术定义数据库下一代交互方式
 <div align="center">
  <p>
    <a href="https://github.com/csunny/DB-GPT">
@@ -12,8 +12,6 @@
 [**English**](README.md)|[**Discord**](https://discord.gg/ea6BnZkY) 
 </div>

-[![Star History Chart](https://api.star-history.com/svg?repos=csunny/DB-GPT)](https://star-history.com/#csunny/DB-GPT)
-
 ## DB-GPT 是什么？

 随着大模型的发布迭代，大模型变得越来越智能，在使用大模型的过程当中，遇到极大的数据安全与隐私挑战。在利用大模型能力的过程中我们的私密数据跟环境需要掌握自己的手里，完全可控，避免任何的数据隐私泄露以及安全风险。基于此，我们发起了DB-GPT项目，为所有以数据库为基础的场景，构建一套完整的私有大模型解决方案。 此方案因为支持本地部署，所以不仅仅可以应用于独立私有环境，而且还可以根据业务模块独立部署隔离，让大模型的能力绝对私有、安全、可控。
@@ -51,7 +49,22 @@ DB-GPT 是一个开源的以数据库为基础的GPT实验项目，使用本地

 ## 效果演示

-示例通过 RTX 4090 GPU 演示，[YouTube 地址](https://www.youtube.com/watch?v=1PWI6F89LPo)
+示例通过 RTX 4090 GPU 演示
+<p align="center">
+  <img src="./assets/演示.gif" width="680px" />
+</p>
+
+<p align="center">
+  <img src="./assets/auto_sql.gif" width="680px" />
+</p>
+
+<p align="center">
+  <img src="./assets/chat_url_zh.gif" width="680px" />
+</p>
+
+<p align="center">
+  <img src="./assets/new_knownledge.gif" width="680px" />
+</p>

 ## 架构方案
 DB-GPT基于 [FastChat](https://github.com/lm-sys/FastChat) 构建大模型运行环境，并提供 vicuna 作为基础的大语言模型。此外，我们通过LangChain提供私域知识库问答能力。同时我们支持插件模式, 在设计上原生支持Auto-GPT插件。 
@@ -220,3 +233,6 @@ Run the Python interpreter and type the commands:
 ## Licence

 The MIT License (MIT)
+
+[![Star History Chart](https://api.star-history.com/svg?repos=csunny/DB-GPT)](https://star-history.com/#csunny/DB-GPT)
+
--- a/assets/wechat.jpg
+++ b/assets/wechat.jpg
--- a/pilot/configs/model_config.py
+++ b/pilot/configs/model_config.py
@@ -34,7 +34,6 @@ LLM_MODEL_CONFIG = {
    "chatglm-6b-int4": os.path.join(MODEL_PATH, "chatglm-6b-int4"),
    "chatglm-6b": os.path.join(MODEL_PATH, "chatglm-6b"),
    "text2vec-base": os.path.join(MODEL_PATH, "text2vec-base-chinese"),
-    "sentence-transforms": os.path.join(MODEL_PATH, "all-MiniLM-L6-v2"),
    "guanaco-33b-merged": os.path.join(MODEL_PATH, "guanaco-33b-merged"),
    "proxyllm": "proxyllm",
 }
--- a/pilot/conversation.py
+++ b/pilot/conversation.py
@@ -295,7 +295,7 @@ default_conversation = conv_default


 chat_mode_title = {
-    "sql_generate_diagnostics": get_lang_text("sql_analysis_and_diagnosis"),
+    "sql_generate_diagnostics": get_lang_text("sql_generate_diagnostics"),
    "chat_use_plugin": get_lang_text("chat_use_plugin"),
    "knowledge_qa": get_lang_text("knowledge_qa"),
 }
--- a/pilot/language/lang_content_mapping.py
+++ b/pilot/language/lang_content_mapping.py
@@ -44,13 +44,13 @@ lang_dicts = {
        "learn_more_markdown": "The service is a research preview intended for non-commercial use only. subject to the model [License](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) of Vicuna-13B",
        "model_control_param": "Model Parameters",
        "sql_generate_mode_direct": "Execute directly",
-        "sql_generate_mode_none": "Execute without model",
+        "sql_generate_mode_none": "Execute without mode",
        "max_input_token_size": "Maximum output token size",
        "please_choose_database": "Please choose database",
        "sql_generate_diagnostics": "SQL Generation & Diagnostics",
        "knowledge_qa_type_llm_native_dialogue": "LLM native dialogue",
        "knowledge_qa_type_default_knowledge_base_dialogue": "Default documents",
-        "knowledge_qa_type_add_knowledge_base_dialogue": "Added documents",
+        "knowledge_qa_type_add_knowledge_base_dialogue": "New documents",
        "knowledge_qa_type_url_knowledge_dialogue": "Chat with url",
        "dialogue_use_plugin": "Dialogue Extension",
        "create_knowledge_base": "Create Knowledge Base",
@@ -60,7 +60,7 @@ lang_dicts = {
        "sql_vs_setting": "In the automatic execution mode, DB-GPT can have the ability to execute SQL, read data from the network, automatically store and learn",
        "chat_use_plugin": "Plugin Mode",
        "select_plugin": "Select Plugin",
-        "knowledge_qa": "Documents QA",
+        "knowledge_qa": "Documents Chat",
        "configure_knowledge_base": "Configure Documents",
        "url_input_label": "Please input url",
        "new_klg_name": "New document name",
--- a/pilot/model/guanaco_stream_llm.py
+++ b/pilot/model/guanaco_stream_llm.py
@@ -1,54 +0,0 @@
-import torch
-from threading import Thread
-from transformers import TextIteratorStreamer, StoppingCriteriaList, StoppingCriteria
-
-
-def guanaco_stream_generate_output(model, tokenizer, params, device, context_len=2048):
-    """Fork from: https://github.com/KohakuBlueleaf/guanaco-lora/blob/main/generate.py"""
-    tokenizer.bos_token_id = 1
-    print(params)
-    stop = params.get("stop", "###")
-    prompt = params["prompt"]
-    query = prompt
-    print("Query Message: ", query)
-
-    input_ids = tokenizer(query, return_tensors="pt").input_ids
-    input_ids = input_ids.to(model.device)
-
-    streamer = TextIteratorStreamer(
-        tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True
-    )
-
-    tokenizer.bos_token_id = 1
-    stop_token_ids = [0]
-
-    class StopOnTokens(StoppingCriteria):
-        def __call__(
-            self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs
-        ) -> bool:
-            for stop_id in stop_token_ids:
-                if input_ids[0][-1] == stop_id:
-                    return True
-            return False
-
-    stop = StopOnTokens()
-
-    generate_kwargs = dict(
-        input_ids=input_ids,
-        max_new_tokens=512,
-        temperature=1.0,
-        do_sample=True,
-        top_k=1,
-        streamer=streamer,
-        repetition_penalty=1.7,
-        stopping_criteria=StoppingCriteriaList([stop]),
-    )
-
-    t = Thread(target=model.generate, kwargs=generate_kwargs)
-    t.start()
-
-    out = ""
-    for new_text in streamer:
-        out += new_text
-        yield new_text
-    return out
--- a/pilot/server/chat_adapter.py
+++ b/pilot/server/chat_adapter.py
@@ -60,20 +60,6 @@ class ChatGLMChatAdapter(BaseChatAdpter):
        return chatglm_generate_stream


-class GuanacoChatAdapter(BaseChatAdpter):
-    """Model chat adapter for Guanaco"""
-
-    def match(self, model_path: str):
-        return "guanaco" in model_path
-
-    def get_generate_stream_func(self):
-        from pilot.model.llm_out.guanaco_stream_llm import (
-            guanaco_stream_generate_output,
-        )
-
-        return guanaco_generate_output
-
-
 class CodeT5ChatAdapter(BaseChatAdpter):

    """Model chat adapter for CodeT5"""
--- a/pilot/server/webserver.py
+++ b/pilot/server/webserver.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
+import signal
 import threading
 import traceback
 import argparse
@@ -206,7 +207,13 @@ def post_process_code(code):
 def get_chat_mode(selected, param=None) -> ChatScene:
    if chat_mode_title["chat_use_plugin"] == selected:
        return ChatScene.ChatExecution
-    elif chat_mode_title["knowledge_qa"] == selected:
+    elif chat_mode_title["sql_generate_diagnostics"] == selected:
+        sql_mode = param
+        if sql_mode == conversation_sql_mode["auto_execute_ai_response"]:
+            return ChatScene.ChatWithDbExecute
+        else:
+            return ChatScene.ChatWithDbQA
+    else:
        mode = param
        if mode == conversation_types["default_knownledge"]:
            return ChatScene.ChatKnowledge
@@ -216,12 +223,6 @@ def get_chat_mode(selected, param=None) -> ChatScene:
            return ChatScene.ChatUrlKnowledge
        else:
            return ChatScene.ChatNormal
-    else:
-        sql_mode = param
-        if sql_mode == conversation_sql_mode["auto_execute_ai_response"]:
-            return ChatScene.ChatWithDbExecute
-        else:
-            return ChatScene.ChatWithDbQA


 def chatbot_callback(state, message):
@@ -245,12 +246,13 @@ def http_bot(
    logger.info(
        f"User message send!{state.conv_id},{selected},{plugin_selector},{mode},{sql_mode},{db_selector},{url_input}"
    )
-    if chat_mode_title["knowledge_qa"] == selected:
-        scene: ChatScene = get_chat_mode(selected, mode)
+    if chat_mode_title["sql_generate_diagnostics"] == selected:
+        scene: ChatScene = get_chat_mode(selected, sql_mode)
    elif chat_mode_title["chat_use_plugin"] == selected:
        scene: ChatScene = get_chat_mode(selected)
    else:
-        scene: ChatScene = get_chat_mode(selected, sql_mode)
+        scene: ChatScene = get_chat_mode(selected, mode)
+
    print(f"chat scene:{scene.value}")

    if ChatScene.ChatWithDbExecute == scene:
@@ -403,58 +405,6 @@ def build_single_model_ui():
    tabs.select(on_select, None, selected)

    with tabs:
-        tab_sql = gr.TabItem(get_lang_text("sql_generate_diagnostics"), elem_id="SQL")
-        with tab_sql:
-            # TODO A selector to choose database
-            with gr.Row(elem_id="db_selector"):
-                db_selector = gr.Dropdown(
-                    label=get_lang_text("please_choose_database"),
-                    choices=dbs,
-                    value=dbs[0] if len(models) > 0 else "",
-                    interactive=True,
-                    show_label=True,
-                ).style(container=False)
-
-            # db_selector.change(fn=db_selector_changed, inputs=db_selector)
-
-            sql_mode = gr.Radio(
-                [
-                    get_lang_text("sql_generate_mode_direct"),
-                    get_lang_text("sql_generate_mode_none"),
-                ],
-                show_label=False,
-                value=get_lang_text("sql_generate_mode_none"),
-            )
-            sql_vs_setting = gr.Markdown(get_lang_text("sql_vs_setting"))
-            sql_mode.change(fn=change_sql_mode, inputs=sql_mode, outputs=sql_vs_setting)
-
-        tab_plugin = gr.TabItem(get_lang_text("chat_use_plugin"), elem_id="PLUGIN")
-        # tab_plugin.select(change_func)
-        with tab_plugin:
-            print("tab_plugin in...")
-            with gr.Row(elem_id="plugin_selector"):
-                # TODO
-                plugin_selector = gr.Dropdown(
-                    label=get_lang_text("select_plugin"),
-                    choices=list(plugins_select_info().keys()),
-                    value="",
-                    interactive=True,
-                    show_label=True,
-                    type="value",
-                ).style(container=False)
-
-                def plugin_change(
-                    evt: gr.SelectData,
-                ):  # SelectData is a subclass of EventData
-                    print(f"You selected {evt.value} at {evt.index} from {evt.target}")
-                    print(f"user plugin:{plugins_select_info().get(evt.value)}")
-                    return plugins_select_info().get(evt.value)
-
-                plugin_selected = gr.Textbox(
-                    show_label=False, visible=False, placeholder="Selected"
-                )
-                plugin_selector.select(plugin_change, None, plugin_selected)
-
        tab_qa = gr.TabItem(get_lang_text("knowledge_qa"), elem_id="QA")
        with tab_qa:
            mode = gr.Radio(
@@ -517,6 +467,58 @@ def build_single_model_ui():
                            get_lang_text("upload_and_load_to_klg")
                        )

+        tab_sql = gr.TabItem(get_lang_text("sql_generate_diagnostics"), elem_id="SQL")
+        with tab_sql:
+            # TODO A selector to choose database
+            with gr.Row(elem_id="db_selector"):
+                db_selector = gr.Dropdown(
+                    label=get_lang_text("please_choose_database"),
+                    choices=dbs,
+                    value=dbs[0] if len(models) > 0 else "",
+                    interactive=True,
+                    show_label=True,
+                ).style(container=False)
+
+            # db_selector.change(fn=db_selector_changed, inputs=db_selector)
+
+            sql_mode = gr.Radio(
+                [
+                    get_lang_text("sql_generate_mode_direct"),
+                    get_lang_text("sql_generate_mode_none"),
+                ],
+                show_label=False,
+                value=get_lang_text("sql_generate_mode_none"),
+            )
+            sql_vs_setting = gr.Markdown(get_lang_text("sql_vs_setting"))
+            sql_mode.change(fn=change_sql_mode, inputs=sql_mode, outputs=sql_vs_setting)
+
+        tab_plugin = gr.TabItem(get_lang_text("chat_use_plugin"), elem_id="PLUGIN")
+        # tab_plugin.select(change_func)
+        with tab_plugin:
+            print("tab_plugin in...")
+            with gr.Row(elem_id="plugin_selector"):
+                # TODO
+                plugin_selector = gr.Dropdown(
+                    label=get_lang_text("select_plugin"),
+                    choices=list(plugins_select_info().keys()),
+                    value="",
+                    interactive=True,
+                    show_label=True,
+                    type="value",
+                ).style(container=False)
+
+                def plugin_change(
+                    evt: gr.SelectData,
+                ):  # SelectData is a subclass of EventData
+                    print(f"You selected {evt.value} at {evt.index} from {evt.target}")
+                    print(f"user plugin:{plugins_select_info().get(evt.value)}")
+                    return plugins_select_info().get(evt.value)
+
+                plugin_selected = gr.Textbox(
+                    show_label=False, visible=False, placeholder="Selected"
+                )
+                plugin_selector.select(plugin_change, None, plugin_selected)
+
    with gr.Blocks():
        chatbot = grChatbot(elem_id="chatbot", visible=False).style(height=550)
        with gr.Row():
@@ -648,6 +650,11 @@ def async_db_summery():
    thread.start()


+def signal_handler(sig, frame):
+    print("in order to avoid chroma db atexit problem")
+    os._exit(0)
+
+
 if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", type=str, default="0.0.0.0")
@@ -664,6 +671,7 @@ if __name__ == "__main__":
    cfg = Config()

    dbs = cfg.local_db.get_database_list()
+    signal.signal(signal.SIGINT, signal_handler)
    async_db_summery()
    cfg.set_plugins(scan_plugins(cfg, cfg.debug_mode))

--- a/pilot/source_embedding/pdf_embedding.py
+++ b/pilot/source_embedding/pdf_embedding.py
@@ -4,10 +4,10 @@ from typing import List

 from langchain.document_loaders import PyPDFLoader
 from langchain.schema import Document
+from langchain.text_splitter import SpacyTextSplitter

 from pilot.configs.config import Config
 from pilot.source_embedding import SourceEmbedding, register
-from pilot.source_embedding.chn_document_splitter import CHNDocumentSplitter

 CFG = Config()

@@ -24,11 +24,11 @@ class PDFEmbedding(SourceEmbedding):
    @register
    def read(self):
        """Load from pdf path."""
-        # loader = UnstructuredPaddlePDFLoader(self.file_path)
        loader = PyPDFLoader(self.file_path)
-        textsplitter = CHNDocumentSplitter(
-            pdf=True, sentence_size=CFG.KNOWLEDGE_CHUNK_SIZE
-        )
+        # textsplitter = CHNDocumentSplitter(
+        #     pdf=True, sentence_size=CFG.KNOWLEDGE_CHUNK_SIZE
+        # )
+        textsplitter = SpacyTextSplitter(pipeline='zh_core_web_sm', chunk_size=1000, chunk_overlap=200)
        return loader.load_and_split(textsplitter)

    @register
--- a/pilot/vector_store/chroma_store.py
+++ b/pilot/vector_store/chroma_store.py
@@ -1,7 +1,6 @@
 import os

 from langchain.vectorstores import Chroma
-
 from pilot.configs.model_config import KNOWLEDGE_UPLOAD_ROOT_PATH
 from pilot.logs import logger
 from pilot.vector_store.vector_store_base import VectorStoreBase
--- a/plugins/DB-GPT-Plugin-ByteBase.zip
+++ b/plugins/DB-GPT-Plugin-ByteBase.zip