Standalone Deployment

(standalone-index)=

Install Prepare

git clone https://github.com/eosphoros-ai/DB-GPT.git

cd DB-GPT

Create conda environment

conda create -n dbgpt_env python=3.10

conda activate dbgpt_env

Install Default Requirements

# Install Default Requirements
pip install -e ".[default]"

Download and Prepare LLM Model and Embedding Model

 If you don't have high performance hardware server

you can use openai api, tongyi api , bard api, etc.

mkdir models && cd models

# download embedding model, eg: text2vec-large-chinese
git clone https://huggingface.co/GanymedeNil/text2vec-large-chinese

set proxy api in .env

#set LLM_MODEL TYPE
LLM_MODEL=proxyllm
#set your Proxy Api key and Proxy Server url
PROXY_API_KEY={your-openai-sk}
PROXY_SERVER_URL=https://api.openai.com/v1/chat/completions

If you have high performance hardware server

mkdir models && cd models

# # download embedding model, eg: vicuna-13b-v1.5 or  
git clone https://huggingface.co/lmsys/vicuna-13b-v1.5

# download embedding model, eg: text2vec-large-chinese
git clone https://huggingface.co/GanymedeNil/text2vec-large-chinese

popd

Start all services with a single command.

LLM_MODEL=vicuna-13b-v1.5 
dbgpt start webserver --port 6006

By default, the "dbgpt start webserver" command will start the Webserver, Model Controller, and Model Worker in a single Python process. Here, we specify the service to be started on port 6006.

View and validate the model service in the command line, you can use the following commands

1.list the started model services and deployed Model Workers, you can use the following command

dbgpt model list

output is:

+-----------------+------------+------------+------+---------+---------+-----------------+----------------------------+
|    Model Name   | Model Type |    Host    | Port | Healthy | Enabled | Prompt Template |       Last Heartbeat       |
+-----------------+------------+------------+------+---------+---------+-----------------+----------------------------+
| vicuna-13b-v1.5 |    llm     | 172.17.0.9 | 6006 |   True  |   True  |                 | 2023-10-16T19:49:59.201313 |
|  WorkerManager  |  service   | 172.17.0.9 | 6006 |   True  |   True  |                 | 2023-10-16T19:49:59.246756 |
+-----------------+------------+------------+------+---------+---------+-----------------+----------------------------+

The WorkerManager is the management process for Model Workers

validate the deployed model in the command line, you can use the following command

dbgpt model chat --model_name vicuna-13b-v1.5

Then an interactive page will be launched where you can have a conversation with the deployed LLM in the terminal.

Chatbot started with model vicuna-13b-v1.5. Type 'exit' to leave the chat.


You: Hello
Bot: Hello! How can I assist you today?

You:

3.0 KiB Raw Blame History