github/DB-GPT
1
0
Fork 0
mirror of https://github.com/csunny/DB-GPT.git synced 2026-07-17 01:58:47 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0da65cd435a7cde9744e261069d0dbf206e973cd
DB-GPT/docs/getting_started/observability.md
FangYin Cheng 87f2484171 doc(core): Add Trace document
2023-10-12 12:02:56 +08:00

23 KiB
Raw Blame History

Debugging

DB-GPT provides a set of tools to help you troubleshoot and resolve some of the issues you may encounter.

Trace Logs

DB-GPT writes some critical system runtime information to trace logs. By default, these are located in logs/dbgpt*.jsonl.

DB-GPT also offers a command-line tool, dbgpt trace, to help you analyze these trace logs. You can see its specific usage with the command dbgpt trace --help.

Viewing Chat Details

You can use the dbgpt trace chat command to view chat details. By default, it will display the latest chat message.

Viewing Service Runtime Information

dbgpt trace chat --hide_conv

You will see an output like:

+------------------------+--------------------------+-----------------------------+------------------------------------+
| Config Key (Webserver) | Config Value (Webserver) | Config Key (EmbeddingModel) |   Config Value (EmbeddingModel)    |
+------------------------+--------------------------+-----------------------------+------------------------------------+
|          host          |         0.0.0.0          |          model_name         |              text2vec              |
|          port          |           5000           |          model_path         | /app/models/text2vec-large-chinese |
|         daemon         |          False           |            device           |                cuda                |
|         share          |          False           |     normalize_embeddings    |                None                |
|    remote_embedding    |          False           |                             |                                    |
|       log_level        |           None           |                             |                                    |
|         light          |          False           |                             |                                    |
+------------------------+--------------------------+-----------------------------+------------------------------------+
+--------------------------+-----------------------------+----------------------------+------------------------------+
| Config Key (ModelWorker) |  Config Value (ModelWorker) | Config Key (WorkerManager) | Config Value (WorkerManager) |
+--------------------------+-----------------------------+----------------------------+------------------------------+
|        model_name        |       vicuna-13b-v1.5       |         model_name         |       vicuna-13b-v1.5        |
|        model_path        | /app/models/vicuna-13b-v1.5 |         model_path         | /app/models/vicuna-13b-v1.5  |
|          device          |             cuda            |        worker_type         |             None             |
|        model_type        |         huggingface         |        worker_class        |             None             |
|     prompt_template      |             None            |         model_type         |         huggingface          |
|     max_context_size     |             4096            |            host            |           0.0.0.0            |
|         num_gpus         |             None            |            port            |             5000             |
|      max_gpu_memory      |             None            |           daemon           |            False             |
|      cpu_offloading      |            False            |  limit_model_concurrency   |              5               |
|        load_8bit         |            False            |         standalone         |             True             |
|        load_4bit         |            False            |          register          |             True             |
|        quant_type        |             nf4             |    worker_register_host    |             None             |
|     use_double_quant     |             True            |      controller_addr       |    http://127.0.0.1:5000     |
|      compute_dtype       |             None            |       send_heartbeat       |             True             |
|    trust_remote_code     |             True            |     heartbeat_interval     |              20              |
|         verbose          |            False            |         log_level          |             None             |
+--------------------------+-----------------------------+----------------------------+------------------------------+

Viewing the Latest Chat Message

dbgpt trace chat --hide_run_params

You will see an output like:

+-------------------------------------------------------------------------------------------------------------------------------------------+
|                                                             Chat Trace Details                                                            |
+----------------+--------------------------------------------------------------------------------------------------------------------------+
|      Key       |                                                       Value Value                                                        |
+----------------+--------------------------------------------------------------------------------------------------------------------------+
|    trace_id    |                                           5d1900c3-5aad-4159-9946-fbb600666530                                           |
|    span_id     |                        5d1900c3-5aad-4159-9946-fbb600666530:14772034-bed4-4b4e-b43f-fcf3a8aad6a7                         |
|    conv_uid    |                                           5e456272-68ac-11ee-9fba-0242ac150003                                           |
|   user_input   |                                                       Who are you?                                                       |
|   chat_mode    |                                                       chat_normal                                                        |
|  select_param  |                                                           None                                                           |
|   model_name   |                                                     vicuna-13b-v1.5                                                      |
|  temperature   |                                                           0.6                                                            |
| max_new_tokens |                                                           1024                                                           |
|      echo      |                                                          False                                                           |
|  llm_adapter   |                        FastChatLLMModelAdaperWrapper(fastchat.model.model_adapter.VicunaAdapter)                         |
|  User prompt   | A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polit |
|                |                             e answers to the user's questions. USER: Who are you? ASSISTANT:                             |
|  Model output  |  You can call me Vicuna, and I was trained by Large Model Systems Organization (LMSYS) researchers as a language model.  |
+----------------+--------------------------------------------------------------------------------------------------------------------------+

Viewing Chat Details and Call Chain

dbgpt trace chat --hide_run_params --tree

You will see an output like:


Invoke Trace Tree:

Operation: DB-GPT-Web-Entry (Start: 2023-10-12 03:06:43.180, End: None)
  Operation: get_chat_instance (Start: 2023-10-12 03:06:43.258, End: None)
  Operation: get_chat_instance (Start: 2023-10-12 03:06:43.258, End: 2023-10-12 03:06:43.424)
  Operation: stream_generator (Start: 2023-10-12 03:06:43.425, End: None)
    Operation: BaseChat.stream_call (Start: 2023-10-12 03:06:43.426, End: None)
      Operation: WorkerManager.generate_stream (Start: 2023-10-12 03:06:43.426, End: None)
        Operation: DefaultModelWorker.generate_stream (Start: 2023-10-12 03:06:43.428, End: None)
          Operation: DefaultModelWorker_call.generate_stream_func (Start: 2023-10-12 03:06:43.430, End: None)
          Operation: DefaultModelWorker_call.generate_stream_func (Start: 2023-10-12 03:06:43.430, End: 2023-10-12 03:06:48.518)
        Operation: DefaultModelWorker.generate_stream (Start: 2023-10-12 03:06:43.428, End: 2023-10-12 03:06:48.518)
      Operation: WorkerManager.generate_stream (Start: 2023-10-12 03:06:43.426, End: 2023-10-12 03:06:48.518)
    Operation: BaseChat.stream_call (Start: 2023-10-12 03:06:43.426, End: 2023-10-12 03:06:48.519)
  Operation: stream_generator (Start: 2023-10-12 03:06:43.425, End: 2023-10-12 03:06:48.519)
Operation: DB-GPT-Web-Entry (Start: 2023-10-12 03:06:43.180, End: 2023-10-12 03:06:43.257)
+-------------------------------------------------------------------------------------------------------------------------------------------+
|                                                             Chat Trace Details                                                            |
+----------------+--------------------------------------------------------------------------------------------------------------------------+
|      Key       |                                                       Value Value                                                        |
+----------------+--------------------------------------------------------------------------------------------------------------------------+
|    trace_id    |                                           5d1900c3-5aad-4159-9946-fbb600666530                                           |
|    span_id     |                        5d1900c3-5aad-4159-9946-fbb600666530:14772034-bed4-4b4e-b43f-fcf3a8aad6a7                         |
|    conv_uid    |                                           5e456272-68ac-11ee-9fba-0242ac150003                                           |
|   user_input   |                                                       Who are you?                                                       |
|   chat_mode    |                                                       chat_normal                                                        |
|  select_param  |                                                           None                                                           |
|   model_name   |                                                     vicuna-13b-v1.5                                                      |
|  temperature   |                                                           0.6                                                            |
| max_new_tokens |                                                           1024                                                           |
|      echo      |                                                          False                                                           |
|  llm_adapter   |                        FastChatLLMModelAdaperWrapper(fastchat.model.model_adapter.VicunaAdapter)                         |
|  User prompt   | A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polit |
|                |                             e answers to the user's questions. USER: Who are you? ASSISTANT:                             |
|  Model output  |  You can call me Vicuna, and I was trained by Large Model Systems Organization (LMSYS) researchers as a language model.  |
+----------------+--------------------------------------------------------------------------------------------------------------------------+

Viewing Chat Details Based on trace_id

dbgpt trace chat --hide_run_params --trace_id ec30d733-7b35-4d61-b02e-2832fd2e29ff

You will see an output like:

+-------------------------------------------------------------------------------------------------------------------------------------------+
|                                                             Chat Trace Details                                                            |
+----------------+--------------------------------------------------------------------------------------------------------------------------+
|      Key       |                                                       Value Value                                                        |
+----------------+--------------------------------------------------------------------------------------------------------------------------+
|    trace_id    |                                           ec30d733-7b35-4d61-b02e-2832fd2e29ff                                           |
|    span_id     |                        ec30d733-7b35-4d61-b02e-2832fd2e29ff:0482a0c5-38b3-4b38-8101-e42489f90ccd                         |
|    conv_uid    |                                           87a722de-68ae-11ee-9fba-0242ac150003                                           |
|   user_input   |                                                          Hello                                                           |
|   chat_mode    |                                                       chat_normal                                                        |
|  select_param  |                                                           None                                                           |
|   model_name   |                                                     vicuna-13b-v1.5                                                      |
|  temperature   |                                                           0.6                                                            |
| max_new_tokens |                                                           1024                                                           |
|      echo      |                                                          False                                                           |
|  llm_adapter   |                        FastChatLLMModelAdaperWrapper(fastchat.model.model_adapter.VicunaAdapter)                         |
|  User prompt   | A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polit |
|                |                                e answers to the user's questions. USER: Hello ASSISTANT:                                 |
|  Model output  | Hello! How can I help you today? Is there something specific you want to know or talk about? I'm here to answer any ques |
|                |                                     tions you might have, to the best of my ability.                                     |
+----------------+--------------------------------------------------------------------------------------------------------------------------+

More chat Usage

dbgpt trace chat --help

Usage: dbgpt trace chat [OPTIONS] [FILES]...

  Show conversation details

Options:
  --trace_id TEXT                 Specify the trace ID to analyze. If None,
                                  show latest conversation details
  --tree                          Display trace spans as a tree
  --hide_conv                     Hide your conversation details
  --hide_run_params               Hide run params
  --output [text|html|csv|latex|json]
                                  The output format
  --help                          Show this message and exit.

Viewing Call Tree Based on trace_id

dbgpt trace tree --trace_id ec30d733-7b35-4d61-b02e-2832fd2e29ff

You will see an output like:

Operation: DB-GPT-Web-Entry (Start: 2023-10-12 03:22:10.592, End: None)
  Operation: get_chat_instance (Start: 2023-10-12 03:22:10.594, End: None)
  Operation: get_chat_instance (Start: 2023-10-12 03:22:10.594, End: 2023-10-12 03:22:10.658)
  Operation: stream_generator (Start: 2023-10-12 03:22:10.659, End: None)
    Operation: BaseChat.stream_call (Start: 2023-10-12 03:22:10.659, End: None)
      Operation: WorkerManager.generate_stream (Start: 2023-10-12 03:22:10.660, End: None)
        Operation: DefaultModelWorker.generate_stream (Start: 2023-10-12 03:22:10.675, End: None)
          Operation: DefaultModelWorker_call.generate_stream_func (Start: 2023-10-12 03:22:10.676, End: None)
          Operation: DefaultModelWorker_call.generate_stream_func (Start: 2023-10-12 03:22:10.676, End: 2023-10-12 03:22:16.130)
        Operation: DefaultModelWorker.generate_stream (Start: 2023-10-12 03:22:10.675, End: 2023-10-12 03:22:16.130)
      Operation: WorkerManager.generate_stream (Start: 2023-10-12 03:22:10.660, End: 2023-10-12 03:22:16.130)
    Operation: BaseChat.stream_call (Start: 2023-10-12 03:22:10.659, End: 2023-10-12 03:22:16.130)
  Operation: stream_generator (Start: 2023-10-12 03:22:10.659, End: 2023-10-12 03:22:16.130)
Operation: DB-GPT-Web-Entry (Start: 2023-10-12 03:22:10.592, End: 2023-10-12 03:22:10.673)

Listing Trace Information

Listing All Trace Information

dbgpt trace list

You will see an output like:

+--------------------------------------+---------------------------------------------------------------------------+-----------------------------------+------------------+
|               Trace ID               |                                  Span ID                                  |           Operation Name          | Conversation UID |
+--------------------------------------+---------------------------------------------------------------------------+-----------------------------------+------------------+
| eaf4830f-976f-45a4-9a50-244f3ab6f9e1 | eaf4830f-976f-45a4-9a50-244f3ab6f9e1:f650065f-f761-4790-99f7-8109c15f756a |           run_webserver           |       None       |
| eaf4830f-976f-45a4-9a50-244f3ab6f9e1 | eaf4830f-976f-45a4-9a50-244f3ab6f9e1:b2ff279e-0557-4b2d-8959-85e25dcfe94e |        EmbeddingLoader.load       |       None       |
| eaf4830f-976f-45a4-9a50-244f3ab6f9e1 | eaf4830f-976f-45a4-9a50-244f3ab6f9e1:b2ff279e-0557-4b2d-8959-85e25dcfe94e |        EmbeddingLoader.load       |       None       |
| eaf4830f-976f-45a4-9a50-244f3ab6f9e1 | eaf4830f-976f-45a4-9a50-244f3ab6f9e1:3e8b1b9d-5ef2-4382-af62-6b2b21cc04fd | WorkerManager._start_local_worker |       None       |
| eaf4830f-976f-45a4-9a50-244f3ab6f9e1 | eaf4830f-976f-45a4-9a50-244f3ab6f9e1:3e8b1b9d-5ef2-4382-af62-6b2b21cc04fd | WorkerManager._start_local_worker |       None       |
| eaf4830f-976f-45a4-9a50-244f3ab6f9e1 | eaf4830f-976f-45a4-9a50-244f3ab6f9e1:4c280ec9-0fd6-4ee8-b79f-1afcab0f9901 |      DefaultModelWorker.start     |       None       |
+--------------------------------------+---------------------------------------------------------------------------+-----------------------------------+------------------+

Listing Trace Information by Trace Type

dbgpt trace list --span_type chat

You will see an output like:

+--------------------------------------+---------------------------------------------------------------------------+-------------------+--------------------------------------+
|               Trace ID               |                                  Span ID                                  |   Operation Name  |           Conversation UID           |
+--------------------------------------+---------------------------------------------------------------------------+-------------------+--------------------------------------+
| 5d1900c3-5aad-4159-9946-fbb600666530 | 5d1900c3-5aad-4159-9946-fbb600666530:14772034-bed4-4b4e-b43f-fcf3a8aad6a7 | get_chat_instance | 5e456272-68ac-11ee-9fba-0242ac150003 |
| 5d1900c3-5aad-4159-9946-fbb600666530 | 5d1900c3-5aad-4159-9946-fbb600666530:14772034-bed4-4b4e-b43f-fcf3a8aad6a7 | get_chat_instance | 5e456272-68ac-11ee-9fba-0242ac150003 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:0482a0c5-38b3-4b38-8101-e42489f90ccd | get_chat_instance | 87a722de-68ae-11ee-9fba-0242ac150003 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:0482a0c5-38b3-4b38-8101-e42489f90ccd | get_chat_instance | 87a722de-68ae-11ee-9fba-0242ac150003 |
+--------------------------------------+---------------------------------------------------------------------------+-------------------+--------------------------------------+

Searching Trace Information

dbgpt trace list --search Hello

You will see an output like:

+--------------------------------------+---------------------------------------------------------------------------+----------------------------------------------+--------------------------------------+
|               Trace ID               |                                  Span ID                                  |                Operation Name                |           Conversation UID           |
+--------------------------------------+---------------------------------------------------------------------------+----------------------------------------------+--------------------------------------+
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:0482a0c5-38b3-4b38-8101-e42489f90ccd |              get_chat_instance               | 87a722de-68ae-11ee-9fba-0242ac150003 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:0482a0c5-38b3-4b38-8101-e42489f90ccd |              get_chat_instance               | 87a722de-68ae-11ee-9fba-0242ac150003 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:03de6c87-34d6-426a-85e8-7d46d475411e |             BaseChat.stream_call             |                 None                 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:03de6c87-34d6-426a-85e8-7d46d475411e |             BaseChat.stream_call             |                 None                 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:19593596-b4c7-4d15-a3c1-0924d86098dd | DefaultModelWorker_call.generate_stream_func |                 None                 |
| ec30d733-7b35-4d61-b02e-2832fd2e29ff | ec30d733-7b35-4d61-b02e-2832fd2e29ff:19593596-b4c7-4d15-a3c1-0924d86098dd | DefaultModelWorker_call.generate_stream_func |                 None                 |
+--------------------------------------+---------------------------------------------------------------------------+----------------------------------------------+--------------------------------------+

More list Usage

dbgpt trace list --help

Usage: dbgpt trace list [OPTIONS] [FILES]...

  List your trace spans

Options:
  --trace_id TEXT                 Specify the trace ID to list
  --span_id TEXT                  Specify the Span ID to list.
  --span_type TEXT                Specify the Span Type to list.
  --parent_span_id TEXT           Specify the Parent Span ID to list.
  --search TEXT                   Search trace_id, span_id, parent_span_id,
                                  operation_name or content in metadata.
  -l, --limit INTEGER             Limit the number of recent span displayed.
  --start_time TEXT               Filter by start time. Format: "YYYY-MM-DD
                                  HH:MM:SS.mmm"
  --end_time TEXT                 Filter by end time. Format: "YYYY-MM-DD
                                  HH:MM:SS.mmm"
  --desc                          Whether to use reverse sorting. By default,
                                  sorting is based on start time.
  --output [text|html|csv|latex|json]
                                  The output format
  --help                          Show this message and exit.
Reference in New Issue View Git Blame Copy Permalink