github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-17 23:18:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,897 Commits 59 Branches 51 Tags
grpo-profile
Commit Graph

420 Commits

Author SHA1 Message Date
YeAnbang
26d859f68e [feat] Support DAPO (#6263) 
* update help information

* update style

* fix

* minor fix

* support PP training

* add pp support

* remove unused code

* address conversation

* fix memory leakage support tp+pp

* move empty cache

* move empty cache

* add DAPO support

* remove format reward

* fix filtering, still buggy

* small fix

* add DAPO support

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tested multi-node training; fix bind_batch bug

* fix conversation; support sleep mode

* support reusing excessive samples

* add dynamic batching control flag

* add dynamic batching control flag

* refactored

* fix logging

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-04-25 17:39:17 +08:00
Tong Li
b823c6eec7 [feat] Add final save at the end (#6274) 
* add final save

* default 1 episode
 2025-04-23 10:03:46 +08:00
Tong Li
03f4b1dde3 add prompt template (#6273) 
Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-04-22 10:39:47 +08:00
YeAnbang
9467c10690 [hot-fix] Fix memory leakage bug, support TP+PP (#6258) 
* update help information

* update style

* fix

* minor fix

* support PP training

* add pp support

* remove unused code

* address conversation

* fix memory leakage support tp+pp

* move empty cache

* move empty cache

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-04-10 10:52:18 +08:00
YeAnbang
ed43a4be04 [Distributed RLHF] Integration of PP (#6257) 
* update help information

* update style

* fix

* minor fix

* support PP training

* add pp support

* remove unused code

* address conversation

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-04-09 13:23:24 +08:00
YeAnbang
50153005b4 [feat] add microbatch forwarding (#6251) 
* add microbatch forwarding

* fix forward microbatch

* fix producer OOM

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change project name

* fix temperature annealing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address conversation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-03-28 10:24:58 +08:00
YeAnbang
2aa7385c88 update logging 2025-03-21 16:12:07 +08:00
YeAnbang
d8eaf0d473 simplify vllm preprocessing input ids 2025-03-21 15:03:10 +08:00
YeAnbang
0472f44163 fix logprob, add filtering, temperature annealing, lr descent 2025-03-21 10:24:24 +08:00
YeAnbang
7ee4452f8c fix vllm 2025-03-19 17:11:10 +08:00
duanjunwen
7795d4c50d [Feature] Support Distributed LogProb for GRPO Training (#6247) 
* [fix] fix qwen VocabParallelLMHead1D and gather output

* fix tp bug

* fix consumer

* [feat] Support Distributed LogProb for GRPO Training

* [fix] fix loss func

* [fix] fix log prob plugin

* [fix] fix qwen modeling param

* [fix] rm comments

* [fix] rm hard-code;fix non-dist version

* [fix] fix test file param name and benchmark tp gather output=True/False

* [fix] rm non-dist version in dist log prob

* [fix] fix comments

* [fix] fix dis log prob plugin

* [fix] fix test case

* [fix] fix qwen VocabParallelLMHead1D and gather output

* [fix] fix DistLogProb comments

* [fix] restore tp size

* [fix] fix comments

* [fix] fix comment; fix LogSoftmax usage

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-03-18 17:47:55 +08:00
YeAnbang
bc0171d392 fix transformers backend 2025-03-14 18:12:35 +08:00
Tong Li
57b49da5e4 setup update 2025-03-13 16:52:15 +08:00
Tong Li
45ac6c6cb2 print results 2025-03-13 16:51:22 +08:00
Tong Li
4702d57841 convert to 8 generation 2025-03-13 16:49:02 +08:00
Tong Li
afddfde2dd fix consumer 2025-03-13 14:55:26 +08:00
Tong Li
704866a240 detach 2025-03-11 16:17:02 +08:00
Tong Li
47d6493778 add response length 2025-03-11 13:06:09 +08:00
Tong Li
abca66e69f fix reward score 2025-03-11 10:17:32 +08:00
Tong Li
71a0181fce update reward 2025-03-10 14:19:10 +08:00
Tong Li
754b16dfbf update reward fn 2025-03-10 14:18:22 +08:00
Tong Li
9d9d51614e update grpo 2025-03-10 14:12:04 +08:00
pre-commit-ci[bot]
eb6337f07f [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-03-06 08:29:59 +00:00
Tong Li
22cc1558a8 Merge branch 'grpo-latest' of github.com:hpcaitech/ColossalAI into grpo-latest 2025-03-06 16:28:47 +08:00
Tong Li
0590f10fb7 update select algo 2025-03-06 16:27:13 +08:00
Tong Li
0cc0c843ed add save 2025-03-06 16:26:14 +08:00
pre-commit-ci[bot]
ab5b6d8432 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-03-06 06:30:27 +00:00
Tong Li
0f566cc2d4 add algo selection 2025-03-06 14:29:22 +08:00
Tong Li
812f4b7750 update loader 2025-03-06 11:44:42 +08:00
Tong Li
7f2ceac5c3 update example 2025-03-06 10:54:23 +08:00
Tong Li
d03cdea949 update reward fn 2025-03-06 10:53:48 +08:00
Tong Li
678f5a9eca update loss 2025-03-06 10:53:03 +08:00
Tong Li
b96d69055e grpo consumer 2025-03-06 10:51:27 +08:00
Tong Li
c15225bc52 modify data loader 2025-03-06 10:49:44 +08:00
Tong Li
070907dd7f polish 2025-02-28 10:16:42 +08:00
Tong Li
f736d747e3 update grpo 2025-02-25 18:12:04 +08:00
Tong Li
ffd3878a1e add simple grpo 2025-02-23 22:54:26 +08:00
Tong Li
8e6c9a4ab3 add reward related function 2025-02-23 11:02:54 +08:00
Hongxin Liu
de282dd694 [feature] fit RL style generation (#6213) 
* [feature] fit rl style generation

* [doc] add docstr

* [doc] add docstr
 2025-02-21 17:28:19 +08:00
Hongxin Liu
43c9b5fb44 [chat] add distributed impl (#6210) 2025-02-21 15:24:23 +08:00
Hongxin Liu
f73ae55394 [application] add lora sft example data (#6198) 2025-02-18 20:18:18 +08:00
Tong Li
f8b9e88484 [application] Update README (#6196) 
* remove unused ray

* remove unused readme

* update readme

* update readme

* update

* update

* add link

* update readme

* update readme

* fix link

* update code

* update cititaion

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update readme

* update project

* add images

* update link

* update note

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-02-18 20:17:56 +08:00
Hongxin Liu
d54642a263 [application] add lora sft example (#6192) 
* [application] add lora sft example

* update requirements

* update readme

* update comment

* update ci
 2025-02-18 13:06:38 +08:00
YeAnbang
d20c8ffd97 Add GRPO and Support RLVR for PPO (#6186) 
* add grpo, support rlvr

* add grpo, support rlvr

* tested deepseek r1 pipeline

* add ci

* verify grpo r1

* verify grpo r1

* update readme, remove unused code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove path

* clean code

* fix circular import

* fix ci OOM

* fix ci OOM

* skip kto tp, fix qwen generation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-02-18 09:43:36 +08:00
flybird11111
aaafb38851 [Device]Support npu (#6159) 
* support npu

* support pretrain

support pretrain

fix

* support lora

fix

fix

* support chatglm

fix

fxi

fix

[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

fix

fix

[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

fix

[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

fix

fix

fix

* Update train.py

* Update train.py

* fix

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

* fix

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-12-17 15:42:39 +08:00
Tong Li
30a9443132 [Coati] Refine prompt for better inference (#6117) 
* refine prompt

* update prompt

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-11-08 11:00:37 +08:00
Tong Li
7a60161035 update readme (#6116) 2024-11-06 17:24:08 +08:00
Tong Li
89a9a600bc [MCTS] Add self-refined MCTS (#6098) 
* add reasoner

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update code

* delete llama

* update prompts

* update readme

* update readme

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-24 17:51:19 +08:00
Tong Li
4c8e85ee0d [Coati] Train DPO using PP (#6054) 
* update dpo

* remove unsupport plugin

* update msg

* update dpo

* remove unsupport plugin

* update msg

* update template

* update dataset

* add pp for dpo

* update dpo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add dpo fn

* update dpo

* update dpo

* update dpo

* update dpo

* minor update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update loss

* update help

* polish code

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-11 19:32:00 +08:00
Camille Zhong
f9546ba0be [ColossalEval] support for vllm (#6056) 
* support vllm

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* modify vllm and update readme

* run pre-commit

* remove dupilicated lines and refine code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update param name

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refine code

* update readme

* refine code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-09-18 17:09:45 +08:00