github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-08-23 10:11:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,913 Commits 58 Branches 51 Tags 237 MiB
grpo_optimization
Commit Graph

49 Commits

Author SHA1 Message Date
YeAnbang
c2561f826a fix bugs 2025-06-20 15:44:13 +08:00
YeAnbang
ff6696a9bb support n_behind, add profiling 2025-06-20 03:14:00 +00:00
YeAnbang
e3d56cbd86 implement memory efficient logprob 2025-06-18 10:24:48 +00:00
YeAnbang
30a6859f77 optimize pp log_softmax OOM 2025-06-13 18:21:54 +08:00
Tong Li
8992def757
fix pp memory issue (#6344) 
Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-06-11 17:54:18 +08:00
pre-commit-ci[bot]
9ca920c1af [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-06-09 01:48:20 +00:00
YeAnbang
dc3033e68a support code generation tasks 2025-06-05 18:05:22 +08:00
pre-commit-ci[bot]
0d008110e7 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-05-29 10:16:56 +00:00
YeAnbang
7b921acc8a merge grpo-latest 2025-05-29 18:14:43 +08:00
YeAnbang
ee939d9aa5 address conversation 2025-05-29 10:25:59 +08:00
Tong Li
c8b368c294
add overlength sample count (#6332) 
Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-05-28 19:18:09 +08:00
YeAnbang
4c3656870a address conversation 2025-05-28 17:34:11 +08:00
YeAnbang
78a06f5ce3 fix missing tags parameter 2025-05-21 10:51:32 +08:00
YeAnbang
37663386bc fix metric calculation 2025-05-20 18:14:05 +08:00
Tong Li
32afa7bf29
fix empty tensor (#6319) 
Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-05-20 17:41:44 +08:00
YeAnbang
116621d004 merge reward and eval 2025-05-19 11:53:47 +08:00
YeAnbang
107470a360 fix logging rollouts 2025-05-17 21:12:58 +08:00
YeAnbang
03b41d6fb5 upgrade reward functions 2025-05-16 18:04:38 +08:00
YeAnbang
11a5854b50 remove redundant code and fix bugs 2025-05-16 14:08:23 +08:00
Tong Li
1644adf684 handle empty index 2025-05-15 20:32:05 +08:00
YeAnbang
a528921944 move prompt-level-filtering to buffer side 2025-05-15 18:30:32 +08:00
YeAnbang
55eee129d2 move prompt-level-filtering to buffer side 2025-05-15 18:16:50 +08:00
YeAnbang
957e3a521a disable wandb tb syncing 2025-05-15 16:52:31 +08:00
YeAnbang
4ec73298b2 use consumer global step 2025-05-15 14:15:40 +08:00
YeAnbang
094f119b3a merge 2025-05-14 18:13:47 +08:00
YeAnbang
50070c1e84 move logging to producer 2025-05-14 18:10:57 +08:00
Tong Li
aca547623f
[feat] Support prompt level dynamic (#6300) 
* adjust to dynamic prompt bs

* remove debug

* update pad seq (#6303)

Co-authored-by: Tong Li <tong.li35271158@gmail.com>

* adjust to dynamic prompt bs

* remove debug

* fix dp issue

* fix

* fix default settings

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-05-14 16:40:35 +08:00
YeAnbang
47a7dc7142 Support evaluation during training 2025-05-14 11:03:11 +08:00
YeAnbang
eb6b5dd62e
[fix] revert reward update and evaluation (#6295) 
* Revert "rewrite reward fn"

This reverts commit d06042b434.

* Revert "upgrade reward math verification"

This reverts commit a6085ff676.

* Revert "fix bug"

This reverts commit 01640ebd65.

* Revert "reuse comm-group"

This reverts commit bd61918dcf.

* Revert "Support evaluation during training"

This reverts commit 57a88395fe.
 2025-05-07 10:56:47 +08:00
YeAnbang
57a88395fe Support evaluation during training 2025-04-30 18:31:49 +08:00
YeAnbang
14f237ce7e
[feat] Support boxed math reward (#6284) 
* fix pp+tp, fix dataloader

* fixed plugin micro-batch size

* support boxed reward

* add boxed reward

* fix pp state dict incomplete issue

* Revert "fix pp state dict incomplete issue"

This reverts commit 6c1b3b694f.
 2025-04-29 16:46:47 +08:00
YeAnbang
2ca1e3c630
fix pp+tp, fix dataloader (#6280) 2025-04-28 17:10:00 +08:00
Tong Li
28795f560c
fix save issue (#6279) 
Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-04-27 17:54:06 +08:00
YeAnbang
26d859f68e
[feat] Support DAPO (#6263) 
* update help information

* update style

* fix

* minor fix

* support PP training

* add pp support

* remove unused code

* address conversation

* fix memory leakage support tp+pp

* move empty cache

* move empty cache

* add DAPO support

* remove format reward

* fix filtering, still buggy

* small fix

* add DAPO support

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tested multi-node training; fix bind_batch bug

* fix conversation; support sleep mode

* support reusing excessive samples

* add dynamic batching control flag

* add dynamic batching control flag

* refactored

* fix logging

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-04-25 17:39:17 +08:00
YeAnbang
9467c10690
[hot-fix] Fix memory leakage bug, support TP+PP (#6258) 
* update help information

* update style

* fix

* minor fix

* support PP training

* add pp support

* remove unused code

* address conversation

* fix memory leakage support tp+pp

* move empty cache

* move empty cache

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-04-10 10:52:18 +08:00
YeAnbang
ed43a4be04
[Distributed RLHF] Integration of PP (#6257) 
* update help information

* update style

* fix

* minor fix

* support PP training

* add pp support

* remove unused code

* address conversation

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-04-09 13:23:24 +08:00
YeAnbang
50153005b4
[feat] add microbatch forwarding (#6251) 
* add microbatch forwarding

* fix forward microbatch

* fix producer OOM

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change project name

* fix temperature annealing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address conversation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-03-28 10:24:58 +08:00
YeAnbang
2aa7385c88 update logging 2025-03-21 16:12:07 +08:00
YeAnbang
0472f44163 fix logprob, add filtering, temperature annealing, lr descent 2025-03-21 10:24:24 +08:00
YeAnbang
7ee4452f8c fix vllm 2025-03-19 17:11:10 +08:00
duanjunwen
7795d4c50d
[Feature] Support Distributed LogProb for GRPO Training (#6247) 
* [fix] fix qwen VocabParallelLMHead1D and gather output

* fix tp bug

* fix consumer

* [feat] Support Distributed LogProb for GRPO Training

* [fix] fix loss func

* [fix] fix log prob plugin

* [fix] fix qwen modeling param

* [fix] rm comments

* [fix] rm hard-code;fix non-dist version

* [fix] fix test file param name and benchmark tp gather output=True/False

* [fix] rm non-dist version in dist log prob

* [fix] fix comments

* [fix] fix dis log prob plugin

* [fix] fix test case

* [fix] fix qwen VocabParallelLMHead1D and gather output

* [fix] fix DistLogProb comments

* [fix] restore tp size

* [fix] fix comments

* [fix] fix comment; fix LogSoftmax usage

---------

Co-authored-by: Tong Li <tong.li35271158@gmail.com>
 2025-03-18 17:47:55 +08:00
Tong Li
47d6493778 add response length 2025-03-11 13:06:09 +08:00
Tong Li
9d9d51614e update grpo 2025-03-10 14:12:04 +08:00
pre-commit-ci[bot]
eb6337f07f [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-03-06 08:29:59 +00:00
Tong Li
0cc0c843ed add save 2025-03-06 16:26:14 +08:00
Tong Li
b96d69055e grpo consumer 2025-03-06 10:51:27 +08:00
Tong Li
070907dd7f polish 2025-02-28 10:16:42 +08:00
Tong Li
f736d747e3 update grpo 2025-02-25 18:12:04 +08:00
Tong Li
ffd3878a1e add simple grpo 2025-02-23 22:54:26 +08:00