github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-01 09:07:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,821 Commits 58 Branches 51 Tags
feat/ppo
Commit Graph

53 Commits

Author SHA1 Message Date
pre-commit-ci[bot]
6e096362ef [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-03-07 10:43:03 +00:00
YeAnbang
c8e13a9403 run pre-commit 2025-03-07 18:40:31 +08:00
YeAnbang
d31f9e4d0f run pre-commit 2025-03-07 18:30:19 +08:00
YeAnbang
6a6634b6e8 add ppo 2025-03-07 18:29:34 +08:00
pre-commit-ci[bot]
eb6337f07f [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2025-03-06 08:29:59 +00:00
Tong Li
0cc0c843ed add save 2025-03-06 16:26:14 +08:00
Tong Li
0f566cc2d4 add algo selection 2025-03-06 14:29:22 +08:00
Tong Li
812f4b7750 update loader 2025-03-06 11:44:42 +08:00
Tong Li
d03cdea949 update reward fn 2025-03-06 10:53:48 +08:00
Tong Li
678f5a9eca update loss 2025-03-06 10:53:03 +08:00
Tong Li
b96d69055e grpo consumer 2025-03-06 10:51:27 +08:00
Tong Li
c15225bc52 modify data loader 2025-03-06 10:49:44 +08:00
Tong Li
070907dd7f polish 2025-02-28 10:16:42 +08:00
Tong Li
f736d747e3 update grpo 2025-02-25 18:12:04 +08:00
Tong Li
ffd3878a1e add simple grpo 2025-02-23 22:54:26 +08:00
Tong Li
8e6c9a4ab3 add reward related function 2025-02-23 11:02:54 +08:00
Hongxin Liu
de282dd694 [feature] fit RL style generation (#6213) 
* [feature] fit rl style generation

* [doc] add docstr

* [doc] add docstr
 2025-02-21 17:28:19 +08:00
Hongxin Liu
43c9b5fb44 [chat] add distributed impl (#6210) 2025-02-21 15:24:23 +08:00
Tong Li
f8b9e88484 [application] Update README (#6196) 
* remove unused ray

* remove unused readme

* update readme

* update readme

* update

* update

* add link

* update readme

* update readme

* fix link

* update code

* update cititaion

* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update readme

* update project

* add images

* update link

* update note

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-02-18 20:17:56 +08:00
Hongxin Liu
d54642a263 [application] add lora sft example (#6192) 
* [application] add lora sft example

* update requirements

* update readme

* update comment

* update ci
 2025-02-18 13:06:38 +08:00
YeAnbang
d20c8ffd97 Add GRPO and Support RLVR for PPO (#6186) 
* add grpo, support rlvr

* add grpo, support rlvr

* tested deepseek r1 pipeline

* add ci

* verify grpo r1

* verify grpo r1

* update readme, remove unused code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove path

* clean code

* fix circular import

* fix ci OOM

* fix ci OOM

* skip kto tp, fix qwen generation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2025-02-18 09:43:36 +08:00
Tong Li
30a9443132 [Coati] Refine prompt for better inference (#6117) 
* refine prompt

* update prompt

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-11-08 11:00:37 +08:00
Tong Li
89a9a600bc [MCTS] Add self-refined MCTS (#6098) 
* add reasoner

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update code

* delete llama

* update prompts

* update readme

* update readme

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-24 17:51:19 +08:00
Tong Li
4c8e85ee0d [Coati] Train DPO using PP (#6054) 
* update dpo

* remove unsupport plugin

* update msg

* update dpo

* remove unsupport plugin

* update msg

* update template

* update dataset

* add pp for dpo

* update dpo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add dpo fn

* update dpo

* update dpo

* update dpo

* update dpo

* minor update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update loss

* update help

* polish code

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-10-11 19:32:00 +08:00
Tong Li
39e2597426 [ColossalChat] Add PP support (#6001) 
* support pp training

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update rm

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update test case

* fix

* change to 4

* fix eval

* test

* add pp

* hotfix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* support pp training

* update rm

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update test case

* fix

* change to 4

* fix eval

* test

* add pp

* hotfix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

* skip pp eval

* update all reduce

* update sft

* update ignore

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update no cache

* add eval

* remove fi

* remove debug

* remove parentheses to avoid warning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "add eval"

This reverts commit 3ab2f6fa32.

* add all reduce

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-08-21 10:47:39 +08:00
YeAnbang
ed97d3a5d3 [Chat] fix readme (#5989) 
* fix readme

* fix readme, tokenization fully tested

* fix readme, tokenization fully tested

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: root <root@notebook-8f919155-6035-47b4-9c6f-1be133b9e2c9-0.notebook-8f919155-6035-47b4-9c6f-1be133b9e2c9.colossal-ai.svc.cluster.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-08-12 14:55:17 +08:00
YeAnbang
0b2d55c4ab Support overall loss, update KTO logging 2024-08-02 06:51:38 +00:00
YeAnbang
30f4e31a33 [Chat] Fix lora (#5946) 
* fix merging

* remove filepath

* fix style
 2024-07-31 14:10:17 +08:00
YeAnbang
6fd9e86864 fix style 2024-07-29 01:29:18 +00:00
YeAnbang
de1bf08ed0 fix style 2024-07-26 10:07:15 +00:00
YeAnbang
8a3ff4f315 fix style 2024-07-26 09:55:15 +00:00
YeAnbang
150505cbb8 Merge branch 'kto' of https://github.com/hpcaitech/ColossalAI into kto 2024-07-19 10:11:05 +00:00
YeAnbang
d49550fb49 refactor tokenization 2024-07-19 10:10:48 +00:00
Tong Li
d08c99be0d Merge branch 'main' into kto 2024-07-19 15:23:31 +08:00
Tong Li
f585d4e38e [ColossalChat] Hotfix for ColossalChat (#5910) 
* add ignore and tiny llama

* fix path issue

* run style

* fix issue

* update bash

* add ignore and tiny llama

* fix path issue

* run style

* fix issue

* update bash

* fix ddp issue

* add Qwen 1.5 32B
 2024-07-19 13:40:07 +08:00
YeAnbang
544b7a38a1 fix style, add kto data sample 2024-07-18 08:38:56 +00:00
YeAnbang
09d5ffca1a add kto 2024-07-18 07:54:11 +00:00
YeAnbang
b3594d4d68 fix orpo cross entropy loss 2024-07-15 02:12:05 +00:00
YeAnbang
e7a8634636 fix eval 2024-07-11 03:35:03 +00:00
YeAnbang
d888c3787c add benchmark for sft, dpo, simpo, orpo. Add benchmarking result. Support lora with gradient checkpoint 2024-07-10 10:17:08 +00:00
YeAnbang
16f3451fe2 Merge branch 'main' of https://github.com/hpcaitech/ColossalAI into rlhf_SimPO 2024-07-10 02:32:07 +00:00
pre-commit-ci[bot]
7c2f79fa98 [pre-commit.ci] pre-commit autoupdate (#5572) 
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/PyCQA/autoflake: v2.2.1 → v2.3.1](https://github.com/PyCQA/autoflake/compare/v2.2.1...v2.3.1)
- [github.com/pycqa/isort: 5.12.0 → 5.13.2](https://github.com/pycqa/isort/compare/5.12.0...5.13.2)
- [github.com/psf/black-pre-commit-mirror: 23.9.1 → 24.4.2](https://github.com/psf/black-pre-commit-mirror/compare/23.9.1...24.4.2)
- [github.com/pre-commit/mirrors-clang-format: v13.0.1 → v18.1.7](https://github.com/pre-commit/mirrors-clang-format/compare/v13.0.1...v18.1.7)
- [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.6.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v4.6.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-01 17:16:41 +08:00
YeAnbang
8aad064fe7 fix style 2024-06-27 07:29:33 +00:00
YeAnbang
c8d1b4a968 add orpo 2024-06-27 07:20:28 +00:00
YeAnbang
f3de5a025c remove debug code 2024-06-24 05:16:29 +00:00
YeAnbang
0b2d6275c4 fix dataloader 2024-06-24 05:10:44 +00:00
YeAnbang
82aecd6374 add SimPO 2024-06-24 02:12:20 +00:00
YeAnbang
0d7ff10ea5 replace the customized dataloader setup with the build-in one 2024-06-07 09:43:42 +00:00
pre-commit-ci[bot]
1b880ce095 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2024-06-07 07:01:31 +00:00
YeAnbang
0b4a33548c moupdate ci tests, st ci test cases passed, tp failed in generation for ppo, sp is buggy 2024-06-07 07:01:31 +00:00