mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-01-17 07:58:15 +00:00

Files

Hongxin Liu 7f8b16635b [misc] refactor launch API and tensor constructor (#5666 )

* [misc] remove config arg from initialize

* [misc] remove old tensor contrusctor

* [plugin] add npu support for ddp

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [devops] fix doc test ci

* [test] fix test launch

* [doc] update launch doc

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

2024-04-29 10:40:11 +08:00

model

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

utils

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

arguments.py

[bug] fix get_default_parser in examples (#4764 )

2023-09-21 10:42:25 +08:00

bert_dataset_provider.py

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

evaluation.py

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

hostfile

[example] reorganize for community examples (#3557 )

2023-04-14 16:27:48 +08:00

loss.py

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

nvidia_bert_dataset_provider.py

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

pretrain_utils.py

[misc] update pre-commit and run all files (#4752 )

2023-09-19 14:20:26 +08:00

README.md

fix typo examples/community/roberta (#3925 )

2023-06-08 14:28:34 +08:00

run_pretrain_resume.sh

[example] reorganize for community examples (#3557 )

2023-04-14 16:27:48 +08:00

run_pretrain.sh

[example] reorganize for community examples (#3557 )

2023-04-14 16:27:48 +08:00

run_pretraining.py

[misc] refactor launch API and tensor constructor (#5666 )

2024-04-29 10:40:11 +08:00

README.md

Pretraining

Pretraining roberta through running the script below. Detailed parameter descriptions can be found in the arguments.py. data_path_prefix is absolute path specifies output of preprocessing. You have to modify the hostfile according to your cluster.

bash run_pretrain.sh

--hostfile: servers' host name from /etc/hosts
--include: servers which will be used
--nproc_per_node: number of process(GPU) from each server
--data_path_prefix: absolute location of train data, e.g., /h5/0.h5
--eval_data_path_prefix: absolute location of eval data
--tokenizer_path: tokenizer path contains huggingface tokenizer.json, e.g./tokenizer/tokenizer.json
--bert_config: config.json which represent model
--mlm: model type of backbone, bert or deberta_v2

if resume training from earlier checkpoint, run the script below.

bash run_pretrain_resume.sh

--resume_train: whether to resume training
--load_pretrain_model: absolute path which contains model checkpoint
--load_optimizer_lr: absolute path which contains optimizer checkpoint