github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-07-19 18:00:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Dev/zero offload (#5858)

Browse Source 
* fix llama

* fix llama
This commit is contained in:
Wang Binluo 2024-06-26 16:07:06 +08:00 committed by GitHub
parent de3f67d128
commit 868afdb311
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 0 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
colossalai/shardformer/modeling/llama.py
View File

@ -803,8 +803,6 @@ def get_lm_forward_with_dist_cross_entropy(shard_config: ShardConfig):
shift_logits, shift_logits,
shift_labels, shift_labels,
process_group=shard_config.tensor_parallel_process_group, process_group=shard_config.tensor_parallel_process_group,
vocab_size=self.lm_head.out_features,
dtype=self.model.dtype,
) )
if not return_dict: if not return_dict: