github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-05 19:13:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

[hotfix]: add pp sanity check and fix mbs arg (#5268)

Browse Source 
* fix: fix misleading mbs arg

* feat: add pp sanity check

* fix: fix 1f1b sanity check
This commit is contained in:
Wenhao Chen
2024-01-15 15:57:40 +08:00
committed by GitHub
parent c174c4fc5f
commit ef4f0ee854
5 changed files with 21 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
colossalai/pipeline/schedule/interleaved_pp.py
View File

@@ -72,6 +72,10 @@ class InterleavedSchedule(PipelineSchedule):
assert self.last_batch_size is None or self.last_batch_size == self.batch_size
assert self.batch_size == self.microbatch_size * self.num_microbatch
assert (
self.num_microbatch % self.stage_manager.num_stages == 0
), "Number of microbatch should be an integer multiple of number of pipeline parallel devices"
if self.forward_only:
self.num_microbatch = (self.batch_size - 1) // self.microbatch_size + 1
# NOTE: disable metadata cache when batch size changes (not valid anymore)