41 Commits

Author	SHA1	Message	Date
duanjunwen	cfade4c36d	[feat] Linear1D_COL/ROW support zbv WeightGradStore;	2024-10-14 07:02:43 +00:00
duanjunwen	0ca16d5cbe	[fix] fix llama, mixtral benchmark zbv loss none bug; update mixtral & llama policy and modeling;	2024-10-11 07:32:43 +00:00
duanjunwen	1342a983b1	[fix] rm print & comments;	2024-09-26 11:05:27 +00:00
duanjunwen	64ceea746f	[fix] remove chunk 0 stage 0 bwd b; u don't have to cal micrbatch's dx;	2024-09-26 10:50:44 +00:00
duanjunwen	a92e16719b	[fix] fix zerobubble; support shardformer model type;	2024-09-26 06:11:56 +00:00
duanjunwen	83163fa70c	[fix] fix traverse; traverse dict --> traverse tensor List;	2024-09-25 06:38:11 +00:00
duanjunwen	fc8b016887	[fix] fix stage_indices;	2024-09-25 06:15:45 +00:00
duanjunwen	7e6f793c51	[fix] fix detach_output_obj clone;	2024-09-24 08:08:32 +00:00
duanjunwen	6c1e1550ae	[fix] fix dumb clone;	2024-09-23 06:43:49 +00:00
duanjunwen	c114d1429a	[fix] fix detach clone release order;	2024-09-23 04:00:24 +00:00
duanjunwen	da3220f48c	[fix] fix pipeline util func deallocate --> release_tensor_data; fix bwd_b loss bwd branch;	2024-09-20 09:48:35 +00:00
duanjunwen	1739df423c	[fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'	2024-09-20 07:34:43 +00:00
duanjunwen	b6616f544e	[fix] rm comments;	2024-09-20 07:29:41 +00:00
duanjunwen	c6d6ee39bd	[fix] use tree_flatten replace dict traverse;	2024-09-20 07:18:49 +00:00
duanjunwen	26783776f1	[fix] fix input_tensors buffer append input_obj(dict) --> Tuple (microbatch, input_obj) , and all bwd b related cal logic;	2024-09-20 06:41:19 +00:00
duanjunwen	a115106f8d	[fix] fix bwd w input;	2024-09-19 08:10:05 +00:00
duanjunwen	349272c71f	[fix] updatw bwd b&w input; dict --> list[torch.Tensor]	2024-09-19 07:47:01 +00:00
duanjunwen	6ee9584b9a	[fix] fix require_grad & deallocate call;	2024-09-19 05:53:03 +00:00
duanjunwen	3dbad102cf	[fix] fix zerobubble pp for shardformer type input;	2024-09-18 07:14:34 +00:00
duanjunwen	ce58d8e8bf	[fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap;	2024-09-09 08:19:58 +00:00
duanjunwen	7568b34626	[fix] fix redundant detach & clone; add buffer assertation in the end;	2024-09-09 08:04:28 +00:00
duanjunwen	e6e1a97a6d	[fix] fix requir grad position and detach position and input&output local buffer append position;	2024-09-04 03:31:08 +00:00
duanjunwen	20503cdfdf	[fix] rm requir_grad for output;	2024-09-03 09:24:40 +00:00
duanjunwen	b4103f125c	[fix] fix detach output & release output;	2024-09-03 09:09:41 +00:00
duanjunwen	4c1f81c683	[fix] fix bwd step if condition; remove useless comments and format info;	2024-09-03 08:56:08 +00:00
duanjunwen	ab643c9af7	[fix] rm output.data after send fwd;	2024-09-03 14:12:17 +08:00
duanjunwen	591a13bf7e	[fix] fix optim bwd;	2024-09-02 11:19:42 +00:00
duanjunwen	6d18d38d5c	[feat] update test; rm comments;	2024-09-02 09:50:47 +00:00
duanjunwen	a7b767b071	[fix] fix communication_map;	2024-08-30 05:56:02 +00:00
duanjunwen	8eb6eac225	[fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;	2024-08-30 05:42:43 +00:00
duanjunwen	6af81d8c0d	[feat] add fwd_bwd_step, run_fwd_only;	2024-08-30 02:47:52 +00:00
duanjunwen	48ba22dbfd	[feat] fix optimizer bwd b & w; support return accum loss & output	2024-08-29 08:54:45 +00:00
duanjunwen	4c4b01b859	[feat] add optim backward_b_by_grad	2024-08-29 03:16:59 +00:00
duanjunwen	8b37323f16	[feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass;	2024-08-27 09:31:38 +00:00
duanjunwen	9e0bd1af00	[fix] fix ci test; add pytest;	2024-08-27 08:00:23 +00:00
duanjunwen	283c9ff5d2	[fix] rm useless assign and comments;	2024-08-27 07:31:58 +00:00
duanjunwen	1b4bb2beeb	[feat] add comments for ZBV func;	2024-08-27 07:11:50 +00:00
duanjunwen	5e09c8b4e1	[feat] split communication and calculation; fix pop empty send_bwd_buffer error;	2024-08-27 06:29:13 +00:00
duanjunwen	1d75045c37	[feat] add test run_fwd_bwd automatic scheduling;	2024-08-26 11:21:56 +00:00
duanjunwen	c18ef060cf	[feat] add dw test;	2024-08-23 06:04:12 +00:00
duanjunwen	ee9baedadf	[feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;	2024-08-22 10:25:34 +00:00