57 Commits

Author	SHA1	Message	Date
duanjunwen	41fdd2139b	[fix] rm unused comments	2024-11-18 16:48:21 +08:00
duanjunwen	9a21f87ed6	[fix] fix wait handle in run_fwd_bwd	2024-11-18 02:50:14 +00:00
duanjunwen	0fb500c7d4	[fix] rm debug info; update llama policy; update wait handle	2024-11-15 09:47:05 +00:00
duanjunwen	cf86c1b1c5	[fix] fix zbv wait_handle	2024-11-15 07:56:14 +00:00
duanjunwen	5c2ebbfd48	[fix] fix mixtral modeling & policy; update wait handles; doing benchmarking for llama hybrid;	2024-11-15 05:58:56 +00:00
duanjunwen	1bc4dba3a3	[fix] fix p2p error in zbv	2024-11-14 09:40:38 +00:00
duanjunwen	80b04d7855	[feat] support mixtral policy with zbv tp_Linear & non_tp_Linear	2024-11-12 07:28:49 +00:00
duanjunwen	337debcf2a	[feat] fix testcase;	2024-11-11 11:34:29 +00:00
duanjunwen	12919de424	[fix] fix send_tensor_metadata & send_grad_metadata;	2024-11-11 08:54:39 +00:00
duanjunwen	0d6d40ccc6	[fix] fix zbv llama pp4	2024-11-06 03:35:12 +00:00
duanjunwen	4fc92aa77d	[feat] support no_tp Linear for sharderformer.llama	2024-11-05 05:55:42 +00:00
duanjunwen	fa3ccda8ee	[fix] fix send recv signature;	2024-10-29 03:33:58 +00:00
duanjunwen	fafe049b83	[fix] fix handle name; rm useless comments;	2024-10-29 03:24:15 +00:00
duanjunwen	d0ec221b38	[fix\ fix fail case test_shard_llama	2024-10-25 02:28:55 +00:00
duanjunwen	2eca112c90	[feat] support meta cache, meta_grad_send, meta_tensor_send; fix runtime too long in Recv Bwd; benchmark for llama + Hybrid(tp+pp);	2024-10-24 07:30:19 +00:00
duanjunwen	9912cc8c07	[fix] fix bwd b; now bwd w only for Layer replaced by Linear1D_Col/Row; other layer perform a fully bwd;	2024-10-15 06:26:01 +00:00
duanjunwen	cfade4c36d	[feat] Linear1D_COL/ROW support zbv WeightGradStore;	2024-10-14 07:02:43 +00:00
duanjunwen	0ca16d5cbe	[fix] fix llama, mixtral benchmark zbv loss none bug; update mixtral & llama policy and modeling;	2024-10-11 07:32:43 +00:00
duanjunwen	1342a983b1	[fix] rm print & comments;	2024-09-26 11:05:27 +00:00
duanjunwen	64ceea746f	[fix] remove chunk 0 stage 0 bwd b; u don't have to cal micrbatch's dx;	2024-09-26 10:50:44 +00:00
duanjunwen	a92e16719b	[fix] fix zerobubble; support shardformer model type;	2024-09-26 06:11:56 +00:00
duanjunwen	83163fa70c	[fix] fix traverse; traverse dict --> traverse tensor List;	2024-09-25 06:38:11 +00:00
duanjunwen	fc8b016887	[fix] fix stage_indices;	2024-09-25 06:15:45 +00:00
duanjunwen	7e6f793c51	[fix] fix detach_output_obj clone;	2024-09-24 08:08:32 +00:00
duanjunwen	6c1e1550ae	[fix] fix dumb clone;	2024-09-23 06:43:49 +00:00
duanjunwen	c114d1429a	[fix] fix detach clone release order;	2024-09-23 04:00:24 +00:00
duanjunwen	da3220f48c	[fix] fix pipeline util func deallocate --> release_tensor_data; fix bwd_b loss bwd branch;	2024-09-20 09:48:35 +00:00
duanjunwen	1739df423c	[fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'	2024-09-20 07:34:43 +00:00
duanjunwen	b6616f544e	[fix] rm comments;	2024-09-20 07:29:41 +00:00
duanjunwen	c6d6ee39bd	[fix] use tree_flatten replace dict traverse;	2024-09-20 07:18:49 +00:00
duanjunwen	26783776f1	[fix] fix input_tensors buffer append input_obj(dict) --> Tuple (microbatch, input_obj) , and all bwd b related cal logic;	2024-09-20 06:41:19 +00:00
duanjunwen	a115106f8d	[fix] fix bwd w input;	2024-09-19 08:10:05 +00:00
duanjunwen	349272c71f	[fix] updatw bwd b&w input; dict --> list[torch.Tensor]	2024-09-19 07:47:01 +00:00
duanjunwen	6ee9584b9a	[fix] fix require_grad & deallocate call;	2024-09-19 05:53:03 +00:00
duanjunwen	3dbad102cf	[fix] fix zerobubble pp for shardformer type input;	2024-09-18 07:14:34 +00:00
duanjunwen	ce58d8e8bf	[fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap;	2024-09-09 08:19:58 +00:00
duanjunwen	7568b34626	[fix] fix redundant detach & clone; add buffer assertation in the end;	2024-09-09 08:04:28 +00:00
duanjunwen	e6e1a97a6d	[fix] fix requir grad position and detach position and input&output local buffer append position;	2024-09-04 03:31:08 +00:00
duanjunwen	20503cdfdf	[fix] rm requir_grad for output;	2024-09-03 09:24:40 +00:00
duanjunwen	b4103f125c	[fix] fix detach output & release output;	2024-09-03 09:09:41 +00:00
duanjunwen	4c1f81c683	[fix] fix bwd step if condition; remove useless comments and format info;	2024-09-03 08:56:08 +00:00
duanjunwen	ab643c9af7	[fix] rm output.data after send fwd;	2024-09-03 14:12:17 +08:00
duanjunwen	591a13bf7e	[fix] fix optim bwd;	2024-09-02 11:19:42 +00:00
duanjunwen	6d18d38d5c	[feat] update test; rm comments;	2024-09-02 09:50:47 +00:00
duanjunwen	a7b767b071	[fix] fix communication_map;	2024-08-30 05:56:02 +00:00
duanjunwen	8eb6eac225	[fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;	2024-08-30 05:42:43 +00:00
duanjunwen	6af81d8c0d	[feat] add fwd_bwd_step, run_fwd_only;	2024-08-30 02:47:52 +00:00
duanjunwen	48ba22dbfd	[feat] fix optimizer bwd b & w; support return accum loss & output	2024-08-29 08:54:45 +00:00
duanjunwen	4c4b01b859	[feat] add optim backward_b_by_grad	2024-08-29 03:16:59 +00:00
duanjunwen	8b37323f16	[feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass;	2024-08-27 09:31:38 +00:00