Develop/experiments (#59)

* Add gradient accumulation, fix lr scheduler

* fix FP16 optimizer and adapted torch amp with tensor parallel (#18)

* fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes

* fixed trainer

* Revert "fixed trainer"

This reverts commit 2e0b0b7699.

* improved consistency between trainer, engine and schedule (#23)

Co-authored-by: 1SAA <c2h214748@gmail.com>

* Split conv2d, class token, positional embedding in 2d, Fix random number in ddp
Fix convergence in cifar10, Imagenet1000

* Integrate 1d tensor parallel in Colossal-AI (#39)

* fixed 1D and 2D convergence (#38)

* optimized 2D operations

* fixed 1D ViT convergence problem

* Feature/ddp (#49)

* remove redundancy func in setup (#19) (#20)

* use env to control the language of doc (#24) (#25)

* Support TP-compatible Torch AMP and Update trainer API (#27)

* Add gradient accumulation, fix lr scheduler

* fix FP16 optimizer and adapted torch amp with tensor parallel (#18)

* fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes

* fixed trainer

* Revert "fixed trainer"

This reverts commit 2e0b0b7699.

* improved consistency between trainer, engine and schedule (#23)

Co-authored-by: 1SAA <c2h214748@gmail.com>

Co-authored-by: 1SAA <c2h214748@gmail.com>
Co-authored-by: ver217 <lhx0217@gmail.com>

* add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29)

* add explanation for ViT example (#35) (#36)

* support torch ddp

* fix loss accumulation

* add log for ddp

* change seed

* modify timing hook

Co-authored-by: Frank Lee <somerlee.9@gmail.com>
Co-authored-by: 1SAA <c2h214748@gmail.com>
Co-authored-by: binmakeswell <binmakeswell@gmail.com>

* Feature/pipeline (#40)

* remove redundancy func in setup (#19) (#20)

* use env to control the language of doc (#24) (#25)

* Support TP-compatible Torch AMP and Update trainer API (#27)

* Add gradient accumulation, fix lr scheduler

* fix FP16 optimizer and adapted torch amp with tensor parallel (#18)

* fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes

* fixed trainer

* Revert "fixed trainer"

This reverts commit 2e0b0b7699.

* improved consistency between trainer, engine and schedule (#23)

Co-authored-by: 1SAA <c2h214748@gmail.com>

Co-authored-by: 1SAA <c2h214748@gmail.com>
Co-authored-by: ver217 <lhx0217@gmail.com>

* add an example of ViT-B/16 and remove w_norm clipping in LAMB (#29)

* add explanation for ViT example (#35) (#36)

* optimize communication of pipeline parallel

* fix grad clip for pipeline

Co-authored-by: Frank Lee <somerlee.9@gmail.com>
Co-authored-by: 1SAA <c2h214748@gmail.com>
Co-authored-by: binmakeswell <binmakeswell@gmail.com>

* optimized 3d layer to fix slow computation ; tested imagenet performance with 3d; reworked lr_scheduler config definition; fixed launch args; fixed some printing issues; simplified apis of 3d layers (#51)

* Update 2.5d layer code to get a similar accuracy on imagenet-1k dataset

* update api for better usability (#58)

update api for better usability

Co-authored-by: 1SAA <c2h214748@gmail.com>
Co-authored-by: ver217 <lhx0217@gmail.com>
Co-authored-by: puck_WCR <46049915+WANG-CR@users.noreply.github.com>
Co-authored-by: binmakeswell <binmakeswell@gmail.com>
Co-authored-by: アマデウス <kurisusnowdeng@users.noreply.github.com>
Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>

tests/test_data/test_cifar10_dataset.py

@@ -5,39 +5,50 @@ import os
 from pathlib import Path
 
 import pytest
+from torchvision import transforms
 from torch.utils.data import DataLoader
 
-from colossalai.builder import build_dataset
+from colossalai.builder import build_dataset, build_transform
 from colossalai.context import Config
 
-train_data = dict(
+TRAIN_DATA = dict(
     dataset=dict(
-        type='CIFAR10Dataset',
+        type='CIFAR10',
         root=Path(os.environ['DATA']),
         train=True,
-        download=True,
-        transform_pipeline=[
-            dict(type='ToTensor'),
-            dict(type='Normalize',
-                 mean=(0.5, 0.5, 0.5),
-                 std=(0.5, 0.5, 0.5))
-        ]),
-    dataloader=dict(batch_size=4, shuffle=True, num_workers=2)
+        download=True
+    ),
+    dataloader=dict(batch_size=4, shuffle=True, num_workers=2),
+    transform_pipeline=[
+        dict(type='ToTensor'),
+        dict(type='Normalize',
+             mean=(0.5, 0.5, 0.5),
+             std=(0.5, 0.5, 0.5)
+             )
+    ]
 )
 
 
 @pytest.mark.cpu
 def test_cifar10_dataset():
-    global train_data
-    config = Config(train_data)
-    dataset = build_dataset(config.dataset)
-    dataloader = DataLoader(dataset=dataset, **config.dataloader)
+    config = Config(TRAIN_DATA)
+    dataset_cfg = config.dataset
+    dataloader_cfg = config.dataloader
+    transform_cfg = config.transform_pipeline
+
+    # build transform
+    transform_pipeline = [build_transform(cfg) for cfg in transform_cfg]
+    transform_pipeline = transforms.Compose(transform_pipeline)
+    dataset_cfg['transform'] = transform_pipeline
+
+    # build dataset
+    dataset = build_dataset(dataset_cfg)
+
+    # build dataloader
+    dataloader = DataLoader(dataset=dataset, **dataloader_cfg)
     data_iter = iter(dataloader)
     img, label = data_iter.next()
 
-    assert isinstance(img, list) and isinstance(label, list), \
-        f'expected the img and label to be list but got {type(img)} and {type(label)}'
-
 
 if __name__ == '__main__':
     test_cifar10_dataset()