Layer integration (#83)

* integrated parallel layers for ease of building models

* integrated 2.5d layers

* cleaned codes and unit tests

* added log metric by step hook; updated imagenet benchmark; fixed some bugs

* reworked initialization; cleaned codes
Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>

colossalai/nn/metric/_utils.py

+import torch
+
+def calc_acc(logits, targets):
+    preds = torch.argmax(logits, dim=-1)
+    correct = torch.sum(targets == preds)
+    return correct

examples/simclr_cifar10_data_parallel/config.py

@@ -19,5 +19,5 @@ dataset = dict(
 )
 
 gradient_accumulation=2
-gradient_clipping=1.0
+clip_grad_norm=1.0