GitLab

Move rearranging query_key_value and key_value values in old checkpoints to when the checkpoint is loaded instead of runtime..

Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)

Small bugfix in bert_model.py: make sure word_embeddings is initialized before instantiating lm_head

Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods

Also, use mpu.get_*_model_parallel_size() instead of args.*_model_parallel_size

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments