GitLab

added across rank sync for checkpoint iteration laod, fixed type for timing, and validation iterations

Some tests expect a clean model parallel slate and complain if a previous test left something behind; this change clears more variables that the tests complain about.

- Accumulate encoder hidden state gradient to handle skip connection
- Correctly compute the number of layers in encoder / decoder for T5 model
- Ensure e weights are initialized the same way in embeddings
- Synchronize embedding gradients across encoder and decoder for T5 model
- Support for checkpoint loading and saving

hidden_size % attention_heads == 0 is handled above when dealing with kv_channels.

Adding check for decoder sequence length.