GitLab

Refactor gpt2

See merge request ADLR/megatron-lm!15

Fixed edge case with multiple end of sequence in one sequence

See merge request ADLR/megatron-lm!12

refactored for code reuse

See merge request ADLR/megatron-lm!11

Major refactoring, combining gpt2 and bert

See merge request ADLR/megatron-lm!8

Move vocab padding to utils

See merge request ADLR/megatron-lm!6

fixed deserializing issue with old checkpoint

See merge request ADLR/megatron-lm!5

moved few common elements between bert and gpt to utils

See merge request ADLR/megatron-lm!4

created megatron package

See merge request ADLR/megatron-lm!3

Use DDP command line argument instead of source flag in pretrain_bert.py.

See merge request ADLR/megatron-lm!2

Note that there is currently an issue with bert using Torch DDP.

Updating public repo with latest changes.

See merge request ADLR/megatron-lm!1

Fixes #7.

Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

* threaded tf_dl+presplit sentences+shuffled dataset with resume

* elaborate in readme