GitLab

refactored merge_mp_partitions.py

See merge request ADLR/megatron-lm!47

Do not cast return type of indexed dataset

See merge request ADLR/megatron-lm!48

Do not cast data returned from indexed_dataset to int64, rely on caller to cast to appropriate type.

preprocess_data:
- Adds ability to not split sentences. This is used for gpt2 datasets.

- Adds ability to create multiple datasets from different json keys,
this is current untested.

indexed_dataset:
- Add new "get" function to get a portion of an entry.

Lint megatron/data/dataset_utils.py

See merge request ADLR/megatron-lm!42

Refactoring text generation

See merge request ADLR/megatron-lm!39

Refactoring evaluate gpt2

See merge request ADLR/megatron-lm!38

Major refactor: args, global variables, tokenizer

See merge request ADLR/megatron-lm!36