GPT2 base on megatron-deepspeed

8ec5d678 · hepj987 · 8ec5d678 · 8ec5d678 · 8ec5d678 · 8ec5d678
Commit 8ec5d678 authored Apr 03, 2023 by hepj987
20 changed files
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_masked_softmax_cuda.cu
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_masked_softmax_cuda.cu
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_masked_softmax_hip.cpp
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_masked_softmax_hip.cpp
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_masked_softmax_hip.hip
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_masked_softmax_hip.hip
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.cpp
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.cpp
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/tests/__init__.py
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/tests/__init__.py
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/tests/test_fused_kernels.py
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/tests/test_fused_kernels.py
--- a/megatron-deepspeed_dtk22.10/megatron/fused_kernels/type_shim.h
+++ b/megatron-deepspeed_dtk22.10/megatron/fused_kernels/type_shim.h
--- a/megatron-deepspeed_dtk22.10/megatron/global_vars.py
+++ b/megatron-deepspeed_dtk22.10/megatron/global_vars.py
--- a/megatron-deepspeed_dtk22.10/megatron/indexer.py
+++ b/megatron-deepspeed_dtk22.10/megatron/indexer.py
--- a/megatron-deepspeed_dtk22.10/megatron/initialize.py
+++ b/megatron-deepspeed_dtk22.10/megatron/initialize.py
--- a/megatron-deepspeed_dtk22.10/megatron/learning_rates.py
+++ b/megatron-deepspeed_dtk22.10/megatron/learning_rates.py
--- a/megatron-deepspeed_dtk22.10/megatron/logging.py
+++ b/megatron-deepspeed_dtk22.10/megatron/logging.py
--- a/megatron-deepspeed_dtk22.10/megatron/memory.py
+++ b/megatron-deepspeed_dtk22.10/megatron/memory.py
--- a/megatron-deepspeed_dtk22.10/megatron/microbatches.py
+++ b/megatron-deepspeed_dtk22.10/megatron/microbatches.py
--- a/megatron-deepspeed_dtk22.10/megatron/model/__init__.py
+++ b/megatron-deepspeed_dtk22.10/megatron/model/__init__.py
--- a/megatron-deepspeed_dtk22.10/megatron/model/bert_model.py
+++ b/megatron-deepspeed_dtk22.10/megatron/model/bert_model.py