megatron-lm0.3.2 based on dtk-22.10

1016e98a · zhuww · 6c3f6c7b · 1016e98a · 1016e98a · 1016e98a
Commit 1016e98a authored Feb 18, 2023 by zhuww
20 changed files
--- a/megatron/fused_kernels/build/scaled_softmax.o
+++ b/megatron/fused_kernels/build/scaled_softmax.o
--- a/megatron/fused_kernels/build/scaled_softmax_cuda.so
+++ b/megatron/fused_kernels/build/scaled_softmax_cuda.so
--- a/megatron/fused_kernels/build/scaled_softmax_hip.cuda.o
+++ b/megatron/fused_kernels/build/scaled_softmax_hip.cuda.o
--- a/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax.o
+++ b/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax.o
--- a/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax_cuda.so
+++ b/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax_cuda.so
--- a/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax_hip.cuda.o
+++ b/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax_hip.cuda.o
--- a/megatron/fused_kernels/compat.h
+++ b/megatron/fused_kernels/compat.h
--- a/megatron/fused_kernels/fused_weight_gradient_dense.cpp
+++ b/megatron/fused_kernels/fused_weight_gradient_dense.cpp
--- a/megatron/fused_kernels/fused_weight_gradient_dense.cu
+++ b/megatron/fused_kernels/fused_weight_gradient_dense.cu
--- a/megatron/fused_kernels/layer_norm_cuda.cpp
+++ b/megatron/fused_kernels/layer_norm_cuda.cpp
--- a/megatron/fused_kernels/layer_norm_cuda_kernel.cu
+++ b/megatron/fused_kernels/layer_norm_cuda_kernel.cu
--- a/megatron/fused_kernels/layer_norm_hip_kernel.hip
+++ b/megatron/fused_kernels/layer_norm_hip_kernel.hip
--- a/megatron/fused_kernels/scaled_masked_softmax.cpp
+++ b/megatron/fused_kernels/scaled_masked_softmax.cpp
--- a/megatron/fused_kernels/scaled_masked_softmax.h
+++ b/megatron/fused_kernels/scaled_masked_softmax.h
--- a/megatron/fused_kernels/scaled_masked_softmax_cuda.cu
+++ b/megatron/fused_kernels/scaled_masked_softmax_cuda.cu
--- a/megatron/fused_kernels/scaled_masked_softmax_hip.hip
+++ b/megatron/fused_kernels/scaled_masked_softmax_hip.hip
--- a/megatron/fused_kernels/scaled_softmax.cpp
+++ b/megatron/fused_kernels/scaled_softmax.cpp
--- a/megatron/fused_kernels/scaled_softmax_cuda.cu
+++ b/megatron/fused_kernels/scaled_softmax_cuda.cu
--- a/megatron/fused_kernels/scaled_softmax_hip.hip
+++ b/megatron/fused_kernels/scaled_softmax_hip.hip
--- a/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp
+++ b/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp