Merge pull request #5310 from hpcaitech/feature/npu

Feature/npu

Merge pull request #5310 from hpcaitech/feature/npu
Feature/npu
8823cc48 · Frank Lee · GitHub · bce9499e · 73f4dc57 · 8823cc48
Unverified Commit 8823cc48 authored Jan 29, 2024 by Frank Lee Committed by GitHub Jan 29, 2024
20 changed files
--- a/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu
+++ b/colossalai/kernel/cuda_native/csrc/multi_tensor_l2norm_kernel.cu
--- a/colossalai/kernel/cuda_native/csrc/multi_tensor_lamb.cu
+++ b/colossalai/kernel/cuda_native/csrc/multi_tensor_lamb.cu
--- a/colossalai/kernel/cuda_native/csrc/multi_tensor_scale_kernel.cu
+++ b/colossalai/kernel/cuda_native/csrc/multi_tensor_scale_kernel.cu
--- a/colossalai/kernel/cuda_native/csrc/multi_tensor_sgd_kernel.cu
+++ b/colossalai/kernel/cuda_native/csrc/multi_tensor_sgd_kernel.cu
--- a/colossalai/kernel/cuda_native/csrc/scaled_masked_softmax.cpp
+++ b/colossalai/kernel/cuda_native/csrc/scaled_masked_softmax.cpp
--- a/colossalai/kernel/cuda_native/csrc/scaled_masked_softmax.h
+++ b/colossalai/kernel/cuda_native/csrc/scaled_masked_softmax.h
--- a/colossalai/kernel/cuda_native/csrc/scaled_masked_softmax_cuda.cu
+++ b/colossalai/kernel/cuda_native/csrc/scaled_masked_softmax_cuda.cu
--- a/colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax.cpp
+++ b/colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax.cpp
--- a/colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax.h
+++ b/colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax.h
--- a/colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax_cuda.cu
+++ b/colossalai/kernel/cuda_native/csrc/scaled_upper_triang_masked_softmax_cuda.cu
--- a/colossalai/kernel/cuda_native/csrc/type_shim.h
+++ b/colossalai/kernel/cuda_native/csrc/type_shim.h
--- a/colossalai/kernel/cuda_native/scaled_softmax.py
+++ b/colossalai/kernel/cuda_native/scaled_softmax.py
--- a/extensions/cuda_extension.py
+++ b/extensions/cuda_extension.py
--- a/extensions/flash_attention/__init__.py
+++ b/extensions/flash_attention/__init__.py
--- a/extensions/flash_attention/flash_attention_dao_cuda.py
+++ b/extensions/flash_attention/flash_attention_dao_cuda.py
--- a/extensions/flash_attention/flash_attention_npu.py
+++ b/extensions/flash_attention/flash_attention_npu.py
--- a/extensions/flash_attention/flash_attention_xformers_cuda.py
+++ b/extensions/flash_attention/flash_attention_xformers_cuda.py
--- a/extensions/layernorm/__init__.py
+++ b/extensions/layernorm/__init__.py
--- a/op_builder/layernorm.py
+++ b/op_builder/layernorm.py
--- a/extensions/moe/__init__.py
+++ b/extensions/moe/__init__.py