Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
gemm_xdl_fp16.cpp 13.3 KB
Newer Older
Chao Liu's avatar
add license in file (#303)  
Chao Liu committed
1 
// SPDX-License-Identifier: MIT
Rostyslav Geyyer's avatar
Add a gpu gemm reference kernel (#1528)  
Rostyslav Geyyer committed
2 
// Copyright (c) 2018-2024, Advanced Micro Devices, Inc. All rights reserved.
Chao Liu's avatar
add license in file (#303)  
Chao Liu committed
3
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
4 
#include "common.hpp"
Chao Liu's avatar
Absolute include path (#281)  
Chao Liu committed
5
Adam Osewski's avatar
Refactor device op implementations into `impl` subdirectory. (#420)  
Adam Osewski committed
6
7 
#include "ck/tensor_operation/gpu/device/impl/device_gemm_xdl.hpp"
#include "ck/tensor_operation/gpu/device/impl/device_gemm_xdl_cshuffle.hpp"
Chao Liu's avatar
Absolute include path (#281)  
Chao Liu committed
8
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
9
10
11 
using ADataType        = ck::half_t;
using BDataType        = ck::half_t;
using AccDataType      = float;
zjing14's avatar
Improve 4k gemm perf (#1047)  
zjing14 committed
12 
using CShuffleDataType = ck::half_t;
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
13 
using CDataType        = ck::half_t;
Chao Liu's avatar
fix ReLU formula (#61)  
Chao Liu committed
14
Raman R jana's avatar
Wavelet (inter-wave consumer-producer) GEMM (#310)  
Raman R jana committed
15
16 
using F16 = ck::half_t;
Chao Liu's avatar
GEMM with Multiple Source, GEMM+Bias+Add+FastGeLU example and ckProfiler (#241)  
Chao Liu committed
17 
using ALayout = Row;
root's avatar
Add new mfma instructions and examples  
root committed
18
19 
using BLayout = Row; // DeviceGemmXdl or DeviceGemm_Xdl_CShuffle with ck::LoopScheduler::Interwave
// using BLayout = Col;    // DeviceGemm_Xdl_CShuffle with ck::LoopScheduler::Default
Chao Liu's avatar
GEMM with Multiple Source, GEMM+Bias+Add+FastGeLU example and ckProfiler (#241)  
Chao Liu committed
20 
using CLayout = Row;
Chao Liu's avatar
fix ReLU formula (#61)  
Chao Liu committed
21
Chao Liu's avatar
GEMM with Multiple Source, GEMM+Bias+Add+FastGeLU example and ckProfiler (#241)  
Chao Liu committed
22
23
24 
using AElementOp = PassThrough;
using BElementOp = PassThrough;
using CElementOp = PassThrough;
Chao Liu's avatar
fix ReLU formula (#61)  
Chao Liu committed
25
Chao Liu's avatar
Compile for gfx908 and gfx90a (#130)  
Chao Liu committed
26 
static constexpr auto GemmDefault = ck::tensor_operation::device::GemmSpecialization::Default;
Chao Liu's avatar
Add gridwise GEMM pipeline (#89)  
Chao Liu committed
27
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
28 
// clang-format off
Po Yen Chen's avatar
GEMM pipeline v2 (#317)  
Po Yen Chen committed
29 
using DeviceGemmInstance0 = ck::tensor_operation::device::DeviceGemmXdl
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
30
31
32
33 
// ######|     AData|     BData|     CData|     AccData| ALayout| BLayout| CLayout|           A|           B|           C|          GEMM| Block|  MPer|  NPer| K0Per| K1| MPer| NPer| MXdl| NXdl|  ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockLds|  BBlockTransfer| BBlockTransfer| BBlockTransfer| BlockTransfer| BBlockTransfer| BBlockTransfer| BBlockLds| CThreadTransfer| CThreadTransfer|
// ######|      Type|      Type|      Type|        Type|        |        |        | Elementwise| Elementwise| Elementwise|Spacialization|  Size| Block| Block| Block|   |  XDL|  XDL|  Per|  Per|   ThreadCluster|  ThreadCluster| SrcAccessOrder|   SrcVectorDim|      SrcScalar|      DstScalar| AddExtraM|   ThreadCluster|  ThreadCluster| SrcAccessOrder|  SrcVectorDim|      SrcScalar|      DstScalar| AddExtraN| SrcDstVectorDim|       DstScalar|
// ######|          |          |          |            |        |        |        |   Operation|   Operation|   Operation|              |      |      |      |      |   |     |     | Wave| Wave| Lengths_K0_M_K1|   ArrangeOrder|               |               |      PerVector|   PerVector_K1|          | Lengths_K0_N_K1|   ArrangeOrder|               |              |      PerVector|   PerVector_K1|          |                |       PerVector|
// ######|          |          |          |            |        |        |        |            |            |            |              |      |      |      |      |   |     |     |     |     |                |               |               |               |               |               |          |                |               |               |              |               |               |          |                |                |
root's avatar
Add new mfma instructions and examples  
root committed
34
35
36
37
38
39
40
41
42
43
44 
#if !defined(CK_USE_AMD_MFMA_GFX950)
    //     < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   256,   128,     4,  8,   32,   32,    4,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 32, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              4,              8,      true,               7,               1>;
    //     < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   128,   128,     4,  8,   32,   32,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 32, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              4,              8,      true,               7,               1>;
//         < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   128,   128,     4,  8,   32,   32,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 64, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              2,              8,      true,               7,               1>;
    //     < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   64,   64,     4,  16,   16,   16,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 32, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              2,              8,      true,               7,               1>;
    //     < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   64,   64,     4,  16,   16,   16,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 64, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              1,              8,      true,               7,               1>;
#else
         < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   256,   128,     4,  16,   32,   32,    4,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 32, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              4,              8,      true,               7,               1>;
    //     < ADataType, BDataType, CDataType, AccDataType, ALayout, BLayout, CLayout,  AElementOp,  BElementOp,  CElementOp,   GemmDefault,   256,   64,   64,     4,  32,   16,   16,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,      true,     S<4, 32, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              2,              8,      true,               7,               1>;
#endif
    // clang-format on
Po Yen Chen's avatar
GEMM pipeline v2 (#317)  
Po Yen Chen committed
45
root's avatar
Add new mfma instructions and examples  
root committed
46 
    // clang-format off
Po Yen Chen's avatar
GEMM pipeline v2 (#317)  
Po Yen Chen committed
47 
using DeviceGemmInstance1 = ck::tensor_operation::device::DeviceGemm_Xdl_CShuffle
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
48
49
50
51 
// ######| ALayout| BLayout| CLayout|     AData|     BData|     CData|     AccData|         CShuffle|           A|           B|           C|           GEMM| NumGemmK| Block|  MPer|  NPer|  KPer| AK1| BK1| MPer| NPer| MXdl| NXdl|  ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockTransfer| ABlockLds|  BBlockTransfer| BBlockTransfer| BBlockTransfer| BlockTransfer| BBlockTransfer| BBlockTransfer| BBlockLds|    CShuffle|    CShuffle| CBlockTransferClusterLengths|  CBlockTransfer|
// ######|        |        |        |      Type|      Type|      Type|        Type|         DataType| Elementwise| Elementwise| Elementwise| Spacialization| Prefetch|  Size| Block| Block| Block|    |    |  XDL|  XDL|  Per|  Per|   ThreadCluster|  ThreadCluster| SrcAccessOrder|   SrcVectorDim|      SrcScalar|      DstScalar| AddExtraM|   ThreadCluster|  ThreadCluster| SrcAccessOrder|  SrcVectorDim|      SrcScalar|      DstScalar| AddExtraN| MXdlPerWave| NXdlPerWave|         _MBlock_MWaveMPerXdl| ScalarPerVector|
// ######|        |        |        |          |          |          |            |                 |   Operation|   Operation|   Operation|               |    Stage|      |      |      |      |    |    |     |     | Wave| Wave| Lengths_K0_M_K1|   ArrangeOrder|               |               |      PerVector|   PerVector_K1|          | Lengths_K0_N_K1|   ArrangeOrder|               |              |      PerVector|   PerVector_K1|          |  PerShuffle|  PerShuffle|         _NBlock_NWaveNPerXdl|   _NWaveNPerXdl|
// ######|        |        |        |          |          |          |            |                 |            |            |            |               |         |      |      |      |      |    |    |     |     |     |     |                |               |               |               |               |               |          |                |               |               |              |               |               |          |            |            |                             |                |
root's avatar
Add new mfma instructions and examples  
root committed
52
53
54
55
56
57
58
59
60
61
62 
#if !defined(CK_USE_AMD_MFMA_GFX950)
    //     < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   256,   128,    32,   8,   2,   32,   32,    4,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<8, 32, 1>,     S<0, 2, 1>,     S<0, 2, 1>,             1,              4,              2,         0,           1,           2,              S<1, 16, 1, 16>,               8, ck::LoopScheduler::Interwave, ck::PipelineVersion::v1>;
    //     < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   256,   128,    32,   8,   8,   32,   32,    4,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               8>;
    //     < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   128,   128,    32,   8,   8,   32,   32,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               8>;
    //     < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   128,   128,    64,   8,   8,   32,   32,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               8>;
    //     < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   64,   64,    64,   16,   16,   16,   16,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               4>;
         < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   64,   64,    128,   16,   16,   16,   16,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               4>;
#else
    //     < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   128,   128,    64,   16,   16,   32,   32,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               8>;
         < ALayout, BLayout, CLayout, ADataType, BDataType, CDataType, AccDataType, CShuffleDataType,  AElementOp,  BElementOp,  CElementOp,    GemmDefault,        1,   256,   64,   64,    128,   32,   32,   16,   16,    2,    2,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,              2,              8,              8,         1,     S<4, 64, 1>,     S<1, 0, 2>,     S<1, 0, 2>,             2,              8,              8,         1,           1,           1,               S<1, 32, 1, 8>,               4>;
#endif
Chao Liu's avatar
fix ReLU formula (#61)  
Chao Liu committed
63
64 
// clang-format on
root's avatar
Add new mfma instructions and examples  
root committed
65
66 
// using DeviceGemmInstance = DeviceGemmInstance1;
using DeviceGemmInstance = DeviceGemmInstance0;
Po Yen Chen's avatar
GEMM pipeline v2 (#317)  
Po Yen Chen committed
67
Chao Liu's avatar
GEMM+Bias+ReLU+Add (#76)  
Chao Liu committed
68 
using ReferenceGemmInstance = ck::tensor_operation::host::
ltqin's avatar
Add FP64 XDL GEMM built-in function (#199)  
ltqin committed
69 
    ReferenceGemm<ADataType, BDataType, CDataType, AccDataType, AElementOp, BElementOp, CElementOp>;
Chao Liu's avatar
ckProfiler and device-level XDL GEMM operator (#48)  
Chao Liu committed
70
Rostyslav Geyyer's avatar
Add a gpu gemm reference kernel (#1528)  
Rostyslav Geyyer committed
71
72
73
74
75
76
77
78
79
80
81 
using ReferenceGemmInstanceGPU = ck::tensor_operation::device::ReferenceGemm<ALayout,
                                                                             BLayout,
                                                                             CLayout,
                                                                             ADataType,
                                                                             BDataType,
                                                                             CDataType,
                                                                             AccDataType,
                                                                             AElementOp,
                                                                             BElementOp,
                                                                             CElementOp>;
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
82 
#include "run_gemm_example.inc"
Jianfeng Yan's avatar
Conv3d new (#94)  
Jianfeng Yan committed
83
Po Yen Chen's avatar
Add examples of Gemm (data type: int4) (#367)  
Po Yen Chen committed
84 
int main(int argc, char* argv[]) { return !run_gemm_example(argc, argv); }