Optimize the performance of GPTQ

d4bccff3 · xuxzh1 · ee3d6944 · d4bccff3
Commit d4bccff3 authored Jan 24, 2025 by xuxzh1 🎱
Show whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

server/exllamav2_kernels/exllamav2_kernels/cuda/q_gemm.cu server/exllamav2_kernels/exllamav2_kernels/cuda/q_gemm.cu +1 -1

No files found.
--- a/server/exllamav2_kernels/exllamav2_kernels/cuda/q_gemm.cu
+++ b/server/exllamav2_kernels/exllamav2_kernels/cuda/q_gemm.cu
@@ -10,7 +10,7 @@
 #include "quant/qdq_6.cuh"
 #include "quant/qdq_8.cuh"

-#define GPTQ_BLOCK_KN_SIZE 128
+#define GPTQ_BLOCK_KN_SIZE 256
 #define GPTQ_BLOCK_M_SIZE_MAX 8
 #define GPTQ_MAX_GROUPS_IN_BLOCK (GPTQ_BLOCK_KN_SIZE / 32)