Merge pull request #1013 from kvcache-ai/work-concurrent

In v0.2.4 version, we’ve added highly desired multi-concurrency support to the community through a major refactor of the whole architecture.

Merge pull request #1013 from kvcache-ai/work-concurrent
In v0.2.4 version, we’ve added highly desired multi-concurrency support to the community through a major refactor of the whole architecture.
a41d2163 · wang jiahao · GitHub · f142f4df · 4ed9744e · a41d2163
Unverified Commit a41d2163 authored Apr 02, 2025 by wang jiahao Committed by GitHub Apr 02, 2025
20 changed files
--- a/ktransformers/ktransformers_ext/cuda/gptq_marlin/gptq_marlin.cu
+++ b/ktransformers/ktransformers_ext/cuda/gptq_marlin/gptq_marlin.cu
--- a/ktransformers/ktransformers_ext/cuda/gptq_marlin/gptq_marlin.cuh
+++ b/ktransformers/ktransformers_ext/cuda/gptq_marlin/gptq_marlin.cuh
--- a/ktransformers/ktransformers_ext/cuda/gptq_marlin/gptq_marlin_dtypes.cuh
+++ b/ktransformers/ktransformers_ext/cuda/gptq_marlin/gptq_marlin_dtypes.cuh
--- a/ktransformers/ktransformers_ext/cuda/gptq_marlin/ops.h
+++ b/ktransformers/ktransformers_ext/cuda/gptq_marlin/ops.h
--- a/ktransformers/ktransformers_ext/cuda/setup.py
+++ b/ktransformers/ktransformers_ext/cuda/setup.py
--- a/ktransformers/ktransformers_ext/cuda/test_dequant.py
+++ b/ktransformers/ktransformers_ext/cuda/test_dequant.py
--- a/ktransformers/ktransformers_ext/examples/test_attention.py
+++ b/ktransformers/ktransformers_ext/examples/test_attention.py
--- a/ktransformers/ktransformers_ext/examples/test_linear.py
+++ b/ktransformers/ktransformers_ext/examples/test_linear.py
--- a/ktransformers/ktransformers_ext/examples/test_mlp.py
+++ b/ktransformers/ktransformers_ext/examples/test_mlp.py
--- a/ktransformers/ktransformers_ext/examples/test_moe.py
+++ b/ktransformers/ktransformers_ext/examples/test_moe.py
--- a/ktransformers/ktransformers_ext/ext_bindings.cpp
+++ b/ktransformers/ktransformers_ext/ext_bindings.cpp
--- a/ktransformers/ktransformers_ext/operators/kvcache/kvcache.h
+++ b/ktransformers/ktransformers_ext/operators/kvcache/kvcache.h
--- a/ktransformers/ktransformers_ext/operators/kvcache/kvcache_attn.cpp
+++ b/ktransformers/ktransformers_ext/operators/kvcache/kvcache_attn.cpp
--- a/ktransformers/ktransformers_ext/operators/kvcache/kvcache_load_dump.cpp
+++ b/ktransformers/ktransformers_ext/operators/kvcache/kvcache_load_dump.cpp
--- a/ktransformers/ktransformers_ext/operators/kvcache/kvcache_read_write.cpp
+++ b/ktransformers/ktransformers_ext/operators/kvcache/kvcache_read_write.cpp
--- a/ktransformers/ktransformers_ext/operators/kvcache/kvcache_utils.cpp
+++ b/ktransformers/ktransformers_ext/operators/kvcache/kvcache_utils.cpp
--- a/ktransformers/ktransformers_ext/operators/llamafile/conversion.h
+++ b/ktransformers/ktransformers_ext/operators/llamafile/conversion.h
--- a/ktransformers/ktransformers_ext/operators/llamafile/linear.cpp
+++ b/ktransformers/ktransformers_ext/operators/llamafile/linear.cpp
--- a/ktransformers/ktransformers_ext/operators/llamafile/linear.h
+++ b/ktransformers/ktransformers_ext/operators/llamafile/linear.h
--- a/ktransformers/ktransformers_ext/operators/llamafile/mlp.cpp
+++ b/ktransformers/ktransformers_ext/operators/llamafile/mlp.cpp