Merge pull request #1013 from kvcache-ai/work-concurrent

In v0.2.4 version, we’ve added highly desired multi-concurrency support to the community through a major refactor of the whole architecture.

Merge pull request #1013 from kvcache-ai/work-concurrent
In v0.2.4 version, we’ve added highly desired multi-concurrency support to the community through a major refactor of the whole architecture.
a41d2163 · wang jiahao · GitHub · f142f4df · 4ed9744e · a41d2163
Unverified Commit a41d2163 authored Apr 02, 2025 by wang jiahao Committed by GitHub Apr 02, 2025
20 changed files
--- a/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-fp8-linear-ggml-experts.yaml
+++ b/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-fp8-linear-ggml-experts.yaml
--- a/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml
+++ b/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml
--- a/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml
+++ b/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml
--- a/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
+++ b/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--- a/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml
+++ b/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml
--- a/ktransformers/optimize/optimize_rules/Moonlight-16B-A3B-serve.yaml
+++ b/ktransformers/optimize/optimize_rules/Moonlight-16B-A3B-serve.yaml
--- a/ktransformers/server/args.py
+++ b/ktransformers/server/args.py
--- a/ktransformers/server/backend/args.py
+++ b/ktransformers/server/backend/args.py
--- a/ktransformers/server/backend/context_manager.py
+++ b/ktransformers/server/backend/context_manager.py
--- a/ktransformers/server/backend/interfaces/balance_serve.py
+++ b/ktransformers/server/backend/interfaces/balance_serve.py
--- a/ktransformers/server/backend/interfaces/ktransformers.py
+++ b/ktransformers/server/backend/interfaces/ktransformers.py
--- a/ktransformers/server/backend/interfaces/transformers.py
+++ b/ktransformers/server/backend/interfaces/transformers.py
--- a/ktransformers/server/balance_serve/inference/__init__.py
+++ b/ktransformers/server/balance_serve/inference/__init__.py
--- a/ktransformers/server/balance_serve/inference/config.py
+++ b/ktransformers/server/balance_serve/inference/config.py
--- a/ktransformers/server/balance_serve/inference/distributed/__init__.py
+++ b/ktransformers/server/balance_serve/inference/distributed/__init__.py
--- a/ktransformers/server/balance_serve/inference/distributed/communication_op.py
+++ b/ktransformers/server/balance_serve/inference/distributed/communication_op.py
--- a/ktransformers/server/balance_serve/inference/distributed/cuda_wrapper.py
+++ b/ktransformers/server/balance_serve/inference/distributed/cuda_wrapper.py
--- a/ktransformers/server/balance_serve/inference/distributed/custom_all_reduce.py
+++ b/ktransformers/server/balance_serve/inference/distributed/custom_all_reduce.py
--- a/ktransformers/server/balance_serve/inference/distributed/custom_all_reduce_utils.py
+++ b/ktransformers/server/balance_serve/inference/distributed/custom_all_reduce_utils.py
--- a/ktransformers/server/balance_serve/inference/distributed/parallel_state.py
+++ b/ktransformers/server/balance_serve/inference/distributed/parallel_state.py