Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ktransformers
Commits
c3d0ac80
Commit
c3d0ac80
authored
May 14, 2025
by
qiyuxinlin
Browse files
update readme
parent
ee524b0f
Changes
4
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
9 additions
and
1 deletion
+9
-1
doc/en/balance-serve.md
doc/en/balance-serve.md
+2
-0
doc/en/install.md
doc/en/install.md
+2
-0
doc/en/llama4.md
doc/en/llama4.md
+3
-1
doc/zh/DeepseekR1_V3_tutorial_zh.md
doc/zh/DeepseekR1_V3_tutorial_zh.md
+2
-0
No files found.
doc/en/balance-serve.md
View file @
c3d0ac80
...
...
@@ -100,8 +100,10 @@ git submodule update --init --recursive
# Install single NUMA dependencies
USE_BALANCE_SERVE
=
1 bash ./install.sh
pip
install
third_party/custom_flashinfer/
# For those who have two cpu and 1T RAM(Dual NUMA):
USE_BALANCE_SERVE
=
1
USE_NUMA
=
1 bash ./install.sh
pip
install
third_party/custom_flashinfer/
```
## Running DeepSeek-R1-Q4KM Models
...
...
doc/en/install.md
View file @
c3d0ac80
...
...
@@ -117,11 +117,13 @@ Download source code and compile:
```shell
USE_BALANCE_SERVE=1 bash ./install.sh
pip install third_party/custom_flashinfer/
```
-
For Multi-concurrency with two cpu and 1T RAM:
```shell
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
pip install third_party/custom_flashinfer/
```
-
For Windows (Windows native temporarily deprecated, please try WSL)
...
...
doc/en/llama4.md
View file @
c3d0ac80
...
...
@@ -67,9 +67,11 @@ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.o
```
bash
# Install single NUMA dependencies
USE_BALANCE_SERVE
=
1 bash ./install.sh
USE_BALANCE_SERVE
=
1 bash ./
pip
install
third_party/custom_flashinfer/
# For those who have two cpu and 1T RAM(Dual NUMA):
USE_BALANCE_SERVE
=
1
USE_NUMA
=
1 bash ./install.sh
pip
install
third_party/custom_flashinfer/
```
### 4. Use our custom config.json
...
...
doc/zh/DeepseekR1_V3_tutorial_zh.md
View file @
c3d0ac80
...
...
@@ -127,8 +127,10 @@ cd ktransformers
git submodule update
--init
--recursive
# 如果使用双 numa 版本
USE_BALANCE_SERVE
=
1
USE_NUMA
=
1 bash ./install.sh
pip
install
third_party/custom_flashinfer/
# 如果使用单 numa 版本
USE_BALANCE_SERVE
=
1 bash ./install.sh
pip
install
third_party/custom_flashinfer/
# 启动命令
python ktransformers/server/main.py
--model_path
<your model path>
--gguf_path
<your gguf path>
--cpu_infer
62
--optimize_config_path
<inject rule path>
--port
10002
--chunk_size
256
--max_new_tokens
1024
--max_batch_size
4
--port
10002
--cache_lens
32768
--backend_type
balance_serve
```
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment