Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
DeepSeek-V4
Commits
8dc93b44
"googlemock/vscode:/vscode.git/clone" did not exist on "7c2bd3af981e9f257a0b61ce664953008c105dad"
Commit
8dc93b44
authored
Apr 25, 2026
by
luopl
Browse files
Delete README.md
parent
5e61101f
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
0 additions
and
26 deletions
+0
-26
inference/README.md
inference/README.md
+0
-26
No files found.
inference/README.md
deleted
100644 → 0
View file @
5e61101f
# Inference code for DeepSeek models
First convert huggingface model weight files to the format of this project.
```
bash
export
EXPERTS
=
256
export
MP
=
4
export
CONFIG
=
config.json
python convert.py
--hf-ckpt-path
${
HF_CKPT_PATH
}
--save-path
${
SAVE_PATH
}
--n-experts
${
EXPERTS
}
--model-parallel
${
MP
}
```
Then chat with DeepSeek model at will!
```
bash
torchrun
--nproc-per-node
${
MP
}
generate.py
--ckpt-path
${
SAVE_PATH
}
--config
${
CONFIG
}
--interactive
```
Or batch inference from file.
```
bash
torchrun
--nproc-per-node
${
MP
}
generate.py
--ckpt-path
${
SAVE_PATH
}
--config
${
CONFIG
}
--input-file
${
FILE
}
```
Or multi nodes inference.
```
bash
torchrun
--nnodes
${
NODES
}
--nproc-per-node
$((
MP
/
NODES
))
--node-rank
$RANK
--master-addr
$ADDR
generate.py
--ckpt-path
${
SAVE_PATH
}
--config
${
CONFIG
}
--input-file
${
FILE
}
```
If you want to use fp8, just remove
`"expert_dtype": "fp4"`
in
`config.json`
and specify
`--expert-dtype fp8`
in
`convert.py`
.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment