Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
change
sglang
Commits
76d14f8c
Unverified
Commit
76d14f8c
authored
Dec 30, 2024
by
Lzhang-hub
Committed by
GitHub
Dec 30, 2024
Browse files
add 2*h20 node serving example for deepseek v3 (#2650)
Co-authored-by:
Yineng Zhang
<
me@zhyncs.com
>
parent
b08c308e
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
10 additions
and
0 deletions
+10
-0
benchmark/deepseek_v3/README.md
benchmark/deepseek_v3/README.md
+10
-0
No files found.
benchmark/deepseek_v3/README.md
View file @
76d14f8c
...
@@ -51,6 +51,16 @@ response = client.chat.completions.create(
...
@@ -51,6 +51,16 @@ response = client.chat.completions.create(
)
)
print(response)
print(response)
```
```
### Example serving with 2 H20*8
For example, there are two H20 nodes, each with 8 GPUs. The first node's IP is
`10.0.0.1`
, and the second node's IP is
`10.0.0.2`
.
```
bash
# node 1
GLOO_SOCKET_IFNAME
=
eth0 python
-m
sglang.launch_server
--model-path
deepseek-ai/DeepSeek-V3
--tp
16
--nccl-init
10.0.0.1:5000
--nnodes
2
--node-rank
0
--trust-remote-code
# node 2
GLOO_SOCKET_IFNAME
=
eth0 python
-m
sglang.launch_server
--model-path
deepseek-ai/DeepSeek-V3
--tp
16
--nccl-init
10.0.0.1:5000
--nnodes
2
--node-rank
1
--trust-remote-code
```
## DeepSeek V3 Optimization Plan
## DeepSeek V3 Optimization Plan
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment