Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
change
sglang
Commits
2e6346fc
Unverified
Commit
2e6346fc
authored
Jan 08, 2025
by
Chayenne
Committed by
GitHub
Jan 08, 2025
Browse files
Docs:Update the style of llma 3.1 405B docs (#2789)
parent
977f785d
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
12 additions
and
10 deletions
+12
-10
docs/backend/server_arguments.md
docs/backend/server_arguments.md
+0
-1
docs/index.rst
docs/index.rst
+2
-2
docs/references/llama_405B.md
docs/references/llama_405B.md
+10
-7
No files found.
docs/backend/server_arguments.md
View file @
2e6346fc
...
@@ -31,4 +31,3 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct
...
@@ -31,4 +31,3 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct
# Node 1
# Node 1
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1
```
```
docs/index.rst
View file @
2e6346fc
...
@@ -56,9 +56,9 @@ The core features include:
...
@@ -56,9 +56,9 @@ The core features include:
references/hyperparameter_tuning.md
references/hyperparameter_tuning.md
references/benchmark_and_profiling.md
references/benchmark_and_profiling.md
references/custom_chat_template.md
references/custom_chat_template.md
references/llama_405B.md
references/modelscope.md
references/contribution_guide.md
references/contribution_guide.md
references/troubleshooting.md
references/troubleshooting.md
references/faq.md
references/faq.md
references/learn_more.md
references/learn_more.md
references/llama_405B.md
references/modelscope.md
docs/references/llama_405B.md
View file @
2e6346fc
# Example: Run Llama 3.1 405B
# Run Llama 3.1 405B
## Run 405B (fp8) on a Single Node
```
bash
```
bash
# Run 405B (fp8) on a single node
python
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3.1-405B-Instruct-FP8
--tp
8
python
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3.1-405B-Instruct-FP8
--tp
8
```
```
## Run 405B (fp16) on Two Nodes
```
bash
```
bash
#
Run 405B (fp16) on two nodes
#
on the first node, replace 172.16.4.52:20000 with your own node ip address and port
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
python3
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3.1-405B-Instruct
--tp
16
--nccl-init-addr
172.16.4.52:20000
--nnodes
2
--node-rank
0
python3
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3.1-405B-Instruct
--tp
16
--nccl-init-addr
172.16.4.52:20000
--nnodes
2
--node-rank
0
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
# on the second node, replace 172.18.45.52:20000 with your own node ip address and port
python3
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3.1-405B-Instruct
--tp
16
--nccl-init-addr
172.16.4.52:20000
--nnodes
2
--node-rank
1
```
python3
-m
sglang.launch_server
--model-path
meta-llama/Meta-Llama-3.1-405B-Instruct
--tp
16
--nccl-init-addr
172.18.45.52:20000
--nnodes
2
--node-rank
1
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment