Unverified Commit 2e6346fc authored by Chayenne's avatar Chayenne Committed by GitHub
Browse files

Docs:Update the style of llma 3.1 405B docs (#2789)

parent 977f785d
...@@ -31,4 +31,3 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct ...@@ -31,4 +31,3 @@ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct
# Node 1 # Node 1
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1
``` ```
...@@ -56,9 +56,9 @@ The core features include: ...@@ -56,9 +56,9 @@ The core features include:
references/hyperparameter_tuning.md references/hyperparameter_tuning.md
references/benchmark_and_profiling.md references/benchmark_and_profiling.md
references/custom_chat_template.md references/custom_chat_template.md
references/llama_405B.md
references/modelscope.md
references/contribution_guide.md references/contribution_guide.md
references/troubleshooting.md references/troubleshooting.md
references/faq.md references/faq.md
references/learn_more.md references/learn_more.md
references/llama_405B.md
references/modelscope.md
# Example: Run Llama 3.1 405B # Run Llama 3.1 405B
## Run 405B (fp8) on a Single Node
```bash ```bash
# Run 405B (fp8) on a single node
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8
``` ```
## Run 405B (fp16) on Two Nodes
```bash ```bash
# Run 405B (fp16) on two nodes # on the first node, replace 172.16.4.52:20000 with your own node ip address and port
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 0 python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 0
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port # on the second node, replace 172.18.45.52:20000 with your own node ip address and port
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 1
```
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.18.45.52:20000 --nnodes 2 --node-rank 1
```
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment