Unverified Commit 8adbc78b authored by Zachary Streeter's avatar Zachary Streeter Committed by GitHub
Browse files

added llama and cleaned up (#3503)

parent 45e3a7bc
# AMD Configuration and Setup for SGLang # SGLang on AMD
## Introduction ## Introduction
...@@ -99,9 +99,29 @@ drun sglang_image \ ...@@ -99,9 +99,29 @@ drun sglang_image \
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities. With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.
## Running DeepSeek-V3 ## Examples
The only difference in running DeepSeek-V3 is when starting the server. ### Running DeepSeek-V3
The only difference in running DeepSeek-V3 is when starting the server. Here's an example command:
```bash
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \ # <- here
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
### Running Llama3.1
Running Llama3.1 is nearly identical. The only difference is in the model specified when starting the server, shown by the following example command:
```bash ```bash
drun -p 30000:30000 \ drun -p 30000:30000 \
...@@ -110,9 +130,13 @@ drun -p 30000:30000 \ ...@@ -110,9 +130,13 @@ drun -p 30000:30000 \
--env "HF_TOKEN=<secret>" \ --env "HF_TOKEN=<secret>" \
sglang_image \ sglang_image \
python3 -m sglang.launch_server \ python3 -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3 # <- here \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here
--tp 8 \ --tp 8 \
--trust-remote-code \ --trust-remote-code \
--host 0.0.0.0 \ --host 0.0.0.0 \
--port 30000 --port 30000
``` ```
### Warmup Step
When the server displays "The server is fired up and ready to roll!", it means the startup is successful.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment