Unverified Commit 8adbc78b authored by Zachary Streeter's avatar Zachary Streeter Committed by GitHub
Browse files

added llama and cleaned up (#3503)

parent 45e3a7bc
# AMD Configuration and Setup for SGLang
# SGLang on AMD
## Introduction
......@@ -99,9 +99,29 @@ drun sglang_image \
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.
## Running DeepSeek-V3
## Examples
The only difference in running DeepSeek-V3 is when starting the server.
### Running DeepSeek-V3
The only difference in running DeepSeek-V3 is when starting the server. Here's an example command:
```bash
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \ # <- here
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
### Running Llama3.1
Running Llama3.1 is nearly identical. The only difference is in the model specified when starting the server, shown by the following example command:
```bash
drun -p 30000:30000 \
......@@ -110,9 +130,13 @@ drun -p 30000:30000 \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3 # <- here \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
### Warmup Step
When the server displays "The server is fired up and ready to roll!", it means the startup is successful.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment