".github/vscode:/vscode.git/clone" did not exist on "6aebf44f47bc73ac34344fb7b5d941790c11d39d"
Unverified Commit 3efc8e2d authored by mRSun15's avatar mRSun15 Committed by GitHub
Browse files

add attention backend supporting matrix in the doc (#5211)


Co-authored-by: default avatarStefan He <hebiaobuaa@gmail.com>
parent 27a009bb
# Attention Backend
## Supporting matrix for different attention backend
| **Backend** | **Page Size > 1** | **Spec Decoding** | **MLA** | **Sliding Window** | **MultiModal** |
|--------------------------|-------------------|-------------------|--------|--------------------|------------|
| **FlashInfer** | ✅ | ✅ | ✅ | ✅ | ✅ |
| **FA3** | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Triton** | ❌ | ✅ | ✅ | ❌ | ❌ |
| **Torch Native** | ❌ | ❌ | ❌ | ❌ | ❌ |
## User guide
#### Launch command for different attention backend.
- FlashInfer (Default for Non-Hopper Machines, e.g., A100, A40)
```bash
python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --attention-backend flashinfer
python3 -m sglang.launch_server --tp 8 --model deepseek-ai/DeepSeek-V3 --attention-backend flashinfer --trust-remote-code
```
- FlashAttention 3 (Default for Hopper Machines, e.g., H100, H200, H20)
```bash
python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --attention-backend fa3
python3 -m sglang.launch_server --tp 8 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --attention-backend fa3
```
- Triton
```bash
python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --attention-backend triton
python3 -m sglang.launch_server --tp 8 --model deepseek-ai/DeepSeek-V3 --attention-backend triton --trust-remote-code
```
- Torch Native
```bash
python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --attention-backend torch_native
```
......@@ -32,6 +32,7 @@ The core features include:
backend/sampling_params.md
backend/hyperparameter_tuning.md
backend/structured_outputs_for_reasoning_models.ipynb
backend/attention_backend.md
.. toctree::
:maxdepth: 1
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment