Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
deepseek.md 9.35 KB
Newer Older
Chayenne's avatar
Docs: Move dpsk docs forward a step (#3894)  
Chayenne committed
1 
# DeepSeek Usage
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
2
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
3 
SGLang provides several optimizations specifically designed for the DeepSeek model to boost its inference speed. This document outlines current optimizations for DeepSeek. Additionally, the SGLang team is actively developing enhancements for [DeepSeek V3](https://github.com/sgl-project/sglang/issues/2591).
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
4
Chayenne's avatar
Docs: Reorngaize dpsk links (#3900)  
Chayenne committed
5
6 
Special thanks to Meituan's Search & Recommend Platform Team and Baseten's Model Performance Team for implementing the model, and DataCrunch for providing GPU resources.
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
7 
## Launch DeepSeek V3 with SGLang
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
8
Chayenne's avatar
Docs: Reorngaize dpsk links (#3900)  
Chayenne committed
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86 
SGLang is recognized as one of the top engines for [DeepSeek model inference](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3). To run DeepSeek V3/R1 models, the requirements are as follows:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Weight Configurations</title>
    <style>
        table {
            width: 100%;
            border-collapse: collapse;
            margin: 20px 0;
        }
        th, td {
            border: 1px solid #ddd;
            padding: 12px;
            text-align: left;
        }
        th {
            background-color: #f2f2f2;
        }
        tr:nth-child(even) {
            background-color: #f9f9f9;
        }
    </style>
</head>
<body>
    <table>
        <thead>
            <tr>
                <th>Weight Type</th>
                <th>Configuration</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td rowspan="3"><b>Full precision FP8 (recommended)</b></td>
                <td>8 x H200</td>
            </tr>
            <tr>
                <td>8 x MI300X</td>
            </tr>
            <tr>
                <td>2 x 8 x H100/800/20</td>
            </tr>
            <tr>
                <td rowspan="4">Full precision BF16</td>
                <td>2 x 8 x H200</td>
            </tr>
            <tr>
                <td>2 x 8 x MI300X</td>
            </tr>
            <tr>
                <td>4 x 8 x H100/800/20</td>
            </tr>
            <tr>
                <td>4 x 8 x A100/A800</td>
            </tr>
            <tr>
                <td rowspan="2">Quantized weights (AWQ)</td>
                <td>8 x H100/800/20</td>
            </tr>
            <tr>
                <td>8 x A100/A800</td>
            </tr>
        </tbody>
    </table>
</body>
</html>

Detailed commands for reference:

- [8 x H200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
- [8 x MI300X](https://docs.sglang.ai/references/amd.html#running-deepseek-v3)
- [2 x 8 x H200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h208-nodes)
- [4 x 8 x A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
- [8 x A100 (AWQ)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
87
88
89
90
91 

### Download Weights

If you encounter errors when starting the server, ensure the weights have finished downloading. It's recommended to download them beforehand or restart multiple times until all weights are downloaded. Please refer to [DeepSeek V3]([https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#installation--launch](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base#61-inference-with-deepseek-infer-demo-example-only)) offical guide to download the weights.
Shenggui Li's avatar
[docs] added torch.compile cache to dpsk manual (#3737)  
Shenggui Li committed
92
93
94
95 
### Caching `torch.compile`

The DeepSeek series have huge model weights, it takes some time to compile the model with `torch.compile` for the first time if you have added the flag `--enable-torch-compile`. By default, `torch.compile` will automatically cache the FX graph and Triton in `/tmp/torchinductor_root`, which might be cleared according to the [system policy](https://serverfault.com/questions/377348/when-does-tmp-get-cleared). You can export the environment variable `TORCHINDUCTOR_CACHE_DIR` to save compilation cache in your desired directory to avoid unwanted deletion. You can also share the cache with other machines to reduce the compilation time. You may refer to the [PyTorch official documentation](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [SGLang Documentation](./torch_compile_cache.md) for more details.
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
96
97
98
99
100
101
102
103
104
105 
### Launch with One node of 8 H200

Please refer to [the example](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended). **Note that Deepseek V3 is already in FP8. So we should not run it with any quantization arguments like `--quantization fp8 --kv-cache-dtype fp8_e5m2`.** Also, `--enable-dp-attention` can be useful to improve for Deepseek V3/R1's throughput. Please refer to [Data Parallelism Attention](https://docs.sglang.ai/references/deepseek.html#multi-head-latent-attention-mla-throughput-optimizations) for detail.

### Running examples on Multi-node

- [Serving with two H20*8 nodes](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h208-nodes).

- [Serving with two H200*8 nodes and docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker).
Jhin's avatar
Update DeepSeek V3 Doc (#3541)  
Jhin committed
106
107 
- [Serving with four A100*8 nodes](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes).
Shi Shuai's avatar
Fix: Runtime error for function calling (#3300)  
Shi Shuai committed
108 
## Optimizations
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
109
110 

### Multi-head Latent Attention (MLA) Throughput Optimizations
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
111
112
113
114 

**Description**: [MLA](https://arxiv.org/pdf/2405.04434) is an innovative attention mechanism introduced by the DeepSeek team, aimed at improving inference efficiency. SGLang has implemented specific optimizations for this, including:

- **Weight Absorption**: By applying the associative law of matrix multiplication to reorder computation steps, this method balances computation and memory access and improves efficiency in the decoding phase.
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
115
Baizhou Zhang's avatar
[doc] Update document for flashinfer mla (#3907)  
Baizhou Zhang committed
116 
- **Flashinfer MLA Wrapper**: By providing `--enable-flashinfer-mla` argument, the server will use MLA kernels customized by Flashinfer. This optimization can be significant under long context scenarios. More details can be referred to [this document](https://docs.flashinfer.ai/api/mla.html).
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
117
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
118 
- **FP8 Quantization**: W8A8 FP8 and KV Cache FP8 quantization enables efficient FP8 inference. Additionally, we have implemented Batched Matrix Multiplication (BMM) operator to facilitate FP8 inference in MLA with weight absorption.
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
119
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
120
121
122 
- **CUDA Graph & Torch.compile**: Both MLA and Mixture of Experts (MoE) are compatible with CUDA Graph and Torch.compile, which reduces latency and accelerates decoding speed for small batch sizes.

Overall, with these optimizations, we have achieved up to a 7x acceleration in output throughput compared to the previous version.
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
123
Shi Shuai's avatar
Improve: Token-In Token-Out Usage for RLHF (#2843)  
Shi Shuai committed
124
125
126 
<p align="center">
  <img src="https://lmsys.org/images/blog/sglang_v0_3/deepseek_mla.svg" alt="Multi-head Latent Attention for DeepSeek Series Models">
</p>
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
127
Didier Durand's avatar
fix deepseek_v3 typo (#3497)  
Didier Durand committed
128 
**Usage**: MLA optimization is enabled by default, to disable, use `--disable-mla`.
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
129
130
131 

**Reference**: Check [Blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for more details.
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
132 
### Data Parallelism Attention
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
133
134 

**Description**: This optimization involves data parallelism (DP) for the MLA attention mechanism of DeepSeek Series Models, which allows for a significant reduction in the KV cache size, enabling larger batch sizes. Each DP worker independently handles different types of batches (prefill, decode, idle), which are then synchronized before and after processing through the Mixture-of-Experts (MoE) layer.
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
135
Shi Shuai's avatar
Improve: Token-In Token-Out Usage for RLHF (#2843)  
Shi Shuai committed
136
137
138 
<p align="center">
  <img src="https://lmsys.org/images/blog/sglang_v0_4/dp_attention.svg" alt="Data Parallelism Attention for DeepSeek Series Models">
</p>
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
139
Didier Durand's avatar
fix deepseek_v3 typo (#3497)  
Didier Durand committed
140 
**Usage**: This optimization is aimed at improving throughput and should be used for scenarios with high QPS (Queries Per Second). Data Parallelism Attention optimization can be enabled by `--enable-dp-attention` for DeepSeek Series Models.
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
141
Chayenne's avatar
Docs: Add Performance Demonstaration for DPA (#3005)  
Chayenne committed
142
143
144
145 
<p align="center">
  <img src="https://lmsys.org/images/blog/sglang_v0_4/deepseek_coder_v2.svg" alt="Data Parallelism Attention Performance Comparison">
</p>
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
146
147 
**Reference**: Check [Blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models).
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
148 
### Multi Node Tensor Parallelism
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
149
Xiaotong Jiang's avatar
[Doc]: Deepseek reference docs (#2787)  
Xiaotong Jiang committed
150
151
152 
**Description**: For users with limited memory on a single node, SGLang supports serving DeepSeek Series Models, including DeepSeek V3, across multiple nodes using tensor parallelism. This approach partitions the model parameters across multiple GPUs or nodes to handle models that are too large for one node's memory.

**Usage**: Check [here](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-2-h208) for usage examples.
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
153
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
154 
### Block-wise FP8
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
155
156
157
158 

**Description**: SGLang implements block-wise FP8 quantization with two key optimizations:

- **Activation**: E4M3 format using per-token-per-128-channel sub-vector scales with online casting.
Chayenne's avatar
Docs/CI: Enable Fake Finish for Docs Only PR (#3350)  
Chayenne committed
159
Chayenne's avatar
Doc: add block-wise FP8 in dpsk model reference (#2830)  
Chayenne committed
160
161
162 
- **Weight**: Per-128x128-block quantization for better numerical stability.

**Usage**: turn on by default for DeepSeek V3 models.
Shenggui Li's avatar
[improve] made timeout configurable (#3803)  
Shenggui Li committed
163
164
165 

## FAQ
Shenggui Li's avatar
[doc] added quantization doc for dpsk (#3843)  
Shenggui Li committed
166 
1. **Question**: What should I do if model loading takes too long and NCCL timeout occurs?
Shenggui Li's avatar
[improve] made timeout configurable (#3803)  
Shenggui Li committed
167
Shenggui Li's avatar
[doc] added quantization doc for dpsk (#3843)  
Shenggui Li committed
168 
    **Answer**: You can try to add `--dist-timeout 3600` when launching the model, this allows for 1-hour timeout.