reasoning_outputs.md 14.2 KB
Newer Older
1
# Reasoning Outputs
2
3
4

vLLM offers support for reasoning models like [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1), which are designed to generate outputs containing both reasoning steps and final conclusions.

5
6
7
8
Reasoning models return an additional `reasoning` field in their outputs, which contains the reasoning steps that led to the final conclusion. This field is not present in the outputs of other models.

!!! warning
    `reasoning` used to be called `reasoning_content`. For now, `reasoning_content` will continue to work. However, we encourage you to migrate to `reasoning` in case `reasoning_content` is removed in future.
9
10
11
12
13

## Supported Models

vLLM currently supports the following reasoning models:

14
| Model Series | Parser Name | Structured Output Support | Tool Calling |
15
| ------------ | ----------- | ---------------- | ----------- |
16
| [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) | `deepseek_r1` | `json`, `regex` | ❌ |
17
| [DeepSeek-V3.1](https://huggingface.co/collections/deepseek-ai/deepseek-v31-68a491bed32bd77e7fca048f) | `deepseek_v3` | `json`, `regex` | ❌ |
18
19
| [ERNIE-4.5-VL series](https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT) | `ernie45` | `json`, `regex` | ❌ |
| [ERNIE-4.5-21B-A3B-Thinking](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking) | `ernie45` | `json`, `regex` | ✅ |
20
| [GLM-4.5 series](https://huggingface.co/collections/zai-org/glm-45-687c621d34bda8c9e4bf503b) | `glm45` | `json`, `regex` | ✅ |
21
| [Holo2 series](https://huggingface.co/collections/Hcompany/holo2) | `holo2` | `json`, `regex` | ✅ |
22
| [Hunyuan A13B series](https://huggingface.co/collections/tencent/hunyuan-a13b-685ec38e5b46321e3ea7c4be) | `hunyuan_a13b` | `json`, `regex` | ✅ |
23
| [IBM Granite 3.2 language models](https://huggingface.co/collections/ibm-granite/granite-32-language-models-67b3bc8c13508f6d064cff9a) | `granite` | ❌ | ❌ |
24
| [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
25
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
26
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
27

28
!!! note
29
    IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
30
    The reasoning feature for the Qwen3 series is enabled by default. To disable it, you must pass `enable_thinking=False` in your `chat_template_kwargs`.
31
    DeepSeek-V3.1 tool calling is supported in non-thinking mode.
32
    Holo2 reasoning is enabled by default. To disable it, you must also pass `thinking=False` in your `chat_template_kwargs`.
33
34
35

## Quickstart

36
To use reasoning models, you need to specify the `--reasoning-parser` flags when making a request to the chat completion endpoint. The `--reasoning-parser` flag specifies the reasoning parser to use for extracting reasoning content from the model output.
37
38

```bash
Reid's avatar
Reid committed
39
40
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --reasoning-parser deepseek_r1
41
42
43
44
```

Next, make a request to the model that should return the reasoning content in the response.

45
??? code
46

47
48
    ```python
    from openai import OpenAI
49

50
51
52
    # Modify OpenAI's API key and API base to use vLLM's API server.
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
53

54
55
56
57
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
58

59
60
    models = client.models.list()
    model = models.data[0].id
61

62
63
64
65
66
67
    # Round 1
    messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
    # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
    # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
    # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
    response = client.chat.completions.create(model=model, messages=messages)
68

69
    reasoning = response.choices[0].message.reasoning
70
71
    content = response.choices[0].message.content

72
    print("reasoning:", reasoning)
73
74
    print("content:", content)
    ```
75

76
The `reasoning` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.
77
78
79

## Streaming chat completions

80
Streaming chat completions are also supported for reasoning models. The `reasoning` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
81

82
??? console "Json"
83
84
85
86
87
88
89
90
91
92
93
94
95

    ```json
    {
        "id": "chatcmpl-123",
        "object": "chat.completion.chunk",
        "created": 1694268190,
        "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
        "system_fingerprint": "fp_44709d6fcb",
        "choices": [
            {
                "index": 0,
                "delta": {
                    "role": "assistant",
96
                    "reasoning": "is",
97
98
99
100
101
102
103
                },
                "logprobs": null,
                "finish_reason": null
            }
        ]
    }
    ```
104

105
OpenAI Python client library does not officially support `reasoning` attribute for streaming output. But the client supports extra attributes in the response. You can use `hasattr` to check if the `reasoning` attribute is present in the response. For example:
106

107
??? code
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

    ```python
    from openai import OpenAI

    # Modify OpenAI's API key and API base to use vLLM's API server.
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"

    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    models = client.models.list()
    model = models.data[0].id

    messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
    # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
    # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
    # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
128
129
130
131
132
    stream = client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
    )
133
134

    print("client: Start streaming chat completions...")
135
    printed_reasoning = False
136
137
138
    printed_content = False

    for chunk in stream:
139
        # Safely extract reasoning and content from delta,
140
        # defaulting to None if attributes don't exist or are empty strings
141
142
        reasoning = (
            getattr(chunk.choices[0].delta, "reasoning", None) or None
143
144
        )
        content = getattr(chunk.choices[0].delta, "content", None) or None
145

146
147
148
149
150
        if reasoning is not None:
            if not printed_reasoning:
                printed_reasoning = True
                print("reasoning:", end="", flush=True)
            print(reasoning, end="", flush=True)
151
152
153
154
155
156
157
        elif content is not None:
            if not printed_content:
                printed_content = True
                print("\ncontent:", end="", flush=True)
            # Extract and print the content
            print(content, end="", flush=True)
    ```
158

159
Remember to check whether the `reasoning` exists in the response before accessing it. You could check out the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
160

161
162
## Tool Calling

163
The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning`.
164

165
??? code
166
167
168
169
170
171

    ```python
    from openai import OpenAI

    client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location", "unit"],
                }
            },
187
        }
188
    ]
189

190
191
192
193
    response = client.chat.completions.create(
        model=client.models.list().data[0].id,
        messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
        tools=tools,
194
        tool_choice="auto",
195
    )
196

197
198
    print(response)
    tool_call = response.choices[0].message.tool_calls[0].function
199

200
    print(f"reasoning: {response.choices[0].message.reasoning}")
201
202
203
    print(f"Function called: {tool_call.name}")
    print(f"Arguments: {tool_call.arguments}")
    ```
204

205
For more examples, please refer to [examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py](../../examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py).
206

207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
## Server-Level Default Chat Template Kwargs

You can set default `chat_template_kwargs` at the server level using the `--default-chat-template-kwargs` CLI argument. This is useful for configuring reasoning behavior across all requests without requiring clients to specify it in each request.

### Disabling Thinking Mode by Default

For models like Qwen3 where thinking is enabled by default, you can disable it server-wide:

```bash
vllm serve Qwen/Qwen3-8B \
    --reasoning-parser qwen3 \
    --default-chat-template-kwargs '{"enable_thinking": false}'
```

### Enabling Thinking Mode by Default

For models like IBM Granite 3.2 or DeepSeek-V3.1 where thinking is disabled by default, you can enable it server-wide:

```bash
vllm serve ibm-granite/granite-3.2-2b-instruct \
    --reasoning-parser granite \
    --default-chat-template-kwargs '{"thinking": true}'
```

### Request-Level Override

Request-level `chat_template_kwargs` always take priority over server defaults. For example, if the server is started with `enable_thinking=false`, a client can still enable it for a specific request:

```python
response = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}}  # Overrides server default
)
```

243
244
245
## Limitations

- The reasoning content is only available for online serving's chat completion endpoint (`/v1/chat/completions`).
246
247
248

## How to support a new reasoning model

249
You can add a new `ReasoningParser` similar to [vllm/reasoning/deepseek_r1_reasoning_parser.py](../../vllm/reasoning/deepseek_r1_reasoning_parser.py).
250

251
??? code
252
253
254
255
256

    ```python
    # import the required packages

    from vllm.reasoning import ReasoningParser, ReasoningParserManager
257
258
    from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
    from vllm.entrypoints.openai.engine.protocol import DeltaMessage
259
260
261
262
263

    # define a reasoning parser and register it to vllm
    # the name list in register_module can be used
    # in --reasoning-parser.
    class ExampleParser(ReasoningParser):
264
        def __init__(self, tokenizer: TokenizerLike):
265
266
            super().__init__(tokenizer)

267
        def extract_reasoning_streaming(
268
269
270
271
272
273
274
            self,
            previous_text: str,
            current_text: str,
            delta_text: str,
            previous_token_ids: Sequence[int],
            current_token_ids: Sequence[int],
            delta_token_ids: Sequence[int],
275
        ) -> DeltaMessage | None:
276
277
278
279
280
281
282
283
            """
            Instance method that should be implemented for extracting reasoning
            from an incomplete response; for use when handling reasoning calls and
            streaming. Has to be an instance method because  it requires state -
            the current tokens/diffs, but also the information about what has
            previously been parsed and extracted (see constructor)
            """

284
        def extract_reasoning(
285
286
287
288
            self,
            model_output: str,
            request: ChatCompletionRequest | ResponsesRequest,
        ) -> tuple[str | None, str | None]:
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
            """
            Extract reasoning content from a complete model-generated string.

            Used for non-streaming responses where we have the entire model response
            available before sending to the client.

            Parameters:
            model_output: str
                The model-generated string to extract reasoning content from.

            request: ChatCompletionRequest
                The request object that was used to generate the model_output.

            Returns:
            tuple[Optional[str], Optional[str]]
                A tuple containing the reasoning content and the content.
            """
306
307
308
309
310
311
    # Register the reasoning parser
    ReasoningParserManager.register_lazy_module(
        name="example",
        module_path="vllm.reasoning.example_reasoning_parser",
        class_name="ExampleParser",
    )
312
    ```
313

314
Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in [vllm/reasoning/deepseek_r1_reasoning_parser.py](../../vllm/reasoning/deepseek_r1_reasoning_parser.py).
315

316
??? code
317

318
319
320
    ```python
    @dataclass
    class DeepSeekReasoner(Reasoner):
321
        """
322
323
324
325
326
327
328
329
330
331
        Reasoner for DeepSeek R series models.
        """
        start_token_id: int
        end_token_id: int

        start_token: str = "<think>"
        end_token: str = "</think>"

        @classmethod
        def from_tokenizer(cls, tokenizer: PreTrainedTokenizer) -> Reasoner:
332
333
334
335
            return cls(
                start_token_id=tokenizer.encode("<think>", add_special_tokens=False)[0],
                end_token_id=tokenizer.encode("</think>", add_special_tokens=False)[0],
            )
336
337
338

        def is_reasoning_end(self, input_ids: list[int]) -> bool:
            return self.end_token_id in input_ids
339
340
341

        def is_reasoning_end_streaming(self, input_ids: list[int], delta_ids: list[int]) -> bool:
            return self.end_token_id in delta_token_ids
342
343
        ...
    ```
344

345
The structured output engine like [xgrammar](https://github.com/mlc-ai/xgrammar) will use `end_token_id` to check if the reasoning content is present in the model output and skip the structured output if it is the case.
346

347
Finally, you can enable reasoning for the model by using the `--reasoning-parser` flags.
348
349

```bash
350
vllm serve <model_tag> --reasoning-parser example
351
```