reasoning_outputs.md 12.1 KB
Newer Older
1
# Reasoning Outputs
2
3
4

vLLM offers support for reasoning models like [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1), which are designed to generate outputs containing both reasoning steps and final conclusions.

5
Reasoning models return an additional `reasoning_content` field in their outputs, which contains the reasoning steps that led to the final conclusion. This field is not present in the outputs of other models.
6
7
8
9
10

## Supported Models

vLLM currently supports the following reasoning models:

11
12
| Model Series | Parser Name | Structured Output Support | Tool Calling |
|--------------|-------------|------------------|-------------|
13
| [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) | `deepseek_r1` | `json`, `regex` | ❌ |
14
| [DeepSeek-V3.1](https://huggingface.co/collections/deepseek-ai/deepseek-v31-68a491bed32bd77e7fca048f) | `deepseek_v3` | `json`, `regex` | ❌ |
15
16
| [ERNIE-4.5-VL series](https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT) | `ernie45` | `json`, `regex` | ❌ |
| [ERNIE-4.5-21B-A3B-Thinking](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking) | `ernie45` | `json`, `regex` | ✅ |
17
| [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
18
| [IBM Granite 3.2 language models](https://huggingface.co/collections/ibm-granite/granite-32-language-models-67b3bc8c13508f6d064cff9a) | `granite` | ❌ | ❌ |
19
20
21
| [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
| [Hunyuan A13B series](https://huggingface.co/collections/tencent/hunyuan-a13b-685ec38e5b46321e3ea7c4be) | `hunyuan_a13b` | `json`, `regex` | ✅ |
| [GLM-4.5 series](https://huggingface.co/collections/zai-org/glm-45-687c621d34bda8c9e4bf503b) | `glm45` | `json`, `regex` | ✅ |
22

23
!!! note
24
    IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
25
    The reasoning feature for the Qwen3 series is enabled by default. To disable it, you must pass `enable_thinking=False` in your `chat_template_kwargs`.
26
    DeepSeek-V3.1 tool calling is supported in non-thinking mode.
27
28
29

## Quickstart

30
To use reasoning models, you need to specify the `--reasoning-parser` flags when making a request to the chat completion endpoint. The `--reasoning-parser` flag specifies the reasoning parser to use for extracting reasoning content from the model output.
31
32

```bash
Reid's avatar
Reid committed
33
34
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --reasoning-parser deepseek_r1
35
36
37
38
```

Next, make a request to the model that should return the reasoning content in the response.

39
??? code
40

41
42
    ```python
    from openai import OpenAI
43

44
45
46
    # Modify OpenAI's API key and API base to use vLLM's API server.
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
47

48
49
50
51
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
52

53
54
    models = client.models.list()
    model = models.data[0].id
55

56
57
58
59
60
61
    # Round 1
    messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
    # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
    # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
    # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
    response = client.chat.completions.create(model=model, messages=messages)
62

63
64
65
66
67
68
    reasoning_content = response.choices[0].message.reasoning_content
    content = response.choices[0].message.content

    print("reasoning_content:", reasoning_content)
    print("content:", content)
    ```
69
70
71
72
73
74
75

The `reasoning_content` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.

## Streaming chat completions

Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).

76
??? console "Json"
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

    ```json
    {
        "id": "chatcmpl-123",
        "object": "chat.completion.chunk",
        "created": 1694268190,
        "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
        "system_fingerprint": "fp_44709d6fcb",
        "choices": [
            {
                "index": 0,
                "delta": {
                    "role": "assistant",
                    "reasoning_content": "is",
                },
                "logprobs": null,
                "finish_reason": null
            }
        ]
    }
    ```
98

99
OpenAI Python client library does not officially support `reasoning_content` attribute for streaming output. But the client supports extra attributes in the response. You can use `hasattr` to check if the `reasoning_content` attribute is present in the response. For example:
100

101
??? code
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

    ```python
    from openai import OpenAI

    # Modify OpenAI's API key and API base to use vLLM's API server.
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"

    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    models = client.models.list()
    model = models.data[0].id

    messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
    # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
    # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
    # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
122
123
124
125
126
    stream = client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
    )
127
128
129
130
131
132

    print("client: Start streaming chat completions...")
    printed_reasoning_content = False
    printed_content = False

    for chunk in stream:
133
134
135
136
137
138
        # Safely extract reasoning_content and content from delta,
        # defaulting to None if attributes don't exist or are empty strings
        reasoning_content = (
            getattr(chunk.choices[0].delta, "reasoning_content", None) or None
        )
        content = getattr(chunk.choices[0].delta, "content", None) or None
139
140
141
142
143
144
145
146
147
148
149
150
151

        if reasoning_content is not None:
            if not printed_reasoning_content:
                printed_reasoning_content = True
                print("reasoning_content:", end="", flush=True)
            print(reasoning_content, end="", flush=True)
        elif content is not None:
            if not printed_content:
                printed_content = True
                print("\ncontent:", end="", flush=True)
            # Extract and print the content
            print(content, end="", flush=True)
    ```
152

153
Remember to check whether the `reasoning_content` exists in the response before accessing it. You could check out the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
154

155
156
157
158
## Tool Calling

The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning_content`.

159
??? code
160
161
162
163
164
165

    ```python
    from openai import OpenAI

    client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location", "unit"],
                }
            },
181
        }
182
    ]
183

184
185
186
187
    response = client.chat.completions.create(
        model=client.models.list().data[0].id,
        messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
        tools=tools,
188
        tool_choice="auto",
189
    )
190

191
192
    print(response)
    tool_call = response.choices[0].message.tool_calls[0].function
193

194
195
196
197
    print(f"reasoning_content: {response.choices[0].message.reasoning_content}")
    print(f"Function called: {tool_call.name}")
    print(f"Arguments: {tool_call.arguments}")
    ```
198

199
For more examples, please refer to <gh-file:examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py>.
200

201
202
203
## Limitations

- The reasoning content is only available for online serving's chat completion endpoint (`/v1/chat/completions`).
204
205
206

## How to support a new reasoning model

Reid's avatar
Reid committed
207
You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
208

209
??? code
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233

    ```python
    # import the required packages

    from vllm.reasoning import ReasoningParser, ReasoningParserManager
    from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
                                                DeltaMessage)

    # define a reasoning parser and register it to vllm
    # the name list in register_module can be used
    # in --reasoning-parser.
    @ReasoningParserManager.register_module(["example"])
    class ExampleParser(ReasoningParser):
        def __init__(self, tokenizer: AnyTokenizer):
            super().__init__(tokenizer)

        def extract_reasoning_content_streaming(
            self,
            previous_text: str,
            current_text: str,
            delta_text: str,
            previous_token_ids: Sequence[int],
            current_token_ids: Sequence[int],
            delta_token_ids: Sequence[int],
234
        ) -> DeltaMessage | None:
235
236
237
238
239
240
241
242
243
            """
            Instance method that should be implemented for extracting reasoning
            from an incomplete response; for use when handling reasoning calls and
            streaming. Has to be an instance method because  it requires state -
            the current tokens/diffs, but also the information about what has
            previously been parsed and extracted (see constructor)
            """

        def extract_reasoning_content(
244
245
246
247
            self,
            model_output: str,
            request: ChatCompletionRequest | ResponsesRequest,
        ) -> tuple[str | None, str | None]:
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
            """
            Extract reasoning content from a complete model-generated string.

            Used for non-streaming responses where we have the entire model response
            available before sending to the client.

            Parameters:
            model_output: str
                The model-generated string to extract reasoning content from.

            request: ChatCompletionRequest
                The request object that was used to generate the model_output.

            Returns:
            tuple[Optional[str], Optional[str]]
                A tuple containing the reasoning content and the content.
            """
    ```
266

267
Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
268

269
??? code
270

271
272
273
    ```python
    @dataclass
    class DeepSeekReasoner(Reasoner):
274
        """
275
276
277
278
279
280
281
282
283
284
        Reasoner for DeepSeek R series models.
        """
        start_token_id: int
        end_token_id: int

        start_token: str = "<think>"
        end_token: str = "</think>"

        @classmethod
        def from_tokenizer(cls, tokenizer: PreTrainedTokenizer) -> Reasoner:
285
286
287
288
            return cls(
                start_token_id=tokenizer.encode("<think>", add_special_tokens=False)[0],
                end_token_id=tokenizer.encode("</think>", add_special_tokens=False)[0],
            )
289
290
291
292
293

        def is_reasoning_end(self, input_ids: list[int]) -> bool:
            return self.end_token_id in input_ids
        ...
    ```
294

295
The structured output engine like [xgrammar](https://github.com/mlc-ai/xgrammar) will use `end_token_id` to check if the reasoning content is present in the model output and skip the structured output if it is the case.
296

297
Finally, you can enable reasoning for the model by using the `--reasoning-parser` flags.
298
299

```bash
300
vllm serve <model_tag> --reasoning-parser example
301
```